Military and Veterans Disability System: Pilot Has Achieved Some Goals, but Further Planning and Monitoring Needed
Published: 2010-12-06T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Since 2007, the Departments of Defense (DOD) and Veterans Affairs (VA) have been testing a new disability evaluation system designed to integrate their separate processes and thereby expedite veterans' benefits for wounded, ill, and injured servicemembers. Having piloted the integrated disability evaluation system (IDES) at 27 military facilities, they are now planning for its expansion military-wide. Part of the National Defense Authorization Act for Fiscal Year 2008 required GAO to report on DOD and VA's implementation of policies on disability evaluations. This report examines: (1) the results of the agencies' evaluation of the IDES pilot, (2) challenges in implementing the IDES pilot to date, and (3) whether DOD and VA's plans to expand the IDES adequately address potential future challenges. GAO analyzed data from DOD and VA, conducted site visits at 10 military facilities, and interviewed DOD and VA officials.
section_title: What GAO Found
paragraphs: In their evaluation of the IDES pilot as of February 2010, DOD and VA concluded that it had improved servicemember satisfaction relative to the existing "legacy" system and met their established goal of delivering VA benefits to active duty and reserve component servicemembers within 295 and 305 days, respectively, on average. While these results are promising, average case processing times have steadily increased since the February 2010 evaluation. At 296 days for active duty servicemembers, as of August 2010, processing time for the IDES is still an improvement over the 540 days that DOD and VA estimated the legacy process takes to deliver VA benefits to members. However, the full extent of improvement of the IDES over the legacy system is unknown because (1) the 540-day estimate was based on a small, nonrepresentative sample of cases and (2) limitations in legacy case data prevent a comprehensive comparison of timeliness, as well as appeal rates. Piloting of the IDES has revealed several implementation challenges that have contributed to delays in the process, the most significant being insufficient staffing by DOD and VA. Staffing shortages were severe at a few pilot sites that experienced caseload surges. For example, at one of these sites, due to a lack of VA medical staff, it took 140 days on average to complete one of the key features of the pilot--the single exam--compared with the agencies' goal to complete this step of the process in 45 days. The single exam posed other challenges that contributed to process delays, such as exam summaries that did not contain sufficient information for VA to determine the servicemember's benefits and disagreements between DOD and VA medical staff about diagnoses for servicemembers' medical conditions. Cases with these problems were returned for further attention, adding time to the process. Pilot sites also experienced logistical challenges, such as incorporating VA staff at military facilities and housing and managing personnel going through the process. As DOD and VA prepare to expand the IDES worldwide, they have made preparations to address a number of these challenges, but these efforts have yet to be tested, and not all challenges have been addressed. To address staffing shortages and ensure timely processing, VA is developing a contract for additional medical examiners, and DOD and VA are requiring local staff to develop written contingency plans for handling surges in caseloads. However, the agencies lack strategies for meeting some key challenges, such as ensuring enough military physicians to handle anticipated workloads. They also do not have a comprehensive monitoring plan for identifying problems as they occur--such as staffing shortages and insufficiencies in medical exams--in order to take remedial actions as early as possible.
section_title: What GAO Recommends
paragraphs: GAO is making several recommendations to improve DOD and VA's planning for expansion of the new disability evaluation system, including developing a systematic monitoring process and ensuring that adequate staff is in place. DOD and VA generally concurred with GAO's recommendations and provided technical comments that GAO incorporated into the report as appropriate.

section_title: Letter
paragraphs: Over 40,000 servicemembers have been wounded in the wars in Iraq and  Afghanistan, as of October 2010. After receiving medical treatment, many  wounded servicemembers must navigate a complex disability evaluation  system that begins with the Department of Defense (DOD) determining  whether they are medically fit to continue their military service. If they are  found unfit, servicemembers continue through the system to obtain a  determination of their eligibility for military disability benefits. Once  servicemembers are discharged from the military, they may also be eligible  to receive disability benefits from the Department of Veterans Affairs  (VA), but they must first undergo an entirely separate VA disability  evaluation process. A series of articles in 2007 by The Washington Post  concerning conditions at Walter Reed Army Medical Center, and  subsequent reports from numerous high-level commissions and review  groups, highlighted problems with the DOD and VA disability evaluations  systems.  These included long delays, duplication in DOD and VA  processes, confusion among servicemembers, and distrust of systems  regarded as adversarial by servicemembers and veterans.
In response to these concerns, DOD and VA jointly designed a new  disability evaluation system that integrates DOD and VA processes, with  the goal of expediting the delivery of benefits to servicemembers. DOD  and VA began pilot testing the integrated disability evaluation system  (IDES) in November 2007 at three Washington, D.C., area military  treatment facilities and, by March 2010, added 24 more facilities to the  pilot. DOD and VA are now planning to expand the piloted system to 28  additional facilities, as a first step toward replacing the military’s  existing—or “legacy”—disability evaluation system with the IDES  worldwide.
In January 2008, Congress enacted the National Defense Authorization Act  for Fiscal Year 2008 (NDAA 2008) requiring DOD and VA, to the extent  feasible, to jointly develop and implement a comprehensive policy on  improvements to the care, management, and transition of recovering  servicemembers, including improvements to the agencies’ disability  evaluation systems. The NDAA 2008 also required GAO to report on the  progress DOD and VA have made in developing and implementing the  comprehensive policy. In agreement with cognizant congressional staff,  we reviewed DOD and VA’s progress in implementing policies related to  their disability evaluation systems, focusing on the agencies’ joint pilot of  the IDES. Specifically, we examined: (1) the results of DOD and VA’s  evaluation of the pilot, (2) challenges in implementing the piloted system  to date, and (3) DOD and VA plans to expand the piloted system and  whether those plans adequately address potential challenges.
To examine DOD and VA’s evaluation of the IDES pilot, we identified the  goals that DOD and VA expected the pilot to achieve and reviewed their  assessment of whether those goals were met. As part of this work, we  assessed the reliability of two types of data that DOD and VA planned to  use as the basis of their pilot evaluation—case data from both pilot and  legacy disability evaluation systems, as well as data from surveys DOD  conducted to gauge servicemember satisfaction. We obtained the case  data and survey data as of early 2010, the same cutoff dates that DOD and  VA used for their pilot evaluation. To identify challenges in implementing  the piloted system to date, we visited 10 of the 27 military treatment  facilities participating in the pilot. We selected these 10 facilities to obtain  perspectives from sites in different military services and geographical  regions and with varying caseloads and organizational structures. For all  of the research objectives, we conducted interviews with key officials  involved in the pilot at DOD, VA, and each of the military services.  Furthermore, we analyzed pilot case data and reviewed reports, guidance,  plans, and other documents. We also reviewed relevant federal laws and  regulations. We conducted this performance audit from November 2009 to  December 2010, in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
The NDAA 2008 also requires us to certify whether we had timely access  to sufficient information to make informed judgments on the matters  covered by our report. We were provided sufficient information in a  timely manner to make informed judgments on the audit objectives  covered in this report.
subsections: section_title: Background
subsections: section_title: The DOD Legacy Disability Evaluation System
paragraphs: The military’s legacy disability evaluation process begins at a military  treatment facility when a physician identifies a condition that may  interfere with a servicemember’s ability to perform his or her duties. On  the basis of medical examinations and the servicemember’s medical  records, a medical evaluation board (MEB) identifies and documents any  conditions that may limit a servicemember’s ability to serve in the military.
The servicemember’s case is then evaluated by a physical evaluation board  (PEB) to make a determination of fitness or unfitness for duty. Each of the  services conducts this process for its servicemembers. The Army has three  PEBs, which are located at Fort Sam Houston, Texas; Walter Reed Army  Medical Center in Washington, D.C.; and Fort Lewis, Washington. The  Navy and Air Force each have one PEB: the Navy’s is located at the  Washington Navy Yard in Washington, D.C., and the Air Force’s is located  in San Antonio, Texas. The PEB process begins with an “informal” PEB— an administrative review of the case file by PEB adjudicators without the  presence of the servicemember. If the servicemember is found to be unfit  due to medical conditions incurred in the line of duty, the informal PEB  assigns the servicemember a combined percentage rating for those unfit  conditions, and the servicemember is discharged from duty. Disability  ratings range from 0 (least severe) to 100 percent (most severe) in  increments of 10 percent. Depending on the overall disability rating and  number of years of active duty or equivalent service, the servicemember  found unfit with compensable conditions is entitled to either monthly  disability retirement benefits or lump sum disability severance pay.
Servicemembers have opportunities to appeal the results of their disability  evaluations. If servicemembers are dissatisfied with the informal PEB’s  decisions, they may request a hearing with a “formal” PEB. If they then  disagree with the formal PEB’s findings, they can, under certain  conditions, appeal to the reviewing authority of the PEB.
As servicemembers navigate DOD’s disability evaluation system, they  interface with staff who play key roles in supporting them through the  process. Military physicians involved in the MEB process play a  fundamental role because they are responsible for documenting in the  disability evaluation case file the medical conditions that may limit a  servicemember’s ability to serve in the military. To prepare this  documentation, military physicians may require that servicemembers  obtain additional medical evidence from specialty physicians, such as a  psychiatrist. Throughout the MEB and PEB processes, board liaisons serve  a key role by explaining the process to servicemembers and constructing  the case files. The liaisons inform servicemembers of board results and of  deadlines at key decision points in the process. The military also provides  legal counsel to advise and represent servicemembers going through the  disability evaluation process, although servicemembers may retain their  own representative at their own expense.
section_title: The VA Disability Claims Process
paragraphs: In addition to receiving disability benefits from DOD, veterans with  service-connected disabilities may receive compensation from VA for lost  earnings capacity. In contrast to DOD’s disability evaluation system, which  evaluates only medical conditions affecting servicemembers’ fitness for  duty, VA evaluates all medical conditions claimed by the veteran, whether  or not they were previously evaluated by the military services’ medical  evaluation process. Although a servicemember may file a VA claim while  still in the military, he or she can only obtain disability compensation from  VA as a veteran.
VA’s disability compensation claims process starts when a veteran submits  a claim to VA’s Veterans Benefits Administration (VBA). The claim lists the  medical conditions that the veteran believes are service-connected. For  each claimed condition, VA must determine if credible evidence is  available to support the veteran’s contention of service connection. A  service representative assists the veteran in gathering the relevant  evidence to evaluate the claim, which may include the veteran’s military  service records and treatment records from VA medical facilities and  private medical service providers. Also, if necessary for reaching a  decision on a claim, VBA arranges for the veteran to receive a medical  examination conducted by clinicians (including physicians, nurse  practitioners, or physician assistants) certified to perform the exams  under VA’s Compensation and Pension program. Once a claim has all of  the necessary evidence, a VA rating specialist evaluates the claim and  determines whether the claimant is eligible for benefits. If so, the rating  specialist assigns a percentage rating. If VA finds that a veteran has one or  more service-connected disabilities with a combined rating of at least 10  percent, the agency will pay monthly compensation. The veteran can claim  additional benefits over time, for example, if a service-connected disability  worsens or surfaces at a later point in time.
section_title: The Integrated Disability Evaluation System
paragraphs: In November 2007, DOD and VA began piloting the IDES, a joint disability  evaluation system to eliminate duplication in their separate systems and to  expedite receipt of VA benefits for wounded, ill, and injured  servicemembers. The IDES merges DOD and VA processes, so that  servicemembers begin their VA disability claim while they undergo their  DOD disability evaluation, rather than sequentially, making it possible for  them to receive VA disability benefits shortly after leaving military service.  Specifically, the IDES    merges DOD and VA’s separate exam processes into a single exam process  conducted to VA standards. This single exam—which may involve more  than one medical examination (for example, by different specialists)—in  conjunction with the servicemembers’ medical records, is used by military  service PEBs to make a determination of servicemembers’ fitness for  continued military service, and by VA as evidence of service-connected  disabilities. The single exam may be performed by medical staff working  for either VA, DOD, or a private provider contracted with either agency.    consolidates DOD and VA’s separate rating phases into one VA rating  phase. If the informal PEB has determined that a servicemember is unfit  for duty, VA rating specialists prepare two ratings—one for the conditions  that DOD determined made a servicemember unfit for duty, which DOD  uses to provide military disability benefits, and the other for all service- connected disabilities, which VA uses to determine VA disability benefits.  Ratings for the IDES are prepared by rating specialists at VA’s Baltimore  and Seattle regional offices.    provides VA case managers to perform outreach and nonclinical case  management and explain VA results and processes to servicemembers.
By consolidating DOD and VA’s separate medical exams and ratings, the  IDES eliminates several steps from the existing “legacy” systems (see fig. 1).
In designing the IDES, DOD and VA established goals to provide VA  benefits to active duty servicemembers within 295 days of being referred  into the system, and to reserve component members within 305 days. In  establishing the 295- and 305-day goals, they also established timeliness  goals for the specific steps of the IDES process (see fig. 2).
DOD and VA first piloted the IDES at 3 Washington, D.C., area military  treatment facilities, beginning in November 2007 (see table 1). They added  18 military facilities to the pilot in fiscal year 2009 and 6 in fiscal year 2010.  DOD and VA stated that expansion to additional sites was intended to  assess the IDES system in a variety of geographic areas and to test the  agencies’ capacity to handle additional caseload. According to DOD, the 27  pilot sites represented almost half of the servicemembers in the military  services’ disability evaluation systems.
section_title: Pilot Evaluation Results Are Promising, but the Degree of Improvement Achieved Is Unknown
subsections: section_title: DOD and VA’s Evaluation Shows That the Pilot Is Achieving Some of Its Goals
paragraphs: In their planning documents for the IDES pilot, DOD and VA stated that  they were basing their evaluation of the effectiveness of the IDES pilot on  whether it has achieved three key goals relative to the legacy process:  increased servicemember satisfaction, improved case-processing time, and  a reduction in servicemember appeal rates. In addition, they also  examined IDES program costs. To determine whether they have achieved  their goals, the agencies surveyed servicemembers in the IDES pilot and  legacy systems and are using a data system—called the Veterans Tracking  Application (VTA)—that enables them to track case processing time and  appeals. They have been monitoring their progress on these goals through  weekly reports.
In August 2010, DOD and VA officials issued an interim report to Congress  summarizing their evaluation results to date. In this report, the agencies  concluded that servicemembers who went through the IDES pilot were  more satisfied than those who went through the legacy system, and that  the IDES process met the agencies’ goals of delivering VA benefits to  active duty servicemembers within 295 days and to reserve component  servicemembers within 305 days. Specifically, they reported that, as of  February 2010, the IDES process took an average of 274 days to complete  for active duty servicemembers and 281 days for reserve component  members who, according to the interim report, comprise 15 percent of  IDES participants. Furthermore, they concluded that the IDES pilot has  achieved a faster processing time than the legacy system, which they  estimated to be 540 days.
While overall results were promising, data presented in the report had  some limitations, and the report itself did not include certain analyses. For  example, DOD officials told us that the 540-day estimate for the legacy  process was based upon a review of a small and nonrepresentative sample  of legacy cases during the agencies’ “table top” planning exercise in  August 2007. In addition, although DOD officials told us that they  planned to compare average processing times of pilot cases with a broader sample of legacy cases, and to determine whether fewer servicemembers  are appealing the findings of informal PEBs and formal PEBs in the pilot  compared with the legacy, the interim report did not include these  comparisons. In addition, in their planning documents for the IDES pilot,  DOD and VA indicated that they were establishing a goal to deliver VA  benefits to 80 percent of members in the IDES pilot within the 295- and  305-day time frames. However, their interim report did not discuss  whether this goal was met.
Our review of DOD and VA’s data and weekly reports generally confirm  DOD and VA’s findings, as of early 2010. However, while the agencies have  largely met their overall goal to increase servicemember satisfaction and  met their timeliness goal as of February 2010, since that time, case  processing times have been steadily increasing as the caseload has  increased. In addition, not all of the service branches are achieving the  same results.
Servicemember satisfaction: Our review of the survey data that DOD used  for the interim report (as of February 2010), as well as a recent weekly  report, indicate that, on average, servicemembers in the IDES process  have had higher satisfaction levels than those who went through the  legacy process. In addition, a higher percentage of servicemembers who  went through the IDES process felt that the process was fair compared  with those who went through the legacy system. However,  servicemembers in the Air Force who went through the IDES pilot  indicated less satisfaction with the process than those who went through  the legacy system, though Air Force members represented a small  proportion of pilot cases—about 7 percent of those enrolled in the pilot.  We reviewed the agencies’ survey methodology and generally found their  survey design and conclusions to be sound (see app. I for further  information on our review).
Average case processing times: The agencies have been meeting their 295- day and 305-day timeliness goals for much of the past 2 years, but more  recent weekly reports indicate case processing time has been increasing  and that they are now missing their goal for active duty members. As of  August 29, 2010, the agencies missed the goal for active duty  servicemembers by 1 day, while still meeting the 305-day goal for reserve  component members by 7 days. Processing times have increased as  caseload has increased, from about 5,750 active cases in February to about  9,650 cases in August 2010. We reviewed the reliability of the VTA data  upon which the agencies based their analyses and generally found these  data to be sufficiently reliable for purposes of these analyses.
The increases in overall case processing time and caseloads mirror the  trends at individual sites. For each pilot site, case processing times have  generally increased as workloads have increased. For example, figure 3  shows the case processing times 1 year or more after implementation and  in August 2010 for the first seven pilot sites.
Of the four military services, only the Army and Navy were achieving the  295- and 305-day goals on average, as of February 2010, and only the Army  was achieving these goals as of August 2010. Because the Army comprises  a large proportion of cases (approximately 60 percent of IDES pilot cases  that have completed the whole process), it has lowered the overall average  processing time to near or below the established goals. Figure 4 shows the  average case processing times for active duty, by service, as of August  2010. (See app. II for reserve component.)
As of February 2010, the agencies also had not met the goal of processing  80 percent of all pilot cases within targeted time frames. Specifically,  about 60 percent of active duty pilot cases have been completed within  295 days, according to our analysis of the agencies’ case data intended for  their interim report. Further, none of the four military services have  achieved this goal, although the Army has had the highest rate of cases (66  percent) meeting the goal, while only 42 percent of Air Force cases were  processed within the time frame (see fig. 5 for active duty and app. II for  reserve component).
section_title: Extent of Improvement Over the Legacy System Is Unknown Due to Gaps in Legacy Data
paragraphs: DOD and VA planned to compare the case processing times of  servicemembers in the IDES pilot and servicemembers who, between  fiscal years 2005 and 2009, were enrolled in the legacy system at pilot sites  prior to pilot implementation, but significant gaps in the legacy case data  preclude reliable comparisons. DOD compiled the legacy case data from  each of the military services and the VA, but the military services each had  slightly different disability evaluation processes, used different data  systems, and did not track the same information. As a result, information  needed to conduct a comparison is not available for all services. For  example, the Navy, Marine Corps, and Air Force legacy data do not have  information on when the servicemember was referred into the disability  evaluation system and, as a result, case-processing time for the legacy  system DOD-wide cannot be known. DOD officials said they planned to  estimate legacy case processing time by approximating the dates that  servicemembers in the Navy, Marine Corps, and Air Force were referred  into the disability evaluation process, but their methodology was based on  a limited number of Army cases (see app. I for further information). In  addition, for legacy cases across all military services, VA was not able to  provide data on the date VA benefits were delivered, so total case  processing time from referral to delivery of VA benefits cannot be  measured. However, while legacy case data are not sufficiently reliable for  comparison with the IDES overall, the Army’s legacy data appear to be  reliable on some key processing dates, making some limited comparisons  possible. Our analysis of Army legacy data suggests that, under the legacy  process, active duty Army cases took 369 days to complete the DOD  legacy process and reach the VA rating phase—though this figure does not  include time to complete the VA rating and provide the benefits to  servicemembers—compared with 266 days to deliver VA benefits to  servicemembers under the pilot, according to the agencies’ August weekly  report. However, Army comparisons cannot be generalized to the other  services.
The agencies also planned to compare servicemembers’ appeal rates in the  pilot and legacy systems, but similar gaps in the legacy data preclude a  comparison DOD-wide. For example, the legacy data that DOD compiled  did not contain data on appeals of informal PEB decisions to the formal  PEB in the Navy and Marines, and consequently the rate of appeals across  the military in the legacy system is unknown. While the Army’s appeals  data appear to be more reliable, potentially making some limited  comparisons possible, the agencies’ method for comparing pilot appeals  with legacy has limitations. DOD officials told us they are planning to  compare the proportion of informal PEB decisions that were appealed to a  formal PEB hearing in the pilot and legacy systems. However, this will not  take into account that, under the legacy system, a servicemember could  appeal the informal PEB’s decision for two reasons—because they were  dissatisfied with the fitness decision or the disability rating the PEB  assigned, while in the IDES, they can only appeal the informal PEB  decision to a formal PEB if they are dissatisfied with the fitness decision.  Under the IDES, servicemembers who disagree with the disability rating  can appeal to VA for a rating reconsideration. By not including appeals to  VA for rating reconsiderations, the agencies may overestimate the  decrease in appeals in the IDES pilot. For example, our analysis of data as  of early 2010 for the Army indicates that Army members in the pilot  appealed 7.5 percent of informal PEB decisions. However, when appeals  to VA are factored in, 13 percent of Army members in the pilot filed an  appeal, which is the same proportion as in the legacy system (see fig. 6).
In addition to evaluating the three goals, DOD and VA initially planned a  cost-benefit analysis of the IDES program but have only completed an  analysis of costs. According to data provided to us in August 2010, DOD  projects that costs directly associated with implementing the IDES will be  $63 million greater per year when compared with the legacy system, after  full expansion of the IDES. In October 2010, VA reported to us total IDES  cost estimates of approximately $50 million for fiscal year 2011—about  $33 million for VBA, which provides VA case managers and rating staff to  the IDES, and $17 million for the Veterans Health Administration (VHA),  which provides medical staff to perform the single exams. These  analyses did not quantify the value of potential benefits created by the  pilot, for example time savings from DOD physicians no longer needing to  perform disability examinations, which allows them to perform other  duties.
section_title: Pilot Sites Experienced Several Challenges
paragraphs: As DOD and VA tested the IDES at different facilities and added caselo to the pilot, they encountered several challenges that led to delays in  certain phases of the process. Among these were insufficient staffing,  challenges in conducting the single exams, logistical challenges related  integrating VA staff, as well as housing and managing servicemembers  going through the IDES. DOD and VA were able to address some, but not  all, of these challenges as they arose.
subsections: section_title: DOD and VA Did Not Sufficiently Staff Many Key Positions in the IDES Pilot
paragraphs: DOD and VA have not provided sufficient numbers of staff in many of th IDES locations, affecting their ability to complete certain phases of the  IDES process within the goals they established. Officials at most of the 1 pilot sites we visited said they have experienced staffing shortages to a t  least some extent, with a few sites—Fort Ca particular—experiencing severe shortages.  rson and Fort Stewart, in  VA or contract examiners: At three pilot sites we visited—Fort Carson,  Fort Polk, and Fort Stewart—local officials said that a lack of VA or VA  contractor staff who could perform the required single medical exams led  to bottlenecks in the process. For example, as of August 2010, exams at  Fort Carson have taken an average of 140 days to complete for active duty servicemembers, according to the agencies’ data, far from achieving their  goal to complete single medical exams within 45 days (see fig. 7; s ee also  app. II for processing times for reserve component members). Across all  pilot sites, exams have taken 68 days to complete for active duty  servicemembers, on average, with 8 of the 27 pilot sites meeting the 45 goal. examiners. For instance, Fort Carson’s IDES process was particularly hampered by a lack of mental health specialists; in contrast, VA officials  serving the Fort Polk pilot site said perform specialty medical exams but did not have enough examiners to  complete general medical exams.
The 8 pilot sites that met the 45-day goal for completing single exams include 2 Air Force  sites, 5 Army sites, and 1 Navy site that met the 45-day goal for servicemembers in both the  Navy and Marine Corps. One additional site (Camp Pendleton) met the 45-day goal for Navy  members but did not meet it for Marine Corps members.
The Navy PEB determines fitness decisions for servicemembers in the Marine Corps.
VA rating staff: Officials at the Baltimore rating office—one of the two VA  offices that conduct disability ratings for the IDES pilot—expressed  significant concerns that they were understaffed, in part due to staff  turnover. DOD and VA data show that, overall, the VA rating offices are  not meeting the agencies’ goal to complete ratings within 15 days, taking  39 days on average for active duty servicemembers and 42 days for reserve  component members. We could not determine case processing times at  each individual VA rating office, since DOD and VA’s weekly monitoring  reports do not provide processing times for the rating phase by office. The  weekly reports also do not provide data on caseloads at each office.  Although the Baltimore office currently has fewer rating staff than Seattle,  VA officials said that it has prepared ratings for the majority of IDES pilot  cases, based on the way in which VA has allocated cases between the two  offices. The Baltimore office handles cases for the Air Force, Navy,  Marines, and 5 of the 15 Army pilot sites, while the Seattle office conducts  ratings for the remaining 10 Army pilot sites. VA officials said that to  address staffing shortages in Baltimore, they have assigned staff from  other VA offices to assist the Baltimore office.
VA case managers: DOD and VA have set a target for each VA case  manager to handle no more than 30 cases at a time, but two sites we  visited—Fort Carson and Fort Stewart—appeared to be far from these  targets. At Fort Carson, three VA case managers told us they were  handling about 900 cases when we visited in April 2010, for a caseload  ratio of roughly 1:300. At the time of our visit in June 2010, Fort Stewart  had over 750 active cases with two VA case managers, for a caseload ratio  of approximately 1:375. Although local officials we spoke with at both  sites told us that the numbers of VA case managers were insufficient, an  official at VA’s central office told us that VA bases staffing of case  managers on the number of new (not pending) cases each month, and the  agencies’ data indicates the average number of new cases per VA case  manager has been about 25 at each site. The VA official said that the  reason local case managers felt understaffed was likely due to other  process inefficiencies. In addition, the official told us VA can reassign staff  from other VA programs to assist case managers at IDES pilot sites as  needed. At some of the other pilot sites we visited, local officials also told  us they had concerns at times about the numbers of VA case managers  available to handle the site’s caseload, but VA was able to add staff. VA  case managers at two Air Force sites we visited—Travis and Vance Air  Force Bases—indicated that their caseloads were manageable. We were  unable to independently determine the extent to which VA is meeting its  caseload target because VA does not collect national data on actual  caseloads per case manager.
DOD board liaisons: At most of the sites we visited, local officials  expressed concerns about insufficient numbers of DOD board liaisons,  who serve as servicemembers’ DOD case managers. DOD guidance has  been inconsistent on the caseload target for DOD board liaisons. While  DOD’s operations manual for the IDES pilot sets a caseload target of at  most 30 cases per board liaison, guidance on the general disability  evaluation system sets the target at a maximum of 20 cases per liaison.  DOD and VA’s documents related to planning for IDES expansion indicate  that DOD is striving for a 1:20 caseload target in the IDES. However, 19 of  the 27 pilot sites did not meet the 1:30 caseload target, and 23 did not meet  the 1:20 target (see fig. 10).
Local DOD and VA officials attributed staffing shortages to higher than  anticipated caseloads and difficulty finding qualified staff in rural areas. At  several of the pilot sites we visited, officials said that caseloads were  higher than the initial estimates that they had based staffing levels upon.  DOD officials said that they had based caseload estimates on a 1-year  history of caseload at each site. While some sites have added staff as  caseloads increased, others, such as Fort Polk, located in central  Louisiana, have had difficulty finding qualified staff, particularly  physicians, in this rural area.
Two of the pilot sites we visited—Fort Carson and Fort Stewart—were  particularly challenged to provide staff in response to surges in caseload,  which occurred when Army units were preparing to deploy to combat  zones. Through the Army’s predeployment medical assessment process,  large numbers of servicemembers were determined to be unable to deploy  due to a medical condition and were referred to the IDES within a short  period of time, overwhelming the staff. These two sites were unable to  quickly increase staffing levels, particularly clinicians performing the  single exam. The VA medical center conducting the single medical exams  for Fort Carson experienced turnover among its examiners at the same  time that the caseload surged, while at Fort Stewart, the contractor  performing the single medical exams had difficulties finding qualified  physicians in a rural area of Georgia. To address caseload surges,  examiners were reassigned from other locations to the pilot sites. For  example, VA officials told us they assigned examiners from other VA  medical centers to the Fort Carson IDES and established a contract with a  private-sector provider to complete the exams that VA examiners would  normally have performed for veterans in the area claiming VA disability  compensation. At Fort Stewart, the contractor told us that they had  reassigned examiners from their Atlanta clinic to Fort Stewart.
section_title: Insufficiency of Exam Summaries and Disagreements about Medical Diagnoses and Ratings Can Prolong Case Processing Time
paragraphs: Issues related to the completeness and clarity of single exam summaries  were an additional cause of delays in the VA rating phase of the IDES  process. Officials from VA rating offices said that some exam summaries  did not contain information necessary to make a rating or fitness decision,  or were unclear as to the examiners’ diagnoses and conclusions. As a  result, VA rating office staff must ask the examiner to clarify the summary  or add information and, in some cases, redo the exam, adding time to the  process. In addition, VA rating staff told us that it is sometimes unclear  who they should contact if they identify insufficiencies in an exam  summary and finding the appropriate person also adds time. However, the  extent to which insufficient exam summaries caused delays in the IDES  process is unknown because DOD and VA’s VTA system does not track  whether an exam summary had to be returned to the examiner or whether  it was resolved. Due to these limitations, VA officials told us that VA rating  staff have created logs of outstanding insufficient exams and sent them to  VA examiners to correct.
VA officials attributed the problems with exam summaries to several  factors, including the difficulty of conducting exams for IDES pilot cases,  which may entail evaluating many complex medical conditions and may  involve several physicians and specialists. In addition, VA officials  indicated that, at sites with exam backlogs, such as at Fort Carson, it may  be difficult for examiners to ensure quality when are trying to complete  exams quickly. Furthermore, VA staff noted that some errors were  common, such as missing information for musculoskeletal conditions and  traumatic brain injury, suggesting that some examiners may not be aware  of the information required for certain types of medical conditions.  Finally, while examiners are supposed to receive the servicemember’s  complete medical records prior to the date of the exam, some VA  examiners also told us that they did not receive the records in time for the  exam in some cases, or the records were not well-organized. As a result,  they lacked key information, such as the servicemember’s medical history  and results of laboratory tests. According to the agencies’ operations  manual for the IDES pilot, the DOD board liaison should compile the  complete medical records within 10 days of an active duty servicemember  being referred to the IDES, but some DOD officials we spoke with said  that it is sometimes difficult to obtain all of the records, particularly when  servicemembers have received treatment from private-sector physicians.
In addition, while the single exam in the IDES eliminates duplicative exams  performed by DOD and VA in the legacy system, it raises the potential for  there to be disagreements about diagnoses of servicemembers’ conditions,  with implications for their disability ratings, as well as processing times.  DOD officials we spoke with in our interviews and site visits also said that  their physicians sometimes disagree with VA medical diagnoses, particularly  for mental health conditions, and this has extended processing times for  some cases. In addition, since medical diagnoses are a basis for VA’s  disability ratings, DOD may subsequently disagree with the ratings VA  completed for determining DOD disability benefits. The number of cases  with disagreements about diagnoses and ratings, and the extent to which  they have increased processing time, are unknown because the VTA system  does not track when a case has had such disagreements. However, officials  at 4 of the 10 pilot sites we visited said that military physicians have  disagreed with VA diagnoses in at least some cases. In addition, PEB  officials in two of the three military services—the Army and the Navy—said  that they have sometimes disagreed with the rating VA produced for  determining DOD disability benefits.
An example can illustrate the implications of differences in diagnoses.  Officials at Army pilot sites informed us about cases in which a military  physician had treated members for a mental condition, such as anxiety or  depressive disorder. However, when the members went to see the VA  examiners for their single exam, the examiners diagnosed them with  posttraumatic stress disorder (PTSD). When such cases were sent to the  PEB, it returned them to the MEB because it was unclear to the PEB  which conditions should be the basis of their decision on the  servicemembers’ fitness for duty. The cases then languished because the  military physicians experienced difficulties resolving the discrepancy with  the VA diagnosis.
To address such processing delays, the Army issued guidance in February  2010 stating that MEB physicians should review all of the medical records  (including the results of the single exam) and determine whether to revise  their diagnoses. If after doing so the MEB physician maintains that their  original diagnosis is accurate, they should write a memorandum  summarizing the basis of their decision, and the PEB should accept the  MEB’s diagnosis. Some Army officials we spoke with believe that this  guidance has been helpful for enabling cases to move forward when there  are differences in diagnoses. The other services do not have written  guidance on how to address differences in diagnoses, though Navy  officials told us that they have provided verbal guidance to their  physicians, and Air Force officials said they have not had cases with  significant disagreements about diagnoses.
In some cases, due to the differences in diagnoses, DOD has also disagreed  with the rating that VA prepared for DOD disability benefits, particularly in  cases involving servicemembers with mental health conditions. For  example, Army and Navy officials told us about cases in which the PEB  found the servicemember unfit due to a mental condition, such as major  depression, and asked VA to complete a rating for this condition.  However, VA returned a rating for occupational and social impairment  caused by PTSD, since the examiner had diagnosed the member with  PTSD. DOD requires a rating for only the conditions for which the member  was found unfit for duty because it can only provide disability benefits for  those conditions. However, according to VA regulations for rating mental  disorders, VA does not rate each mental health condition individually;  rather, VA bases its rating on the degree to which the combination of  symptoms of mental disorders cause occupational and social  impairment. As such, when rating mental health conditions for IDES  cases, VA officials said that rating specialists would consider both the  symptoms of mental conditions diagnosed by DOD physicians and those  identified by the VA examiner. Both Army and Navy PEB officials said that  they generally accept VA ratings in these cases, even though the rating is  not for the unfitting conditions alone. However, they noted that, if they  feel the VA rating is in error, there is no guidance on how disagreements  about servicemembers’ ratings should be resolved. Army and Navy  officials said that they may return the case to VA and informally requ est  that they reconsider the case, though Navy PEB officials said that they are  hesitant to do so because it may further delay the case.
DOD and VA officials attributed disagreements about diagnoses to several  factors. They noted that VA examiners may not have received or reviewed  the servicemembers’ medical records prior to the exam, and therefore may  not be aware of the medical conditions for which the members had been  previously diagnosed and treated. In addition, DOD and VA identify  conditions for different purposes in the disability evaluation system. While  DOD identifies conditions that make a servicemember unable to perform  their duties, VA identifies all service-connected conditions. As such, VA  examiners are likely to identify a broader set of conditions than DOD’s  physicians. In addition, local officials we spoke with in some of our site  visits said that servicemembers may be more willing to disclose all of their  medical conditions to VA than to DOD because VA could potentially  compensate them for all of the conditions. Furthermore, VA officials noted  that servicemembers’ health conditions may have changed between the  time DOD physicians identified the conditions and VA performed the  exam. Finally, DOD and VA officials said that differences in opinions about  diagnoses are common among physicians, particularly in the mental health  field. For example, they noted that it be can be difficult to distinguish  PTSD from anxiety, depression, and other mental health conditions.
section_title: Pilot Sites Faced Various Logistical Challenges Integrating VA Staff
paragraphs: DOD and VA officials at several pilot sites said that they experienced some  logistical challenges integrating VA staff at the military facilities. At a few  sites, it took time for VA staff to receive common access cards needed to  access the military facilities and to use the facilities’ computer systems.  During the time that VA staff did not have access cards, they were unable  to access VA computer systems, such as those for establishing the VA  claim, requesting exams, and viewing exam results, via DOD’s network.
In addition, DOD and VA staff noted several difficulties using the agencies’  multiple information technology (IT) systems to process cases. While the  agencies both use the VTA system to manage cases, VA also has IT  systems for completing certain tasks, and the military services also have  their own case tracking systems. This causes DOD and VA staff to have to  enter the same data multiple times into different IT systems. In addition,  some VA staff working on military bases reported that using the military  services’ computer systems to access VA systems has significantly slowed  down computer processing speeds. Finally, DOD and VA staff cannot  directly access each others’ systems, making it more cumbersome for case  managers to determine the status of servicemembers’ cases. For example,  without access to VA’s system for managing exams, DOD board liaisons  cannot readily provide servicemembers with information on when or  where their exams are scheduled and must contact VA case managers to  obtain the information. A few sites we visited were able to address some  IT issues. For example, at Fort Polk, VA officials said they were adding a  new telecommunications line to provide faster computer processing  speeds for their staff.
In addition, VA physicians working at military facilities need to be  credentialed by DOD before they can begin working on base, which  involves verification of their education, license, and clinical history. Some  VA officials said that this process could take 1 month or longer to  complete.
section_title: Extended Periods in the Military Disability Evaluation Process Posed Housing and Other Challenges at Some Pilot Sites
paragraphs: Although many DOD and VA officials we interviewed at central offices and  pilot sites felt that the IDES process expedited the delivery of VA benefits  to servicemembers, several also indicated that it may increase the amount  of time servicemembers are in the military’s disability evaluation process.  Data on legacy cases are not sufficiently reliable to determine whether this  is the case military-wide, but Army data appear to be sufficiently reliable  to allow for some limited analysis. Our analysis of Army pilot and legacy  data as of early 2010 shows that compared with legacy cases, active duty  cases in the pilot took on average 39 more days to reach the end of the  PEB phase—the last step of the DOD disability evaluation process before  servicemembers begin transitioning from military service or, if found fit,  back to duty. For reserve component cases in the Army, IDES pilot cases  took on average 17 more days to reach the end of the PEB phase,  compared with legacy cases. It was not possible to conduct this analysis  for the other military services because their legacy data lacked  information on when servicemembers were referred into the disability  evaluation system.
Some DOD officials noted that the increased time that servicemembers are  in the military’s disability evaluation process means that they must be  cared for and managed for a longer period. Officials in our site visits and  interviews said that some pilot sites have had challenges housing  servicemembers in the IDES, in part due to servicemembers being in the  process longer. For some servicemembers in the disability evaluation  system, the military services may move them to temporary medical units  or, for those needing longer-term medical care or complex case  management, to special medical units such as a Warrior Transition Unit in  the Army or Wounded Warrior Regiment in the Marine Corps. However,  these units were full at a few pilot sites we visited, or members in the IDES  did not meet the criteria for entering the special medical units. Where  servicemembers remain with their units while going through the disability  evaluation system, the units cannot replace them with able-bodied  members. Officials at Fort Carson said that this created a challenge for  combat units. Because most servicemembers in the IDES did not meet the  criteria for entering Warrior Transition Units, combat units had to find  another organizational unit to take charge of members in the IDES so they  could replace them with soldiers ready and able to deploy to combat  areas. In addition, officials at Naval Medical Center San Diego and Fort  Carson said that some members are not gainfully employed by their units  and, left idle while waiting to complete their disability evaluation process,  are more likely to engage in negative behavior, potentially resulting in  their being discharged due to misconduct and a forfeiture of disability  benefits. We were unable to assess the extent or cause of this problem  because the VTA system that tracks servicemembers in the IDES does not  capture sufficient detail on reasons for servicemembers dropping out of  the IDES, or which organizational unit(s) the servicemember was assigned  to while in the IDES. DOD officials also noted that servicemembers benefit  from continuing to receive their salaries and benefits while their case  undergoes scrutiny by two agencies, though some also acknowledged that  these additional salaries and benefits create costs for DOD.
section_title: DOD and VA Expansion Plans Address Some Though Not All Challenges
subsections: section_title: DOD and VA Have Incorporated Many Lessons Learned into Their Planning for Worldwide Expansion of the IDES but Lack Concrete Plans for Addressing Some Challenges
paragraphs: DOD and VA plan to expand the IDES to sites worldwide on an ambitious  timetable—to 113 sites during fiscal year 2011, a pace of about 1 site every  3 days. Expansion is scheduled to occur in four stages, beginning with   28 sites in the southeastern and western United States by the end of  December 2010.
DOD and VA have many efforts under way to prepare for IDES expansion.  At each site, local DOD and VA officials are expected to work together to  prepare for implementation. This includes completing a site assessment  matrix—a checklist of information DOD and VA officials at each site  should obtain and preparations they should make. While most pilot sites  had used a site assessment matrix to prepare for IDES implementation, the  agencies completed a significant revision of the matrix in August 2010, and  they now request additional information and documentation to address  areas where prior IDES sites had experienced challenges. In addition,  while during the pilot phase local DOD and VA officials were encouraged  to develop written agreements on IDES procedures, the matrix now  requests that a written agreement be completed prior to implementing the  IDES. Finally, senior-level local DOD and VA officials will be expected to  sign the site assessment matrix to certify that a site is ready for IDES  implementation. This differs from the pilot phase where, according to  DOD and VA officials, some sites implemented the IDES without having  been fully prepared. In addition, in September 2010, the military services  and VA held preimplementation training conferences for local DOD and  VA staff. At the time of our review, the first 28 expansion sites were  completing their site assessment matrices.
Through the new site assessment matrix and other initiatives under way,  DOD and VA are addressing several of the challenges identified in the pilot  phase. These include ensuring sufficient exam and case management staff,  being prepared to deal with surges in caseloads, addressing exam  sufficiency issues, and making adequate logistical arrangements.
Ensuring sufficient exam resources: The matrix asks whether a site can  complete single exams within the IDES’ 45-day time frame and within  DOD’s TRICARE access standards.  The matrix asks for detailed  information, such as who will conduct the exams (VA, VA contractor, or  military providers), where the exams will be conducted, and VA’s  anticipated overall volume of disability compensation and pension exams  in the area. In addition to the matrix, VA has several initiatives under way  to increase resources and expedite exams. VA plans to award a new  contract under which it can acquire examiners for sites that do not have  sufficient staff to perform exams, such as sites located where VA does not  have medical facilities or in rural areas where VA has had difficulty hiring  staff. VA has also recently changed its exam policy so that exams  performed by nurse practitioners or physician assistants certified to  perform disability exams no longer have to be cosigned by a physician,  which is expected to expedite completion of more exam reports.
Ensuring sufficient VA rating staff: VA officials said that they have hired  new staff to replace those that recently left the Baltimore rating office and  anticipate hiring a small number of additional staff. Based on caseload  projections, they expect that, once the additional staff are hired, the  Baltimore office will be close to having sufficient rating staff. Although VA  officials said that the Baltimore office conducted ratings for a majority of  cases during the IDES pilot phase, they have projected that the workload  will be divided almost evenly between the Baltimore and Seattle offices  once the IDES is fully expanded worldwide.
Ensuring sufficient DOD PEB adjudicators: Air Force officials informed  us they added adjudicators for the informal PEB and have since eliminated  their case backlog. They are currently adding adjudicators for the formal  PEB. Navy PEB officials also said that they are adding adjudicators  through activation of reserve component personnel for special work and  expected that they would be in place by November 2010.
Ensuring sufficient case management staffing: The site assessment  matrix also asks whether local facilities will have sufficient trained DOD  board liaison staff to meet a 1:20 caseload ratio and sufficient VA case  managers to meet a 1:30 caseload ratio. In addition, according to DOD  officials, each of the military services is increasing its board liaison  staffing levels to achieve 1:20 caseload ratios. VA officials said that they  plan to hire an additional 73 case managers.
Coping with caseload surges: The matrix asks sites to provide a longer  and more detailed caseload history—a 2-year, month-by-month history— as opposed to the 1-year history that DOD based its caseload projections  on during the pilot phase. In addition, the matrix asks sites to anticipate  any surges in caseloads, such as those due to seasonal trends. Sites are  also expected to provide a written contingency plan for dealing with  caseload surges. In addition, the matrix asks sites to develop a system for  communicating updates, such as information on expected caseload surges,  to stakeholders. VA officials also said that the Army has agreed to keep  them better informed of deployments that could result in caseload surges.  Further, VA officials noted that they are developing a plan for addressing  the additional need for examiners during surges, through which VA offices  with lower demand for disability exams would send examiners to an IDES  site experiencing a surge in exam workloads.
Ensuring the sufficiency of single exams: The site assessment matrix  asks sites whether all staff who will conduct exams are trained to VA  standards and certified by VA to conduct disability compensation and  pension exams. In addition, VA has begun the process of revising its exam  templates, to better ensure that examiners include the information needed  for a VA disability rating decision and enable them to complete their exam  reports in less time. Finally, a VA official stated that VA is examining  whether it can add capabilities to the VTA system that would enable staff  to identify where problems with exams have occurred and track the  progress of their resolution. For sites that choose to have military  physicians perform the single exams, VA officials said that they have  provided materials to DOD from their national training program, and DOD  has made these materials accessible on its Web site. To help improve the  ability of DOD board liaisons to obtain servicemembers’ medical and  personnel records prior to the exam, DOD officials said that they are  revising their policies to require reserve component units to provide the  records when a reserve member is referred to the IDES.
Ensuring adequate logistics at IDES sites: The site assessment matrix  asks sites whether they have the logistical arrangements needed to  implement the IDES, including necessary facilities, IT, and transportation  for servicemembers to exam locations. For example, the matrix asks  whether the military treatment facility will address the needs of VA staff  for access cards, identification badges, and security clearances, and  whether all VA medical providers will be credentialed and privileged to  practice at the DOD facility. In terms of IT, the matrix asks whether DOD  sites will enable VA staff access to VA information systems needed to  perform their duties. The matrix also asks sites to identify IT contacts  from both VA and DOD so that they may work together to resolve IT  problems. Furthermore, DOD and VA are developing a general  memorandum of agreement on IDES information sharing. This agreement  is intended to enable DOD and VA staff access to each other’s IT systems,  for example, to allow DOD staff to track the status of VA exams. DOD  officials also said that they are developing two new IT solutions.  According to officials, one system currently being tested would help  military treatment facilities better manage their cases. Another IT solution,  still at a preliminary stage of development, would integrate the VTA with  the services’ case tracking systems so as to reduce multiple data entry.
However, in some areas, DOD and VA’s efforts to prepare for IDES  expansion do not fully address some challenges or are not yet complete.
Ensuring sufficient military physician staffing: While DOD and VA are  taking steps to address shortages of examiners, case managers, and  adjudicators, they do not yet have strategies or plans to address potential  shortages of military physicians for completing MEB determinations. For  example, the site assessment matrix does not include a question about the  sufficiency of military providers to handle expected numbers of MEB  cases at the site, or ask sites to identify strategies for ensuring sufficient  military physicians if there is a caseload surge or staff turnover.
Ensuring sufficient housing and organizational oversight for IDES  participants: Although the site assessment matrix asks sites whether they  will have sufficient temporary housing available for servicemembers going  through the IDES, the matrix requires only a yes or no response and does  not ensure that sites will have conducted a thorough review of their  housing capacity prior to implementing the IDES. For example, sites are  not asked about the capacity of their medical hold units or special units  for wounded servicemembers, or to identify other options if their existing  units do not have sufficient capacity for their projected IDES caseload. In  addition, the site assessment matrix does not address whether sites have  plans for ensuring that IDES participants are gainfully employed or  sufficiently supported by their organizational units.
Addressing differences in diagnoses: According to a DOD official, as part  of its revision of its IDES operations manual, DOD is currently developing  guidance on how staff should address differences in diagnoses between  military physicians and VA examiners, and between military PEBs and VA  disability rating staff. DOD anticipated issuing the new guidance in  September 2010, but at the time of our review had not yet done so. In  addition, a VA official stated that VA is developing new procedures for  identifying cases with potential for multiple mental health diagnoses and  will ask VA examiners to review the servicemembers’ medical records and  reconcile differing diagnoses. However, since the new guidance and  procedures are still being developed, we cannot determine whether they  will resolve discrepancies or disagreements. Significantly, DOD and VA do  not have a mechanism for tracking disagreements about diagnoses and  ratings, and consequently, may not be able to determine whether the  guidance sufficiently addresses the discrepancies or whether it requires  further revision.
section_title: DOD and VA Lack a Mechanism for Monitoring Problems That May Emerge with Full Implementation
paragraphs: As DOD and VA move quickly to implement the IDES worldwide, they  have some mechanisms in place to monitor challenges that may arise in  the IDES. DOD officials said that they expect to continue holding  postimplementation “hotwash” meetings, in which they review individual  sites’ implementation. In addition, DOD and VA will continue to regularly  collect and report data on caseloads, processing times, and  servicemember satisfaction. Furthermore, the new site assessment matrix  asks sites to develop plans for VA and DOD local staff to meet weekly for  the first 60 to 90 days after implementing the IDES, then no less than  monthly to address any identified challenges. VA officials also said that  they will continue to prepare a report on an annual basis on challenges in  the IDES. To prepare this report, they will obtain input and data from local  DOD and VA officials.
However, DOD and VA do not have a system-wide monitoring mechanism  to help ensure that steps they took to address challenges are sufficient and  to identify problems in a more timely basis. For example, they do not  collect data centrally on staffing levels relative to caseload. Consequently,  despite efforts to acquire additional staff, as local sites experience staffing  turnover in the future, DOD and VA central offices may not become aware  that a site is short-staffed until their monitoring reports show lengthy  processing times. As a result, DOD and VA may be delayed in taking  corrective action, since it takes time to assess what types of staff are  needed at a site and to hire or reassign staff. In addition, without  information on when or how often other problems occur, such as  insufficient exam summaries or disagreements about diagnoses, DOD and  VA managers may not be able to target additional training or guidance  where needed. Furthermore, while DOD and VA report data on processing  times by phase of the process, military treatment facility, and military  service, their monitoring reports do not show processing times or  caseloads for each VA rating office and each of the five PEBs (three Army  and one each for the Navy and Air Force), limiting their ability to identify  if specific rating or PEB offices are experiencing challenges.
DOD and VA also lack mechanisms or forums for systematically sharing  information on challenges as well as best practices. For example, while  the site assessment matrix indicates that sites are expected to hold  periodic meetings to identify local challenges, DOD and VA have not  established a process for local sites to systematically report those  challenges to DOD and VA management and for lessons learned to be  systematically shared system-wide. During the pilot phase, VA surveyed  pilot sites on a monthly basis about challenges they faced in completing  single exams. Such a practice has the potential to provide useful feedback  if extended to other IDES challenges.
section_title: Conclusions
paragraphs: By merging two duplicative disability evaluation systems, the IDES shows  promise for expediting the delivery of VA benefits to servicemembers  leaving the military due to a disability. Servicemembers who proceed  through the process are able to leave the military with greater financial  security, since they receive disability benefits from both agencies shortly  after discharge. Further, having both DOD and VA personnel involved in  reviewing each disability evaluation may result in a more thorough  scrutiny of cases and informed decisions on behalf of servicemembers.
However, piloting of the system at 27 sites has revealed several significant  challenges that require careful management attention and oversight before  DOD and VA expand the system military-wide. DOD and VA are currently  taking steps to address many of these challenges, and the agencies have  developed a site implementation process that encourages local DOD and  VA officials to identify and resolve local challenges prior to transitioning  to the new system. However, given the agencies’ ambitious  implementation schedule—more than 100 sites in a year—it is unclear  whether all of these challenges will be fully dealt with before DOD and VA  deploy the integrated system to additional military facilities. For example,  it is unclear whether sites will have sufficient military physicians to  complete key steps of the process in a timely manner. Insufficient staffing  of any one part of the process is likely to lead to bottlenecks, delaying not  only servicemembers’ receipt of disability benefits, but also their  separation from the military and reentry into civilian life. In addition,  DOD’s preparations of sites for the IDES do not ensure that military  facilities have adequate capacity or plans for housing and providing  organizational oversight over servicemembers in the IDES, who potentially  could remain at the locations for extended periods of time. Furthermore,  while integrating VA medical exams into DOD’s disability evaluation  system eliminates duplicative exams, it raises the potential for there to be  disagreements about diagnoses of servicemembers’ conditions, with  implications for servicemembers’ disability ratings and their DOD  disability compensation. While DOD is developing guidance to address  such disagreements, it is important that the agencies have a thorough  understanding of how often and why these disagreements occur and  continually review whether their new guidance adequately addresses this  issue so as to be able to make improvements where needed.
Successful implementation of any program requires effective monitoring.  DOD and VA currently have mechanisms to track numbers of cases  processed, timeliness, and servicemember satisfaction, but they do not  routinely monitor factors—such as staffing levels relative to caseload,  disagreements about diagnoses, and insufficient exam summaries—that  can delay the process. In addition, they do not monitor timeliness and  caseloads for some of the key IDES offices, namely each VA rating office  and each PEB. Ultimately, the success or failure of the IDES will depend  on DOD and VA’s ability to sufficiently staff local sites, the VA rating  offices, and the PEBs, and to resolve other challenges not only at the  initiation of the transition to IDES but also on an ongoing, long-term basis.  By not monitoring staffing and other risk factors, DOD and VA may not be  able to ensure that their efforts to address these factors are sufficient or to  identify problems as they emerge and take immediate steps to address  them before they become major problems.
section_title: Recommendations for Executive Action
paragraphs: To ensure that the IDES is sufficiently staffed and that military treatment  facilities are prepared to house personnel in the IDES, we recommend that  the Secretary of Defense direct the military services to conduct thorough  assessments prior to each site’s implementation of the IDES of the  following three issues:  the adequacy of staffing of military physicians for completing MEB  determinations at military treatment facilities; contingency plans should  be developed to address potential staffing shortfalls, for example, due to  staff turnover or caseload surges;  the availability of housing for servicemembers in the IDES at military  facilities; alternative housing options should be identified if sites do not  have adequate capacity; and  the capacity of organizational units to absorb servicemembers undergoing  the disability evaluation; plans should be in place to ensure  servicemembers are appropriately and constructively engaged.
To improve their agencies’ ability to resolve differences about diagnoses of  servicemembers’ conditions, and to determine whether their new guidance  sufficiently addresses these disagreements, we recommend that the  Secretaries of Defense and Veterans Affairs take the following two actions:    conduct a study to assess the prevalence and causes of such    establish a mechanism to continuously monitor disagreements about  diagnoses between military physicians and VA examiners and between  PEBs and VA rating offices.
To enable their agencies to take early action on problems at IDES sites  postimplementation, we recommend that the Secretaries of Defense and  Veterans Affairs develop a system-wide monitoring mechanism to identify  challenges as they arise in all DOD and VA facilities and offices involved in  the IDES. This system could include:    continuous collection and analysis of data on DOD and VA staffing levels,  sufficiency of exam summaries, and diagnostic disagreements;    monitoring of available data on caseloads and case processing time by  individual VA rating office and PEB; and    a formal mechanism for agency officials at local DOD and VA facilities to  communicate challenges and best practices to DOD and VA headquarters  offices.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DOD and VA for review and  comment. The agencies provided written comments, which are  reproduced in appendixes III and IV. DOD and VA generally concurred  with our recommendations. Each agency also provided technical  comments, which we incorporated as appropriate.
DOD concurred with our recommendation to ensure that, before the IDES  is implemented at each new site, a thorough assessment be done of the  site’s staffing adequacy, the availability of housing for servicemembers in  the IDES, and the capacity of organizational units to appropriately and  constructively engage servicemembers in the IDES. However, DOD stated  that the IDES site assessment matrix addresses plans to ensure that  servicemembers are gainfully employed while in the IDES. We changed  our report to more clearly indicate that the site assessment matrix does  not, in fact, address such plans. We believe that specifically identifying this  in the matrix could help local DOD officials, including servicemembers’  unit commanders, focus on ensuring gainful employment or other support.
DOD concurred, and VA concurred in principle, with our recommendation  to study and establish mechanisms to monitor diagnostic differences. VA  identified a plan to study the prevalence and causes of diagnostic  differences and determine by July 1, 2011, whether mechanisms are  needed. DOD stated that it expects, as diagnostic differences are  monitored and studied, that the agencies will address and resolve many of  the issues identified in our report. We agree that the planned study could  yield valuable insights on how to resolve diagnostic differences but  emphasize that continuous monitoring of such differences over a period of  time may be needed to assess the extent and nature of such differences, as  well as the success of any actions to address them.
Both agencies concurred with our recommendation to develop monitoring  mechanisms to help them take early actions on problems that may arise at  IDES sites postimplementation. VA stated that the VTA system currently  has data that can be monitored by PEB and VA rating site, and DOD said  its weekly monitoring report could be modified to present these data. Also,  VHA plans to monitor the IDES exam workload, including numbers of  exam requests compared with forecasts, exam timeliness, and insufficient  exams. Implementation is scheduled for December 31, 2010. In terms of  identifying site implementation problems for quick resolution, DOD stated  that the military services bring sites’ challenges and best practices to the  Disability Advisory Council, a DOD body that includes VA representatives,  which is being re-chartered as part of the Benefits Executive Council, a  subgroup of the VA-DOD Joint Executive Council. VA and DOD’s plans  sound promising and consistent with our recommendations provided that  they allow for ongoing monitoring of site staffing levels and create a  systematic way for local DOD and VA staff to communicate their  challenges or best practices, enabling the agencies to identify and address  problems at an early stage.
We are sending copies of this report to the appropriate congressional  committees, the Secretary of Defense, the Secretary of Veterans Affairs,  and other interested parties. The report is also available at no charge on  the GAO Web site at www.gao.gov.
If you or your staff members have any questions about this report, please  contact me at (202) 512-7215 or at bertonid@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. Staff members who made key contributions in  this report are listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: In conducting our review of the integrated disability evaluation system  (IDES) piloted by the Departments of Defense (DOD) and Veterans Affairs  (VA), our objectives were to examine (1) the results of DOD and VA’s  evaluation of the IDES pilot, (2) challenges in implementing the piloted  system to date, and (3) DOD and VA plans to expand the piloted system  and whether those plans adequately address potential challenges. We  conducted this performance audit from November 2009 to December 2010,  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Review of DOD and VA’s Evaluation of the IDES Pilot
paragraphs: To address objective 1, we reviewed DOD and VA policy guidance, reports,  and analysis plans to determine how the agencies are evaluating the pilot’s  effectiveness and to obtain information on their results. We also reviewed  the relevant requirements of the National Defense Authorization Act of  2008 as it pertains to this review. In addition, we interviewed officials  responsible for the evaluation at DOD’s Office of the Deputy Under  Secretary of Defense for Wounded Warrior Care & Transition Policy  (WWCTP), DOD’s Defense Manpower Data Center, and two organizations  that DOD has contracted with to perform the evaluation—Booz Allen  Hamilton and Westat. We then tested the reliability of the data the  agencies are using for their evaluation—data from surveys of  servicemembers, IDES case data from the Veterans Tracking Application  (VTA) system, and legacy case data that DOD’s WWCTP obtained from the  military services. Finally, we conducted some analyses of IDES and legacy  case data for the Army to compare the two systems on timeliness and  appeal rates, using elements of the data that we found to be reliable, but  these comparisons have limitations and are not generalizable to other  military services. The sections below describe our data reliability work  and our analysis of Army data in further detail.
subsections: section_title: Review of Satisfaction Survey Data Reliability
paragraphs: DOD and VA have been surveying servicemembers going through the IDES  pilot, and a comparison group of veterans who went through the standard  “legacy” disability evaluation system, to determine whether the IDES pilot  has improved servicemember satisfaction. The agencies survey all  servicemembers in the IDES pilot at three points in time—following their  completion of the medical evaluation board (MEB) of the disability  evaluation process, completion of the physical evaluation board (PEB),  and during the transition phase. To create a comparison group, the  agencies sampled veterans who have been through the legacy system at  current pilot sites. Their sampling methods were designed to ensure that  the pilot and legacy groups were of comparable size and had similar  proportions of servicemembers found unfit for duty. DOD and VA are  analyzing the differences between the pilot and legacy groups’ average  responses on four “survey composites,” or general categories composed of  several survey questions: overall experience, fairness, DOD board liaison  officer customer service, and VA case manager customer service.
We reviewed the reliability of surveys DOD and VA are using to obtain  information on satisfaction levels by examining their survey design and  analysis. To do so, we interviewed officials at DOD’s Defense Manpower  Data Center and Westat responsible for implementing the survey, as well  as officials at WWCTP and Booz Allen Hamilton responsible for designing  the survey and analyzing the survey data. We also reviewed the survey  instruments, response rates, data analysis plans, analysis results, and  survey data as of February 28, 2010. We found DOD’s survey  methodology—and the data derived using that methodology—to be  reliable for purposes of comparing servicemembers’ satisfaction levels in  the IDES and legacy disability evaluation systems.
section_title: Pilot and Legacy Case Data Reliability Tests
paragraphs: DOD and VA are collecting data on IDES pilot cases through the VTA and  are using these data to conduct ongoing monitoring of case processing  times and appeal rates, with the results presented in weekly reports. VA  manages VTA, but evaluation of the data is primarily conducted by staff at  DOD’s WWCTP and Booz Allen Hamilton.
For their August 2010 interim report to Congress, DOD staff created a data  set used to compare pilot and legacy processing times and appeal rates.  This data set included IDES pilot cases as of February 28, 2010, with the  earliest case started in November 2007. The data set also included data, as  of January 31, 2010, on legacy cases started between fiscal years 2005 and  2009 at the first 21 sites operating the IDES pilot, prior to pilot  implementation. The agencies also matched legacy case data from each of  the military services with VA data, in order to capture additional  processing time it took for servicemembers to navigate the VA disability  claims process. Because the data set was created from February 2010 pilot  data, it only included about one-third of the IDES pilot cases that were  completed as of August 29, 2010. The February 2010 data set included  cases from 17 of the 27 current pilot sites, and 7 of the 17 sites—including  some of the pilot sites with the largest caseloads such as Fort Carson and  Camp Lejeune—had fewer than 20 completed cases each when the data  set was created.
To assess whether the data DOD and VA are using for their monitoring and  evaluation are reliable, we obtained the early 2010 data set that the  agencies planned to use for their evaluation report to Congress. We  restricted our reliability assessments to the specific variables that the  agencies used in their analyses. Following steps detailed below, we found  that the IDES pilot case data were sufficiently reliable for our analyses,  but that the legacy case data were incomplete with respect to data  elements key to measuring case processing time and appeal rates.
To assess the reliability of the agencies’ IDES pilot data, we interviewed  VA database managers responsible for VTA, reviewed VTA manuals and  guidance, conducted electronic tests of the data and, for a small, random  sample of cases, checked the data against case files.
Through our interviews and document reviews, we concluded that the  agencies have sufficient internal controls to give reasonable assurance  that the data are complete.
Our electronic testing of the data generally found low rates of missing data  and errors in completed IDES cases. In these tests, we considered a data  element to be sufficiently reliable for purposes of using in our report if 15  percent or less of the data were missing or had errors. Using this standard,  we determined that one data element for IDES cases—the date that  servicemembers separated from the military—was not reliable, because:  (1) it was missing in 19 percent of completed cases and (2) in cases where  the date was present, more than 30 percent appeared to have errors (for  example, the date was before a step of the process that it should have  followed).
We also conducted a trace-to-file process to determine whether date fields  in the VTA system were an accurate reflection of the information in the  IDES case files. Specifically, we compared 12 date fields in the VTA  against a random sample of paper files for 54 completed cases: 24 from the  three Army PEBs, 10 from the Air Force PEB, and 20 from the Navy PEB  (10 Navy cases and 10 Marine Corps). In comparing these dates, we  allowed for a 10 percent discrepancy in dates—i.e., a difference of 2 to 10  days, depending on the date and phase of the process—to allow for the  possibility that dates may have been entered into the database after an  event took place. The trace-to-file process resulted in an overall accuracy  rate of 84 percent. For five data elements key to DOD and VA’s evaluation  of the IDES pilot, we found that VTA dates reflected dates in the case files  85 percent of the time or better. For six key data elements—i.e., the end  dates of the exam and MEB phases, the start of the PEB phase, the date a  VA rating request was made, the date of the final disposition, and the date  servicemembers received VA benefits—the VTA dates matched case file  dates between 70 to 85 percent of the time. Although we considered these  dates sufficiently reliable to include in this report, these dates should be  interpreted with more caution. The separation date was accurate less than  70 percent of the time and did not meet our standards of reliability.
To assess the reliability of the legacy data that the agencies planned to  compare the IDES pilot against, we tested the data electronically, and  found that data for key dates and appeals indicators had significant gaps  because the services did not collect the same information for legacy cases  that were collected for pilot cases (see table 2). For example, only Army  cases had information on when servicemembers were referred to the MEB  process. In addition, the legacy data did not include the date on when  servicemembers received VA benefits—which is necessary for measuring  the full length of the legacy process. Without sufficient data on the  beginning (when servicemembers were referred into the system) or end of  the process (when they received VA benefits), we concluded that the full  case processing time in the legacy system cannot be known. We also  concluded that comparisons could not be made between the legacy and  IDES pilot on appeal rates because only Army and Air Force cases had  information on whether servicemembers appealed the informal PEB  decisions.
section_title: GAO’s Review of the Agencies’ Comparison of Pilot and Legacy Data
paragraphs: In addition to reviewing the reliability of the IDES pilot and legacy data,  we reviewed how DOD and VA are using the data for their comparisons of  the two disability evaluation systems. Through interviews with officials at  DOD’s WWCTP and Booz Allen Hamilton and documents they provided us,  we understand that DOD planned to address gaps in the legacy data by: (1)  approximating the referral dates in Air Force, Marine Corps, and Navy  cases using Army data and (2) using dates when cases were ready to be  rated by VA to approximate the end of the process. Specifically, to  approximate referral dates, they said they would use the average time for  Army cases between when the servicemember was referred and when the  MEB documentation identifying the servicemember’s potentially unfitting  medical conditions (i.e., the narrative summary) was completed, which  they calculated to be 60 days. For Navy and Marine Corps cases, they then  subtracted 60 days from the date of the narrative summary to estimate a  referral date and, for Air Force cases, they did so from the date of the MEB  decision. However, because only 11 percent of Army legacy cases had a  narrative summary date, the estimate of 60 days is based on a small  number of cases (see table 3). To address the lack of data on the date VA  benefits were delivered, DOD planned to use the date that VA determined  a case was ready to be rated to approximate the end of the process,  though this would underestimate the length of time it took to deliver VA  benefits in the legacy process.
section_title: GAO Analysis of IDES Case Data
paragraphs: For objective 1, we presented information on average processing time in  the IDES, both overall and by military service, using information presented  by DOD and VA in their weekly monitoring reports. Where information  was not available in the weekly reports, we conducted our own analysis  using the early 2010 data set that DOD and VA intended to use for their  report to Congress. Specifically, we used these data to determine the  proportion of pilot cases meeting the 295-day goal for active duty  servicemembers and the 305-day goal for reserve servicemembers.
In addition, although limitations in the legacy data preclude reliable  comparisons between the IDES pilot and legacy systems for all the  military services, the Army legacy data on when servicemembers were  referred into the IDES were sufficiently complete to make some limited  comparisons. Specifically, we analyzed Army legacy data to determine  how long the legacy process took, on average, between when  servicemembers were referred to the process and when VA was ready to  conduct the disability rating. We limited our analysis to cases in which a  VA claim was filed between 2006 and 2009 because data on when VA was  ready to conduct the rating was missing for a substantial number of cases  where the VA claim was filed in 2005 and 2010. We compared this legacy  average with the total pilot case processing time through to delivery of VA  benefits, but we noted that the legacy average does not account for time  for VA to complete the rating and deliver the benefits. We also analyzed  Army data on appeals in order to illustrate the limitations of DOD’s plan to  compare only appeals to the informal PEB in the pilot and legacy systems  and not take into account appeals of rating decisions to VA. We conducted  this analysis using the legacy data and pilot case data as of early 2010,  since DOD and VA’s weekly reports do not contain information on appeals  to VA.
section_title: Identifying Challenges in Implementing the IDES at Pilot Sites
paragraphs: To identify challenges in implementing the IDES during the pilot phase, we  visited 10 of the 27 military treatment facilities participating in the pilot. At  the site visits, we interviewed officials involved in implementing the IDES  from both DOD and VA, including military facility commanders and  administrators, DOD board liaisons, military physicians involved in MEB  determinations, DOD legal staff, VA case workers, VA or contract  examiners, and administrators at VA medical clinics and VA regional  offices. We selected the 10 facilities to obtain perspectives from sites in  different military services and geographical regions and that varied in  terms of disability evaluation caseloads and how their single exams were  conducted (by DOD, VA, or a VA contractor) (see table 4).
We also interviewed various offices at DOD and VA involved in  implementing the IDES pilot. At DOD, this included WWCTP; Office of the  Assistant Secretary of Defense for Health Affairs; Office of the Assistant  Secretary of Defense for Reserve Affairs; Air Force Physical Disability  Division; Army Physical Disability Agency; Navy Physical Evaluation  Board; Office of the Air Force Surgeon General; Army Medical Command;  and Navy Bureau of Medicine and Surgery. At VA, we interviewed officials  in the Veterans Benefits Administration, Veterans Health Administration,  and VA/DOD Collaboration Service.
Furthermore, we reviewed relevant documents, including DOD and VA  policies and guidance and records of “hotwash” meetings, which DOD and  VA held shortly after implementing the IDES at pilot sites to identify  implementation successes and challenges. We also reviewed data on  processing times for the single exams, MEB determinations, informal PEB  decisions, and VA ratings, as reported in the agencies’ weekly monitoring  reports. In addition, we reviewed relevant federal laws and regulations.
To determine whether the IDES process extended the time that  servicemembers remained in military service, we analyzed the legacy and  pilot case data from the early 2010 data set, but we identified several  limitations with the data. As noted earlier, the date servicemembers  separated from the military was missing for 19 percent of completed IDES  pilot cases. Further, as shown in table 5, only Air Force cases contained  data on the separation date in the legacy data. Also noted earlier, only the  Army legacy data contained information on when servicemembers were  referred into the legacy process. As a result, for Army cases, we compared  the average length of time it took cases to reach a final PEB decision in the  legacy and pilot, since this date was sufficiently complete in both the  legacy and pilot data. The PEB decision is the last phase of the disability  evaluation process before a servicemember either begins to transition  from military service, or if they are found fit, returns to their unit.
section_title: Examining DOD and VA’s Plans for Expanding the IDES
paragraphs: To identify the agencies’ preparations for worldwide expansion of the  IDES, we reviewed documents on DOD and VA’s expansion strategy, their  site assessment matrix, and weekly monitoring reports which, beginning in  July 2010, tracked key implementation time frames, both nationally and at  individual military treatment facilities. Our interviews with officials  involved in the pilot at DOD, VA, and each of the military services also  provided us with information on the agencies’ expansion plans. We also  reviewed relevant federal laws and regulations.
We determined the adequacy of the agencies’ planning efforts by assessing  whether their plans addressed the challenges we had identified in  objective 2. We also determined whether the plans incorporated internal  controls described in GAO’s Standards for Internal Control in the Federal  Government and best practices for program implementation identified in  academic literature.
section_title: Appendix II: IDES Pilot Processing Times for Reserve Component Servicemembers
paragraphs: The figures below show case processing times in the IDES pilot for reserve  component servicemembers. Figure 11 shows the average number of days  it took to complete the process—i.e., to deliver VA benefits to reserve  component servicemembers, as of August 2010. Figure 12 shows the  percentage of cases that met the DOD and VA goal to deliver VA benefits  within 305 days, as of February 2010. Figures 13-15 show the average  length of time it took, as of August 2010, to complete phases of the IDES  process—i.e., the single exam, the MEB documentation, and the informal  PEB decision, respectively—each of which have taken longer, on average,  than the goals established by DOD and VA.
section_title: Appendix III: Comments from the Department of Defense
section_title: Appendix IV: Comments from the Department of Veterans Affairs
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: Michele Grgich (Assistant Director), Yunsian Tai (Analyst-in-Charge),  Jeremy Conley, and Greg Whitney made significant contributions to this  report. Walter Vance and Vanessa Taylor provided assistance with  research methodology and data analysis. Bonnie Anderson, Rebecca  Beale, Mark Bird, Brenda Farrell, Valerie Melvin, Patricia Owens, and  Randall Williamson provided subject matter expertise. Susan Bernstein  and Kathleen van Gelder provided writing assistance. James Bennett  provided graphics assistance. Roger Thomas provided legal counsel.