Military Disability System: Improved Monitoring Needed to Better Track and Manage Performance
Published: 2012-08-28T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Since 2007, DOD and VA have jointly operated IDES--which is intended to expedite benefits for injured servicemembers. IDES replaced the departments' separate processes for evaluating servicemembers for fitness and disability. Initially a pilot at 3 military treatment facilities, IDES is now in place at military treatment facilities worldwide. In previous reports, GAO identified a number of challenges as IDES expanded to more facilities, including staffing shortages and difficultly meeting timeliness goals. In light of IDES' expansion, GAO was asked to examine: (1) the extent to which DOD and VA are meeting IDES timeliness and servicemember satisfaction performance goals, and (2) steps the agencies are taking to improve IDES performance. GAO analyzed IDES timeliness and customer satisfaction data, visited six IDES sites with varying performance, and interviewed DOD and VA officials.
section_title: What GAO Found
paragraphs: Case processing times under the Integrated Disability Evaluation System (IDES) have increased over time, and measures of servicemember satisfaction have shortcomings. Since 2008, annual average processing times for IDES cases have steadily climbed, while the percentage of cases meeting established timeliness goals declined. Average case processing times reached 394 and 420 days for active and reserve component members in fiscal year 2011--compared to goals of 295 and 305 days, respectively, and just 19 percent of active duty and 18 percent of guard or reserve servicemembers completed the process and received benefits within established goals. Of the four phases comprising IDES, the medical evaluation board phase increasingly fell short of timeliness goals, while the physical evaluation board phase, although meeting goals, was taking increasingly more time to complete. With respect to servicemember satisfaction with the IDES process, GAO found shortcomings in how these data are collected and reported, such as unduly limiting who is eligible to receive a survey and computing average satisfaction scores in a manner that may overstate them. Department of Defense (DOD) officials told GAO they are considering alternatives for gauging satisfaction with the process. DOD and Veterans Affairs (VA) are taking steps to improve IDES performance, but progress to date is uneven and it is too early to assess their overall impact. For example, VA increased resources for completing exams and disability ratings while the Army is hiring additional staff for its medical evaluation boards. VA has met exam timeliness goals in the past several months, but other resources have yet to translate into lower processing times. DOD and VA are pursuing system upgrades so that staff and managers at IDES facilities can better track and manage the progress of servicemembers' cases. IDES officials have been working with the military services to correct case data that were inaccurately entered into VA's IDES tracking system, but have not yet identified a permanent solution to improve the accuracy of data input. Finally, DOD, with VA's assistance, is in the early stages of an in-depth review of the entire IDES process and its supporting IT systems. This effort is intended to improve understanding of how each step contributes to overall processing times and identify opportunities to streamline the process and supporting systems. However, timeframes for completing the review or issuing recommendations have yet to be established.
section_title: What GAO Recommends
paragraphs: To improve monitoring of IDES timeliness and satisfaction, GAO recommends that DOD and VA work together to (1) develop plans for completing the ongoing business process review and implementing any resulting recommendations and (2) improve the accuracy of case information at the point of data entry; and that (3) DOD consider alternative approaches to measuring satisfaction. DOD and VA concurred with GAO's recommendations.

section_title: Letter
paragraphs: Servicemembers who are injured in war or as the result of accidents and  illnesses may face a difficult transition as they leave the military and  become veterans. In response to concerns that wounded, ill, or injured  servicemembers had to undergo two complex disability evaluations—first  by the Department of Defense (DOD) then by the Department of Veterans  Affairs (VA)—DOD and VA jointly designed a new integrated disability  evaluation process to expedite the delivery of benefits to  servicemembers. In November 2007, DOD and VA began pilot testing the  integrated disability evaluation system (IDES) at three military treatment  facilities in the Washington, D.C. area, and expanded the number of sites  over time. As of October 1, 2011, IDES had replaced the military services’  existing—or “legacy”—disability evaluation systems for almost all new  disability cases.
Past GAO work highlighted challenges DOD and VA experienced while  piloting the IDES and recommended a number of improvements. For  instance, we reported in December 2010 that insufficient staff and  logistical challenges contributed to delays in completing IDES cases and  recommended the agencies take steps to ensure adequate staffing levels  and develop a systematic process for monitoring caseloads. In response  to ongoing concerns with IDES performance, this report provides  information on (1) the extent to which DOD and VA are meeting IDES  performance goals, and (2) steps DOD and VA are taking to improve  IDES performance.
In conducting our work, we obtained DOD timeliness and customer  satisfaction data from the inception of IDES in 2007 to December 2011.  We assessed the reliability of these data and analyzed them to look for  changes in performance over time; factors that may help or hinder  performance; and relationships between servicemember satisfaction and  case outcomes and timeliness. We supplemented these analyses with  site visits to six military treatment facilities, where we spoke with DOD  and VA staff as well as some servicemembers involved in the IDES  process. We selected these facilities to obtain perspectives from sites in  different military services and geographical regions and with varying  caseloads and performance outcomes. For both research objectives, we  interviewed key officials involved with IDES at DOD, VA, and each of the  military services, and reviewed pertinent reports, guidance, plans,  relevant federal laws, regulations, directives, and other documents. We  conducted this performance audit from May 2011 to August 2012, in  accordance with generally accepted government auditing standards.  These standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe the  evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: The Disability Evaluation Process
paragraphs: The IDES process begins at a military treatment facility when a physician  identifies one or more conditions that may interfere with a  servicemember’s ability to perform his or her duties. The process  involves four main phases: the Medical Evaluation Board (MEB), the  Physical Evaluation Board (PEB), transition out of military service  (transition), and VA benefits.
MEB phase: In this phase, medical examinations are conducted and  decisions are made by the MEB regarding a servicemember’s ability to  continue to serve in the military. This phase involves four stages: (1) the  servicemember is counseled by a DOD board liaison on what to expect  during the IDES process; (2) the servicemember is counseled by a VA  case manager on what to expect during the IDES process and medical  exams are scheduled; (3) medical exams are conducted according to VA  standards for exams for disability compensation by VA, DOD, or  contractor physicians, and (4) exam results are used by the MEB to  identify conditions that limit the servicemember’s ability to serve in the  military.an impartial medical review by a physician not on the MEB, or both.
Also during this stage, a servicemember can seek a rebuttal, or  PEB phase: In this subsequent phase, decisions are made about the  servicemember’s fitness for duty, disability rating and DOD and VA  disability benefits, and the servicemember has opportunities to appeal  those decisions. This includes: (1) the informal PEB stage, an  administrative review of the case file by the relevant military branch’s PEB  without the presence of the servicemember; (2) VA rating stage, where a  VA rating specialist prepares a rating that covers the conditions that DOD  determined made a servicemember unfit for duty and any other conditions  claimed by the servicemember to VA. This rating is prepared for use by both agencies in determining disability benefits. In addition,  servicemembers have several opportunities to appeal different aspects of  their disability evaluations: a servicemember dissatisfied with the decision  on whether he or she is fit for duty may request a hearing with a “formal”  PEB; a servicemember who disagrees with the formal PEB fitness  decision can, under certain conditions, appeal to the reviewing authority  of the PEB; and a servicemember can ask for VA to reconsider its rating,  but only for conditions found unfitting by the PEB.
Transition phase: If the servicemember is found unfit to serve, he or she  enters the transition phase and begins the process of separating from the  military. During this time, the servicemember may take accrued leave.  Also, DOD board liaisons and VA case managers provide counseling on  available benefits and services, such as job assistance.
VA benefits phase: A servicemember found unfit and separated from  service becomes a veteran and enters the VA benefits phase. VA  finalizes its disability rating after receiving evidence of the  servicemember’s separation from military service. VA then starts to award  monthly disability compensation to the veteran.
DOD and VA established timeliness goals for the IDES process to provide  VA benefits to active duty servicemembers within 295 days of being  referred into the process, and to reserve component members within 305  days (see fig. 1). DOD and VA also established interim timeliness goals  for each phase and stage of the IDES process. The overall timeframes  are intended to represent an improvement over the legacy disability  evaluation system, which was estimated to take 540 days to complete.
In addition to timeliness, the agencies also established a performance  goal of having 80 percent of servicemembers satisfied with the IDES  process. DOD measures satisfaction through surveys conducted after the  completion of the MEB, PEB, and transition phases. Each survey consists  of approximately 30 questions, including 4 questions that ask about the  servicemember’s satisfaction with the overall IDES process up to that  point. Reported satisfaction rates for each phase are based on an  average of responses to these four questions, and reported overall  satisfaction with IDES (which is used to track the percent satisfied under  the performance goal) is an average of satisfaction rates for the three  phases.
section_title: Rollout of IDES and Enrollment
paragraphs: From the original 3 pilot military treatment facilities in the Washington,   the IDES has expanded to 139 military treatment facilities in  D.C., area,the U.S. and several other countries. DOD and VA first added 24 military  treatment facilities to the pilot in fiscal years 2009 and 2010, bringing the  pilot total to 27. In 2010, DOD and VA leadership decided to implement  the IDES world-wide, and did so in 4 stages between October 2010 and  September 2011, adding 112 military treatment facilities. As IDES  expanded, the number of new cases enrolled in IDES has also increased,  totaling 18,651 in fiscal year 2011 (see fig. 2).
IDES caseloads vary by service, but the Army manages the bulk of IDES  cases. Of new cases referred to IDES in fiscal year 2011, about 64  percent were in the Army, and much of the growth in caseload has been  in the Army. Additionally, active duty servicemembers make up the  majority of IDES cases, with about 88 percent of new cases in fiscal year  2011 involving this group (see fig. 3).
section_title: IDES Processing Times Increased Over Time, While Measures of Servicemember Satisfaction Have Shortcomings
subsections: section_title: Overall IDES Case Processing Times Steadily Increased Since the Start of IDES
paragraphs: IDES timeliness has worsened since the inception of the program. Since  fiscal year 2008, the average number of days for servicemember cases to  be processed and receive benefits increased from 283 to 394 for active  duty cases (compared to the goal of 295 days) and from 297 to 420, for  reserve component cases (compared to the goal of 305 days) (see   fig. 4).
Along with increasing average processing times, the percent of IDES  cases awarded benefits within timeliness goals has steadily declined.  DOD’s and VA’s current goal is to complete 60 percent of IDES cases on  time. In fiscal year 2008, an average of 63 percent of cases for active  duty servicemembers and 65 percent for reservists completed the  process and received benefits within the timeliness goals; by fiscal year  2011 this was down to 19 and 18 percent respectively (see fig. 5). These  trends also hold when considering all cases that completed the IDES  process regardless of outcome, although overall processing times were  shorter. (See app. III for more information on case processing times  regardless of outcome.)
When examining timeliness across the four phases that make up IDES,  data show that average processing time regularly fell short of goals for  three—MEB, Transition, and VA Benefits. For example, for cases that  completed the MEB phase in fiscal year 2011, active duty and reserve  component members’ cases took an average of 181 and 188 days  respectively to be processed, compared to goals of 100 and 140 days.  For the PEB phase, processing times increased over time, but were still  within the established goal of 120 days. Along with increasing average  processing times, the percentage of cases meeting goals for most phases  has generally declined (see fig. 6). In particular, the MEB and Transition  phases have lower percentages of cases meeting goals than the other  phases in most years, especially for active duty cases.
As noted above, the MEB phase was a key contributor to increases in  overall processing times between 2008 and 2011 for both active duty  servicemembers and reservists for cases that have completed the IDES  process regardless of outcome (table 1).
To obtain a better understanding of more recent timeliness trends within  the MEB phase, GAO analyzed MEB timeliness of all cases—all fiscal  years combined—that completed the MEB process by sorting them into  two groups: (1) those that completed the entire IDES process, and (2)  those that had not yet completed IDES but completed the MEB phase. As  shown in figure 7, for the group that completed IDES, 30 percent of active  duty servicemembers and 18 percent of reservists missed the goal by  more than 90 days. For those still in IDES, representing more recent data,  the picture is slightly better for active duty servicemembers with 37  percent of cases meeting the MEB goal and 25 percent missing the goal  by more than 90 days. However, the percentage of reserve component  members who missed the goal by more than 90 days increased from 18  to 28 percent.
For those servicemembers who were still enrolled in the MEB phase as of  December 2011, the data show that 41 percent of active duty and 33  percent of reserve component servicemember cases had already missed  the goal processing times (see fig. 8). Of these, 15 percent of active duty  and 10 percent of reservist component servicemember cases missed the  goal by more than 90 days.
Within the MEB phase, significant delays have occurred in completing  medical examinations (medical exam stage) and delivering an MEB  decision (the MEB stage). For cases completing the MEB phase in fiscal  year 2011, 31 percent of active duty and 29 percent of reservist cases  met the 45-day goal for the medical exam stage and 20 percent of active  duty and 17 percent of reservist cases met the 35-day goal for the MEB  stage. Officials at some sites we visited told us that MEB phase goals  were difficult to meet and not realistic given current resources. For  example:
Some military officials noted that they did not have sufficient numbers  of doctors to write the narrative summaries of exam results needed to  complete the MEB stage in a timely manner. One facility noted that  while they have 7 doctors, they would need 11 additional doctors and  10 technician assistants to process cases through the initial medical  exam and other additional disability specific examinations in a timely  manner. Further, officials at another Army base we visited noted that  there was a shortage of doctors and DOD board liaisons and that they  had difficulty recruiting such staff due to the remote location of the  base.
At all the facilities we visited, officials told us DOD board liaisons and  VA case managers had large case loads. While DOD has established  a goal of 1 board liaison for every 20 servicemembers, the ratios  varied widely by military treatment facility with a range from 1:1 as the  lowest to the highest of 1:75 according to recent data. Because of  high case loads and a reported increase in the complexity of cases,  staff at one facility reported a liaison to servicemember ratio of 1:80  and noted that liaisons must often prioritize cases to deal with the  most pressing issues first. As a result, cases that might otherwise be  quick to process take longer simply because they are waiting to be  processed. Liaisons are often working overtime and weekends to  keep up with cases.
Monthly data produced by DOD subsequent to the data we analyzed  show significantly improved timeliness for the medical exam stage (66  percent of active duty cases met the goal in June 2012) and some  improvement for the MEB stage (40 percent of active duty cases met the  goal in the month of June 2012). However, it is too early to tell whether  these improvements will continue going forward. (See app. III for DOD  reported monthly data, October 2011 – June 2012.)
Since fiscal year 2008, the majority of cases have completed the PEB  phase under the goal of 120 days, however, PEB timeliness has still  worsened over time. In 2011, 78 percent of active duty and 62 percent of  reservist cases that completed the entire IDES process met the PEB goal.
The average processing time was 93 days for active duty  servicemembers and 116 for reservists (see table 2).
Despite meeting the overall PEB goal in fiscal year 2011, established  goals were not met for any of the interim PEB stages, including the  informal PEB and VA rating stages which are the two stages all  servicemembers must complete. For all cases that completed the PEB  phase in fiscal year 2011, only 38 percent of active duty and 38 percent of  reservists’ cases received an informal PEB decision within the 15 days  allotted. Further, only 32 percent of active duty and 27 percent of reservist  cases received a preliminary VA rating within the 15-day goal. (see   table 3).
Regarding delays with the VA rating, VA officials told us that staffing has  been a challenge at their IDES rating sites and that this has slowed case  processing. Monthly data produced by DOD subsequent to the data we  analyzed show similar trends for the informal PEB and VA preliminary  rating stages. As of June 2012 (most recent data available), active duty  cases showed slight improvements in timeliness for the informal PEB  stage (41 percent of cases meeting the goal and processing times  averaging 24 days). The VA rating stage, on the other hand, showed  slight declines in timeliness (31 percent of cases meeting the established  goal and processing times of 35 days) relative to FY 2011 averages for  active duty servicemembers. However, as noted before, it is too early to  tell the extent to which such trends will continue. (See app. III for DOD  reported monthly data, October – June 2012.)
Also during this phase, IDES planners allocated the majority of overall  PEB processing time (75 out of the 120 days) for appeals—including a  formal PEB hearing and a reconsideration of the VA ratings. According to  officials, while the three appeal stages do not happen for every case,  appeals can significantly increase processing times for any one case.  However, only 20 percent of cases completed in fiscal year 2011 actually  had any appeals; calling into question DOD and VA’s assumption on the  prevalence and average effect of appeals, and potentially masking  processing delays in other mandatory parts of the PEB phase.
The transition phase has consistently taken longer than its 45-day goal— almost twice as long on average. While processing times improved  slightly for cases that completed this phase in fiscal year 2011 (from 79  days in fiscal year 2010 to 76 days in fiscal year 2011 for active duty  cases), timeliness has remained consistently problematic since fiscal year  2008 (see table 4).
DOD lacks comprehensive data on how servicemembers spend their time  in the transition phase, which includes many different activities related to  separation from the military. These activities vary widely depending on  the case. For example, during this phase servicemembers receive  mandatory training such as job training through the Transition Assistance  Program and may also receive counseling such as pre-discharge  Vocational Rehabilitation and Employment counseling. In addition,  servicemembers may be placed on temporary duty while house hunting,  or to allow for a servicemember’s children to complete the school year  before moving. Servicemembers may also take earned leave time—to  which they are entitled—before separating from the service. For example,  an Army official said that Army policy allows servicemembers to take up  to 90 days of earned leave prior to separating, and that average leave  time was about 80 days. Because many of these activities can occur  simultaneously or in small intermittent segments of time, DOD officials  said it is difficult to track which activities servicemembers participate in or  determine how much time each activity takes. DOD is exploring options  for better tracking how time is spent in this phase. Because a potentially  substantial amount of the time in this phase may be for the personal  benefit of servicemembers, DOD recently began reporting time in IDES  with and without the transition phase included.
Processing time improved somewhat for the benefits phase (48 days in  fiscal year 2010 to 38 days in fiscal year 2011), but continued to exceed  the 30-day goal for active duty servicemembers (see table 5).
Several factors may contribute to delays in this final phase. VA officials  told us that cases cannot be closed without the proper discharge forms  and that sometimes they do not receive this information in a timely  manner from the military services. Additionally, if data are missing from  the IDES tracking system (e.g., the servicemember already separated,  but this was not recorded in the database), processing time will continue  to accrue for cases that remain open in the system. Officials could not  provide data on the extent to which these factors had an impact on  processing times for pending cases, but said that once errors are  detected and addressed, reported processing times are also corrected.
section_title: Shortcomings in the Design and Administration of Servicemember Surveys Hamper their Usefulness
paragraphs: In addition to timeliness, DOD and VA evaluate IDES performance using  the results of servicemember satisfaction surveys. In principle, all  members have an opportunity to complete satisfaction surveys at the end  of the MEB, PEB, and transition phases; however, under current survey  procedures servicemembers become ineligible to complete a survey for  either the PEB or transition phases if they did not complete a survey in an  earlier phase. Additionally, servicemembers who start but do not  complete a phase are not surveyed. As such, DOD may be missing  opportunities to obtain input from servicemembers who did not complete  a prior survey or exited IDES in the middle of a phase. Further,  response rates may be affected because DOD does not survey  servicemembers once they separate from the service and become  veterans. While it is not necessary for DOD to survey all servicemembers  at the end of every phase, the percentage and characteristics of  servicemembers covered by the survey (i.e., who completed a phase and  were ultimately interviewed) may be insufficient to establish that the  survey results are representative of servicemember satisfaction,  especially for later phases. (See table 6 for response and coverage  rates.) DOD officials recently told us that they will consider alternative  survey eligibility requirements, including working with the Office of  Management and Budget for permission to interview veterans. (For  additional information regarding the timing of the survey, see app. II).
In addition, alternate survey measures show lower satisfaction rates  than those reported by DOD. Using DOD’s measure, we found an  overall satisfaction rate of about 67 percent since the inception of  IDES. DOD defines a servicemember as satisfied if the average of his  or her responses across several surveys is above 3 on a 5-point  scale, with 3 denoting neither satisfied nor dissatisfied. However,  using our alternate measure that defines servicemembers as satisfied  only when all of their responses are 4 or above, we calculated the  satisfaction rate to be about 24 percent (see fig. 9).
Our calculation is a more conservative measure of satisfaction, because it  rules out the possibility that a servicemember is deemed “satisfied” even  when he or she is dissatisfied on one or more questions in the scale.  While not incorrect, DOD’s scale can mask pockets of servicemember  dissatisfaction. For example, an individual may indicate that he or she is  very dissatisfied with one phase of the program, but satisfied with other  phases, and the overall satisfaction score can be the same as one for a  servicemember who is generally satisfied across all phases of the  process. Measuring satisfaction, or even dissatisfaction, in different ways  may provide a more complete picture of satisfaction and how it varies in  different circumstances, and thus may reveal areas where DOD could  focus on improving management and performance.
Finally, using either DOD’s or our calculated measure, we found that  overall satisfaction did not vary much according to differences in the  experiences of servicemembers. For example, our model estimated that  satisfaction varied by no more than approximately five percentage points  across branch, component, disenrollment outcome, sex, MEB exam  provider, enlisted and officer personnel classes, and the number of  claimed and referred conditions. While lack of variation could be a  positive outcome signaling consistent treatment, it could equally mean  that the survey does not measure opinions in enough detail to  discriminate among servicemembers’ experiences. Either way, such  results provide little insight into identifying areas for improvement or  effective practices. Further, while we found some association between  servicemembers satisfaction and the timeliness of their case processing,  we also found many servicemembers were highly dissatisfied even when  their cases were completed on time, and many were highly satisfied even  when their cases were not. For example, 68 percent of those who said  that PEB timeliness was “very poor” completed the phase on time, and 55  percent of those who said that MEB timeliness was “very good” did not  complete on time. The lack of variation and/or correlation between  satisfaction and experiences of servicemembers—coupled with low  coverage rates—raise questions about the value of the survey results as  a performance measure and program evaluation tool. (See app. II for  more information on servicemember satisfaction results.)
DOD is reconsidering its options for measuring customer satisfaction, but  has yet to select a particular approach. As noted above, possible changes  might include widening the criteria for who is eligible for the survey,  modifying survey questions, changing when and how the survey is  delivered, and changing how satisfaction is calculated. Officials already  concluded that the survey, in its current form, is not a useful management  tool for determining what changes are needed in IDES and said that it is  expensive to administer—costing approximately $4.3 million in total since  the start of the IDES pilot. Navy officials told us they believed that the  satisfaction surveys could be made more useful if they knew whether  servicemember’s satisfaction was actually influenced by the  servicemember’s desired or actual outcome of the IDES process. Further,  Army officials already determined that the DOD survey is of limited value,  and are proceeding with plans to field their own survey in the hopes of  obtaining more detailed information at the facility level. Because of fiscal  constraints, DOD suspended the survey in December 2011, but officials  told us that they hope to resume collecting data in fiscal year 2013.
We identified two potential alternatives to assessing servicemember  experiences.
Surveying a sample of servicemembers: While a census gives each  servicemember a chance to describe his or her experiences with  IDES, DOD could collect the same data at a lower cost by surveying a  probability sample of servicemembers. If appropriately designed and  executed, a sample would accurately represent all groups of  servicemembers and produce the necessary data for important  subgroups, such as facilities or branches. Since the cost of  administering a survey is strongly related to the number of people  surveyed, probability sampling could also allow DOD to assess  servicemember experiences while substantially reducing data  collection costs.
Exit interviews: In-depth interviews with servicemembers, completed  at disenrollment from IDES, could also yield more detailed and  actionable information about the program. Although the current survey  includes open-ended questions, it is primarily designed to collect  standardized, quantitative measures of satisfaction with broad aspects  of IDES, such as fairness and the performance of DOD board liaisons  and VA case managers. As a result, the survey provides a limited  amount of detailed feedback on particular facilities, staff members,  and stages of the process that managers might use to improve the  servicemember experience, decrease processing times, or reduce  cost. In contrast, semi-structured exit interviews would allow  servicemembers to provide this type of qualitative, detailed feedback.  Interviewing servicemembers at the end of the process would also  allow servicemembers to assess their overall experiences with IDES  rather than at an earlier stage, without having completed the entire  process. Exit interviews could also reach servicemembers who exit  IDES without completing the process such as those who are returned  to duty. Exit interviews, however, have the potential to be labor  intensive and expensive.
section_title: Recent Actions and Ongoing Initiatives May Improve IDES Performance, but It Is Too Early to Assess Their Overall Impact
subsections: section_title: DOD and VA Took Steps to Address Previously Identified IDES Challenges
paragraphs: DOD and VA have undertaken a number of actions to address IDES  challenges—many of which we identified in our past work. Some  actions—such as increased oversight and staffing—represent important  steps in the right direction, but progress is uneven in some areas.
Increased monitoring and oversight: We identified the need for agency  leadership to provide continuous oversight of IDES in 2008 and the  need for system-wide monitoring mechanisms in 2010. Since then,  agency leadership has established mechanisms to improve  communication, monitoring, and accountability.
The secretaries of DOD and VA have met several times since  February 2011 to discuss progress in improving IDES timeliness and  have tasked their agencies to find ways to streamline the process so  that the timeliness goals can be shortened. The secretaries also  tasked their agencies to expand the use of expedited disability  evaluations for severely combat-wounded servicemembers; and develop a system to electronically transfer case files between DOD  and VA locations.
Senior Army and Navy officials regularly hold conferences to assess  performance and address performance issues, including at specific  facilities. With respect to the Army, meetings are led by the Army’s  vice-chief of staff and VA’s chief of staff, and include reviews of  performance where regional and local facility commanders provide  feedback on best practices and challenges. For example, recent  Army-VA conferences focused on delays in completion of preliminary  ratings for Army PEBs by VA’s Seattle rating site, efforts by the Army  to increase MEB staffing, development of Army-wide IDES  standardization guidance, and Army-VA electronic records  interchange. Periodic meetings are also held between senior Navy  medical and VA officials to discuss performance issues at Navy  military treatment facilities.
VA holds its own biweekly conferences with local staff responsible for  VA’s portion of the process. These conferences are supplemented by a  bi-weekly IDES “dashboard” that tracks performance data for portions of  the IDES for which VA is responsible. According to VA officials, in  addition to identifying best practices, these conferences focus on sites  with performance problems and identify potential corrective actions. For  example, officials said a recent conference addressed delays at Fort  Benning, Georgia, and discussed how they could be reduced. VA officials  noted that examiner staff were reassigned to this site and worked on  weekends to address the problems at this site. In addition, senior VA  health care officials hold periodic conferences with officials responsible  for exams at IDES sites, to monitor performance.
Ensuring sufficient medical exam resources: In our December 2010  report, we noted that VA struggled to provide enough medical examiners  (both VA employees and contractors) to meet demand and deliver exam  summaries within its 45-day goal. For example, significant deficiencies in  examiner staffing (particularly for mental health exams) at Fort Carson  contributed to exams for active duty members taking an average of 140  days. To improve exam timeliness, VA hired more examiners and is  devoting more resources at those sites where VA clinicians perform IDES  exams. In addition, in July 2011, VA awarded a revised compensation  and pension (including IDES) contract that provides more flexibility for VA  to have contractors perform IDES exams at sites needing additional  resources. As a result, VA can use contractors to conduct exams for  regional offices beyond the 10 offices for which the contractor normally  provides services. Also, VA contracted with 5 companies to provide short- term exam assistance at IDES sites needing it. Further, VA procedures  allow reserve component servicemembers in remote locations to receive  exams close to their homes. VA exam timeliness has improved and the  agency met its 45-day goal for active component members in every  month from August 2011 through June 2012. VA officials attributed  improved exam timeliness, in part, to additional exam resources provided  to IDES sites. (See app. III for additional information on fiscal year 2012  timeliness.)
Ensuring sufficient exam summaries: In our December 2010 report, we  noted that some cases were delayed because VA medical exam  summaries were not complete and clear enough for use in making rating  and fitness decisions and needed to be sent back to examiners for  additional work. VA officials told us that they have been reinforcing the  importance of training and communication between rating staff and  medical examiners as ways to improve exam summary sufficiency. For  example, VA identified types of information which, if missing from an  exam summary, would cause it to be insufficient, and has been training  examiners to include such information. Additionally, VA noted that VTA  now has the ability to track cases with insufficient exams by allowing staff  to annotate information on exam summaries. However, staff are not  required to provide this information and rules and procedures for its use  have not been established.
Ensuring sufficient MEB staffing: In our December 2010 report, we noted  that some sites had insufficient MEB physicians, leading to delays in  completing the MEB phase. At that time, most of the 27 pilot sites were  not meeting the 35-day goal, with average times for active component  cases as high as 109 days. Meanwhile, DOD did not have sufficient board  liaison staff to handle IDES caseloads. The Army is in the midst of a  major hiring initiative intended to more than double staffing for its MEBs  over its October 2011 level, which will include additional board liaison and  MEB physician positions. The Army reported having 610 full-time  equivalent MEB staff positions in October 2011, and planned to hire up to  1,410; this would include 172 MEB physician and 513 board liaison  positions. The Army also planned to hire an average of one contact  representative per board liaison; these staff members assist the board  liaisons with clerical functions, freeing more of the liaisons’ time for  counseling servicemembers. As of June 2012, the Army had filled 1,219  (86 percent) of the planned 1,410 positions.
Ensuring sufficient VA rating staff: In our December 2010 report, we  noted that VA had insufficient staff at one of its rating sites to handle the  demand for preliminary ratings, rating reconsiderations, and final VA  benefit decisions. VA officials said that the agency has more than tripled  the staffing at its IDES rating sites–from 78 to 262 positions. Further, VA  has moved staff resources to IDES rating sites from other VA regional  offices to provide short-term help in working down rating backlogs.  Recent monthly data show an increase in the number of preliminary VA  ratings completed, and a slight improvement in processing times.  However, as noted before, it is too early to tell the extent to which such  trends will continue. (See app. III for additional information on fiscal year  2012 timeliness.)
Improving completeness of reserve component members’ records:  Service officials noted that incomplete medical records and administrative  documentation, especially for reserve component members, often  contribute to delays in the early IDES stages, including the VA exam  stage. For example, a reserve unit may not have complete medical  records for a member who received care from a private provider. When  the servicemember enters the IDES, a board liaison is responsible for  obtaining the private provider records before handing off the case to VA  for exams. To address issues with reserve component servicemembers’  records, the Army established an interim office in Pinellas Park, Florida in  January 2011. For reserve component servicemembers who may require  IDES referral, this office is tasked with obtaining records from the  member’s reserve unit; reviewing them to identify missing information;  and, if necessary, requiring the reserve unit to obtain additional records to  complete the case file. Staff at this office also determine whether the  member needs IDES referral. Army officials indicated that this office is  expected to help reduce the backlog of Army reserve component cases in  the IDES. However, Army officials noted that they are providing training to  reserve units to improve their ability to maintain complete records on their  servicemembers and eventually, the Army may discontinue this office if  no longer needed.
Improving MEB documentation and decisions: In response to delays in  completing the MEB stage, the Navy and Army have initiatives underway  to help ensure the timely completion of narrative summaries and MEB  decisions. For example, the Navy piloted electronic narrative summary  preparation at Naval Hospital Camp Lejeune, North Carolina. In May  2012, after determining that the piloted process led to improved MEB  completion timeliness, the Navy deployed electronic narrative summary  preparation Navy-wide. In March 2011, the Army also deployed an  abbreviated MEB narrative summary format, intended to provide better  information for MEB and PEB decision making while helping reduce  delays in the completion of these summaries by MEB clinicians.  Incorporating feedback from its MEBs and PEBs, the Army expects the  revised IDES template to reduce redundant information, make summaries  simpler and easier to use, and standardize summary preparation across  their sites.
Resolving diagnostic differences: In our December 2010 report, we  identified differences in diagnoses between DOD physicians and VA  examiners, especially regarding mental health conditions, as a potential  source of delay in IDES. We also noted inconsistencies among services  in providing guidance and a lack of a tracking mechanism for determining  the extent of diagnostic differences. In response to our recommendation,  DOD commissioned a study on the subject. The resulting report  confirmed the lack of data on the extent and nature of such differences,  and noted that the Army has established guidance more comprehensive  than the guidance DOD was developing. It also recommended that DOD  or the other services develop similar guidance. A DOD official told us that  consistent guidance across the services, similar to the Army’s, was  included in DOD’s December 2011 IDES manual. Also, in response to our  recommendation, VA took steps to modify the VTA database used to  track IDES to collect information on diagnostic differences. The VTA  upgrade was completed in June 2012 after several delays. The report  also recommended that DOD and VA establish a committee to improve  the accuracy of posttraumatic stress disorder ratings. DOD noted that  training on diagnostic differences has been incorporated into its  continuing medical education curriculum for military clinicians, but DOD  considers the issue of posttraumatic stress disorder ratings largely  resolved. Meanwhile, the Army’s new IDES narrative summary template  includes a section where the MEB clinician identifies any inconsistencies  in the case record, including any diagnostic differences with VA  examiners.
section_title: DOD and VA Are Addressing Shortcomings in Information Systems, but Efforts to Date Are Limited
paragraphs: DOD and VA are working to remedy shortcomings in information systems  that support the IDES process. These shortcomings include VTA’s lack of  capability for local sites to track cases, and the potential for erroneous  and missing data in VTA, affecting timeliness measurement. However,  some efforts related to information systems are causing work  inefficiencies, are still in progress, or otherwise are limited.
Improving local IDES reporting capability: DOD and VA are implementing  solutions to improve the ability of local military treatment facilities to track  their IDES cases, but multiple initiatives may result in redundant work  efforts. Officials told us that the VTA—which is the primary means of  tracking the completion of IDES cases—has limited reporting capabilities  and staff at local facilities are unable to use it for monitoring the cases for  which they are responsible. DOD and VA developed VTA improvements  that will allow DOD board liaisons and VA case managers— and their  supervisors—to track the status of their cases. VA included these  operational reporting improvements in its June 2012 VTA upgrade. In the  meantime, staff at many IDES sites have been using their own local  systems to track cases and alleviate limitations in VTA. Further, the  military services have been moving ahead with their own solutions. For  instance, the Army has deployed its own information system for MEBs  and PEBs Army-wide. In addition, DOD has also been piloting its own  tracking system at 9 IDES sites. As a result, staff at IDES sites we  visited reported having to enter the same data into multiple systems. For  example, board liaisons at Army MEBs Fort Meade and Joint Base Lewis- McChord reported entering data into VTA and the Army’s new system,  while board liaisons at Andrews Air Force Base reported entering data  into VTA and DOD’s pilot data system.
Improving IDES data quality: DOD is taking steps to improve the quality of  data in VTA. Our analysis of VTA data identified erroneous or missing  dates in at least 4 percent of the cases reviewed. Officials told us that  VTA lacks adequate controls to prevent erroneous data entry, and that  incorrect dates may be entered, or dates may not be entered at all, which  can result in inaccurate timeliness data. For example, Army officials noted  that some cases shown in VTA as very old were actually closed, but were  missing key dates. In September 2011, DOD began a focused effort with  the services to correct erroneous and missing case data in VTA. Officials  noted that the Air Force and Navy completed substantial efforts to correct  the issues identified at that time, but Army efforts continue. DOD and  Army officials noted that additional staff resources are being devoted to  cleaning up Army VTA data. While improved local tracking and reporting  capabilities will help facilities identify and correct erroneous data, keeping  VTA data accurate will be an ongoing challenge due to a lack of data  entry controls. While DOD is currently assisting the services, DOD  officials said they expect that eventually the services will be responsible  for identifying and fixing data errors.
DOD and VA are also pursuing options to allow them to save time by  replacing the shipping of paper case files among facilities with electronic  file transfers. Requirements for an electronic case file transfer solution  have been completed and DOD and VA officials expect to begin piloting it  in August 2012. As a short-term solution, the Army and VA began using  an Army file transfer Web site to move IDES records between the Army’s  PEBs and the Seattle VA rating site in March 2012. According to VA  officials, this could save several days currently spent shipping paper files  between these offices. VA officials noted that the same Web site is being  used for transfers between the Navy PEB and Providence rating site.  Meanwhile, the secretaries of Defense and Veterans Affairs tasked their  staffs to develop standards for electronic IDES case files by July 2012.
section_title: DOD and VA are Pursuing Broader Solutions to Improving IDES Performance
paragraphs: Based on concerns of the Secretaries of DOD and VA about IDES delays,  the departments have undertaken additional initiatives to achieve time  savings for servicemembers. For example, in response to the secretaries’  February 2011 directive to streamline the process, DOD and VA officials  proposed a remodeled IDES process. In December 2011, senior agency  leadership decided to postpone the pilot of a remodeled IDES process,  and instead tasked the agencies to explore other ways to streamline the  process. As a result, DOD, with VA’s assistance, began a business  process review to better understand how IDES is operating and identify  best practices for possible implementation. This review incorporates  several efforts, including visits to 8 IDES sites to examine how the  process was operating and identify best practices.includes the following:
Process simulation model: Using data from site visits and VTA, DOD  is developing a simulation model of the IDES process. According to a  DOD official, this process model will allow the agencies to assess the  impact of potential situations or changes on IDES processing times,  such as surges in workloads or changes in staffing.
Fusion diagram: DOD is developing this diagram to identify the  various sources of IDES data—including VA claim forms and narrative  summaries—and different information technology systems that play a  role in supporting the IDES process. Officials said this diagram would  allow them to better understand and identify overlaps and gaps in  data systems.
Ultimately, according to DOD officials, this business process review could  lead to short- and long-term recommendations to improve IDES  performance, potentially including changes to the different steps in the  IDES process, performance goals, and staffing levels; and possibly the  procurement of a new information system to support process  improvements. However, a DOD official noted that these efforts are in  their early stages, and thus there is no timetable yet for completing the  review or providing recommendations to senior DOD and VA leadership.  DOD officials indicated that they expect this to be a continuous IDES  improvement process, including further site visits.
Finally, DOD is also developing guidance to expand implementation of an  expedited disability evaluation process for servicemembers with  catastrophic, combat-related conditions by allowing it to be operated at  more military treatment facilities. DOD created this expedited process in  January 2009 for servicemembers who suffer catastrophic, combat- related disabilities. Under an agreement with VA, the services can rate  such members as 100 percent disabled without the need to use VA’s  rating schedule. However, according to DOD officials, the services report  that no eligible servicemembers are using this process. Instead,  servicemembers are having their cases expedited through the IDES  informally. The revisions to DOD’s policy would allow the expedited  process to be used at additional military treatment facilities beyond the  original 4 facilities.of a rewrite of DOD’s key guidance documents, and was undergoing  review at the time of our review.
section_title: Conclusions
paragraphs: By merging two duplicative disability evaluation systems, IDES shows  promise for expediting the delivery of DOD and VA benefits to injured  servicemembers and is considered by many to be an improvement over  the legacy process it replaced. However, nearly 5 years after its inception  as a pilot, delays continue to affect the system and the contribution of  various, complex factors to timeliness is not fully understood. Recent  efforts by DOD and VA to better understand how different IDES  processes contribute to timeliness are promising and may provide the  departments with an opportunity to reassess resource levels and  timeframes, and to make adjustments if needed. This information will also  help to ensure that DOD and VA are making the best use of limited  resources to improve IDES performance. However, it is not clear when  these efforts will be complete or if any recommended actions will be  implemented. DOD has also begun rethinking its approach to determining  servicemember satisfaction with IDES. Our analysis of customer  satisfaction data suggests that there are opportunities for improving the  representativeness of the survey information collected and reconsidering  the cost-effectiveness of the current lengthy surveys. Finally, providing  local facilities the capability to track and generate reports on the status of  their cases is long overdue and may empower local staff to better address  challenges. However, tracking reports are only as good as the data that  are entered into VTA, and DOD and VA can ensure the quality of these  data through continuous monitoring. Meanwhile, the DOD-led business  process review should identify and ultimately eliminate any redundant or  inefficient information systems for tracking cases as well as for other  IDES purposes.
section_title: Recommendations for Executive Action
paragraphs: 1) To ensure that servicemember cases are processed and are awarded  benefits in a timely manner, we recommend that the Secretaries of  Defense and Veterans Affairs work together to develop timeframes for  completing the IDES business process review and implementing any  resulting recommendations.  2) To improve DOD’s ability to measure servicemembers’ satisfaction with  the IDES process, we recommend that the Secretary of Defense develop  alternative approaches for collecting more meaningful and representative  information in a cost effective manner.  3) To ensure that IDES management decisions continue to be based  upon reliable and accurate data, we recommend that the Secretaries of  Defense and Veterans Affairs work together to develop a strategy to  continuously monitor and remedy issues with VTA timeliness information.  This could include issuing guidance to facilities or developing best  practices on preventing and correcting data entry errors; and developing  reporting capabilities in VTA to alert facilities to potential issues with their  data.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DOD and VA for review and  comment. In their written comments, which are reproduced in appendixes  IV and V, DOD and VA both concurred with our recommendations. VA  also provided technical comments, which we incorporated as appropriate.
While concurring with our recommendations, DOD also commented that  our discussion of IDES surveys contained inaccuracies, but did not  specify the inaccurate information in our draft report. In a subsequent  communication, DOD officials noted that our draft inaccurately described  DOD’s decision to not survey veterans. We corrected this information  accordingly. Further, while VA concurred with our recommendation that it  work with DOD to develop timeframes for completing the IDES business  process review and implementing any resulting recommendations, VA  stated that DOD is leading the business process review, and therefore  should develop the timeframes for completing the review. We have  revised this report to clarify that DOD is leading the business process  review, but we did not alter the recommendation because we believe that  it is important for VA to work closely with DOD, including in developing  review timeframes.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to the  appropriate congressional committees, the Secretary of Defense, the  Secretary of Veterans Affairs, and other interested parties. The report is  also available at no charge on the GAO Web site at www.gao.gov.
If you or your staff members have any questions about this report, please  contact me at (202) 512-7215 or at bertonid@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found  on the last page of this report. Staff members who made key contributions  to this report are listed in appendix VI.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: In conducting our review of the Integrated Disability Evaluation System  (IDES), our objectives were to examine (1) the extent to which the  Departments of Defense (DOD) and Veterans Affairs (VA) are meeting  IDES performance goals, and (2) steps DOD and VA are taking to  improve IDES performance. We conducted this performance audit from  May 2011 to August 2012, in accordance with generally accepted  government auditing standards. These standards require that we plan  and perform the audit to obtain sufficient, appropriate evidence to provide  a reasonable basis for our findings and conclusions based on our audit  objectives. We believe the evidence obtained provides a reasonable  basis for our findings and conclusions based on our audit objectives.
subsections: section_title: Review of IDES Timeliness Data
paragraphs: To determine the extent to which IDES is meeting established timeliness  goals, we analyzed data collected through VA’s Veterans Tracking  Application (VTA) database. While VA manages VTA, both DOD and VA  staff enter data into VTA, and the evaluation of IDES data is primarily  conducted by staff at DOD’s Office of Warrior Care Policy (WCP). WCP  provided us with a dataset that was current as of January 1, 2012 and  contained data spanning back to the inception of IDES in late 2007. This  data export included data on a total of 39,260 cases. Of these cases,  34,185 were active duty servicemembers and 5,068 were Reserve/Guard  servicemembers. This VTA data set contained demographic data for  each individual IDES case as well as a record of dates for when  servicemembers reached various milestones in IDES. Overall and  interim IDES timeliness calculations are based on computing the number  of days elapsed between appropriate milestone dates. For example,  overall timeliness for servicemembers that receive benefits is calculated  as the number of days between the individual being referred into the  IDES and the date on which his or her VA benefits letter is issued. We  met with staff at WCP to ensure we used appropriate variables when  calculating timeliness. We also met with officials at VA to discuss the  calculations used to determine the timeliness of cases.
We took a number of steps to assess the reliability of VTA data and  ultimately found the data to be sufficiently reliable for the purposes of our  audit. Past GAO worknumber of steps to follow up on past assessments of VTA.   relied on VTA data, and therefore we took a
We interviewed DOD and VA and determined that internal controls on  VTA data had not changed substantially since our past review.
We conducted electronic testing of the VTA data and generally found  low rates of missing data or erroneous dates pertinent to our  analysis—approximately 4 percent of cases.  For IDES cases in  which we found missing dates or dates out of sequence, we excluded  those cases from all of our analyses. While there were some  instances in which the erroneous dates may be justified, we excluded  the entire case from our analysis if any such dates appeared at any  point in the VTA database. Such data included cases in which (1)  there was no MEB referral date signifying the start of IDES process,  and (2) the ending date preceded the beginning date of the IDES  phase (resulting in timeliness calculations appearing as a negative  amount of time).
We also conducted a limited trace-to-file process to determine  whether date fields in VTA were an accurate reflection of the  information in IDES case files. Specifically, we compared VTA dates  in 15 IDES cases completed in fiscal year 2011 against the dates in  the corresponding paper files. In comparing dates, we allowed for a  discrepancy of 5 days in dates to allow for the possibility that dates  may have been entered into the database after an event took place.  Ninety-three percent of the dates we traced back to the original file  documents were found to be accurate, that is falling within our 5 day  allowance.
section_title: Analysis Conducted Using VTA Data
paragraphs: For the cases meeting our criteria for reliability, we analyzed timeliness  data for those cases that had completed the entire IDES process or had  completed each of the four IDES phases. We specifically:  Identified the total number of cases enrolled each fiscal year from FY  2008 through 2011, by active as well as National Guard and reserve  servicemembers, and by military branch of service.
Identified the number of cases that completed the entire IDES process  for each fiscal year from fiscal year 2008 through fiscal year 2011. We  analyzed completed cases in two different ways: (1) those who  completed the process and received VA benefits and (2) those who  completed the IDES with any outcome (such as permanent  retirement, Temporary Disability Retirement List, return to duty, etc.).  In order to be able to make comparisons across cases with different  outcomes for a given point in time, we defined fiscal year by using the  VTA variable “final disposition date”. We did this because most  completed cases—regardless of outcome—have a final disposition  date in VTA. In contrast to our approach, VA use the “VA benefit date”  variable to determine fiscal year of completion for cases resulting in  benefits. As such, their number of cases and timeliness calculations  by fiscal year differed from ours, although overall trends are similar.
Identified the number of cases that completed each phase of IDES  and the interim stages within each phase, again by fiscal year (fiscal  years 2008 through 2011).
Computed timeliness statistics for the completion of the IDES  process, phases, and stages against the performance goals set by  DOD and VA, such as average days and percent meeting goals.
Computed number and percent of cases where a servicemember  appealed a decision made during the IDES process, by fiscal year.
For the purposes of this report, GAO opted to not include reserve  component time spent in the VA benefit phase in our calculations for  overall time because the 30 days allotted for this phase is not included in  the 305-day overall goal for the reserve component.
GAO also performed analyses similar to those above, except that we  grouped cases according to the year in which they were enrolled in IDES.  (See app. II for more detail on these analyses.) Additionally, we analyzed  timeliness for cases that had not yet completed the MEB stage as of the  date we received the VTA data.
section_title: Review of IDES Customer Satisfaction Data
paragraphs: To determine the extent to which IDES is meeting its customer  satisfaction goals, we analyzed data collected from IDES customer  satisfaction surveys conducted at the end of three phases: MEB, PEB  and Transition. These surveys are administered by telephone by  contractors hired by DOD. The dataset we received contained survey  responses for individual servicemembers from the beginning of the IDES  pilot to December 2011, at which time administration of the survey was  suspended. Additionally, we matched individual survey responses with  information from VTA to gain additional understanding into how customer  satisfaction varied according to different factors such as timeliness and  case outcome. We matched survey and VTA data using the unique case  identifier attached to each IDES case, maintaining the anonymity of the  servicemembers. See appendix II for the results of additional analyses we  conducted using survey data and survey data matched with VTA data.
In the course of our review we concluded that the survey data were  sufficiently reliable for our purposes. We interviewed relevant officials at  DOD and their contractors about eligibility requirements and the  administration of the surveys. Further, we met with DOD and their  contractors on multiple occasions to discuss the calculations used to  determine response rates for the survey and servicemembers’ level of  satisfaction. See appendix II for more details on GAO’s review of  response rates.
section_title: Identifying Challenges and Actions Taken to Improve Performance
paragraphs: To identify challenges in implementing IDES as well as steps taken to  improve performance, we visited six military treatment facilities. During  the site visits, we interviewed officials involved in implementing IDES from  both DOD and VA, including military facility commanders and  administrators, DOD board liaisons, military physicians involved in MEB  determinations, DOD legal staff, VA case workers, VA or contract  examiners, and administrators at VA medical clinics and regional offices.  Additionally, we interviewed servicemembers who were currently enrolled  in the IDES process. We selected the six facilities to obtain perspectives  from sites in different military services, geographic areas, and their ability  to meet timeliness goals for different phases of the process (see table 7).  In addition, we visited the Air Force’s Formal Physical Evaluation Board at  Randolph Air Force Base, Texas. During this visit we observed a hearing  and met with board members to obtain a better understanding of the  process.
section_title: Appendix II: Additional Timeliness and Satisfaction Analyses
paragraphs: This appendix provides additional information on the timeliness of the  IDES process and servicemember satisfaction with it. First, we use  timeliness data to examine whether changes over time in processing  times and the percentage of cases meeting timeliness goals look any  different when cases are grouped according to the fiscal year in which the  cases were first enrolled rather than the fiscal year in which the cases  were completed. Second, we use survey data to examine different  measurements of servicemember satisfaction with IDES, how satisfaction  varied according to various servicemember characteristics, response and  coverage rates for the servicemembers surveyed, and how the survey  respondents differed from nonrespondents.
With respect to timeliness, we find generally similar trends for cases  grouped by fiscal year of enrollment versus fiscal year of completion, with  some key differences. Organizing cases by completion date results in  shorter average processing times in 2008, since only those cases that are  processed quickly could be completed in the first year of IDES. As such,  organizing cases by enrollment date provides a better estimate of the  processing times for the early IDES cases. However, this approach  results in shorter processing times in 2011, the most recent full year of  the program, since only cases that finish quickly can be analyzed.
With respect to satisfaction, we find that the particular index used to  summarize servicemembers’ responses can affect the proportion reported  as being “satisfied” or “dissatisfied” with IDES overall. DOD’s index  suggests that 67 percent of servicemembers have been satisfied since  the IDES program began, but a reasonable alternative measure we  developed suggests that only 24 percent of servicemembers have been  satisfied. Using this measure, satisfaction varies only slightly across many  important groups of servicemembers, such as by disenrollment outcome,  suggesting that available program data cannot precisely explain  satisfaction outcomes. Also, servicemembers surveyed may not represent  the servicemembers who completed the different phases of IDES well  enough to generalize to them, given the low response rates to the MEB  survey and fact that being selected for latter (PEB and Transition) surveys  were conditional on completing the MEB survey.
subsections: section_title: Overall IDES Timeliness: Processing Times and Percent of Cases Meeting Timeliness Goals
paragraphs: Average IDES processing times for completed cases resulting in benefits  generally worsened since 2008, especially for active duty cases,  regardless of whether cases are grouped by the fiscal year in which they  were completed (fig. 10) or by the fiscal year in which they were enrolled  (fig. 11). The notable exception is when fiscal year 2011 is the year of  enrollment. However, caution must be used when examining cases  enrolled in 2011 because over 15,600 service members of the 18,651 (or  at least 84 percent) who entered IDES in fiscal year 2011 did not have an  outcome in 2011 and were enrolled in IDES as of January 1, 2012,  potentially changing the distribution of processing times as they proceed  through IDES.
We also examine average IDES processing times according to year of  completion (see fig. 12) and year of enrollment for cases (see fig. 13) for  all completed cases regardless of outcome. As with cases that resulted in  benefits, for cases resulting in any outcome we find that average  processing times increased since 2008—again with the exception of fiscal  year 2011 for reasons discussed earlier—although average processing  times are somewhat shorter than when only servicemembers receiving  benefits are included (fig. 11).
Figures 14 and 15 show that regardless of whether cases are organized  by year of completion or enrollment, the percent of completed cases  resulting in benefits that were not timely increased between fiscal year  2008 and 2010 for both active duty servicemembers and members in the  Reserves or National Guard. As with the average processing times,  caution must be used when examining cases enrolled in fiscal year 2011  (fig. 15), since only those cases that are processed quickly are observed  in the last year. Similarly, caution also must be used when examining  cases in 2008 (fig. 14), since the only cases that are included in the first  year are those that completed IDES quickly.
Figures 16 and 17 show how average processing times for each of the  four phases of IDES have changed over the four fiscal years when cases  are grouped by the fiscal year in which they completed a given phase and  when cases are grouped by the fiscal year in which they were enrolled or  started a given phase.according to the fiscal year in which the different phases were completed,  processing times increased for all phases except the Transition phase.  Figure 17 shows a roughly similar pattern of increases in processing  times in all but the Transition phase, though processing times in 2011 are  skewed for the reason mentioned above. Figures 18 and 19 show how  the percentages meeting the timeliness goals for each of the four phases  of IDES have changed over the four fiscal years when cases are grouped  by the fiscal year in which they completed a given phase and when cases   Figure 16 shows that when cases are grouped  are grouped by the fiscal year in which they were enrolled or started a  given phase. Figure 18 shows that the percent of cases meeting  timeliness goals decreased over the four years for the MEB and PEB  phases, although a high percent of cases met PEB goals. However, the  Transition and Benefits phases fluctuated up and down and both were  favorable across some years. Figure 19 also shows decreases in  percentages of cases meeting timeliness goals at the MEB and PEB  phases when cases are grouped by fiscal year of starting a given phase.  The fluctuations in the timeliness of the Transition and Benefits phases  were more prevalent when cases were grouped in this manner.
Low response and coverage rates for servicemember satisfaction surveys  administered after each phase of IDES raise concerns about how well the  satisfaction survey results represented the larger population of  servicemembers who completed one or more phases.
DOD surveys servicemembers after they complete the MEB, PEB, and  Transition phases of IDES. The department attempts to survey all  servicemembers who complete each phase, but only if they completed  the prior surveys. For example, the MEB survey must be completed  before a servicemember is eligible to complete the PEB survey.
Using the data available to us, and as table 8 below shows, we found that  9,604 of the 25,212 servicemembers who completed the MEB phase  were surveyed, for a 38 percent response and coverage rate.18,296 servicemembers who completed the PEB phase, only 8,968 of  them completed the prior MEB survey and were eligible for the PEB  survey and of these only 4,795 were surveyed. Using DOD’s eligibility  criteria, the response rate for the PEB survey was roughly 54 percent  (4,795 of 8,968). However, the coverage rate for all servicemembers who  completed the PEB phase (regardless of whether they completed the  prior survey) was only 26 percent (4,795 of 18,296). Similarly, the  response rate for the Transition survey was 72 percent while the  coverage rate was only 23 percent (See table 8).
As table 9 below shows, there were some sizable differences between  respondents and nonrespondents, especially for the PEB and Transition  surveys. For example, respondents to the transition survey spent more  time than nonrespondents in the Transition phase, were less likely to be  separated with benefits, and were more likely to be placed on the  Permanent Disability Retired List. These differences, combined with low  response and coverage rates, raise the possibility of biased responses.
The particular measure used to assess servicemember satisfaction can  affect the proportion reported as “satisfied” with the IDES program.  Depending on the measure used, satisfaction is about 2.8 times lower  than what DOD has reported, and many servicemembers classified as  “satisfied” express moderate dissatisfaction with some aspects of the  process.
DOD has reported average servicemember satisfaction with IDES overall  and with three phases of the process, i.e., MEB, PEB, and Transition  phases. In so doing, DOD has developed indices of satisfaction on  several broad dimensions, such as satisfaction with the overall  experience and fairness, which combine responses to selected survey  questions. Although the number of questions used in each index vary  depending on the number of phases completed, each index classifies  servicemembers as “satisfied” or “dissatisfied” using the average of their  responses across all questions in the index. Each question’s scale ranges  from 1 to 5, with 1 denoting “very dissatisfied” (or a similar negative  response), 5 denoting “very satisfied,” and 3 denoting “neither satisfied  nor dissatisfied.” DOD reports that a servicemember is “satisfied” if his or  her average response across all items in the scale exceeds 3. Table 10  summarizes the responses to each question that DOD uses in its overall  satisfaction index at each phase (as of August 2011).
DOD’s indices are one reasonable method of summarizing  servicemember opinions. In quarterly performance reports, DOD notes  that it has used factor analysis, a form of latent variable statistical models  to assess the reliability of its scales. While we did not review DOD’s  models, we independently found that DOD’s overall index of satisfaction  with IDES was highly reliable. (Specifically, using Cronbach’s alpha, the  index was highly correlated with a single latent dimension at α = 0.92.)  This supports DOD’s choice to measure the single concept of  “satisfaction” by averaging the ordinal servicemember responses.
Nevertheless, the average survey response can obscure variation in the  responses that make up the index. For example, suppose that a  servicemember said she was “very satisfied” (response of 5) on two of the  four questions in the index, “dissatisfied” on one (response of 2) and “very  dissatisfied” on the last one (response of 1). With an average response  over 3, the DOD measure would classify her as “satisfied,” despite the  fact that she was “somewhat dissatisfied” or “very dissatisfied” with two of  the four aspects of IDES that DOD considers important. The grouping rule  considers this servicemember equally happy with IDES as someone who  says they are “satisfied” with all four aspects in the index.
To assess the extent to which DOD’s index might mask dissatisfaction,  we calculated the proportion of questions in the scale on which  servicemembers whom DOD classified as “satisfied” gave neutral or  negative responses (1, 2, or 3). We found that half of these  servicemembers gave neutral or negative answers to at least 25 percent  of the items in the index, and a quarter gave such answers to at least 41  percent of the items. For these servicemembers, the DOD index may  suggest more satisfaction than the underlying survey questions would  support.
We further assessed the sensitivity of DOD’s index by comparing it  against a different (i.e., GAO’s) measure of satisfaction: whether a  servicemember is “somewhat” or “very satisfied” (or gives a similarly  positive response) on all items in DOD’s scale of overall IDES  satisfaction. Our measure is more conservative than DOD’s, because  ours only includes positive responses and uses a broader cutpoint (two  response categories) to distinguish between “satisfied” and “not satisfied”  servicemembers. (In contrast, DOD calculates average satisfaction on an  ordinal scale of 1 to 5, and then uses a cutpoint at 3.) Our measure is not  inherently more valid, however, and has its own weaknesses. In  particular, we classify a servicemember as “not satisfied” if she gives a  neutral or negative response to just one of the four items in DOD’s scale.
When we analyzed overall satisfaction using both measures, we found  that overall, servicemembers are 2.8 times less satisfied on our measure  than on DOD’s (i.e., 23.8 versus 67 percent). Further, only about 20 to 30  percent of servicemembers are “satisfied” with each aspect of the IDES  process that DOD considers important across most of the subgroups we  analyzed, while DOD classifies about 60 to 70 percent of such  servicemembers as “satisfied” on average. In the next section, we present  further information on variation in satisfaction across servicemember  groups.
section_title: Explaining Servicemember Satisfaction Outcomes
paragraphs: Although the servicemember survey provides numerous measures of  satisfaction, it is also important to explain variation in satisfaction  outcomes—i.e. why some servicemembers are more satisfied than  others. Explaining variation can connect dissatisfaction with poor program  performance and help identify specific reforms to improve the  experiences of servicemembers who typically have been less satisfied.  However, the available program data cannot precisely explain outcomes  when used in this type of explanatory analysis. Using the available data,  we could predict satisfaction only 1.9 percentage points better after  controlling for multiple factors than what we would have achieved by  chance (65.5 percent vs. 63.7 percent of satisfied responses predicted  correctly).
In order to further explain variation in satisfaction, we matched the survey  responses to the data that DOD and VA maintain on the processing of  each servicemember’s case, known as the VTA data. This database  primarily measures the time it took servicemembers to complete each  phase of the IDES process. A small number of other program and  demographic variables are also available, such as service branch,  component, and the number of conditions claimed and referred. Using the  matched survey and VTA data, we estimated the association between  satisfaction and observable factors that could potentially explain variation  in servicemembers’ experiences.
Table 11 below (columns 2-4) presents these associations for both DOD’s  and GAO’s overall measures of satisfaction. The “raw data” estimates are  simply the proportion of servicemembers in a particular group who were  satisfied according to either measure. In the fourth column (“model  estimates”), we estimate this proportion holding constant all of the other  factors listed, using a statistical model. Specifically, the estimates are in- sample mean predicted probabilities of giving a satisfied response on the  GAO satisfaction index from a logistic model of satisfaction. The  covariates are given by indicators of whether the servicemember  belonged to each group in column 4. The maximum likelihood estimators  allowed the probability of satisfaction, given the covariates, to be  dependent across observations within the 26 cross-classified groups of  PEB location and MEB medical treatment facility. This adjusted for the  possibility that servicemembers were similarly satisfied if they were  processed in the same locations, given similar values on the observed  covariates.
Regardless of which measure is used (DOD’s or GAO’s), satisfaction  varied only modestly across many important groups of servicemembers.  Our model estimates that the GAO measure of satisfaction varied by no  more than approximately five percentage points across branch,  component, disenrollment outcome, sex, MEB exam provider, enlisted  and officer personnel classes, and the number of claimed and referred  conditions, although differences across MEB treatment facilities and PEB  locations were larger. This can be seen as a positive outcome, if this  correlation implies that DOD and VA administer the program consistently  across servicemembers and locations. However, the lack of variation also  could suggest that the survey items do not measure opinions in enough  detail to discriminate among servicemembers’ experiences.
Also shown in table 11, satisfaction had a stronger association with case  processing time (time spent in IDES) than some of the other factors we  examined. Servicemembers whose case processing times were among  the quickest 25 percent were about 2.3 times as likely to be satisfied (on  the GAO scale) than those whose times were among the 25 percent of  cases with the longest overall timeframes (i.e., 41 versus 18 percent).  Nevertheless, only 41 percent of those servicemembers whose cases  were processed most quickly were satisfied (holding constant the other  factors). This suggests that servicemembers’ opinions about IDES may  be only loosely related to the amount of time they spent in IDES, as  discussed in the next section below.
Although the average case processing time has generally increased since  2008, when we look at satisfaction by fiscal year, servicemember  satisfaction shows evidence of improvement since fiscal year 2008.  Specifically, our measure of satisfaction from the model increased by 15  percentage points since 2008, roughly doubling from 13 to 28. Because  the model estimates control for various other factors, these results  suggest that servicemember views of the IDES process have improved  over time, rather than the possibility that IDES has simply processed  different types of cases.
Satisfaction does not vary by a large amount across many MEB treatment  facilities, but there are exceptions. Our model estimates that about 18 to  26 percent of servicemembers were satisfied at most facilities. However,  there were pockets of greater satisfaction. Specifically, servicemembers  had more positive experiences at Forts Belvoir, Bragg, Campbell, Drum,  Hood, and Polk, with satisfaction estimated to have ranged from 28 to 45  percentage points. Fort Meade had the lowest satisfaction at 15 percent.  These estimates hold constant time spent in IDES and other factors in  column 4 and, thus, partially account for the types of cases each facility  processes.
DOD and VA measure IDES timeliness directly in VTA and as part of the  overall servicemember satisfaction scale. These overlapping measures  let us compare servicemembers’ opinions to their actual experiences in  the program. To do this, we calculated processing times at each phase of  IDES for servicemembers who expressed varying degrees of satisfaction  with the timeliness of their case processing at that phase. In addition, we  analyzed whether servicemembers who were satisfied with the overall  IDES process were more or less likely to meet timeliness goals. Table 12  provides these statistics.
As shown in table 12, satisfaction generally stayed the same or  decreased as processing times increased. The median days spent in the  MEB and PEB phases were 35 and 38 percent lower, respectively,  among those servicemembers who said that MEB and PEB timeliness  was “very good” as compared to those who said it was “very poor.” The  former group was 170 percent more likely to have met the MEB  timeliness goal and 30 percent more likely to have met the PEB  timeliness goal. Similarly, the case for a median servicemember—whom  we classified as “satisfied” with the overall IDES process—was completed  15 percent more quickly and was 49 percent more likely to have met the  timeliness goal than the median servicemember who was “dissatisfied.”  The model estimates in table 11 confirm that the GAO measure of  satisfaction and timeliness (time spent in IDES) are negatively related  even when holding constant several other variables.
Perceived and actual timeliness had little association at the Transition  phase. Across all levels of satisfaction with timeliness, the median  processing time varied by no more than 4 days, and the proportion  meeting the timeliness goal varied by no more than 4 percentage points.  The use of personal leave is one plausible explanation for the  unresponsiveness of servicemember satisfaction to actual processing  times in the Transition phase. A servicemember might not have been  dissatisfied with delays if taking leave was the reason, rather than the  IDES process itself.
Despite the associations between actual and perceived timeliness at the  MEB and PEB phases, there were many servicemembers who were  satisfied or dissatisfied with timeliness that spent similar amounts of time  in the program. For example, 68 percent of those who said that PEB  timeliness was “very poor” completed the phase on time, and 55 percent  of those who said that MEB timeliness was “very good” did not complete  on time. Among servicemembers who said that MEB timeliness was “very  good,” the middle 80 percent of processing times ranged from 62 days to  223 days. The same range for servicemembers who said MEB timeliness  was “very poor” was 88 to 323 days. As table 12 shows, a similar pattern  holds for the PEB phase. Although servicemembers tend to be more  satisfied in MEB and PEB when their cases take less time, many of them  are highly dissatisfied even when their cases take an unusually short  amount of time (and vice versa). In the Transition phase, however, 40  percent of servicemembers who said that timeliness was “very good”  were processed in 91 to 657 days—a more lengthy range than at the  other phases. The large range and relationship with satisfaction may  reflect the use of servicemember leave.
The fact that many servicemembers are similarly satisfied with timeliness,  even though they can have widely different processing times, has broader  implications for measuring the performance of IDES. DOD’s timeliness  goals may not be meaningful to servicemembers or necessarily reflect  high-quality service. Alternatively, servicemembers may not use  reasonable standards to assess the time required to process their cases,  or they may not accurately perceive the time they have spent in the  program. In these scenarios, the value of servicemember satisfaction as a  performance measure becomes less certain.
The relationship between perceived and actual timeliness may simply  reflect a large amount of unobserved heterogeneity across  servicemembers. For example, a servicemember whose case has been in  IDES for an extremely long time might still be highly satisfied with  timeliness if the case was complex or personal leave was taken during  the process. Neither the survey nor the VTA data measure these or other  such characteristics that might affect the program’s key performance  measures.
section_title: Factors Affecting Survey Results and Implications for Program Evaluation
paragraphs: The lack of variation in satisfaction across servicemember groups and  according case timeliness might be seen as a positive outcome, and may  suggest that DOD and VA administer the program consistently across  servicemembers and locations. However, the lack of variation also could  suggest shortcomings in the design and administration of the survey, or in  data limitations that, alone or together, may reduce the usefulness of  survey data for program evaluation. For example:
Survey questions: The survey questions may not be sufficiently  detailed to measure important differences among  servicemembers’ experiences. For example, the survey includes  12 questions (4 per survey) that measure broad opinions about  IDES, and DOD subsequently averages these responses  together. This approach may limit the survey’s capacity to  describe IDES experiences in sufficient detail.
Precision of DOD indices: DOD reports measures of overall  satisfaction with IDES for each phase, using the questions in table  9. However, these measures include one question that asks  respondents to “evaluate their overall experience since entering  the IDES process,” which could be influenced by experiences in  prior phases. Consequently, the satisfaction measures reported  for each phase could represent a combination of servicemembers’  experiences in that phase and prior phases.
Completing two surveys at once: DOD officials told us that a  servicemember may be surveyed for the PEB and Transition  phases in one session. In these instances a large amount of time  may have passed since the servicemember completed the PEB  phase and it may be more difficult for the servicemember to  isolate his or her satisfaction with a particular phase.
Survey design: The satisfaction survey is primarily designed to  measure performance, not explain it. The survey includes many  highly correlated questions measuring satisfaction with the overall  process or broad components of it, such as DOD board liaisons,  VA case managers, or timeliness. While multiple questions can  improve the statistical reliability and validity of DOD’s performance  measures, they require costly survey administration time that  could be used for other purposes, such as to measure a larger  number of variables that could explain servicemember satisfaction  or case processing times.
VTA data limitations: The VTA administrative data that we  matched to survey data primarily measure processing times and  basic servicemember demographics, such as service branch,  component, and treatment facility. The data support detailed  reporting of performance measures, but they do not measure  similarly detailed information on the nature of each case that might  allow DOD and VA to understand the reasons for lengthy case  processing times or to identify cases that might become delayed  and ensure that they remain on schedule. For example, the  database does not measure the type or severity of referred  medical conditions in detail, the nature of delays experienced  early in the process, or the use of servicemember leave. In  addition, little information is available on staffing at or caseloads  for MEB and PEB locations, DOD board liaisons, or VA case  managers, which might help to explain or predict performance.
Low response and coverage rates: The response and coverage  rates of the satisfaction survey further limit the degree to which  DOD can generalize the data obtained to the population of  servicemembers who participate in IDES. In particular, the survey  does not assess the views of servicemembers who disenroll from  the process before finishing a stage or those who do not complete  prior waves of the survey. Including servicemembers who do not  complete all waves would complicate longitudinal analysis,  however.
section_title: Appendix III: Monthly DOD Timeliness Data for Active Duty Cases in Fiscal Year 2012
paragraphs: Table 13 presents data reported by DOD on average processing time for  active duty cases completed during part of fiscal year 2012—Oct. 2011 to  June 2012. DOD’s data are provided as a supplement the analyses GAO  conducted for fiscal years 2008 through 2011. We did not evaluate the  reliability of these data and cannot predict the extent to which any trends  will continue for the rest of the fiscal year.
section_title: Appendix IV: Comments from the Department of Defense
section_title: Appendix V: Comments from the Department of Veterans Affairs
section_title: Appendix VI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: Michele Grgich (Assistant Director), Daniel Concepcion, Melissa Jaynes,  and Greg Whitney made significant contributions to all aspects of this  report. Also contributing to this report were Bonnie Anderson, James  Bennett, Mark Bird, Joanna Chan, Brenda Farrell, Jamila Jones Kennedy,  Douglas Sloane, Almeta Spencer, Vanessa Taylor, Jeffrey Tessin, Roger  Thomas, Walter Vance, Kathleen van Gelder, and Sonya Vartivarian.
section_title: Related GAO Products
paragraphs: Military Disability System: Preliminary Observations on Efforts to Improve  Performance. GAO-12-718T (Washington, D.C.: May 23, 2012).
Military and Veterans Disability System: Worldwide Deployment of  Integrated System Warrants Careful Monitoring. GAO-11-633T  (Washington, D.C.: May 4, 2011).
Military and Veterans Disability System: Pilot Has Achieved Some Goals,  but Further Planning and Monitoring Needed. GAO-11-69 (Washington,  D.C.: December 6, 2010).
Military and Veterans Disability System: Preliminary Observations on  Evaluation and Planned Expansion of DOD/VA Pilot. GAO-11-191T  (Washington, D.C.: November 18, 2010).
Veterans’ Disability Benefits: Further Evaluation of Ongoing Initiatives  Could Help Identify Effective Approaches for Improving Claims  Processing. GAO-10-213 (Washington, D.C.: January 29, 2010).
Recovering Servicemembers: DOD and VA Have Jointly Developed the  Majority of Required Policies but Challenges Remain. GAO-09-728  (Washington, D.C.: July 8, 2009).
Recovering Servicemembers: DOD and VA Have Made Progress to  Jointly Develop Required Policies but Additional Challenges Remain.  GAO-09-540T (Washington, D.C.: April 29, 2009).
Military Disability System: Increased Supports for Servicemembers and  Better Pilot Planning Could Improve the Disability Evaluation Process.  GAO-08-1137 (Washington, D.C.: September 24, 2008).
DOD and VA: Preliminary Observations on Efforts to Improve Care  Management and Disability Evaluations for Servicemembers.  GAO-08-514T (Washington, D.C.: February 27, 2008).
DOD and VA: Preliminary Observations on Efforts to Improve Health Care  and Disability Evaluations for Returning Servicemembers. GAO-07-1256T  (Washington, D.C.: September 26, 2007).
Military Disability System: Improved Oversight Needed to Ensure  Consistent and Timely Outcomes for Reserve and Active Duty Service  Members. GAO-06-362 (Washington, D.C.: March 31, 2006).