Military Disability System: Improved Monitoring Needed to Better Track and Manage Performance
Released: 2012-09-19T00:00:00

Letter

Background

The Disability Evaluation Process

The IDES process begins at a military treatment facility when a physician identifies one or more conditions that may interfere with a servicemember’s ability to perform his or her duties. The process involves four main phases: the Medical Evaluation Board (MEB), the Physical Evaluation Board (PEB), transition out of military service (transition), and VA benefits.

MEB phase: In this phase, medical examinations are conducted and decisions are made by the MEB regarding a servicemember’s ability to continue to serve in the military. This phase involves four stages: (1) the servicemember is counseled by a DOD board liaison on what to expect during the IDES process; (2) the servicemember is counseled by a VA case manager on what to expect during the IDES process and medical exams are scheduled; (3) medical exams are conducted according to VA standards for exams for disability compensation by VA, DOD, or contractor physicians, and (4) exam results are used by the MEB to identify conditions that limit the servicemember’s ability to serve in the military.an impartial medical review by a physician not on the MEB, or both.

Also during this stage, a servicemember can seek a rebuttal, or PEB phase: In this subsequent phase, decisions are made about the servicemember’s fitness for duty, disability rating and DOD and VA disability benefits, and the servicemember has opportunities to appeal those decisions. This includes: (1) the informal PEB stage, an administrative review of the case file by the relevant military branch’s PEB without the presence of the servicemember; (2) VA rating stage, where a VA rating specialist prepares a rating that covers the conditions that DOD determined made a servicemember unfit for duty and any other conditions claimed by the servicemember to VA. This rating is prepared for use by both agencies in determining disability benefits. In addition, servicemembers have several opportunities to appeal different aspects of their disability evaluations: a servicemember dissatisfied with the decision on whether he or she is fit for duty may request a hearing with a “formal” PEB; a servicemember who disagrees with the formal PEB fitness decision can, under certain conditions, appeal to the reviewing authority of the PEB; and a servicemember can ask for VA to reconsider its rating, but only for conditions found unfitting by the PEB.

Transition phase: If the servicemember is found unfit to serve, he or she enters the transition phase and begins the process of separating from the military. During this time, the servicemember may take accrued leave. Also, DOD board liaisons and VA case managers provide counseling on available benefits and services, such as job assistance.

VA benefits phase: A servicemember found unfit and separated from service becomes a veteran and enters the VA benefits phase. VA finalizes its disability rating after receiving evidence of the servicemember’s separation from military service. VA then starts to award monthly disability compensation to the veteran.

DOD and VA established timeliness goals for the IDES process to provide VA benefits to active duty servicemembers within 295 days of being referred into the process, and to reserve component members within 305 days (see fig. 1). DOD and VA also established interim timeliness goals for each phase and stage of the IDES process. The overall timeframes are intended to represent an improvement over the legacy disability evaluation system, which was estimated to take 540 days to complete.

In addition to timeliness, the agencies also established a performance goal of having 80 percent of servicemembers satisfied with the IDES process. DOD measures satisfaction through surveys conducted after the completion of the MEB, PEB, and transition phases. Each survey consists of approximately 30 questions, including 4 questions that ask about the servicemember’s satisfaction with the overall IDES process up to that point. Reported satisfaction rates for each phase are based on an average of responses to these four questions, and reported overall satisfaction with IDES (which is used to track the percent satisfied under the performance goal) is an average of satisfaction rates for the three phases.

Rollout of IDES and Enrollment

From the original 3 pilot military treatment facilities in the Washington, the IDES has expanded to 139 military treatment facilities in D.C., area,the U.S. and several other countries. DOD and VA first added 24 military treatment facilities to the pilot in fiscal years 2009 and 2010, bringing the pilot total to 27. In 2010, DOD and VA leadership decided to implement the IDES world-wide, and did so in 4 stages between October 2010 and September 2011, adding 112 military treatment facilities. As IDES expanded, the number of new cases enrolled in IDES has also increased, totaling 18,651 in fiscal year 2011 (see fig. 2).

IDES caseloads vary by service, but the Army manages the bulk of IDES cases. Of new cases referred to IDES in fiscal year 2011, about 64 percent were in the Army, and much of the growth in caseload has been in the Army. Additionally, active duty servicemembers make up the majority of IDES cases, with about 88 percent of new cases in fiscal year 2011 involving this group (see fig. 3).

IDES Processing Times Increased Over Time, While Measures of Servicemember Satisfaction Have Shortcomings

Overall IDES Case Processing Times Steadily Increased Since the Start of IDES

IDES timeliness has worsened since the inception of the program. Since fiscal year 2008, the average number of days for servicemember cases to be processed and receive benefits increased from 283 to 394 for active duty cases (compared to the goal of 295 days) and from 297 to 420, for reserve component cases (compared to the goal of 305 days) (see fig. 4).

Along with increasing average processing times, the percent of IDES cases awarded benefits within timeliness goals has steadily declined. DOD’s and VA’s current goal is to complete 60 percent of IDES cases on time. In fiscal year 2008, an average of 63 percent of cases for active duty servicemembers and 65 percent for reservists completed the process and received benefits within the timeliness goals; by fiscal year 2011 this was down to 19 and 18 percent respectively (see fig. 5). These trends also hold when considering all cases that completed the IDES process regardless of outcome, although overall processing times were shorter. (See app. III for more information on case processing times regardless of outcome.)

When examining timeliness across the four phases that make up IDES, data show that average processing time regularly fell short of goals for three—MEB, Transition, and VA Benefits. For example, for cases that completed the MEB phase in fiscal year 2011, active duty and reserve component members’ cases took an average of 181 and 188 days respectively to be processed, compared to goals of 100 and 140 days. For the PEB phase, processing times increased over time, but were still within the established goal of 120 days. Along with increasing average processing times, the percentage of cases meeting goals for most phases has generally declined (see fig. 6). In particular, the MEB and Transition phases have lower percentages of cases meeting goals than the other phases in most years, especially for active duty cases.

As noted above, the MEB phase was a key contributor to increases in overall processing times between 2008 and 2011 for both active duty servicemembers and reservists for cases that have completed the IDES process regardless of outcome (table 1).

To obtain a better understanding of more recent timeliness trends within the MEB phase, GAO analyzed MEB timeliness of all cases—all fiscal years combined—that completed the MEB process by sorting them into two groups: (1) those that completed the entire IDES process, and (2) those that had not yet completed IDES but completed the MEB phase. As shown in figure 7, for the group that completed IDES, 30 percent of active duty servicemembers and 18 percent of reservists missed the goal by more than 90 days. For those still in IDES, representing more recent data, the picture is slightly better for active duty servicemembers with 37 percent of cases meeting the MEB goal and 25 percent missing the goal by more than 90 days. However, the percentage of reserve component members who missed the goal by more than 90 days increased from 18 to 28 percent.

For those servicemembers who were still enrolled in the MEB phase as of December 2011, the data show that 41 percent of active duty and 33 percent of reserve component servicemember cases had already missed the goal processing times (see fig. 8). Of these, 15 percent of active duty and 10 percent of reservist component servicemember cases missed the goal by more than 90 days.

Within the MEB phase, significant delays have occurred in completing medical examinations (medical exam stage) and delivering an MEB decision (the MEB stage). For cases completing the MEB phase in fiscal year 2011, 31 percent of active duty and 29 percent of reservist cases met the 45-day goal for the medical exam stage and 20 percent of active duty and 17 percent of reservist cases met the 35-day goal for the MEB stage. Officials at some sites we visited told us that MEB phase goals were difficult to meet and not realistic given current resources. For example:

Some military officials noted that they did not have sufficient numbers of doctors to write the narrative summaries of exam results needed to complete the MEB stage in a timely manner. One facility noted that while they have 7 doctors, they would need 11 additional doctors and 10 technician assistants to process cases through the initial medical exam and other additional disability specific examinations in a timely manner. Further, officials at another Army base we visited noted that there was a shortage of doctors and DOD board liaisons and that they had difficulty recruiting such staff due to the remote location of the base.

At all the facilities we visited, officials told us DOD board liaisons and VA case managers had large case loads. While DOD has established a goal of 1 board liaison for every 20 servicemembers, the ratios varied widely by military treatment facility with a range from 1:1 as the lowest to the highest of 1:75 according to recent data. Because of high case loads and a reported increase in the complexity of cases, staff at one facility reported a liaison to servicemember ratio of 1:80 and noted that liaisons must often prioritize cases to deal with the most pressing issues first. As a result, cases that might otherwise be quick to process take longer simply because they are waiting to be processed. Liaisons are often working overtime and weekends to keep up with cases.

Monthly data produced by DOD subsequent to the data we analyzed show significantly improved timeliness for the medical exam stage (66 percent of active duty cases met the goal in June 2012) and some improvement for the MEB stage (40 percent of active duty cases met the goal in the month of June 2012). However, it is too early to tell whether these improvements will continue going forward. (See app. III for DOD reported monthly data, October 2011 – June 2012.)

Since fiscal year 2008, the majority of cases have completed the PEB phase under the goal of 120 days, however, PEB timeliness has still worsened over time. In 2011, 78 percent of active duty and 62 percent of reservist cases that completed the entire IDES process met the PEB goal.

The average processing time was 93 days for active duty servicemembers and 116 for reservists (see table 2).

Despite meeting the overall PEB goal in fiscal year 2011, established goals were not met for any of the interim PEB stages, including the informal PEB and VA rating stages which are the two stages all servicemembers must complete. For all cases that completed the PEB phase in fiscal year 2011, only 38 percent of active duty and 38 percent of reservists’ cases received an informal PEB decision within the 15 days allotted. Further, only 32 percent of active duty and 27 percent of reservist cases received a preliminary VA rating within the 15-day goal. (see table 3).

Regarding delays with the VA rating, VA officials told us that staffing has been a challenge at their IDES rating sites and that this has slowed case processing. Monthly data produced by DOD subsequent to the data we analyzed show similar trends for the informal PEB and VA preliminary rating stages. As of June 2012 (most recent data available), active duty cases showed slight improvements in timeliness for the informal PEB stage (41 percent of cases meeting the goal and processing times averaging 24 days). The VA rating stage, on the other hand, showed slight declines in timeliness (31 percent of cases meeting the established goal and processing times of 35 days) relative to FY 2011 averages for active duty servicemembers. However, as noted before, it is too early to tell the extent to which such trends will continue. (See app. III for DOD reported monthly data, October – June 2012.)

Also during this phase, IDES planners allocated the majority of overall PEB processing time (75 out of the 120 days) for appeals—including a formal PEB hearing and a reconsideration of the VA ratings. According to officials, while the three appeal stages do not happen for every case, appeals can significantly increase processing times for any one case. However, only 20 percent of cases completed in fiscal year 2011 actually had any appeals; calling into question DOD and VA’s assumption on the prevalence and average effect of appeals, and potentially masking processing delays in other mandatory parts of the PEB phase.

The transition phase has consistently taken longer than its 45-day goal— almost twice as long on average. While processing times improved slightly for cases that completed this phase in fiscal year 2011 (from 79 days in fiscal year 2010 to 76 days in fiscal year 2011 for active duty cases), timeliness has remained consistently problematic since fiscal year 2008 (see table 4).

DOD lacks comprehensive data on how servicemembers spend their time in the transition phase, which includes many different activities related to separation from the military. These activities vary widely depending on the case. For example, during this phase servicemembers receive mandatory training such as job training through the Transition Assistance Program and may also receive counseling such as pre-discharge Vocational Rehabilitation and Employment counseling. In addition, servicemembers may be placed on temporary duty while house hunting, or to allow for a servicemember’s children to complete the school year before moving. Servicemembers may also take earned leave time—to which they are entitled—before separating from the service. For example, an Army official said that Army policy allows servicemembers to take up to 90 days of earned leave prior to separating, and that average leave time was about 80 days. Because many of these activities can occur simultaneously or in small intermittent segments of time, DOD officials said it is difficult to track which activities servicemembers participate in or determine how much time each activity takes. DOD is exploring options for better tracking how time is spent in this phase. Because a potentially substantial amount of the time in this phase may be for the personal benefit of servicemembers, DOD recently began reporting time in IDES with and without the transition phase included.

Processing time improved somewhat for the benefits phase (48 days in fiscal year 2010 to 38 days in fiscal year 2011), but continued to exceed the 30-day goal for active duty servicemembers (see table 5).

Several factors may contribute to delays in this final phase. VA officials told us that cases cannot be closed without the proper discharge forms and that sometimes they do not receive this information in a timely manner from the military services. Additionally, if data are missing from the IDES tracking system (e.g., the servicemember already separated, but this was not recorded in the database), processing time will continue to accrue for cases that remain open in the system. Officials could not provide data on the extent to which these factors had an impact on processing times for pending cases, but said that once errors are detected and addressed, reported processing times are also corrected.

Shortcomings in the Design and Administration of Servicemember Surveys Hamper their Usefulness

In addition to timeliness, DOD and VA evaluate IDES performance using the results of servicemember satisfaction surveys. In principle, all members have an opportunity to complete satisfaction surveys at the end of the MEB, PEB, and transition phases; however, under current survey procedures servicemembers become ineligible to complete a survey for either the PEB or transition phases if they did not complete a survey in an earlier phase. Additionally, servicemembers who start but do not complete a phase are not surveyed. As such, DOD may be missing opportunities to obtain input from servicemembers who did not complete a prior survey or exited IDES in the middle of a phase. Further, response rates may be affected because DOD does not survey servicemembers once they separate from the service and become veterans. While it is not necessary for DOD to survey all servicemembers at the end of every phase, the percentage and characteristics of servicemembers covered by the survey (i.e., who completed a phase and were ultimately interviewed) may be insufficient to establish that the survey results are representative of servicemember satisfaction, especially for later phases. (See table 6 for response and coverage rates.) DOD officials recently told us that they will consider alternative survey eligibility requirements, including working with the Office of Management and Budget for permission to interview veterans. (For additional information regarding the timing of the survey, see app. II).

In addition, alternate survey measures show lower satisfaction rates than those reported by DOD. Using DOD’s measure, we found an overall satisfaction rate of about 67 percent since the inception of IDES. DOD defines a servicemember as satisfied if the average of his or her responses across several surveys is above 3 on a 5-point scale, with 3 denoting neither satisfied nor dissatisfied. However, using our alternate measure that defines servicemembers as satisfied only when all of their responses are 4 or above, we calculated the satisfaction rate to be about 24 percent (see fig. 9).

Our calculation is a more conservative measure of satisfaction, because it rules out the possibility that a servicemember is deemed “satisfied” even when he or she is dissatisfied on one or more questions in the scale. While not incorrect, DOD’s scale can mask pockets of servicemember dissatisfaction. For example, an individual may indicate that he or she is very dissatisfied with one phase of the program, but satisfied with other phases, and the overall satisfaction score can be the same as one for a servicemember who is generally satisfied across all phases of the process. Measuring satisfaction, or even dissatisfaction, in different ways may provide a more complete picture of satisfaction and how it varies in different circumstances, and thus may reveal areas where DOD could focus on improving management and performance.

Finally, using either DOD’s or our calculated measure, we found that overall satisfaction did not vary much according to differences in the experiences of servicemembers. For example, our model estimated that satisfaction varied by no more than approximately five percentage points across branch, component, disenrollment outcome, sex, MEB exam provider, enlisted and officer personnel classes, and the number of claimed and referred conditions. While lack of variation could be a positive outcome signaling consistent treatment, it could equally mean that the survey does not measure opinions in enough detail to discriminate among servicemembers’ experiences. Either way, such results provide little insight into identifying areas for improvement or effective practices. Further, while we found some association between servicemembers satisfaction and the timeliness of their case processing, we also found many servicemembers were highly dissatisfied even when their cases were completed on time, and many were highly satisfied even when their cases were not. For example, 68 percent of those who said that PEB timeliness was “very poor” completed the phase on time, and 55 percent of those who said that MEB timeliness was “very good” did not complete on time. The lack of variation and/or correlation between satisfaction and experiences of servicemembers—coupled with low coverage rates—raise questions about the value of the survey results as a performance measure and program evaluation tool. (See app. II for more information on servicemember satisfaction results.)

DOD is reconsidering its options for measuring customer satisfaction, but has yet to select a particular approach. As noted above, possible changes might include widening the criteria for who is eligible for the survey, modifying survey questions, changing when and how the survey is delivered, and changing how satisfaction is calculated. Officials already concluded that the survey, in its current form, is not a useful management tool for determining what changes are needed in IDES and said that it is expensive to administer—costing approximately $4.3 million in total since the start of the IDES pilot. Navy officials told us they believed that the satisfaction surveys could be made more useful if they knew whether servicemember’s satisfaction was actually influenced by the servicemember’s desired or actual outcome of the IDES process. Further, Army officials already determined that the DOD survey is of limited value, and are proceeding with plans to field their own survey in the hopes of obtaining more detailed information at the facility level. Because of fiscal constraints, DOD suspended the survey in December 2011, but officials told us that they hope to resume collecting data in fiscal year 2013.

We identified two potential alternatives to assessing servicemember experiences.

Surveying a sample of servicemembers: While a census gives each servicemember a chance to describe his or her experiences with IDES, DOD could collect the same data at a lower cost by surveying a probability sample of servicemembers. If appropriately designed and executed, a sample would accurately represent all groups of servicemembers and produce the necessary data for important subgroups, such as facilities or branches. Since the cost of administering a survey is strongly related to the number of people surveyed, probability sampling could also allow DOD to assess servicemember experiences while substantially reducing data collection costs.

Exit interviews: In-depth interviews with servicemembers, completed at disenrollment from IDES, could also yield more detailed and actionable information about the program. Although the current survey includes open-ended questions, it is primarily designed to collect standardized, quantitative measures of satisfaction with broad aspects of IDES, such as fairness and the performance of DOD board liaisons and VA case managers. As a result, the survey provides a limited amount of detailed feedback on particular facilities, staff members, and stages of the process that managers might use to improve the servicemember experience, decrease processing times, or reduce cost. In contrast, semi-structured exit interviews would allow servicemembers to provide this type of qualitative, detailed feedback. Interviewing servicemembers at the end of the process would also allow servicemembers to assess their overall experiences with IDES rather than at an earlier stage, without having completed the entire process. Exit interviews could also reach servicemembers who exit IDES without completing the process such as those who are returned to duty. Exit interviews, however, have the potential to be labor intensive and expensive.

Recent Actions and Ongoing Initiatives May Improve IDES Performance, but It Is Too Early to Assess Their Overall Impact

DOD and VA Took Steps to Address Previously Identified IDES Challenges

DOD and VA have undertaken a number of actions to address IDES challenges—many of which we identified in our past work. Some actions—such as increased oversight and staffing—represent important steps in the right direction, but progress is uneven in some areas.

Increased monitoring and oversight: We identified the need for agency leadership to provide continuous oversight of IDES in 2008 and the need for system-wide monitoring mechanisms in 2010. Since then, agency leadership has established mechanisms to improve communication, monitoring, and accountability.

The secretaries of DOD and VA have met several times since February 2011 to discuss progress in improving IDES timeliness and have tasked their agencies to find ways to streamline the process so that the timeliness goals can be shortened. The secretaries also tasked their agencies to expand the use of expedited disability evaluations for severely combat-wounded servicemembers; and develop a system to electronically transfer case files between DOD and VA locations.

Senior Army and Navy officials regularly hold conferences to assess performance and address performance issues, including at specific facilities. With respect to the Army, meetings are led by the Army’s vice-chief of staff and VA’s chief of staff, and include reviews of performance where regional and local facility commanders provide feedback on best practices and challenges. For example, recent Army-VA conferences focused on delays in completion of preliminary ratings for Army PEBs by VA’s Seattle rating site, efforts by the Army to increase MEB staffing, development of Army-wide IDES standardization guidance, and Army-VA electronic records interchange. Periodic meetings are also held between senior Navy medical and VA officials to discuss performance issues at Navy military treatment facilities.

VA holds its own biweekly conferences with local staff responsible for VA’s portion of the process. These conferences are supplemented by a bi-weekly IDES “dashboard” that tracks performance data for portions of the IDES for which VA is responsible. According to VA officials, in addition to identifying best practices, these conferences focus on sites with performance problems and identify potential corrective actions. For example, officials said a recent conference addressed delays at Fort Benning, Georgia, and discussed how they could be reduced. VA officials noted that examiner staff were reassigned to this site and worked on weekends to address the problems at this site. In addition, senior VA health care officials hold periodic conferences with officials responsible for exams at IDES sites, to monitor performance.

Ensuring sufficient medical exam resources: In our December 2010 report, we noted that VA struggled to provide enough medical examiners (both VA employees and contractors) to meet demand and deliver exam summaries within its 45-day goal. For example, significant deficiencies in examiner staffing (particularly for mental health exams) at Fort Carson contributed to exams for active duty members taking an average of 140 days. To improve exam timeliness, VA hired more examiners and is devoting more resources at those sites where VA clinicians perform IDES exams. In addition, in July 2011, VA awarded a revised compensation and pension (including IDES) contract that provides more flexibility for VA to have contractors perform IDES exams at sites needing additional resources. As a result, VA can use contractors to conduct exams for regional offices beyond the 10 offices for which the contractor normally provides services. Also, VA contracted with 5 companies to provide short- term exam assistance at IDES sites needing it. Further, VA procedures allow reserve component servicemembers in remote locations to receive exams close to their homes. VA exam timeliness has improved and the agency met its 45-day goal for active component members in every month from August 2011 through June 2012. VA officials attributed improved exam timeliness, in part, to additional exam resources provided to IDES sites. (See app. III for additional information on fiscal year 2012 timeliness.)

Ensuring sufficient exam summaries: In our December 2010 report, we noted that some cases were delayed because VA medical exam summaries were not complete and clear enough for use in making rating and fitness decisions and needed to be sent back to examiners for additional work. VA officials told us that they have been reinforcing the importance of training and communication between rating staff and medical examiners as ways to improve exam summary sufficiency. For example, VA identified types of information which, if missing from an exam summary, would cause it to be insufficient, and has been training examiners to include such information. Additionally, VA noted that VTA now has the ability to track cases with insufficient exams by allowing staff to annotate information on exam summaries. However, staff are not required to provide this information and rules and procedures for its use have not been established.

Ensuring sufficient MEB staffing: In our December 2010 report, we noted that some sites had insufficient MEB physicians, leading to delays in completing the MEB phase. At that time, most of the 27 pilot sites were not meeting the 35-day goal, with average times for active component cases as high as 109 days. Meanwhile, DOD did not have sufficient board liaison staff to handle IDES caseloads. The Army is in the midst of a major hiring initiative intended to more than double staffing for its MEBs over its October 2011 level, which will include additional board liaison and MEB physician positions. The Army reported having 610 full-time equivalent MEB staff positions in October 2011, and planned to hire up to 1,410; this would include 172 MEB physician and 513 board liaison positions. The Army also planned to hire an average of one contact representative per board liaison; these staff members assist the board liaisons with clerical functions, freeing more of the liaisons’ time for counseling servicemembers. As of June 2012, the Army had filled 1,219 (86 percent) of the planned 1,410 positions.

Ensuring sufficient VA rating staff: In our December 2010 report, we noted that VA had insufficient staff at one of its rating sites to handle the demand for preliminary ratings, rating reconsiderations, and final VA benefit decisions. VA officials said that the agency has more than tripled the staffing at its IDES rating sites–from 78 to 262 positions. Further, VA has moved staff resources to IDES rating sites from other VA regional offices to provide short-term help in working down rating backlogs. Recent monthly data show an increase in the number of preliminary VA ratings completed, and a slight improvement in processing times. However, as noted before, it is too early to tell the extent to which such trends will continue. (See app. III for additional information on fiscal year 2012 timeliness.)

Improving completeness of reserve component members’ records: Service officials noted that incomplete medical records and administrative documentation, especially for reserve component members, often contribute to delays in the early IDES stages, including the VA exam stage. For example, a reserve unit may not have complete medical records for a member who received care from a private provider. When the servicemember enters the IDES, a board liaison is responsible for obtaining the private provider records before handing off the case to VA for exams. To address issues with reserve component servicemembers’ records, the Army established an interim office in Pinellas Park, Florida in January 2011. For reserve component servicemembers who may require IDES referral, this office is tasked with obtaining records from the member’s reserve unit; reviewing them to identify missing information; and, if necessary, requiring the reserve unit to obtain additional records to complete the case file. Staff at this office also determine whether the member needs IDES referral. Army officials indicated that this office is expected to help reduce the backlog of Army reserve component cases in the IDES. However, Army officials noted that they are providing training to reserve units to improve their ability to maintain complete records on their servicemembers and eventually, the Army may discontinue this office if no longer needed.

Improving MEB documentation and decisions: In response to delays in completing the MEB stage, the Navy and Army have initiatives underway to help ensure the timely completion of narrative summaries and MEB decisions. For example, the Navy piloted electronic narrative summary preparation at Naval Hospital Camp Lejeune, North Carolina. In May 2012, after determining that the piloted process led to improved MEB completion timeliness, the Navy deployed electronic narrative summary preparation Navy-wide. In March 2011, the Army also deployed an abbreviated MEB narrative summary format, intended to provide better information for MEB and PEB decision making while helping reduce delays in the completion of these summaries by MEB clinicians. Incorporating feedback from its MEBs and PEBs, the Army expects the revised IDES template to reduce redundant information, make summaries simpler and easier to use, and standardize summary preparation across their sites.

Resolving diagnostic differences: In our December 2010 report, we identified differences in diagnoses between DOD physicians and VA examiners, especially regarding mental health conditions, as a potential source of delay in IDES. We also noted inconsistencies among services in providing guidance and a lack of a tracking mechanism for determining the extent of diagnostic differences. In response to our recommendation, DOD commissioned a study on the subject. The resulting report confirmed the lack of data on the extent and nature of such differences, and noted that the Army has established guidance more comprehensive than the guidance DOD was developing. It also recommended that DOD or the other services develop similar guidance. A DOD official told us that consistent guidance across the services, similar to the Army’s, was included in DOD’s December 2011 IDES manual. Also, in response to our recommendation, VA took steps to modify the VTA database used to track IDES to collect information on diagnostic differences. The VTA upgrade was completed in June 2012 after several delays. The report also recommended that DOD and VA establish a committee to improve the accuracy of posttraumatic stress disorder ratings. DOD noted that training on diagnostic differences has been incorporated into its continuing medical education curriculum for military clinicians, but DOD considers the issue of posttraumatic stress disorder ratings largely resolved. Meanwhile, the Army’s new IDES narrative summary template includes a section where the MEB clinician identifies any inconsistencies in the case record, including any diagnostic differences with VA examiners.

DOD and VA Are Addressing Shortcomings in Information Systems, but Efforts to Date Are Limited

DOD and VA are working to remedy shortcomings in information systems that support the IDES process. These shortcomings include VTA’s lack of capability for local sites to track cases, and the potential for erroneous and missing data in VTA, affecting timeliness measurement. However, some efforts related to information systems are causing work inefficiencies, are still in progress, or otherwise are limited.

Improving local IDES reporting capability: DOD and VA are implementing solutions to improve the ability of local military treatment facilities to track their IDES cases, but multiple initiatives may result in redundant work efforts. Officials told us that the VTA—which is the primary means of tracking the completion of IDES cases—has limited reporting capabilities and staff at local facilities are unable to use it for monitoring the cases for which they are responsible. DOD and VA developed VTA improvements that will allow DOD board liaisons and VA case managers— and their supervisors—to track the status of their cases. VA included these operational reporting improvements in its June 2012 VTA upgrade. In the meantime, staff at many IDES sites have been using their own local systems to track cases and alleviate limitations in VTA. Further, the military services have been moving ahead with their own solutions. For instance, the Army has deployed its own information system for MEBs and PEBs Army-wide. In addition, DOD has also been piloting its own tracking system at 9 IDES sites. As a result, staff at IDES sites we visited reported having to enter the same data into multiple systems. For example, board liaisons at Army MEBs Fort Meade and Joint Base Lewis- McChord reported entering data into VTA and the Army’s new system, while board liaisons at Andrews Air Force Base reported entering data into VTA and DOD’s pilot data system.

Improving IDES data quality: DOD is taking steps to improve the quality of data in VTA. Our analysis of VTA data identified erroneous or missing dates in at least 4 percent of the cases reviewed. Officials told us that VTA lacks adequate controls to prevent erroneous data entry, and that incorrect dates may be entered, or dates may not be entered at all, which can result in inaccurate timeliness data. For example, Army officials noted that some cases shown in VTA as very old were actually closed, but were missing key dates. In September 2011, DOD began a focused effort with the services to correct erroneous and missing case data in VTA. Officials noted that the Air Force and Navy completed substantial efforts to correct the issues identified at that time, but Army efforts continue. DOD and Army officials noted that additional staff resources are being devoted to cleaning up Army VTA data. While improved local tracking and reporting capabilities will help facilities identify and correct erroneous data, keeping VTA data accurate will be an ongoing challenge due to a lack of data entry controls. While DOD is currently assisting the services, DOD officials said they expect that eventually the services will be responsible for identifying and fixing data errors.

DOD and VA are also pursuing options to allow them to save time by replacing the shipping of paper case files among facilities with electronic file transfers. Requirements for an electronic case file transfer solution have been completed and DOD and VA officials expect to begin piloting it in August 2012. As a short-term solution, the Army and VA began using an Army file transfer Web site to move IDES records between the Army’s PEBs and the Seattle VA rating site in March 2012. According to VA officials, this could save several days currently spent shipping paper files between these offices. VA officials noted that the same Web site is being used for transfers between the Navy PEB and Providence rating site. Meanwhile, the secretaries of Defense and Veterans Affairs tasked their staffs to develop standards for electronic IDES case files by July 2012.

DOD and VA are Pursuing Broader Solutions to Improving IDES Performance

Based on concerns of the Secretaries of DOD and VA about IDES delays, the departments have undertaken additional initiatives to achieve time savings for servicemembers. For example, in response to the secretaries’ February 2011 directive to streamline the process, DOD and VA officials proposed a remodeled IDES process. In December 2011, senior agency leadership decided to postpone the pilot of a remodeled IDES process, and instead tasked the agencies to explore other ways to streamline the process. As a result, DOD, with VA’s assistance, began a business process review to better understand how IDES is operating and identify best practices for possible implementation. This review incorporates several efforts, including visits to 8 IDES sites to examine how the process was operating and identify best practices.includes the following:

Process simulation model: Using data from site visits and VTA, DOD is developing a simulation model of the IDES process. According to a DOD official, this process model will allow the agencies to assess the impact of potential situations or changes on IDES processing times, such as surges in workloads or changes in staffing.

Fusion diagram: DOD is developing this diagram to identify the various sources of IDES data—including VA claim forms and narrative summaries—and different information technology systems that play a role in supporting the IDES process. Officials said this diagram would allow them to better understand and identify overlaps and gaps in data systems.

Ultimately, according to DOD officials, this business process review could lead to short- and long-term recommendations to improve IDES performance, potentially including changes to the different steps in the IDES process, performance goals, and staffing levels; and possibly the procurement of a new information system to support process improvements. However, a DOD official noted that these efforts are in their early stages, and thus there is no timetable yet for completing the review or providing recommendations to senior DOD and VA leadership. DOD officials indicated that they expect this to be a continuous IDES improvement process, including further site visits.

Finally, DOD is also developing guidance to expand implementation of an expedited disability evaluation process for servicemembers with catastrophic, combat-related conditions by allowing it to be operated at more military treatment facilities. DOD created this expedited process in January 2009 for servicemembers who suffer catastrophic, combat- related disabilities. Under an agreement with VA, the services can rate such members as 100 percent disabled without the need to use VA’s rating schedule. However, according to DOD officials, the services report that no eligible servicemembers are using this process. Instead, servicemembers are having their cases expedited through the IDES informally. The revisions to DOD’s policy would allow the expedited process to be used at additional military treatment facilities beyond the original 4 facilities.of a rewrite of DOD’s key guidance documents, and was undergoing review at the time of our review.

Conclusions

By merging two duplicative disability evaluation systems, IDES shows promise for expediting the delivery of DOD and VA benefits to injured servicemembers and is considered by many to be an improvement over the legacy process it replaced. However, nearly 5 years after its inception as a pilot, delays continue to affect the system and the contribution of various, complex factors to timeliness is not fully understood. Recent efforts by DOD and VA to better understand how different IDES processes contribute to timeliness are promising and may provide the departments with an opportunity to reassess resource levels and timeframes, and to make adjustments if needed. This information will also help to ensure that DOD and VA are making the best use of limited resources to improve IDES performance. However, it is not clear when these efforts will be complete or if any recommended actions will be implemented. DOD has also begun rethinking its approach to determining servicemember satisfaction with IDES. Our analysis of customer satisfaction data suggests that there are opportunities for improving the representativeness of the survey information collected and reconsidering the cost-effectiveness of the current lengthy surveys. Finally, providing local facilities the capability to track and generate reports on the status of their cases is long overdue and may empower local staff to better address challenges. However, tracking reports are only as good as the data that are entered into VTA, and DOD and VA can ensure the quality of these data through continuous monitoring. Meanwhile, the DOD-led business process review should identify and ultimately eliminate any redundant or inefficient information systems for tracking cases as well as for other IDES purposes.

Recommendations for Executive Action

1) To ensure that servicemember cases are processed and are awarded benefits in a timely manner, we recommend that the Secretaries of Defense and Veterans Affairs work together to develop timeframes for completing the IDES business process review and implementing any resulting recommendations. 2) To improve DOD’s ability to measure servicemembers’ satisfaction with the IDES process, we recommend that the Secretary of Defense develop alternative approaches for collecting more meaningful and representative information in a cost effective manner. 3) To ensure that IDES management decisions continue to be based upon reliable and accurate data, we recommend that the Secretaries of Defense and Veterans Affairs work together to develop a strategy to continuously monitor and remedy issues with VTA timeliness information. This could include issuing guidance to facilities or developing best practices on preventing and correcting data entry errors; and developing reporting capabilities in VTA to alert facilities to potential issues with their data.

Agency Comments and Our Evaluation

We provided a draft of this report to DOD and VA for review and comment. In their written comments, which are reproduced in appendixes IV and V, DOD and VA both concurred with our recommendations. VA also provided technical comments, which we incorporated as appropriate.

While concurring with our recommendations, DOD also commented that our discussion of IDES surveys contained inaccuracies, but did not specify the inaccurate information in our draft report. In a subsequent communication, DOD officials noted that our draft inaccurately described DOD’s decision to not survey veterans. We corrected this information accordingly. Further, while VA concurred with our recommendation that it work with DOD to develop timeframes for completing the IDES business process review and implementing any resulting recommendations, VA stated that DOD is leading the business process review, and therefore should develop the timeframes for completing the review. We have revised this report to clarify that DOD is leading the business process review, but we did not alter the recommendation because we believe that it is important for VA to work closely with DOD, including in developing review timeframes.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees, the Secretary of Defense, the Secretary of Veterans Affairs, and other interested parties. The report is also available at no charge on the GAO Web site at www.gao.gov.

If you or your staff members have any questions about this report, please contact me at (202) 512-7215 or at bertonid@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Staff members who made key contributions to this report are listed in appendix VI.

Appendix I: Objectives, Scope, and Methodology

In conducting our review of the Integrated Disability Evaluation System (IDES), our objectives were to examine (1) the extent to which the Departments of Defense (DOD) and Veterans Affairs (VA) are meeting IDES performance goals, and (2) steps DOD and VA are taking to improve IDES performance. We conducted this performance audit from May 2011 to August 2012, in accordance with generally accepted government auditing standards. These standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Review of IDES Timeliness Data

To determine the extent to which IDES is meeting established timeliness goals, we analyzed data collected through VA’s Veterans Tracking Application (VTA) database. While VA manages VTA, both DOD and VA staff enter data into VTA, and the evaluation of IDES data is primarily conducted by staff at DOD’s Office of Warrior Care Policy (WCP). WCP provided us with a dataset that was current as of January 1, 2012 and contained data spanning back to the inception of IDES in late 2007. This data export included data on a total of 39,260 cases. Of these cases, 34,185 were active duty servicemembers and 5,068 were Reserve/Guard servicemembers. This VTA data set contained demographic data for each individual IDES case as well as a record of dates for when servicemembers reached various milestones in IDES. Overall and interim IDES timeliness calculations are based on computing the number of days elapsed between appropriate milestone dates. For example, overall timeliness for servicemembers that receive benefits is calculated as the number of days between the individual being referred into the IDES and the date on which his or her VA benefits letter is issued. We met with staff at WCP to ensure we used appropriate variables when calculating timeliness. We also met with officials at VA to discuss the calculations used to determine the timeliness of cases.

We took a number of steps to assess the reliability of VTA data and ultimately found the data to be sufficiently reliable for the purposes of our audit. Past GAO worknumber of steps to follow up on past assessments of VTA. relied on VTA data, and therefore we took a

We interviewed DOD and VA and determined that internal controls on VTA data had not changed substantially since our past review.

We conducted electronic testing of the VTA data and generally found low rates of missing data or erroneous dates pertinent to our analysis—approximately 4 percent of cases. For IDES cases in which we found missing dates or dates out of sequence, we excluded those cases from all of our analyses. While there were some instances in which the erroneous dates may be justified, we excluded the entire case from our analysis if any such dates appeared at any point in the VTA database. Such data included cases in which (1) there was no MEB referral date signifying the start of IDES process, and (2) the ending date preceded the beginning date of the IDES phase (resulting in timeliness calculations appearing as a negative amount of time).

We also conducted a limited trace-to-file process to determine whether date fields in VTA were an accurate reflection of the information in IDES case files. Specifically, we compared VTA dates in 15 IDES cases completed in fiscal year 2011 against the dates in the corresponding paper files. In comparing dates, we allowed for a discrepancy of 5 days in dates to allow for the possibility that dates may have been entered into the database after an event took place. Ninety-three percent of the dates we traced back to the original file documents were found to be accurate, that is falling within our 5 day allowance.

Analysis Conducted Using VTA Data

For the cases meeting our criteria for reliability, we analyzed timeliness data for those cases that had completed the entire IDES process or had completed each of the four IDES phases. We specifically: Identified the total number of cases enrolled each fiscal year from FY 2008 through 2011, by active as well as National Guard and reserve servicemembers, and by military branch of service.

Identified the number of cases that completed the entire IDES process for each fiscal year from fiscal year 2008 through fiscal year 2011. We analyzed completed cases in two different ways: (1) those who completed the process and received VA benefits and (2) those who completed the IDES with any outcome (such as permanent retirement, Temporary Disability Retirement List, return to duty, etc.). In order to be able to make comparisons across cases with different outcomes for a given point in time, we defined fiscal year by using the VTA variable “final disposition date”. We did this because most completed cases—regardless of outcome—have a final disposition date in VTA. In contrast to our approach, VA use the “VA benefit date” variable to determine fiscal year of completion for cases resulting in benefits. As such, their number of cases and timeliness calculations by fiscal year differed from ours, although overall trends are similar.

Identified the number of cases that completed each phase of IDES and the interim stages within each phase, again by fiscal year (fiscal years 2008 through 2011).

Computed timeliness statistics for the completion of the IDES process, phases, and stages against the performance goals set by DOD and VA, such as average days and percent meeting goals.

Computed number and percent of cases where a servicemember appealed a decision made during the IDES process, by fiscal year.

For the purposes of this report, GAO opted to not include reserve component time spent in the VA benefit phase in our calculations for overall time because the 30 days allotted for this phase is not included in the 305-day overall goal for the reserve component.

GAO also performed analyses similar to those above, except that we grouped cases according to the year in which they were enrolled in IDES. (See app. II for more detail on these analyses.) Additionally, we analyzed timeliness for cases that had not yet completed the MEB stage as of the date we received the VTA data.

Review of IDES Customer Satisfaction Data

To determine the extent to which IDES is meeting its customer satisfaction goals, we analyzed data collected from IDES customer satisfaction surveys conducted at the end of three phases: MEB, PEB and Transition. These surveys are administered by telephone by contractors hired by DOD. The dataset we received contained survey responses for individual servicemembers from the beginning of the IDES pilot to December 2011, at which time administration of the survey was suspended. Additionally, we matched individual survey responses with information from VTA to gain additional understanding into how customer satisfaction varied according to different factors such as timeliness and case outcome. We matched survey and VTA data using the unique case identifier attached to each IDES case, maintaining the anonymity of the servicemembers. See appendix II for the results of additional analyses we conducted using survey data and survey data matched with VTA data.

In the course of our review we concluded that the survey data were sufficiently reliable for our purposes. We interviewed relevant officials at DOD and their contractors about eligibility requirements and the administration of the surveys. Further, we met with DOD and their contractors on multiple occasions to discuss the calculations used to determine response rates for the survey and servicemembers’ level of satisfaction. See appendix II for more details on GAO’s review of response rates.

Identifying Challenges and Actions Taken to Improve Performance

To identify challenges in implementing IDES as well as steps taken to improve performance, we visited six military treatment facilities. During the site visits, we interviewed officials involved in implementing IDES from both DOD and VA, including military facility commanders and administrators, DOD board liaisons, military physicians involved in MEB determinations, DOD legal staff, VA case workers, VA or contract examiners, and administrators at VA medical clinics and regional offices. Additionally, we interviewed servicemembers who were currently enrolled in the IDES process. We selected the six facilities to obtain perspectives from sites in different military services, geographic areas, and their ability to meet timeliness goals for different phases of the process (see table 7). In addition, we visited the Air Force’s Formal Physical Evaluation Board at Randolph Air Force Base, Texas. During this visit we observed a hearing and met with board members to obtain a better understanding of the process.

Appendix II: Additional Timeliness and Satisfaction Analyses

This appendix provides additional information on the timeliness of the IDES process and servicemember satisfaction with it. First, we use timeliness data to examine whether changes over time in processing times and the percentage of cases meeting timeliness goals look any different when cases are grouped according to the fiscal year in which the cases were first enrolled rather than the fiscal year in which the cases were completed. Second, we use survey data to examine different measurements of servicemember satisfaction with IDES, how satisfaction varied according to various servicemember characteristics, response and coverage rates for the servicemembers surveyed, and how the survey respondents differed from nonrespondents.

With respect to timeliness, we find generally similar trends for cases grouped by fiscal year of enrollment versus fiscal year of completion, with some key differences. Organizing cases by completion date results in shorter average processing times in 2008, since only those cases that are processed quickly could be completed in the first year of IDES. As such, organizing cases by enrollment date provides a better estimate of the processing times for the early IDES cases. However, this approach results in shorter processing times in 2011, the most recent full year of the program, since only cases that finish quickly can be analyzed.

With respect to satisfaction, we find that the particular index used to summarize servicemembers’ responses can affect the proportion reported as being “satisfied” or “dissatisfied” with IDES overall. DOD’s index suggests that 67 percent of servicemembers have been satisfied since the IDES program began, but a reasonable alternative measure we developed suggests that only 24 percent of servicemembers have been satisfied. Using this measure, satisfaction varies only slightly across many important groups of servicemembers, such as by disenrollment outcome, suggesting that available program data cannot precisely explain satisfaction outcomes. Also, servicemembers surveyed may not represent the servicemembers who completed the different phases of IDES well enough to generalize to them, given the low response rates to the MEB survey and fact that being selected for latter (PEB and Transition) surveys were conditional on completing the MEB survey.

Overall IDES Timeliness: Processing Times and Percent of Cases Meeting Timeliness Goals

Average IDES processing times for completed cases resulting in benefits generally worsened since 2008, especially for active duty cases, regardless of whether cases are grouped by the fiscal year in which they were completed (fig. 10) or by the fiscal year in which they were enrolled (fig. 11). The notable exception is when fiscal year 2011 is the year of enrollment. However, caution must be used when examining cases enrolled in 2011 because over 15,600 service members of the 18,651 (or at least 84 percent) who entered IDES in fiscal year 2011 did not have an outcome in 2011 and were enrolled in IDES as of January 1, 2012, potentially changing the distribution of processing times as they proceed through IDES.

We also examine average IDES processing times according to year of completion (see fig. 12) and year of enrollment for cases (see fig. 13) for all completed cases regardless of outcome. As with cases that resulted in benefits, for cases resulting in any outcome we find that average processing times increased since 2008—again with the exception of fiscal year 2011 for reasons discussed earlier—although average processing times are somewhat shorter than when only servicemembers receiving benefits are included (fig. 11).

Figures 14 and 15 show that regardless of whether cases are organized by year of completion or enrollment, the percent of completed cases resulting in benefits that were not timely increased between fiscal year 2008 and 2010 for both active duty servicemembers and members in the Reserves or National Guard. As with the average processing times, caution must be used when examining cases enrolled in fiscal year 2011 (fig. 15), since only those cases that are processed quickly are observed in the last year. Similarly, caution also must be used when examining cases in 2008 (fig. 14), since the only cases that are included in the first year are those that completed IDES quickly.

Figures 16 and 17 show how average processing times for each of the four phases of IDES have changed over the four fiscal years when cases are grouped by the fiscal year in which they completed a given phase and when cases are grouped by the fiscal year in which they were enrolled or started a given phase.according to the fiscal year in which the different phases were completed, processing times increased for all phases except the Transition phase. Figure 17 shows a roughly similar pattern of increases in processing times in all but the Transition phase, though processing times in 2011 are skewed for the reason mentioned above. Figures 18 and 19 show how the percentages meeting the timeliness goals for each of the four phases of IDES have changed over the four fiscal years when cases are grouped by the fiscal year in which they completed a given phase and when cases Figure 16 shows that when cases are grouped are grouped by the fiscal year in which they were enrolled or started a given phase. Figure 18 shows that the percent of cases meeting timeliness goals decreased over the four years for the MEB and PEB phases, although a high percent of cases met PEB goals. However, the Transition and Benefits phases fluctuated up and down and both were favorable across some years. Figure 19 also shows decreases in percentages of cases meeting timeliness goals at the MEB and PEB phases when cases are grouped by fiscal year of starting a given phase. The fluctuations in the timeliness of the Transition and Benefits phases were more prevalent when cases were grouped in this manner.

Low response and coverage rates for servicemember satisfaction surveys administered after each phase of IDES raise concerns about how well the satisfaction survey results represented the larger population of servicemembers who completed one or more phases.

DOD surveys servicemembers after they complete the MEB, PEB, and Transition phases of IDES. The department attempts to survey all servicemembers who complete each phase, but only if they completed the prior surveys. For example, the MEB survey must be completed before a servicemember is eligible to complete the PEB survey.

Using the data available to us, and as table 8 below shows, we found that 9,604 of the 25,212 servicemembers who completed the MEB phase were surveyed, for a 38 percent response and coverage rate.18,296 servicemembers who completed the PEB phase, only 8,968 of them completed the prior MEB survey and were eligible for the PEB survey and of these only 4,795 were surveyed. Using DOD’s eligibility criteria, the response rate for the PEB survey was roughly 54 percent (4,795 of 8,968). However, the coverage rate for all servicemembers who completed the PEB phase (regardless of whether they completed the prior survey) was only 26 percent (4,795 of 18,296). Similarly, the response rate for the Transition survey was 72 percent while the coverage rate was only 23 percent (See table 8).

As table 9 below shows, there were some sizable differences between respondents and nonrespondents, especially for the PEB and Transition surveys. For example, respondents to the transition survey spent more time than nonrespondents in the Transition phase, were less likely to be separated with benefits, and were more likely to be placed on the Permanent Disability Retired List. These differences, combined with low response and coverage rates, raise the possibility of biased responses.

The particular measure used to assess servicemember satisfaction can affect the proportion reported as “satisfied” with the IDES program. Depending on the measure used, satisfaction is about 2.8 times lower than what DOD has reported, and many servicemembers classified as “satisfied” express moderate dissatisfaction with some aspects of the process.

DOD has reported average servicemember satisfaction with IDES overall and with three phases of the process, i.e., MEB, PEB, and Transition phases. In so doing, DOD has developed indices of satisfaction on several broad dimensions, such as satisfaction with the overall experience and fairness, which combine responses to selected survey questions. Although the number of questions used in each index vary depending on the number of phases completed, each index classifies servicemembers as “satisfied” or “dissatisfied” using the average of their responses across all questions in the index. Each question’s scale ranges from 1 to 5, with 1 denoting “very dissatisfied” (or a similar negative response), 5 denoting “very satisfied,” and 3 denoting “neither satisfied nor dissatisfied.” DOD reports that a servicemember is “satisfied” if his or her average response across all items in the scale exceeds 3. Table 10 summarizes the responses to each question that DOD uses in its overall satisfaction index at each phase (as of August 2011).

DOD’s indices are one reasonable method of summarizing servicemember opinions. In quarterly performance reports, DOD notes that it has used factor analysis, a form of latent variable statistical models to assess the reliability of its scales. While we did not review DOD’s models, we independently found that DOD’s overall index of satisfaction with IDES was highly reliable. (Specifically, using Cronbach’s alpha, the index was highly correlated with a single latent dimension at α = 0.92.) This supports DOD’s choice to measure the single concept of “satisfaction” by averaging the ordinal servicemember responses.

Nevertheless, the average survey response can obscure variation in the responses that make up the index. For example, suppose that a servicemember said she was “very satisfied” (response of 5) on two of the four questions in the index, “dissatisfied” on one (response of 2) and “very dissatisfied” on the last one (response of 1). With an average response over 3, the DOD measure would classify her as “satisfied,” despite the fact that she was “somewhat dissatisfied” or “very dissatisfied” with two of the four aspects of IDES that DOD considers important. The grouping rule considers this servicemember equally happy with IDES as someone who says they are “satisfied” with all four aspects in the index.

To assess the extent to which DOD’s index might mask dissatisfaction, we calculated the proportion of questions in the scale on which servicemembers whom DOD classified as “satisfied” gave neutral or negative responses (1, 2, or 3). We found that half of these servicemembers gave neutral or negative answers to at least 25 percent of the items in the index, and a quarter gave such answers to at least 41 percent of the items. For these servicemembers, the DOD index may suggest more satisfaction than the underlying survey questions would support.

We further assessed the sensitivity of DOD’s index by comparing it against a different (i.e., GAO’s) measure of satisfaction: whether a servicemember is “somewhat” or “very satisfied” (or gives a similarly positive response) on all items in DOD’s scale of overall IDES satisfaction. Our measure is more conservative than DOD’s, because ours only includes positive responses and uses a broader cutpoint (two response categories) to distinguish between “satisfied” and “not satisfied” servicemembers. (In contrast, DOD calculates average satisfaction on an ordinal scale of 1 to 5, and then uses a cutpoint at 3.) Our measure is not inherently more valid, however, and has its own weaknesses. In particular, we classify a servicemember as “not satisfied” if she gives a neutral or negative response to just one of the four items in DOD’s scale.

When we analyzed overall satisfaction using both measures, we found that overall, servicemembers are 2.8 times less satisfied on our measure than on DOD’s (i.e., 23.8 versus 67 percent). Further, only about 20 to 30 percent of servicemembers are “satisfied” with each aspect of the IDES process that DOD considers important across most of the subgroups we analyzed, while DOD classifies about 60 to 70 percent of such servicemembers as “satisfied” on average. In the next section, we present further information on variation in satisfaction across servicemember groups.

Explaining Servicemember Satisfaction Outcomes

Although the servicemember survey provides numerous measures of satisfaction, it is also important to explain variation in satisfaction outcomes—i.e. why some servicemembers are more satisfied than others. Explaining variation can connect dissatisfaction with poor program performance and help identify specific reforms to improve the experiences of servicemembers who typically have been less satisfied. However, the available program data cannot precisely explain outcomes when used in this type of explanatory analysis. Using the available data, we could predict satisfaction only 1.9 percentage points better after controlling for multiple factors than what we would have achieved by chance (65.5 percent vs. 63.7 percent of satisfied responses predicted correctly).

In order to further explain variation in satisfaction, we matched the survey responses to the data that DOD and VA maintain on the processing of each servicemember’s case, known as the VTA data. This database primarily measures the time it took servicemembers to complete each phase of the IDES process. A small number of other program and demographic variables are also available, such as service branch, component, and the number of conditions claimed and referred. Using the matched survey and VTA data, we estimated the association between satisfaction and observable factors that could potentially explain variation in servicemembers’ experiences.

Table 11 below (columns 2-4) presents these associations for both DOD’s and GAO’s overall measures of satisfaction. The “raw data” estimates are simply the proportion of servicemembers in a particular group who were satisfied according to either measure. In the fourth column (“model estimates”), we estimate this proportion holding constant all of the other factors listed, using a statistical model. Specifically, the estimates are in- sample mean predicted probabilities of giving a satisfied response on the GAO satisfaction index from a logistic model of satisfaction. The covariates are given by indicators of whether the servicemember belonged to each group in column 4. The maximum likelihood estimators allowed the probability of satisfaction, given the covariates, to be dependent across observations within the 26 cross-classified groups of PEB location and MEB medical treatment facility. This adjusted for the possibility that servicemembers were similarly satisfied if they were processed in the same locations, given similar values on the observed covariates.

Regardless of which measure is used (DOD’s or GAO’s), satisfaction varied only modestly across many important groups of servicemembers. Our model estimates that the GAO measure of satisfaction varied by no more than approximately five percentage points across branch, component, disenrollment outcome, sex, MEB exam provider, enlisted and officer personnel classes, and the number of claimed and referred conditions, although differences across MEB treatment facilities and PEB locations were larger. This can be seen as a positive outcome, if this correlation implies that DOD and VA administer the program consistently across servicemembers and locations. However, the lack of variation also could suggest that the survey items do not measure opinions in enough detail to discriminate among servicemembers’ experiences.

Also shown in table 11, satisfaction had a stronger association with case processing time (time spent in IDES) than some of the other factors we examined. Servicemembers whose case processing times were among the quickest 25 percent were about 2.3 times as likely to be satisfied (on the GAO scale) than those whose times were among the 25 percent of cases with the longest overall timeframes (i.e., 41 versus 18 percent). Nevertheless, only 41 percent of those servicemembers whose cases were processed most quickly were satisfied (holding constant the other factors). This suggests that servicemembers’ opinions about IDES may be only loosely related to the amount of time they spent in IDES, as discussed in the next section below.

Although the average case processing time has generally increased since 2008, when we look at satisfaction by fiscal year, servicemember satisfaction shows evidence of improvement since fiscal year 2008. Specifically, our measure of satisfaction from the model increased by 15 percentage points since 2008, roughly doubling from 13 to 28. Because the model estimates control for various other factors, these results suggest that servicemember views of the IDES process have improved over time, rather than the possibility that IDES has simply processed different types of cases.

Satisfaction does not vary by a large amount across many MEB treatment facilities, but there are exceptions. Our model estimates that about 18 to 26 percent of servicemembers were satisfied at most facilities. However, there were pockets of greater satisfaction. Specifically, servicemembers had more positive experiences at Forts Belvoir, Bragg, Campbell, Drum, Hood, and Polk, with satisfaction estimated to have ranged from 28 to 45 percentage points. Fort Meade had the lowest satisfaction at 15 percent. These estimates hold constant time spent in IDES and other factors in column 4 and, thus, partially account for the types of cases each facility processes.

DOD and VA measure IDES timeliness directly in VTA and as part of the overall servicemember satisfaction scale. These overlapping measures let us compare servicemembers’ opinions to their actual experiences in the program. To do this, we calculated processing times at each phase of IDES for servicemembers who expressed varying degrees of satisfaction with the timeliness of their case processing at that phase. In addition, we analyzed whether servicemembers who were satisfied with the overall IDES process were more or less likely to meet timeliness goals. Table 12 provides these statistics.

As shown in table 12, satisfaction generally stayed the same or decreased as processing times increased. The median days spent in the MEB and PEB phases were 35 and 38 percent lower, respectively, among those servicemembers who said that MEB and PEB timeliness was “very good” as compared to those who said it was “very poor.” The former group was 170 percent more likely to have met the MEB timeliness goal and 30 percent more likely to have met the PEB timeliness goal. Similarly, the case for a median servicemember—whom we classified as “satisfied” with the overall IDES process—was completed 15 percent more quickly and was 49 percent more likely to have met the timeliness goal than the median servicemember who was “dissatisfied.” The model estimates in table 11 confirm that the GAO measure of satisfaction and timeliness (time spent in IDES) are negatively related even when holding constant several other variables.

Perceived and actual timeliness had little association at the Transition phase. Across all levels of satisfaction with timeliness, the median processing time varied by no more than 4 days, and the proportion meeting the timeliness goal varied by no more than 4 percentage points. The use of personal leave is one plausible explanation for the unresponsiveness of servicemember satisfaction to actual processing times in the Transition phase. A servicemember might not have been dissatisfied with delays if taking leave was the reason, rather than the IDES process itself.

Despite the associations between actual and perceived timeliness at the MEB and PEB phases, there were many servicemembers who were satisfied or dissatisfied with timeliness that spent similar amounts of time in the program. For example, 68 percent of those who said that PEB timeliness was “very poor” completed the phase on time, and 55 percent of those who said that MEB timeliness was “very good” did not complete on time. Among servicemembers who said that MEB timeliness was “very good,” the middle 80 percent of processing times ranged from 62 days to 223 days. The same range for servicemembers who said MEB timeliness was “very poor” was 88 to 323 days. As table 12 shows, a similar pattern holds for the PEB phase. Although servicemembers tend to be more satisfied in MEB and PEB when their cases take less time, many of them are highly dissatisfied even when their cases take an unusually short amount of time (and vice versa). In the Transition phase, however, 40 percent of servicemembers who said that timeliness was “very good” were processed in 91 to 657 days—a more lengthy range than at the other phases. The large range and relationship with satisfaction may reflect the use of servicemember leave.

The fact that many servicemembers are similarly satisfied with timeliness, even though they can have widely different processing times, has broader implications for measuring the performance of IDES. DOD’s timeliness goals may not be meaningful to servicemembers or necessarily reflect high-quality service. Alternatively, servicemembers may not use reasonable standards to assess the time required to process their cases, or they may not accurately perceive the time they have spent in the program. In these scenarios, the value of servicemember satisfaction as a performance measure becomes less certain.

The relationship between perceived and actual timeliness may simply reflect a large amount of unobserved heterogeneity across servicemembers. For example, a servicemember whose case has been in IDES for an extremely long time might still be highly satisfied with timeliness if the case was complex or personal leave was taken during the process. Neither the survey nor the VTA data measure these or other such characteristics that might affect the program’s key performance measures.

Factors Affecting Survey Results and Implications for Program Evaluation

The lack of variation in satisfaction across servicemember groups and according case timeliness might be seen as a positive outcome, and may suggest that DOD and VA administer the program consistently across servicemembers and locations. However, the lack of variation also could suggest shortcomings in the design and administration of the survey, or in data limitations that, alone or together, may reduce the usefulness of survey data for program evaluation. For example:

Survey questions: The survey questions may not be sufficiently detailed to measure important differences among servicemembers’ experiences. For example, the survey includes 12 questions (4 per survey) that measure broad opinions about IDES, and DOD subsequently averages these responses together. This approach may limit the survey’s capacity to describe IDES experiences in sufficient detail.

Precision of DOD indices: DOD reports measures of overall satisfaction with IDES for each phase, using the questions in table 9. However, these measures include one question that asks respondents to “evaluate their overall experience since entering the IDES process,” which could be influenced by experiences in prior phases. Consequently, the satisfaction measures reported for each phase could represent a combination of servicemembers’ experiences in that phase and prior phases.

Completing two surveys at once: DOD officials told us that a servicemember may be surveyed for the PEB and Transition phases in one session. In these instances a large amount of time may have passed since the servicemember completed the PEB phase and it may be more difficult for the servicemember to isolate his or her satisfaction with a particular phase.

Survey design: The satisfaction survey is primarily designed to measure performance, not explain it. The survey includes many highly correlated questions measuring satisfaction with the overall process or broad components of it, such as DOD board liaisons, VA case managers, or timeliness. While multiple questions can improve the statistical reliability and validity of DOD’s performance measures, they require costly survey administration time that could be used for other purposes, such as to measure a larger number of variables that could explain servicemember satisfaction or case processing times.

VTA data limitations: The VTA administrative data that we matched to survey data primarily measure processing times and basic servicemember demographics, such as service branch, component, and treatment facility. The data support detailed reporting of performance measures, but they do not measure similarly detailed information on the nature of each case that might allow DOD and VA to understand the reasons for lengthy case processing times or to identify cases that might become delayed and ensure that they remain on schedule. For example, the database does not measure the type or severity of referred medical conditions in detail, the nature of delays experienced early in the process, or the use of servicemember leave. In addition, little information is available on staffing at or caseloads for MEB and PEB locations, DOD board liaisons, or VA case managers, which might help to explain or predict performance.

Low response and coverage rates: The response and coverage rates of the satisfaction survey further limit the degree to which DOD can generalize the data obtained to the population of servicemembers who participate in IDES. In particular, the survey does not assess the views of servicemembers who disenroll from the process before finishing a stage or those who do not complete prior waves of the survey. Including servicemembers who do not complete all waves would complicate longitudinal analysis, however.

Appendix III: Monthly DOD Timeliness Data for Active Duty Cases in Fiscal Year 2012

Table 13 presents data reported by DOD on average processing time for active duty cases completed during part of fiscal year 2012—Oct. 2011 to June 2012. DOD’s data are provided as a supplement the analyses GAO conducted for fiscal years 2008 through 2011. We did not evaluate the reliability of these data and cannot predict the extent to which any trends will continue for the rest of the fiscal year.

Appendix IV: Comments from the Department of Defense

Appendix V: Comments from the Department of Veterans Affairs

Appendix VI: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

Michele Grgich (Assistant Director), Daniel Concepcion, Melissa Jaynes, and Greg Whitney made significant contributions to all aspects of this report. Also contributing to this report were Bonnie Anderson, James Bennett, Mark Bird, Joanna Chan, Brenda Farrell, Jamila Jones Kennedy, Douglas Sloane, Almeta Spencer, Vanessa Taylor, Jeffrey Tessin, Roger Thomas, Walter Vance, Kathleen van Gelder, and Sonya Vartivarian.

Related GAO Products

Military Disability System: Preliminary Observations on Efforts to Improve Performance. GAO-12-718T (Washington, D.C.: May 23, 2012).

Military and Veterans Disability System: Worldwide Deployment of Integrated System Warrants Careful Monitoring. GAO-11-633T (Washington, D.C.: May 4, 2011).

Military and Veterans Disability System: Pilot Has Achieved Some Goals, but Further Planning and Monitoring Needed. GAO-11-69 (Washington, D.C.: December 6, 2010).

Military and Veterans Disability System: Preliminary Observations on Evaluation and Planned Expansion of DOD/VA Pilot. GAO-11-191T (Washington, D.C.: November 18, 2010).

Veterans’ Disability Benefits: Further Evaluation of Ongoing Initiatives Could Help Identify Effective Approaches for Improving Claims Processing. GAO-10-213 (Washington, D.C.: January 29, 2010).

Recovering Servicemembers: DOD and VA Have Jointly Developed the Majority of Required Policies but Challenges Remain. GAO-09-728 (Washington, D.C.: July 8, 2009).

Recovering Servicemembers: DOD and VA Have Made Progress to Jointly Develop Required Policies but Additional Challenges Remain. GAO-09-540T (Washington, D.C.: April 29, 2009).

Military Disability System: Increased Supports for Servicemembers and Better Pilot Planning Could Improve the Disability Evaluation Process. GAO-08-1137 (Washington, D.C.: September 24, 2008).

DOD and VA: Preliminary Observations on Efforts to Improve Care Management and Disability Evaluations for Servicemembers. GAO-08-514T (Washington, D.C.: February 27, 2008).

DOD and VA: Preliminary Observations on Efforts to Improve Health Care and Disability Evaluations for Returning Servicemembers. GAO-07-1256T (Washington, D.C.: September 26, 2007).

Military Disability System: Improved Oversight Needed to Ensure Consistent and Timely Outcomes for Reserve and Active Duty Service Members. GAO-06-362 (Washington, D.C.: March 31, 2006).