Military and Veterans Disability System: Pilot Has Achieved Some Goals, but Further Planning and Monitoring Needed
Released: 2010-12-06T00:00:00

Letter

Background

The DOD Legacy Disability Evaluation System

The military’s legacy disability evaluation process begins at a military treatment facility when a physician identifies a condition that may interfere with a servicemember’s ability to perform his or her duties. On the basis of medical examinations and the servicemember’s medical records, a medical evaluation board (MEB) identifies and documents any conditions that may limit a servicemember’s ability to serve in the military.

The servicemember’s case is then evaluated by a physical evaluation board (PEB) to make a determination of fitness or unfitness for duty. Each of the services conducts this process for its servicemembers. The Army has three PEBs, which are located at Fort Sam Houston, Texas; Walter Reed Army Medical Center in Washington, D.C.; and Fort Lewis, Washington. The Navy and Air Force each have one PEB: the Navy’s is located at the Washington Navy Yard in Washington, D.C., and the Air Force’s is located in San Antonio, Texas. The PEB process begins with an “informal” PEB— an administrative review of the case file by PEB adjudicators without the presence of the servicemember. If the servicemember is found to be unfit due to medical conditions incurred in the line of duty, the informal PEB assigns the servicemember a combined percentage rating for those unfit conditions, and the servicemember is discharged from duty. Disability ratings range from 0 (least severe) to 100 percent (most severe) in increments of 10 percent. Depending on the overall disability rating and number of years of active duty or equivalent service, the servicemember found unfit with compensable conditions is entitled to either monthly disability retirement benefits or lump sum disability severance pay.

Servicemembers have opportunities to appeal the results of their disability evaluations. If servicemembers are dissatisfied with the informal PEB’s decisions, they may request a hearing with a “formal” PEB. If they then disagree with the formal PEB’s findings, they can, under certain conditions, appeal to the reviewing authority of the PEB.

As servicemembers navigate DOD’s disability evaluation system, they interface with staff who play key roles in supporting them through the process. Military physicians involved in the MEB process play a fundamental role because they are responsible for documenting in the disability evaluation case file the medical conditions that may limit a servicemember’s ability to serve in the military. To prepare this documentation, military physicians may require that servicemembers obtain additional medical evidence from specialty physicians, such as a psychiatrist. Throughout the MEB and PEB processes, board liaisons serve a key role by explaining the process to servicemembers and constructing the case files. The liaisons inform servicemembers of board results and of deadlines at key decision points in the process. The military also provides legal counsel to advise and represent servicemembers going through the disability evaluation process, although servicemembers may retain their own representative at their own expense.

The VA Disability Claims Process

In addition to receiving disability benefits from DOD, veterans with service-connected disabilities may receive compensation from VA for lost earnings capacity. In contrast to DOD’s disability evaluation system, which evaluates only medical conditions affecting servicemembers’ fitness for duty, VA evaluates all medical conditions claimed by the veteran, whether or not they were previously evaluated by the military services’ medical evaluation process. Although a servicemember may file a VA claim while still in the military, he or she can only obtain disability compensation from VA as a veteran.

VA’s disability compensation claims process starts when a veteran submits a claim to VA’s Veterans Benefits Administration (VBA). The claim lists the medical conditions that the veteran believes are service-connected. For each claimed condition, VA must determine if credible evidence is available to support the veteran’s contention of service connection. A service representative assists the veteran in gathering the relevant evidence to evaluate the claim, which may include the veteran’s military service records and treatment records from VA medical facilities and private medical service providers. Also, if necessary for reaching a decision on a claim, VBA arranges for the veteran to receive a medical examination conducted by clinicians (including physicians, nurse practitioners, or physician assistants) certified to perform the exams under VA’s Compensation and Pension program. Once a claim has all of the necessary evidence, a VA rating specialist evaluates the claim and determines whether the claimant is eligible for benefits. If so, the rating specialist assigns a percentage rating. If VA finds that a veteran has one or more service-connected disabilities with a combined rating of at least 10 percent, the agency will pay monthly compensation. The veteran can claim additional benefits over time, for example, if a service-connected disability worsens or surfaces at a later point in time.

The Integrated Disability Evaluation System

In November 2007, DOD and VA began piloting the IDES, a joint disability evaluation system to eliminate duplication in their separate systems and to expedite receipt of VA benefits for wounded, ill, and injured servicemembers. The IDES merges DOD and VA processes, so that servicemembers begin their VA disability claim while they undergo their DOD disability evaluation, rather than sequentially, making it possible for them to receive VA disability benefits shortly after leaving military service. Specifically, the IDES merges DOD and VA’s separate exam processes into a single exam process conducted to VA standards. This single exam—which may involve more than one medical examination (for example, by different specialists)—in conjunction with the servicemembers’ medical records, is used by military service PEBs to make a determination of servicemembers’ fitness for continued military service, and by VA as evidence of service-connected disabilities. The single exam may be performed by medical staff working for either VA, DOD, or a private provider contracted with either agency. consolidates DOD and VA’s separate rating phases into one VA rating phase. If the informal PEB has determined that a servicemember is unfit for duty, VA rating specialists prepare two ratings—one for the conditions that DOD determined made a servicemember unfit for duty, which DOD uses to provide military disability benefits, and the other for all service- connected disabilities, which VA uses to determine VA disability benefits. Ratings for the IDES are prepared by rating specialists at VA’s Baltimore and Seattle regional offices. provides VA case managers to perform outreach and nonclinical case management and explain VA results and processes to servicemembers.

By consolidating DOD and VA’s separate medical exams and ratings, the IDES eliminates several steps from the existing “legacy” systems (see fig. 1).

In designing the IDES, DOD and VA established goals to provide VA benefits to active duty servicemembers within 295 days of being referred into the system, and to reserve component members within 305 days. In establishing the 295- and 305-day goals, they also established timeliness goals for the specific steps of the IDES process (see fig. 2).

DOD and VA first piloted the IDES at 3 Washington, D.C., area military treatment facilities, beginning in November 2007 (see table 1). They added 18 military facilities to the pilot in fiscal year 2009 and 6 in fiscal year 2010. DOD and VA stated that expansion to additional sites was intended to assess the IDES system in a variety of geographic areas and to test the agencies’ capacity to handle additional caseload. According to DOD, the 27 pilot sites represented almost half of the servicemembers in the military services’ disability evaluation systems.

Pilot Evaluation Results Are Promising, but the Degree of Improvement Achieved Is Unknown

DOD and VA’s Evaluation Shows That the Pilot Is Achieving Some of Its Goals

In their planning documents for the IDES pilot, DOD and VA stated that they were basing their evaluation of the effectiveness of the IDES pilot on whether it has achieved three key goals relative to the legacy process: increased servicemember satisfaction, improved case-processing time, and a reduction in servicemember appeal rates. In addition, they also examined IDES program costs. To determine whether they have achieved their goals, the agencies surveyed servicemembers in the IDES pilot and legacy systems and are using a data system—called the Veterans Tracking Application (VTA)—that enables them to track case processing time and appeals. They have been monitoring their progress on these goals through weekly reports.

In August 2010, DOD and VA officials issued an interim report to Congress summarizing their evaluation results to date. In this report, the agencies concluded that servicemembers who went through the IDES pilot were more satisfied than those who went through the legacy system, and that the IDES process met the agencies’ goals of delivering VA benefits to active duty servicemembers within 295 days and to reserve component servicemembers within 305 days. Specifically, they reported that, as of February 2010, the IDES process took an average of 274 days to complete for active duty servicemembers and 281 days for reserve component members who, according to the interim report, comprise 15 percent of IDES participants. Furthermore, they concluded that the IDES pilot has achieved a faster processing time than the legacy system, which they estimated to be 540 days.

While overall results were promising, data presented in the report had some limitations, and the report itself did not include certain analyses. For example, DOD officials told us that the 540-day estimate for the legacy process was based upon a review of a small and nonrepresentative sample of legacy cases during the agencies’ “table top” planning exercise in August 2007. In addition, although DOD officials told us that they planned to compare average processing times of pilot cases with a broader sample of legacy cases, and to determine whether fewer servicemembers are appealing the findings of informal PEBs and formal PEBs in the pilot compared with the legacy, the interim report did not include these comparisons. In addition, in their planning documents for the IDES pilot, DOD and VA indicated that they were establishing a goal to deliver VA benefits to 80 percent of members in the IDES pilot within the 295- and 305-day time frames. However, their interim report did not discuss whether this goal was met.

Our review of DOD and VA’s data and weekly reports generally confirm DOD and VA’s findings, as of early 2010. However, while the agencies have largely met their overall goal to increase servicemember satisfaction and met their timeliness goal as of February 2010, since that time, case processing times have been steadily increasing as the caseload has increased. In addition, not all of the service branches are achieving the same results.

Servicemember satisfaction: Our review of the survey data that DOD used for the interim report (as of February 2010), as well as a recent weekly report, indicate that, on average, servicemembers in the IDES process have had higher satisfaction levels than those who went through the legacy process. In addition, a higher percentage of servicemembers who went through the IDES process felt that the process was fair compared with those who went through the legacy system. However, servicemembers in the Air Force who went through the IDES pilot indicated less satisfaction with the process than those who went through the legacy system, though Air Force members represented a small proportion of pilot cases—about 7 percent of those enrolled in the pilot. We reviewed the agencies’ survey methodology and generally found their survey design and conclusions to be sound (see app. I for further information on our review).

Average case processing times: The agencies have been meeting their 295- day and 305-day timeliness goals for much of the past 2 years, but more recent weekly reports indicate case processing time has been increasing and that they are now missing their goal for active duty members. As of August 29, 2010, the agencies missed the goal for active duty servicemembers by 1 day, while still meeting the 305-day goal for reserve component members by 7 days. Processing times have increased as caseload has increased, from about 5,750 active cases in February to about 9,650 cases in August 2010. We reviewed the reliability of the VTA data upon which the agencies based their analyses and generally found these data to be sufficiently reliable for purposes of these analyses.

The increases in overall case processing time and caseloads mirror the trends at individual sites. For each pilot site, case processing times have generally increased as workloads have increased. For example, figure 3 shows the case processing times 1 year or more after implementation and in August 2010 for the first seven pilot sites.

Of the four military services, only the Army and Navy were achieving the 295- and 305-day goals on average, as of February 2010, and only the Army was achieving these goals as of August 2010. Because the Army comprises a large proportion of cases (approximately 60 percent of IDES pilot cases that have completed the whole process), it has lowered the overall average processing time to near or below the established goals. Figure 4 shows the average case processing times for active duty, by service, as of August 2010. (See app. II for reserve component.)

As of February 2010, the agencies also had not met the goal of processing 80 percent of all pilot cases within targeted time frames. Specifically, about 60 percent of active duty pilot cases have been completed within 295 days, according to our analysis of the agencies’ case data intended for their interim report. Further, none of the four military services have achieved this goal, although the Army has had the highest rate of cases (66 percent) meeting the goal, while only 42 percent of Air Force cases were processed within the time frame (see fig. 5 for active duty and app. II for reserve component).

Extent of Improvement Over the Legacy System Is Unknown Due to Gaps in Legacy Data

DOD and VA planned to compare the case processing times of servicemembers in the IDES pilot and servicemembers who, between fiscal years 2005 and 2009, were enrolled in the legacy system at pilot sites prior to pilot implementation, but significant gaps in the legacy case data preclude reliable comparisons. DOD compiled the legacy case data from each of the military services and the VA, but the military services each had slightly different disability evaluation processes, used different data systems, and did not track the same information. As a result, information needed to conduct a comparison is not available for all services. For example, the Navy, Marine Corps, and Air Force legacy data do not have information on when the servicemember was referred into the disability evaluation system and, as a result, case-processing time for the legacy system DOD-wide cannot be known. DOD officials said they planned to estimate legacy case processing time by approximating the dates that servicemembers in the Navy, Marine Corps, and Air Force were referred into the disability evaluation process, but their methodology was based on a limited number of Army cases (see app. I for further information). In addition, for legacy cases across all military services, VA was not able to provide data on the date VA benefits were delivered, so total case processing time from referral to delivery of VA benefits cannot be measured. However, while legacy case data are not sufficiently reliable for comparison with the IDES overall, the Army’s legacy data appear to be reliable on some key processing dates, making some limited comparisons possible. Our analysis of Army legacy data suggests that, under the legacy process, active duty Army cases took 369 days to complete the DOD legacy process and reach the VA rating phase—though this figure does not include time to complete the VA rating and provide the benefits to servicemembers—compared with 266 days to deliver VA benefits to servicemembers under the pilot, according to the agencies’ August weekly report. However, Army comparisons cannot be generalized to the other services.

The agencies also planned to compare servicemembers’ appeal rates in the pilot and legacy systems, but similar gaps in the legacy data preclude a comparison DOD-wide. For example, the legacy data that DOD compiled did not contain data on appeals of informal PEB decisions to the formal PEB in the Navy and Marines, and consequently the rate of appeals across the military in the legacy system is unknown. While the Army’s appeals data appear to be more reliable, potentially making some limited comparisons possible, the agencies’ method for comparing pilot appeals with legacy has limitations. DOD officials told us they are planning to compare the proportion of informal PEB decisions that were appealed to a formal PEB hearing in the pilot and legacy systems. However, this will not take into account that, under the legacy system, a servicemember could appeal the informal PEB’s decision for two reasons—because they were dissatisfied with the fitness decision or the disability rating the PEB assigned, while in the IDES, they can only appeal the informal PEB decision to a formal PEB if they are dissatisfied with the fitness decision. Under the IDES, servicemembers who disagree with the disability rating can appeal to VA for a rating reconsideration. By not including appeals to VA for rating reconsiderations, the agencies may overestimate the decrease in appeals in the IDES pilot. For example, our analysis of data as of early 2010 for the Army indicates that Army members in the pilot appealed 7.5 percent of informal PEB decisions. However, when appeals to VA are factored in, 13 percent of Army members in the pilot filed an appeal, which is the same proportion as in the legacy system (see fig. 6).

In addition to evaluating the three goals, DOD and VA initially planned a cost-benefit analysis of the IDES program but have only completed an analysis of costs. According to data provided to us in August 2010, DOD projects that costs directly associated with implementing the IDES will be $63 million greater per year when compared with the legacy system, after full expansion of the IDES. In October 2010, VA reported to us total IDES cost estimates of approximately $50 million for fiscal year 2011—about $33 million for VBA, which provides VA case managers and rating staff to the IDES, and $17 million for the Veterans Health Administration (VHA), which provides medical staff to perform the single exams. These analyses did not quantify the value of potential benefits created by the pilot, for example time savings from DOD physicians no longer needing to perform disability examinations, which allows them to perform other duties.

Pilot Sites Experienced Several Challenges

As DOD and VA tested the IDES at different facilities and added caselo to the pilot, they encountered several challenges that led to delays in certain phases of the process. Among these were insufficient staffing, challenges in conducting the single exams, logistical challenges related integrating VA staff, as well as housing and managing servicemembers going through the IDES. DOD and VA were able to address some, but not all, of these challenges as they arose.

DOD and VA Did Not Sufficiently Staff Many Key Positions in the IDES Pilot

DOD and VA have not provided sufficient numbers of staff in many of th IDES locations, affecting their ability to complete certain phases of the IDES process within the goals they established. Officials at most of the 1 pilot sites we visited said they have experienced staffing shortages to a t least some extent, with a few sites—Fort Ca particular—experiencing severe shortages. rson and Fort Stewart, in VA or contract examiners: At three pilot sites we visited—Fort Carson, Fort Polk, and Fort Stewart—local officials said that a lack of VA or VA contractor staff who could perform the required single medical exams led to bottlenecks in the process. For example, as of August 2010, exams at Fort Carson have taken an average of 140 days to complete for active duty servicemembers, according to the agencies’ data, far from achieving their goal to complete single medical exams within 45 days (see fig. 7; s ee also app. II for processing times for reserve component members). Across all pilot sites, exams have taken 68 days to complete for active duty servicemembers, on average, with 8 of the 27 pilot sites meeting the 45 goal. examiners. For instance, Fort Carson’s IDES process was particularly hampered by a lack of mental health specialists; in contrast, VA officials serving the Fort Polk pilot site said perform specialty medical exams but did not have enough examiners to complete general medical exams.

The 8 pilot sites that met the 45-day goal for completing single exams include 2 Air Force sites, 5 Army sites, and 1 Navy site that met the 45-day goal for servicemembers in both the Navy and Marine Corps. One additional site (Camp Pendleton) met the 45-day goal for Navy members but did not meet it for Marine Corps members.

The Navy PEB determines fitness decisions for servicemembers in the Marine Corps.

VA rating staff: Officials at the Baltimore rating office—one of the two VA offices that conduct disability ratings for the IDES pilot—expressed significant concerns that they were understaffed, in part due to staff turnover. DOD and VA data show that, overall, the VA rating offices are not meeting the agencies’ goal to complete ratings within 15 days, taking 39 days on average for active duty servicemembers and 42 days for reserve component members. We could not determine case processing times at each individual VA rating office, since DOD and VA’s weekly monitoring reports do not provide processing times for the rating phase by office. The weekly reports also do not provide data on caseloads at each office. Although the Baltimore office currently has fewer rating staff than Seattle, VA officials said that it has prepared ratings for the majority of IDES pilot cases, based on the way in which VA has allocated cases between the two offices. The Baltimore office handles cases for the Air Force, Navy, Marines, and 5 of the 15 Army pilot sites, while the Seattle office conducts ratings for the remaining 10 Army pilot sites. VA officials said that to address staffing shortages in Baltimore, they have assigned staff from other VA offices to assist the Baltimore office.

VA case managers: DOD and VA have set a target for each VA case manager to handle no more than 30 cases at a time, but two sites we visited—Fort Carson and Fort Stewart—appeared to be far from these targets. At Fort Carson, three VA case managers told us they were handling about 900 cases when we visited in April 2010, for a caseload ratio of roughly 1:300. At the time of our visit in June 2010, Fort Stewart had over 750 active cases with two VA case managers, for a caseload ratio of approximately 1:375. Although local officials we spoke with at both sites told us that the numbers of VA case managers were insufficient, an official at VA’s central office told us that VA bases staffing of case managers on the number of new (not pending) cases each month, and the agencies’ data indicates the average number of new cases per VA case manager has been about 25 at each site. The VA official said that the reason local case managers felt understaffed was likely due to other process inefficiencies. In addition, the official told us VA can reassign staff from other VA programs to assist case managers at IDES pilot sites as needed. At some of the other pilot sites we visited, local officials also told us they had concerns at times about the numbers of VA case managers available to handle the site’s caseload, but VA was able to add staff. VA case managers at two Air Force sites we visited—Travis and Vance Air Force Bases—indicated that their caseloads were manageable. We were unable to independently determine the extent to which VA is meeting its caseload target because VA does not collect national data on actual caseloads per case manager.

DOD board liaisons: At most of the sites we visited, local officials expressed concerns about insufficient numbers of DOD board liaisons, who serve as servicemembers’ DOD case managers. DOD guidance has been inconsistent on the caseload target for DOD board liaisons. While DOD’s operations manual for the IDES pilot sets a caseload target of at most 30 cases per board liaison, guidance on the general disability evaluation system sets the target at a maximum of 20 cases per liaison. DOD and VA’s documents related to planning for IDES expansion indicate that DOD is striving for a 1:20 caseload target in the IDES. However, 19 of the 27 pilot sites did not meet the 1:30 caseload target, and 23 did not meet the 1:20 target (see fig. 10).

Local DOD and VA officials attributed staffing shortages to higher than anticipated caseloads and difficulty finding qualified staff in rural areas. At several of the pilot sites we visited, officials said that caseloads were higher than the initial estimates that they had based staffing levels upon. DOD officials said that they had based caseload estimates on a 1-year history of caseload at each site. While some sites have added staff as caseloads increased, others, such as Fort Polk, located in central Louisiana, have had difficulty finding qualified staff, particularly physicians, in this rural area.

Two of the pilot sites we visited—Fort Carson and Fort Stewart—were particularly challenged to provide staff in response to surges in caseload, which occurred when Army units were preparing to deploy to combat zones. Through the Army’s predeployment medical assessment process, large numbers of servicemembers were determined to be unable to deploy due to a medical condition and were referred to the IDES within a short period of time, overwhelming the staff. These two sites were unable to quickly increase staffing levels, particularly clinicians performing the single exam. The VA medical center conducting the single medical exams for Fort Carson experienced turnover among its examiners at the same time that the caseload surged, while at Fort Stewart, the contractor performing the single medical exams had difficulties finding qualified physicians in a rural area of Georgia. To address caseload surges, examiners were reassigned from other locations to the pilot sites. For example, VA officials told us they assigned examiners from other VA medical centers to the Fort Carson IDES and established a contract with a private-sector provider to complete the exams that VA examiners would normally have performed for veterans in the area claiming VA disability compensation. At Fort Stewart, the contractor told us that they had reassigned examiners from their Atlanta clinic to Fort Stewart.

Insufficiency of Exam Summaries and Disagreements about Medical Diagnoses and Ratings Can Prolong Case Processing Time

Issues related to the completeness and clarity of single exam summaries were an additional cause of delays in the VA rating phase of the IDES process. Officials from VA rating offices said that some exam summaries did not contain information necessary to make a rating or fitness decision, or were unclear as to the examiners’ diagnoses and conclusions. As a result, VA rating office staff must ask the examiner to clarify the summary or add information and, in some cases, redo the exam, adding time to the process. In addition, VA rating staff told us that it is sometimes unclear who they should contact if they identify insufficiencies in an exam summary and finding the appropriate person also adds time. However, the extent to which insufficient exam summaries caused delays in the IDES process is unknown because DOD and VA’s VTA system does not track whether an exam summary had to be returned to the examiner or whether it was resolved. Due to these limitations, VA officials told us that VA rating staff have created logs of outstanding insufficient exams and sent them to VA examiners to correct.

VA officials attributed the problems with exam summaries to several factors, including the difficulty of conducting exams for IDES pilot cases, which may entail evaluating many complex medical conditions and may involve several physicians and specialists. In addition, VA officials indicated that, at sites with exam backlogs, such as at Fort Carson, it may be difficult for examiners to ensure quality when are trying to complete exams quickly. Furthermore, VA staff noted that some errors were common, such as missing information for musculoskeletal conditions and traumatic brain injury, suggesting that some examiners may not be aware of the information required for certain types of medical conditions. Finally, while examiners are supposed to receive the servicemember’s complete medical records prior to the date of the exam, some VA examiners also told us that they did not receive the records in time for the exam in some cases, or the records were not well-organized. As a result, they lacked key information, such as the servicemember’s medical history and results of laboratory tests. According to the agencies’ operations manual for the IDES pilot, the DOD board liaison should compile the complete medical records within 10 days of an active duty servicemember being referred to the IDES, but some DOD officials we spoke with said that it is sometimes difficult to obtain all of the records, particularly when servicemembers have received treatment from private-sector physicians.

In addition, while the single exam in the IDES eliminates duplicative exams performed by DOD and VA in the legacy system, it raises the potential for there to be disagreements about diagnoses of servicemembers’ conditions, with implications for their disability ratings, as well as processing times. DOD officials we spoke with in our interviews and site visits also said that their physicians sometimes disagree with VA medical diagnoses, particularly for mental health conditions, and this has extended processing times for some cases. In addition, since medical diagnoses are a basis for VA’s disability ratings, DOD may subsequently disagree with the ratings VA completed for determining DOD disability benefits. The number of cases with disagreements about diagnoses and ratings, and the extent to which they have increased processing time, are unknown because the VTA system does not track when a case has had such disagreements. However, officials at 4 of the 10 pilot sites we visited said that military physicians have disagreed with VA diagnoses in at least some cases. In addition, PEB officials in two of the three military services—the Army and the Navy—said that they have sometimes disagreed with the rating VA produced for determining DOD disability benefits.

An example can illustrate the implications of differences in diagnoses. Officials at Army pilot sites informed us about cases in which a military physician had treated members for a mental condition, such as anxiety or depressive disorder. However, when the members went to see the VA examiners for their single exam, the examiners diagnosed them with posttraumatic stress disorder (PTSD). When such cases were sent to the PEB, it returned them to the MEB because it was unclear to the PEB which conditions should be the basis of their decision on the servicemembers’ fitness for duty. The cases then languished because the military physicians experienced difficulties resolving the discrepancy with the VA diagnosis.

To address such processing delays, the Army issued guidance in February 2010 stating that MEB physicians should review all of the medical records (including the results of the single exam) and determine whether to revise their diagnoses. If after doing so the MEB physician maintains that their original diagnosis is accurate, they should write a memorandum summarizing the basis of their decision, and the PEB should accept the MEB’s diagnosis. Some Army officials we spoke with believe that this guidance has been helpful for enabling cases to move forward when there are differences in diagnoses. The other services do not have written guidance on how to address differences in diagnoses, though Navy officials told us that they have provided verbal guidance to their physicians, and Air Force officials said they have not had cases with significant disagreements about diagnoses.

In some cases, due to the differences in diagnoses, DOD has also disagreed with the rating that VA prepared for DOD disability benefits, particularly in cases involving servicemembers with mental health conditions. For example, Army and Navy officials told us about cases in which the PEB found the servicemember unfit due to a mental condition, such as major depression, and asked VA to complete a rating for this condition. However, VA returned a rating for occupational and social impairment caused by PTSD, since the examiner had diagnosed the member with PTSD. DOD requires a rating for only the conditions for which the member was found unfit for duty because it can only provide disability benefits for those conditions. However, according to VA regulations for rating mental disorders, VA does not rate each mental health condition individually; rather, VA bases its rating on the degree to which the combination of symptoms of mental disorders cause occupational and social impairment. As such, when rating mental health conditions for IDES cases, VA officials said that rating specialists would consider both the symptoms of mental conditions diagnosed by DOD physicians and those identified by the VA examiner. Both Army and Navy PEB officials said that they generally accept VA ratings in these cases, even though the rating is not for the unfitting conditions alone. However, they noted that, if they feel the VA rating is in error, there is no guidance on how disagreements about servicemembers’ ratings should be resolved. Army and Navy officials said that they may return the case to VA and informally requ est that they reconsider the case, though Navy PEB officials said that they are hesitant to do so because it may further delay the case.

DOD and VA officials attributed disagreements about diagnoses to several factors. They noted that VA examiners may not have received or reviewed the servicemembers’ medical records prior to the exam, and therefore may not be aware of the medical conditions for which the members had been previously diagnosed and treated. In addition, DOD and VA identify conditions for different purposes in the disability evaluation system. While DOD identifies conditions that make a servicemember unable to perform their duties, VA identifies all service-connected conditions. As such, VA examiners are likely to identify a broader set of conditions than DOD’s physicians. In addition, local officials we spoke with in some of our site visits said that servicemembers may be more willing to disclose all of their medical conditions to VA than to DOD because VA could potentially compensate them for all of the conditions. Furthermore, VA officials noted that servicemembers’ health conditions may have changed between the time DOD physicians identified the conditions and VA performed the exam. Finally, DOD and VA officials said that differences in opinions about diagnoses are common among physicians, particularly in the mental health field. For example, they noted that it be can be difficult to distinguish PTSD from anxiety, depression, and other mental health conditions.

Pilot Sites Faced Various Logistical Challenges Integrating VA Staff

DOD and VA officials at several pilot sites said that they experienced some logistical challenges integrating VA staff at the military facilities. At a few sites, it took time for VA staff to receive common access cards needed to access the military facilities and to use the facilities’ computer systems. During the time that VA staff did not have access cards, they were unable to access VA computer systems, such as those for establishing the VA claim, requesting exams, and viewing exam results, via DOD’s network.

In addition, DOD and VA staff noted several difficulties using the agencies’ multiple information technology (IT) systems to process cases. While the agencies both use the VTA system to manage cases, VA also has IT systems for completing certain tasks, and the military services also have their own case tracking systems. This causes DOD and VA staff to have to enter the same data multiple times into different IT systems. In addition, some VA staff working on military bases reported that using the military services’ computer systems to access VA systems has significantly slowed down computer processing speeds. Finally, DOD and VA staff cannot directly access each others’ systems, making it more cumbersome for case managers to determine the status of servicemembers’ cases. For example, without access to VA’s system for managing exams, DOD board liaisons cannot readily provide servicemembers with information on when or where their exams are scheduled and must contact VA case managers to obtain the information. A few sites we visited were able to address some IT issues. For example, at Fort Polk, VA officials said they were adding a new telecommunications line to provide faster computer processing speeds for their staff.

In addition, VA physicians working at military facilities need to be credentialed by DOD before they can begin working on base, which involves verification of their education, license, and clinical history. Some VA officials said that this process could take 1 month or longer to complete.

Extended Periods in the Military Disability Evaluation Process Posed Housing and Other Challenges at Some Pilot Sites

Although many DOD and VA officials we interviewed at central offices and pilot sites felt that the IDES process expedited the delivery of VA benefits to servicemembers, several also indicated that it may increase the amount of time servicemembers are in the military’s disability evaluation process. Data on legacy cases are not sufficiently reliable to determine whether this is the case military-wide, but Army data appear to be sufficiently reliable to allow for some limited analysis. Our analysis of Army pilot and legacy data as of early 2010 shows that compared with legacy cases, active duty cases in the pilot took on average 39 more days to reach the end of the PEB phase—the last step of the DOD disability evaluation process before servicemembers begin transitioning from military service or, if found fit, back to duty. For reserve component cases in the Army, IDES pilot cases took on average 17 more days to reach the end of the PEB phase, compared with legacy cases. It was not possible to conduct this analysis for the other military services because their legacy data lacked information on when servicemembers were referred into the disability evaluation system.

Some DOD officials noted that the increased time that servicemembers are in the military’s disability evaluation process means that they must be cared for and managed for a longer period. Officials in our site visits and interviews said that some pilot sites have had challenges housing servicemembers in the IDES, in part due to servicemembers being in the process longer. For some servicemembers in the disability evaluation system, the military services may move them to temporary medical units or, for those needing longer-term medical care or complex case management, to special medical units such as a Warrior Transition Unit in the Army or Wounded Warrior Regiment in the Marine Corps. However, these units were full at a few pilot sites we visited, or members in the IDES did not meet the criteria for entering the special medical units. Where servicemembers remain with their units while going through the disability evaluation system, the units cannot replace them with able-bodied members. Officials at Fort Carson said that this created a challenge for combat units. Because most servicemembers in the IDES did not meet the criteria for entering Warrior Transition Units, combat units had to find another organizational unit to take charge of members in the IDES so they could replace them with soldiers ready and able to deploy to combat areas. In addition, officials at Naval Medical Center San Diego and Fort Carson said that some members are not gainfully employed by their units and, left idle while waiting to complete their disability evaluation process, are more likely to engage in negative behavior, potentially resulting in their being discharged due to misconduct and a forfeiture of disability benefits. We were unable to assess the extent or cause of this problem because the VTA system that tracks servicemembers in the IDES does not capture sufficient detail on reasons for servicemembers dropping out of the IDES, or which organizational unit(s) the servicemember was assigned to while in the IDES. DOD officials also noted that servicemembers benefit from continuing to receive their salaries and benefits while their case undergoes scrutiny by two agencies, though some also acknowledged that these additional salaries and benefits create costs for DOD.

DOD and VA Expansion Plans Address Some Though Not All Challenges

DOD and VA Have Incorporated Many Lessons Learned into Their Planning for Worldwide Expansion of the IDES but Lack Concrete Plans for Addressing Some Challenges

DOD and VA plan to expand the IDES to sites worldwide on an ambitious timetable—to 113 sites during fiscal year 2011, a pace of about 1 site every 3 days. Expansion is scheduled to occur in four stages, beginning with 28 sites in the southeastern and western United States by the end of December 2010.

DOD and VA have many efforts under way to prepare for IDES expansion. At each site, local DOD and VA officials are expected to work together to prepare for implementation. This includes completing a site assessment matrix—a checklist of information DOD and VA officials at each site should obtain and preparations they should make. While most pilot sites had used a site assessment matrix to prepare for IDES implementation, the agencies completed a significant revision of the matrix in August 2010, and they now request additional information and documentation to address areas where prior IDES sites had experienced challenges. In addition, while during the pilot phase local DOD and VA officials were encouraged to develop written agreements on IDES procedures, the matrix now requests that a written agreement be completed prior to implementing the IDES. Finally, senior-level local DOD and VA officials will be expected to sign the site assessment matrix to certify that a site is ready for IDES implementation. This differs from the pilot phase where, according to DOD and VA officials, some sites implemented the IDES without having been fully prepared. In addition, in September 2010, the military services and VA held preimplementation training conferences for local DOD and VA staff. At the time of our review, the first 28 expansion sites were completing their site assessment matrices.

Through the new site assessment matrix and other initiatives under way, DOD and VA are addressing several of the challenges identified in the pilot phase. These include ensuring sufficient exam and case management staff, being prepared to deal with surges in caseloads, addressing exam sufficiency issues, and making adequate logistical arrangements.

Ensuring sufficient exam resources: The matrix asks whether a site can complete single exams within the IDES’ 45-day time frame and within DOD’s TRICARE access standards. The matrix asks for detailed information, such as who will conduct the exams (VA, VA contractor, or military providers), where the exams will be conducted, and VA’s anticipated overall volume of disability compensation and pension exams in the area. In addition to the matrix, VA has several initiatives under way to increase resources and expedite exams. VA plans to award a new contract under which it can acquire examiners for sites that do not have sufficient staff to perform exams, such as sites located where VA does not have medical facilities or in rural areas where VA has had difficulty hiring staff. VA has also recently changed its exam policy so that exams performed by nurse practitioners or physician assistants certified to perform disability exams no longer have to be cosigned by a physician, which is expected to expedite completion of more exam reports.

Ensuring sufficient VA rating staff: VA officials said that they have hired new staff to replace those that recently left the Baltimore rating office and anticipate hiring a small number of additional staff. Based on caseload projections, they expect that, once the additional staff are hired, the Baltimore office will be close to having sufficient rating staff. Although VA officials said that the Baltimore office conducted ratings for a majority of cases during the IDES pilot phase, they have projected that the workload will be divided almost evenly between the Baltimore and Seattle offices once the IDES is fully expanded worldwide.

Ensuring sufficient DOD PEB adjudicators: Air Force officials informed us they added adjudicators for the informal PEB and have since eliminated their case backlog. They are currently adding adjudicators for the formal PEB. Navy PEB officials also said that they are adding adjudicators through activation of reserve component personnel for special work and expected that they would be in place by November 2010.

Ensuring sufficient case management staffing: The site assessment matrix also asks whether local facilities will have sufficient trained DOD board liaison staff to meet a 1:20 caseload ratio and sufficient VA case managers to meet a 1:30 caseload ratio. In addition, according to DOD officials, each of the military services is increasing its board liaison staffing levels to achieve 1:20 caseload ratios. VA officials said that they plan to hire an additional 73 case managers.

Coping with caseload surges: The matrix asks sites to provide a longer and more detailed caseload history—a 2-year, month-by-month history— as opposed to the 1-year history that DOD based its caseload projections on during the pilot phase. In addition, the matrix asks sites to anticipate any surges in caseloads, such as those due to seasonal trends. Sites are also expected to provide a written contingency plan for dealing with caseload surges. In addition, the matrix asks sites to develop a system for communicating updates, such as information on expected caseload surges, to stakeholders. VA officials also said that the Army has agreed to keep them better informed of deployments that could result in caseload surges. Further, VA officials noted that they are developing a plan for addressing the additional need for examiners during surges, through which VA offices with lower demand for disability exams would send examiners to an IDES site experiencing a surge in exam workloads.

Ensuring the sufficiency of single exams: The site assessment matrix asks sites whether all staff who will conduct exams are trained to VA standards and certified by VA to conduct disability compensation and pension exams. In addition, VA has begun the process of revising its exam templates, to better ensure that examiners include the information needed for a VA disability rating decision and enable them to complete their exam reports in less time. Finally, a VA official stated that VA is examining whether it can add capabilities to the VTA system that would enable staff to identify where problems with exams have occurred and track the progress of their resolution. For sites that choose to have military physicians perform the single exams, VA officials said that they have provided materials to DOD from their national training program, and DOD has made these materials accessible on its Web site. To help improve the ability of DOD board liaisons to obtain servicemembers’ medical and personnel records prior to the exam, DOD officials said that they are revising their policies to require reserve component units to provide the records when a reserve member is referred to the IDES.

Ensuring adequate logistics at IDES sites: The site assessment matrix asks sites whether they have the logistical arrangements needed to implement the IDES, including necessary facilities, IT, and transportation for servicemembers to exam locations. For example, the matrix asks whether the military treatment facility will address the needs of VA staff for access cards, identification badges, and security clearances, and whether all VA medical providers will be credentialed and privileged to practice at the DOD facility. In terms of IT, the matrix asks whether DOD sites will enable VA staff access to VA information systems needed to perform their duties. The matrix also asks sites to identify IT contacts from both VA and DOD so that they may work together to resolve IT problems. Furthermore, DOD and VA are developing a general memorandum of agreement on IDES information sharing. This agreement is intended to enable DOD and VA staff access to each other’s IT systems, for example, to allow DOD staff to track the status of VA exams. DOD officials also said that they are developing two new IT solutions. According to officials, one system currently being tested would help military treatment facilities better manage their cases. Another IT solution, still at a preliminary stage of development, would integrate the VTA with the services’ case tracking systems so as to reduce multiple data entry.

However, in some areas, DOD and VA’s efforts to prepare for IDES expansion do not fully address some challenges or are not yet complete.

Ensuring sufficient military physician staffing: While DOD and VA are taking steps to address shortages of examiners, case managers, and adjudicators, they do not yet have strategies or plans to address potential shortages of military physicians for completing MEB determinations. For example, the site assessment matrix does not include a question about the sufficiency of military providers to handle expected numbers of MEB cases at the site, or ask sites to identify strategies for ensuring sufficient military physicians if there is a caseload surge or staff turnover.

Ensuring sufficient housing and organizational oversight for IDES participants: Although the site assessment matrix asks sites whether they will have sufficient temporary housing available for servicemembers going through the IDES, the matrix requires only a yes or no response and does not ensure that sites will have conducted a thorough review of their housing capacity prior to implementing the IDES. For example, sites are not asked about the capacity of their medical hold units or special units for wounded servicemembers, or to identify other options if their existing units do not have sufficient capacity for their projected IDES caseload. In addition, the site assessment matrix does not address whether sites have plans for ensuring that IDES participants are gainfully employed or sufficiently supported by their organizational units.

Addressing differences in diagnoses: According to a DOD official, as part of its revision of its IDES operations manual, DOD is currently developing guidance on how staff should address differences in diagnoses between military physicians and VA examiners, and between military PEBs and VA disability rating staff. DOD anticipated issuing the new guidance in September 2010, but at the time of our review had not yet done so. In addition, a VA official stated that VA is developing new procedures for identifying cases with potential for multiple mental health diagnoses and will ask VA examiners to review the servicemembers’ medical records and reconcile differing diagnoses. However, since the new guidance and procedures are still being developed, we cannot determine whether they will resolve discrepancies or disagreements. Significantly, DOD and VA do not have a mechanism for tracking disagreements about diagnoses and ratings, and consequently, may not be able to determine whether the guidance sufficiently addresses the discrepancies or whether it requires further revision.

DOD and VA Lack a Mechanism for Monitoring Problems That May Emerge with Full Implementation

As DOD and VA move quickly to implement the IDES worldwide, they have some mechanisms in place to monitor challenges that may arise in the IDES. DOD officials said that they expect to continue holding postimplementation “hotwash” meetings, in which they review individual sites’ implementation. In addition, DOD and VA will continue to regularly collect and report data on caseloads, processing times, and servicemember satisfaction. Furthermore, the new site assessment matrix asks sites to develop plans for VA and DOD local staff to meet weekly for the first 60 to 90 days after implementing the IDES, then no less than monthly to address any identified challenges. VA officials also said that they will continue to prepare a report on an annual basis on challenges in the IDES. To prepare this report, they will obtain input and data from local DOD and VA officials.

However, DOD and VA do not have a system-wide monitoring mechanism to help ensure that steps they took to address challenges are sufficient and to identify problems in a more timely basis. For example, they do not collect data centrally on staffing levels relative to caseload. Consequently, despite efforts to acquire additional staff, as local sites experience staffing turnover in the future, DOD and VA central offices may not become aware that a site is short-staffed until their monitoring reports show lengthy processing times. As a result, DOD and VA may be delayed in taking corrective action, since it takes time to assess what types of staff are needed at a site and to hire or reassign staff. In addition, without information on when or how often other problems occur, such as insufficient exam summaries or disagreements about diagnoses, DOD and VA managers may not be able to target additional training or guidance where needed. Furthermore, while DOD and VA report data on processing times by phase of the process, military treatment facility, and military service, their monitoring reports do not show processing times or caseloads for each VA rating office and each of the five PEBs (three Army and one each for the Navy and Air Force), limiting their ability to identify if specific rating or PEB offices are experiencing challenges.

DOD and VA also lack mechanisms or forums for systematically sharing information on challenges as well as best practices. For example, while the site assessment matrix indicates that sites are expected to hold periodic meetings to identify local challenges, DOD and VA have not established a process for local sites to systematically report those challenges to DOD and VA management and for lessons learned to be systematically shared system-wide. During the pilot phase, VA surveyed pilot sites on a monthly basis about challenges they faced in completing single exams. Such a practice has the potential to provide useful feedback if extended to other IDES challenges.

Conclusions

By merging two duplicative disability evaluation systems, the IDES shows promise for expediting the delivery of VA benefits to servicemembers leaving the military due to a disability. Servicemembers who proceed through the process are able to leave the military with greater financial security, since they receive disability benefits from both agencies shortly after discharge. Further, having both DOD and VA personnel involved in reviewing each disability evaluation may result in a more thorough scrutiny of cases and informed decisions on behalf of servicemembers.

However, piloting of the system at 27 sites has revealed several significant challenges that require careful management attention and oversight before DOD and VA expand the system military-wide. DOD and VA are currently taking steps to address many of these challenges, and the agencies have developed a site implementation process that encourages local DOD and VA officials to identify and resolve local challenges prior to transitioning to the new system. However, given the agencies’ ambitious implementation schedule—more than 100 sites in a year—it is unclear whether all of these challenges will be fully dealt with before DOD and VA deploy the integrated system to additional military facilities. For example, it is unclear whether sites will have sufficient military physicians to complete key steps of the process in a timely manner. Insufficient staffing of any one part of the process is likely to lead to bottlenecks, delaying not only servicemembers’ receipt of disability benefits, but also their separation from the military and reentry into civilian life. In addition, DOD’s preparations of sites for the IDES do not ensure that military facilities have adequate capacity or plans for housing and providing organizational oversight over servicemembers in the IDES, who potentially could remain at the locations for extended periods of time. Furthermore, while integrating VA medical exams into DOD’s disability evaluation system eliminates duplicative exams, it raises the potential for there to be disagreements about diagnoses of servicemembers’ conditions, with implications for servicemembers’ disability ratings and their DOD disability compensation. While DOD is developing guidance to address such disagreements, it is important that the agencies have a thorough understanding of how often and why these disagreements occur and continually review whether their new guidance adequately addresses this issue so as to be able to make improvements where needed.

Successful implementation of any program requires effective monitoring. DOD and VA currently have mechanisms to track numbers of cases processed, timeliness, and servicemember satisfaction, but they do not routinely monitor factors—such as staffing levels relative to caseload, disagreements about diagnoses, and insufficient exam summaries—that can delay the process. In addition, they do not monitor timeliness and caseloads for some of the key IDES offices, namely each VA rating office and each PEB. Ultimately, the success or failure of the IDES will depend on DOD and VA’s ability to sufficiently staff local sites, the VA rating offices, and the PEBs, and to resolve other challenges not only at the initiation of the transition to IDES but also on an ongoing, long-term basis. By not monitoring staffing and other risk factors, DOD and VA may not be able to ensure that their efforts to address these factors are sufficient or to identify problems as they emerge and take immediate steps to address them before they become major problems.

Recommendations for Executive Action

To ensure that the IDES is sufficiently staffed and that military treatment facilities are prepared to house personnel in the IDES, we recommend that the Secretary of Defense direct the military services to conduct thorough assessments prior to each site’s implementation of the IDES of the following three issues: the adequacy of staffing of military physicians for completing MEB determinations at military treatment facilities; contingency plans should be developed to address potential staffing shortfalls, for example, due to staff turnover or caseload surges; the availability of housing for servicemembers in the IDES at military facilities; alternative housing options should be identified if sites do not have adequate capacity; and the capacity of organizational units to absorb servicemembers undergoing the disability evaluation; plans should be in place to ensure servicemembers are appropriately and constructively engaged.

To improve their agencies’ ability to resolve differences about diagnoses of servicemembers’ conditions, and to determine whether their new guidance sufficiently addresses these disagreements, we recommend that the Secretaries of Defense and Veterans Affairs take the following two actions: conduct a study to assess the prevalence and causes of such establish a mechanism to continuously monitor disagreements about diagnoses between military physicians and VA examiners and between PEBs and VA rating offices.

To enable their agencies to take early action on problems at IDES sites postimplementation, we recommend that the Secretaries of Defense and Veterans Affairs develop a system-wide monitoring mechanism to identify challenges as they arise in all DOD and VA facilities and offices involved in the IDES. This system could include: continuous collection and analysis of data on DOD and VA staffing levels, sufficiency of exam summaries, and diagnostic disagreements; monitoring of available data on caseloads and case processing time by individual VA rating office and PEB; and a formal mechanism for agency officials at local DOD and VA facilities to communicate challenges and best practices to DOD and VA headquarters offices.

Agency Comments and Our Evaluation

We provided a draft of this report to DOD and VA for review and comment. The agencies provided written comments, which are reproduced in appendixes III and IV. DOD and VA generally concurred with our recommendations. Each agency also provided technical comments, which we incorporated as appropriate.

DOD concurred with our recommendation to ensure that, before the IDES is implemented at each new site, a thorough assessment be done of the site’s staffing adequacy, the availability of housing for servicemembers in the IDES, and the capacity of organizational units to appropriately and constructively engage servicemembers in the IDES. However, DOD stated that the IDES site assessment matrix addresses plans to ensure that servicemembers are gainfully employed while in the IDES. We changed our report to more clearly indicate that the site assessment matrix does not, in fact, address such plans. We believe that specifically identifying this in the matrix could help local DOD officials, including servicemembers’ unit commanders, focus on ensuring gainful employment or other support.

DOD concurred, and VA concurred in principle, with our recommendation to study and establish mechanisms to monitor diagnostic differences. VA identified a plan to study the prevalence and causes of diagnostic differences and determine by July 1, 2011, whether mechanisms are needed. DOD stated that it expects, as diagnostic differences are monitored and studied, that the agencies will address and resolve many of the issues identified in our report. We agree that the planned study could yield valuable insights on how to resolve diagnostic differences but emphasize that continuous monitoring of such differences over a period of time may be needed to assess the extent and nature of such differences, as well as the success of any actions to address them.

Both agencies concurred with our recommendation to develop monitoring mechanisms to help them take early actions on problems that may arise at IDES sites postimplementation. VA stated that the VTA system currently has data that can be monitored by PEB and VA rating site, and DOD said its weekly monitoring report could be modified to present these data. Also, VHA plans to monitor the IDES exam workload, including numbers of exam requests compared with forecasts, exam timeliness, and insufficient exams. Implementation is scheduled for December 31, 2010. In terms of identifying site implementation problems for quick resolution, DOD stated that the military services bring sites’ challenges and best practices to the Disability Advisory Council, a DOD body that includes VA representatives, which is being re-chartered as part of the Benefits Executive Council, a subgroup of the VA-DOD Joint Executive Council. VA and DOD’s plans sound promising and consistent with our recommendations provided that they allow for ongoing monitoring of site staffing levels and create a systematic way for local DOD and VA staff to communicate their challenges or best practices, enabling the agencies to identify and address problems at an early stage.

We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Secretary of Veterans Affairs, and other interested parties. The report is also available at no charge on the GAO Web site at www.gao.gov.

If you or your staff members have any questions about this report, please contact me at (202) 512-7215 or at bertonid@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Staff members who made key contributions in this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

In conducting our review of the integrated disability evaluation system (IDES) piloted by the Departments of Defense (DOD) and Veterans Affairs (VA), our objectives were to examine (1) the results of DOD and VA’s evaluation of the IDES pilot, (2) challenges in implementing the piloted system to date, and (3) DOD and VA plans to expand the piloted system and whether those plans adequately address potential challenges. We conducted this performance audit from November 2009 to December 2010, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Review of DOD and VA’s Evaluation of the IDES Pilot

To address objective 1, we reviewed DOD and VA policy guidance, reports, and analysis plans to determine how the agencies are evaluating the pilot’s effectiveness and to obtain information on their results. We also reviewed the relevant requirements of the National Defense Authorization Act of 2008 as it pertains to this review. In addition, we interviewed officials responsible for the evaluation at DOD’s Office of the Deputy Under Secretary of Defense for Wounded Warrior Care & Transition Policy (WWCTP), DOD’s Defense Manpower Data Center, and two organizations that DOD has contracted with to perform the evaluation—Booz Allen Hamilton and Westat. We then tested the reliability of the data the agencies are using for their evaluation—data from surveys of servicemembers, IDES case data from the Veterans Tracking Application (VTA) system, and legacy case data that DOD’s WWCTP obtained from the military services. Finally, we conducted some analyses of IDES and legacy case data for the Army to compare the two systems on timeliness and appeal rates, using elements of the data that we found to be reliable, but these comparisons have limitations and are not generalizable to other military services. The sections below describe our data reliability work and our analysis of Army data in further detail.

Review of Satisfaction Survey Data Reliability

DOD and VA have been surveying servicemembers going through the IDES pilot, and a comparison group of veterans who went through the standard “legacy” disability evaluation system, to determine whether the IDES pilot has improved servicemember satisfaction. The agencies survey all servicemembers in the IDES pilot at three points in time—following their completion of the medical evaluation board (MEB) of the disability evaluation process, completion of the physical evaluation board (PEB), and during the transition phase. To create a comparison group, the agencies sampled veterans who have been through the legacy system at current pilot sites. Their sampling methods were designed to ensure that the pilot and legacy groups were of comparable size and had similar proportions of servicemembers found unfit for duty. DOD and VA are analyzing the differences between the pilot and legacy groups’ average responses on four “survey composites,” or general categories composed of several survey questions: overall experience, fairness, DOD board liaison officer customer service, and VA case manager customer service.

We reviewed the reliability of surveys DOD and VA are using to obtain information on satisfaction levels by examining their survey design and analysis. To do so, we interviewed officials at DOD’s Defense Manpower Data Center and Westat responsible for implementing the survey, as well as officials at WWCTP and Booz Allen Hamilton responsible for designing the survey and analyzing the survey data. We also reviewed the survey instruments, response rates, data analysis plans, analysis results, and survey data as of February 28, 2010. We found DOD’s survey methodology—and the data derived using that methodology—to be reliable for purposes of comparing servicemembers’ satisfaction levels in the IDES and legacy disability evaluation systems.

Pilot and Legacy Case Data Reliability Tests

DOD and VA are collecting data on IDES pilot cases through the VTA and are using these data to conduct ongoing monitoring of case processing times and appeal rates, with the results presented in weekly reports. VA manages VTA, but evaluation of the data is primarily conducted by staff at DOD’s WWCTP and Booz Allen Hamilton.

For their August 2010 interim report to Congress, DOD staff created a data set used to compare pilot and legacy processing times and appeal rates. This data set included IDES pilot cases as of February 28, 2010, with the earliest case started in November 2007. The data set also included data, as of January 31, 2010, on legacy cases started between fiscal years 2005 and 2009 at the first 21 sites operating the IDES pilot, prior to pilot implementation. The agencies also matched legacy case data from each of the military services with VA data, in order to capture additional processing time it took for servicemembers to navigate the VA disability claims process. Because the data set was created from February 2010 pilot data, it only included about one-third of the IDES pilot cases that were completed as of August 29, 2010. The February 2010 data set included cases from 17 of the 27 current pilot sites, and 7 of the 17 sites—including some of the pilot sites with the largest caseloads such as Fort Carson and Camp Lejeune—had fewer than 20 completed cases each when the data set was created.

To assess whether the data DOD and VA are using for their monitoring and evaluation are reliable, we obtained the early 2010 data set that the agencies planned to use for their evaluation report to Congress. We restricted our reliability assessments to the specific variables that the agencies used in their analyses. Following steps detailed below, we found that the IDES pilot case data were sufficiently reliable for our analyses, but that the legacy case data were incomplete with respect to data elements key to measuring case processing time and appeal rates.

To assess the reliability of the agencies’ IDES pilot data, we interviewed VA database managers responsible for VTA, reviewed VTA manuals and guidance, conducted electronic tests of the data and, for a small, random sample of cases, checked the data against case files.

Through our interviews and document reviews, we concluded that the agencies have sufficient internal controls to give reasonable assurance that the data are complete.

Our electronic testing of the data generally found low rates of missing data and errors in completed IDES cases. In these tests, we considered a data element to be sufficiently reliable for purposes of using in our report if 15 percent or less of the data were missing or had errors. Using this standard, we determined that one data element for IDES cases—the date that servicemembers separated from the military—was not reliable, because: (1) it was missing in 19 percent of completed cases and (2) in cases where the date was present, more than 30 percent appeared to have errors (for example, the date was before a step of the process that it should have followed).

We also conducted a trace-to-file process to determine whether date fields in the VTA system were an accurate reflection of the information in the IDES case files. Specifically, we compared 12 date fields in the VTA against a random sample of paper files for 54 completed cases: 24 from the three Army PEBs, 10 from the Air Force PEB, and 20 from the Navy PEB (10 Navy cases and 10 Marine Corps). In comparing these dates, we allowed for a 10 percent discrepancy in dates—i.e., a difference of 2 to 10 days, depending on the date and phase of the process—to allow for the possibility that dates may have been entered into the database after an event took place. The trace-to-file process resulted in an overall accuracy rate of 84 percent. For five data elements key to DOD and VA’s evaluation of the IDES pilot, we found that VTA dates reflected dates in the case files 85 percent of the time or better. For six key data elements—i.e., the end dates of the exam and MEB phases, the start of the PEB phase, the date a VA rating request was made, the date of the final disposition, and the date servicemembers received VA benefits—the VTA dates matched case file dates between 70 to 85 percent of the time. Although we considered these dates sufficiently reliable to include in this report, these dates should be interpreted with more caution. The separation date was accurate less than 70 percent of the time and did not meet our standards of reliability.

To assess the reliability of the legacy data that the agencies planned to compare the IDES pilot against, we tested the data electronically, and found that data for key dates and appeals indicators had significant gaps because the services did not collect the same information for legacy cases that were collected for pilot cases (see table 2). For example, only Army cases had information on when servicemembers were referred to the MEB process. In addition, the legacy data did not include the date on when servicemembers received VA benefits—which is necessary for measuring the full length of the legacy process. Without sufficient data on the beginning (when servicemembers were referred into the system) or end of the process (when they received VA benefits), we concluded that the full case processing time in the legacy system cannot be known. We also concluded that comparisons could not be made between the legacy and IDES pilot on appeal rates because only Army and Air Force cases had information on whether servicemembers appealed the informal PEB decisions.

GAO’s Review of the Agencies’ Comparison of Pilot and Legacy Data

In addition to reviewing the reliability of the IDES pilot and legacy data, we reviewed how DOD and VA are using the data for their comparisons of the two disability evaluation systems. Through interviews with officials at DOD’s WWCTP and Booz Allen Hamilton and documents they provided us, we understand that DOD planned to address gaps in the legacy data by: (1) approximating the referral dates in Air Force, Marine Corps, and Navy cases using Army data and (2) using dates when cases were ready to be rated by VA to approximate the end of the process. Specifically, to approximate referral dates, they said they would use the average time for Army cases between when the servicemember was referred and when the MEB documentation identifying the servicemember’s potentially unfitting medical conditions (i.e., the narrative summary) was completed, which they calculated to be 60 days. For Navy and Marine Corps cases, they then subtracted 60 days from the date of the narrative summary to estimate a referral date and, for Air Force cases, they did so from the date of the MEB decision. However, because only 11 percent of Army legacy cases had a narrative summary date, the estimate of 60 days is based on a small number of cases (see table 3). To address the lack of data on the date VA benefits were delivered, DOD planned to use the date that VA determined a case was ready to be rated to approximate the end of the process, though this would underestimate the length of time it took to deliver VA benefits in the legacy process.

GAO Analysis of IDES Case Data

For objective 1, we presented information on average processing time in the IDES, both overall and by military service, using information presented by DOD and VA in their weekly monitoring reports. Where information was not available in the weekly reports, we conducted our own analysis using the early 2010 data set that DOD and VA intended to use for their report to Congress. Specifically, we used these data to determine the proportion of pilot cases meeting the 295-day goal for active duty servicemembers and the 305-day goal for reserve servicemembers.

In addition, although limitations in the legacy data preclude reliable comparisons between the IDES pilot and legacy systems for all the military services, the Army legacy data on when servicemembers were referred into the IDES were sufficiently complete to make some limited comparisons. Specifically, we analyzed Army legacy data to determine how long the legacy process took, on average, between when servicemembers were referred to the process and when VA was ready to conduct the disability rating. We limited our analysis to cases in which a VA claim was filed between 2006 and 2009 because data on when VA was ready to conduct the rating was missing for a substantial number of cases where the VA claim was filed in 2005 and 2010. We compared this legacy average with the total pilot case processing time through to delivery of VA benefits, but we noted that the legacy average does not account for time for VA to complete the rating and deliver the benefits. We also analyzed Army data on appeals in order to illustrate the limitations of DOD’s plan to compare only appeals to the informal PEB in the pilot and legacy systems and not take into account appeals of rating decisions to VA. We conducted this analysis using the legacy data and pilot case data as of early 2010, since DOD and VA’s weekly reports do not contain information on appeals to VA.

Identifying Challenges in Implementing the IDES at Pilot Sites

To identify challenges in implementing the IDES during the pilot phase, we visited 10 of the 27 military treatment facilities participating in the pilot. At the site visits, we interviewed officials involved in implementing the IDES from both DOD and VA, including military facility commanders and administrators, DOD board liaisons, military physicians involved in MEB determinations, DOD legal staff, VA case workers, VA or contract examiners, and administrators at VA medical clinics and VA regional offices. We selected the 10 facilities to obtain perspectives from sites in different military services and geographical regions and that varied in terms of disability evaluation caseloads and how their single exams were conducted (by DOD, VA, or a VA contractor) (see table 4).

We also interviewed various offices at DOD and VA involved in implementing the IDES pilot. At DOD, this included WWCTP; Office of the Assistant Secretary of Defense for Health Affairs; Office of the Assistant Secretary of Defense for Reserve Affairs; Air Force Physical Disability Division; Army Physical Disability Agency; Navy Physical Evaluation Board; Office of the Air Force Surgeon General; Army Medical Command; and Navy Bureau of Medicine and Surgery. At VA, we interviewed officials in the Veterans Benefits Administration, Veterans Health Administration, and VA/DOD Collaboration Service.

Furthermore, we reviewed relevant documents, including DOD and VA policies and guidance and records of “hotwash” meetings, which DOD and VA held shortly after implementing the IDES at pilot sites to identify implementation successes and challenges. We also reviewed data on processing times for the single exams, MEB determinations, informal PEB decisions, and VA ratings, as reported in the agencies’ weekly monitoring reports. In addition, we reviewed relevant federal laws and regulations.

To determine whether the IDES process extended the time that servicemembers remained in military service, we analyzed the legacy and pilot case data from the early 2010 data set, but we identified several limitations with the data. As noted earlier, the date servicemembers separated from the military was missing for 19 percent of completed IDES pilot cases. Further, as shown in table 5, only Air Force cases contained data on the separation date in the legacy data. Also noted earlier, only the Army legacy data contained information on when servicemembers were referred into the legacy process. As a result, for Army cases, we compared the average length of time it took cases to reach a final PEB decision in the legacy and pilot, since this date was sufficiently complete in both the legacy and pilot data. The PEB decision is the last phase of the disability evaluation process before a servicemember either begins to transition from military service, or if they are found fit, returns to their unit.

Examining DOD and VA’s Plans for Expanding the IDES

To identify the agencies’ preparations for worldwide expansion of the IDES, we reviewed documents on DOD and VA’s expansion strategy, their site assessment matrix, and weekly monitoring reports which, beginning in July 2010, tracked key implementation time frames, both nationally and at individual military treatment facilities. Our interviews with officials involved in the pilot at DOD, VA, and each of the military services also provided us with information on the agencies’ expansion plans. We also reviewed relevant federal laws and regulations.

We determined the adequacy of the agencies’ planning efforts by assessing whether their plans addressed the challenges we had identified in objective 2. We also determined whether the plans incorporated internal controls described in GAO’s Standards for Internal Control in the Federal Government and best practices for program implementation identified in academic literature.

Appendix II: IDES Pilot Processing Times for Reserve Component Servicemembers

The figures below show case processing times in the IDES pilot for reserve component servicemembers. Figure 11 shows the average number of days it took to complete the process—i.e., to deliver VA benefits to reserve component servicemembers, as of August 2010. Figure 12 shows the percentage of cases that met the DOD and VA goal to deliver VA benefits within 305 days, as of February 2010. Figures 13-15 show the average length of time it took, as of August 2010, to complete phases of the IDES process—i.e., the single exam, the MEB documentation, and the informal PEB decision, respectively—each of which have taken longer, on average, than the goals established by DOD and VA.

Appendix III: Comments from the Department of Defense

Appendix IV: Comments from the Department of Veterans Affairs

Appendix V: GAO Contact and Staff Acknowledgments

Staff Acknowledgments

Michele Grgich (Assistant Director), Yunsian Tai (Analyst-in-Charge), Jeremy Conley, and Greg Whitney made significant contributions to this report. Walter Vance and Vanessa Taylor provided assistance with research methodology and data analysis. Bonnie Anderson, Rebecca Beale, Mark Bird, Brenda Farrell, Valerie Melvin, Patricia Owens, and Randall Williamson provided subject matter expertise. Susan Bernstein and Kathleen van Gelder provided writing assistance. James Bennett provided graphics assistance. Roger Thomas provided legal counsel.