Veterans Disability Benefits: Clearer Information for Veterans and Additional Performance Measures Could Improve Appeal Process
Published: 2011-09-29T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Veterans Affairs (VA) has struggled to provide timely reviews for veterans who appeal decisions on their disability compensation claims. A veteran appeals to the VA regional office that made the initial decision, and if still dissatisfied, to the Board of Veterans Appeals (Board). An appeal to the Board adds more than 2 years, on average, to the wait for a decision on the appeal. To resolve more appeals at the regional level and avoid waits at the Board, VA, in 2001, established the Decision Review Officer (DRO) review as an alternative to the traditional regional office appeal review. A DRO is given authority to grant additional benefits after reviewing an appeal based on a difference of opinion with the original decision. In contrast, under the traditional review, new evidence is generally required for a grant of additional benefits. GAO examined (1) the extent to which veterans choose a DRO review, (2) outcomes for DRO reviews, and (3) VA's challenges in managing DROs. GAO analyzed Board data, surveyed managers in all 57 regional offices, visited 4 offices, and interviewed veterans.
section_title: What GAO Found
paragraphs: According to VA data, which has only tracked DRO involvement since 2003, veterans chose a DRO review in 61 percent (534,439) of all appeals filed from 2003 to 2010. Veterans who sought assistance with their appeal from a veteran service organization or other qualified representatives were more likely to choose a DRO review than those without a representative. Without assistance, veterans may not fully understand their two appeal options. GAO found that the letter VA uses to inform veterans of their options does not highlight key deadlines or differences between the two options. According to more than half of surveyed regional office managers, most veterans could not make an informed choice on the options based just on the letter. The DRO review process has helped some veterans get additional benefits at the regional office level, but has not reduced the percentage of appeals continuing on to the Board--the primary purpose of the program. In fiscal years 2003 through 2008, 21 percent of DRO reviews resulted in a full grant of benefits compared to 17 percent of traditional reviews. A full grant of benefits ends, or resolves, the appeal at the regional level. However, appeals may also be resolved at the regional level if veterans who do not receive full grants decide not to continue their appeal to the Board. VA gave DROs the flexibility to interact informally with veterans in part so they could explain when the benefits already granted are appropriate given the law. However, while DRO reviews led to the grant of full benefits at a higher rate, a higher percentage of veterans not granted benefits through traditional review voluntarily ended their appeals. As a result, in fiscal years 2003 through 2008 the overall percentage of appeals resolved at the regional level was about the same for DRO and traditional reviews--about 70 percent for both. VA faces challenges in how to most effectively use and train DROs. Since the DRO process and position were established, DRO duties have expanded beyond reviewing appeals to performing additional tasks such as quality review. However, VA officials have not reached consensus on how to balance DROs' time among different tasks. VA has no performance goal or measure for appeal resolution at the regional level that could help it determine whether it is achieving the most effective balance between different tasks. In addition, VA headquarters offers no nationwide, standardized training for new DROs, which according to managers and DROs would be beneficial, as they often lack experience with other tasks that DROs frequently perform such as conducting hearings. Ninetythree percent of surveyed regional managers said a nationally standardized training for new DROs would be beneficial.
section_title: What GAO Recommends
paragraphs: GAO recommends VA (1) revise its appeals election letter, (2) develop an appeal resolution goal at the regional level, and (3) develop a training curriculum on DRO duties. In its comments, VA concurred fully with GAO's first and third recommendations but only partially with the second. VA expressed concerns about an appeal resolution goal, including that it could encourage the unjustified granting of benefits. GAO feels that VA's quality control process minimizes this risk.

section_title: Letter
paragraphs: The Department of Veterans Affairs (VA) paid more than an estimated  $36 billion in disability compensation in fiscal year 2010 to over three  million veterans who incurred new or aggravated existing disabilities  during their military service. Veterans apply for disability compensation  benefits through one of VA’s 57 regional offices. Those who disagree with  VA’s initial decision on their claim may appeal to the regional office that  made the decision. If still dissatisfied with the regional office’s decision on  their appeal, they can continue their appeal to the Board of Veterans  Appeals (Board) in Washington, D.C. In fiscal year 2010, VA received  more than 140,000 appeals—a 45 percent increase over fiscal year  2007—and the Board has struggled with an increasing backlog of cases.  On average, veterans wait 8 months after filing an appeal to receive a  decision from the regional office, and those who continue to the Board  wait over two more years for a decision.
In an effort to resolve more appeals at the regional level, thus shortening  total wait times for veterans, and improve the quality of appeals which go  on to the Board, in 2001, VA established the Decision Review Officer  (DRO) review—an alternative review process at the regional level. Prior  to that time, regional offices reviewed initial claim decisions and generally  could only overturn them when they received new evidence; otherwise  appeals had to go to the Board for consideration of a grant of additional  benefits. Under the alternative process, DROs (who are senior staff) have  the authority to overturn an initial disability claim decision without any new  evidence based only on difference of opinion. However, veterans still  have the option of choosing traditional review, which is VA’s term for the  process that existed prior to 2001. As of July 2010, there were over 400  DROs nationwide. At your request, this report examines aspects of the  DRO program. We evaluated (1) the extent to which veterans choose a  DRO review as opposed to a traditional review, (2) outcomes for veterans  who choose a DRO review, and (3) challenges VA faces in managing  DROs.
To address these items, we reviewed relevant federal laws, VA  regulations, policy manuals, and regional office materials, as well as  previous VA studies, GAO reports, and other documents relevant to  DROs and the appeal process. We analyzed appeals management data  from the Veterans Appeals Control and Locator System (VACOLS), an  electronic Board database, for fiscal years 2003 through 2010. Although  the DRO process was established in 2001, VACOLS began tracking DRO  involvement in appeals in fiscal year 2003. VA extracted the VACOLS  data for us on April 7, 2011. We assessed the data and determined they  were reliable for the purposes of this study. We administered an online  survey to all 57 regional office managers, and conducted phone  interviews with a randomly selected but nongeneralizeable sample of 40  veterans who had recently appealed their disability claim decision through  either a DRO or a traditional review. We conducted site visits to four of  VA’s regional offices—Atlanta, Georgia; Providence, Rhode Island; Waco,  Texas; and Salt Lake City, Utah—and interviewed VA staff and veterans  service organizations (VSO) representatives who assist veterans with  their claims and appeals. We selected offices based on several factors,  including geographic location, number of staff, timeliness of appeal  processing for appeals in which DRO was selected, and participation in a  pilot study of new criteria for assessing DRO performance.
We conducted this performance audit from July 2010 through September  2011 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives. Appendix I contains a  more detailed description of our scope and methodology.
subsections: section_title: Background
subsections: section_title: Disability Claims Process
paragraphs: To apply for disability compensation, a veteran submits a claim to VA’s  Veterans Benefits Administration (VBA). A Veterans Service  Representative at one of VA’s regional offices reviews the claim and  assists the veteran in gathering required evidence, including military  service records and medical treatment records from VA facilities and  private providers. A Veteran Service Representative’s responsibilities  may include establishing claims files, generating notification letters to  veterans, assisting veterans in obtaining the evidence needed to support  their claims, and assisting in processing appeals of claim decisions. A  Rating Veterans Service Representative (RVSR) then evaluates the  evidence and, if the RVSR finds the veteran to be eligible, determines the  percentage rating for purposes of compensation. Disability compensation  varies with the degree of disability and the number of a veteran’s  dependents, and is paid monthly. Monthly base benefits in 2011 for an  individual range from $123 for 10 percent disability to $2,673 for 100  percent disability. A veteran may cite multiple medical issues in a claim,  for example, post-traumatic stress disorder, knee impairment, and  hearing loss. The RVSR may grant all, some, or none of the issues in a  claim. The veteran can obtain help by submitting a claim and navigating  the process with a VSO representative, private attorney, or agent  accredited by VA to assist veterans. In fiscal year 2010, VA received  over one million disability compensation claims, a 46 percent increase  from fiscal year 2007.
section_title: Appeal Process
paragraphs: Regional office review of notice of disagreement (initial appeal  stage). If a veteran disagrees with the regional office’s initial decision on  a claim, he or she appeals by submitting a written notice of disagreement  to the regional office. (See fig. 1 for a flowchart of appeals process at the  VA.) A veteran may choose a traditional review or a DRO review by  responding to an appeal process request letter sent by VA. VA sends  this letter to a veteran after receiving his or her notice of disagreement;  the letter informs the veteran of the DRO review and traditional review  options, requests the veteran choose one, and states how to obtain  representation if the veteran does not already have it. If the veteran  chooses the traditional review, the reviewer, who may be a RVSR or  DRO, examines the claim file and any new evidence that the veteran  submits and may hold a formal, transcribed hearing with the veteran. The  reviewer may overturn the original decision based only on (1) new  evidence or (2) a clear and unmistakable error made in the original  decision. However, if a veteran chooses the DRO review, a DRO  conducts a de novo review of the claim, meaning a new and complete  review without deference to the original decision, and can revise that  decision without new evidence or a clear and unmistakable error—in  other words, based on a difference of opinion. In addition to formal  hearings, DROs may hold informal conferences with the veteran or the  veteran’s representative to discuss an appeal, including why benefits  already awarded are appropriate. Ultimately, in either process, the  reviewer may: (1) award a full grant, in which all claimed benefits are  awarded at the maximum level; (2) award a partial grant, in which some  benefits are granted but not necessarily at the maximum level or for all  claimed medical issues; or (3) confirm the original decision on the claim,  in which no further benefits are granted on any issues. If a full grant is  awarded, the appeal ends. Otherwise, the regional office issues a written  explanation to the veteran in a statement of the case.
Regional office preparation of appeal for the Board. Regardless of  whether a DRO or a traditional review is chosen, if a veteran disagrees  with the regional office’s appeal decision, he or she may file what VA calls  a substantive appeal to the Board, which is processed initially by the  regional office. The substantive appeal is a document that the veteran  completes to explain the issues being appealed and why the veteran  believes the VA decided his or her case incorrectly. After receiving this  document and reviewing any additional supporting evidence provided by  the veteran, the regional office may award additional benefits. A veteran  may submit new evidence multiple times and each submission requires  the regional office to re-evaluate the claim. When the regional office  determines no further work is necessary, it certifies the appeal as  complete and transfers it to the Board.
Board review of appeal. When the Board receives the file, it may grant  or deny the claim. If the Board finds it cannot make a decision until the  regional office does additional work (e.g., requesting a more recent  medical exam) it sends, or remands, the case back to the regional office  or to the VBA Appeals Management Center in Washington, D.C., which  develops evidence and adjudicates the claim. A veteran dissatisfied with  the Board’s decision can appeal, in succession, to the U.S. Court of  Appeals for Veterans Claims, to the Court of Appeals for the Federal  Circuit, and finally to the Supreme Court of the United States.
VA created the DRO position and implemented the DRO review process  nationally in 2001 after a 1997–1998 pilot in 12 regional offices. In the  pilot, DROs were responsible for performing reviews of all appeals and  had the ability to make a new decision based solely on a difference of  opinion with the original decision. VA promulgated final regulations when  it implemented DRO review nationwide in 2001, which made the process  optional and required the veteran to expressly choose it. Later in 2001,  VA headquarters issued guidance to the regional offices expanding the  responsibilities of DROs to include mentoring and training other disability  claims staff and working with regional office managers to identify error  trends and training needs.
section_title: Veterans More Often Choose the DRO Review, but Some May Not Fully Understand Their Options
paragraphs: Since fiscal year 2003, when VA started tracking DRO involvement in  appeals, data show more veterans have chosen a DRO review than a  traditional review for appeals of decisions on their disability compensation  claims. From fiscal years 2003 through 2010, veterans chose a DRO  review in 534,439 appeals, or 61 percent of all appeals filed over this time  period, according to Board data. The percentage of appeals in which a  DRO review was chosen increased each year, from 54 percent of all  appeals in fiscal year 2003 to 65 percent in fiscal year 2010 (see fig. 2).  However, there was significant variation across VA’s regional offices. For  example, during the 8-year time period, the regional office in Columbia,  South Carolina, had the lowest percentage of DRO reviews chosen—32  percent—and San Diego, California, the highest—87 percent. More than  half of the regional offices had the DRO review chosen in at least 50  percent of all appeals filed.
Our review suggests a key factor in the choice of a DRO review is  whether a veteran has assistance from a third party, such as a VSO  representative or a private attorney. Sixty-three percent of all appeals  filed by veterans with representation chose a DRO review compared to 44  percent of those filed by veterans without representation across all  regional offices. Controlling for selected factors that may affect the choice  of a DRO review, veterans with representation are still more likely to  choose a DRO review for their appeals. Additionally, we found that  representation may also be one of the factors that contribute to the  variation in the percentage of veterans choosing a DRO review across  regional offices. As the percentage of appeals with representation  increased, the percentage of DRO reviews also increased across regional  offices. However, representation alone does not fully explain the variation  in DRO reviews chosen across regional offices. Other factors that we did  not analyze, such as demographic factors, may contribute to the variation  across regional offices.
Representatives may recommend one appeal option over another to their  veterans. Of the 40 veterans we interviewed, 22 told us that  representatives from a VSO or state or county agency helped them  decide between the DRO and the traditional review, and 17 of the 22 said  that the assistance they received was very or somewhat helpful in  deciding between the appeal options. Most VSO representatives we  interviewed said they usually recommend a DRO review because they  believe it is faster or because a DRO may be more likely to grant the  claim. Some also said they recommend a DRO review over a traditional  review because a DRO will thoroughly review all the evidence and a  veteran has the opportunity to make his or her case to the regional office.  VSO representatives at one regional office we visited said that they  recommend a DRO review if they believe, based on experience, that the  DRO likely to be assigned the case is more open to working with them.
However, some VSO representatives also said that they recommend a  traditional review under certain circumstances. For example, if an appeal  has ambiguous evidence, they may recommend a traditional review  because it may result in an appeal going to the Board faster and because  they believe the Board has more flexibility than a DRO to grant benefits in  such cases. Also, VSO representatives at one site visit told us they  generally recommend a traditional review because in their view a DRO  review takes too much time.
Of all appeals filed since fiscal year 2003, 53 percent included the fact  that they chose the DRO review option on the notice of disagreement.  According to VSO representatives we interviewed, because veterans’  representatives have power of attorney, they can choose an appeal  option at the same time they file an appeal for a veteran rather than  waiting for the veteran to receive VA’s appeal process request letter. A  DRO review may be chosen on the same day an appeal is filed in order to  save time because the regional office does not have to send the appeal  process request letter and wait for the veteran’s response before  beginning its review.
Veterans filing 13 percent of appeals chose an appeal option without  assistance from a representative, and these veterans may not fully  understand their options as described in the VA appeal process request  letter, some VA officials and VSO representatives told us. The VBA  manual provides a template of the letter for the regional offices, which  prepare and mail it to a veteran after receiving the notice of  disagreement. VA headquarters officials told us that while regional  offices have flexibility to add information to the template letter, they  cannot exclude any information, and headquarters prefers that the  regional offices use the template letter. (For VA’s template letter, see app.  II.) Some VA officials and VSO representatives said veterans may not be  able to make an informed decision based on the letter alone. Specifically,  29 of 55 regional office managers we surveyed (53 percent) answered  that they believed no or only some veterans could make an informed  decision using only the VA letter. The other regional office managers we  surveyed (47 percent) said that all or most veterans could make an  informed decision using only the VA letter (see fig. 3). Of the DROs we  spoke with, 10 of 17 said that in their view, no or only some veterans  could make informed decisions based on the letter alone, and most VSO  representatives at the regional offices we visited said that few or no  veterans could do so. Representatives from one VSO told us veterans  may phone after receiving the letter because they do not understand its  description of appeal options. Some managers in our survey and VA staff  we interviewed suggested changes to the template, for example,  shortening its length, using simpler language and less jargon, using  graphics or tables to aid understanding, explaining the benefits of  representation, and advising veterans they can request informal or formal  hearings if they choose a DRO review. They also said the letter could  better explain differences between DRO and traditional reviews,  particularly in the treatment of evidence and the authority given a DRO to  render a new decision without additional evidence.
Our analysis of the template in the VBA manual, based on criteria  established for federal government communication with the public, found  the letter did not clearly explain appeal options and the appeal process.  Although we found the letter met several criteria for clear communication  by federal agencies and programs, such as simple structure, concise  headings, and avoidance of complex language overall, we found it did not  meet some other criteria, including:    defining terms that could be unfamiliar to the veteran, such as  “decision review officer,” “de novo reviews,” “clear and unmistakable  error,” and “substantive appeal”;   highlighting key deadlines for responding by including them at the  beginning of the letter, so the veteran is clear that VA needs a  response within 60 days; and   using tables or graphics to explain appeal options and enhance  recipient’s understanding (see fig. 4).
One of the regional offices we visited made changes to the template that  address some of these issues, for example, including instructions on how  to request a hearing and italicizing important information, such as the 60- day deadline for a veteran to respond with an appeal choice.
Of the 40 veterans we interviewed, 14 remembered receiving and were  able to describe the appeal process request letter, and 9 of those told us  that the letter explained the appeal options only somewhat clearly or not  at all clearly. Five said it explained the options very clearly. Some of the  veterans we spoke with offered suggestions on how VA could improve the  letter, for example, by providing a point of contact for veterans with  questions, explaining when they might next hear from VA about their  appeal, and more thoroughly describing the two appeal options. VA  headquarters officials said VBA management and attorneys reviewed the  letter template but they did not test it with focus groups of veterans  because VBA does not have contracts to conduct focus groups.  According to federal guidelines for clear communication with the public,  testing documents should be an integral part of an agency’s plain- language planning and writing process.
section_title: The DRO Review More Often Results in Additional Benefits, but Does Not Reduce Appeals Going to the Board
subsections: section_title: DRO Reviews More Often Resulted in Additional Benefits than Traditional Reviews
paragraphs: Veterans who chose a DRO review were more likely than those who  chose a traditional review to get additional disability compensation  benefits after initial review of their notices of disagreement. From fiscal  years 2003 through 2010 at least some additional benefits were awarded  at the regional level in 32 percent of DRO reviews compared to 23  percent of traditional regional office reviews, according to our analysis of  Board data (see fig.5). We found that having a DRO review was  associated with a greater chance of being awarded some additional  benefits even after controlling for some other factors that may affect  appeal outcomes, such as whether a veteran had a representative or the  regional office in which the appeal was filed. Veterans awarded  additional benefits at this appeal stage receive these benefits immediately  even if they continue their appeals, according to a VA official.
DRO reviews took about a month longer than traditional reviews to  complete, or 266 days compared to 235 days, on average, for the time  period from fiscal years 2003 through 2009 (see fig. 6). It is possible  DRO reviews take longer because they involve a more thorough  examination of the evidence. For example, one DRO we spoke with said  such reviews take longer because they require a completely fresh look at  all evidence in the original claim. Regardless of whether veterans choose  a DRO or a traditional review, their waiting time for a decision on the  notice of disagreement is considerably less than if they continue their  appeal to the Board. For appeals filed from fiscal years 2003 through  2007, the time from filing of the notice of disagreement to Board decision  was more than 1,000 days on average.
Although veterans choosing a DRO review were more likely to gain some  additional benefits at the regional office level, in comparison to a  traditional review, DRO review does not appear to reduce the number of  appeals continuing to the Board—which was an important goal for VA in  introducing this appeal option. The resolution rate at the regional office  level for appeals in which a DRO review was selected has been about the  same as the rate for traditional reviews. Of the 593,526 appeals of  disability claim decisions receiving either DRO or traditional reviews from  fiscal years 2003 through 2008 that had a final regional office outcome,  72 percent ended at the regional office level. The remainder continued to  the Board. For appeals in which a DRO review was selected, 71 percent  ended at the regional office level compared to 73 percent of appeals in  which a traditional review was chosen.The resolution rate for DRO and  traditional reviews was roughly the same in each year during this 6-year  period (see fig. 7). Resolution rates varied across VA’s regional offices.  For both DRO and traditional reviews, resolution rates ranged from less  than 50 percent to more than 80 percent in individual offices, with the  majority having rates of more than 70 percent for both types of reviews.  VA officials offered several possible explanations for the variation across  offices, for example, the extent to which VSO representatives are  proactive in submitting necessary evidence.
The negligible difference between appeal resolution rates at the regional  office level for DRO and traditional reviews in part reflects veterans’  decisions on how to proceed if they do not receive a full grant of  requested benefits. Although a slightly higher percentage of DRO reviews  resulted in full grants of benefits for veterans, veterans choosing DRO  review were less likely than those choosing traditional review to end their  appeals when not granted full benefits. Board data show that for appeals  filed from fiscal years 2003 through 2008 the most common reason for  resolution at the regional office level was a veteran’s failure to return a  form to continue the appeal (failure to respond) after receiving VA’s  explanation of its decision and of what steps to take to move the appeal  to the Board (see fig. 8). On this front, traditional reviews were  somewhat more likely than DRO reviews to be resolved through veterans’  failure to respond—41 percent versus 36 percent, respectively. The  second most common reason for appeal resolution at the regional level  was a full grant of benefits, which automatically ends the process. In  contrast to the first issue, DRO reviews were somewhat more likely than  traditional reviews to be resolved through a full grant of benefits—21  percent versus 17 percent, respectively. Most decisions to grant full  benefits were made at the initial appeal stage, while some were made  during the stage in which the regional office prepares the appeal for the  Board. The remaining appeal resolutions were due to veterans or their  representatives withdrawing their appeals by contacting VA or to the  death of the veteran.
While we found no difference in how veterans responded to a partial grant  of benefits based on their choice of appeal option, we found that veterans  who chose a traditional review appeared to respond differently than those  who chose a DRO review when no additional benefits were granted at the  initial appeal stage. Board data for fiscal years 2003 through 2008 show  that when no additional benefits were granted (i.e., the original decision  was upheld), appeals in which a traditional review was chosen were  somewhat more likely than those in which a DRO review was chosen to  end at the regional office level—57 percent versus 51 percent,  respectively. By contrast, in cases that resulted in the award of partial  benefits, there was no difference based on which appeal option had been  chosen—75 percent of both traditional appeals and DRO appeals were  ended at the regional level. Representation may be a factor in why  veterans selecting a traditional review are more likely to end their appeals  after receiving no additional benefits from the regional office. Fifty-nine  percent of appeals which received no additional benefits at the initial  appeal stage and in which the veteran had no representative were ended  by the veteran without going on to the Board, compared to 52 percent of  such appeals in which the veteran had a representative. As noted  previously, veterans without representation are more likely to choose a  traditional review.
VA regional office staff told us that certain aspects of the DRO review can  be effective in resolving appeals before they continue to the Board.  Managers we surveyed rated DROs’ authority to make a completely new  (de novo) decision on a claim and to hold informal conferences with  veterans and representatives as the most effective tools for resolving  appeals (see fig. 9). Of the 17 DROs we interviewed, 15 rated their  authority to make a completely new decision as very or moderately  effective in resolving appeals, 10 rated informal conferences as very or  moderately effective, and 7 rated formal hearings as very or moderately  effective. Many DROs in our site visits told us their authority to make a  completely new decision on a claim allows them to reverse original  decisions without submission of additional evidence by the veteran.  DROs also said informal conferences may help resolve appeals because  they enable DROs to obtain information from a veteran’s representative  or explain to a representative and veteran why no additional benefits can  be granted.
While many regional staff believe DRO reviews have a greater potential  to resolve appeals, we found that the appeal resolution rate for DRO and  traditional reviews are quite similar, perhaps in part because VA’s criteria  for assessing and rewarding individual DROs’ performance do not always  encourage them to use their unique authorities to resolve appeals. VA  currently assesses the performance of individual DROs using four criteria:  quality of work, productivity, customer service, and timeliness. The  existing criteria do not specifically encourage appeal resolution at the  regional level, according to 60 percent of surveyed managers and 10 of  the 17 DROs we interviewed. For example, managers in one office we  visited said the current criteria encourage DROs to complete tasks that  help them meet their productivity requirement but may not necessarily  lead to appeal resolution.
The department is currently revising its criteria for assessing DRO  performance based on a 90-day pilot program in eight regional offices  conducted in fiscal year 2010, and it expects to implement new criteria  nationwide in fiscal year 2012. According to VA officials, the revisions are  intended in part to focus DROs’ attention more on resolving appeals at  the earliest possible stage of the process. VA officials told us the new  criteria under development will better encourage appeal resolution by  restructuring the way work credits—needed to meet the productivity  requirement—are awarded to DROs. For example, under the existing  criteria, work credits are awarded for holding informal conferences, but  under the new criteria additional credits would be awarded when the  conference results in resolving the appeal. In addition, VA officials said  the new criteria aim to promote appeal resolution by more explicitly  promoting regular communication with veterans’ representatives. They  said that even though such communication can help resolve appeals by  helping representatives understand, for example, why further benefits  cannot be granted in a particular case, DROs have become less focused  on communication over the years due to production pressures.
section_title: VA Faces Challenges in Defining Role of and Training for DROs
subsections: section_title: Lack of Performance Measures for Appeal Resolution Hinders VA’s Ability to Assess Use of DROs
paragraphs: Since the DRO pilot program in 1997–1998, VA has expanded DRO  duties to include tasks not related to appeals. According to the estimates  of regional office managers who responded to our survey, DROs overall  tend to spend the majority of their time on appeal-related tasks, but also  spend a significant amount of time on training other staff and performing  quality reviews (see fig.10). Time spent on different tasks varies across  regional offices. For example, the proportion of time spent on conducting  de novo reviews of appeals—that is, a completely new evaluation of a  claim without deference to the original decision—ranged from 3 to 70  percent in individual offices, and on formal training ranged from 0 to 40  percent. The four offices we visited typically assigned individual DROs to  specific responsibilities, such as reviewing appeals, training other staff,  and conducting quality reviews of other staff’s work, rather than having  each DRO perform the full range of tasks. Managers in one office said  specialization is helpful because it permits DROs to focus on training  without the distraction of appeals work. Regional office managers may  also assign DROs to different duties at different times to meet changing  office needs. For example, one office we visited had recently assigned  more DROs to training and quality review to address problems with its  RVSRs’ performance, and managers said preliminary results showed an  improvement in RVSR quality scores after this reallocation of DRO  resources.
Since the implementation of the DRO review process in 2001, there have  been differing opinions within VA about the proper balance between  processing appeals and performing other DRO tasks. In 2006, an internal  VA study group that assessed the impact of the DRO review process  recommended that VA limit the scope of DRO duties to reviewing appeals  and have them perform de novo reviews of all appeals—not just those in  which a DRO review was selected. Their report found that there was a  sufficient number of DROs to perform de novo reviews of all appeals if  DROs’ duties were limited to appeals-related tasks. Officials told us that  VA management decided not to implement these recommendations  because of concerns that there was an insufficient number of DROs to  review all appeals and a belief that other tasks are also important DRO  functions. There was also a mix of opinions on this topic among regional  office managers we surveyed and interviewed. Almost two-thirds of those  surveyed stated that eliminating the DRO review election process and  having DROs perform only de novo reviews of all appeals would  somewhat or greatly improve their effectiveness. On the other hand,  managers said it could be difficult to have their DROs spend more time on  such reviews. About 60 percent of surveyed managers said they would  prefer to allocate more DRO time to performing de novo reviews, but  some respondents noted that they cannot allocate DROs’ time optimally  because of staff and resource limitations and the need to assign DROs to  other duties.
While VA has expanded the role of the DROs so they are contributing to  other VA goals beyond appeal resolution, the department has not  developed performance measures to assess whether DROs are  successful in meeting their original purpose of reducing the number of  appeals to the Board. The measures that VA uses to assess national and  regional office performance in processing appeals do not include the  proportion of appeals resolved at regional offices. Existing VA measures  for national and regional office appeals performance involve the number  of appeals awaiting a decision, number of days appeals have been  pending a decision, and percentage of appeals that are remanded from  the Board. We have previously noted that agencies should develop  performance measures that are linked to agency goals. Linking  performance measures to broader agency goals at each operating level  reinforces the importance of these goals and may help managers identify  areas where problems exist and corrective action is required.
VA already collects some data related to appeal resolution, but some VA  officials question the value of a performance measure related to appeal  resolution. The department tracks data on the number of appeals  resolved at different stages of the appeal process—for example, through  the veteran’s failure to respond to the statement of the case or after the  veteran’s submission of a substantive appeal to the Board—nationally  and by regional office. It makes these data available to regional offices.  About half the regional office managers surveyed said an appeal  resolution goal at the regional office level would somewhat or greatly  enhance DROs’ effectiveness. However, VA officials told us they have not  established a performance measure for appeal resolution because this  outcome is not entirely under the control of the DRO, as a veteran has a  right to continue his or her appeal to the Board. Some VA managers and  staff in the offices we visited also expressed a concern that such a goal  could have a negative impact. For example, it could push DROs to grant  benefits that are not completely justified or to pressure veterans to  withdraw their appeals. While we understand this potential concern, we  note that VA has quality review procedures in place to ensure the quality  and accuracy of DRO decisions. Specifically, VA reviews on average five  cases per month that each DRO has worked on. VA assesses each case  against several criteria, including whether all claimed issues were  addressed, whether all applicable evidence was discussed, and whether  all issues were correctly granted or denied. Some regional office  managers agreed that setting an appeal resolution goal would not  encourage DROs to grant unjustified benefits, because, for example,  existing quality review procedures would prevent this from happening.
section_title: VA Provides No Nationwide Specialized Training for New DROs
paragraphs: VA has not developed a nationwide training curriculum for DROs to help  them learn the duties of the position. The training available to DROs from  VA headquarters is the same training offered to RVSRs (staff who  evaluate initial claims). VA requires its claims processing staff, including  DROs, to take 85 hours of training annually. Headquarters and regional  office officials told us DROs typically take the same courses as RVSRs.  As part of its training for claims processors, VA has three courses for all  regional office staff who process appeals: an orientation to the appeals  team, a course on the appeal process, and a course on how to use the  Board’s appeals tracking system. VA headquarters officials told us they  have not developed training specifically for new DROs because they  should already be technical experts in evaluating claims when promoted.  However, they acknowledged that the department has not completed an  analysis of DRO tasks specifically to determine the training needs of this  position. According to generally accepted criteria, a key step in  developing successful training in the federal government is for an agency  to understand the skills that its workforce needs to achieve agency goals.  Furthermore, these criteria stress the need for agencies to incorporate  continuous, life-long learning into their training—even experienced staff  may still need additional training in certain areas.
A number of managers and DROs in the regional offices we visited said  the skills DROs learn before promotion may not be sufficient to perform  their new tasks. Several managers told us new DROs, based on their  prior experience, are indeed technical experts in evaluating claims;  however, managers and DROs said new DROs may lack experience with  certain responsibilities, such as holding formal hearings and informal  conferences, training other staff, processing appeals, and balancing  multiple priorities. Regional managers at three of the four offices we  visited said the appeals courses offered by VA are not sufficient to teach  new DROs how to perform their duties; a manager in one office said  these courses provide a general overview and are not primarily used to  train them. DROs in the four offices we visited learn their duties primarily  on the job and by observing more experienced colleagues, according to  managers. However, some offices have developed more formal training  for DROs. Forty percent of surveyed regional managers said their office  has developed specific training that address such topics as the appeals  process, conducting formal hearings and informal conferences, reviewing  other staff’s work, and policy and regulatory changes.
Despite regional office efforts to provide targeted training, our survey of  regional managers found that 93 percent believe a nationally  standardized training program for newly promoted DROs would improve  their effectiveness, and 16 of the 17 DROs we interviewed also said a  national training program would be helpful. Managers in two regional  offices we visited said a training program developed by VA’s central office  would ensure consistency in how DROs are trained across the nation.  Managers responding to our survey indicated that tailored training on a  number of topics would enhance the ability of new DROs to perform their  job duties (see fig. 11).
A formal, centrally designed training program—especially for new  DROs—has been proposed within VA. Under this proposal, according to  one VA official involved with the management of the DROs nationally,  new DROs would receive instruction in areas such as training other staff  and communicating with veterans and their representatives. The training  might be administered in a central location for new DROs from multiple  regional offices, or developed centrally and then administered by regional  offices. However, VA does not yet have formal plans to move ahead with  the development of such a program, according to VA officials.
section_title: Conclusions
paragraphs: Review of veterans’ appeals by VA’s DROs has had some positive  impacts for veterans, and it is possible that more veterans would choose  a DRO review if they fully understood the distinction between the DRO  and traditional reviews. Because VA’s outreach letter lacks information  and clarity that veterans need to make an informed choice between the  options, some—in particular, those without a representative—may not be  taking advantage of the DRO review process. However, looking more  broadly, VA has not achieved its original goal for the DRO review process  of reducing the number of appeals continuing to the Board and thereby  shortening the time that veterans wait for appeal decisions. Indeed, while  VA is taking steps to place a greater focus on appeal resolution in the  criteria it uses to assess the performance of individual DROs, it has yet to  establish any national or regional office performance measures related to  resolving appeals at the regional level. Certainly, VA must balance  competing demands on DROs, senior claims processors in its regional  offices, to both resolve appeals and train and supervise less experienced  staff, but without a more strategic approach—which includes goals and  measures for resolution of appeals—VA does not know it is fully  leveraging DRO reviews and, ultimately, their effectiveness for the  veterans VA serves. Regardless of the proper balance between their  different responsibilities, DROs may not be contributing as much as they  could be, either to the goal of appeal resolution or to the goal of  developing newer claims processing staff, because they do not always  receive training specific to their duties.
section_title: Recommendations for Executive Action
paragraphs: To clarify information for veterans and ensure the most effective use of  DROs, the Secretary of Veterans Affairs should direct the Veterans  Benefits Administration to take the following three actions:  revise the sample appeals election letter in its policy manual to define  unfamiliar terms and emphasize key deadlines, and test any revised  letter’s clarity with veterans before implementing it;   establish national and regional office performance measures related  to appeal resolution at the regional level and ensure that sufficient  quality review procedures are in place to prevent DROs from granting  unjustified benefits; and   assess the knowledge and skills that DROs need to perform their  varied responsibilities, determine if any gaps exist in the training  currently available, and, if necessary, develop a training curriculum or  program tailored to DROs.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to VA for review and comment.  In its  written comments (see app. III), VA concurred fully with two of our  recommendations and partially with our recommendation regarding an  appeal resolution performance measure. The department concurred with  our recommendation to revise its sample appeal election letter. It reported  that it has formed a workgroup to assess and recommend changes to the  appeal election process, potentially including changes to the letter. VA  also concurred with our recommendation to assess the knowledge and  skills that DROs need and if necessary develop a training curriculum for  them. VA said it plans to develop additional training based on the results  of skills certification tests administered to DROs and on a job task  analysis conducted to aid the design of these certification tests.
VA partially concurred with our recommendation to establish national and  regional performance measures related to the resolution of appeals at the  regional office level. The department noted that—as explained previously  in our report—it is revising the criteria used to assess the performance of  individual DROs to place greater emphasis on resolving appeals at the  earliest stage possible, for example through communication with veterans  and their representatives. However, it also stated that resolution of  appeals at the regional level should not be a performance measure used  to assess the impact of the DROs, for several reasons including (1) the  DROs also play an important role in training, mentoring, and quality  review, so an appeal resolution measure does not capture their full  impact; (2) each veteran has a right to continue an appeal to the Board if  dissatisfied with the regional office decision, and whether a veteran  continues is beyond a DRO’s control; and (3) establishing an appeal  resolution performance measure could encourage the granting of benefits  that are not justified. We acknowledge that DROs impact VA operations  beyond appeals. Yet VA’s own documents—including its 2006 review of  the DRO program—state that the department’s primary goal in  establishing the DRO review process was to increase the percentage of  appeals resolved before continuing to the Board, and the changes VA is  now making to its criteria for measuring individual DRO performance  indicate that appeal resolution remains a major goal.  We are not  recommending that an appeal resolution performance measure replace  existing VA performance measures related to appeals or to other VA  processes on which DROs could have an impact. Our recommendation is  that VA add such a measure to gauge whether the program is meeting its  specific intended goal and help the department assess whether further  adjustments are needed.  We also acknowledge that it is not possible to  prevent all appeals from continuing to the Board; such an objective may  not even be desirable. However, based on its changes to the criteria for  individual DRO performance, VA clearly believes that changes in  incentives can encourage behaviors that are likely to resolve more  appeals early on. Finally, as VA also noted in its comments, individual  DROs must meet rigorous accuracy standards in their appeals work,  which should mitigate against the granting of unjustified benefits.
However, if VA does not believe that its existing quality control measures  are sufficient to guard against the granting of unjustified benefits, we  would encourage the department to consider what additional quality  control measures it could implement to ensure that the addition of a new  performance measure would not have the unintended consequence of  increasing the award of unjustified benefits.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until one day from the  report date.  At that time, we will send copies of this report to the relevant  congressional committees, the Secretary of Veterans Affairs, and other  interested parties. This report is also available at no charge on GAO’s  website at http://www.gao.gov.
If you or your staff have any questions regarding this report, please  contact me at (202) 512-7215 or bertonid@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found  on the last page of this report. Staff members who made key contributions  to this report are listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: We were asked to examine (1) the extent to which veterans choose a  Decision Review Officer (DRO) review as opposed to a traditional review,  (2) outcomes for veterans who choose a DRO review, and (3) challenges  the Department of Veterans Affairs (VA) faces in managing DROs. To  answer these objectives, we reviewed relevant laws and regulations, and  VA documents including procedural manuals, internal studies, and the  sample appeal election letter that VA headquarters provides for its  regional offices. We interviewed cognizant officials from the Veterans  Benefits Administration (VBA) and the Board of Veterans’ Appeals  (Board). We interviewed managers, DROs, and veteran service  organization (VSO) staff in 4 regional offices, and we conducted a web- based survey of managers in all of VA’s 57 regional offices. We obtained  and analyzed administrative data from the Board on outcomes and  processing times for appeals of disability compensation claims. Finally, to  learn about veterans’ perspectives on the DRO review election process  we conducted phone interviews with a small sample of veterans who had  appealed their disability compensation claims.
We conducted this review from July 2010 to September 2011 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Analysis of VA Appeal Process Request Letter
paragraphs: We analyzed the VBA’s appeal process request letter as a factor that may  affect veterans’ decisions about the appeal options. For our analysis we  used the sample letter in the VBA manual since VBA central office  officials said that regional offices use the sample as a template for the  letters they mail to veterans. We selected 16 criteria for clear federal  communication using information from the Plain Language Guidelines, a  previous GAO report on VBA letter clarity, and the VBA manual (see  table 1). We reviewed the letter and came to a consensus about which  criteria were met and which were not met.
We conducted site visits to 4 of VA’s 57 regional offices: Atlanta, Georgia;  Providence, Rhode Island; Salt Lake City, Utah; and Waco, Texas. We  conducted these visits to learn about regional office practices in utilizing  DROs and regional staff’s opinions on such topics as how veterans  choose between appeal options, factors affecting DRO review outcomes,  and challenges facing VA in utilizing DROs. In each regional office, we  interviewed regional office managers, at least one appeals coach,  multiple DROs, and local VSO staff. In total, we interviewed 17 DROs  across the 4 offices, with varying levels of experience in their position. In  some offices we also interviewed the regional office training coordinator.  We judgmentally selected these offices to achieve variation in several  factors, including geographic location, number of staff, timeliness of  appeal processing for appeals in which DRO was selected, and  participation in a pilot study of revised criteria for assessing DRO  performance (see fig. 12). What we report about these sites may not  necessarily be representative of other VA regional offices.
To analyze data on VA’s appellate workload, including the number of  appeals in which DRO and traditional reviews were selected, the  outcomes for DRO and traditional reviews, and the processing times for  DRO and traditional reviews, we obtained record-level appeals data  extracted on April 7, 2011, from the Veterans Appeals Control and  Locator System (VACOLS). We limited our analysis to appeals of  disability compensation claims. We also limited our analysis to original  appeals, as opposed to, for example, appeals that had been remanded by  the Board. Using the record-level data, we generated nationwide annual  data on appeals filed from fiscal years 2003 through 2010. The data for  each fiscal year includes all notices of disagreement filed with the VA  during that fiscal year. We looked at data back to fiscal year 2003, even  though the DRO review process was established in 2001, because the  VACOLS data element that identifies appeals in which DRO review was  selected was only added to VACOLS at the end of 2002. For some  analyses, we excluded data for appeals filed in recent fiscal years  because a high proportion of these appeals were still pending some  action by the regional office. To assess the reliability of the record-level  appeals data, we (1) reviewed documentation on VACOLS including the  data dictionary, (2) interviewed Board officials about VACOLS and any  data reliability issues, and (3) performed electronic testing. We found the  data to be sufficiently reliable for our reporting objectives.
section_title: Statistical Analysis of Factors Affecting Choice of DRO Review
paragraphs: To investigate the relationship between DRO review selection and  representation, using VACOLS data, we first examined descriptive  statistics at the regional office level. We found substantial variation in  rates of DRO election, as well as rates of representation among veterans  filing appeals, across regional offices. Using linear multiple regression  analysis, we estimated that at the regional office level, a one percentage  point increase in the proportion of veterans with representation was  associated with approximately half a percentage point increase in the rate  of DRO election across regional offices, after controlling for fiscal year.  However, variation in representation levels across regional offices did not  fully explain variation in DRO election across offices.
Our next analysis focused on appeals level data. Descriptive data show  that, prior to controlling for other factors, veterans with representation  were more likely to choose a DRO review than those without  representation. Specifically, table 2 shows that approximately 63 percent  of those veterans with representation requested a DRO reviews,  compared to approximately 44 percent of veterans with no representation.
An alternative way to examine the likelihood of DRO election among  veterans with and without representation is the odds of requesting a DRO  review. Prior to controlling for other factors, the odds that a veteran with  no representation requesting a DRO review is defined as the proportion of  veterans requesting a DRO review divided by the proportion requesting a  traditional review, or 0.79 (44.1/55.9). In comparison, among veterans  with representation, the odds of requesting a DRO review were 1.72  (63.3/36.7). To compare the relative proportion of veterans with and  without representation that requested a DRO review, we can construct an  odds ratio that compares the odds of DRO selection to the reference  group of veterans without representation, which is 2.18 (1.72/0.79). In  other words, prior to controlling for other factors, the unadjusted odds that  a veteran with representation requesting a DRO review were 2.18 times  that of the odds that a veteran without representation requesting such  review.
Although unadjusted odds provide useful summary information, they do  not account for multiple other factors that could also affect a veteran’s  choice of review. Logistic regression analysis can be used to estimate the  odds ratio comparing the likelihood of DRO selection among veterans  with and without representation, after also controlling for other factors that  could potentially influence the choice of review type. We estimated a  logistic regression model of DRO selection controlling for other factors  including the fiscal year of the appeal, the regional office of the appeal,  whether the veteran filing the appeal had representation, how many  distinct medical issues were associated with each appeal, and what type  of body systems were included in the issues under appeal. We were not  able to control for certain other factors, such as the timing of DRO  election, because we lacked comparative data for traditional reviews.  There may be systematic differences between veterans who decide not to  appeal a ruling and those who appeal a ruling and thus chose between  DRO and traditional reviews. Additionally, we lacked information to  control for certain factors such as a veteran’s branch of service or  demographic characteristics.
After controlling for fiscal year, regional office, number of issues in each  appeal, and body systems related to the issues under appeal, we found  that appeals filed by veterans with representation were substantially more  likely than those filed by veterans without representation to elect the DRO  review process. Per the last column of table 2, we can see that, after  controlling for other factors, the odds that an appeal filed by a veteran  with representation underwent a DRO review were approximately two  times higher than the odds that an appeal filed by a veteran without  representation underwent a DRO review. This result was statistically  significant at the p<0.05 level.
Other factors were also significantly associated with a veteran’s choice of  review. Table 2 shows that even after controlling for representation,  appeals filed at different regional offices varied substantially in the odds  that they underwent a DRO review. After controlling for other factors, the  odds that an appeal elected a DRO review were approximately four times  higher in San Diego, California, than the odds at Albuquerque, New  Mexico, and the odds that an appeal filed at Columbia, South Carolina,  elected DRO were approximately 75 percent lower than those filed in  Albuquerque. Additionally, compared to appeals with only one issue,  veterans filing appeals with multiple issues had higher odds of electing a  DRO review after controlling for other factors. After controlling for other  factors, the odds that an appeal elected a DRO review increased  significantly over time. For example, the odds that an appeal filed in fiscal  year 2010 underwent a DRO review were about 63 percent higher than  the odds for an appeal filed in fiscal year 2003. For the most part, appeals  that included issues related to some body systems had odds of DRO  review similar to appeals that did not mention those systems, although  the difference was statistically significant for several issues.
As noted previously, we could not control for the full range of variables  that could conceivably affect a veteran’s choice of review type, including  the timing of DRO election, a veteran’s branch of service or demographic  information, or what factors affected whether a veteran decided to appeal  rather than accept an initial ruling. To the extent that these appeal- specific factors are correlated with both the choice of a DRO review rather  than traditional review and other variables in our model, our model  estimates may be biased. We tested several alternative specifications of  the model, such as reclassifying those whose representation status was  unknown to either having or not representation, and found that the  direction and magnitude of our results were similar across models. We  further confirmed that the model estimates were not sensitive to the  inclusion or exclusion of body system or recategorization of the variable  measuring the number of issues in an appeal. Overall, while we are  confident that our models of the type of review provides some explanatory  power and are robust to alternative specifications, they are limited in their  ability to substantially explain individual veterans’ decisions to choose a  DRO review rather than a traditional review.
We used appeals level VACOLS data to examine whether DRO reviews  were more or less likely to have a full or partial award granted than  traditional reviews. Table 3 shows the likelihood of grant awards by  review type (DRO or traditional). Although 22.7 percent of appeals that  underwent a traditional review received a full or partial award, 31.5  percent of appeals that underwent a DRO review received a full or partial  award. Comparing the odds of a full or partial award for cases that  underwent different review processes, the unadjusted odds ratio  comparing the odds of an award for a DRO review to those of a traditional  review is 1.56. In other words, prior to controlling for other factors that  could affect the odds of an award, the odds that a DRO-reviewed appeal  received an award were 56 percent higher than those for a traditional- reviewed appeal.
We used logistic regression analysis to assess if DRO reviews still had  higher odds of receiving a full or partial grant, even after controlling for  selected factors that could affect the likelihood of an award grant. We  controlled for fiscal year; regional office; number of issues in each appeal,  and body systems related to the issues under appeal; and veterans’  representation status. After controlling for these factors, we found that  DRO-reviewed appeals still were significantly more likely than traditional  reviews to receive an award. Specifically, table 3 shows that the odds that  an appeal that underwent a DRO review received a full or partial award  were still approximately 60 percent higher than an appeal that went  underwent a traditional review after controlling for other factors (odds ratio  1.58). Besides a veteran’s choice of appeal option, some other variables  were also correlated with higher odds of a receiving an award. For  example, appeals filed in certain regional offices were significantly more  likely to receive grants than appeals in other offices.
We also found that, after controlling for review type and other factors, the  odds that an appeal filed by a veteran with representation won a full or  partial award were quite similar to that of an appeal filed by a veteran  without representation (odds ratio 1.03, or approximately 3 percent higher  odds). Furthermore, compared to appeals with only one issue, veterans  filing appeals with multiple issues had notably lower odds of receiving a  full or partial award after controlling for other factors (between  approximately 25 and 50 percent lower than the odds of appeals with one  issue).
We tested a variety of alternative model specifications with different  variables and populations. Given that a large portion of recent cases may  not have been resolved as of the time our data were produced, we tested  our model excluding the fiscal years 2009 and 2010 to avoid biasing our  results towards those cases that could be quickly resolved. We confirmed  that a DRO review still had higher odds of an award even after  reclassifying the dependent variable into full grant versus partial or no  grant. We further confirmed that the model estimates were not sensitive  to the inclusion or exclusion of body system or recategorization of the  variable measuring the number of issues in an appeal. We also tested a  model with an interaction term between representation status and review  type. These models confirmed that, compared to traditional cases with or  without representation, appeals that went through DRO review were more  likely to receive a full or partial grant. Overall, although our models  provide some explanatory power and are robust to alternative  specifications, we acknowledge that they are limited in their ability to  substantially explain the outcome of the award process, especially in light  of known omitted variables related to the specific case.
To obtain managerial views on the appeal process and use of DROs in  VBA’s 57 regional offices, and to understand variations among offices, we  conducted a web-based survey of one regional office manager, or  assistant manager if so designated by their director, at each office.
Survey development. We developed survey questions with input from an  official at VA’s Office of Field Operations and GAO subject matter experts  on survey design and performance management. To pretest the  questionnaire, we conducted in-depth probing interviews and held  debriefing sessions with four regional office managers by telephone.  Pretest participants were selected to represent variety in regional office  sizes and to include offices that participated in VBA’s pilot program for  new DRO performance measures. We conducted these pretests to  determine if the questions were burdensome or difficult to understand and  if they measured what we intended. On the basis of the feedback from the  pretests and these other knowledgeable entities, we modified the  questions as appropriate.
Survey implementation. We obtained e-mail addresses of managers  from VA’s Office of Field Operations. This office sent a message to  prospective respondents on December 20, 2010, encouraging  participation in the upcoming survey. We then began the survey by e- mailing passwords and links to the web-based questionnaire on January  4, 2011. To obtain candid responses and a high response rate, we  pledged not to link the responses presented in our report to individual  survey participants, and we followed up with two e-mails to initial  nonrespondents, the first on January 13, 2011, and the second on  January 31, 2011. Additionally, we contacted those managers who had  not responded to the survey e-mails by telephone from February 3  through February 18, 2011. We also contacted some respondents by e- mail to clarify unclear, inconsistent, or incomplete responses. We  received usable responses from 56 of the 57 managers, for a 98 percent  response rate, and ended the survey on February 22, 2011.
Analysis of responses. We used computer programs verified to be  written correctly by an independent GAO analyst to analyze the  responses. We provided respondents with an opportunity to answer  several open-ended questions. The responses to those questions were  categorized and coded for content by a GAO analyst, while a second  analyst verified that the first analyst had coded the responses  appropriately. Some comments were coded into more than one category  since some respondents commented on more than one topic. As a result,  the number of coded items is not equal to the number of respondents who  provided comments.
Analysis of survey error. Because we identified and selected all 57  regional offices for our survey, our data is not subject to errors due to  selecting only a sample or failing to include a portion of the population in  the sample. The practical difficulties of conducting any survey may  introduce errors due to measurement, nonresponse, or data processing;  however, steps taken during survey development, implementation, and  analysis minimize the chance of such errors. Additionally, because of the  high response rate, the risk of nonresponse error was further minimized,  and because the sole nonresponding office was also one of the pretest  sites, we were able to determine that the site’s characteristics and pretest  answers did not differ greatly from respondents’ answers. Inclusion of  what would likely have been their final answers, as determined by their  pretest responses, would not materially affect overall results. As a result,  we conclude that there is no material risk of nonresponse bias in our  survey.
section_title: Phone Interviews with Veterans
paragraphs: We conducted structured phone interviews with veterans to gather  information on the factors that affect veterans’ appeals decisions, such as  the clarity of the VBA letter appeals process election letter, assistance  from VSOs, and other specific reasons for choosing a DRO review or a  traditional review. The team worked with a survey methodologist and  communication expert in the development of a phone script and interview  questions for the veterans, which were pretested with six veterans. The  finalized phone script and questions included screening questions to  determine if veterans understood the questions or recalled information  accurately.
To develop our sample, we obtained VACOLS data from the Board on  veterans who had filed appeals of their disability compensation decisions  between February 1, 2010 and July 31, 2010. The data provided by the  Board included the date of the appeal; whether the veteran had selected  a DRO or a traditional review; the date on which the DRO review was  selected; the veteran’s name; and, when available, the veteran’s address  and phone number. The file contained 77,542 unique appeals, which  corresponds to something less than 77,542 unique veterans, because  one veteran may have filed multiple appeals. We removed from the list all  veterans associated with a regional office that was a participant in VA’s  Expedited Claim Adjudication pilot project in 2008 since all veterans who  took part in this pilot automatically received a DRO review, so our  interview questions—which focus on how veterans made the decision to  elect DRO review—would not have been relevant for them. Four regional  offices were part of this pilot: Nashville, Tennessee; Seattle, Washington;  Lincoln, Nebraska; and Philadelphia, Pennsylvania. The final file, without  any veterans associated with the Expedited Claim Adjudication sites,  contained 71,891 appeals. We drew a random sample of 200 appeals  from this final data file. The sample of appeals represented 200 unique  veterans. The team used a random sample to protect against selection  bias.
Before conducting the interviews, we tried to obtain contact information  for veterans with missing phone numbers or addresses, but were not able  to identify contact information in all cases. When addresses were  available, we sent notification letters to veterans before beginning the  interviews. We conducted the interviews with veterans from December  2010 to April 2011. When contacting the veterans, the interviewer read  the phone script and interview questions and documented the responses.  In cases where a spouse or other family member stated that the veteran  was not able to participate in the interview, we asked to speak with the  person most knowledgeable of the appeal and conducted the interview  with this person. We also spoke with surviving spouses or children who  were appealing a disability claim of a deceased veteran. We successfully  completed interviews with 40 veterans—28 of the veterans had chosen a  DRO review of their appeal and 12 had chosen a traditional review. The  results of our interviews cannot be generalized to the overall population of  veterans who filed appeals between February 2010 and July 2010. (See  table 4 for detailed information on the implementation of the phone  interviews.)
section_title: Appendix II: VA Appeal Process Request Letter Template
section_title: Appendix III: Comments from the Department of Veterans Affairs
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Shelia Drake, Assistant Director;  Lorin Obler; Susan Aschoff; Susan Baker; Jamie Berryhill; Jessica  Botsford; Juliann Gorse; Mimi Nguyen; Anna Maria Ortiz; Patricia Owens;  Carol Petersen; Carl Ramirez; and Walter Vance made key contributions  to this report.
section_title: Related GAO Products
paragraphs: Military and Veterans Disability System: Worldwide Deployment of  Integrated System Warrants Careful Monitoring. GAO-11-633T.  Washington, D.C.: May 4, 2011.
High-Risk Series: An Update. GAO-11-278. Washington, D.C.: February  16, 2011.
Military and Veterans Disability System: Pilot Has Achieved Some Goals,  but Further Planning and Monitoring Needed. GAO-11-69. Washington,  D.C.: December 6, 2010.
Veterans’ Disability Benefits: Expanded Oversight Would Improve  Training for Experienced Claims Processors. GAO-10-445. Washington,  D.C.: April 30, 2010.
Veterans’ Disability Benefits: VA Has Improved Its Programs for  Measuring Accuracy and Consistency but Challenges Remain.  GAO-10-530T. Washington, D.C.: March 24, 2010.
Veterans’ Disability Benefits: Further Evaluation of Ongoing Initiatives  Could Help Identify Effective Approaches for Improving Claims  Processing. GAO-10-213. Washington, D.C.: January 29, 2010.
Military Disability System: Increased Supports for Servicemembers and  Better Pilot Planning Could Improve the Disability Evaluation Process.  GAO-08-1137. Washington, D.C.: September 24, 2008.
Veterans’ Benefits: Increased Focus on Evaluation and Accountability  Would Enhance Training and Performance Management for Claims  Processors. GAO-08-561. Washington, D.C.: May 27, 2008.
Veterans’ Disability Benefits: VA Can Improve Its Procedures for  Obtaining Military Service Records. GAO-07-98. Washington, D.C.:  December 12, 2006.
Veterans’ Benefits: Further Changes in VBA’s Field Office Structure  Could Help Improve Disability Claims Processing. GAO-06-149.  Washington, D.C.: December 9, 2005.
Veterans’ Disability Benefits: Claims Processing Challenges and  Opportunities for Improvements. GAO-06-283T. Washington, D.C.:  December 7, 2005.
Managing for Results: Enhancing Agency Use of Performance  Information for Management Decision Making. GAO-05-927. Washington,  D.C.: September 9, 2005.
VA Disability Benefits: Board of Veterans’ Appeals Has Made  Improvements in Quality Assurance, but Challenges Remain for VA in  Assuring Consistency. GAO-05-655T. Washington, D.C.: May 5, 2005.
Veterans’ Benefits: More Transparency Needed to Improve Oversight of  VBA’s Compensation and Pension Staffing Levels. GAO-05-47.  Washington, D.C.: November 15, 2004.
Human Capital: A Guide for Assessing Strategic Training and  Development Efforts in the Federal Government. GAO-04-546G.  Washington, D.C.: March 2004.
Veterans’ Benefits: Improvements Needed in the Reporting and Use of  Data on the Accuracy of Disability Claims Decisions. GAO-03-1045.  Washington, D.C.: September 30, 2003.
Veterans Benefits Administration: Clarity of Letters to Claimants Needs to  Be Improved. GAO-02-395. Washington, D.C.: April 23, 2002.
Human Capital: Practices That Empowered and Involved Employees.  GAO-01-1070. Washington, D.C.: September 14, 2001.