Social Security Administration: More Effort Needed to Assess Consistency of Disability Decisions
Published: 2004-07-02T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Each year, about 2.5 million people file claims with the Social Security Administration (SSA) for disability benefits. If the claim is denied at the initial level, the claimant may appeal to the hearings level. The hearings level has allowed more than half of all appealed claims, an allowance rate that has raised concerns about the consistency of decisions made at the two levels. To help ensure consistency, SSA began a "process unification" initiative in 1994 and recently announced a new proposal to strengthen its disability programs. This report examines (1) the status of SSA's process unification initiative, (2) SSA's assessments of possible inconsistencies in decisions between adjudication levels, and (3) whether SSA's new proposal incorporates changes to improve consistency in decisions between adjudication levels.
section_title: What GAO Found
paragraphs: SSA has only partially implemented its process unification initiative. Although the agency initially made improvements in its policies and training intended to address inconsistency in decisions made at the two adjudication levels, it has not continued to actively pursue these efforts. Further, as part of this initiative, the agency implemented a review of hearings level decisions to identify ways to improve training and policies, but no new improvements were made as a result of the review. Finally, the agency began tests of two process changes intended to improve the consistency of decision making between the two adjudication levels. One test, which is ongoing, was not well designed and therefore will not provide conclusive results. The other test was abandoned because of implementation difficulties. SSA's assessments have not provided a clear understanding of the extent and causes of possible inconsistencies in decisions between adjudication levels. The two measures SSA uses to monitor inconsistency of decisions have weaknesses, such as not accounting for the many factors that can affect decision outcomes, and therefore do not provide a true picture of the changes in consistency. Furthermore, SSA has not sufficiently assessed the causes of possible inconsistency. For example, SSA conducted an analysis in 1994 that identified potential areas of inconsistency, but it did not employ more sophisticated techniques--such as multivariate analyses, followed by in-depth case studies--that would allow the agency to identify and address the key areas and leading causes of possible inconsistency. SSA has yet to repeat or expand upon this 10-year-old study. SSA's new proposal incorporates changes intended to improve consistency in decisions between levels. However, challenges may hinder its implementation. Most stakeholder groups for adjudicators and claimant representatives told us that a number of aspects of the proposal hold promise for improving consistency. These included one change, being tested as part of the process unification initiative, that requires state adjudicators to more fully develop and document their decisions, as well as several new changes, such as providing both adjudication levels with equal access to medical expertise. However, stakeholder groups also told us that insufficient resources and other obstacles might hinder the implementation of some changes. Adding to uncertainties about the proposal's overall success is its dependence on a new electronic folder system that would allow cases to be easily accessed by various adjudicators across the country. However, this technically complex project has not been fully tested.

section_title: Letter
paragraphs: The Social Security Administration (SSA) is the nation’s largest provider   of income assistance to individuals with disabilities, paying $91 billion in  federal benefits to 11.4 million beneficiaries with a disability and their  families in 2003. Each year, about 2.5 million people file claims with SSA  for disability benefits. State agencies called Disability Determination  Services (DDS) decide whether claimants meet SSA’s definition of  disability by applying SSA’s decision-making criteria. If a DDS ultimately  decides, after an initial determination and then a reconsideration of this  decision, that a claimant does not meet SSA’s definition for disability, the  claimant may appeal to the hearings level, where an SSA Administrative  Law Judge (ALJ) reviews the claim to decide if the claimant should be  allowed benefits. About one-third of disability claims denied at the state  level were appealed to the hearings level; of these, SSA’s ALJs have  allowed over one-half, with annual allowance rates fluctuating between 58  percent and 72 percent since 1985. While it is appropriate that some  appealed claims, such as those in which a claimant’s impairment has  worsened and prohibits work, be allowed benefits, representatives from  SSA, the Congress, and interest groups have long been concerned that the  high rate of claims allowed at the hearings level may indicate that decision  makers at the two levels are interpreting and applying SSA’s criteria  differently. If this is the case, adjudicators at the two levels may be making  inconsistent decisions that result in similar cases receiving dissimilar  decisions.
Concerned about the possibility that adjudicators are making inconsistent  decisions, SSA embarked on a “process unification” initiative in 1994 with  the goal of ensuring that adjudicators at both levels consistently apply  SSA’s policy guidance and make similar decisions on similar cases. Partly  on the basis of early studies of potential causes of inconsistent decisions,  SSA included in its process unification initiative efforts to provide  consistent guidance to all adjudicators, clarify policy, provide training, test  potential process changes, and perform a new quality review of  allowances decided by ALJs. However, SSA continues to face challenges in  its efforts to provide consistent disability decisions. These challenges, and  others associated with modernizing its disability programs, contributed to  our decision to include federal disability programs on our list of high-risk  government programs. In September 2003, SSA’s Commissioner unveiled  a new proposal that laid out the vision of a long-term strategy for  improving the disability decision-making process and helping people with  disabilities return to work. Several of the changes in the new proposal are  intended to improve the accuracy, timeliness, and consistency of  decisions, such as having the DDS decision makers more fully develop and  document their decisions, providing for centralized quality review of all  decisions, and providing both adjudication levels equal access to medical  expertise.
In response to your interest in the effectiveness of SSA’s past and future  efforts to improve and assess the consistency of decisions between levels,  we evaluated these agency efforts. Specifically, we examined (1) the status  of SSA’s process unification initiative, (2) SSA’s assessments of possible  inconsistencies in decisions between adjudication levels, and (3) whether  SSA’s new proposal incorporates changes to improve consistency in  decisions between adjudication levels.
To assess the extent to which SSA has implemented its planned activities  under the initiative, we evaluated agency documentation describing SSA’s  process unification efforts. To evaluate SSA’s efforts to assess consistency  in decisions between levels, we interviewed officials from SSA’s Office of  Quality Assurance and Performance Assessment (OQA) and reviewed  summary data and reports from SSA’s quality assurance and performance  management systems, including findings from SSA’s Disability Hearings  Quality Review, which is a quality review of DDS adjudicators’ and ALJs’  decisions and the associated case files. While we evaluated SSA’s methods  and approaches for assessing the consistency of decisions, we generally  did not trace figures cited by SSA back to their original source  documents. To further evaluate SSA’s process unification efforts and its  new proposal, we interviewed selected officials who have firsthand  knowledge about these issues, including SSA officials, staff of the Social  Security Advisory Board, and leaders of SSA stakeholder groups  representing state and hearings office adjudicative staff and claimant  attorneys. We also reviewed recent testimony and other documents on the  consistency of SSA’s decisions from the Social Security Advisory Board  and other stakeholder groups. We conducted our work between February  2003 and March 2004 in accordance with generally accepted government  auditing standards. Our ability to evaluate SSA’s new proposal has been  limited by a lack of detailed information, such as specific information on  how changes will be implemented and their costs, because the agency is  still in the process of developing and refining its proposal.
subsections: section_title: Background
paragraphs: SSA operates the Disability Insurance (DI) and Supplemental Security  Income (SSI) programs—the two largest programs providing cash benefits  to people with disabilities. The law defines disability for both programs as  the inability to engage in any substantial gainful activity by reason of a  severe physical or mental impairment that is medically determinable and is  expected to last at least 12 months or result in death. The programs have  grown substantially, from 10.7 million beneficiaries and $61 billion in  benefits in 1995 to 11.4 million beneficiaries and $91 billion in federal  benefits to individuals with disabilities in 2003.  While disability benefits  account for only 15 percent of SSA’s total benefit payments for its Old-Age,  Survivors and Disability Insurance (OASDI) programs, administering the  disability benefits accounted for 45 percent of the agency’s annual  administrative expenses.  The relatively high cost of administering the DI  program reflects the complex and demanding nature of making disability  decisions. SSA estimates that the cost of the disability programs will rise  substantially in the near future as the baby boom generation reaches its  disability-prone years.
The disability determination process begins at a field office, where an SSA  representative determines whether a claimant meets the programs’ non- medical eligibility criteria. Claims meeting these criteria are forwarded to  the state DDS to determine if a claimant meets the agency’s definition of  disability. At the DDS, the disability examiner takes the lead, or works as a  team with the medical or psychological consultants, to analyze a  claimant’s documentation, gather additional evidence as appropriate, and  approve or deny the claim. A denied claimant may ask the DDS to  reconsider its finding, at which point a different DDS team reviews the  claim. If the claim is denied again, the claimant may appeal the  determination to SSA’s Office of Hearings and Appeals (OHA), where it  will be reviewed by an ALJ. The ALJ usually conducts a hearing in which  the claimant and others may testify and present new evidence. In making  the disability decision, the ALJ uses information from the hearing and  from the state DDS, including the findings of the DDS medical consultant.  A claimant whose appeal is denied may request a review by SSA’s Appeals  Council and, if denied again, may file suit in federal court.   Figure 1 provides an overview of SSA’s disability decision-making process  and outcomes for 2003.
SSA uses a sequential evaluation process when determining disability.  First, SSA field office representatives determine whether a claimant is  performing substantial gainful work. If not, DDS or ALJ adjudicators will  assess the severity of a claimant’s medical condition(s) to determine  whether it meets or equals the criteria in SSA’s regulations (commonly  referred to as the medical listings). For a claimant whose conditions do  not meet or equal the listings, adjudicators then focus on the functional  consequences of the claimant’s medically determined impairments—that  is, whether the claimant can perform work he or she has done in the past,  and, if not, whether the claimant can perform other work in the national  economy.
Concerns about the rate of appeals for hearings, ALJs’ allowance rates,  and the accuracy and consistency of ALJ decisions led the Congress to  direct SSA to conduct a study in 1980 to determine the extent to which  hearings decisions conformed to legal requirements and binding SSA  policy. Since the allowance rates at the hearings level could be influenced  by many factors, such as the introduction of new evidence, the purpose of  the 1980 study was to present the same evidence on cases to different  reviewers representing different adjudication levels. In determining the  extent to which decision makers agreed on whether to allow or deny  benefits, the study concluded that different levels of decision makers had  significantly different allowance rates. Specifically, the ALJs decided to  allow 64 percent of the cases, whereas the SSA’s central office quality  assurance reviewers, comprising medical consultants and disability  examiners, decided that only 13 percent of cases should be allowed. The  study identified several possible causes of the disparity, including  inconsistency in the standards and procedures, interpretation of the  standards, and weight given to the evidence. The study also found that  disability decisions are complex and necessarily involve some degree of  subjectivity by adjudicators.
To help address concerns raised by this and other studies, SSA began its  process unification efforts to ensure that both levels more consistently  interpreted and applied SSA’s policy guidance. SSA’s plans for its process  unification initiative were part of SSA’s larger effort to redesign its  disability claims process and were modified over time. SSA’s process  unification plans included six major efforts, as described in table 1.
In 1997, we reported on the possible reasons for the inconsistency of  decisions between the initial and hearings levels. Our report found that  differences in state DDSs’ and ALJs’ views on the claimants’ functional  abilities was a key factor in explaining why ALJs allowed benefits on  appealed cases. We also reported that poorly documented state DDS  evaluations of the claims were of limited use to ALJs and SSA quality  reviews did not focus on identifying inconsistency in decisions. To support  SSA’s process unification efforts, the report recommended that SSA, using  available systems and data collected so far, move quickly ahead to  implement its quality assurance initiative to provide consistent feedback  to DDS and ALJ adjudicators as soon as possible. In addition, we  recommended that SSA expand its effort to return cases to a DDS for  review when new evidence is introduced on appeal. Last, we  recommended that SSA set goals for measuring the effectiveness of  process unification in reducing inconsistent decisions.
More recently, the Social Security Advisory Board issued a 2001 report  that identified many factors that could potentially affect the overall  consistency of disability decision making between adjudication levels.  Some of the factors the board suggested as potentially affecting  consistency included:  the fact that most claims are decided based on a paper review of case  evidence without face-to-face contact with an adjudicator until a  claimant has an ALJ hearing,  involvement of attorneys and other claimant representatives at the ALJ  hearing,  the fact that claimants are allowed to introduce new evidence and  allegations at each stage of the appeals process,    differences in quality assurance procedures applied to initial- and    differences in the training given to ALJs and state examiners, and   lack of clear and unified policy guidance from SSA.
Despite SSA’s process unification efforts and related studies to improve  the consistency of decisions, recent ALJ allowance rates—which declined  after process unification began, but started increasing in 1999 to reach   61 percent in fiscal year 2003—still raise questions as to whether initial-  and hearings-level decision makers are consistently applying the agency’s  guidance. In addition to inconsistent application of SSA’s policy guidance,  there are several other reasons why a large number of ALJ allowances are  made. For example, some ALJ allowances should be expected because, by  law, cases can remain open throughout the hearings process, allowing new  evidence to be submitted that may not have been available to the state  adjudicators. Such new evidence could show that the claimant’s condition  has worsened and prohibits work. Also, SSA’s decision-making criteria  require that a great deal of professional judgment be applied. As a result,  some allowances at the hearings level could simply reflect the differing  judgments of two adjudicators reviewing a case. While a claimant’s  deteriorating health, changes in the characteristics of a claim over time,  and the complexity of disability decisions may help to explain some of the  ALJ allowances, studies have not sufficiently explained why consistently  over half the cases appealed to the hearings level are allowed. Instead,  studies indicate that systemic differences in the assessment of claims at  both adjudication levels are contributing to the ALJ allowance rate. For  example, our 1997 report noted a difference in state DDSs’ and ALJs’ views  on the claimant’s functional abilities was a key factor in explaining why  ALJs allowed cases on appeal.
Inconsistency in decisions may create several problems. High hearings  allowance rates may create the perception that the hearings level is  applying SSA’s criteria less strictly than the initial level and create an  incentive for claimants to appeal to an ALJ for a more favorable decision.  If deserving claimants must appeal to the hearings level for benefits, this  situation increases the burden on claimants, who must wait, on average,  almost a year for a hearing decision and frequently incur extra costs to pay  for legal representation. In addition, to the extent that the ALJ allowance  rates include inappropriate allowances, SSA could be incurring  unwarranted program costs. Although SSA has tried to address these  problems, its inability to resolve them has contributed to our decision to  include federal disability programs on our list of high-risk government  programs.
Renewing its effort to address long-standing and critical problems with the  disability programs, SSA’s Commissioner recently announced a new  proposal to improve these programs. (See app. I for an excerpt of the  announcement that describes the newly proposed decision-making  process.) In addition to proposing demonstration projects that provide  work incentives and supports to help people with disabilities return to  work, SSA has proposed significant changes to both the process of  adjudicating disability claims and the structure and management of the  agency’s quality management system to improve the timeliness, accuracy,  and consistency of the disability decision-making process. The agency  believes that several of these changes will help to improve consistency  between DDS and ALJ decisions. For example, SSA plans to provide more  centralized end-of-line quality reviews. According to SSA, the proposed  quality reviews should help to hold adjudicators more accountable for  their decisions and ensure that they consistently apply SSA’s policies as  well as help the agency detect and amend those policy areas leading to  inconsistent decisions. Table 2 provides a description of SSA’s proposed  changes to improve the disability decision-making process.
SSA does not plan to implement its proposed changes before it has  successfully implemented its Accelerated Electronic Disability (AeDib)  system. This major initiative should allow adjudication staff in states and  throughout the agency, regardless of geographic location, to access case  information electronically through the use of an electronic disability  folder. The initiative is intended to reduce delays that result from mailing,  locating, and organizing paper folders. SSA also expects this new system  to provide critical management information for analyzing and reducing  inconsistencies in disability decisions. SSA is implementing the new  system and plans to give adjudicators time to adjust to this change before  implementing its new proposal.  SSA’s implementation of the new  proposal will therefore be no earlier than October 2005. In the meantime,  SSA continues to discuss the proposal with stakeholders and plans to  further refine it before implementation.
section_title: SSA Has Partially Implemented Its Process Unification Initiative
paragraphs: SSA has partially implemented its process unification initiative. Although  the agency initially made improvements in its policies and training  intended to improve the consistency of decisions between adjudication  levels, it has not continued to actively pursue these efforts. As part of the  initiative, the agency also implemented a review of ALJs’ allowance  decisions to identify additional ways to improve training and policies, but  no new changes were made as a result of findings from the review. Finally,  the agency also began two tests of process changes to help improve the  consistency of decisions, but one ongoing test with design problems is not  likely to lead to any conclusive results and the other test has been  abandoned.
subsections: section_title: SSA Made Early Progress Improving Policies and Training, but Has Not Actively Pursued These Efforts
paragraphs: While SSA initially made progress carrying out efforts to improve policies  and training to better ensure the consistency of decisions, the agency has  not continued to actively pursue these efforts. SSA quickly accomplished  most of its planned efforts to clarify policy guidance. In 1996, SSA issued  nine process unification rulings to clarify policy areas it found to be  contributing to inconsistent decisions. For example, one ruling provided  all adjudicators with guidance on how to weigh and document their  evaluation of the treating physician’s opinions when making a disability  decision. SSA successfully went through the regulatory process several  years later and published three new regulations to strengthen its process  unification rulings, but was unable to agree on a fourth regulation  regarding the weight to be given to the treating physician’s opinion when  evaluating a claim.
SSA planned to develop a single presentation of policy guidance to replace  the different sources used by each level, but has since abandoned full  implementation of these plans in favor of a more limited approach. DDS  adjudicators currently follow a detailed set of policy and procedural  guidelines, whereas ALJs rely directly on statutes, regulations, and  rulings for guidance in making disability decisions. To help ensure that  inconsistent guidance was not contributing to inconsistent DDS and ALJ  decisions, SSA began issuing guidance in the same wording to all  adjudicators in 1996. Although SSA had also planned to address  differences in policy guidance issued before 1996 and to eventually  combine existing adjudication policy documents into a single document, it  ultimately decided not to take these additional steps. According to SSA,  further efforts to unify the policy guidance used by both levels would be a  massive undertaking and not worth the cost because the guidance issued  since 1996 had already addressed important policy areas that were leading  to inconsistent decisions. While some stakeholder groups representing  adjudicators tended to agree with SSA’s position, the Social Security  Advisory Board and other groups still believe the agency should take  additional steps to provide a unified policy guide to all adjudicators.  Instead of creating one policy manual for all adjudicators, SSA told us that  it plans to undertake a comprehensive effort to evaluate and improve its  disability policies to make them less susceptible to differing  interpretations and to ensure they are up to date. A more comprehensive  approach could address key weaknesses in SSA’s disability program that  we previously highlighted in our performance and accountability series,  and thereby help to modernize federal disability programs to better meet  the needs of Americans with disabilities.
Early on, SSA also provided extensive cross-training of DDS and ALJ  adjudicators, although the scope of its efforts has since diminished. To  help all adjudicators understand how to appropriately apply process  unification rulings, SSA provided extensive and mandatory training in   1996 and 1997 to 15,000 disability adjudicators (including DDS examiners,  physicians, ALJs, and quality assurance staff). The training was provided  to adjudicators at all levels of the process in three of the most complex  disability areas—assessment of symptoms, treatment of expert opinions,  and assessment of claimants’ remaining capacity to work (i.e., residual  functional capacity). While this training was intended to be ongoing, SSA’s  training efforts have diminished significantly since 1997. Stakeholder  groups representing DDS adjudicators told us that SSA’s training does not  sufficiently cover process unification issues. In addition, our review of  DDS and OHA participation in video training revealed inconsistent  participation in training by adjudicators. To provide ongoing training to  both adjudication levels and other components involved in the claims  process, SSA has used interactive video technology. Almost all the state  DDS sites and about 85 percent of OHA offices have this technology.  However, in reviewing participation for two recent courses, we found for  those sites with this interactive technology only 31 percent of DDS sites  and 16 percent of OHA sites logged on for a course on the role of  consultative examinations, and 18 percent of DDS sites and 4 percent of  OHA sites logged on for a monthly disability hour training class.  According to SSA, neither DDS nor OHA adjudicators are generally  required to attend courses. In line with these findings, our recent report on  the human capital challenges facing DDSs found gaps in the key  knowledge and skills of their adjudicators in the same areas SSA had  earlier identified as critical to making consistent decisions, and we  recommended that SSA work with DDSs to close these gaps.
Despite SSA’s early efforts to improve policy guidance and provide  training, stakeholder groups representing state adjudicators told us that  many states are not performing the additional development and  documentation of decisions required by the process unification rulings.  They also told us that the rulings have added significantly to the time,  complexity, and subjectivity of the decision-making process, while  insufficient resources have limited their ability to fully implement the  rulings’ requirements. In addition, claimant lawsuits against three state  DDSs have alleged that DDS adjudicators were not following SSA’s rulings  or other decision-making guidance. In settling these lawsuits, SSA agreed  to have these states fully develop and document cases. However,  according to DDS stakeholder groups, SSA has not ensured that states  have sufficient resources to meet ruling requirements, which they believe  may lead to inconsistency in decisions among states. Furthermore, SSA’s  quality assurance process does not help ensure compliance because  reviewers of DDS decisions are not required to identify and return to the  DDSs cases that are not fully documented in accordance with the rulings.  SSA’s procedures require only that the reviewers return cases that have a  deficiency that could result in an incorrect decision.
section_title: SSA’s Review of ALJ Allowances Has Not Resulted in Improvements to Policy and Training
paragraphs: As part of its initiative, the agency has also implemented a quality review  of ALJ decisions, but the review has not proved useful for identifying any  new changes to SSA’s policies or training that would help to address the  inconsistency of decisions. This review—referred to as the ALJ Pre- effectuation Review—involves a sequential review by SSA’s OQA and the  Appeals Council of certain ALJ allowances that have not yet been finalized  (i.e., the claimant has not yet been awarded benefits). In selecting  allowances for review, OQA uses an error-prone profile developed from its  analysis of errors detected when reviewing DDS allowances. SSA began  testing the new review of ALJs’ decisions in 1996 and implemented it as an  annual review in 1998. From fiscal years 1998 through 2002, OQA reviewed  27,148 ALJ allowances and of these, OQA found fault with about 35  percent and referred them to the Appeals Council. The Appeals Council  screens the allowances for its own review and selects those in which the  prior actions may not have been proper, fair, or in accordance with the law  or the ALJ’s decision was not supported by substantial evidence. If the  council finds fault with the ALJ’s decision, it will deny the claimant  benefits or return the claim to the ALJ to have the identified problems  corrected. If the council does not find fault with the ALJ’s decision, the  claimant will be awarded benefits.
In addition to identifying inappropriate ALJ allowances, SSA intended to  use the new quality review to identify areas of inconsistency between  adjudication levels and ways to improve policies and training to address  those inconsistencies. Specifically, OQA identified cases where it found  fault with the ALJ decision, but the Appeals Council, after screening them,  did not accept them for review. OQA then forwarded these cases to a  panel of staff from the various components involved in SSA’s claims  process to determine whether the inconsistent assessment of these cases  by OQA and the Appeals Council indicated the need to clarify policies,  issue new policies, or provide training to improve the consistency of  decisions. However, according to a SSA official, this review did not  identify any new areas of inconsistency that required improvements to  policy and training. Weaknesses in the design of the review may have  contributed to SSA’s inability to identify new policy areas contributing to  inconsistency. For example, rather than reviewing a random sample of all  ALJ decisions, this review focused on allowances. Further, the review  looked only at ALJ allowances that were selected using a DDS error-prone  profile, i.e., a profile that is based upon cases in which quality reviewers  did not agree with the DDS adjudicators’ decisions. As a result, SSA  selected and reviewed nonrandom allowance decisions with case  characteristics that the agency may have already suspected were  associated with inconsistent decisions. In 1999, the panel was disbanded  because members had other priorities needing attention. OQA told us that  it continued to perform a limited review of cases viewed differently by  OQA and the Appeals Council. More recently, OQA began an effort to  summarize the results of its review and expected to issue a report of its  findings in April 2004. As of April 2004, this report had not been issued.
section_title: SSA Began Two Tests of Process Changes to Improve the Consistency of Decisions, but Neither Test Was Successfully Completed
paragraphs: SSA began two tests of potential changes to the process to help improve  the consistency of decisions, but neither test was successfully completed.  The changes tested were (1) more fully developing and documenting  decisions made at the initial level and (2) sending appealed cases that  involve new medical information back to the initial level to be reevaluated.
SSA wanted to test having DDSs more fully develop and document  decisions because it believed that DDS decisions, especially denials, are  often not well documented. SSA wanted to test whether better  explanations of why benefits were denied would improve the accuracy of  DDS decisions and consistency of decisions between adjudication levels.  SSA first implemented a pilot of this change to explore alternatives for  developing and documenting decisions. Then SSA tested this change,  along with other process changes, in a larger test, called the prototype  initiative. Concurrently, SSA tested other process changes, such as the  elimination of a reconsideration step and a predecision DDS interview  with the claimant.
The prototype test had limitations for predicting the impact of  documented decisions. For example, SSA’s decision to test several  changes together left the agency without clear information on what impact  fully developed decisions would have on the decision-making process  without the other process changes. SSA’s test design also did not build in  an ALJ feedback mechanism to provide sufficient information on the  usefulness of more fully documented decisions. SSA continues to test this  change along with other changes and, despite limited information on the  best approach for and impact of this change, currently plans to implement  more fully documented decisions as part of the Commissioner’s new  proposal to improve SSA’s disability programs.
SSA also began, but ultimately abandoned, a test in which appealed cases  with new medical information submitted prior to the hearing were to be  sent back to the initial level so that the evidence could be evaluated by  medical consultants residing at the DDSs. Since medical expertise resides  in the DDS and not at the hearings level, SSA decided to test whether  “remanding,” or sending cases to the DDS for evaluation, might result in a  more consistent review of medical evidence. SSA believed that this  change, in turn, could help improve the consistency of decisions because  the new medical information might be contributing to ALJ allowances.  However, the change also had the potential to increase the time claimants  with remanded claims would have to wait for final decisions because  claims that were not allowed by the DDSs had to be returned to OHA for  hearings. SSA began remanding cases in July 1997, with a 1-year goal of  remanding 100,000 cases, but after 10 months, it had remanded fewer than  9,000. In implementing this test, SSA encountered several difficulties. For  example, it had difficulty identifying the claims to be remanded and  ensuring the ALJs, who had authority over the claims, would remand the  claims to the DDSs. The ALJs’ resistance to remanding claims to the DDSs  may be due in part to concerns that remanding would not lead to many  allowances by the DDSs and would result in many claims being returned  to OHA, thereby increasing the time many claimants would have to wait  for a final decision from OHA. Realizing that the agency would not be able  to reach its remanding goal, the agency decided to discontinue this test.
section_title: SSA Lacks a Clear Understanding of the Extent and Causes of Inconsistency between Levels
paragraphs: SSA’s assessments have not provided the agency with a clear  understanding of the extent and causes of possible inconsistencies in  decisions between adjudication levels. The two measures SSA uses to  monitor changes in the extent of inconsistency of decisions have  weaknesses and therefore do not provide a true picture of the changes in  consistency. In addition, SSA has not sufficiently assessed the causes of  possible inconsistency. The agency conducted an analysis in 1994 that  identified some potential areas of inconsistency. However, although SSA  continues to collect information that would support this analysis, it has  not repeated this initial effort, nor has it expanded on it by employing  more sophisticated assessment techniques.
subsections: section_title: SSA Attempts to Monitor Changes in the Extent of Inconsistency, but the Measures It Uses Provide an Incomplete Picture
paragraphs: SSA has made some efforts to monitor changes in the extent of  inconsistency between the initial and hearings levels, including tracking  trends in allowance rates at different levels and conducting special  reviews of ALJ decisions. Together, according to SSA, these measures and  assessments suggest that the consistency between levels has improved  since the agency began implementing its process unification initiative.  However, because of methodological weaknesses, these measures provide,  at best, a partial picture of trends in the consistency of decisions between  adjudication levels.
SSA tracks trends in the proportion of all allowances decided at each level  to assess the consistency of decisions between levels. The agency collects  information on the number of allowances granted to claimants at each  level of the process, tracks the proportion of claims allowed at the initial  level relative to the hearings level, and looks at trends in these proportions  over a period of several years. According to data from SSA, the proportion  of overall allowances that occurred at the initial level has increased since  process unification was implemented. As shown in figure 2, in fiscal year  1996, 72 percent of all allowances were granted at the initial level. This  proportion increased in most subsequent years, and by fiscal year 2003,   77 percent of all allowances were granted at the initial level. Officials from  OQA, the office responsible for reviewing, evaluating, and assessing the  integrity and quality of the administration of SSA’s programs, view the  relative shift toward earlier allowances as an indicator that consistency  between adjudication levels has improved, and they believe that process  unification efforts have contributed to these results.
However, SSA’s measure of tracking yearly changes in the proportion of  allowances at each level is a simplistic and inconclusive indicator of  trends in the consistency of decisions because it does not control for the  multitude of factors that can affect allowance rates at either adjudication  level in any given year and over time. For example, SSA uses “snapshot”  data in looking at the proportion of allowances granted at each level,  meaning that it looks at the number of claimants and allowances at each  level during a given year, rather than following a 1-year cohort of initial  claimants through the entire process and capturing the proportion of  allowances for that cohort decided at each level. Because SSA uses data  that illustrate allowance rates at a given moment in time, it captures a  different pool of claimants in the process at each level, and the resulting  allowance rates are subject to a different set of demographic and case  characteristics. Over time, the pool of claimants may change because of  factors such as a downturn in the economy, which can cause more people  with less severe impairments to claim benefits or appeal initial denial  decisions. In addition, snapshot data may be significantly affected by  fluctuations in productivity at either adjudication level caused by process  changes that are unrelated to process unification and that affect only one  level.
SSA has collected other data to further assess trends in the consistency of  decisions. Since 1993, the agency has conducted a biennial case review as  part of its Disability Hearing Quality Review process. This review  consists of medical consultants and disability examiners in SSA’s central  office evaluating a sample of ALJs’ decisions plus supporting  documentation to determine whether the ALJ has adequately supported  his or her decision. In evaluating the ALJ decisions, these medical  consultants and disability examiners use the same standards as those used  by initial-level adjudicators to adjudicate claims, which are from the  official SSA program policy and operations guidance found in POMS. To  some degree, therefore, the medical examiners and disability reviewers  serve as a proxy for initial-level adjudicators, and their decisions are  representative of how initial-level examiners should be deciding claims.
While unpublished results from the biennial case reviews indicate an  increase in supportable ALJ allowances, such findings focus on the ALJ  level and therefore provide only a partial picture of trends in consistency.  The reviews indicated that medical consultants and disability examiners  have found that supportable ALJ allowances increased from 36 percent in  fiscal year 1993-94 to 57 percent in fiscal year 1999-2000. OQA officials told  us that this increase suggests an improvement in consistency between  adjudication levels because it indicates that disability examiners using  initial-level standards and ALJs increasingly agree on how like cases  should be decided. However, SSA’s assessment provides only a partial  picture because it does not reflect trend information on the extent to  which ALJs have found DDS decisions to be supportable, to ensure that  both levels are making more consistent decisions. Although the 1994  report of findings from the initial biennial case review included the results  of a special probe in which ALJs reviewed 165 DDS reconsideration denial  decisions, the sample was not representative, and therefore results could  not serve as a baseline for developing trend information. In 2003, SSA  began another probe, in which ALJs reviewed 400 DDS reconsideration  denial determinations, but the agency does not plan to release its findings  until summer 2004.
Although SSA has limited information on how ALJs view DDS decisions,  other information collected by the agency suggests that consistency of  decision making at the initial level might not be improving. For example,  OQA reviewers routinely assess the accuracy and supportability of DDS  decisions. A recent SSA study of these data shows that the accuracy of  DDS denial decisions—those decisions most likely to be appealed to the  hearings level—has declined by 4 percentage points over a 1- year period.  Another review of DDS decisions by OQA reviewers also suggests a lack of  improvement at the initial level. Specifically, the extent to which quality  reviewers found that DDS reconsideration denials appealed to the  hearings level were supported declined from 71 percent in fiscal year   1993-94 to 68 percent in fiscal year 1999-2000.
section_title: SSA’s Assessments Have Not Identified the Areas and Causes of Possible Inconsistency in Decisions between Adjudication Levels
paragraphs: Despite some efforts to assess inconsistency in decisions, shortcomings in  SSA’s analyses also limit its ability to identify areas and causes of possible  inconsistency. Most notably, over the last 10 years, SSA has not updated  its prior analyses of information from its initial biennial case review that  helped identify problem areas. In addition, SSA has not improved on its  case review and analysis by ensuring that reviewers assess all relevant  case evidence used to make decisions, or performed more sophisticated  analysis to identify the areas and causes of inconsistency in decisions.  Other efforts—including the review of ALJ allowances and a probe of DDS  reconsideration denials—have yet to yield useful information.
In 1994, for its initial biennial case review report, the agency took its first  step in identifying areas of possible inconsistency by identifying two  characteristics about the claimants and their cases over which initial-level  reviewers tended to disagree with ALJs. Specifically, the 1994 report  concluded that teams of reviewing medical consultants and disability  examiners sometimes viewed cases involving mental impairments  differently than the reviewing ALJs. In addition, these two sets of  reviewers tended to have different views on the severity of claimants’  impairments and their resulting capacity to work. According to the official  responsible for overseeing the review, the findings in this initial report  provided important support for SSA’s process unification efforts as well as  the agency’s efforts to redesign the disability claims process.
SSA continues to conduct the biennial case reviews; however, the agency  has not continued to analyze and identify areas that are viewed differently  by different adjudication levels. Specifically, SSA no longer identifies the  particular case characteristics over which reviewers from the two levels  tend to disagree. As a result, SSA does not know whether previously  identified problem areas are still present. Moreover, SSA no longer  publishes any information from the medical consultant and disability  examiner biennial case reviews, even though it has performed some  limited analysis of the supportability of decisions made by adjudicators.  By not continuing to publish its analysis and findings, the agency makes it  difficult to ensure the reliability of its methods and results, and leaves  stakeholders outside the agency, including disability groups, without a  means for understanding SSA’s assessment efforts and progress in  improving the consistency of decisions. The SSA office conducting the  study has told us that, because of downsizing and competing priorities, it  has no current plans to further analyze and publish these data.
Further, in its ongoing biennial case reviews, SSA does not make full use  of available case information that would be useful in identifying areas and  causes of inconsistency. Specifically, medical consultants and disability  reviewers do not listen to tapes of the hearings and therefore do not  review the entire case as presented to the original ALJ. Although reviewing  medical consultants and disability examiners read the ALJs’ explanations  for their original decisions, which should include the most important  factors behind the ALJs’ decisions, the reviewers do not evaluate the oral  evidence independently. An SSA official with whom we spoke indicated  that some evidence entered by witnesses at the hearing might not be  accompanied by other hard copy sources of the same information.  Therefore, reviewers would not consider information potentially relevant  to the ALJ’s decision that could be used to identify areas and causes of  inconsistency.
SSA also does not make full use of the information it collects because it  has not employed analytical tools that would improve its ability to identify  areas and causes of inconsistency. For example, SSA’s biennial case  reviews provide a rich dataset that lends itself to regression analysis to  identify areas and possible causes of inconsistency between levels.  Regression analysis would allow the agency to better pinpoint any  significant case characteristics affecting decisions and to more clearly  identify the underlying causes of inconsistency. Specifically, among the  data collected in this review are such variables as the types of impairments  the claimant has, the types of relevant medical evidence, and additional  impairments presented at the hearing. Multivariate analysis, such as a  multiple regression model, could allow SSA to assess how these and many  other factors, relative to one another, contribute to whether a case results  in a similar outcome at both levels. However, SSA has not employed this  more sophisticated multivariate technique, citing resource constraints and  competing priorities. We recognize the methodological complexities of  analyzing disability decisions, and we previously recommended that SSA  establish an advisory panel of external experts from a range of disciplines  to provide leadership, oversight, and technical assistance to the agency.  Otherwise, in forgoing such analysis, the agency will continue to miss an  opportunity to better pinpoint areas and some possible causes of  inconsistency in decisions between the two adjudication levels, and to lay  the foundation for further investigation.
Another tool SSA has not sufficiently employed for identifying areas and  causes of inconsistency is in-depth case studies involving both levels of  adjudication. Case studies, in which different adjudicators review the  same test case, can be a means for unearthing causes for inconsistency by  getting adjudicators from both levels to acknowledge and address  discrepancies in the ways they view cases. SSA has performed case studies  in the past to ascertain differences in policy interpretation between DDS  examiners and quality reviewers. However, SSA does not routinely have  both DDS examiners and ALJs perform in-depth review of the same  sample of cases, despite this method’s potential for helping identify causes  of inconsistency between the two adjudication levels. OQA officials told us  that case studies are a very resource-intensive tool because they need a  sufficient number of cases from which to generalize. Therefore, the agency  is reluctant to use this approach to help it understand the causes of  inconsistency between adjudication levels. However, using multivariate  analyses of the biennial case review data could help the agency to more  effectively target its in-depth case studies on those areas found to be  leading to inconsistent decisions and thereby increase its success at  identifying the causes of inconsistency.
SSA conducts other analyses of inconsistency between levels, but to date  these efforts have yielded limited information concerning areas and  possible causes of inconsistency. For example, as part of SSA’s ALJ Pre- Effectuation Review, two different levels of reviewers have evaluated  thousands of cases. However, limitations in the review methodology, such  as not using a random sample of ALJ decisions, do not allow the agency to  use this review to identify the leading causes of inconsistency. SSA  recently began an evaluation of this effort and plans to publish its findings  and recommendations in April 2004. Another analysis currently under way,  a special 400-case review, might help identify areas of inconsistency at the  initial level, but it has yet to be completed. Begun in 2003, this review by  ALJs of DDS reconsideration denial determinations is expressly aimed at  assessing inconsistency between adjudication levels. SSA expects to gain  some understanding of why about 60 percent of cases denied by the initial  level and appealed to the hearings level are allowed. The agency plans to  publish its findings in summer 2004.
section_title: SSA’s New Proposal Incorporates Efforts to Improve the Consistency of Decisions, but Challenges May Impede Successful Implementation
paragraphs: Some changes included in SSA’s new proposal to overhaul its disability  claims process may improve the consistency of DDS and ALJ decisions,  but challenges may hinder the implementation of the proposal. The new  proposal includes several changes to the disability claims process that the  agency and stakeholder groups representing adjudicators and claimant  representatives believe offer promise for improving the consistency of  DDS and ALJ decisions. However, past difficulties in improving the  process, as well as stakeholder concerns about limited resources and  other obstacles, indicate that some difficulties may arise in the  development and implementation of SSA’s new proposal.
subsections: section_title: The Agency and Most Stakeholders Viewed Several Aspects of the New Proposal as Offering Promise for Improving Consistency
paragraphs: SSA told us that several aspects of the new proposal may improve the  consistency of decisions, and although opinions varied among stakeholder  groups, most thought the following four proposed changes have the  potential to improve the consistency of decisions between adjudication  levels: (1) requiring state adjudicators to more fully develop and document  their decisions, (2) centralizing the agency’s approach to quality control,  (3) providing both adjudication levels with equal access to more  centralized medical expertise, and (4) requiring ALJs to address agency  reports that either recommend denying the claim or outline the evidence  needed to fully support the claim.
Representatives from most stakeholder groups with whom we spoke told  us that having state adjudicators more fully develop and document their  decisions may help to improve the consistency of DDS and ALJ decisions.  Specifically, stakeholders said that more developed decisions may provide  ALJs with a better understanding of the DDS decision and enable them to  more fully consider this information when evaluating a case. According to  the agency and stakeholders, this change may contribute to a more  consistent interpretation and application of SSA’s decision-making  criteria. They also mentioned that well-developed decisions by DDS  examiners could assist SSA in holding adjudicators accountable for case  development and decisions, such as enabling quality reviewers to more  effectively assess the appropriateness of the DDSs decisions. Unlike SSA’s  earlier attempt at more fully developing decisions as part of process  unification, SSA plans to incorporate a reviewing official into the process  whose assessment of all appealed DDS decisions can provide feedback on  the extent to which cases are being fully developed.
In addition, the agency and many stakeholders told us that they believe  centralizing the agency’s quality control system may help resolve some  problems contributing to inconsistent decisions between the two levels.  For example, they believed that it may help ensure a more consistent  review of cases across the country and between adjudication levels.  According to both stakeholders and other experts within and outside of  SSA (including SSA’s Deputy Commissioner of Disability and Income  Security and a consulting group that reviewed SSA’s quality assurance  system), the current quality control and case review process encourages  adjudicators at the initial level to inappropriately deny cases, while  encouraging adjudicators at the hearings level to inappropriately allow  cases. Specifically, by overemphasizing a review of DDS allowances to  help control the cost of benefits, the agency has unintentionally  encouraged DDS examiners to deny cases. Conversely, SSA’s review of  ALJ decisions consists mostly of SSA’s Appeals Council reviewing cases  denied by ALJs, thereby providing an incentive for ALJs to allow cases. By  centralizing the quality control system and making other changes to the  process, SSA believes that it can remove the current incentives that  contribute to inconsistency.
The third proposed change that the agency and most stakeholder groups  believe may improve consistency is SSA’s plan to provide both  adjudication levels with equal access to more centralized medical experts,  organized by clinical specialty. Although located in the regions, these  experts should be able to review cases from across the country with the  successful completion of SSA’s AeDib initiative—an electronic folder  initiative for exchanging case information currently being implemented by  SSA. By making experts in a range of specialties available to assist both  levels of adjudicators in their decision making, SSA and stakeholders  believe that adjudicators could more consistently apply SSA’s decision- making criteria, in addition to acquiring better medical evidence.
Finally, the agency and most stakeholder groups told us that the  requirement to have an ALJ’s decision address the recommendations from  a reviewing official’s report to either deny or more fully develop the claim  may increase consistency between levels. Under the new proposal, SSA  plans to introduce a reviewing official into the process to evaluate all  appealed DDS claims. The official will allow claims that meet SSA’s  definition of disability and, for the remaining claims, will develop a report  that either (1) contains reasons for denying the claim or (2) outlines the  evidence needed to fully support the claim. The ALJ’s decision must  address issues raised in the reviewing official’s report. Stakeholders  believed that this change could, as intended by SSA, hold adjudicators  more accountable for their decisions and provide adjudicators with  feedback on the reasons decisions tend to differ between levels to improve  the quality and consistency of their decisions.
Although there was less agreement among stakeholder groups on the  potential effect that other aspects of the new proposal may have on the  consistency of decisions, some groups thought that other changes could  result in improved consistency between DDS and ALJ decisions. For  example, the Social Security Advisory Board and two groups representing  the DDSs thought that the proposed in-line quality control, if implemented  effectively at all levels, could have a positive impact on consistency by  ensuring that adjudicators adhere to the rulings and regulations  throughout the decision-making process. One stakeholder group added  that in-line quality control could also help the agency identify problem  areas, including areas in which policy is applied inconsistently or where  more training is needed.
section_title: Resource and Other Constraints May Limit SSA’s Ability to Successfully Implement Some Changes in the New Proposal
paragraphs: According to stakeholder groups—and based on SSA’s prior experience  with making significant changes to its claims process—insufficient  resources and other obstacles may prove to be major challenges for the  agency in developing and implementing aspects of its new proposal. For  example, experience with the process unification initiative has shown that  limited state resources have hindered the agency’s ability to have state  adjudicators fully document decisions. To address this issue, SSA plans to  reduce the states’ workloads by decreasing the number of claims to be  decided by the DDSs. Specifically, SSA expects that establishing regional  expert review units to make quick decisions for claimants who are  obviously disabled will substantially decrease the states’ workloads.  However, SSA has not developed and provided stakeholders with  estimates of the administrative cost for more fully documenting decisions  and other planned changes, and stakeholder groups were not convinced  that the reduction in claims was sufficient to offset resources needed to  fully document their decisions. Although the agency has had some recent  success in increasing its 2004 administrative budget, and is confident that  it will be successful in acquiring the resources it needs to implement the  proposal, the significance of stakeholders’ concerns about funding cannot  be assessed until SSA fully develops its proposal and associated cost  estimates.
Experience has also shown that another proposed change, developing a  centralized quality control system for both adjudication levels, could be a  major challenge for the agency. In 1994, SSA began efforts to create a  unified and comprehensive quality control system as part of its redesign  efforts, but made little progress, in part because of considerable  disagreement among internal and external stakeholders on how to  accomplish this difficult objective. To get external assistance in  developing an effective quality assurance system, SSA contracted with an  independent consulting firm to assess SSA’s quality assurance practices  used in the disability claims process. In 2001, concluding that SSA could  achieve its quality objectives for the disability program only by adopting a  broad, modern view of quality management, the consulting firm  recommended SSA abandon its current system and design a new quality  management system focused on building quality into the process. The  agency agreed that it was appropriate to transform the existing quality  assurance system and established an executive work group to decide a  future course of action. The agency is working with another consulting  group to further develop the changes recently proposed by the  Commissioner. However, after 10 years of efforts to develop a more  unified quality review system, SSA has not yet formulated changes to its  quality review system, beyond the brief and general descriptions provided  in the Commissioner’s new proposal.
Other obstacles also add to the complexity and difficulty of implementing  the proposal. For example, stakeholder groups have raised concerns about  SSA’s ability to successfully implement its proposed change to provide  equal access for all adjudicators to more centralized medical expertise by  removing medical expertise from the state DDSs and providing expertise  in regional offices. Stakeholder groups were concerned that SSA would  not be able to attract and retain sufficient medical experts to meet the  agency’s needs. They told us that states are currently experiencing  problems attracting medical experts because SSA’s compensation rates  are too low. State adjudicators, who currently work with medical experts  directly at DDS offices, were also concerned that removing these experts  and placing experts in SSA regional offices would impair the states’  effectiveness and efficiency. By placing experts in regional offices, state  disability examiners would no longer have on-site access to these experts  who help facilitate the states’ adjudication of claims and provide on-the- job training and mentoring to DDS examiners.
Stakeholders have also raised questions about SSA’s ability to ensure that  ALJs’ decisions fully respond to the reviewing officials’ reports and the  ultimate effectiveness of this change. Stakeholder groups representing  ALJs and claimant representatives believed that the requirements may  have the potential to impinge on an ALJ’s legal responsibility to ensure a  claimant receives a fair hearing and an independent decision. Other groups  have raised concerns about SSA’s ability to ensure that ALJs will  adequately address recommendations in the reviewing officials’ reports to  help ensure that this requirement leads to more consistent decisions.  Although these concerns have been raised, the Commissioner has clearly  stated that the intent of the proposal is to improve service to claimants,  including providing fair and accurate decisions, and that changes will not  impinge on the independence of ALJs.
In addition, several stakeholder groups also told us that staffing the new  reviewing official positions with attorneys, as SSA intends to do, would be  expensive. To the extent that SSA has difficulty filling these positions, the  agency could create a slowdown or bottleneck in the process that could  increase the time claimants must wait for a decision. Furthermore,  according to one stakeholder group, SSA’s new quality assurance process  will need to ensure that this new position does not create another source  of inconsistent interpretation and application of SSA’s decision-making  criteria.
Several groups representing hearings level adjudicators and claimant  representatives were also concerned about other aspects of the  Commissioner’s new proposal, such as the proposed elimination of the  Appeals Council and the claimants’ loss of the right to appeal an ALJ  decision to the council. The Appeals Council currently reviews about  100,000 appealed ALJ decisions annually. For these claims, the council  provides an additional appellate step for addressing claimants’ objections  to the ALJs’ decisions, reviewing new medical information on the claims  and reducing the number of claims appealed directly to the federal courts.  According to one stakeholder group, the council also performs other  important functions, such as reviewing claims for surviving children or  spouses of workers who were insured under the disability insurance and  retirement program. The council also reviews cases remanded from  federal courts. This stakeholder group also told us that as SSA refines its  proposal it will need to articulate how all of the council’s functions will be  handled under the new process.
Adding to uncertainties about the proposal’s success is its dependence on  the successful development and implementation of the AeDib system—a  highly complex and as yet unproven system using electronic folders to  share information with all entities involved in disability determinations.  SSA does not plan to implement its newly proposed changes before it has  completed a national rollout of its electronic disability system, scheduled  to be completed by October 2005. The new electronic disability system  represents an important step toward a paperless and more efficient  sharing of information by multiple partners involved in the disability  claims process, including SSA and state officials, as well as physicians and  other members of the medical community who provide needed medical  evidence. SSA also expects this new system to provide critical  management information for analyzing and reducing inconsistencies in  disability decisions. As we previously reported, SSA has made progress  developing the new system. However, its approach involves risks that  could jeopardize the agency’s successful transition to an electronic  disability claims process. For example, SSA recently began a national  rollout of the electronic disability system without fully evaluating pilot test  results or ensuring the resolution of all critical problems. Skipping such  important steps in development and implementation leaves the new  system vulnerable to problems in its performance and reliability. In  addition, problems with implementation of this system could delay the  implementation of SSA’s new proposal.
SSA recognizes that transforming its massive and complex disability  programs and achieving the benefits envisioned by the Commissioner will  be a challenging undertaking. The agency is refining its proposal and, as  part of this process, is actively seeking input from stakeholder groups. The  Commissioner and her staff have met directly with stakeholder groups to  understand and begin to address their concerns. As the agency refines its  proposal, the significance of both stakeholder concerns and previous  problems SSA has experienced improving its programs should become  clearer.
section_title: Conclusions
paragraphs: When SSA’s Commissioner announced her new proposal to overhaul the  disability programs, the agency acknowledged the importance of making  similar decisions on similar cases and making the right decision as early in  the process as possible. SSA has good cause to focus on the consistency of  decisions between adjudication levels. Incorrect denials at the initial level  that are appealed increase both the time claimants must wait for a  decision and the cost of deciding cases. Incorrect denials that are not  appealed may leave needy individuals without a financial or medical safety  net. Conversely, incorrect allowances at any adjudication level could  substantially increase the cost of providing disability benefits.
While the agency has made some effort to assess the inconsistency in  decisions between levels, its efforts have not provided the agency with a  clear understanding of the extent and leading causes of possible  inconsistencies in the interpretation and application of disability guidance.
For example, SSA’s assessment of ALJ error-prone allowances has not  proven to be effective at identifying new areas and causes of  inconsistency. SSA also has not updated its more effective approach of  analyzing its Disability Hearings Quality Review data to identify problem  areas and help improve its understanding of the factors that may be  contributing to inconsistency. Further, SSA’s analysis lacked sophisticated  statistical techniques and in-depth analysis of cases by adjudicators at  both levels, which together would have allowed SSA to better identify and  address the areas and leading causes of inconsistency. Moreover, by not  having examiners and medical consultants perform a complete review of  all relevant information before an ALJ, SSA has limited its ability to  understand the areas and causes of possible inconsistency.
Without better information on the areas and causes of possible  inconsistency, the agency cannot ensure that the Commissioner’s new  proposal will help to resolve this complex and long-standing concern. By  taking immediate actions to improve its understanding of the leading  causes of possible inconsistency in decisions, the agency will have  information needed to evaluate and possibly refine its new proposal,  including its plans to build an effective quality assurance system that can  both detect and prevent inconsistencies in decisions. This information will  help the agency to target its limited resources and take decisive steps to  build a claims process that provides claimants with the accurate,  consistent, and timely decisions they deserve, as envisioned in the  Commissioner’s proposal.
section_title: Recommendations
paragraphs: To move successfully forward with agency efforts to make more  consistent decisions, including efforts incorporated in the Commissioner’s  proposal for an improved disability claims process and quality assurance  system, we recommend that SSA quickly expand its assessment of the  areas and causes of inconsistency in decisions between adjudication  levels. In doing so, SSA should consider making near-term and cost- effective enhancements to its current approach for assessing the  consistency of decisions, including:  1.  Reestablish ongoing analyses of case characteristics as part of its  biennial case review, in line with efforts undertaken for the review  report published in 1994.  2.  Perform more sophisticated multivariate analysis on the biennial  case review data in order to pinpoint the most significant case  characteristics influencing allowance decisions and to distinguish  factors that might be contributing either appropriately or  inappropriately to allowance decisions.  3.  Expand the biennial case review by requiring disability examiners  and medical consultants to review the hearing tapes to ensure that  reviewers have the complete case before them (including the types  and sources of testimonial evidence provided during the hearings)  when evaluating the ALJs’ decisions.  4.  Have adjudicators and reviewers from each level study cases in  depth to help pinpoint the causes of inconsistency, once potential  areas of inconsistency between levels are identified.  5.  Publish the methods and findings of all analyses, to keep internal  and external stakeholders aware of the agency’s efforts to assess  consistency and demonstrate improvement over time.  6.  Use the information from these improved analyses to develop a  more focused and effective strategy for ensuring uniform  application of SSA’s guidance and to improve the consistency of  decisions. To accomplish this, SSA should clarify guidance for  making disability decisions and develop mandatory training for  adjudicators on issues identified as contributing to inconsistency.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to SSA for comment.  SSA expressed  several reservations about the recommendations, findings, and  conclusions of our report. Primarily, SSA took issue with: (1) our  characterization of the agency’s progress over the past several years in  analyzing and reducing the inconsistency of decisions, (2) our  recommendation that the agency incorporate multivariate analysis into its  assessments, and (3) our finding that the agency has not acted on the  results of its reviews of decisions. SSA indicated that it would reevaluate  our recommendations as the design of its Commissioner’s new approach  to disability decision making evolves. However, the agency did agree to  pilot one recommendation—that quality reviewers assess hearing tapes  when evaluating the ALJs’ decisions—as part of a quality review.
One of SSA’s main concerns was that our report did not fully discuss the  progress SSA had achieved in analyzing and reducing the inconsistency in  decision making between adjudication levels. For example, SSA  commented that our report dismissed the 21-percentage point increase in  the quality reviewers’ support rate of ALJ decisions, conducted as part of  SSA’s biennial case reviews over the last 10 years. SSA also pointed to  findings from its ALJ peer reviews as additional evidence that the quality  and consistency of SSA’s decisions had improved. In addition, SSA  asserted that its comparison of the relative proportion of allowances at the  DDS and ALJ levels, along with high accuracy rates, indicated that  adjudicators were making the right decisions sooner in the process—a  goal of both process unification and the Commissioner’s new disability  approach. Although our report incorporates results from the analyses  cited by SSA, our conclusion about the improvement in consistency  between levels is not as optimistic as SSA’s because of weaknesses in  SSA’s assessments. As we reported, SSA’s analysis of the quality reviewers’  assessment of ALJ cases has been limited for 10 years to calculating ALJ  support rates. SSA has not used available data to determine the potential  areas of inconsistency between levels or the extent to which changes in  the ALJ support rate is related to improvements in consistency of  decisions between adjudication levels. SSA’s assessment also lacks a  reliable method for determining whether DDS decisions are more  consistent with ALJ decisions, for example, by having ALJs regularly  review a statistical sample of DDS decisions. Lastly, as we pointed out,  changes in the proportion of overall allowances made by the DDS and ALJ  levels cannot serve as a reliable indicator for measuring the consistency of  decisions between levels, because many factors can affect these  proportions, such as significant fluctuations in the number of decisions  made at each adjudication level.
SSA also expressed its reservations about the benefits of multivariate  analysis in its evaluation of decision making. SSA asserted that its analyses  over the past 10 years have provided the agency with a solid  understanding of how certain variables influence disability decision  making and that the multivariate analyses we recommended would not  identify the causes and effects of inconsistent decision making at different  levels of this complex process. We agree with SSA that the disability  decision-making process is complex and that multivariate analysis alone  cannot establish all the causes and effects of inconsistent decision making.   However, because multivariate analysis takes into account the influence of  a number of relevant variables for each decision, this analytical technique  can provide a more accurate understanding of areas and causes of  inconsistency in decisions than methods previously employed by SSA.   Such analyses, followed by in-depth case studies by adjudicators at both  levels, which we also recommended, would bring SSA closer to  understanding and resolving the inconsistency of decisions between  adjudication levels. Therefore, we continue to believe that by performing  the analyses we recommend, the agency will have a better understanding  of the extent and causes of inconsistency, and that SSA’s Commissioner  should quickly implement our recommendations to ensure that her new  approach effectively addresses the consistency of decisions between  adjudication levels.
Finally, SSA disagreed with our finding that it has not acted on the results  of its reviews of decisions. SSA noted that it has made changes to address  training needs that have been identified by its reviews. Specifically, SSA  indicated that it has provided a series of interactive video training (IVT)  sessions focusing on problematic areas noted in the ALJ peer review  reports.  We acknowledge that SSA has conducted ALJ peer reviews and  used findings from its reviews to develop and provide training to ALJs.   However, we did not include these findings in our report, because our  objectives were limited to reporting efforts undertaken by SSA to assess or  improve the consistency of decisions between adjudication levels or to  implement its process unification initiative. SSA’s ALJ peer review is  conducted to identify problems with the quality of ALJ hearing process  and decisions, not to identify problems with the inconsistency of decisions  between levels. Conversely, our report included information on SSA’s ALJ  pre-effectuation review, because it was part of SSA’s process unification  initiative. According to information provided to us by SSA during our  audit, although this review was intended to help identify policy and  training areas that were associated with inconsistent decisions between  adjudication levels, it was not effective at identifying any new areas to be  pursued by the agency. This finding, along with those provided throughout  the report, supports our recommendations to SSA that the agency perform  additional analysis to determine the causes of potential inconsistency  between adjudication levels and to clarify guidance and provide  mandatory training to address any identified causes.
In addition, SSA provided several other general and technical comments  about the draft report. These additional comments, as well as our response  to them, are provided in appendix II.
Copies of this report are being sent to the Commissioner of SSA,  appropriate congressional committees, and other interested parties.  The  report is also available at no charge on GAO’s Web site at   http://www.gao.gov. If you have any questions about this report, please  contact me at (202) 512-7215. Other contacts and staff acknowledgments  are listed in appendix III.
section_title: Appendix I: Excerpt of SSA’s Testimony Announcing Its New Proposal to Improve Its Disability Decision-Making Process
paragraphs: In designing my approach to improve the overall disability determination  process, I was guided by three questions the President posed during our  first meeting to discuss the disability programs.
Why does it take so long to make a disability decision?
Why can’t people who are obviously disabled get a decision  immediately?
Why would anyone want to go back to work after going through such a  long process to receive benefits?
I realized that designing an approach to fully address the central and  important issues raised by the President required a focus on two over- arching operational goals: (1) to make the right decision as early in the  process as possible; and (2) to foster return to work at all stages of the  process. I also decided to focus on improvements that could be  effectuated by regulation and to ensure that no SSA employee would be  adversely affected by my approach. My reference to SSA employees  includes State Disability Determination Service employees and  Administrative Law Judges (ALJs).
As I developed my approach for improvement, I met with and talked to  many people—SSA employees and other interested organizations,  individually and in small and large groups—to listen to their concerns  about the current process at both the initial and appeals levels and their  recommendations for improvement. I became convinced that  improvements must be looked at from a system-wide perspective and, to  be successful, perspectives from all parts of the system must be  considered. I believe an open and collaborative process is critically  important to the development of disability process improvements. To that  end, members of my staff and I visited our regional offices, field offices,  hearing offices, and State Disability Determination Services, and private  disability insurers to identify and discuss possible improvements to the  current process.
Finally, a number of organizations provided written recommendations for  changing the disability process. Most recently, the Social Security  Advisory Board issued a report prepared by outside experts making  recommendations for process change. My approach for changing the  disability process was developed after a careful review of these  discussions and written recommendations. As we move ahead, I look  forward to working within the Administration and with Congress, as well  as interested organizations and advocacy groups. I would now like to  highlight some of the major and recurring recommendations made by  these various parties.
The need for additional resources to eliminate the backlog and reduce the  lengthy processing time was a common theme. This important issue is  being addressed through my Service Delivery Plan, starting with the  President’s FY 2004 budget submission which is currently before  Congress. Another important and often heard concern was the necessity of  improving the quality of the administrative record. DDSs expressed  concerns about receiving incomplete applications from the field office;  ALJs expressed concerns about the quality of the adjudicated record they  receive and emphasized the extensive pre-hearing work required to  thoroughly and adequately present the case for their consideration. In  addition, the number of remands by the Appeals Council and the Federal  Courts make clear the need for fully documenting the administrative  hearing record.
Applying policy consistently in terms of: 1) the DDS decision and ALJ  decision; 2) variations among state DDSs; and 3) variations among  individual ALJs—was of great concern. Concerns related to the  effectiveness of the existing regional quality control reviews and ALJ peer  review were also expressed. Staff from the Judicial Conference expressed  strong concern that the process assure quality prior to the appeal of cases  to the Federal Courts.
ALJs and claimant advocacy and claimant representative organizations  strongly recommended retaining the de novo hearing before an ALJ.  Department of Justice litigators and the Judicial Conference stressed the  importance of timely case retrieval, transcription, and transmission. Early  screening and analysis of cases to make expedited decisions for clear  cases of disability was emphasized time and again as was the need to  remove barriers to returning to work.
My approach for disability process improvement is designed to address  these concerns. It incorporates some of the significant features of the  current disability process. For example, initial claims for disability will  continue to be handled by SSA’s field offices. The State Disability  Determination Services will continue to adjudicate claims for benefits, and  Administrative Law Judges will continue to conduct hearings and issue  decisions. My approach envisions some significant differences.
I intend to propose a quick decision step at the very earliest stages of the  claims process for people who are obviously disabled. Cases will be sorted  based on disabling conditions for early identification and expedited action.
Examples of such claimants would be those with ALS, aggressive cancers,  and end-stage renal disease. Once a disability claim has been completed at  an SSA field office, these Quick Decision claims would be adjudicated in  Regional Expert Review Units across the country, without going to a State  Disability Determination Service. This approach would have the two-fold  benefit of allowing the claimant to receive a decision as soon as possible,  and allowing the State DDSs to devote resources to more complex claims.
Centralized medical expertise within the Regional Expert Review Units  would be available to disability decision makers at all levels, including the  DDSs and the Office of Hearings and Appeals (OHA). These units would be  organized around clinical specialties such as musculoskeletal,  neurological, cardiac, and psychiatric. Most of these units would be  established in SSA’s regional offices.
The initial claims not adjudicated through the Quick Decision process  would be decided by the DDSs. However, I would also propose some  changes in the initial claims process that would require changes in the way  DDSs are operating. An in-line quality review process managed by the  DDSs and a centralized quality control unit would replace the current SSA  quality control system. I believe a shift to in-line quality review would  provide greater opportunities for identifying problem areas and  implementing corrective actions and related training. The Disability  Prototype would be terminated and the DDS Reconsideration step would  be eliminated. Medical expertise would be provided to the DDSs by the  Regional Expert Review units that I described earlier.
State DDS examiners would be required to fully document and explain the  basis for their determination. More complete documentation should result  in more accurate initial decisions. The increased time required to  accomplish this would be supported by redirecting DDS resources freed  up by the Quick Decision cases being handled by the expert units, the  elimination of the Reconsideration step, and the shift in medical expertise  responsibilities to the regional units.
A Reviewing Official (RO) position would be created to evaluate claims at  the next stage of the process. If a claimant files a request for review of the  DDS determination, the claim would be reviewed by an SSA Reviewing  Official. The RO, who would be an attorney, would be authorized to issue  an allowance decision or to concur in the DDS denial of the claim. If the  claim is not allowed by the RO, the RO will prepare either a  Recommended Disallowance or a Pre-Hearing Report. A Recommended  Disallowance would be prepared if the RO believes that the evidence in  the record shows that the claimant is ineligible for benefits. It would set  forth in detail the reasons the claim should be denied. A Pre-Hearing  Report would be prepared if the RO believes that the evidence in the  record is insufficient to show that the claimant is eligible for benefits but  also fails to show that the claimant is ineligible for benefits. The report  would outline the evidence needed to fully support the claim. Disparity in  decisions at the DDS level has been a long-standing issue and the SSA  Reviewing Official and creation of Regional Expert Medical Units would  promote consistency of decisions at an earlier stage in the process.
If requested by a claimant whose claim has been denied by an RO, an ALJ  would conduct a de novo administrative hearing. The record would be  closed following the ALJ hearing. If, following the conclusion of the  hearing, the ALJ determines that a claim accompanied by a Recommended  Disallowance should be allowed, the ALJ would describe in detail in the  written opinion the basis for rejecting the RO’s Recommended  Disallowance. If, following the conclusion of the hearing, the ALJ  determines that a claim accompanied by a Pre-Hearing Report should be  allowed, the ALJ would describe the evidence gathered during the hearing  that responds to the description of the evidence needed to successfully  support the claim contained in the Pre-Hearing Report.
Because of the consistent finding that the Appeals Council review adds  processing time and generally supports the ALJ decision, the Appeals  Council stage of the current process would be eliminated. Quality control  for disability claims would be centralized with end-of-line reviews and ALJ  oversight. If an ALJ decision is not reviewed by the centralized quality  control staff, the decision of the ALJ will become a final agency action. If  the centralized quality control review disagrees with an allowance or  disallowance determination made by an ALJ, the claim would be referred  to an Oversight Panel for determination of the claim. The Oversight Panel  would consist of two Administrative Law Judges and one Administrative  Appeals Judge. If the Oversight Panel affirms the ALJ’s decision, it  becomes the final agency action. If the Panel reverses the ALJ’s decision,  the oversight Panel decision becomes the final agency action. As is  currently the case, claimants would be able to appeal any final agency  action to a Federal Court.
At the same time these changes are being implemented to improve the  process, we plan to conduct several demonstration projects aimed at  helping people with disabilities return to work. These projects would  support the President’s New Freedom Initiative and provide work  incentives and opportunities earlier in the process.
Early Intervention demonstration projects will provide medical and cash  benefits and employment supports to Disability Insurance (DI) applicants  who have impairments reasonably presumed to be disabling and elect to  pursue work rather than proceeding through the disability determination  process. Temporary Allowance demonstration projects will provide  immediate cash and medical benefits for a specified period (12-24 months)  to applicants who are highly likely to benefit from aggressive medical care.  Interim Medical Benefits demonstration projects will provide health  insurance coverage to certain applicants throughout the disability  determination process. Eligible applicants will be those without such  insurance whose medical condition is likely to improve with medical  treatment or where consistent, treating source evidence will be necessary  to enable SSA to make a benefit eligibility determination. Ongoing  Employment Supports to assist beneficiaries to obtain and sustain  employment will be tested, including a Benefit Offset demonstration to  test to effects of allowing DI beneficiaries to work without total loss of  benefits by reducing their monthly benefit $1 for every $2 of earnings  above a specified level and Ongoing Medical Benefits demonstration to  test the effects of providing ongoing health insurance coverage to  beneficiaries who wish to work but have no other affordable access to  health insurance.
I believe these changes and demonstrations will address the major  concerns I highlighted earlier. I also believe they offer a number of  important improvements:    People who are obviously disabled will receive quick decisions.
Adjudicative accountability will be reinforced at every step in the  process.
Processing time will be reduced by at least 25%.
Decisional consistency and accuracy will be increased.
Barriers for those who can and want to work would be removed.
Describing my approach for improving the process is the first step of what  I believe must be—and will work to make—a collaborative process. I will  work within the Administration, with Congress, the State Disability  Determination Services and interested organizations and advocacy groups  before putting pen to paper to write regulations. As I said earlier, and I say  again that to be successful, perspectives from all parts of the system must  be considered.
Later today, I will conduct a briefing for Congressional staff of the Ways  and Means and Senate Finance Committees. I will also brief SSA and DDS  management. In addition, next week I will provide a video tape of the  management briefing describing my approach for improvement to all SSA  regional, field, and hearing offices, State Disability Determination Services,  and headquarters and regional office employees involved in the disability  program. Tomorrow, I will be conducting briefings for representatives of  SSA employee unions and interested organizations and advocacy groups,  and I will schedule meetings to provide an opportunity for those  representatives to express their views and provide assistance in working  through details, as the final package of process improvements is fully  developed.
I believe that if we work together, we will create a disability system that  responds to the challenge inherent in the President’s questions. We will  look beyond the status quo to the possibility of what can be. We will  achieve our ultimate goal of providing accurate, timely service for the  American people.
section_title: Appendix II: Comments from the Social Security Administration
subsections: section_title: GAO Comments
paragraphs: 1.  We maintain that our report fully and fairly describes SSA’s progress in  analyzing and addressing the underlying causes of inconsistent  decisions between state DDS examiners and ALJs. Our research  included an extensive review of agency documentation and interviews  with SSA officials, as well as stakeholder groups for adjudicators and  claimant representatives, to develop a complete understanding of the  agency’s efforts to assess and improve the consistency of decisions  between adjudication levels. Also, in agreement with our requestor, we  sought to expand the review to include SSA’s new approach to  improving its disability programs, so that we could provide the  Congress with an understanding of how SSA’s future plans may help to  address this issue.  2.  We provided information on the various reviews and analyses of  disability decisions to assess the consistency of decisions between  adjudication levels conducted by SSA over the last 10 years, but none  of these reviews have clearly identified the causes of inconsistency in  decisions between adjudication levels.
3.
Our report has not overlooked the data cited by SSA.  Nevertheless,  our conclusion about the improvement in consistency between levels  indicated by the data is not as optimistic as SSA’s because of  weaknesses in SSA’s assessments. As we reported, for 10 years SSA’s  analysis of the quality reviewers’ assessment of ALJ cases has been  limited to calculating ALJ support rates. SSA has not used available  data to determine the potential areas of inconsistency between levels  nor the extent to which changes in the ALJ support rate are related to  improvements in consistency of decisions between adjudication levels.   SSA’s assessment only provides a general indication of overall changes  in consistency at one adjudication level.  4.  Our report recognizes that SSA’s disability decision-making process is  complex. Because of this complexity, we believe that multivariate  analysis is an appropriate assessment tool that would allow SSA to  assess the effect of multiple factors. In recommending this  sophisticated tool, we were careful not to imply that causes and effects  of inconsistent decision making can be established with certainty.  However, we believe that such an analysis will help SSA understand  the relative importance of the variety of factors that affect its decision- making process. After identifying areas of inconsistency, SSA can  target these areas with in-depth case analyses to pinpoint the causes of  inconsistency and develop a more effective strategy for addressing  inconsistency. On the basis of our review of SSA’s analyses to date, we  do not agree with the implications of SSA’s comments that it has a  solid understanding of how certain variables influence disability  decision making, and therefore does not need to conduct additional,  more sophisticated analyses.  5.  We agree with SSA that the proportion of allowances made at each  level can provide some insight into the allowance rate dynamic.  However, as we reported, we do not believe that it can serve as a  reliable indicator of the agency’s progress in achieving more consistent  decisions between the DDS and OHA levels. The allowance data  provided by SSA simply show that the relative proportion of  allowances made at the DDS level increased in comparison with the  OHA level, but SSA has not performed any additional analysis to show  that these changes have any relationship to improved consistency in  decision making between the two adjudication levels. Additional  analysis is needed because a myriad of factors, such as changes in the  economy, can affect allowance rates. Although SSA claims that over  this period of time the economy has been “relatively stable,” without  performing any additional analysis it cannot eliminate changes in the  economy or demographics of claimants as an influence on the  allowance rates at each level. In addition, SSA has not analyzed how  other factors, such as changes in productivity and total number of  decisions made at each level, may be influencing the allowance data.   6.  The allowance rate data provided by SSA in its comments is very  similar to that provided by SSA earlier to us and included in our report  in figure 2. The figures we reported for the proportion of allowances  made by the DDS and OHA levels for fiscal years 1997 and 1998 vary in  comparison with those provided by SSA by one percentage point. We  have not changed the figures in our report because we believe that  these slight differences simply reflect that we reported data based  upon fiscal, not calendar, years.  7.  In our report, our statements that SSA has not made changes as a  result of findings from its reviews were specifically related to SSA’s  ALJ pre-effectuation review. We included information on this review  because it was part of SSA’s process unification initiative and was  intended to identify policy and training areas associated with  inconsistent decisions between adjudication levels. During our review,  we were told by an SSA official that the ALJ pre-effectuation review  was not successful at identifying new areas of inconsistency to be  addressed by SSA. In its comments, SSA cites a review unrelated to  assessing the inconsistency of decisions between levels, the ALJ peer  review, to assert that it has used reviews to identify training issues to  improve the quality of decisions. The lack of success with the ALJ pre- effectuation review—along with other findings showing a limited  understanding of the cause of inconsistency—supports our  recommendations to SSA to perform additional analysis and to clarify  guidance and provide mandatory training to address any identified  causes of inconsistency between adjudication levels.  8.  We applaud SSA’s plans to use the electronic disability system to  capture critical management information to address decisional  variance or inconsistency, which could provide a wealth of useful  information for the agency. We have adjusted our report’s text to  reflect this additional purpose. We continue to believe that SSA should  not wait for the development of this system, but should proceed to  perform multivariate analysis, using available data from its biennial  case reviews, to start identifying areas of potential inconsistency  between adjudication levels.  9.  We applaud SSA’s deep commitment to improving the disability  decision-making process, but believe that additional efforts to  understand the causes of potential inconsistencies in decision making  would help to inform the design of the Commissioner’s new approach  and should, therefore, be undertaken immediately.  10.  We generally agree with the technical comments provided and changed  the text accordingly.
section_title: Appendix III: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: In addition to the individuals mentioned above, the following staff  members made major contributions to this report: Michael Morris, Corinna  Nicolaou, Walter Vance, and Rebecca Woiwode. Douglas Sloane provided  assistance with methodological issues, and Daniel Schwimer provided  legal support.