Social Security Disability: Additional Measures and Evaluation Needed to Enhance Accuracy and Consistency of Hearings Decisions
Released: 2018-01-08T00:00:00

Letter

Background

The Social Security Administration’s (SSA) Disability Insurance (DI) and Supplemental Security Income (SSI) programs are the two largest federal programs providing cash assistance to people with disabilities.

The DI program, established in 1956, provides monthly payments to working-age adults (and their dependents or survivors) who are unable to work due to a long-term disability.

The SSI program, established in 1972, is a means-tested income assistance program that provides monthly payments to adults or children who are aged, blind, or have other disabilities and whose income and assets fall below a certain level.

Individuals with low incomes and assets who also have a sufficient work history may qualify for the DI and SSI programs concurrently. In this case, the individual’s SSI payment is generally offset by the amount of the DI payment.

In fiscal year 2016, according to SSA, about 10.8 million disabled workers and their family members received about $143 billion in DI benefits, and an estimated 8.2 million individuals received almost $59 billion in SSI benefits (of those, 2.6 million received SSI in addition to DI or Old-Age and Survivors benefits).

Disability Criteria

Although DI and SSI have different purposes and target populations, the disability criteria for adults are the same for both programs. To be considered eligible for either program as an adult, a person must have a medically determinable physical or mental impairment that (1) has lasted or is expected to last for at least a continuous period of 1 year or result in death, and (2) prevents them from engaging in any substantial gainful activity (SGA).

The disability decision-making process includes five sequential steps (see fig. 1). First, SSA determines if a claimant is working and screens out (denies) claimants who earn over a specified amount. Second, SSA determines whether the claimant has an impairment severe enough to significantly limit his or her ability to do basic work activities and expected to last more than 12 months or result in death, and denies claimants who do not meet these criteria. At the third step, SSA determines whether a claimant’s impairment meets or is equivalent to an impairment listed in SSA’s Listings of Impairments. If a claimant “meets” or “equals” one of the listed impairments, they are allowed benefits. If not, SSA proceeds to the last two steps and assesses whether a claimant, given their impairment, can do their past work (step four) or other work that exists in significant numbers in the national economy (step five).

Over time, more of SSA’s disability decisions have been made at the last two steps in the process, which require more judicial discretion than decisions made at steps 1 through 3, according to SSA. In 2000, 29 percent of decisions were made at steps 4 and 5, according to an SSA report. By 2014, nearly half—49 percent—of all decisions were made at these steps.

Disability Application and Appeals Process

To apply for benefits, a claimant must file an application online, by telephone, or mail, or in person at a local Social Security office. If field office staff determine that the claimant meets the nonmedical eligibility criteria, they forward the claim to the appropriate state Disability Determination Services (DDS) office. DDS staff—generally a team comprised of disability examiners and medical consultants—review medical and other evidence provided by the claimant, obtaining additional evidence as needed, and make the initial disability determination. In fiscal year 2016, SSA received more than 2.5 million disability claims.

If the claimant is not satisfied with this determination, in most states he or she may request a reconsideration of the decision within the same DDS office. If the claimant is dissatisfied with the reconsideration, he or she may request a hearing before an administrative law judge (ALJ). In one of several initiatives to improve the disability determination process, SSA has eliminated the reconsideration step of the process in 10 states, allowing the claimant to appeal the initial decision directly to an ALJ. In fiscal year 2016, claimants appealed more than 698,000 decisions to the hearings level, and SSA issued more than 637,000 dispositions (including allowances, denials, and dismissals). (See fig. 2).

Within SSA’s Office of Disability Adjudication and Review (ODAR), there are approximately 1,500 ALJs who are located in 166 hearing offices across the country, as well as at five National Hearing Centers. In general, cases are randomly assigned to ALJs within the area each hearing office serves, in the order in which the requests for a hearing are received. The ALJ reviews the claimant’s file, including any additional evidence the claimant submitted after the initial determination, and generally conducts a hearing. At the hearing, the ALJ may hear testimony from the claimant, medical experts on the claimant’s medical condition, and vocational experts regarding the claimant’s past work and jobs currently available in significant numbers in the national economy. The majority of claimants are represented at these hearings by an attorney or nonattorney representative, such as a professional disability representative, relative, or social worker.

If the claimant is not satisfied with the ALJ decision, he or she may request a review by SSA’s Appeals Council, which is the final administrative appeal within SSA. The Appeals Council may grant, deny, or dismiss a request for review. If it agrees to review the case, the Appeals Council may uphold, modify, or reverse the ALJ’s decision, or it may remand the case back to the ALJ to hold another hearing and issue a new decision. In fiscal year 2016, the Appeals Council reviewed more than 154,000 ALJ decisions and remanded 13 percent of them.

Hearings Backlogs and Processing Times in Recent Years

Hearings-level backlogs and processing times have increased between fiscal years 2010 and 2016. The number of annual requests for a hearing before an ALJ peaked in fiscal 2011, and declined in each subsequent year, through fiscal year 2016. Despite this decline, SSA has not been able to keep pace with the demand, in terms of dispositions— the number of cases the agency decided or dismissed—in each of those years after 2010 (see figure 3). By the end of fiscal year 2016, SSA reported there were about 1.1 million pending cases. Average processing times for hearings-level decisions also increased during this same time period, from 426 days to 543 days.

During these years, the number of ALJs declined, along with the number of case dispositions per month. For example, SSA reported it employed 1,356 ALJs in fiscal year 2013, and these judges had an average of 48 case dispositions per month. In fiscal year 2015, 1,265 judges had an average of 44 case dispositions per month. Also during this time period, SSA reduced its reliance on senior attorney adjudicators (SAA) to make fully-favorable, on-the-record decisions (that is, decisions in which a hearing is not necessary because the documentary evidence alone supports a decision that is fully favorable to the claimant).

According to SSA, its backlog will be eliminated when the national average processing time for a hearing decision is 270 days. In January 2016, SSA issued a plan to achieve this goal by the end of fiscal year 2020. However, in its fiscal year 2018 performance plan, SSA set a goal for processing hearings decisions in 600 days (up from a target of 485 days in fiscal year 2010). SSA reported that the increase in average processing times is due to the increase in the number of pending cases. Since SSA generally processes cases in the order in which they are received, they focus on the oldest cases first, which increases the average processing time for closed cases.

Requirements for Hiring, Overseeing, and Disciplining SSA Administrative Law Judges

The role of ALJ was created by the Administrative Procedure Act, which was enacted in 1946 to ensure fairness and due process in federal agency proceedings involving rulemaking and adjudications. ALJs serve in a number of executive branch agencies, although SSA employs the vast majority. ALJs preside and make decisions at formal adjudicatory proceedings.

One of the primary goals behind the creation of the ALJ position is to ensure that judges can conduct hearings free from influence or coercion from the agency. Although ALJs are hired by and serve as employees of executive branch agencies like SSA, the Office of Personnel Management (OPM) is responsible for the initial examination, certification for selection, and implementation of the three levels of basic pay of ALJs. As part of its responsibilities, OPM sets the minimum qualifications for ALJs, which are that they generally must be licensed attorneys with a minimum of 7 years of experience in litigation and/or administrative law and pass the competitive examination.

The Administrative Procedure Act gave ALJs qualified decisional independence, with some oversight from agencies. Decisional independence means that ALJs can make decisions independently. Federal law also excludes ALJs from performance evaluations and generally requires that disciplinary actions against ALJs be for good cause established and determined by the Merit Systems Protection Board (MSPB). While ALJs have qualified decisional independence, they must follow their agency’s policies and procedures when making decisions. The Administrative Procedure Act also authorized agencies to review ALJ decisions. If SSA determines that an ALJ has not followed its policies and procedures, it can issue a directive to the ALJ to comply and, if that is unsuccessful, bring a disciplinary action before the MSPB.

Allowance Rates Vary Across Judges, Even for Typical Claims

Allowance Rates Have Varied Across Judges and Hearing Offices in Recent Years, Even After Holding Constant a Range of Factors Relevant to the Appeals Process

Allowance rates varied across administrative law judges from fiscal years 2007 through 2015. We defined the “allowance rate” for each judge as the number of claims in which a judge granted the claimant Disability Insurance (DI) and/or Supplemental Security Income (SSI) benefits divided by the total number of decisions issued by the judge (excluding claims that were dismissed). We analyzed about 3.3 million decisions made by administrative law judges on adult Social Security disability appeals over this period.

The average allowance rate across judges fell 15 percentage points over this period—from a peak of 70 percent in 2008 to 55 percent in 2015—but the range in allowance rates across judges remained fairly constant (see fig. 4). Specifically, the range—the difference between judges with high allowance rates (those at the 95th percentile) and judges with low allowance rates (at the 5th percentile)—was 55 percentage points over this period.

This variation in allowance rates persisted, but fell modestly over time, even when we used multivariate statistical methods to hold constant a variety of factors related to the disability appeals process. These factors included characteristics of claimants, judges, and hearing offices, as well as other factors such as the unemployment rates in a claimant’s state, that could otherwise explain differences in allowance rates. Specifically, for the years 2007 through 2015 combined, our analysis estimated that the allowance rate would vary by 46 percentage points for a typical claim, depending on the judge who heard the case. For example, we estimated that the allowance rate for a typical claim heard by a judge with low allowance rates would be 42 percent, compared to 88 percent for a judge with high allowance rates. This estimated range fell from 50 percentage points in 2007 to 45 percentage points in 2015 (see fig. 5). (Appendix I describes this statistical analysis in more detail.)

Allowance rates also varied across hearing offices during the same time period, but this variation was considerably smaller than the variation across judges in every year. The estimated range across the entire period was 19 percentage points across hearing offices (see fig. 6), compared to a 46 percentage-point estimated range across judges. Accounting for differences in allowance rates across offices ensured that the variation across judges did not reflect characteristics of their offices (such as the types or severity of disability claims received by their offices).

SSA officials noted that the variation in allowance rates we observed across judges was not surprising, nor was the modest narrowing in this range over time. Administrative law judges usually hear complex appeals that may not be clear-cut allowances or denials. As a result, according to SSA officials, given judges’ decisional independence, different judges could look at cases with similar fact patterns and circumstances and come to different conclusions. At the same time, officials also pointed to several factors potentially related to the modest narrowing in the range of allowance rates. First, they noted that SSA started conducting quality assurance reviews of a random sample of allowances in 2011— previously, such cases were not reviewed. In addition, they said that Social Security’s disability programs and administrative law judges were under increased public and Congressional scrutiny following a high-profile fraud case in 2011 involving a judge and an attorney representative. Further, officials said that the expanding use of electronic case files and data analytics within SSA made it possible for the agency to enhance monitoring of decision-making and share this information with judges. Finally, while SSA cannot direct judges to decide cases in a particular way, officials suggested that some judges may have “self-corrected” their approach to decision-making, given all of these factors.

Our multivariate analyses had some limitations, but it provides more information than simple comparisons in allowance rates across judges. For example, the SSA data we used for this analysis do not include a measure of the severity of a claimant’s impairment or their remaining ability to work, which could help explain why one claim with a particular impairment was allowed while another was denied. The data also do not include a standardized measure for the nature of claimants’ prior work (such as the skill level or extent of physical labor), which is also relevant for the disability decision. Nevertheless, our multivariate analysis enabled us to compare allowance rates across judges and hearing offices for typical claims. In addition, SSA’s practice of assigning cases randomly to judges makes it more likely that the remaining variation we found across judges reflects the unique effect of having a particular judge hear a case, rather than other factors. As a result, even though we could not account for all factors that could explain differences in allowance rates, random assignment increases the chances that such factors were similar across all of the cases heard by individual judges.

Numerous Factors, Particularly Those Representing SSA’s Disability Criteria, Are Associated with Variation in Allowance Rates

Although variation in allowance rates persisted across judges, even after controlling for certain factors, many of the factors we identified had meaningful associations with the chance that a claimant was allowed benefits. These factors represent criteria in SSA’s disability decision- making process, such as the claimant’s age, impairment, prior work, and education. We also identified factors that did not have such associations.

Certain claimant characteristics—such as older ages or certain impairments—were associated with higher allowance rates.

Age: Claimants’ chances of being allowed benefits increased with age, even holding constant other factors. For example, a 55-year-old claimant was allowed benefits at a rate 4.3 times higher than a typical 35-year-old claimant. This association is consistent with Social Security’s vocational guidelines, which are generally more lenient for older claimants. As part of SSA’s five-step process to determine eligibility for adult disability benefits, SSA uses a set of rules to evaluate how a claimant’s age, education, and work experience affect their remaining capacity for work. SSA’s criteria vary across four primary age groups—45-49, 50-54, 55-59, and 60 and older. The criteria are less stringent for claimants in older age groups than they are for younger claimants, because the rules assume that individuals at older ages may be less able to transition to other work.

Impairment: Certain impairments were also strongly associated with the chance of being allowed benefits (see fig. 7). For example, claimants with primary impairments recorded in SSA’s data of heart failure or multiple sclerosis were allowed benefits at rates 4.2 and 5 times higher, respectively, than typical claimants with asthma. From fiscal years 2007 through 2015, the allowance rates for claimants with heart failure or multiple sclerosis were 78 and 80 percent, respectively, compared to 44 percent for asthma.

Critical or terminal case: Claimants with critical or terminal cases were allowed benefits at a rate 1.4 times higher than a typical claimant without a critical or terminal case. Critical and terminal cases are cases that require special processing, such as a terminal illness or a veteran with a 100-percent permanent and total disability compensation rating.

Prior work: Claimants reporting shorter work histories (4 years or less in the last 15 years before applying for disability benefits) were allowed at a rate 0.8 times as high as a typical claimant with 10 or more years of work history. As expected, given the nature of the work requirements for the DI program, the association with prior work history was stronger for that program than for the SSI program.

College education: Claimants who reported having a college-level education or higher were approved at a slightly higher rate (1.1 times higher) than a typical claimant with a high-school education. SSA officials suggested that this association could be an indirect measure of the severity of a claimant’s impairment, a factor for which we did not have data. They said that individuals with higher levels of education often have higher incomes and, therefore, may be less likely to forego their income to apply for disability benefits, were it not for the severity of their disability.

Claim type: DI claimants were allowed at a rate 1.7 times higher than a typical SSI claimant. Across judges, the average allowance rate for DI claimants (67 percent) was higher than for SSI claimants (52 percent) from fiscal years 2007 through 2015, with the allowance rate for claimants applying concurrently for DI and SSI benefits falling in between (58 percent).

Other Participants in the Disability Appeals Process Claimants who had appointed a representative to present their case, or had a medical expert testify at their hearing, were associated with a greater chance of being allowed benefits, but the presence of a vocational expert had the opposite association.

Claimant representative: Similar to findings in our prior work, claimants who had a representative—either an attorney or a nonattorney representative—were allowed at a rate 2.9 times higher than a typical claimant with no representative. SSA officials stated that representatives may have a screening process for potential clients, and under SSA’s fee structure, representatives are paid only if the claimant is awarded benefits. As a result, representatives may tend to take cases they believe will be successful. Officials also stated that a representative can help the claimant by ensuring that the medical evidence and other records are fully developed and help the claimant present their case at a hearing. From fiscal years 2007 through 2015, most claimants (77 percent) had an attorney representative, and 12 percent had a nonattorney representative.

Expert testimony: Claimants whose hearings involved testimony from a medical expert were allowed at a rate 1.6 times higher than a typical claimant without a medical expert present. Medical experts include physicians, psychologists, and other types of medical professionals who provide impartial, expert opinion evidence for an ALJ to consider when making a decision about disability. SSA officials said that the association of medical experts with an increased chance of allowance is expected, given that judges are required to seek the testimony of a medical expert in certain cases, for example, when the judge is considering allowing benefits because the claimant’s impairment may be medically equivalent to one in SSA’s Listing of Impairments. In other cases, involving a medical expert is generally at the judge’s discretion. From fiscal years 2007 through 2015, 12 percent of decisions involved a medical expert.

The presence of a vocational expert had the opposite effect— claimants with a vocational expert testifying were allowed at a rate 0.8 times as high as claimants without a vocational expert testifying. Vocational experts provide objective, expert opinion evidence to the ALJ, primarily at the last two steps of the disability decision-making process where SSA considers whether claimants can do their prior work or transition to other work available in the national economy. Although involving a vocational expert is generally at a judge’s discretion, SSA officials said that they were not surprised by this result, because vocational experts are usually called upon at the final two steps in the disability decision-making process. At that point, claimants had already not been allowed benefits at an earlier step because their impairment(s) did not meet or were not equivalent to an impairment in SSA’s listings. From fiscal years 2007 through 2015, most hearings (85 percent) involved a vocational expert.

Judges with certain characteristics, such as those appointed in earlier years, were associated with a greater chance of allowing benefits.

Appointment cohort: A claimant whose claim was heard by a judge appointed between 1995 and 1999 was allowed at a rate 1.5 times higher than a typical claim heard by a judge appointed after 2010. SSA officials said that, since 2010, they have changed the way they train and mentor new judges, and introduced new tools to help provide a standardized decision-making template. As a result, SSA officials said, more recently hired ALJs may be more aware of agency policies and procedures.

Certain characteristics of hearing offices and other factors also were associated with higher chances of allowance. For example:

Hearing type: Claimants whose hearings were held in person were allowed at a slightly higher rate (1.1 times higher) than a typical claimant with a hearing conducted remotely using videoconference technology. This is equivalent to a 2.8 percentage-point higher probability of being allowed benefits for a claimant whose hearing was held in person, compared to an otherwise typical claimant whose hearing was conducted by videoconference. However, we did not seek to estimate the causal impact of videoconferences on allowance rates, and so did not design our analysis to account for all factors that could affect this relationship. Rather, we accounted for the use of videoconferences solely to further ensure that circumstances were similar across the judges and offices we analyzed. Expanding video service delivery is a key goal for SSA, including plans to partner with other agencies, such as the Department of Veterans Affairs, to increase the number of available video hearing sites beyond those already available at hearing offices and the five National Hearing Centers.

Year of decision: Claimants whose appeals were decided in earlier years were associated with a greater chance of being allowed benefits. While this trend is similar to the raw change over time shown in figure 4, our multivariate analysis showed that this change held even for claimants in similar circumstances. For example, claimants who received decisions in 2007 were allowed at a rate 2.0 times higher than a typical claim in 2015. This is consistent with other studies that have found trends of lower allowance rates in recent years.

Factors Not Associated with Differences in Allowance Rates Some factors were not meaningfully associated with allowance rates when holding other factors constant.

Workload measures: Workload and productivity measures at the hearing office and judge level were not meaningfully associated with allowance rates. This includes the annual percentage of cases that were backlogged (that is, awaiting a judge’s decision for more than 270 days) at each hearing office, as well as the annual number of dispositions (decisions plus dismissals) each judge issued. This may suggest that judges’ decisions to allow or deny cases are not significantly influenced by the number of cases before them, similar to findings in prior research.

Hearing office type: We found no meaningful differences in allowance rates between similar claims heard at one of SSA’s National Hearing Centers or a traditional hearing office, after holding constant other factors (including whether the hearing was held by videoconference). SSA has five National Hearing Centers, which hear cases from across the country by videoconference in order to reduce backlogs in certain hearing offices.

Economic characteristics: The unemployment and poverty rates in the claimant’s state at the time of the ALJ decision were not associated with allowance rates. Higher unemployment rates can result in increased applications for Social Security disability benefits because workers with impairments that could qualify them for the program who experience job loss may find it more difficult to become re-employed during periods of high unemployment and apply for benefits. However, the impact on allowance rates in the research we reviewed is mixed.

SSA’s Efforts to Monitor Accuracy and Consistency of Hearings-Level Decisions Lack Performance Measures and Have Not Been Evaluated

SSA Has Timeliness Measures, but Lacks Public Performance Measures for Accuracy and Consistency

SSA has employed a range of efforts to monitor the accuracy and consistency of hearings decisions, but it lacks performance measures to report publicly on these efforts. SSA’s current strategic plan includes an objective to “improve the quality, consistency, and timeliness” of its disability decisions; however, all of the hearings-level measures supporting this objective are related to timeliness. In a previous report, we developed nine attributes of performance goals and measures based on previously established GAO criteria, as well as relevant federal laws and performance management literature. One key attribute states that an agency’s suite of performance measures should be balanced to cover various priorities. In addition, each measure should cover a priority such as quality, timeliness, and cost of service. However, because SSA’s performance measures do not fully reflect its goals, the overall success of SSA’s efforts in this area may be limited.

SSA previously had performance measures related to hearings-level accuracy, which used data from ALJ peer reviews. These measures were discontinued in fiscal year 2009, when the ALJs conducting the reviews were reassigned to hearing cases. By comparison, SSA continues to have a measure for accuracy at the initial decision-making level (see table 1).

SSA officials stated that they have no plans to add new performance measures related to the accuracy and consistency of hearings decisions to the strategic plan. They said that while they collect and monitor a wide variety of workload and performance measures for day-to-day operations, they have to select a few, representative measures that are meaningful to stakeholders and represent agency-wide efforts to achieve its goals. They stated that the current performance measures meet these requirements.

Although SSA officials said the agency does not publicly report performance measures related to the accuracy and consistency of hearings decisions, they said that SSA uses internal performance measures related to hearings decisions. However, these internal measures to monitor quality and consistency of hearings decisions have limitations and are not shared with the public. Regional chief judges—who oversee the hearing offices and judges within each of SSA’s 10 regions— and others told us that they use a measure known as the “agree rate” to help monitor the quality of a judge’s decisions. This measure is based on the number of cases that have been appealed to the Appeals Council by the claimant or representative as a request for review. The agree rate reflects the percentage of cases in which the Appeals Council—the final level of appeals within SSA—concluded that the ALJ’s decisions were supported by substantial evidence and contained no error of law or abuse of discretion. However, the agree rate has some limitations. For example, as noted earlier, it does not reflect the accuracy of ALJ decisions that the claimant did not appeal. SSA’s Office of the Inspector General (OIG) found that this measure provided information on less than one-quarter of all ALJ dispositions and it is not representative of the ALJ’s entire workload because it is based only on Appeals Council reviews of appealed cases. In addition, a March 2017 SSA OIG report found that SSA has not maintained historical data on agree rates, limiting the agency’s ability to analyze agree rate trends.

SSA uses other internal measures to track consistency. For example, SSA developed an internal early monitoring system that tracks 22 metrics of ALJ performance to identify outliers. For example, three of these metrics (average number of dispositions a judge issues per day, agree rate, and allowance rate) have “alarm thresholds” to indicate when an ALJ’s metrics fall outside of a given threshold. Based on these findings, SSA may conduct a focused quality review (a type of quality assurance review) to ensure the judge’s decisions complied with SSA policies, or follow up with the regional chief judge to determine if additional policy guidance or training is needed.

Although these internal measures are helpful for management to monitor and improve accuracy and consistency, without sharing this or similar information publicly, SSA lacks accountability for improving the quality of hearings-level decisions. In addition, federal internal control standards state that management should externally communicate the necessary quality information to achieve objectives, including to external stakeholders such as Congress and the public. Further, given the persistent variation in allowance rates, SSA may be missing an opportunity to provide the public with information on the results of its efforts to improve the accuracy and consistency of disability decisions.

SSA Has Efforts to Monitor and Improve Accuracy and Consistency of Hearings- Level Decisions, but Quality Reviews May Overlap and Have Not Been Systematically Evaluated

SSA provides training and tools to all ALJs and initiates disciplinary actions where needed, as part of its efforts to monitor and improve accuracy and consistency. SSA also conducts multiple quality assurance reviews, but some of these reviews may overlap and SSA has not evaluated them.

Training, Tools, and Policy Guidance ALJs receive ongoing training and guidance from several sources, including through judicial trainings, mentoring, and policy memorandums. In 2006, SSA implemented a three-phase training program for new ALJs, which includes training on core competencies as well as a formal mentoring program in which new ALJs are paired with experienced ALJs for regular sessions over a nine-month period. Regional managers, judges, and stakeholders we spoke with had positive feedback on the training SSA provides to judges. For example, officials from one stakeholder group told us that they believe training had created more consistency in allowance rates. SSA’s chief judge also issues guidance memorandums to clarify policies related to the hearings process. For example, in July 2013, SSA issued a memorandum establishing expectations for the instructions judges provide to decision writers, who are SSA staff who prepare the draft decisions. SSA officials said that they issued the memorandum in response to an ALJ who was providing low- quality instructions to decision writers and SSA realized it had not provided formal guidance on the topic. In addition, ALJs also receive quarterly continuing education training and have a library of reference materials and on-demand video courses to use as needed.

SSA also uses internal metrics and provides electronic tools to judges to monitor and improve accuracy and consistency. Regional chief judges regularly review management information (MI) reports and develop strategies, such as recommending training, to address identified issues. Beginning in 2011, SSA established an electronic tool called “How MI Doing?”, which allows ALJs to compare their productivity and timeliness metrics to hearing office, regional, or national metrics. The tool also provides data on the agree rate for each judge as well as the hearing office, regional, and national agree rates. Using this tool, judges can also learn the reasons any prior decisions have been remanded, and access on-demand training pertaining to that reason. Regional chief judges we spoke with generally found “How MI Doing?” to be a helpful tool, although SSA does not track judges’ usage and has not formally evaluated its effectiveness. In addition, SSA established the electronic Bench Book (eBB), which is designed to assist users with documenting, analyzing, and making consistent and accurate decisions on hearings-level adult disability cases. However, the SSA OIG recently recommended that SSA evaluate eBB and determine whether to continue it. Regional chief judges we spoke with provided mixed feedback on the use of eBB and its usefulness for ALJs. In fiscal year 2016, nearly 500 ALJs (about one- third) used eBB. In June 2017, SSA officials said that while no formal evaluation of eBB was conducted, they recently received approval to proceed with plans to replace eBB with a similar tool as part of updates to SSA’s case management system.

SSA also addresses identified issues with the accuracy and consistency of hearings decisions by taking disciplinary actions, as needed. SSA can take non-disciplinary or disciplinary action to address performance concerns. Non-disciplinary actions include training and counseling (known as “collegial conversations”). Another non-disciplinary action is a written directive, which SSA can issue to individual judges to improve performance on workload, scheduling or policy compliance. From 2007 through 2016, SSA issued about 1,330 such directives. Nearly all (95 percent) were issued to improve timeliness, while about 2 percent were issued to improve policy compliance. If an ALJ’s conduct or performance does not change or becomes more egregious, SSA continues with progressive discipline including reprimand or seeking disciplinary action from the Merit Systems Protection Board, such as short- or long-term suspension or removal. From 2007 through 2016, there were 98 reprimands, 34 proposed suspensions, and 16 proposed removals, according to SSA.

SSA conducts various quality assurance reviews to improve accuracy and consistency. SSA officials stated that the agency has been enhancing its quality review efforts since 2009. Since then, it has added five types of quality assurance reviews that are conducted by three additional offices within SSA (see fig. 8).

SSA added quality assurance reviews for various reasons. For example, in 2009, SSA’s regional staff under the Office of the Chief Administrative Law Judge began conducting regional inline quality reviews, which involve assessing the extent to which hearing office staff are processing cases and preparing them for hearings in accordance with SSA policy, as well as the policy compliance and legal sufficiency of the draft decision. SSA added this review to enhance its reviews of decisions before they are issued, in an effort to reduce remands. Also in 2009, SSA’s Office of Quality Review began conducting disability case reviews to provide feedback on decision-making accuracy to ALJs. In addition, in 2010, SSA created the Division of Quality under the Appeals Council, a unit focused on conducting reviews on a regular basis of decisions that claimants did not appeal. Prior to 2010, SSA generally only reviewed decisions that claimants appealed through the Appeals Council.

While these quality assurance reviews have somewhat different focuses—for example, some assess aspects of how a case was processed while others review the accuracy of the decisions—they overlap in two key ways. According to prior GAO work, overlap occurs when multiple agencies or programs have similar goals, engage in similar activities or strategies to achieve them, or target similar beneficiaries. Some of SSA’s quality review efforts fit the description of overlap in that they have similar goals and review similar cases. For example:

Similar goals: Several of the reviews have similar goals (see table 2). For example, two of the four entities conducting reviews—the Appeals Council’s Division of Quality and staff in SSA’s 10 regional offices— both review decisions for policy compliance before those decisions go into effect (known as pre-effectuation reviews). While one review looks at the judge’s decision and the other looks at the draft decision prior to the judge’s review and approval, according to officials and documents we reviewed, these reviews share similar goals: to guide training and provide feedback to judges. In addition, all the reviews are designed to assess compliance with SSA policy.

Similar cases: SSA’s five quality assurance reviews look at similar cases, and could potentially include the same cases (see table 3). SSA takes some steps to prevent assessing the same claim in multiple quality assurance reviews. Officials told us that, in conducting focused quality reviews (conducted after the decision is final), they exclude cases that were reviewed in a pre-effectuation review. However, they said that the Division of Quality does not know whether cases it has selected were also subject to a regional inline quality review. They said that additional efforts to prevent multiple reviews of a case are manual in nature, and thus there is still the potential for claims to be reviewed more than once. Further, SSA officials said they did not see a need to prevent multiple reviews of a case, in particular, because some reviews are conducted before the decision is final and others are conducted after the decision is final.

SSA officials stated that opportunities exist to improve coordination across offices conducting quality assurance reviews. We found that several offices coordinated their work in some cases. For example, SSA’s Division of Quality and Office of Quality Review participate in a multi- office workgroup that addresses such issues as policy compliance across the initial and hearings levels of the disability process. In addition, they have also worked together on several studies, including a one-time quality review of 454 claims that were denied at the initial determination level, but were allowed as fully favorable at the hearings level. The Office of Quality Review also reviews the content of selected training for judges. In addition, the Division of Quality provided some initial input when the regional inline review effort was being designed. Prior GAO work has found that enhanced coordination can help to reduce overlap and improve efficiency. Effective October 1, 2017, SSA created a new deputy commissioner-level component, the Office of Analytics, Review and Oversight. This agency reorganization moved six oversight offices into the new component, including the Division of Quality and Office of Quality Review. Officials said the new component will create opportunities for improved coordination between these six offices. While this reorganization creates the opportunity for SSA to assess many of its quality assurance reviews, the regional quality review staff will not be included in the new office, and it is too early to tell how this reorganization will help manage the overlap between SSA’s various quality assurance reviews.

In addition, SSA has struggled to sustain all of its quality assurance reviews due to competing demands for the staff who perform them. For example, SSA placed regional inline quality reviews on hold in September 2016 and again in December 2016, because officials said that the agency needed staff to complete pending decisions before a change in the medical listings for mental impairments took effect in January 2017. Decisions not completed before the new listings took effect would have to be redone. Also, the Office of Quality Review curtailed its Disability Case Reviews in fiscal year 2016 to help prepare the oldest cases for hearings. As a result, only the Appeals Council’s review of appealed ALJ decisions (requests for review) and the Division of Quality’s quality assurance reviews were active in 2016.

Even as SSA has added quality assurance reviews, it has not systematically evaluated the efficiency and effectiveness of all the reviews to determine the extent to which they may be overlapping or complementary. We found that reviews conducted by the four entities have resulted in similar findings, raising questions about the efficiency of these reviews. For example, during the same 3-year period (fiscal years 2013 through 2015), quality reviews conducted by all four entities found problems with judges’ assessment of a claimant’s ability to perform work- related tasks, known as a residual functional capacity assessment. In addition, all four entities found problems with the evaluation of medical opinion evidence.

Moreover, SSA has not conducted a cost-benefit analysis of the five reviews. Officials said that there are no definite plans to do so, although they may consider conducting such an analysis in the future. We found that costs for the quality assurance reviews conducted in fiscal year 2015 were at least $23.7 million, and in fiscal year 2016 were at least $11.7 million (see table 4). By evaluating the quality assurance reviews to determine the extent to which each is needed to monitor and improve accuracy and consistency, SSA would be better positioned to meet its goals within its resources.

In addition, SSA continues to develop and implement initiatives aimed at improving hearing decisions, without evaluating the potential for overlap with existing quality assurance reviews. For example, as part of its backlog reduction plan known as the Compassionate And Responsive Services (CARES) plan, SSA is using computer algorithms for natural language processing to analyze the text of disability decisions and flag potential errors. Although the agency is piloting this effort in the Appeals Council before expanding it to hearing offices, it did not conduct a cost- benefit analysis. SSA officials said that natural language processing could be used to identify cases for further review, similar to its current selective reviews, and that decision writers could use the tool to conduct their own reviews of their draft decisions. SSA officials said that they do not anticipate much overlap between the use of natural language processing and OAO’s pre-effectuation reviews. However, there could be potential for overlap with regional inline reviews, which also review decisions drafted by decision writers. Federal internal control standards state that management should implement control activities through policies. Periodically reviewing policies, procedures, and related control activities for continued relevance and effectiveness in achieving objectives and addressing related risks can help agencies meet this standard.

Conclusions

SSA’s disability programs provide more than $200 billion in benefits for tens of millions of Americans annually, making it one of the largest components of the nation’s social safety net. The hearings and appeals level of the disability decision-making process is particularly important because about one in three people receiving Social Security disability benefits are granted benefits at this level. Given the number of people and the dollars at stake, it is crucial that claimants are treated fairly and their applications are evaluated accurately and consistently across the country, at all levels of the program.

Some of the variation in allowance rates that we found across judges may be expected, given the complexity of the cases and judges’ decisional independence. However, the persistent variation we observed over time, even after accounting for various factors that could otherwise explain allowance rates, might warrant additional attention. SSA is rightly focusing on oversight of judges, but our work suggests that the agency’s emphasis on timeliness over accuracy in its public metrics and the potential overlap in its quality assurance efforts may offer opportunities for improving the accuracy and consistency of hearing decisions.

First, this amount of variation in allowance rates underscores the need for SSA to measure and hold itself accountable for accuracy and consistency. However, without sharing performance information on the accuracy and consistency of its hearings-level decisions, such as the rate at which the Appeals Council agrees with a judge’s decisions, SSA may not be providing the public with adequate information on progress toward its objective to improve the quality, consistency, and timeliness of its disability decisions. Developing a set of performance measures that includes the accuracy and consistency of hearings decisions will help ensure the overall success of the program.

Second, SSA has not systematically considered how each of its quality assurance reviews helps the agency meet its objective to improve the quality of hearings-level decisions. Although the planned consolidation of multiple oversight and quality review offices is a positive step, it will be important for SSA to consider the usefulness of the information yielded by each quality assurance effort, as well as the costs associated with conducting the effort. Evaluating the efficiency and effectiveness of quality assurance activities can help ensure that SSA is using its resources for maximum benefit toward its objective to improve the quality, consistency, and timeliness of its disability decisions.

Recommendations for Executive Action

We are making the following two recommendations to SSA: The Commissioner of SSA should develop a set of public performance measures, to include accuracy and consistency, as well as timeliness, of administrative law judges’ (ALJ) disability decisions. SSA could consider whether existing quality review or monitoring efforts could provide suitable data for such measures. (Recommendation 1)

The Commissioner of SSA should systematically evaluate the efficiency and effectiveness of its quality assurance reviews and take steps to reduce or better manage any unnecessary overlap among them to ensure strategic use of resources. Such steps could include enhancing collaboration where reviews overlap or only conducting the reviews that are most efficient and effective in achieving agency goals for improving accuracy and consistency of ALJ disability decisions. (Recommendation 2)

Agency Comments and Our Evaluation

In commenting on a draft of this report, SSA agreed with our two recommendations to (1) establish public performance measures for the accuracy and consistency of administrative law judges’ decisions, and (2) systematically evaluate its various quality assurance reviews and take steps to reduce or better manage any unnecessary overlap among them. SSA stated that it would address both recommendations as part of a comprehensive assessment and refinement of its oversight roles and processes. SSA made several other comments about one of our conclusions and our analysis of variation in administrative law judge allowance rates, which we discuss below. SSA also provided technical comments, which we incorporated into the report as appropriate.

In its comments, SSA described its evolving oversight activities at the hearings level, including providing policy guidance and training for judges, capturing and utilizing data to gain a better understanding of trends and challenges, and implementing additional oversight review processes, all of which we discussed in our report. SSA’s comments acknowledged that our report describes the steps that the agency has taken to improve oversight, but disagreed with our conclusion that SSA emphasizes timeliness over accuracy. Our final report clarifies that we came to this conclusion based on a review of the performance measures the agency shares with the public in its annual strategic plan and performance reports. As we state in the report, SSA has employed a range of efforts to monitor the accuracy and consistency of hearings decisions, but it lacks performance measures to report publicly on these efforts.

Regarding our analysis of variation in ALJ allowance rates, SSA raised a concern about our finding (on page 26 of the final report) that claimants whose hearings were held in person were slightly more likely (by about 2.8 percentage points) to be allowed benefits than a typical claimant with a hearing held by videoconference. SSA cited its own internal analysis, which found a small (0.6 percentage-point) difference in allowance rates between in-person and videoconference hearings after controlling for a number of factors. It is not surprising, however, that our estimates are somewhat different, since SSA’s internal analysis differs from ours in several ways. The primary purpose of our statistical analysis was to isolate variation in allowance rates due to the unique judge or hearing office assigned to each claim. To do this, we developed a multilevel model using 9 years of data that controls for judge, hearing office, and claimant-level factors associated with allowance rates. On the other hand, SSA’s analysis was specifically designed to look at the difference in allowance rates between in-person and video hearings. SSA’s analysis also covered a shorter, more recent period of time (part of fiscal year 2015, fiscal year 2016, and part of fiscal year 2017), than our study (fiscal years 2007 through 2015). Additionally, the version of the model SSA cited in its comments included hearings held in person or by videoconference only in regular hearing offices, whereas our analysis included hearings held in National Hearing Centers as well as regular hearing offices and controlled for the type of hearing office. These differences notwithstanding, we agree with SSA that the estimated model- adjusted difference in allowance rates between in-person and videoconference hearings in both GAO’s and SSA’s analyses could potentially be explained by unmeasurable factors.

In addition, SSA noted that our measure of variation in judge decisions focused on allowance rates at the extremes of the distribution. Given that our charge was to explore the extent of variation in allowance rates across judges, we believe it was appropriate and important to measure the range of allowance rates between judges with high allowance rates (at the 95th percentile) and those with low allowance rates (at the 5th percentile). This would be more conservative than an approach that looks at allowance rates across all judges, including potential extreme values; and more nuanced than an approach that looks at the number of judges whose allowance rates are higher or lower than a given threshold. Further, our analysis shows that unadjusted allowance rates at the 95th percentile declined over the period of our analysis, from a high of 96 percent in fiscal year 2008 to 82 percent in fiscal year 2015. We saw a comparable decline in allowance rates after applying our multivariate model. To provide additional context, our report figures also show the middle of the distribution (the 25th and 75th percentiles), as well as the average allowance rates. We have also added information to our report further describing this middle range.

Finally, SSA noted that our analysis was not weighted by the number of determinations a judge made, suggesting that judges who decided very few claims, for example, could affect the range in allowance rates or the trends. As we show in Appendix I, Table 7, only 2.3 percent of the judges in our study population heard fewer than 250 claims per year. This group of judges had an unadjusted allowance rate of 61.9 percent, very similar to the allowance rate among judges who heard 500-699 claims per year (61.6 percent). Furthermore, the statistical methods we used to estimate the distributions of allowance rates (multilevel models) adjust the estimates for judges with fewer claims by weighting them more heavily toward the overall approval rate. This mitigates against judges with smaller caseloads, and therefore higher sampling variation, from contributing overestimated allowance rates that might have inflated our estimated variation across judges.

As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Commissioner of Social Security, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7215 or jeszeckc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to assess (1) the extent to which allowance rates vary across administrative law judges, and any factors that are associated with this variation, and (2) the extent to which the Social Security Administration (SSA) has processes to monitor the accuracy and consistency of hearings decisions.

To answer these objectives, we reviewed SSA policies and procedures related to administrative law judge (ALJ) disability hearings and decisions; manuals and documents describing SSA’s case processing systems for each level of SSA’s disability decision-making process, and guidance and training provided to judges for making disability decisions. We interviewed SSA officials in several offices within the Office of Disability Adjudication and Review (ODAR), including the Office of the Chief Administrative Law Judge and the Office of Appellate Operations, as well as conducted semi-structured interviews with Regional Chief Administrative Law Judges in each of SSA’s 10 regions. We also observed administrative law judge hearings in one of SSA’s five National Hearing Centers, in Falls Church, Virginia, (hearings in these offices are conducted by videoconference), as well as in two of SSA’s regular hearings offices, in Washington, D.C., and Seattle, Washington. The purpose of these observations was to gain a better understanding of the hearings process in practice, and to inform our scope and methodology for this study. We selected these sites, which are not generalizable to the population of all hearing offices, for a number of reasons, primarily: (1) to observe hearing offices in different geographic locations and observe both in-person and video teleconference hearings, and (2) to select sites at which a cross-section of cases with different types of disabilities and impairments were available. We attended hearings involving both adult and child claimants with a mix of physical and mental impairments.

This appendix is divided into three parts. The first describes our data sources and analysis of allowance rates across judges and associated factors, the second describes our multivariate statistical model, and the third describes our work related to our second research objective on SSA’s processes to monitor the accuracy and consistency of hearings decisions.

Analysis of Variation in Allowance Rates across Judges and Associated Factors

For this objective, we analyzed data from two primary sources from fiscal years 2007 through 2015: SSA’s administrative data systems for the initial and hearings levels of the disability decision-making process, and the agency’s personnel data system. We also obtained other SSA administrative data on staffing levels and numbers of pending cases in each hearing office. Finally, we obtained data on state poverty and unemployment rates from the U.S. Census Bureau and the Bureau of Labor Statistics, respectively.

SSA Administrative Data Systems

To analyze information on all adult disability decisions made by administrative law judges from fiscal years 2007 through 2015—the most current data available at the time of our analysis—we compiled claims data from several SSA administrative data systems. These data contained information on the outcomes of the disability decisions and the characteristics of claims associated with each decision. Specifically, the information was drawn from the following systems:

831 File and Structured Data Repository: The 831 File pertains to the initial and reconsideration level of the disability determination process, within the state Disability Determination Services (DDS). Data on claimant characteristics we used from this system include the date of the claimant’s initial application for benefits and the claimant’s self-reported years of education. We also received a limited set of data captured from the claimant’s disability application in SSA’s electronic case folder system (Structured Data Repository), including the number of years a claimant reported being employed out of the 15 years before becoming disabled.

Case Processing and Management System (CPMS): This system pertains to the hearings level and was our primary source of information on hearing outcomes, claim, and claimant characteristics. Specifically, this system provided information on claim type (i.e., Disability Insurance, DI; Supplemental Security Income, SSI; or concurrent claim); the outcome of the claim (i.e., dismissed, allowed, or denied) and the date the decision was made; the unique identification number of the administrative law judge (ALJ) who made the decision; whether a medical expert or vocational expert attended the hearing; whether the claimant was represented; the hearing office where the claim was decided and the type of hearing office (i.e., hearing office or National Hearing Center); the claimant’s date of birth; the primary impairment at the time of the hearing level; the presence of a secondary impairment; and whether the case was classified as being a critical case—that is, a case requiring special processing, such as a terminal illness.

We used case identifiers to link the information from each of these databases that pertained to each disability decision we analyzed.

Federal Personnel and Payroll System

We obtained data from SSA’s Federal Personnel and Payroll System (FPPS) database on all administrative law judges who were employed by the agency at any time during the period from January 1, 2005, through December 31, 2015. We obtained information on each judge, such as their date of appointment as an ALJ and the type of appointment (regular career appointment or non-permanent); service computation date; and prior position titles within SSA, if any.

Other SSA Administrative Data

We obtained summary-level data, as of January 2017, from SSA on staffing levels (numbers of ALJs, decision writers, and other support staff) at each hearing office for each fiscal year in our study period (fiscal years 2007 through 2015) from SSA’s Payroll Operational Data Store system. We also obtained data on the numbers of cases left pending at the end of each fiscal year (including the number of cases pending for more than 270 days). SSA provided those data from a management information report that uses CPMS data.

Economic Conditions Data

We used publically available estimates of state poverty rates for each year in our analysis (calendar years 2007 through 2015) from the U.S.

Census Bureau’s American Community Survey (ACS). We considered using estimates at the county level but that approach had limitations. First, we would have been limited to using 3-year or 5-year estimates for all counties, because 1-year ACS estimates are only available for areas with populations of 65,000 or more. Second, the Census Bureau cautions against using estimates for particular time periods that do not align with the periods of its estimates. Although using state-level estimates reduced the geographic precision of the estimates, we gained precision by having annual estimates and the ability to measure potential variation in poverty rates over narrower time intervals.

We also used publically available estimates of state unemployment rates in calendar years 2007 through 2015 from the Bureau of Labor Statistics’ Local Area Unemployment Statistics data. This variable allowed us to control for labor market conditions over time.

Data Reliability

SSA constructed custom files for GAO from several SSA datasets in response to our data requests. We assessed the reliability of the data used in our analyses through electronic testing, analyzing related database documentation, examining the SAS code used by SSA to construct the custom files, and working with agency officials to reconcile discrepancies between the data and documentation that we received. We determined that the 831, Structured Data Repository, and CPMS data on ALJ decisions and claimant characteristics and the FPPS data on ALJ appointments were sufficiently reliable for the purposes of describing the extent of variation in the outcomes of ALJ decisions. We also determined that SSA’s data on pending caseloads and ALJ and decision writer staffing, by year and hearing office, were sufficiently reliable for the purpose of describing hearing office characteristics. Finally, we determined that ACS data on state poverty rates and BLS data on state unemployment rates were sufficiently reliable for the purposes of describing these state economic characteristics.

Scope of Analysis

Our analyses of ALJ decisions excluded various types of decisions from the CPMS data because they were out of scope for our research objectives (e.g., child cases, non-disability cases, or cases that were decided by SSA staff who were not ALJs) or were not typically randomly assigned to judges. We selected cases that should have been assigned randomly to judges, according to SSA policy, because that random assignment made it more likely that variation in allowance rates across judges in our multivariate analysis reflects the unique causal effect of having a particular judge hear a case, rather than other factors that also vary across judges. Our exclusion criteria were similar to those used by an internal SSA study of ALJ allowance rates, conducted in 2017. We excluded cases that were:

Dismissed. Cases can be dismissed for reasons not related to the merits of the case and that are usually beyond the ALJ’s control—for example, the claimant’s failure to file a timely request or to appear at the scheduled hearing (without good cause), or the claimant’s death before the hearing. In addition, data on key factors for these cases, such as the claimant’s impairment, were missing. From fiscal years 2007 through 2015, 1,007,526 claims (16 percent of all claims) were dismissed.

Made “on the record” and not randomly assigned to judges. While most appeals are decided after an ALJ hearing, ALJs and senior attorney adjudicators (SAA) have the authority to issue on-the- record decisions. These are decisions where a hearing is not necessary because the documentary evidence alone supported a fully favorable decision. SSA has created screening criteria, such as the claimant’s age (50 and older) and specific impairments, to help identify possible on-the-record decisions earlier in the process. ALJs and SAAs can also issue on-the-record decisions for cases involving critical need, and claimants and their representatives can request that the ALJ or SAA issue an on-the-record decision. These cases are not assigned randomly to judges. From fiscal years 2007 through 2015, 716,574 claims (11 percent of all claims) were on-the-record decisions, although SSA has issued fewer on-the-record decisions in more recent years.

Issued for children. We excluded claimants younger than 18 at the date of the initial application. We also excluded claimants with missing or invalid age values. From fiscal years 2007 through 2015, 492,158 claims (8 percent of all claims) were for people under 18 or with missing or invalid age values. We excluded child cases from our analysis because they involve different evaluation criteria.

Remanded back to a judge from SSA’s Appeals Council (or federal court). These cases represent decisions that were corrected after an order from the Appeals Council or a federal court after the original ALJ’s decision. In these cases, judges are often addressing a narrow set of issues identified in the remand order. Remanded cases are also not assigned randomly to judges, since the Appeals Council generally sends them back to the judge who originally issued the decision. However, SSA’s Office of the Inspector General (OIG) in 2017 found that about half of the remanded cases in its sample were assigned to a different ALJ than the original ALJ. From fiscal years 2007 through 2015, 293,971 claims (less than 5 percent of all claims) were remands.

Made by senior attorney adjudicators who were not administrative law judges. We excluded decisions made by SAAs. SSA implemented a program in 2007 whereby SAAs located in hearing offices across the country could issue fully favorable on-the- record decisions. According to SSA, this allowed ALJs to focus on cases that are more complex or require a hearing. From fiscal years 2007 through 2015, 227,133 claims (4 percent of all claims) were decided by SAAs.

Appeals of continuing disability reviews (CDR). These cases represent decisions about whether or not to continue benefits for claimants who were previously found eligible for the program. As such, they involve different evaluation criteria. From fiscal years 2007 through 2015, 245,862 claims (4 percent of all claims) were appeals of CDRs.

Non-disability cases. These cases include Social Security retirement and survivor benefit decisions. We excluded such cases because they involve different evaluation criteria from disability claims and represent a small minority of decisions at the hearings level. From fiscal years 2007 through 2015, 25,293 claims (less than 0.5 percent of all claims) were for non-disability cases.

Decided by judges with limited experience. We excluded cases decided by judges within the first year (365 days) after their appointment as an ALJ, as calculated by the difference between their date of appointment and the date of the decision on each claim. We excluded these decisions to help ensure that variation we identified in allowance rates was not due to the judges’ more limited experience deciding Social Security disability claims. From fiscal years 2007 through 2015, 574,307 claims (approximately 9 percent of all claims) were decided by judges with limited experience.

In total, our exclusion criteria reduced the number of records analyzed by about half. Specifically, out of a universe of about 6.3 million records, our study population included about 3.3 million decisions. Nevertheless, the overall allowance rate for our study population over fiscal years 2007 through 2015 was 62 percent, very close to the overall allowance rate for the entire population of claims during this period, which was 64 percent.

Calculation of Allowance Rates

We calculated allowance rates by dividing the number of favorable decisions by the total number of decisions (both unfavorable and favorable). We calculated allowance rates for different units of analysis:

Overall, by program type (Disability Insurance, Supplemental Security Income, and concurrent) and by year and for all years and program types pooled together,

At the judge level, by year and for all years pooled together, and

At the hearing office level, by year and for all years pooled together.

When analyzing our data at the case level, we identified whether the case was favorable or unfavorable to the claimant (that is, whether the claimant was allowed benefits or not). We did not include cases that were dismissed in our study population for two reasons. First, as discussed above, these cases can be dismissed for reasons not related to the merits of the case, and without a review of the medical evidence. Second, SSA’s data on dismissed cases are limited, partially because cases are dismissed without a review of medical evidence. For example, the impairment code from the hearings-level decision was missing for virtually 100 percent of dismissed cases.

For concurrent claims—those in which an individual is applying for Disability Insurance (DI) and Supplemental Security Income (SSI) benefits—we considered a case an allowance if the claimant was approved for either or both programs. Our classification of allowances for concurrent cases differs from SSA’s usual practice (although a 2017 internal study of ALJ allowance rates used the same method as ours). SSA officials said they usually allow the SSI decision to “control” the overall outcome of the case. That is, SSA classifies a concurrent claim as an allowance if the SSI decision is an allowance, regardless of the outcome for the DI claim. Officials said that they chose this method primarily for convenience. This results in a different classification of some cases in which the SSI claim was denied but the DI claim was allowed. In such cases, the claimant is receiving a benefit as a result of their concurrent disability claim but would be classified in SSA’s data as a denial. However, the resulting differences in the number of allowances is very small—less than 4,000 claims over fiscal years 2007 through 2015— and the different definitions did not substantively affect allowance rates in any year.

Random Assignment of Cases to Judges

SSA policy states that cases are generally assigned on a “first in, first out” basis, meaning that cases are assigned to judges in the order in which they are received. Administrative law judges are assigned cases on a rotational basis, with the oldest case in the backlog given to a judge who most recently decided a case. Therefore, as noted in prior research, the initial assignment of cases to judges is random (conditional on applying at a given hearing office at a given time). Judges do not select their cases, nor are claimants able to request another judge after one is assigned. Claimants are generally assigned to hearing offices based on their ZIP code, although some claimants in hearing offices with higher numbers of pending claims may be transferred to one of SSA’s five National Hearing Centers. In those cases, hearings are conducted by videoconference rather than in person, as is traditionally done in SSA’s regular hearing offices. However, the claimant may opt out of a video conference hearing within 30 days of receiving a written notice acknowledging the request for a hearing.

There are some exceptions to the “first in, first out” rule, such as cases that are likely to be dismissed or decided on the record (without a hearing) and critical cases (including terminal illness cases and veterans who have a 100-percent permanent and total disability compensation rating). However, as discussed previously, we have excluded all major categories of exceptions but critical cases from our analyses, and included a variable to identify critical cases in our analyses.

Assignment of Claims to Steps in SSA’s Sequential Disability Decision-Making Process

SSA’s disability decision-making process includes five sequential steps, and one part of our analysis was to determine the step at which each decision was made. In consultation with SSA officials, we used a code in CPMS—called the regulation basis code—to assign each claim to a particular step.

Each claim in CPMS has between one and four regulation basis codes, depending on whether the claim was for a single program (DI or SSI), or a concurrent claim for both. We assigned each claim to one of the five steps in SSA’s disability decision-making process, based on its regulation basis code. Each regulation basis code is associated with one of five steps. Therefore, if a claim had just one regulation basis code, we assigned it to the corresponding step. If a claim had more than one regulation basis code, we used a series of decision rules to select the most appropriate step. Specifically, claims for a single program have up to two regulation basis codes listed, and we used the code that matched the outcome of the case and/or the latest step. We used a similar method for concurrent claims.

We found that approximately 19 percent of all allowances occur at step 3, when SSA determines whether a claimant’s impairment meets or is equivalent to an impairment listed in SSA’s Listing of Impairments. Most (80 percent) of all allowances are made at the final step (step 5), when SSA determines whether the claimant can do any work in the national economy, given the limitations of their impairment and their age, education, and work experience.

Nearly a third (28 percent) of denials are made at step 4, where SSA determines whether the claimant can do their past work, and 62 percent of denials are made at step 5. There are some differences between DI claims and SSI claims in the distribution of allowances and denials over the five steps. SSI allowances occur at step 3 to a greater extent than DI allowances, while SSI denials occur at step 5 to a greater extent than DI denials (see table 5 below).

Statistical Model of Variation in Allowance Rates across Judges and Associated Factors

We developed our multivariate statistical analyses in consultation with GAO statisticians, economists, and social scientists and SSA officials and experts. Our analysis was also informed by a comprehensive review of the literature pertaining to judicial decision-making and, in particular, adjudication for SSA’s disability programs. Specifically, we reviewed more than 90 potentially relevant peer-reviewed academic journal articles, government reports, and nonprofit association and think tank white papers. We selected 39 of these studies or reports for a detailed review of the scope and methodology, key factors or variables used in any empirical analyses, and other relevant findings. We also reviewed relevant SSA Office of the Inspector General (OIG) reports and consulted with SSA and OIG officials, and reviewed prior GAO reports that modelled judicial outcomes. Our statistical model included variables that are either direct or approximate measures for: (1) claimant characteristics that represent criteria used in the disability decision-making process, (2) judge characteristics, (3) other participants in the decision-making process, (4) SSA administrative characteristics, and (5) economic characteristics of the claimant’s state.

Our analysis was purely statistical, in that we did not conduct the legal analysis needed to reach conclusions about what legal factors might have affected a judge’s decision or whether the decision that was reached in any particular case was correct. Similarly, we are not making any predictions about the likely or correct outcome of future individual decisions. Each case is unique in both its facts and circumstances and must be examined on its own merits.

We included factors that represent criteria used in decision-making process, such as the type of claim (DI, SSI, or concurrent) and the claimant’s age, years of education (grouped into equivalent levels: less than high school, high school, some college, and college or higher), and primary impairment.

We included factors related to the judge’s employment as an ALJ, such as the year appointed as a judge, the type of appointment (whether they had a career or temporary, non-permanent assignment), and any prior work history at SSA (specifically, whether they were an attorney or held another position prior to being appointed as an ALJ).

Other participants in the decision-making process

We included factors that represent other participants in the decision- making process, such as the claimant’s use of an attorney or non- attorney representative, or the testimony of a medical or vocational expert at the hearing. Our prior work has shown, for example, that claimants who were represented by an attorney or a person who is not an attorney (such as a relative or professional disability representative) were more likely to be allowed disability benefits than claimants who had no representative.

We included factors related to SSA’s administration of its disability programs, such as the hearing office in which the claim was decided, whether the claim was heard in one of 10 states that do not have a reconsideration step between the initial state-level Disability Determination Service decision and a hearing before an ALJ, and the percentage of pending cases at the hearing office that were pending for more than 270 days (SSA’s definition of a “backlogged” case).

Finally, we assessed economic characteristics of the state in which the claimant resided because some prior research suggests that such factors may be associated with disability application and allowance rates. Specifically, we analyzed:

The unemployment rate in the claimant’s state as of the year of each decision in our analysis, from the Bureau of Labor Statistics’ Local Area Unemployment Statistics data. We selected this factor in order to account for the labor market conditions where claimants live.

The poverty rate in the claimant’s state as of the year of each decision in our analysis, from the Census Bureau’s American Community Survey (ACS).

Goals of Analysis

The primary goal of our analysis was to isolate variation in allowance rates due to the unique judge or hearing office assigned to each claim by controlling for multiple factors that could otherwise affect this variation. Some variation in allowance rates across judges and hearing offices could reflect the distribution of other factors that are correlated with allowances. For example, judges who hear disability cases in regions of the country with higher obesity rates—a known risk factor for disability— may appear to have higher allowance rates than those in regions with low obesity rates. Because judges’ decisions to allow benefits may be related to this or other factors, simple univariate comparisons of allowance rates across judges may reflect characteristics of the cases that judges hear.

To help isolate the potential unique effects of judges, we used multilevel, multivariate statistical models that held constant various factors that could have been associated with allowance rates. We held constant variables available in SSA and other public data sources that were relevant to the claim appeals process, in order to estimate the amount of potential residual variation across judges.

Statistical Model

The data we assembled had a multilevel structure, with applications for disability benefits clustered within the same judges and hearing offices. Judges were associated with multiple hearing offices, because judges sometimes decided cases in multiple hearing offices during the period of our analysis. For example, judges could travel to more remote sites to hear cases on a part-time basis.

The data and outcome of interest suggested that a multilevel or mixed logistic regression model would adequately reflect the data generation process. We developed a mixed model that represented the grouping variables—judge, hearing office, and primary diagnosis code—with random intercepts, similar to prior research. We modeled group variation with random effects primarily for parsimony. Modeling group variation with fixed effects would have required estimating several thousand explicit parameters, one for each group level, which would have consumed many degrees of freedom. Estimating the amount of variation across groups then would have required interpreting many contrasts between pairs of fixed effect estimates. In contrast, modeling group variation using random effects allowed us to represent the variation with probability distributions and a small number of summary (hyper) parameters, such as the standard deviation of the judge random effect.

Substantively, random effects accurately represented the SSA policy of randomly assigning judges to cases in our study population, using a “first in, first out” method. Moreover, we modeled variation across judges and hearing offices as random, which implies that we seek to make inferences about a larger, hypothetical population of judges and hearing offices that could exist if we replicated the study in the future. This seems appropriate, because the application review process could be repeated across many new judges and hearing offices in the future. We do not seek to make inferences limited to the judges and hearing offices at the particular time we assembled data.

We held constant case, judge, and hearing office characteristics using covariates with fixed parameters. The smaller number of parameters associated with these covariates made a fixed effects approach easier to apply and interpret. We assumed that the covariate effects did not vary across groups, so that only the model’s intercept varied randomly. We had no prior expectation that specific covariate effects should have varied across groups. Moreover, increasing the number of random effects would have increased the complexity of the model and could have made it hard to estimate computationally.

We viewed the covariates primarily as controls for isolating variation across judges and offices. We did not attempt to build a comprehensive model that correctly specified how all of the covariates were causally ordered and related to each other and the probability of an allowance. As a result, our estimates of these parameters may not be consistent with those obtained from a more comprehensive modeling effort, or from analyses designed to estimate the causal effects of particular variables, such as the use of videoconferences. In the body of the report, we present alternative explanations and provide context to avoid interpreting the covariate effects as with a high degree of causal certainty. For example, we note that claims with legal representation may have higher approval rates if representatives accept claims with greater merit and, therefore, a greater chance of compensation. Below, we test alternative model specifications for covariates where the causal ordering may be ambiguous, in order to avoid biasing estimates of the judge and office parameters of primary interest.

Certain variables and parameters were applied across multiple versions of the model (described below). Let Y denote the allowance or denial decision for claimant i at any step of the appeals process, with Y = 1 if the ALJ allowed the claim and 0 otherwise. Each model took a typical hierarchical generalized linear form for a binary outcome: The probability of allowance, π, was a function of covariate vectors measuring characteristics of claims, Xijo, characteristics of the ALJs assessing those claims, Xjo, and characteristics of the hearing offices where the decision occurred, X. Claimants were clustered in j = {1, 2, … , J} judges, and judges were clustered in o = {1, 2, …, O} offices. g is the inverse logistic link function.

We included normally distributed random effects, ε(.), for each judge, office, and the claimant’s primary diagnosis, indexed by diagnosis codes d = {1, … , D}. Random effects allowed the intercept for each group, α(.) = α + ε(.), to vary around the population average intercept, α, as a function of the group’s variance, σ(.): To make interpretation and computation easier, we classified all continuous covariates into substantively meaningful categories, and set the omitted reference categories to the sample modes. This transformation implied that the random effect variance, σ(.), described variation across judges and offices for a claim that had the modal value of all other covariates in the model and sample. The reference claim remained constant across models fitted to different subsamples, in order to make inferences about a claim that was typical for the study population.

The center of the data at the modes, α, may not necessarily correspond to an actual claim. For example, all judges do not practice at the modal hearing office, and the modal age for the study population may not be typical for claims made in the modal office. Nevertheless, rescaling facilitates estimation and interpretation of the model, because all inference can be done on α and α(.) directly, using the random effect variance, σ(.), without transformation. This allowed us to concisely describe variation in allowance rates for a hypothetical, typical claim in the joint covariate distribution.

In the body of this report, we summarized variation across judges and offices, holding constant other covariates at their sample means, using the estimated distribution of group intercepts scaled in logits: To describe variation across groups on the probability scale, we estimated the quantiles bounding the middle 50 and 90 percent of the group density on the logit scale and then transformed them using the inverse logistic function, g. This reference case does not represent a feasible claim, because the means of the categorical covariates are just the sample proportions. However, this approach complements the centering of the sample, which allows the group variance parameters, σ(.), to represent variation across groups for a feasible reference case at the sample modes.

Covariates and Subsamples

We fit a sequence of models using different covariates and subsamples, listed below. Fitting several models allowed us to assess how simplifying assumptions, such as ignoring the step at which ALJs made allowance decisions, affected our results. This approach also assessed the stability of estimates across multiple runs of the computational model estimation methods. We describe the substantive meaning of the covariates above, and give their exact measurement when reporting results in table 7 below.

Model 1: Intercepts Only

Model 2: Add Covariates (Unemployment and poverty vary at the state level, not at the office level, but we include them with the office covariates for simplicity.)

Model 3: Claims Decided at Steps 4 or 5 We estimated Model 2 for only those claims decided at steps 4 or 5, according to each claim’s Regulation Basis Code. In these last two steps of the sequential disability decision-making process, the judge determines whether claimants retain the ability to perform their past work or other work in the national economy, given the limitations of their impairment and their age, education, and work experience. SSA officials provided methods to map these codes to steps of the appeals process. Estimating the model for decisions at steps 4 or 5 allowed all parameters to vary at these steps versus all steps in the pooled sample. For example, diagnosis may be less strongly associated with allowances at step 5 than at step 3, while the claimant’s age may be more strongly associated.

Models 4-6: Stratify by Year of Decision and Claim Type To assess how the amount of variation across judges and offices has changed over time, we estimated Model 2 separately for each year of decision, claim type, and the cross-classification of these variables. Stratified models allowed all parameters to vary across claim types and years.

Model 7: Exclude Potentially Endogenous Covariates We excluded covariates from Model 2 that may not be exogenous to the probability of approval. These include representation by an attorney or other person and the presence of a medical or vocational expert. Claims with legal representation may have higher approval rates if representatives tend to accept claims with greater merit and, therefore, a greater chance of compensation. (Representatives typically receive a share of their client’s benefits as compensation.) According to SSA officials, medical and vocational experts may be more likely to testify at a hearing, depending on the judge’s expected ruling on the case. Although judges generally have discretion about whether to involve medical and vocational experts, judges are required to seek the opinion of a medical expert in certain cases. For example, a judge must have a medical expert provide an opinion if the judge is considering allowing benefits because the claimant’s impairment may be medically equivalent to one in SSA’s Listing of Impairments. Excluding these covariates avoids potentially biasing estimates of the judge and office parameters of primary interest.

Results

We provide the estimated distributions of allowance rates across judges, hearing offices, and primary diagnoses, holding all other covariates at their means, in table 6 below. Each row in the table lists results for one specification of the model described above. We derived quantiles of the distributions across groups with the data and estimated model parameters, using the methods above. The standard deviations of the allowance rates on the logit scale are explicit parameters in the model and were directly estimated with the fixed coefficients. We used these distributions to describe variation across judges, offices, and diagnoses in the body of this report and in figures, where we interpret the results in more detail.

Table 7 below provides estimated odds-ratios of allowances for the factors other than judge, hearing office, and diagnosis in our primary model of ALJ allowance rates (Model 2 above), along with sample distributions and raw allowance rates. We used the primary model to support our findings in the body of this report, where we interpret the results below in more detail. Our model included variables that are measures or approximate measures for (1) claimant characteristics that represent criteria used in the disability decision-making process, (2) judge characteristics, (3) other participants in the decision-making process, (4) SSA administrative characteristics, and (5) economic characteristics of the claimant’s state. The interpretation of the odds ratio for a particular variable depends on whether the variable is a dummy variable or a categorical variable. For dummy variables, a statistically significant odds ratio that is greater/less than 1.00 indicates that claimants with that characteristic are more/less likely to be allowed than claimants without it. For categorical variables, a statistically significant odds ratio that is greater/less than 1.00 indicates that claimants in that category are more/less likely to be allowed than the claimants in the reference category.

Evaluation of SSA’s Processes to Monitor Accuracy and Consistency in Hearings Decisions

For objective 2, we reviewed relevant federal laws, regulations, and documentation, and collected testimonial evidence from SSA officials to describe and evaluate the processes that SSA uses to monitor hearing decisions, detect variation, and improve accuracy and consistency. We interviewed SSA officials at different levels, including officials at headquarters, regional, DDS, and field office levels. We reviewed documents such as SSA’s Hearings, Appeals, and Litigation Law (HALLEX) manual, policy memoranda issued by the Chief Administrative Law Judge, monitoring and quality assurance reports, user manuals and guides for electronic tools, SSA OIG reports, and descriptions of processes that are under development. We assessed these monitoring efforts against federal internal control standards and our management and evaluation guide for assessing fragmentation, overlap, and duplication in government programs. We also reviewed SSA’s annual performance plans from fiscal year 2006 through fiscal year 2017 to identify performance measures the agency has established to improve the accuracy and consistency of its hearings decisions. We evaluated the current performance measures using key attributes of performance measures used in prior GAO work and federal internal control standards. In addition to interviews with agency officials, as described above, we also interviewed officials from organizations representing judges, disability claimants, and representatives to obtain their perspectives on SSA’s efforts to monitor and improve accuracy and consistency.

Appendix II: Comments from the Social Security Administration

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, Erin M. Godtland, Assistant Director; Rachael Chamberlin, Analyst-in-Charge; Dana Hopings, LaToya King, Stephen Komadina, Rhiannon Patterson, and Jeff Tessin made significant contributions to the report. In addition, Daniel Bertoni, Deborah Bland, David Chrisinger, Melinda Cordero, Holly Dye, Bill Egar, Alex Galuten, Benjamin Licht, Serena Lo, Mimi Nguyen, Samuel Portnow, Sheila McCoy, and Shana Wallace made valuable contributions.