Social Security Disability: SSA Could Increase Savings by Refining Its Selection of Cases for Disability Review
Released: 2016-03-14T00:00:00

Letter

Background

SSA administers two disability programs that provide monthly cash benefits to eligible individuals: (1) Disability Insurance (DI) for individuals (and their dependents) who have paid into the Disability Insurance Trust Fund and (2) Supplemental Security Income (SSI) for low-income individuals. To be eligible for DI or SSI benefits based on a disability, an individual must have a medically determinable physical or mental impairment that (1) prevents the individual from engaging in any substantial gainful activity and (2) is expected to result in death or has lasted or is expected to last at least 1 year.

Federal law generally requires CDRs to be conducted at least once every 3 years for all DI beneficiaries whose disabilities are not considered permanent, and at intervals determined appropriate by SSA for those whose impairments are considered permanent. For SSI, federal law generally requires SSA to (1) conduct CDRs for infants during their first year of life if they are receiving SSI benefits due in part to low birth weight, and at least once every 3 years for SSI children under age 18 if their impairments are considered likely to improve, and (2) review the cases of all SSI children beginning on their 18th birthday to determine whether they are eligible for disability benefits under adult disability criteria. SSA may waive the requirement to conduct periodic legislatively- required CDRs on a state-by-state basis. SSA may also conduct CDRs that are not required by law as it deems appropriate.

CDR Process

SSA contracts with state Disability Determination Services (DDS) agencies to initially determine whether applicants are disabled and to conduct periodic CDRs to determine whether beneficiaries continue to be disabled. DDS examiners assess whether individuals are eligible for benefits based on several criteria, including their current medical condition and ability to work. At the time beneficiaries enter the DI or SSI programs or continue their benefits following a CDR, a DDS examiner determines beneficiaries’ due dates for a subsequent CDR based on their potential for medical improvement. Beneficiaries classified as “medical improvement expected” are generally scheduled for a CDR within 6 to 18 months, beneficiaries classified as “medical improvement possible” are scheduled once every 3 years, and beneficiaries classified as “medical improvement not expected” are scheduled once every 5 to 7 years.

To cost-effectively manage its CDR workload, SSA conducts CDRs in different ways. In general, beneficiaries with a high likelihood of medical improvement are referred for a full medical review—an in-depth assessment of a beneficiary’s medical and vocational status. Beneficiaries with a low likelihood of medical improvement are, at least initially, sent a questionnaire known as a mailer. If beneficiaries respond to a mailer in certain ways, SSA may refer these individuals for a full medical review. In contrast to mailers, full medical reviews are more labor intensive and expensive. Full medical reviews result in a decision to either cease or continue an individual’s benefits. In fiscal year 2013, the cessation rate for CDRs involving full medical reviews was about 19 percent, whereas the cessation rate for all CDRs including mailers was about 5 percent. Each year, SSA allocates a portion of its program integrity budget to CDRs, which affects the number of full medical reviews and mailers that the agency initiates during the year. When the number of cases due for a CDR exceeds SSA’s capacity to conduct full medical reviews and mailers, the cases not initiated during the year are considered backlogged for future review.

The number of CDRs completed as full medical reviews, as mailers only, or backlogged varied during fiscal years 2003 through 2013 (see fig. 1). After the authority for special funding to process CDRs expired in fiscal year 2002, backlogged CDRs increased from about 100,000 cases in fiscal year 2003 to more than 1 million in fiscal year 2007, reaching a peak of nearly 1.5 million in fiscal year 2009. At the same time, the number of full medical reviews fell from nearly 670,000 in fiscal year 2003 to less than 190,000 in fiscal year 2007 before rebounding to nearly 429,000 in fiscal year 2013.

SSA estimates the accuracy rate of CDRs and separately estimates the cost savings that result from CDRs. In addition to annually reporting the nationwide accuracy rate of all CDRs to the Congress, SSA internally tracks CDR accuracy rates by state and generates estimates for the accuracy of cessations and continuances, separately as well as combined. For fiscal year 2013, SSA reported an accuracy rate of 97.2 percent for CDR decisions. For the same year, SSA reported an estimated ratio of federal program savings to costs for performing CDRs as $15 to $1. Savings from CDRs include federal benefits that would be paid to individuals were it not for a CDR that resulted in a cessation. Such benefits include those from Medicare and Medicaid because in certain situations individuals’ eligibility for DI or SSI confers eligibility for these other programs.

SSA Prioritizes CDRs Using Several Inputs but Does Not Fully Incorporate Potential Cost-Savings

SSA Uses a Variety of Inputs, Including Legal Requirements and Statistical Models, to Prioritize CDRs

Because SSA does not complete all CDRs as scheduled due to competing priorities and existing resources, the agency must decide which cases will receive a full medical review. SSA uses a range of inputs to prioritize which CDRs to conduct, such as:

Statutory requirements: Legal requirements to review SSI children beginning at age 18 to determine if they are eligible for benefits under adult disability criteria, and reviews of children up to 1 year old who are receiving SSI benefits due in part to low birth weight.

SSA policies: Rules established by SSA to guide prioritization. For example, SSA prioritizes cases with particular responses to its mailers and cases with a “medical improvement expected” designation that are coming up for review for the first time.

Statistical models: A set of statistical models that score each case according to the likelihood of medical improvement, typically the sole criterion for ceasing benefits.

SSA’s prioritization process determines which CDRs are initiated and in what form: full medical review or mailer. To begin, SSA initiates full medical reviews of cases that fall into two high-priority categories: first, statutory requirements, and then SSA policies. Once full medical reviews of all high-priority cases are initiated, SSA prioritizes the remaining cases by using its statistical models. Full medical reviews of cases with the highest scores (i.e., highest likelihood of medical improvement) are initiated as resources permit, first by beneficiary group (i.e., DI, SSI children, SSI adults) and then by the statistical scores of cases within the group (see fig. 2). Cases with lower scores (i.e., lower likelihood of medical improvement) receive a mailer or are backlogged for future review.

The extent to which the statistical models have been used to select cases for full medical reviews has varied by year, but the models have been consistently used for determining who receives mailers. Specifically, we found that the extent to which SSA’s statistical models were used to select cases for full medical review was related to the combination of budget fluctuations for CDRs, SSA’s statutory requirements, and agency policies. We estimate that the statistical models were the basis for selecting 11 to 60 percent of full medical reviews completed annually in fiscal years 2003 through 2013 (see fig. 3). In contrast, SSA has used the models consistently since 1993 to determine which cases should receive a mailer.

SSA Tests the Accuracy of Its Models but Has Not Updated Them to Reflect Changing Conditions in Recent Years

Although SSA annually assesses its models’ performance, the agency has not updated the models since 2007. The models’ effectiveness depends on their ability to accurately predict beneficiaries’ likelihood of medical improvement. To test the accuracy of the models, SSA conducts an annual validation process using a sample of completed cases to evaluate how well the models predicted medical improvement. According to SSA officials, the model validation process has shown that the models’ accuracy in predicting medical improvement has not degraded substantially in recent years.

In addition to model validation, in the past SSA has conducted periodic re-estimation of its statistical models to help ensure that they are up-to- date. In re-estimating its models, SSA updates the relationship between existing variables and medical improvement and tests whether new variables should be included. Re-estimation is particularly important when advances in medicine and assistive technology affect people’s ability to work. In recent years, SSA has changed its classification of certain beneficiaries’ impairments to reflect advances in medical knowledge. For example, in 2015, SSA revised its codes for cancer in light of new diagnoses and treatments. Because these codes can appear as variables in the statistical models, it is possible that the models are no longer accurately capturing the effect of cancer-related impairments on the likelihood of medical improvement. Although SSA officials believe that some advances would not markedly affect the accuracy of the models, the agency has not completed a re-estimation to confirm the effect of such changes. In addition, demographic changes in the underlying population of disability beneficiaries, which has grown substantially in recent years due in part to baby boomers reaching their disability-prone years, could also affect the accuracy of the models. The contractor that SSA hired to handle its last model re-estimation in 2007 provided SSA with a set of programs that would allow the agency to re-estimate the models in-house. Regular re-estimation and updating of predictive models is a best practice, and the contractor anticipated that SSA would do so at least every 3 years.

Model accuracy leads to savings for SSA in two ways. First, model accuracy is important for identifying cases that are unlikely to result in medical improvement and can therefore be handled as mailers. According to SSA’s contractor, the models’ last re-estimation in 2007 increased the accuracy of the models while allowing SSA to process over 25,000 additional cases as mailers, potentially saving the agency over $20 million by performing fewer full medical reviews. Second, greater model accuracy means the models are more likely to correctly assign high scores to cases most likely to demonstrate medical improvement, potentially leading to more medical cessations among beneficiaries who receive full medical reviews.

Although SSA acknowledged the importance of re-estimating its models again, it has yet to complete concrete actions toward doing so. In December 2015, SSA officials indicated that they were in the process of re-estimating the models, but the agency had not yet documented its efforts. In addition, according to SSA officials, the agency had not yet established plans to re-estimate the models on a regular basis. Without re-estimating its models on a regular basis, the agency risks losing the predictive accuracy of the models and could compromise its ability to use CDR resources efficiently.

SSA Considers Cost Savings Information to a Limited Extent When Prioritizing CDRs

Although SSA considers cost savings when prioritizing CDR cases, it does not do so in a manner that will maximize potential savings. According to federal internal control standards, federal agencies should ensure effective stewardship of public resources. The order in which SSA prioritizes beneficiary groups for CDRs generally aligns with the average savings per full medical review conducted for those groups in recent years. For example, the two highest priority groups—statutorily required reviews for age-18 redeterminations and low birth-weight children—have the highest average savings in foregone disability benefits as a result of full medical reviews (see fig. 4).

However, the priority ranking of beneficiary groups is not exclusively reliant upon the average savings achieved from conducting full medical reviews, because the agency takes other factors into consideration. For example, the average savings per full medical review of children receiving SSI benefits on the basis of low birth weight is higher than that of SSI children at age 18. Although reviews of 18-year-olds are automatically initiated 2 months before the beneficiary’s 18th birthday, not all reviews of low birth-weight children are conducted as scheduled. Specifically, in fiscal years 2009 through 2014, approximately 3,900 to 21,700 low birth- weight reviews were backlogged annually. In addition, although the SSI Other Children group has a higher average savings in foregone disability benefits than DI beneficiaries, SSI Other Children are prioritized after DI beneficiaries. In fiscal year 2013, SSA conducted more than twice as many full medical reviews on DI beneficiaries as on SSI Other Children beneficiaries and backlogged tens of thousands more full medical reviews for SSI Other Children than for DI beneficiaries. DI cases have been given priority over SSI Other Children partly to protect the Disability Insurance Trust Fund, which is the source of benefit payments to most DI recipients. However, recent action to address the solvency of the Disability Insurance Trust Fund somewhat mitigates this rationale. If SSA had switched the number of full medical reviews conducted for these groups in 2013, it is possible that the agency would have generated over $100 million more in savings.

Furthermore, in focusing on beneficiary groups, SSA’s prioritization process does not capture any differences among subgroups’ average savings in foregone disability benefits as a result of full medical reviews. For example, the DI group can be split into four subgroups, and the average lifetime savings per full medical review among these subgroups differed by as much as approximately $3,000 (or about 21 percent) in recent years (see fig. 5).

The aggregate mix of cases across different beneficiary groups reviewed during a fiscal year directly affects the agency’s total savings from conducting CDRs. If SSA were to shift the mix of discretionary cases it reviews among subgroups within beneficiary groups while still taking likelihood of medical improvement into account, it could realize greater savings. For example, shifting the mix of DI cases reviewed to better align with historical average savings performance among different DI subgroups would likely increase SSA’s total savings. Furthermore, we reported in 2012 that certain subgroups of SSI children beneficiaries, such as those with speech and language disorders as well as other mental impairments, demonstrated higher rates of initial cessation (i.e., prior to the appeals process) stemming from full medical review than other SSI children beneficiaries. As a result, reviews of these subgroups are more likely to contribute to savings for the agency than other non- required SSI children reviews. Without considering average savings at the beneficiary subgroup level, SSA may not be maximizing the total savings it realizes from conducting full medical reviews.

In addition to differences in savings from shifts in the aggregate mix of cases receiving full medical reviews, savings can also differ when comparing individual cases. When an individual’s benefits are ceased as the result of a CDR, the foregone benefits represent savings to the federal government. The amount of savings depends on various factors that affect how much SSA would have paid had the individual continued to receive disability benefits over time. These factors include the individual’s age, life expectancy, and monthly benefit payment. For example, two individuals who are different ages but are otherwise similar (e.g., they live in the same state, have the same benefit amount, and have the same likelihood of medical improvement as determined by SSA’s statistical models) would generate different expected savings from a CDR because the younger individual would likely receive benefits for a longer period of time. Similarly, two individuals who have different benefit amounts but are otherwise similar would generate different expected savings from a CDR because the individual with higher monthly benefits would likely receive greater total benefits over time. Prioritizing the CDR for the younger individual or the individual with a higher benefit level could result in greater savings for SSA. The simplified scenarios below illustrate this point; however, if SSA were to further incorporate such factors for individuals into its CDR prioritization process, a more complete set of inputs and assumptions would be needed (see fig. 6).

Despite the potentially substantial differences in savings among beneficiaries, SSA lacks a mechanism for factoring expected savings from benefit cessation into its CDR prioritization process on a case- specific basis. As a program integrity effort, CDRs are intended to assess the continued eligibility of beneficiaries to ensure that payments are made only to those individuals who should be receiving them, and SSA’s statistical models use an appropriate proxy of eligibility—potential for medical improvement—to prioritize cases for review. However, for beneficiaries with the same likelihood of medical improvement, SSA officials told us the agency does not further differentiate among individuals in the same beneficiary group on the basis of potential benefit savings. In SSA’s current prioritization process, the individuals depicted in the hypothetical scenarios in figure 6 would be equally likely to receive a full medical review because the agency does not consider the potential savings from individual cessations. As demonstrated in the analysis presented above, SSA could miss additional savings because it does not further consider beneficiaries’ potential savings when prioritizing cases for full medical review.

SSA Reviews a Sample of CDRs for Quality but Does Not Comprehensively Analyze and Report Errors

To assess the ability of DDSs to correctly apply policy and fully document CDRs, SSA performs quality reviews of a sample of continuances and cessations. The SSA quality reviewers who perform these reviews have guidance for checking specific elements of the decisions, and they are guided through a step-by-step computer program for conducting and documenting the reviews. SSA’s quality reviewers check CDRs for three types of errors: (1) decision errors, which include incorrect decisions or incomplete evidence to support a decision; (2) date errors, including incorrect benefit cessation dates; and (3) administrative errors. The reviewers return CDRs with decision errors to the DDS to perform additional work but generally correct those with date and administrative errors themselves.

SSA uses these quality reviews for multiple purposes. First, SSA estimates state, regional, and national CDR accuracy rates—the percentage of CDRs estimated to be accurate on the basis of a statistical sample. SSA also uses these accuracy rates to help monitor DDSs’ performance and shares this information with the DDSs. In addition, SSA uses the results of quality reviews to correct identified errors before the DDS decisions take effect.

Although SSA has reported high nationwide CDR accuracy rates in recent years, we identified shortcomings in how SSA prevents errors, defines and reports accuracy, and samples CDRs for quality review:

Preventing errors: Although SSA tracks the number and types of CDR decision errors and disseminates this information to state DDSs, it does not analyze the characteristics of CDR errors to help identify error trends associated with particular types of cases and address root causes. According to SSA officials, SSA probes CDR quality review data to uncover error trends by, for example, general groupings of impairments such as mental disorders. However, SSA does not analyze the data to uncover error trends for specific impairments, beneficiary types, or other characteristics. Federal internal control standards stipulate that management should assess the quality of performance over time and promptly resolve findings from audits and other reviews. According to officials, SSA does not analyze the characteristics of CDRs with errors because CDR accuracy rates are generally high and resources are limited. In addition, officials stated that SSA does not have sufficient data to do statistical modeling for such analyses. However, it is possible to analyze the characteristics of CDRs with errors by comparing relevant percentages without modeling, using data from multiple years if necessary. According to SSA and DDS officials, certain types of cases may be more error-prone than others. For example, cases involving mental impairments are thought to be relatively error-prone because they can be more challenging to document. In addition, officials reported challenges in conducting CDRs of low birth-weight children receiving SSI benefits because of the lack of documentation of other impairments they may have. However, because SSA has not analyzed the incidence of inaccurate CDRs by impairment, beneficiary type, or other characteristics, it cannot efficiently identify common types of errors and their root causes to help the DDSs take steps to prevent them.

Defining and reporting accuracy: In determining CDR accuracy rates, SSA does not include date errors, including incorrect cessation dates. As a result, decision makers do not have a complete picture of the CDR errors that affect disability payments. We have previously reported that to be useful, performance information must be complete, accurate, and valid, among other factors. However, per SSA regulation, the agency does not consider date errors when calculating accuracy rates because date errors do not affect the decision to cease or continue benefits, according to officials.28, 29 Nonetheless, such errors can affect the number of payments a beneficiary receives and thus SSA’s costs. For example, cessation date errors in a CDR can result in some beneficiaries receiving payments for longer or shorter periods of time, and thus accruing overpayments or underpayments for the period in question. Without including date errors in its reported accuracy rates, SSA does not provide its management and other decision makers and the public with complete information about errors that can affect disability payments. In addition, if SSA had counted date errors in CDR cessations, its accuracy rate for cessations in fiscal year 2014 would have fallen 1.6 percentage points from 95.5 percent to 93.9 percent. For some states, the effect of considering these errors is more pronounced. We examined SSA’s fiscal year 2014 cessation accuracy rate estimates and found that for 13 states, the accuracy rates would have decreased by at least 2 percentage points had SSA counted date errors; and, in one state the accuracy rate for cessations would have fallen 7 percentage points, from 95.4 percent to 88.4 percent.

SSA regulations define accuracy in this context as the percentage of cases that do not have to be returned to state DDSs for further development or correction of decisions based on evidence in the files. See 20 C.F.R. §§ 404.1643, 416.1043. In explaining the accuracy standard, SSA stated that its primary purpose was to improve the initial claims process and ensure that only properly entitled claimants receive disability benefits and that its approach was to specify outputs (i.e., performance accuracy), rather than specifying all inputs that could go into the standard.

SSA conducts stewardship reviews which examine the non-medical quality of various decisions related to benefit payments, including date designations. To do so, SSA reviews a sample of individuals receiving payments. In conducting and reporting on these reviews, however, SSA does not specifically focus on CDRs.

Sampling CDRs for quality review: SSA produces accuracy rate estimates by state DDS, but its sampling approach does not reliably and efficiently generate accuracy rate estimates for continuances and cessations separately in every state. According to federal guidance for developing statistical estimates, agencies should develop a sampling plan that is reflective of the level of detail and precision needed of the key estimates. CDR accuracy rates vary by state, and continuances are consistently more accurate than cessations. In fiscal year 2014, for example, the states’ estimated CDR accuracy rates varied from 92.4 percent to 99.8 percent. In the same year, the estimated accuracy rate for continuances was 98.3 percent nationwide, whereas the equivalent for cessations was 95.5 percent. Moreover, the range of accuracy rates across states is much larger for cessations. In fiscal year 2014, state-level accuracy rates for cessations ranged from 78.3 to 100 percent, while the accuracy rates for continuances ranged from 92 to 100 percent. To monitor CDR accuracy, SSA randomly selects about 70 continuances and 70 cessations for quality review each quarter from each state. Despite this sampling approach, SSA officials stated that their sampling design is not intended to produce precise estimates for continuations and cessations separately by state. However, precise accuracy rate estimates for continuations and cessations separately by state are needed to monitor DDS performance because of the difference in accuracy by decision type and because the state DDSs are managed separately.

In analyzing CDR workload and accuracy data, we found that SSA’s sampling approach produced accuracy rate estimates with margins of error that were consistently wide in seven states and consistently narrow in six states for either one type of CDR decision or both. A wide margin of error occurs when there are not enough CDR decisions in the sample to produce a reliable estimate. In these instances, such as in Vermont and Wyoming, we found SSA could not produce an estimate with a margin of error of plus or minus 5 percentage points using its current approach unless it sampled more CDR decisions. When SSA does not sample enough decisions and produces estimates with wide margins of error, decision makers may be relying on misleading information to assess CDR accuracy. Conversely, when SSA samples too many decisions and produces estimates with margins of error that are narrower than necessary to achieve reliable results, the agency may be wasting time and resources on such quality reviews.

SSA Reasonably Estimates CDR Cost Savings but Has Conducted Limited Sensitivity Analysis and Documentation

SSA’s annual process for estimating the cost savings of CDRs—the estimated ratio of federal program savings to costs for performing CDRs—involves many steps. To calculate the federal program savings generated by CDRs in a particular year, SSA estimates the present value of expected future benefits over 40 years that are saved as a result of the reviews. In forecasting these savings, SSA considers benefits from programs administered by SSA (i.e., DI and SSI) as well as programs that are not administered by SSA (i.e., Medicare and Medicaid). To do so, SSA estimates the number of people whose benefits would be ceased by CDRs and considers the effect of appeals in determining these estimates. SSA then estimates the savings associated with the cessations that are forecasted not to be overturned. It considers the age of individuals whose benefits would be ceased, and uses separate models to forecast savings from DI, SSI, Medicare, and Medicaid. SSA also forecasts and accounts for the number and timing of beneficiaries who would stop receiving disability benefits regardless of a CDR. Similarly, it forecasts and excludes the number of former beneficiaries who will successfully reapply for benefits after, for example, a new disabling condition arises. To generate the overall CDR cost savings rate (i.e., the amount saved for every dollar invested in CDRs) for a particular year, SSA divides the present value of future benefit savings by SSA’s actual cost of conducting CDRs during the relevant year (see fig. 7). To determine the cost of conducting CDRs, SSA considers its relevant expenses as well as those of the state DDSs.

We determined that SSA’s methods and assumptions for estimating CDR cost savings were reasonable, but, in certain respects, inconsistent with guidance for conducting cost savings analysis of federal programs. Specifically, we identified two areas of weakness:

Sensitivity analysis: According to federal guidance for conducting cost savings analysis, “major assumptions should be varied and net present value and other outcomes recomputed to determine how sensitive outcomes are to changes in the assumptions.” In reviewing SSA’s approach to estimating CDR cost savings for fiscal year 2012, we determined that SSA did not conduct sensitivity analysis of the overall cost savings rate. However, SSA separately performed some limited sensitivity analysis on savings from DI and SSI, which collectively represented about 82 percent of the savings that SSA forecasted for fiscal year 2012 CDRs. SSA calculated the effect of using inputs from fiscal year 2011, such as the average benefit amount, on the savings estimates for fiscal year 2012. However, SSA did not vary its assumptions (e.g., from optimistic to pessimistic) to generate a range of estimated savings. In addition, SSA has not reported the effect of changing its assumptions about SSI and DI savings on the overall cost savings estimate. According to an SSA official, doing sensitivity analysis on the reported cost savings estimate would require additional coordination with CMS about Medicare and Medicaid. However, SSA could conduct more complete sensitivity analysis by, for example, estimating a combined range of savings from DI and SSI without additional coordination. By not including a range of estimated savings for at least SSA’s programs, decision makers lack data on the extent to which the estimates could vary under different assumptions.

Documentation: According to federal guidance, models used in cost savings analysis should also be well documented and, where possible, available to facilitate independent review. SSA uses multiple complex models to estimate the cost savings of CDRs, but it has limited documentation about its methods, including data sources, assumptions, and limitations that factor into the estimates. For example, SSA has not documented how it estimates the number and timing of beneficiaries who stop receiving disability benefits because of a CDR, as well as the number of former beneficiaries who will successfully reapply for benefits. According to SSA officials, the agency has not yet documented the assumptions and procedures used to calculate CDR cost savings because of competing priorities and limited resources. Consequently, knowledge of SSA’s models is limited to the few SSA actuaries who work with them, and this information is not readily available or transferrable to others, including external reviewers.

Conclusions

In light of SSA’s current backlog of CDRs and the long-term financial challenges of the Disability Insurance Trust Fund, conducting timely, high- quality, and cost-effective CDRs is particularly important. In an effort to use its resources efficiently, SSA applies several sound practices to help prioritize CDRs. However, without further integrating comparative cost savings information in its prioritization process, SSA is missing an opportunity not only to focus on CDRs that are likely to save the federal government the most money, but also to more efficiently use its resources for program integrity work. Maximizing cost savings is not the only goal of this work, but it is an important criterion to help SSA prioritize CDRs and ensure that beneficiaries are being more effectively selected for review.

Further, although SSA has an extensive process for reviewing the quality of CDR decisions and a high overall accuracy rate, until the agency systematically uses available data to identify error-prone cases and root causes, it will be hard-pressed to prevent similar errors from recurring. In addition, absent tracking all meaningful errors that it identifies, such as date errors, the agency and other stakeholders lack an accurate sense of the true error rate of CDRs. Similarly, SSA’s current approach to sampling state decisions means the agency may be relying on misleading performance information for making management decisions.

SSA has demonstrated that CDRs are cost-effective, and it applies sound methods and assumptions for estimating cost savings. However, because it does not vary the assumptions that it uses to estimate a range of potential returns on investment for CDRs, the Congress and other stakeholders do not have complete information on the precision of these estimates and the extent to which they could vary with changes in assumptions. Finally, SSA’s limited documentation about its actuarial models leaves the agency vulnerable in the event of turnover of the few staff who use these models and challenges external reviewers’ ability to understand and audit the integrity of its models.

Recommendations for Executive Action

We recommend that the Acting Commissioner of Social Security: 1. Direct the Deputy Commissioner of Operations to further consider cost savings as part of its prioritization of full medical reviews. Such options could include considering the feasibility of prioritizing different types of beneficiaries on the basis of their estimated average savings and, as appropriate, integrating case-specific indicators of potential cost savings, such as beneficiary age and benefit amount, into its modeling or prioritization process. 2. Direct the Deputy Commissioner of Budget, Finance, Quality, and Management to complete a re-estimation of the statistical models that are used to prioritize CDRs and determine a plan for re-estimating these models on a regular basis to ensure that they reflect current conditions. 3. Direct the Deputy Commissioner of Budget, Finance, Quality, and Management to monitor the characteristics of CDR errors to identify potential root causes and report results to the Disability Determination Services. For example, SSA could analyze CDRs with and without errors to identify trends by impairment, beneficiary type, or other characteristics. 4. Direct the Deputy Commissioner of Budget, Finance, Quality, and Management to regularly track the number and rate of date errors, which can affect benefit payments (e.g., incorrect cessation dates), and consider including those errors in its reported CDR accuracy rates. 5. Direct the Deputy Commissioner of Budget, Finance, Quality, and Management to adjust its approach to sampling CDRs to efficiently produce reliable accuracy rate estimates for continuances and cessations separately in each state. 6. Direct the Chief Actuary to conduct sensitivity analyses on SSI and DI’s contributions to CDR cost savings estimates and report the results reflecting a range of inputs (e.g., from optimistic to pessimistic). 7. Direct the Chief Actuary to better document the methods including data sources, assumptions, and limitations that factor into its estimates of CDR cost savings.

Agency Comments and Our Evaluation

We provided a draft of this report to SSA for review and comment, and its written comments are reproduced as appendix II in this report. SSA stated that it generally agreed with our recommendations, but noted that the level of program integrity funding it receives has affected the number of CDRs performed annually and, at times, the size of the CDR backlog. SSA also noted that our report implied that SSA is not focused on the CDRs that are most likely to save the government money and that the report did not fully convey the accuracy of the agency’s statistical models and its treatment of CDR errors. We agree that SSA’s CDR process is generally designed to use its program integrity resources efficiently and note in our report that SSA applies several sound practices to help prioritize CDRs including annually assessing its statistical models’ performance. However, we maintain additional steps are warranted to ensure ongoing accuracy of the models and to maximize potential savings. We also note that the agency has a process in place to identify and evaluate errors, but maintain that additional steps could be taken to systematically analyze error trends and uncover root causes of errors. SSA agreed with four of our recommendations, partially agreed with one recommendation, and disagreed with two recommendations. The agency’s specific concerns and our responses are described below:

Regarding our recommendation to further consider cost savings as part of its prioritization of full medical reviews, SSA partially agreed. Although SSA agreed that it could look for ways to improve its return on conducting CDRs, it also stated that its statistical models and prioritization process already do much of what we recommend. For example, SSA stated that age is already a strong variable in its statistical models. However, these models predict medical improvement and are not designed to take expected cost savings into account. We continue to believe that to maximize expected cost savings SSA could refine its prioritization process by factoring in actuarial considerations. For example, SSA could consider the effect of a beneficiary’s age on expected costs savings, in addition to its existing statistical models that account for the effect of age on the likelihood of medical improvement.

Regarding our recommendation to complete a re-estimation of the statistical models that are used to prioritize CDRs and determine a plan for re-estimating these models on a regular basis, SSA agreed and stated that it plans to complete its ongoing re- estimation and to document a process for determining when to re- estimate the models in the future.

Regarding our recommendation to monitor the characteristics of CDRs with errors to identify root causes, SSA agreed and stated that it reports all errors to the relevant DDS for corrective action. SSA further stated that its identification of root causes is limited by the relatively few reviewed CDRs that have errors. However, in fiscal year 2014 as an example, SSA identified over 600 CDRs with errors. Although these CDRs make up a small percentage of the CDRs reviewed by SSA that year, the agency could analyze the characteristics of CDRs with errors by comparing relevant percentages without modeling. In addition, SSA could combine data from multiple years if it determined that considering more CDRs with errors would be helpful.

Regarding our recommendation to track the number and rate of date errors and consider including them in its reported CDR accuracy rates, SSA disagreed and stated that, per SSA regulation, the agency does not consider date errors when calculating accuracy rates because date errors do not affect the decision to cease or continue benefits. SSA also stated its stewardship reviews examine the non-medical quality of benefit payment decisions. However, these reviews are not focused on CDRs, and SSA does not report results from them for CDRs specifically. SSA also explained that it does not track the number and rate of date errors because they are infrequent. However, SSA’s regulations do not prevent the agency from tracking date errors, and until it does, SSA cannot definitively determine the frequency of these errors. In addition, we found that considering date errors substantially reduced some states’ estimated CDR accuracy rates. Without tracking these errors, SSA cannot assess their effect and consider whether including them in its reported CDR accuracy rates has merit.

Regarding our recommendation to adjust its approach to sampling CDRs to efficiently produce reliable accuracy rate estimates for continuances and cessations separately in each state, SSA disagreed and stated that some states do not generate enough CDR decisions, particularly cessations, to generate statistically valid samples. However, for states with CDR samples that are consistently too small to produce reliable results, SSA could, for example, pool decisions from more months than it currently does to generate statistically valid samples by state. Conversely, for states with CDR samples that are consistently larger than necessary to efficiently achieve reliable results, SSA could, for example, reduce sample sizes. Because CDR accuracy rates vary by state and cessations are consistently less accurate than continuances, we maintain that SSA should adjust its approach to sampling CDRs.

Regarding our recommendation to conduct sensitivity analyses on SSI and DI’s contributions to CDR cost savings estimates and report the results reflecting a range of inputs, SSA agreed and stated that it will expand on its current sensitivity analyses as time and resources permit.

Regarding our recommendation to better document the methods that factor into its estimates of CDR cost savings, SSA agreed and stated that it will improve and expand its existing documentation as time and resources permit.

SSA also provided technical comments, which we incorporated into the report as appropriate.

As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees and the Acting Commissioner of Social Security. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7215 or bertonid@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

The objectives of this report were to examine (1) how the Social Security Administration (SSA) selects which Continuing Disability Reviews (CDR) to conduct, (2) the extent to which SSA reviews the quality of CDR decisions, and (3) how SSA calculates cost savings from CDRs.

CDR Prioritization

To evaluate how SSA selects which CDRs to conduct, we reviewed relevant federal laws and interviewed SSA officials from the agency’s offices of Public Service and Operations Support, Budget, and Quality Improvement. We also reviewed internal SSA documents on the agency’s approach to prioritizing and processing CDRs, including its use of statistical models.

To evaluate the statistical models that SSA uses to help prioritize CDRs, we reviewed internal SSA documents about the statistical models, including lists of variables, tests of model fit, and detailed technical reports provided by the external contractor that last re-estimated the models. The technical reports provided by the contractor explained how each of the models was developed and tested, including the data sources and variables that were considered and used, how SSA impairment codes were aggregated into impairment groups, and how the functional form and interaction terms were identified. In 2007, the contractor compared the performance of models for adult beneficiaries against that of SSA’s prior models—which had been estimated last in 2005—and only specifications and variables that improved model performance were retained. We evaluated the technical specifications and tests of model fit and predictive accuracy for the models for each of the beneficiary cohorts.

To estimate the proportion of full medical reviews completed because of their score from SSA’s statistical models, we analyzed CDR annual report data from fiscal years 2003 through 2013. We assessed the reliability of these data by reviewing related documentation and interviewing knowledgeable agency officials, and we found these data sufficiently reliable for our purposes. To assist with the analysis, we obtained information from SSA’s Office of Public Service and Operations Support on the number of SSA policy priority cases processed annually. From the total number of full medical reviews completed during a fiscal year, we subtracted completed full medical reviews that were prioritized because they were statutorily required (e.g., reviews of SSI children at age 18 and reviews of children under 1 year old who are receiving SSI benefits due in part to low birth weight) and because of SSA policy (e.g., mailers with certain responses and first-time reviews for beneficiaries in the “medical improvement expected” diary category). To avoid potential double- counting, we did not count the sample of approximately 60,000 cases that SSA initiates annually to validate its statistical models among the policy priority cases because the sample consists of cases across all beneficiary groups, including statutorily required cases. As a result, our calculations may underestimate the number of required priority cases and overestimate the number of cases selected because of the statistical models.

To illustrate the impact of further incorporating cost savings into the prioritization process, we obtained and analyzed data from SSA on the average savings per full medical review by beneficiary group in each of fiscal years 2012 and 2013. Using weighted averages of these data, we calculated the average savings for fiscal years 2012 and 2013 separately and combined. We also developed two hypothetical scenarios that pair near-identical beneficiaries with different ages or monthly benefit payments to demonstrate the effect of considering cost savings on an individual basis. We used information from the fiscal year 2013 statistical supplements on the DI and SSI programs to develop reasonable ages and benefit levels for the hypothetical beneficiaries. We calculated the expected savings in foregone benefits after cessation for each beneficiary by multiplying the monthly benefit by the number of months until the beneficiaries would have aged out of the disability programs.

Quality Reviews of CDR Decisions

To understand the process that SSA uses to review the quality of CDR decisions, we reviewed relevant federal laws, regulations, policies, and procedures; interviewed SSA officials about these policies and procedures; and analyzed SSA’s CDR workload and decision accuracy data. This work included reviewing documentation of the Disability Case Adjudication and Review System (DICARS), the software program in which SSA completes quality reviews. We interviewed SSA officials about how the quality reviews are conducted and how the agency uses the results, and compared the agency’s policies and procedures to generally accepted statistical practices and federal internal control standards. We also interviewed state Disability Determination Services officials about factors that challenge CDR quality. We assessed the reliability of SSA’s CDR workload and decision accuracy data by performing data testing, reviewing related documentation, and interviewing agency officials, and we found the data to be sufficiently reliable for the purposes of this review.

To evaluate the extent to which SSA reviews the quality of CDR decisions, we analyzed SSA’s CDR workload and decision accuracy data to determine whether its method for sampling CDRs and estimating CDR accuracy are consistent with generally accepted statistical practices and SSA’s reporting goals. SSA reports accuracy rate estimates for each state every month using the most recent 3 or 6 months of quality review data. Its goal is to produce estimates with 95 percent confidence intervals that are within plus or minus 5 percentage points of the estimate. We analyzed SSA’s CDR workload and accuracy data, consistent with SSA’s sampling and reporting methods, from June 2013 through April 2015. Specifically, we identified the number of continuance and cessation determinations in each state, the District of Columbia, and Puerto Rico; SSA’s accuracy rate estimates for these determinations for each 6-month period; and the 95 percent confidence interval margins of error for each estimate. To identify states that had estimates that consistently do not achieve the reporting goals, we compared the workload, accuracy rate estimates, and margins of errors to those specified in SSA’s sample design and reporting goals. We calculated margins of error for estimates in which SSA did not provide them. We used a statistical formula that produces appropriate margins of error, including when standard formula do not apply, to determine and examine a margin of error for all estimates. We chose this formula because in some cases CDR accuracy is so high or the sampling fraction is so large that the standard statistical formula used for these purposes would compute margins of error that are not appropriate, such as those resulting in confidence intervals above 100 percent. Since CDR accuracy cannot be greater than 100 percent, the standard formula is not appropriate. We also analyzed CDR workload data from fiscal years 2001 through 2014 to inform our evaluation of SSA’s sampling method.

To determine the effect of date errors on accuracy rates, we analyzed data about CDR date errors and cessations. We considered data from fiscal years 2010 through 2014 to determine the frequency with which date errors occur. We calculated fiscal year 2014 cessation accuracy rate estimates for each state by combining the number of cessation decision errors and the number of date errors and dividing the total by the number of cessations SSA reviewed. SSA’s date error data were not broken down by decision type (i.e., continuance or cessation), but we assigned these errors to cessations because of input from SSA and our corroborating analysis. According to SSA officials, the most common date error on a CDR is a cessation date error and other date errors, such as incorrectly inputting an onset date, can occur in a cessation or continuance but are rare. Our analysis corroborated this information. For example, in fiscal year 2014, of 127 date errors identified in CDRs nationally, 125 of them were cessation date errors. In addition, we calculated margins of error for each estimate to assess the statistical reliability of each estimate. We used the statistical formula that produces appropriate margins of error, consistent with our approach to calculating margins of error in our analysis of SSA’s sampling method.

CDR Cost-Savings Estimate

To evaluate SSA’s approach to calculating cost savings from CDRs, we compared SSA’s estimation process to actuarial standards of practice and federal guidelines for benefit-cost analyses of federal programs. Specifically, we interviewed SSA actuaries about the models and methods they used to perform the cost-savings calculation for fiscal year 2012. We also reviewed portions of the programming code related to these models to corroborate the information from the actuaries. In addition, we examined the assumptions that SSA uses to calculate the present value of future benefits saved from ceasing a person’s benefits as the result of a CDR by examining where and how SSA incorporates assumptions into its calculation process. Finally, we reviewed the fiscal year 2012 CDR cessation data and information SSA provided to the Centers for Medicare & Medicaid Services (CMS) that informed CMS’s estimates of Medicare and Medicaid savings resulting from CDR cessations, but we did not review CMS’s models.

We conducted this performance audit from December 2014 to February 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Social Security Administration

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Erin Godtland (Assistant Director), Joel Green (Analyst-in-Charge), Susan Aschoff, James Bennett, Grace Cho, Alexander Galuten, Emei Li, Isabella Johnson, Rhiannon Patterson, Almeta Spencer, Daren Sweeney, Jeff Tessin, Kristen Timko, Frank Todisco, Sonya Vartivarian, and Shana Wallace made key contributions to this report.