Social Security Disability: SSA Could Increase Savings by Refining Its Selection of Cases for Disability Review
Published: 2016-02-11T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: To help ensure that only eligible individuals receive disability benefits, SSA conducts periodic CDRs to assess beneficiaries' medical condition. CDRs have historically saved the government money. However, in recent years, SSA has had difficulty conducting timely CDRs resulting in a backlog of over 900,000 CDRs in fiscal year 2014. With this backdrop, GAO was asked to study SSA's ability to conduct and manage timely, high-quality CDRs.
This report evaluates, among other things, how SSA selects which CDRs to conduct and the extent to which SSA reviews the quality of CDR decisions.
GAO analyzed CDR data for fiscal years 2003 through 2013 (the most recent year for which complete data were available); assessed SSA's models used to prioritize CDRs; reviewed relevant federal laws, regulations, and SSA documentation about CDR prioritization and accuracy review procedures; and interviewed SSA and state Disability Determination Services officials.
section_title: What GAO Found
paragraphs: The Social Security Administration (SSA) selects cases for continuing disability reviews (CDR) using several inputs, but it does not do so in a manner that maximizes potential savings. SSA first prioritizes CDRs required by law or agency policy such as those for children under 1 year old who are receiving benefits due in part to low birth weight. Then SSA uses statistical models to identify the remaining CDRs to be conducted each year. The models also determine which cases will receive an in-depth review of medical records by the Disability Determination Services—the state agencies that conduct CDRs—versus a lower-cost questionnaire sent directly to the beneficiary. As shown in the figure below, a growing number of cases have been set aside for future review (backlogged) over the last 10 years. Although SSA somewhat considers potential cost savings when selecting cases for in-depth reviews, its approach does not maximize potential savings for the government. For example, estimated average savings from conducting CDRs are higher for some groups of Disability Insurance (DI) beneficiaries than others, but SSA's selection process does not differentiate among these groups. As a result, it may be missing opportunities to efficiently and effectively use federal resources.
SSA reviews a sample of CDRs for quality, but its analysis and reporting of errors is not comprehensive. Specifically, SSA randomly selects CDR decisions to check for a variety of potential errors. For example, SSA regularly monitors and reports on the frequency of errors that affect whether benefits are continued or ceased. However, contrary to federal internal control standards, SSA does not systematically analyze errors to detect and address root causes. Consequently, SSA lacks information that could help improve the quality of the reviews conducted by the Disability Determination Services. Further, in determining CDR accuracy rates, SSA does not count date errors, including incorrect cessation dates, which can affect disability benefit payments. As a result, decision makers do not have a complete picture of the CDR errors that affect disability payments.
section_title: What GAO Recommends
paragraphs: GAO recommends SSA, among other things, further consider cost savings as part of its prioritization of CDRs, analyze the root causes of CDRs with errors, and track date errors. SSA agreed with most of GAO's recommendations, but disagreed that there is a need to track date errors and to adjust its approach to sampling CDRs for quality review. GAO maintains actions are warranted and feasible as discussed in the report.

section_title: Letter
paragraphs: To help ensure that only eligible individuals receive disability benefits, the  Social Security Administration (SSA) is generally required to conduct  periodic continuing disability reviews (CDR) for certain individuals who  are receiving benefits under one or both of its disability programs:  Disability Insurance (DI) and Supplemental Security Income (SSI). These  reviews assess whether individuals remain eligible for benefits based on  several criteria, including their current medical condition and ability to  work. CDRs are an important check on the integrity of the disability  programs and have consistently saved the government more money than  they cost to conduct. However, in recent years, SSA has had difficulty  completing timely reviews, and, as a result, amassed a backlog of over  900,000 CDRs by the end of fiscal year 2014. In addition, the Disability  Insurance Trust Fund has faced financial challenges in recent years.
With this backdrop, concerns have been raised about SSA’s ability to  conduct and manage timely, high-quality CDRs. You asked us to study  this topic. This report examines (1) how SSA selects which CDRs to  conduct, (2) the extent to which SSA reviews the quality of CDR  decisions, and (3) how SSA calculates cost savings from CDRs.
To examine SSA’s process for selecting which cases receive a CDR, we  reviewed legal requirements for conducting certain types of CDRs and  analyzed data on the number and type of CDRs conducted for fiscal  years 2003 through 2013 (the most recent year for which complete data  were available). We also reviewed documentation about how SSA  prioritizes which CDRs to conduct each year or backlog for future review.  Specifically, we examined the statistical models SSA uses to help  prioritize CDRs, and interviewed SSA officials about these practices. To  evaluate the extent to which SSA reviews the quality of CDR decisions,  we reviewed relevant federal laws, regulations, and program policies and  analyzed SSA’s monthly CDR accuracy data by state for June 2013  through April 2015. We also interviewed SSA officials and reviewed  documentation about how SSA conducts CDR quality reviews and uses  the results of these reviews to prevent errors. To evaluate SSA’s  approach to calculating cost savings from CDRs, we interviewed SSA  actuaries and other officials about their process for estimating cost  savings and reviewed their documentation and models. We assessed the  reliability of the CDR case-processing and accuracy data used for our  analyses by, among other things, reviewing related documentation and  testing for obvious errors. We determined that the data were sufficiently  reliable for assessing how SSA prioritizes CDRs and reviews their quality.  We compared SSA’s activities to federal internal control standards and  generally accepted statistical and actuarial practices where relevant.
We conducted this performance audit from December 2014 to February  2016 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: SSA administers two disability programs that provide monthly cash  benefits to eligible individuals: (1) Disability Insurance (DI) for individuals  (and their dependents) who have paid into the Disability Insurance Trust  Fund and (2) Supplemental Security Income (SSI) for low-income  individuals. To be eligible for DI or SSI benefits based on a disability, an  individual must have a medically determinable physical or mental  impairment that (1) prevents the individual from engaging in any  substantial gainful activity and (2) is expected to result in death or has  lasted or is expected to last at least 1 year.
Federal law generally requires CDRs to be conducted at least once every  3 years for all DI beneficiaries whose disabilities are not considered  permanent, and at intervals determined appropriate by SSA for those  whose impairments are considered permanent. For SSI, federal law  generally requires SSA to (1) conduct CDRs for infants during their first  year of life if they are receiving SSI benefits due in part to low birth  weight, and at least once every 3 years for SSI children under age 18 if  their impairments are considered likely to improve, and (2) review the  cases of all SSI children beginning on their 18th birthday to determine  whether they are eligible for disability benefits under adult disability  criteria. SSA may waive the requirement to conduct periodic legislatively- required CDRs on a state-by-state basis. SSA may also conduct CDRs  that are not required by law as it deems appropriate.
subsections: section_title: CDR Process
paragraphs: SSA contracts with state Disability Determination Services (DDS)  agencies to initially determine whether applicants are disabled and to  conduct periodic CDRs to determine whether beneficiaries continue to be  disabled. DDS examiners assess whether individuals are eligible for  benefits based on several criteria, including their current medical  condition and ability to work. At the time beneficiaries enter the DI or SSI  programs or continue their benefits following a CDR, a DDS examiner  determines beneficiaries’ due dates for a subsequent CDR based on their  potential for medical improvement. Beneficiaries classified as “medical  improvement expected” are generally scheduled for a CDR within 6 to 18  months, beneficiaries classified as “medical improvement possible” are  scheduled once every 3 years, and beneficiaries classified as “medical  improvement not expected” are scheduled once every 5 to 7 years.
To cost-effectively manage its CDR workload, SSA conducts CDRs in  different ways. In general, beneficiaries with a high likelihood of medical  improvement are referred for a full medical review—an in-depth  assessment of a beneficiary’s medical and vocational status.  Beneficiaries with a low likelihood of medical improvement are, at least  initially, sent a questionnaire known as a mailer. If beneficiaries respond  to a mailer in certain ways, SSA may refer these individuals for a full  medical review. In contrast to mailers, full medical reviews are more labor  intensive and expensive. Full medical reviews result in a decision to  either cease or continue an individual’s benefits. In fiscal year 2013, the  cessation rate for CDRs involving full medical reviews was about 19  percent, whereas the cessation rate for all CDRs including mailers was  about 5 percent. Each year, SSA allocates a portion of its program  integrity budget to CDRs, which affects the number of full medical reviews  and mailers that the agency initiates during the year. When the number of  cases due for a CDR exceeds SSA’s capacity to conduct full medical  reviews and mailers, the cases not initiated during the year are  considered backlogged for future review.
The number of CDRs completed as full medical reviews, as mailers only,  or backlogged varied during fiscal years 2003 through 2013 (see fig. 1).  After the authority for special funding to process CDRs expired in fiscal  year 2002, backlogged CDRs increased from about 100,000 cases in  fiscal year 2003 to more than 1 million in fiscal year 2007, reaching a  peak of nearly 1.5 million in fiscal year 2009. At the same time, the  number of full medical reviews fell from nearly 670,000 in fiscal year 2003  to less than 190,000 in fiscal year 2007 before rebounding to nearly  429,000 in fiscal year 2013.
SSA estimates the accuracy rate of CDRs and separately estimates the  cost savings that result from CDRs. In addition to annually reporting the  nationwide accuracy rate of all CDRs to the Congress, SSA internally  tracks CDR accuracy rates by state and generates estimates for the  accuracy of cessations and continuances, separately as well as  combined. For fiscal year 2013, SSA reported an accuracy rate of 97.2  percent for CDR decisions. For the same year, SSA reported an  estimated ratio of federal program savings to costs for performing CDRs  as $15 to $1. Savings from CDRs include federal benefits that would be  paid to individuals were it not for a CDR that resulted in a cessation. Such  benefits include those from Medicare and Medicaid because in certain  situations individuals’ eligibility for DI or SSI confers eligibility for these  other programs.
section_title: SSA Prioritizes CDRs Using Several Inputs but Does Not Fully Incorporate Potential Cost-Savings
subsections: section_title: SSA Uses a Variety of Inputs, Including Legal Requirements and Statistical Models, to Prioritize CDRs
paragraphs: Because SSA does not complete all CDRs as scheduled due to  competing priorities and existing resources, the agency must decide  which cases will receive a full medical review. SSA uses a range of inputs  to prioritize which CDRs to conduct, such as:
Statutory requirements: Legal requirements to review SSI children  beginning at age 18 to determine if they are eligible for benefits under  adult disability criteria, and reviews of children up to 1 year old who  are receiving SSI benefits due in part to low birth weight.
SSA policies: Rules established by SSA to guide prioritization. For  example, SSA prioritizes cases with particular responses to its mailers  and cases with a “medical improvement expected” designation that  are coming up for review for the first time.
Statistical models: A set of statistical models that score each case  according to the likelihood of medical improvement, typically the sole  criterion for ceasing benefits.
SSA’s prioritization process determines which CDRs are initiated and in  what form: full medical review or mailer. To begin, SSA initiates full  medical reviews of cases that fall into two high-priority categories: first,  statutory requirements, and then SSA policies. Once full medical reviews  of all high-priority cases are initiated, SSA prioritizes the remaining cases  by using its statistical models. Full medical reviews of cases with the  highest scores (i.e., highest likelihood of medical improvement) are  initiated as resources permit, first by beneficiary group (i.e., DI, SSI  children, SSI adults) and then by the statistical scores of cases within the  group (see fig. 2). Cases with lower scores (i.e., lower likelihood of  medical improvement) receive a mailer or are backlogged for future  review.
The extent to which the statistical models have been used to select cases  for full medical reviews has varied by year, but the models have been  consistently used for determining who receives mailers. Specifically, we  found that the extent to which SSA’s statistical models were used to  select cases for full medical review was related to the combination of  budget fluctuations for CDRs, SSA’s statutory requirements, and agency  policies. We estimate that the statistical models were the basis for  selecting 11 to 60 percent of full medical reviews completed annually in  fiscal years 2003 through 2013 (see fig. 3). In contrast, SSA has used the  models consistently since 1993 to determine which cases should receive  a mailer.
section_title: SSA Tests the Accuracy of Its Models but Has Not Updated Them to Reflect Changing Conditions in Recent Years
paragraphs: Although SSA annually assesses its models’ performance, the agency  has not updated the models since 2007. The models’ effectiveness  depends on their ability to accurately predict beneficiaries’ likelihood of  medical improvement. To test the accuracy of the models, SSA conducts  an annual validation process using a sample of completed cases to  evaluate how well the models predicted medical improvement.  According to SSA officials, the model validation process has shown that  the models’ accuracy in predicting medical improvement has not  degraded substantially in recent years.
In addition to model validation, in the past SSA has conducted periodic  re-estimation of its statistical models to help ensure that they are up-to- date. In re-estimating its models, SSA updates the relationship between  existing variables and medical improvement and tests whether new  variables should be included. Re-estimation is particularly important when  advances in medicine and assistive technology affect people’s ability to  work. In recent years, SSA has changed its classification of certain  beneficiaries’ impairments to reflect advances in medical knowledge. For  example, in 2015, SSA revised its codes for cancer in light of new  diagnoses and treatments. Because these codes can appear as variables  in the statistical models, it is possible that the models are no longer  accurately capturing the effect of cancer-related impairments on the  likelihood of medical improvement. Although SSA officials believe that  some advances would not markedly affect the accuracy of the models,  the agency has not completed a re-estimation to confirm the effect of  such changes. In addition, demographic changes in the underlying  population of disability beneficiaries, which has grown substantially in  recent years due in part to baby boomers reaching their disability-prone  years, could also affect the accuracy of the models. The contractor that  SSA hired to handle its last model re-estimation in 2007 provided SSA  with a set of programs that would allow the agency to re-estimate the  models in-house. Regular re-estimation and updating of predictive models  is a best practice, and the contractor anticipated that SSA would do so at  least every 3 years.
Model accuracy leads to savings for SSA in two ways. First, model  accuracy is important for identifying cases that are unlikely to result in  medical improvement and can therefore be handled as mailers. According  to SSA’s contractor, the models’ last re-estimation in 2007 increased the  accuracy of the models while allowing SSA to process over 25,000  additional cases as mailers, potentially saving the agency over $20 million  by performing fewer full medical reviews. Second, greater model  accuracy means the models are more likely to correctly assign high  scores to cases most likely to demonstrate medical improvement,  potentially leading to more medical cessations among beneficiaries who  receive full medical reviews.
Although SSA acknowledged the importance of re-estimating its models  again, it has yet to complete concrete actions toward doing so. In  December 2015, SSA officials indicated that they were in the process of  re-estimating the models, but the agency had not yet documented its  efforts. In addition, according to SSA officials, the agency had not yet  established plans to re-estimate the models on a regular basis. Without  re-estimating its models on a regular basis, the agency risks losing the  predictive accuracy of the models and could compromise its ability to use  CDR resources efficiently.
section_title: SSA Considers Cost Savings Information to a Limited Extent When Prioritizing CDRs
paragraphs: Although SSA considers cost savings when prioritizing CDR cases, it  does not do so in a manner that will maximize potential savings.  According to federal internal control standards, federal agencies should  ensure effective stewardship of public resources. The order in which  SSA prioritizes beneficiary groups for CDRs generally aligns with the  average savings per full medical review conducted for those groups in  recent years. For example, the two highest priority groups—statutorily  required reviews for age-18 redeterminations and low birth-weight  children—have the highest average savings in foregone disability benefits  as a result of full medical reviews (see fig. 4).
However, the priority ranking of beneficiary groups is not exclusively  reliant upon the average savings achieved from conducting full medical  reviews, because the agency takes other factors into consideration. For  example, the average savings per full medical review of children receiving  SSI benefits on the basis of low birth weight is higher than that of SSI  children at age 18. Although reviews of 18-year-olds are automatically  initiated 2 months before the beneficiary’s 18th birthday, not all reviews of  low birth-weight children are conducted as scheduled. Specifically, in  fiscal years 2009 through 2014, approximately 3,900 to 21,700 low birth- weight reviews were backlogged annually. In addition, although the SSI  Other Children group has a higher average savings in foregone disability  benefits than DI beneficiaries, SSI Other Children are prioritized after DI  beneficiaries. In fiscal year 2013, SSA conducted more than twice as  many full medical reviews on DI beneficiaries as on SSI Other Children  beneficiaries and backlogged tens of thousands more full medical reviews  for SSI Other Children than for DI beneficiaries. DI cases have been  given priority over SSI Other Children partly to protect the Disability  Insurance Trust Fund, which is the source of benefit payments to most DI  recipients. However, recent action to address the solvency of the  Disability Insurance Trust Fund somewhat mitigates this rationale. If  SSA had switched the number of full medical reviews conducted for these  groups in 2013, it is possible that the agency would have generated over  $100 million more in savings.
Furthermore, in focusing on beneficiary groups, SSA’s prioritization  process does not capture any differences among subgroups’ average  savings in foregone disability benefits as a result of full medical reviews.  For example, the DI group can be split into four subgroups, and the  average lifetime savings per full medical review among these subgroups  differed by as much as approximately $3,000 (or about 21 percent) in  recent years (see fig. 5).
The aggregate mix of cases across different beneficiary groups reviewed  during a fiscal year directly affects the agency’s total savings from  conducting CDRs. If SSA were to shift the mix of discretionary cases it  reviews among subgroups within beneficiary groups while still taking  likelihood of medical improvement into account, it could realize greater  savings. For example, shifting the mix of DI cases reviewed to better align  with historical average savings performance among different DI  subgroups would likely increase SSA’s total savings. Furthermore, we  reported in 2012 that certain subgroups of SSI children beneficiaries,  such as those with speech and language disorders as well as other  mental impairments, demonstrated higher rates of initial cessation (i.e.,  prior to the appeals process) stemming from full medical review than  other SSI children beneficiaries. As a result, reviews of these subgroups  are more likely to contribute to savings for the agency than other non- required SSI children reviews. Without considering average savings at the  beneficiary subgroup level, SSA may not be maximizing the total savings  it realizes from conducting full medical reviews.
In addition to differences in savings from shifts in the aggregate mix of  cases receiving full medical reviews, savings can also differ when  comparing individual cases. When an individual’s benefits are ceased as  the result of a CDR, the foregone benefits represent savings to the  federal government. The amount of savings depends on various factors  that affect how much SSA would have paid had the individual continued  to receive disability benefits over time. These factors include the  individual’s age, life expectancy, and monthly benefit payment. For  example, two individuals who are different ages but are otherwise similar  (e.g., they live in the same state, have the same benefit amount, and  have the same likelihood of medical improvement as determined by  SSA’s statistical models) would generate different expected savings from  a CDR because the younger individual would likely receive benefits for a  longer period of time. Similarly, two individuals who have different benefit  amounts but are otherwise similar would generate different expected  savings from a CDR because the individual with higher monthly benefits  would likely receive greater total benefits over time. Prioritizing the CDR  for the younger individual or the individual with a higher benefit level could  result in greater savings for SSA. The simplified scenarios below illustrate  this point; however, if SSA were to further incorporate such factors for  individuals into its CDR prioritization process, a more complete set of  inputs and assumptions would be needed (see fig. 6).
Despite the potentially substantial differences in savings among  beneficiaries, SSA lacks a mechanism for factoring expected savings  from benefit cessation into its CDR prioritization process on a case- specific basis. As a program integrity effort, CDRs are intended to assess  the continued eligibility of beneficiaries to ensure that payments are made  only to those individuals who should be receiving them, and SSA’s  statistical models use an appropriate proxy of eligibility—potential for  medical improvement—to prioritize cases for review. However, for  beneficiaries with the same likelihood of medical improvement, SSA  officials told us the agency does not further differentiate among  individuals in the same beneficiary group on the basis of potential benefit  savings. In SSA’s current prioritization process, the individuals depicted in  the hypothetical scenarios in figure 6 would be equally likely to receive a  full medical review because the agency does not consider the potential  savings from individual cessations. As demonstrated in the analysis  presented above, SSA could miss additional savings because it does not  further consider beneficiaries’ potential savings when prioritizing cases for  full medical review.
section_title: SSA Reviews a Sample of CDRs for Quality but Does Not Comprehensively Analyze and Report Errors
paragraphs: To assess the ability of DDSs to correctly apply policy and fully document  CDRs, SSA performs quality reviews of a sample of continuances and  cessations. The SSA quality reviewers who perform these reviews have  guidance for checking specific elements of the decisions, and they are  guided through a step-by-step computer program for conducting and  documenting the reviews. SSA’s quality reviewers check CDRs for three  types of errors: (1) decision errors, which include incorrect decisions or  incomplete evidence to support a decision; (2) date errors, including  incorrect benefit cessation dates; and (3) administrative errors. The  reviewers return CDRs with decision errors to the DDS to perform  additional work but generally correct those with date and administrative  errors themselves.
SSA uses these quality reviews for multiple purposes. First, SSA  estimates state, regional, and national CDR accuracy rates—the  percentage of CDRs estimated to be accurate on the basis of a statistical  sample. SSA also uses these accuracy rates to help monitor DDSs’  performance and shares this information with the DDSs. In addition, SSA  uses the results of quality reviews to correct identified errors before the  DDS decisions take effect.
Although SSA has reported high nationwide CDR accuracy rates in recent  years, we identified shortcomings in how SSA prevents errors, defines  and reports accuracy, and samples CDRs for quality review:
Preventing errors: Although SSA tracks the number and types of  CDR decision errors and disseminates this information to state DDSs,  it does not analyze the characteristics of CDR errors to help identify  error trends associated with particular types of cases and address  root causes. According to SSA officials, SSA probes CDR quality  review data to uncover error trends by, for example, general  groupings of impairments such as mental disorders. However, SSA  does not analyze the data to uncover error trends for specific  impairments, beneficiary types, or other characteristics. Federal  internal control standards stipulate that management should assess  the quality of performance over time and promptly resolve findings  from audits and other reviews. According to officials, SSA does not  analyze the characteristics of CDRs with errors because CDR  accuracy rates are generally high and resources are limited. In  addition, officials stated that SSA does not have sufficient data to do  statistical modeling for such analyses. However, it is possible to  analyze the characteristics of CDRs with errors by comparing relevant  percentages without modeling, using data from multiple years if  necessary. According to SSA and DDS officials, certain types of  cases may be more error-prone than others. For example, cases  involving mental impairments are thought to be relatively error-prone  because they can be more challenging to document. In addition,  officials reported challenges in conducting CDRs of low birth-weight  children receiving SSI benefits because of the lack of documentation  of other impairments they may have. However, because SSA has not  analyzed the incidence of inaccurate CDRs by impairment, beneficiary  type, or other characteristics, it cannot efficiently identify common  types of errors and their root causes to help the DDSs take steps to  prevent them.
Defining and reporting accuracy: In determining CDR accuracy  rates, SSA does not include date errors, including incorrect cessation  dates. As a result, decision makers do not have a complete picture of  the CDR errors that affect disability payments. We have previously  reported that to be useful, performance information must be complete,  accurate, and valid, among other factors. However, per SSA  regulation, the agency does not consider date errors when calculating  accuracy rates because date errors do not affect the decision to  cease or continue benefits, according to officials.28, 29 Nonetheless,  such errors can affect the number of payments a beneficiary receives  and thus SSA’s costs. For example, cessation date errors in a CDR  can result in some beneficiaries receiving payments for longer or  shorter periods of time, and thus accruing overpayments or  underpayments for the period in question. Without including date  errors in its reported accuracy rates, SSA does not provide its  management and other decision makers and the public with complete  information about errors that can affect disability payments. In  addition, if SSA had counted date errors in CDR cessations, its  accuracy rate for cessations in fiscal year 2014 would have fallen 1.6  percentage points from 95.5 percent to 93.9 percent. For some states,  the effect of considering these errors is more pronounced. We  examined SSA’s fiscal year 2014 cessation accuracy rate estimates  and found that for 13 states, the accuracy rates would have  decreased by at least 2 percentage points had SSA counted date  errors; and, in one state the accuracy rate for cessations would have  fallen 7 percentage points, from 95.4 percent to 88.4 percent.
SSA regulations define accuracy in this context as the percentage of cases that do not  have to be returned to state DDSs for further development or correction of decisions  based on evidence in the files. See 20 C.F.R. §§ 404.1643, 416.1043. In explaining the  accuracy standard, SSA stated that its primary purpose was to improve the initial claims  process and ensure that only properly entitled claimants receive disability benefits and  that its approach was to specify outputs (i.e., performance accuracy), rather than  specifying all inputs that could go into the standard.
SSA conducts stewardship reviews which examine the non-medical quality of various  decisions related to benefit payments, including date designations. To do so, SSA reviews  a sample of individuals receiving payments. In conducting and reporting on these reviews,  however, SSA does not specifically focus on CDRs.
Sampling CDRs for quality review: SSA produces accuracy rate  estimates by state DDS, but its sampling approach does not reliably  and efficiently generate accuracy rate estimates for continuances and  cessations separately in every state. According to federal guidance  for developing statistical estimates, agencies should develop a  sampling plan that is reflective of the level of detail and precision  needed of the key estimates. CDR accuracy rates vary by state, and  continuances are consistently more accurate than cessations. In fiscal  year 2014, for example, the states’ estimated CDR accuracy rates  varied from 92.4 percent to 99.8 percent. In the same year, the  estimated accuracy rate for continuances was 98.3 percent  nationwide, whereas the equivalent for cessations was 95.5 percent.  Moreover, the range of accuracy rates across states is much larger for  cessations. In fiscal year 2014, state-level accuracy rates for  cessations ranged from 78.3 to 100 percent, while the accuracy rates  for continuances ranged from 92 to 100 percent. To monitor CDR  accuracy, SSA randomly selects about 70 continuances and 70  cessations for quality review each quarter from each state. Despite  this sampling approach, SSA officials stated that their sampling  design is not intended to produce precise estimates for continuations  and cessations separately by state. However, precise accuracy rate  estimates for continuations and cessations separately by state are  needed to monitor DDS performance because of the difference in  accuracy by decision type and because the state DDSs are managed  separately.
In analyzing CDR workload and accuracy data, we found that SSA’s  sampling approach produced accuracy rate estimates with margins of  error that were consistently wide in seven states and consistently  narrow in six states for either one type of CDR decision or both. A  wide margin of error occurs when there are not enough CDR  decisions in the sample to produce a reliable estimate. In these  instances, such as in Vermont and Wyoming, we found SSA could not  produce an estimate with a margin of error of plus or minus 5  percentage points using its current approach unless it sampled more  CDR decisions. When SSA does not sample enough decisions and  produces estimates with wide margins of error, decision makers may  be relying on misleading information to assess CDR accuracy.  Conversely, when SSA samples too many decisions and produces  estimates with margins of error that are narrower than necessary to  achieve reliable results, the agency may be wasting time and  resources on such quality reviews.
section_title: SSA Reasonably Estimates CDR Cost Savings but Has Conducted Limited Sensitivity Analysis and Documentation
paragraphs: SSA’s annual process for estimating the cost savings of CDRs—the  estimated ratio of federal program savings to costs for performing  CDRs—involves many steps. To calculate the federal program savings  generated by CDRs in a particular year, SSA estimates the present value  of expected future benefits over 40 years that are saved as a result of the  reviews. In forecasting these savings, SSA considers benefits from  programs administered by SSA (i.e., DI and SSI) as well as programs that  are not administered by SSA (i.e., Medicare and Medicaid). To do so,  SSA estimates the number of people whose benefits would be ceased by  CDRs and considers the effect of appeals in determining these  estimates. SSA then estimates the savings associated with the  cessations that are forecasted not to be overturned. It considers the age  of individuals whose benefits would be ceased, and uses separate  models to forecast savings from DI, SSI, Medicare, and Medicaid. SSA  also forecasts and accounts for the number and timing of beneficiaries  who would stop receiving disability benefits regardless of a CDR.  Similarly, it forecasts and excludes the number of former beneficiaries  who will successfully reapply for benefits after, for example, a new  disabling condition arises. To generate the overall CDR cost savings rate  (i.e., the amount saved for every dollar invested in CDRs) for a particular  year, SSA divides the present value of future benefit savings by SSA’s  actual cost of conducting CDRs during the relevant year (see fig. 7). To  determine the cost of conducting CDRs, SSA considers its relevant  expenses as well as those of the state DDSs.
We determined that SSA’s methods and assumptions for estimating CDR  cost savings were reasonable, but, in certain respects, inconsistent with  guidance for conducting cost savings analysis of federal programs.  Specifically, we identified two areas of weakness:
Sensitivity analysis: According to federal guidance for conducting  cost savings analysis, “major assumptions should be varied and net  present value and other outcomes recomputed to determine how  sensitive outcomes are to changes in the assumptions.” In reviewing  SSA’s approach to estimating CDR cost savings for fiscal year 2012,  we determined that SSA did not conduct sensitivity analysis of the  overall cost savings rate. However, SSA separately performed some  limited sensitivity analysis on savings from DI and SSI, which  collectively represented about 82 percent of the savings that SSA  forecasted for fiscal year 2012 CDRs. SSA calculated the effect of  using inputs from fiscal year 2011, such as the average benefit  amount, on the savings estimates for fiscal year 2012. However, SSA  did not vary its assumptions (e.g., from optimistic to pessimistic) to  generate a range of estimated savings. In addition, SSA has not  reported the effect of changing its assumptions about SSI and DI  savings on the overall cost savings estimate. According to an SSA  official, doing sensitivity analysis on the reported cost savings  estimate would require additional coordination with CMS about  Medicare and Medicaid. However, SSA could conduct more complete  sensitivity analysis by, for example, estimating a combined range of  savings from DI and SSI without additional coordination. By not  including a range of estimated savings for at least SSA’s programs,  decision makers lack data on the extent to which the estimates could  vary under different assumptions.
Documentation: According to federal guidance, models used in cost  savings analysis should also be well documented and, where  possible, available to facilitate independent review. SSA uses multiple  complex models to estimate the cost savings of CDRs, but it has  limited documentation about its methods, including data sources,  assumptions, and limitations that factor into the estimates. For  example, SSA has not documented how it estimates the number and  timing of beneficiaries who stop receiving disability benefits because  of a CDR, as well as the number of former beneficiaries who will  successfully reapply for benefits. According to SSA officials, the  agency has not yet documented the assumptions and procedures  used to calculate CDR cost savings because of competing priorities  and limited resources. Consequently, knowledge of SSA’s models is  limited to the few SSA actuaries who work with them, and this  information is not readily available or transferrable to others, including  external reviewers.
section_title: Conclusions
paragraphs: In light of SSA’s current backlog of CDRs and the long-term financial  challenges of the Disability Insurance Trust Fund, conducting timely, high- quality, and cost-effective CDRs is particularly important. In an effort to  use its resources efficiently, SSA applies several sound practices to help  prioritize CDRs. However, without further integrating comparative cost  savings information in its prioritization process, SSA is missing an  opportunity not only to focus on CDRs that are likely to save the federal  government the most money, but also to more efficiently use its resources  for program integrity work. Maximizing cost savings is not the only goal of  this work, but it is an important criterion to help SSA prioritize CDRs and  ensure that beneficiaries are being more effectively selected for review.
Further, although SSA has an extensive process for reviewing the quality  of CDR decisions and a high overall accuracy rate, until the agency  systematically uses available data to identify error-prone cases and root  causes, it will be hard-pressed to prevent similar errors from recurring. In  addition, absent tracking all meaningful errors that it identifies, such as  date errors, the agency and other stakeholders lack an accurate sense of  the true error rate of CDRs. Similarly, SSA’s current approach to sampling  state decisions means the agency may be relying on misleading  performance information for making management decisions.
SSA has demonstrated that CDRs are cost-effective, and it applies sound  methods and assumptions for estimating cost savings. However, because  it does not vary the assumptions that it uses to estimate a range of  potential returns on investment for CDRs, the Congress and other  stakeholders do not have complete information on the precision of these  estimates and the extent to which they could vary with changes in  assumptions. Finally, SSA’s limited documentation about its actuarial  models leaves the agency vulnerable in the event of turnover of the few  staff who use these models and challenges external reviewers’ ability to  understand and audit the integrity of its models.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Acting Commissioner of Social Security:  1.  Direct the Deputy Commissioner of Operations to further consider  cost savings as part of its prioritization of full medical reviews. Such  options could include considering the feasibility of prioritizing different  types of beneficiaries on the basis of their estimated average savings  and, as appropriate, integrating case-specific indicators of potential  cost savings, such as beneficiary age and benefit amount, into its  modeling or prioritization process.  2.  Direct the Deputy Commissioner of Budget, Finance, Quality, and  Management to complete a re-estimation of the statistical models that  are used to prioritize CDRs and determine a plan for re-estimating  these models on a regular basis to ensure that they reflect current  conditions.  3.  Direct the Deputy Commissioner of Budget, Finance, Quality, and  Management to monitor the characteristics of CDR errors to identify  potential root causes and report results to the Disability Determination  Services. For example, SSA could analyze CDRs with and without  errors to identify trends by impairment, beneficiary type, or other  characteristics.  4.  Direct the Deputy Commissioner of Budget, Finance, Quality, and  Management to regularly track the number and rate of date errors,  which can affect benefit payments (e.g., incorrect cessation dates),  and consider including those errors in its reported CDR accuracy  rates.  5.  Direct the Deputy Commissioner of Budget, Finance, Quality, and  Management to adjust its approach to sampling CDRs to efficiently  produce reliable accuracy rate estimates for continuances and  cessations separately in each state.  6.  Direct the Chief Actuary to conduct sensitivity analyses on SSI and  DI’s contributions to CDR cost savings estimates and report the  results reflecting a range of inputs (e.g., from optimistic to  pessimistic).  7.  Direct the Chief Actuary to better document the methods including  data sources, assumptions, and limitations that factor into its  estimates of CDR cost savings.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to SSA for review and comment, and its  written comments are reproduced as appendix II in this report. SSA  stated that it generally agreed with our recommendations, but noted that  the level of program integrity funding it receives has affected the number  of CDRs performed annually and, at times, the size of the CDR backlog.  SSA also noted that our report implied that SSA is not focused on the  CDRs that are most likely to save the government money and that the  report did not fully convey the accuracy of the agency’s statistical models  and its treatment of CDR errors. We agree that SSA’s CDR process is  generally designed to use its program integrity resources efficiently and  note in our report that SSA applies several sound practices to help  prioritize CDRs including annually assessing its statistical models’  performance. However, we maintain additional steps are warranted to  ensure ongoing accuracy of the models and to maximize potential  savings. We also note that the agency has a process in place to identify  and evaluate errors, but maintain that additional steps could be taken to  systematically analyze error trends and uncover root causes of errors.  SSA agreed with four of our recommendations, partially agreed with one  recommendation, and disagreed with two recommendations. The  agency’s specific concerns and our responses are described below:
Regarding our recommendation to further consider cost savings  as part of its prioritization of full medical reviews, SSA partially  agreed. Although SSA agreed that it could look for ways to  improve its return on conducting CDRs, it also stated that its  statistical models and prioritization process already do much of  what we recommend. For example, SSA stated that age is already  a strong variable in its statistical models. However, these models  predict medical improvement and are not designed to take  expected cost savings into account. We continue to believe that to  maximize expected cost savings SSA could refine its prioritization  process by factoring in actuarial considerations. For example,  SSA could consider the effect of a beneficiary’s age on expected  costs savings, in addition to its existing statistical models that  account for the effect of age on the likelihood of medical  improvement.
Regarding our recommendation to complete a re-estimation of the  statistical models that are used to prioritize CDRs and determine a  plan for re-estimating these models on a regular basis, SSA  agreed and stated that it plans to complete its ongoing re- estimation and to document a process for determining when to re- estimate the models in the future.
Regarding our recommendation to monitor the characteristics of  CDRs with errors to identify root causes, SSA agreed and stated  that it reports all errors to the relevant DDS for corrective action.  SSA further stated that its identification of root causes is limited by  the relatively few reviewed CDRs that have errors. However, in  fiscal year 2014 as an example, SSA identified over 600 CDRs  with errors. Although these CDRs make up a small percentage of  the CDRs reviewed by SSA that year, the agency could analyze  the characteristics of CDRs with errors by comparing relevant  percentages without modeling. In addition, SSA could combine  data from multiple years if it determined that considering more  CDRs with errors would be helpful.
Regarding our recommendation to track the number and rate of  date errors and consider including them in its reported CDR  accuracy rates, SSA disagreed and stated that, per SSA  regulation, the agency does not consider date errors when  calculating accuracy rates because date errors do not affect the  decision to cease or continue benefits. SSA also stated its  stewardship reviews examine the non-medical quality of benefit  payment decisions. However, these reviews are not focused on  CDRs, and SSA does not report results from them for CDRs  specifically. SSA also explained that it does not track the number  and rate of date errors because they are infrequent. However,  SSA’s regulations do not prevent the agency from tracking date  errors, and until it does, SSA cannot definitively determine the  frequency of these errors. In addition, we found that considering  date errors substantially reduced some states’ estimated CDR  accuracy rates. Without tracking these errors, SSA cannot assess  their effect and consider whether including them in its reported  CDR accuracy rates has merit.
Regarding our recommendation to adjust its approach to sampling  CDRs to efficiently produce reliable accuracy rate estimates for  continuances and cessations separately in each state, SSA  disagreed and stated that some states do not generate enough  CDR decisions, particularly cessations, to generate statistically  valid samples. However, for states with CDR samples that are  consistently too small to produce reliable results, SSA could, for  example, pool decisions from more months than it currently does  to generate statistically valid samples by state. Conversely, for  states with CDR samples that are consistently larger than  necessary to efficiently achieve reliable results, SSA could, for  example, reduce sample sizes. Because CDR accuracy rates vary  by state and cessations are consistently less accurate than  continuances, we maintain that SSA should adjust its approach to  sampling CDRs.
Regarding our recommendation to conduct sensitivity analyses on  SSI and DI’s contributions to CDR cost savings estimates and  report the results reflecting a range of inputs, SSA agreed and  stated that it will expand on its current sensitivity analyses as time  and resources permit.
Regarding our recommendation to better document the methods  that factor into its estimates of CDR cost savings, SSA agreed and  stated that it will improve and expand its existing documentation  as time and resources permit.
SSA also provided technical comments, which we incorporated into the  report as appropriate.
As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the appropriate  congressional committees and the Acting Commissioner of Social  Security. In addition, the report will be available at no charge on the GAO  website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-7215 or bertonid@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The objectives of this report were to examine (1) how the Social Security  Administration (SSA) selects which Continuing Disability Reviews (CDR)  to conduct, (2) the extent to which SSA reviews the quality of CDR  decisions, and (3) how SSA calculates cost savings from CDRs.
subsections: section_title: CDR Prioritization
paragraphs: To evaluate how SSA selects which CDRs to conduct, we reviewed  relevant federal laws and interviewed SSA officials from the agency’s  offices of Public Service and Operations Support, Budget, and Quality  Improvement. We also reviewed internal SSA documents on the agency’s  approach to prioritizing and processing CDRs, including its use of  statistical models.
To evaluate the statistical models that SSA uses to help prioritize CDRs,  we reviewed internal SSA documents about the statistical models,  including lists of variables, tests of model fit, and detailed technical  reports provided by the external contractor that last re-estimated the  models. The technical reports provided by the contractor explained how  each of the models was developed and tested, including the data sources  and variables that were considered and used, how SSA impairment  codes were aggregated into impairment groups, and how the functional  form and interaction terms were identified. In 2007, the contractor  compared the performance of models for adult beneficiaries against that  of SSA’s prior models—which had been estimated last in 2005—and only  specifications and variables that improved model performance were  retained. We evaluated the technical specifications and tests of model fit  and predictive accuracy for the models for each of the beneficiary  cohorts.
To estimate the proportion of full medical reviews completed because of  their score from SSA’s statistical models, we analyzed CDR annual report  data from fiscal years 2003 through 2013. We assessed the reliability of  these data by reviewing related documentation and interviewing  knowledgeable agency officials, and we found these data sufficiently  reliable for our purposes. To assist with the analysis, we obtained  information from SSA’s Office of Public Service and Operations Support  on the number of SSA policy priority cases processed annually. From the  total number of full medical reviews completed during a fiscal year, we  subtracted completed full medical reviews that were prioritized because  they were statutorily required (e.g., reviews of SSI children at age 18 and  reviews of children under 1 year old who are receiving SSI benefits due in  part to low birth weight) and because of SSA policy (e.g., mailers with  certain responses and first-time reviews for beneficiaries in the “medical  improvement expected” diary category). To avoid potential double- counting, we did not count the sample of approximately 60,000 cases that  SSA initiates annually to validate its statistical models among the policy  priority cases because the sample consists of cases across all beneficiary  groups, including statutorily required cases. As a result, our calculations  may underestimate the number of required priority cases and  overestimate the number of cases selected because of the statistical  models.
To illustrate the impact of further incorporating cost savings into the  prioritization process, we obtained and analyzed data from SSA on the  average savings per full medical review by beneficiary group in each of  fiscal years 2012 and 2013. Using weighted averages of these data, we  calculated the average savings for fiscal years 2012 and 2013 separately  and combined. We also developed two hypothetical scenarios that pair  near-identical beneficiaries with different ages or monthly benefit  payments to demonstrate the effect of considering cost savings on an  individual basis. We used information from the fiscal year 2013 statistical  supplements on the DI and SSI programs to develop reasonable ages  and benefit levels for the hypothetical beneficiaries. We calculated the  expected savings in foregone benefits after cessation for each beneficiary  by multiplying the monthly benefit by the number of months until the  beneficiaries would have aged out of the disability programs.
section_title: Quality Reviews of CDR Decisions
paragraphs: To understand the process that SSA uses to review the quality of CDR  decisions, we reviewed relevant federal laws, regulations, policies, and  procedures; interviewed SSA officials about these policies and  procedures; and analyzed SSA’s CDR workload and decision accuracy  data. This work included reviewing documentation of the Disability Case  Adjudication and Review System (DICARS), the software program in  which SSA completes quality reviews. We interviewed SSA officials about  how the quality reviews are conducted and how the agency uses the  results, and compared the agency’s policies and procedures to generally  accepted statistical practices and federal internal control standards. We  also interviewed state Disability Determination Services officials about  factors that challenge CDR quality. We assessed the reliability of SSA’s  CDR workload and decision accuracy data by performing data testing,  reviewing related documentation, and interviewing agency officials, and  we found the data to be sufficiently reliable for the purposes of this  review.
To evaluate the extent to which SSA reviews the quality of CDR  decisions, we analyzed SSA’s CDR workload and decision accuracy data  to determine whether its method for sampling CDRs and estimating CDR  accuracy are consistent with generally accepted statistical practices and  SSA’s reporting goals. SSA reports accuracy rate estimates for each  state every month using the most recent 3 or 6 months of quality review  data. Its goal is to produce estimates with 95 percent confidence  intervals that are within plus or minus 5 percentage points of the estimate.  We analyzed SSA’s CDR workload and accuracy data, consistent with  SSA’s sampling and reporting methods, from June 2013 through April  2015. Specifically, we identified the number of continuance and cessation  determinations in each state, the District of Columbia, and Puerto Rico;  SSA’s accuracy rate estimates for these determinations for each 6-month  period; and the 95 percent confidence interval margins of error for each  estimate. To identify states that had estimates that consistently do not  achieve the reporting goals, we compared the workload, accuracy rate  estimates, and margins of errors to those specified in SSA’s sample  design and reporting goals. We calculated margins of error for estimates  in which SSA did not provide them. We used a statistical formula that  produces appropriate margins of error, including when standard formula  do not apply, to determine and examine a margin of error for all  estimates. We chose this formula because in some cases CDR accuracy  is so high or the sampling fraction is so large that the standard statistical  formula used for these purposes would compute margins of error that are  not appropriate, such as those resulting in confidence intervals above 100  percent. Since CDR accuracy cannot be greater than 100 percent, the  standard formula is not appropriate. We also analyzed CDR workload  data from fiscal years 2001 through 2014 to inform our evaluation of  SSA’s sampling method.
To determine the effect of date errors on accuracy rates, we analyzed  data about CDR date errors and cessations. We considered data from  fiscal years 2010 through 2014 to determine the frequency with which  date errors occur. We calculated fiscal year 2014 cessation accuracy rate  estimates for each state by combining the number of cessation decision  errors and the number of date errors and dividing the total by the number  of cessations SSA reviewed. SSA’s date error data were not broken down  by decision type (i.e., continuance or cessation), but we assigned these  errors to cessations because of input from SSA and our corroborating  analysis. According to SSA officials, the most common date error on a  CDR is a cessation date error and other date errors, such as incorrectly  inputting an onset date, can occur in a cessation or continuance but are  rare. Our analysis corroborated this information. For example, in fiscal  year 2014, of 127 date errors identified in CDRs nationally, 125 of them  were cessation date errors. In addition, we calculated margins of error for  each estimate to assess the statistical reliability of each estimate. We  used the statistical formula that produces appropriate margins of error,  consistent with our approach to calculating margins of error in our  analysis of SSA’s sampling method.
section_title: CDR Cost-Savings Estimate
paragraphs: To evaluate SSA’s approach to calculating cost savings from CDRs, we  compared SSA’s estimation process to actuarial standards of practice  and federal guidelines for benefit-cost analyses of federal programs.  Specifically, we interviewed SSA actuaries about the models and  methods they used to perform the cost-savings calculation for fiscal year  2012. We also reviewed portions of the programming code related to  these models to corroborate the information from the actuaries. In  addition, we examined the assumptions that SSA uses to calculate the  present value of future benefits saved from ceasing a person’s benefits as  the result of a CDR by examining where and how SSA incorporates  assumptions into its calculation process. Finally, we reviewed the fiscal  year 2012 CDR cessation data and information SSA provided to the  Centers for Medicare & Medicaid Services (CMS) that informed CMS’s  estimates of Medicare and Medicaid savings resulting from CDR  cessations, but we did not review CMS’s models.
We conducted this performance audit from December 2014 to February  2016 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Social Security Administration
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Erin Godtland (Assistant  Director), Joel Green (Analyst-in-Charge), Susan Aschoff, James  Bennett, Grace Cho, Alexander Galuten, Emei Li, Isabella Johnson,  Rhiannon Patterson, Almeta Spencer, Daren Sweeney, Jeff Tessin,  Kristen Timko, Frank Todisco, Sonya Vartivarian, and Shana Wallace  made key contributions to this report.