Modernizing SSA Disability Programs: Progress Made, but Key Efforts Warrant More Management Focus
Published: 2012-06-19T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: SSA administers two of the largest federal disability programs. GAO designated federal disability programs as a high-risk area, in part because eligibility criteria had not been updated to reflect medical and technological advances and labor market changes. Given the size and cost of its disability programs, SSA needs updated criteria to appropriately determine who qualifies for benefits. GAO has been asked to assess SSAs efforts to update its medical criteria and develop a new occupational information system, and to identify other steps taken to modernize disability determination criteria. To do this, GAO reviewed relevant publications and federal laws and regulations; assessed agency plans, cost estimates, schedules, and other documentation against established project management criteria; and interviewed SSA officials, experts, and stakeholders.
section_title: What GAO Found
paragraphs: The Social Security Administration (SSA) has taken steps that hold promise for improving the process for updating its medical criteria, but continues to face challenges ensuring timely updates. SSA now uses a two-tiered system for ongoing revisions to its medical listings. First, it completes a comprehensive review of all medical conditions listed within each of 14 body systems, making needed revisions. For subsequent updates for a body system, the agency uses a targeted approach, selecting for review and revision only those medical conditions most in need of change. To date, SSA has completed comprehensive revisions for 8 of the 14 body systems and now is reviewing conditions under them to determine where targeted revisions are appropriate. However, some of these targeted revisions have experienced delays. Moreover, SSA has yet to complete comprehensive revisions for six body systems that have been ongoing for 19 to 33 years. SSA officials attributed delays to a lack of staff and expertise, along with the complexity and unpredictability of the regulatory process.
SSA has embarked on an ambitious plan to design by 2016 an occupational information system for use in its disability decision-making process, but has fallen short of best practices for estimating costs, maintaining a schedule, and considering risks and alternatives. SSA currently relies on occupational information developed by the Department of Labor which has not had a major update since 1977. In 2008, SSA initiated a project to develop its own occupational information system (OIS), which SSA expects will provide up-to-date information on the physical and mental demands of work to support its decision-making process. To guide the creation of its OIS, SSA established an advisory panel, collaborated with outside experts and other agencies, and in July 2011 issued a research and development plan detailing relevant activities through 2016. SSA has made progress on some baseline activities in the plan. However, SSAs cost estimate and schedule had key deficiencies, such as not including any estimate of the cost of producing, maintaining, and operating the system, which can inform design options. SSA also did not adequately consider inherent risks or potential alternatives, which could heighten the risk of additional costs or project failure.
Consistent with modern views of disability, SSA has taken some concrete steps toward greater consideration of an individuals ability to function with a disability but faces constraints in fully modernizing. SSA has incorporated some criteria into its medical listings to determine whether a claimants impairments result in functional limitations that can prohibit the ability to work. SSA is also sponsoring research through the National Institutes of Health to evaluate how functional abilities can further be considered in determining disability. One project aims to develop a computerized tool to assist adjudicators in evaluating how various impairments affect an individuals function and ability to work. However, SSA officials maintain that other modern concepts of disability cannot be fully incorporated into SSAs disability decisions. Specifically, SSA faces constraints considering the extent to which assistive devices and workplace accommodations can mitigate work disability, because these are not universally available and SSA lacks the resources to conduct individualized assessments.
section_title: What GAO Recommends
paragraphs: GAO recommends that SSA (1) explicitly identify resources needed to achieve its 5-year time frame for updating its medical listings; (2) follow best practices in its cost estimate, schedule, and risk assessment for the occupational information system; and (3) conduct limited, focused studies on how to more fully consider assistive devices and workplace accommodations in its disability determinations. SSA agreed with the first two recommendations and disagreed with the third, stating that such studies would be inconsistent with Congress intentions. GAO continues to believe the recommendation has merit, as discussed more fully within the report.

section_title: Letter
paragraphs: The Social Security Administration (SSA) manages two of the largest  federal disability benefit programs—Social Security Disability Insurance  and Supplemental Security Income—which together resulted in payments  of more than $165 billion to about 13.5 million people with disabilities and  their families in fiscal year 2010. Given the extensive size and cost of its  disability programs, SSA must have current and appropriate criteria by  which to assess whether an applicant’s medical conditions affect his or  her ability to perform work in the national economy. However, we  designated federal disability programs as high risk, in part because the  medical criteria and occupational information that SSA relies on to make  benefit decisions were found to be out of date. Moreover, we and others  have found SSA’s disability programs—which historically have tended to  equate severity of a medical condition with an inability to work—to be out  of sync with a more modern concept of disability that considers the  presence or lack of assistance that an individual with medical  impairments might need to function more effectively in his or her  environment.
You recently asked us to assess SSA’s plans and efforts for revising its  disability criteria. Accordingly, we examined the status and management  of SSA’s efforts to update its medical listings to reflect current medical  knowledge and develop a new occupational information system to reflect  labor market changes, and we identified other steps taken by SSA to  incorporate a modern view of disability into its eligibility criteria.
To perform this work, we reviewed our prior reports and SSA Office of  Inspector General reports; relevant federal laws and regulations; program  documentation, including policies, procedures, strategic goals, and  supporting project plans and cost estimates; relevant literature; and  position papers and testimonies from disability groups and commissions.  We interviewed SSA officials, key project contractors and stakeholders,  disability experts, and representatives of other agencies that administer  disability programs. We also evaluated SSA’s plans to update its criteria  against sound project management and cost estimating practices. We  narrowed our scope to criteria used for initial adult disability  determinations. See appendix I for more on our scope and methodology.
We conducted this performance audit from April 2011 through June 2012  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: SSA administers two disability programs: the Social Security Disability  Insurance (DI) program, enacted in 1956, and the Supplemental Security  Income (SSI) program, enacted in 1972. In order to be eligible for DI or  SSI benefits based on a disability, an individual must meet the definition  of disability for these programs—that is, they must have a medically  determinable physical or mental impairment that (1) prevents the  individual from engaging in any substantial gainful activity and (2) has  lasted or is expected to last at least 1 year or result in death.
To determine eligibility for both programs, SSA uses a five-step  sequential process that is intended, in part, to expedite disability  decisions when possible and limit administrative costs by conducting less  intensive assessments at earlier steps (see fig. 1).
At steps 1 and 2 of the process, SSA determines whether an applicant  is working and meets income thresholds, as well as the medical  severity of impairments. If not working (or not meeting income  thresholds) and with the determination of a severe impairment, the  applicant moves to step 3 of the process.
At step 3, SSA examiners assess the applicant’s medical impairments  against the Listings of Impairments, also known as the medical  listings, which are organized into 14 major body systems for adults  and reflect medical conditions that have been determined by the  agency to be severe enough to qualify an applicant for benefits.  When using the listings to determine eligibility, SSA generally relies  on information on the applicant’s diagnoses, including laboratory  findings, diagnostic tests, and symptoms, as well as some limited  consideration of the applicant’s functional limitations. If the individual’s  impairment meets or is equal in severity to one or more of those in the  listings, the individual is determined to have a disability at step 3.
If a disability determination is not made at step 3, SSA performs an  assessment of the individual’s physical and mental residual functional  capacity. Based on this assessment, SSA determines whether the individual is able to perform past relevant work (step 4) or any work  that is performed in the national economy (step 5). To inform  determinations at steps 4 and 5, SSA uses a Department of Labor  database—known as the Dictionary of Occupational Titles (DOT),  which is an inventory of occupations performed in the national  economy. At step 5, SSA also uses a set of rules and guidelines,  referred to as the grid rules, to evaluate the combined effect of an  individual’s physical residual functional capacity, age, education, and  work experience.
While originally created for expediency, over time the medical listings used  at step 3 were relied on less to make program determinations as they  became increasingly outdated. In the early years of the program, more than  90 percent of cases were decided based on medical conditions specified in  the listings; in 2010 only 47 percent of allowances were made at step 3.  Experts attribute the decline in allowances based on the medical listings to  changes in the program, workplace, and medical treatment that the medical  listings had not kept pace with. Since the 1990s, we, along with SSA’s  Office of Inspector General and the Social Security Advisory Board, have  expressed concerns that the medical listings being used no longer provide  current and relevant criteria to evaluate disability applicants’ inability to  work. In 2003, we deemed SSA’s and other federal disability programs as  high-risk areas, in part, because their programs continue to emphasize  medical conditions in assessing work capacity, without adequate  consideration of work opportunities afforded by advances in medicine,  technology, and changes in the labor market.
In 2008, we reported that SSA had recently established a new process— referred to by SSA as the “business process”—for revising the listings to  better incorporate feedback into its continuous updates. This process,  which has been in effect since 2003, incorporates feedback from multiple  parties, including medical experts and claims examiners, to update the  medical criteria. Under this process, SSA gathers external feedback from  comments associated with regulatory actions, such as the publication of  advanced notices of proposed rulemaking (advanced notices) and notices  of proposed rulemaking (notices) in the Federal Register.
In addition,  this process includes conducting postimplementation reviews one year  after a revision is made to assess the impacts of a revised listing, areas to  improve, and whether expectations from the revisions have been  achieved.
With respect to information on jobs in the national economy that supports  SSA’s occupational criteria, we and others have reported that the DOT is  outdated, although SSA still relies on it to assess eligibility at steps 4 and   and  5 of the process. The DOT has not had a major update since 1977 the Department of Labor (Labor) replaced it with a new database in 1998  called the Occupational Information Network (O*NET). However, SSA  determined that O*NET is not sufficiently detailed for evaluating DI and  SSI disability claims, so SSA has begun developing its own occupational  information system to better reflect the physical and mental demands of  work in the national economy.
Beyond dated medical criteria and vocational information, numerous  disability experts have expressed concern that SSA’s disability  programs—which statutorily require that an adult’s medical condition  prevents that person from engaging in substantial gainful activity—have  historically tended to equate the severity of medical conditions with  inability to work and thus are out of sync with modern concepts of  disability. Modern concepts focus on an individual’s functional abilities in  the workplace environment, including consideration of the presence or  lack of assistance, for example, per the requirements for reasonable  accommodation by the Americans with Disabilities Act of 1990. These modern views are reflected in the International Classification of  Functioning, Disability and Health (ICF), which is the World Health  Organization’s framework for assessing health and disability. This  framework takes into account the interaction of health conditions and  contextual factors, such as products and technology, attitudes, and  services, on an individual’s functional capacity, rather than viewing  disability solely as a medical or biological issue. Along these lines,  experts have recommended that SSA incorporate more consideration of  individual function in its medical listings when doing so can improve their  use as a screening tool for determining inability to work. Further, several  groups, such as the Social Security Advisory Board and the Urban  Institute, have reported that SSA’s disability programs should focus  more on whether an individual can work given appropriate environmental  or other supports, and that SSA—through its demonstration authority or  other means—could play a role in determining how. These suggestions  are consistent with SSA’s long-term targeted outcomes, which include not  only regularly updating regulations and policies to incorporate the most  recent medical advances, but also making it easier for individuals with  disabilities to return to work.
section_title: SSA Has Improved Its Process for Updating Medical Listings, but Still Faces Challenges with Timely Updates
subsections: section_title: SSA Has Improved Its Strategy to Update Medical Listings
paragraphs: Since our last review in 2008, SSA has made several changes that hold  promise for improving its medical listings updates. First, the agency is  using a two-tiered system for ongoing revisions to the listings. Under this  system, SSA first completes a comprehensive listings update for a body  system, which entails reviewing all the diseases and disorders listed  within that system and making necessary revisions. Second, following a  comprehensive revision, SSA will pursue a more targeted approach—that  is, SSA will conduct ongoing reviews and updates of a smaller number of  medical diseases or disorders within that body system. Agency officials  told us that targeted updates should be completed more quickly than  comprehensive updates, allowing them to focus on the most critical  changes needed. As of early March 2012, SSA had begun the ongoing  review process to identify opportunities for targeted revisions for 8 of the  14 adult body systems that were recently comprehensively revised.
Another change, according to agency officials, is that in 2010, the SSA  Commissioner set a 5-year cycle for updating listings for each body  system. listings under each body system, ranging from 3 to 8 years, but frequently  extended them. SSA officials believe that conducting targeted reviews will  generally allow the agency to conclude any necessary revisions prior to  the 5-year period. Additionally, they expect that using the “business  process,” which requires early public notification of changes and obtaining  necessary data and feedback from internal and external parties, should  help keep continuous reviews on track. See figure 2 for the status and  expiration date by which the listings should be reviewed and updated, if  needed, for the 14 body systems undergoing review for either  comprehensive or possible targeted revisions, as of early March 2012.
The 5-year period will be applied to listings under a body system upon completion of  their current revision. SSA began applying the 5-year period in 2011 when it  comprehensively updated the endocrine body system listings. For other body system  listings updated prior to 2011, SSA generally assigned periods extending beyond 5 years.
For the two-tiered revision process, the beginning point is the date of the last comprehensive or  targeted revision.
SSA officials told us they have grouped the disorders in the special senses and speech listings into  two broad sections—vision and hearing—and two standalone medical conditions. SSA has not yet  comprehensively revised the two standalone medical conditions that cover speech and disturbances  of labyrinthine-vestibular function. The two sections have undergone comprehensive revisions and  are undergoing reviews that began in 2007 for the vision section and in 2010 for the hearing section.  Special senses and speech listings are set to expire in 2015.
According to SSA officials, SSA published limited revisions for the hematological disorders in 1988. It  also published final rules for limited revisions for the mental disorders listings in 2000 and the  musculoskeletal system listings in 2001.
SSA has made another change by more extensively engaging the  medical community to identify ways to improve the medical listings. For  example, SSA contracted with the Institute of Medicine to study its  medical criteria for determining disability and to make recommendations  for improving the timeliness and accuracy of its disability decisions,  resulting in a 2007 report with recommendations and a symposium of  experts in 2010. SSA has addressed some of the Institute of Medicine’s  recommendations, such as making better use of its administrative data to  update criteria and creating a standing committee through the Institute of  Medicine to provide recommendations for listings revisions. In addition,  through the Institute of Medicine, SSA created consensus committees to  conduct research and provide other assistance with updating SSA’s  cardiovascular listings and its Human Immunodeficiency Virus (HIV)  listing, which culminated in two reports with recommendations.
section_title: SSA Has Experienced Delays with Its Revision Process
paragraphs: SSA has made progress, but continues to face delays in completing both  comprehensive and ongoing targeted updates. As of early March 2012,  SSA officials told us they had completed comprehensive revisions of  listings for eight body systems, some of which resulted in significant  changes. For example, in 2011, SSA removed the endocrine body system  listings for all the adult disorders, such as diabetes, because they found  that they were now generally diagnosed early and treated very  successfully. Nevertheless, according to SSA, the agency still needs to  complete the comprehensive revisions for listings of the remaining six  body systems—a process that has been ongoing for the last 19 to 33  years, with numerous extensions beyond the original expiration periods  (see table 1). For example, it has been at least 27 years since SSA  finalized comprehensive revisions for two of the six body system  listings—mental and neurological disorders, which are among those SSA  uses most frequently in its eligibility determination process. SSA has  made progress on four of the six body systems set to expire in 2012, but  will likely miss targeted time frames. SSA issued a notice of proposed  rulemaking to revise its listings of mental disorders in 2010 and has told  us that it plans to finalize this comprehensive revision by the end of 2012,  after its current expiration date in July 2012. SSA is still developing  notices for the neurological and two other body system listings, which are  also set to expire in 2012. Because SSA has generally taken more than a  year to move from publishing the notice to final revision and may also  need additional time for any internal revision and review, as well as a  review by the Office of Management and Budget, it is also unlikely that  SSA will meet its planned 2012 time frames for updating these listings.  See appendix II for details on progress updating these listings.
Updates using SSA’s targeted approach seem to be moving at a faster  pace than the comprehensive revisions, but some delays have occurred  here as well, and more are expected. For example, SSA has already  extended the expiration date for its targeted review of the cardiovascular  system. According to SSA officials, it may also need to extend expiration  dates for listings under two other body systems set to expire in 2012.  Extensions may be needed because SSA has not yet published the  notice of proposed rulemaking for those two body system listings and  may not have time to publish these notifications, respond to comments,  and complete the final updates by their current expiration date.
At the same time that SSA has been experiencing delays completing  timely revisions, agency officials reported challenges with other steps in  the business process. Whereas SSA’s business process includes a  postimplementation review of case data at the 1-year mark to determine  whether expectations from a revision were met, several did not undergo  this review, such as those for skin disorders, genitourinary impairments,  and impairments affecting multiple body systems. SSA officials told us  they only began conducting these reviews in 2010, and to date, SSA has  completed just one in 2011 that involved a targeted sample of 175 cases.  SSA officials told us they are conducting or planning to conduct two more  reviews at the 1-year mark in 2012. While disability experts we  interviewed spoke highly of SSA’s business and targeted review  processes to obtain feedback early on and update the listings more  promptly, ongoing delays raise questions about the agency’s ability to  fully follow its current business process while completing continuous and  timely revisions for all 14 body systems.
SSA officials offered two key reasons for the delays in updating the  listings: (1) limitations in the number and expertise of staff and (2) the  complexity and unpredictability of the regulatory process. According to  SSA officials, revising the medical listings requires research, deliberation,  testing, regulatory review, and consensus with many stakeholders, and  consequently is difficult and time-consuming to achieve. According to an  SSA official in the Office of Medical Listings Improvement, the office is  short-staffed and there is a lack of expertise needed to perform this work.  To address these constraints, SSA has contracted with the Institute of  Medicine to review and develop recommendations for revising two of the  body system listings. While not finalized as of March 2012, SSA officials  reported that the agency plans to renew this contract after it expires in  2012 and extend it to 2015. However, SSA has not yet determined how it  will use the Institute of Medicine to revise the listings or the extent to  which the contract will address staffing shortfalls.
Also contributing to delays is the time required for internal review and  public comment under the regulatory process, which depends largely on  the number and the substance of comments received, according to an  SSA official. Obtaining public comment is one way SSA receives critical  information for identifying areas for revision and obtaining stakeholder  consensus. As such, SSA reviews and responds to each comment that  would result in a significant change, and the time required for doing so  varies depending on the number of comments and resources. For  example, SSA officials told us that the advanced notice of proposed  rulemaking to the listings of mental disorders in 2003 resulted in 500  comments, which took SSA 5 years to incorporate into a draft notice of  proposed changes for regulatory review. SSA officials reported they have  considered options for automating and thereby speeding up the process  of obtaining and reviewing public comment. For example, they told us that  they eventually plan to use a web-based tool to obtain early public  feedback on medical listing updates to help target their limited resources  and more quickly make changes. According to an SSA official, the agency  plans to retain its optional use of the advanced notice of proposed  rulemaking because it can also help to identify appropriate areas to focus  on to ultimately make timely updates.
section_title: SSA Has Begun an Ambitious Project to Develop Updated Occupational Information, but Many Uncertainties Remain
subsections: section_title: SSA is Designing a New Occupational Information System, but Has Fallen behind with Some Activities
paragraphs: In 2008, SSA began a multiyear project to research and design a new  source of occupational information that will replace the outdated  information currently being used to determine if claimants are able to do  their past work or any other work in the national economy. Since the  1960s, SSA has been using the DOT, which contains a list of job titles  found in the national economy and had a last major update in 1977. The  DOT provides SSA with descriptions of the physical demands of work— such as climbing, balancing, and environmental requirements—for each  of the more than 12,000 occupations listed. According to SSA, these  descriptions have been essential to its evaluations of how much a  claimant can do despite his or her impairment and whether this level of  functioning enables the claimant to do his or her past work or any other  work. After its last limited update, Labor decided to replace the DOT with  O*NET, which has far fewer occupational titles compared with the DOT  and has served Labor’s purposes more efficiently. According to an SSA  report, after investigating potential alternatives, SSA decided that O*NET  and other existing databases with occupational information were not  sufficiently detailed and not able to withstand legal challenges for use in  its decision-making process. SSA further decided to develop its own  occupational information system (OIS), which would contain detailed  information as in the DOT, but would also include additional information,  such as the mental demands of work. In addition, SSA has determined  that the OIS will (1) meet its legal, program, and data requirements; (2) be  flexible enough to incorporate changes in its policies and processes; and  (3) be able to be updated to reflect the evolving workplace environment.
In 2008, SSA began taking steps to guide the development of its OIS.  SSA created an internal office and working group, as well as an  Occupational Information Development Advisory Panel. While the number  of panel members has fluctuated over time, in April 2012, 14 external  experts were serving on the panel and represented various affiliations,  such as medicine, disability law, rehabilitation, and industrial  organizational issues. The advisory panel holds quarterly public  meetings and has several subcommittees that review material and make  recommendations to SSA on developing various components of the OIS.  For example, in a 2009 report, the advisory panel supported the need for  SSA to develop a new source of occupational information, rather than  adapt O*NET, and recommended the type of data SSA should collect, as  well as approaches for classifying occupations.
The Occupational Employment Statistics program produces employment and wage  estimates for approximately 800 occupations. The U.S. Census Bureau’s household  surveys include (1) the American Community Survey, which is an ongoing survey that  provides annual data on demographics such as age, education, and disabilities, and (2)  the Current Population Survey, which is primarily a labor force survey, conducted every  month by the U.S. Census Bureau for the Bureau of Labor Statistics and provides data  such as the national employment rate. explore specific subject areas, such as sampling issues. Besides working  with Labor and U.S. Census Bureau officials, SSA officials and panel  members have sought input from other experts and current users of the  DOT, such as SSA disability adjudicators and external rehabilitation  professionals. SSA officials conducted a user needs analysis in 2009 and  have presented on the OIS project at events and conferences.
In July 2011, SSA published a detailed research and development plan  outlining all activities related to researching, developing, and testing the  key components of the OIS in order to implement it by 2016 at an  estimated total cost of $108 million. For example, the plan includes  several baseline activities to identify and study other occupational  information systems and various approaches for analyzing occupations  that may inform or could be leveraged in SSA’s OIS data collection. The  plan also includes activities to identify the primary occupational,  functional, and vocational characteristics of current beneficiaries. Other  key components of the plan include developing descriptions of work  requirements, such as the physical and mental demands for jobs;  developing data collection and analysis strategies; and identifying the  occupations, categories, and definitions that will constitute the structure of  the new OIS. SSA also plans to develop a strategy for piloting how it  would ultimately collect data for the OIS nationwide within this time frame.
As of May 2012, SSA had made progress on many of the baseline  activities outlined in its research and development plan for the OIS. For  example, according to an SSA official, its investigation of existing  occupational information systems, now complete, resulted in useful  information about design issues other organizations have confronted and  mitigated when creating their own system. Additionally, SSA’s preliminary  analysis of its own administrative data identified the most frequently cited  occupations and functional and vocational characteristics of disability  applicants. SSA officials told us the agency will target the occupations  identified in this analysis for its pilot studies of the OIS. Also in 2011, SSA  completed a comprehensive framework for assessing an individual’s  capacity to work—key to informing the OIS content, according to SSA  officials—which was based on recommendations of outside experts as  well as SSA’s policy and program requirements.
While SSA has made progress on several key activities, agency officials  delayed 2011 completion dates for certain activities and anticipate making  additional changes to its timeline as a result of not meeting its staffing  goals for fiscal year 2011 (see table 2). Some activities that were delayed  by several months included finalizing reports for the baseline studies and  conducting a literature review that would inform how occupations might  be analyzed. SSA officials told us that they did not have enough staffing  to complete all of the 2012 planned activities within the estimated  schedule. Further, SSA officials said they did not have the budget to hire  new staff in September 2011. To address this challenge, SSA officials  hired consultants to meet some of their needs. SSA officials also met with  the Office of Personnel Management to explore the possibility of an  interagency agreement that would allow SSA to use one or two of the  office’s industrial organizational psychologists to help on a part-time  basis.
Our Cost Estimating and Assessment Guide identifies a number of best  practices for effective cost estimating and scheduling that should result in  reliable and valid cost estimates that management can use for making  informed decisions. Per these criteria, the success of any program  depends in part on having reliable cost estimates and a reliable schedule.  A reliable cost estimate provides the basis for informed investment  decision making, realistic budget formulation and program resourcing,  meaningful progress measurement, proactive course correction when  warranted, and accountability for results. A reliable schedule defines,  among other things, when work activities will occur, how long they will  take, and how they are related to one another. As such, the schedule not  only provides a road map for systematic execution of a program, but also  provides a means by which to gauge progress, identify and address  potential problems, and promote accountability.
We compared SSA’s cost estimate to three best practices for assuring a  reliable cost estimate and found SSA only minimally or partially met  each of these (see table 3). First, we found SSA’s estimated cost of the  OIS research and development phase—$108 million—was not well- documented. For example, after reviewing all of SSA’s documentation  supporting this cost estimate as of December 2011, we did not find step  by step documentation showing how the estimate was derived so that  someone unfamiliar with the project could use the documentation to  recreate the estimate and get the same results. Well-documented cost  estimates are considered a best practice and without good  documentation, SSA is not in a position to defend the reliability of its  estimate. Additionally, SSA only partially met the best practice of having  an accurate cost estimate. While SSA provided documentation showing  how some of their estimates compared to actual costs and the reasons  for any variances, SSA did not provide adequate documentation for us to  determine, for example, if the estimate took inflation into account or if  there were any calculation errors. Generally speaking, in the absence of a  detailed cost model, third party reviewers cannot be certain that cost  estimate calculations are accurate and account for all costs.
We also found the cost estimate was not comprehensive as it does not  include any costs beyond the research and development phase, such as  the costs of producing, maintaining, and updating the final data system,  which could be significant. SSA officials told us that they are still in the  process of determining what information will be included in the data  system, how it will be collected, and how many occupations will be  covered, each of which will influence the cost of developing and  maintaining the OIS. As such, they maintain that it is too early to estimate  the costs of the final system or the costs for maintaining the system.  According to industry best practices, cost estimates should be  comprehensive and include all costs necessary to achieve agency  objectives and should be updated as the agency proceeds with the  project and gains more information. At this point in the project, estimating  total costs of options under consideration could enhance decision making  by evaluating the potential tradeoffs of different designs. Without any  estimate for the cost of producing the OIS, SSA risks designing a system  that would not be a viable or affordable option to complete. Additionally,  without maintenance cost estimates, SSA is at risk of designing a system  that would be too costly to maintain on a regular basis resulting in  outdated information. Other federal information systems provide some  basis for estimating the cost of producing or maintaining an OIS. For  example, Labor officials indicated that the cost of maintaining O*NET—an  occupational information system that includes approximately 1,000  occupations and uses a paper or web survey to collect data—is roughly  $6 million a year.
We also reviewed SSA’s master schedule for the OIS against nine best  practices for a reliable schedule, and found that SSA did not meet four  and minimally met five of these practices (see table 4). For example,  among those minimally met, we found a significant number of activities  within SSA’s schedule were not logically sequenced in the order that they  were going to be carried out (best practice 2) and activities that were  dependent on completion of a prior activity were not identified (best  practice 5, 6, and 7). For example, SSA officials and experts  acknowledged that activities associated with the “OIS work taxonomy”  part of the research and development plan—activities that together  determine which information should be included in the OIS—are an  essential building block that will inform other OIS activities, such as  developing the “OIS work analysis instrument,” which will determine how  OIS information will ultimately be collected.
Nevertheless, when we tested SSA’s schedule for sequencing and  linkages associated with the OIS work taxonomy activity, we found a  significant delay of almost 1,000 days that should have significantly  delayed the project actually barely affected the final completion date in  the OIS project. Such missing links between key activities in the schedule  represent broken logic that reduces the reliability of the forecasted dates.  The OIS schedule also did not meet the best practice of conducting a  schedule risk analysis (best practice 8), which is an essential tool for  project managers to understand the most important risks to the project  and focus on mitigating them. Based on the schedule provided to us, we  do not have any indication that SSA has considered the vulnerability  within its schedule for meeting time frames for individual activities which  would in turn impact the time frames for the entire project. Without  explicitly identifying risks to its schedule, SSA either does not know or is  not conveying the probability of completing research and development  activities on time, and its relationship to overall costs for this phase.  However, in order to conduct a schedule risk analysis the schedule has to  be properly sequenced and networked with all logic links in place for the  analysis to produce credible results.
Beyond gaps in their cost estimate and schedule of specific activities,  SSA faces broader challenges that could impede the success of the OIS  design and implementation; however, SSA has not done a formal risk  analysis of these challenges. While SSA officials said they had recently  begun discussing various risks and how they might address them, they  said it was premature to provide us with this information. Examples of  potential challenges for OIS that experts and stakeholders we spoke with  cited include:
SSA’s lack of expertise with designing an OIS. SSA does not have  prior experience with designing a complex and, in some respects,  unprecedented occupational information system. As such, SSA will  need to depend on many outside experts and contractors to complete  the system. While the Occupational Information Development  Advisory Panel members can provide some expert counsel, most of  its members lack the technical and scientific background essential to  informing this complicated effort. To help design the OIS, SSA  recently hired a research psychologist and plans to hire two more  individuals in the near future, but may need additional resources. In  March 2012, SSA put out a request for information and, according to  an agency official, received ideas from knowledgeable experts about  how to move forward with technical aspects of the project. Until the  agency secures required and sufficient expertise, SSA may not be  able to move ahead with key technical decisions, such as defining the  number of occupations to include and the data collection  methodology, which may cause further delays.
Cost of maintaining an OIS. SSA has not yet made design decisions  on the OIS that will ultimately define the overall cost of producing,  implementing, and maintaining the OIS system. For example, SSA  officials told us that they were not sure how many occupations they  would include in the OIS, although they believe the total number will  be somewhere between the number of occupations in the O*NET and  the DOT, or between approximately 1,000 and 12,000 occupations. A  2010 study conducted by the National Academy of Sciences  reviewing Labor’s O*NET cited the three major cost drivers of an  occupational information system as (1) size of sample, (2) number of  occupations, and (3) frequency of updates. This study noted that as  the agency increases any one of these cost drivers, it does so at the  expense of the others or of the overall cost. For example, if SSA  chooses to frequently update the entire OIS, the agency may need to  make tradeoffs with the number of occupations or sample size it  chooses if it wishes to contain costs.
Managing large, multiyear projects. In the past, SSA has experienced  difficulties managing complex, multiyear efforts. In prior work, we  reported that SSA has cancelled numerous demonstration projects  due to limitations or weaknesses in design or implementation and  lacked sufficient controls to ensure effective management of its  demonstration projects. We have also reported on challenges that  SSA faced managing two separate redesigns of its disability decision- making process. While the OIS project alone represents a complex  effort, stakeholders have emphasized the importance of this project  progressing in close concert with other ongoing research managed by  another part of SSA, placing greater emphasis on broad and effective  project planning. In light of past experiences and as SSA faces a  potential change in leadership in 2013, the management  shortcomings we identified increase the vulnerability of this complex  project.
Given the range of challenges and potential risks they pose to the  success of the OIS project, it is important to identify and carefully explore  feasible alternatives that may mitigate these risks. Examples of potential  alternatives suggested by experts and other observers in the disability  field include:
Leveraging O*NET. In a report issued in 2010, SSA’s advisory panel  found that it did not believe that O*NET would meet SSA’s needs, in  part, because it is not detailed enough. However, the National  Academy of Sciences also issued a report in 2010 and concluded that  O*NET could be altered to better meet SSA’s needs for disability  adjudication. SSA officials have told us that modifying O*NET would  not result in savings for the agency, but they have not conducted any  analysis to determine this. While most disability experts agree that  O*NET in its current form would not be a suitable source of  occupational information for SSA, the National Academy of Sciences  study noted that there are potential linkages between O*NET and an  SSA OIS that are worthy of more careful exploration on SSA’s part  and that leveraging one system to serve broader purposes could be  most cost-effective overall.
Adjusting the scope of the OIS. SSA could reduce costs by limiting the  scope of the OIS. For example, the OIS could be designed to capture  fewer occupations. SSA officials have told us that since they are early  in their design, they do not know how many occupations will be  included in their OIS. Since this decision is fundamental to informing  the OIS data collection methods, feasibility, and cost, consideration of  alternative scope should be given high priority attention.
Limiting data collection methods. Another key factor that SSA must  consider is the method they will use to collect data for their OIS as  some methods are more costly than others. For example, among the  methods SSA is considering, conducting on-site job analyses of  occupations would be more expensive than surveys, which are also  more resource and time intensive than collecting information through  telephone interviews.
Leveraging resources from other agencies and OIS users. Other  federal agencies with experience in data collection and occupational  information have resources that SSA could leverage. SSA officials  told us that they have been considering how they could benefit from  the infrastructure that other agencies such as Labor and the U.S.  Census Bureau have for data collection, but that they have not  identified specific ways to leverage those resources. Additionally,  while SSA needs to assure the OIS serves its most basic needs, it  may be appropriate for SSA to explore the possibility of cost sharing  or applying users’ fees with organizations that currently rely on the  outdated DOT and could ultimately benefit from using a more modern  and comprehensive OIS database.
section_title: SSA Has Taken Other Steps to Modernize Criteria but Faces Limitations
subsections: section_title: SSA Has Taken Steps to Incorporate Modern Concepts of Disability into Eligibility Criteria
paragraphs: Although SSA’s adult disability programs were initially built upon the  assumption that certain severe medical conditions equate to work  incapacity, through its medical listings updates and ongoing research, SSA  has taken steps to modify its eligibility criteria. SSA now is taking a more  modern view of disability that looks beyond the claimant’s medical condition  by giving greater consideration to his or her functional capacity, that reflects  the International Classification of Functioning, Disability and Health (ICF)  framework. According to the 2007 Institute of Medicine report, a modern  concept of disability should recognize that disability is not just inherent in  the individual and his or her medical condition, but is the result of complex  interactions between the person, the person’s medical impairments,  assistive devices to which they have access, and features of his or her  socioeconomic environment, such as the presence or lack of accessible  transportation and workplace accommodations. Under this concept, as  described in this report, two people with the same impairment might have  different degrees of work disability for a variety of reasons.
Updates to the medical listings have been one vehicle through which SSA  can include an assessment of an individual’s functional abilities to  determine whether an impairment prevents work. For example, as part of  SSA’s comprehensive revision to the listings for the immune system, the  agency included several functional criteria, such as performing activities  of daily living, maintaining social functioning, and completing tasks in a  timely manner despite deficiencies in concentration or persistence.  Generally, SSA officials, adjudicators, and disability experts we spoke  with support incorporating appropriate functional criteria into the medical  listings to facilitate a more reliable assessment of an individual’s ability to  work. However, some have also noted that as SSA continues to more  broadly incorporate functional criteria into listings updates, such efforts  may result in a more subjective assessment by adjudicators that could  increase the difficulty of making step 3 disability determinations.
Since 2008, SSA has had an ongoing interagency agreement with the  National Institutes of Health (NIH) to conduct short- and long-term  research that has informed SSA’s efforts to incorporate functional  information into the disability criteria. For example, in its 2011 annual  report to SSA, NIH presented its findings on the use of functional criteria,  as defined by the ICF, in the adult listings. The objective of the project  was to quantify the comprehensiveness and consistency of functional  criteria among the adult listings and any influence this has in  determination of outcomes. NIH found that while the use of functional  terms were particularly apparent in five body systems, nearly one-half of  the 14 body systems did not capture the influence of health conditions  and impairment on human functioning. NIH is currently working on  another project, which examines the presence and consistency of  functional terms in the listings criteria. It is estimated the project will be  completed by 2013.
Besides exploring how medical listings could further incorporate  functional considerations, SSA is also sponsoring longer term research  through the NIH to develop an automated method to more quickly and  comprehensively evaluate a claimant’s functional abilities for its disability  determinations. Specifically, project researchers are developing a  computer-based tool to rapidly and reliably assess the functional abilities  of individual claimants considering their medical conditions. As  envisioned, the claimant, a medical provider, or both would respond to a  series of questions through the computer-based tool, which would likely  take an hour to complete the entire battery, perhaps less if a more limited  assessment is warranted. The project’s intended scope encompasses six  areas of function consistent with the ICF activity domain: (1) mobility, (2)  interpersonal/social interaction and relationships, (3) self-care, (4)  communication, (5) general tasks and demands (such as multitasking and  carrying out daily routines), and (6) learning and applying knowledge. NIH  researchers said they plan to consider the use of common personal  assistive devices, such as wheelchairs, in developing the tool. As of April  2012, NIH had completed preliminary testing of instruments in two of the  six areas—mobility and interpersonal/social interaction and  relationships—with a sample of claimants and providers and had begun  analyzing the data. SSA and NIH officials anticipate several benefits from  the functional assessment tool, such as providing information on the  impact of impairments more consistently, comprehensively, and early in  the disability determination process. This information would help  adjudicators more accurately assess whether a person can perform  certain kinds of work given his or her functional and occupational  capabilities.
While this research is promising, there are several unknown variables.  SSA officials said they have not yet determined when or how the tool will  be integrated into the disability determination process. NIH officials  indicated they believe that the tool may be most useful early in the  determination process. SSA officials said they expect to pilot the  functional assessment tool after all relevant domains have been tested  and validated, which will likely be by 2016. Additionally, NIH researchers  told us that, ultimately, this tool, which collects information on the  applicant, would need to be aligned with the OIS, which collects  information on the demands of work. SSA and NIH officials reported they  have held meetings to share their progress to date, but both projects are  in the early stages and additional coordination will be needed going  forward.
SSA is in the preliminary stages of taking additional steps to broaden its  use of functional criteria throughout the disability determination process in  response to other findings by NIH. Specifically, NIH evaluated a number  of forms that SSA uses during the disability application and determination  processes to determine how well these forms captured information on  claimant functional activity as described by the ICF. The researchers  found that current SSA assessment processes had major gaps in  covering the ICF concepts. For example, SSA captured only limited  information relative to sensory experiences, such as watching or listening,  learning and applying knowledge, communication, and interpersonal  interactions and relationships, all of which NIH considered particularly  relevant to work. NIH concluded that the gaps need to be addressed in  order to characterize individual functioning more comprehensively in  relationship to the demands of the workplace. Another step the agency  reports that it plans to take is the issuance of a Federal Notice of  Solicitation of Collaboration from federal agencies in developing a  standard for coding functional capacity in federal disability programs  based on the ICF. The notice will explain that SSA believes using the ICF  would help standardize how agencies describe and measure different  aspects of disability, improve the clarity and comparability of research  findings, and strengthen the base of scientific knowledge that guides  public policies and health practices. In addition, SSA has asked the  Institute of Medicine to plan an international symposium focused on how  best to use and assess function in the disability determination process.  SSA officials also stated that the agency may ultimately revise the  residual functional capacity forms based on criteria from the ICF.
section_title: SSA Faces Constraints to Incorporating Modern Concepts
paragraphs: Consistent with modern views of disability, a key consideration in  assessing disability is that the environment can hinder or enhance an  individual’s ability to function. While assistive devices and workplace  accommodations can play a critical role in an individual’s ability to  function in the work environment, SSA does not always consider them in  its assessment of disability. Regarding the incorporation of assistive  devices into the medical listings, SSA officials and experts we spoke with  expressed concern about the extent to which specific technologies should  be incorporated if they are not widely available. SSA officials told us that  they currently incorporate assistive devices into the medical listings once  these devices become standard in the medical community—a threshold  that SSA officials described as generally involving some combination of  availability, accessibility, and insurance coverage. After an assistive  device, such as a prosthetic device for walking, is incorporated into a  listing, adjudicators must evaluate the individual’s ability to walk with the  device being used. For example, evaluations of people who have had  amputations involving a lower extremity or extremities are to be done with  the prosthetic device in place. When we asked if wheelchairs are  considered standard in the medical community, and whether SSA  considers how individuals with wheelchairs might function in today’s  knowledge-based labor market, given their age and education, we heard  conflicting information from SSA officials. Some explained that individuals  in wheelchairs are generally allowed at step 3 on the basis of their  underlying medical condition, without evaluating how their disability might  be assessed at steps 4 or 5. Other officials maintained that according to  SSA’s policy, an individual in a wheelchair would be evaluated for upper  body strength or other medical issues. However, SSA officials we spoke  with said they have not evaluated these types of allowances specifically.
Regarding workplace accommodations, SSA officials said their policy is to  not consider them for several reasons. First, officials cited SSA’s inability  to ensure that workplace accommodations are provided by employers—a  concern shared by other disability experts we interviewed. SSA officials  also indicated the agency would be unable to assess the effectiveness of  workplace accommodations for claimants. Further, officials noted that  SSA already faces resource constraints managing its disability claims  workload and expanding the scope of individualized assessments would  exacerbate those constraints. Finally, they noted that data on the  availability and use of workplace accommodations are lacking.
SSA’s policy notwithstanding, some opportunities exist for SSA to learn  more about the availability of workplace accommodations. For example,  developing the OIS may provide an opportunity to obtain some, albeit  limited, information on workplace accommodations for the disability  determination process. Specifically, while SSA officials reported that they  do not plan to collect specific information on workplace accommodations  as they develop the system, they may collect more specific information on  the physical requirements of different jobs, such as options for sitting or  standing. Some experts we spoke with agreed that while information on  workplace accommodations would be immensely useful to include in the  OIS, given the current scope of that project, they agreed collecting this  information via the OIS would likely be too great a task for SSA to  accomplish with existing budget constraints and time frames. However,  another potentially viable option for collecting more information about  workplace accommodations might be the Disability Research Consortium,  which SSA is creating pursuant to section 1110 of the Social Security Act,  as amended. SSA envisions the consortium as a 5-year cooperative  agreement that will serve as a national resource for fostering high quality  research, communication, and education on matters related to disability  policy, such as identifying or eliminating barriers encountered by people  with disabilities in returning to or maintaining work.
Researchers and disability experts have commented on the limitations of  SSA’s current disability program eligibility criteria to fully consider whether  an individual can function in the workplace due to limited consideration of  assistive devices, including those in the workplace—limitations that we  have also noted. Moreover, officials we spoke with from an organization  of vocational examiners expressed frustration when they see young  individuals who could work with minor accommodations being provided  disability benefits likely throughout their working life, rather than receiving  support to pursue work. Representatives of the organization added that  minor accommodations can include a stool for sitting or devices to assist  with vision impairments. Although giving broad consideration of assistive  devices and workplace accommodations may be difficult to incorporate  into the current disability criteria and process, considering how common  and inexpensive workplace supports may affect work disability seems  feasible and reasonable.
section_title: Conclusions
paragraphs: SSA has taken important and concrete steps toward modernizing its  disability criteria, but varied challenges may prevent the agency from  achieving its goals. Specifically, SSA has a better process in place for  updating medical listings but delays in meeting its own goals will likely  continue unless the agency explicitly identifies and assigns the resources  needed to achieve them. SSA is also taking bold and needed steps to  replace outdated occupational information with a new OIS. However,  especially in an era of governmentwide fiscal constraints, success of its  OIS may hinge on SSA prioritizing its analysis of risks and feasible  alternatives to address potential funding shortfalls and other significant  challenges. Regardless of the shape and scope of the OIS, absent a  complete, reliable, and transparent cost estimate and schedule, SSA risks  schedule and performance shortfalls. Finally, SSA is sponsoring  promising research to increase consideration of functional ability in its  medical listings and other aspects of eligibility criteria—research that  experts believe must be aligned with the design of SSA’s new OIS system  at the earliest opportunity to best serve the disability determination  process. On the other hand, SSA is not considering the full range of  assistive devices and workplace accommodations available today. While  SSA and others raise valid concerns as to their universal availability and  other considerations, in the absence of studies on how certain assistive  devices and workplace accommodations are playing a role in helping  individuals with impairments to stay at work or return to work, and their  costs in comparison with potentially many years of disability benefit  payments, SSA may be missing an opportunity to potentially assist  individuals with disabilities to re-engage with the workforce and to target  finite resources efficiently and effectively.
section_title: Recommendations for Executive Action
paragraphs: 1.  To achieve the goal of updating listings for each body system within  SSA’s 5-year time frame, we recommend that the Commissioner of  Social Security explicitly identify the resources needed to achieve this  goal, such as staff, contractors, and technology aids, and its plans to  overcome any resource limitations. 2.  To ensure that its work to revise occupational information is feasible  and cost effective, and to improve its chance for success, we  recommend that the Commissioner of Social Security: formally assess risks to the success of the OIS—addressing such  challenges as related to controlling cost, acquiring expertise,  managing project complexity, and coordinating with ongoing and  related SSA research—and develop appropriate mitigation  strategies, and develop a comprehensive and reliable cost estimate and schedule  for the life cycle of the project, in accordance with best practices.  3.  To help ensure that SSA’s disability decisions are as equitable and  consistent with modern views of disability as possible, we recommend  the Commissioner of Social Security conduct limited and focused  studies on the availability and effects of considering more fully  assistive devices and workplace accommodations in its disability  determinations.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft copy of this report to SSA and the Departments of  Commerce, Health and Human Services, and Labor for review and  comment. The Department of Commerce did not provide comments.  SSA, Labor, and the Department of Health and Human Services provided  technical comments, which we incorporated as appropriate. SSA also  provided written comments, which are reproduced in appendix V.
SSA agreed with the first two recommendations and disagreed with the  third recommendation that the agency should conduct limited and focused  studies on the availability and effects of considering more fully assistive  devices and workplace accommodations in its disability determinations.  SSA stated that conducting such studies is inconsistent with Congress’  intentions.  Specifically, SSA noted that the Americans with Disabilities  Act of 1990 (ADA), as amended, addresses requirements for workplace  accommodations. SSA referred to a 1999 Supreme Court case which  acknowledged the complexity of SSA involvement in determining the  availability of reasonable accommodations and also noted that ADA  determinations are separate from disability determinations under the  Social Security Act.  SSA asserts that because Congress has made no  effort to change the balance between its disability programs and the ADA  in the past 22 years, it would be inappropriate for SSA to spend its limited  administrative resources on “an initiative that would undermine the  balance Congress chose to strike.”
Notwithstanding SSA’s response, we continue to believe that SSA should  conduct limited and focused studies on the availability and effects of  considering more fully assistive devices and workplace accommodations  in its disability determination process. Although SSA asserts that  workplace accommodations are addressed by the ADA, this act serves a  very different purpose than SSA’s disability programs. We do not think  that the fact that workplace accommodations are addressed by the ADA  would necessarily preclude SSA from potentially considering them in  making disability determinations. Further, although we agree with SSA  that Congress has not explicitly directed the agency to consider  incorporating assistive devices and workplace accommodations into its  disability determinations process, we also note that Congress has not  explicitly prohibited SSA from making such considerations.  Because the  language of the Social Security Act is silent on this issue, and in the  absence of clear Congressional intent, we think it would be reasonable for  SSA to conduct limited studies on the feasibility of considering such  factors.
SSA further cites the 1999 Supreme Court decision in Cleveland v. Policy  Management Systems Corporation to point out that assessing reasonable  accommodation may turn on highly disputed workplace-specific matters,  and that an SSA misjudgment about that detailed matter could deprive a  disabled person of the financial support the statute provides. While this is  true, SSA fails to acknowledge that by not incorporating such factors, it  may be providing benefits to individuals who are capable of working with  accommodations, thereby potentially expending scarce government  resources in a manner that may not be most appropriate. SSA’s response  acknowledges that the agency’s resources are limited—a concern that we  share.  By conducting limited and focused studies on this issue, SSA  would be in a better position to thoughtfully weigh the costs and benefits  of these various policy options before deciding on an appropriate course  of action.
As we discussed in the report, SSA has avenues to research the  availability of select devices or accommodations and the impact of their  inclusion on disability determinations, such as through the SSA funded  Disability Research Consortium, which is charged to conduct research to  identify and eliminate barriers encountered by people with disabilities in  returning to work or maintaining their ability to work. SSA could also  incorporate discussion of assistive devices and workplace  accommodations into its planned symposium with the Institute of  Medicine that will assess the inclusion of functional measures in the  disability determination process. Through these efforts, SSA may be able  to identify certain common and widespread workplace accommodations  that could be incorporated into the disability criteria, just as it incorporates  certain medical treatments and assistive devices into the medical listings  when they become a medical standard. Despite SSA’s efforts to better  address an individual’s functioning in its disability criteria, without at least  evaluating the costs and benefits of considering more assistive devices  and workplace accommodations, SSA is likely to face ongoing barriers in  its efforts to integrate a more modern concept of disability into its  determination process.
As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to the  Commissioner of the Social Security Administration and the Secretaries  of Commerce, Health and Human Services, and Labor; relevant  congressional committees; and other interested parties. In addition, this  report will be available at no charge on GAO’s website at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-7215 or bertonid@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix VI.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: We were asked to assess the Social Security Administration’s (SSA)  plans and efforts to revise its disability criteria. Specifically, we examined  the status and management of efforts to update its medical listings to  reflect current medical knowledge and develop a new occupational  information system (OIS) to reflect labor market changes, and we  identified other steps taken by SSA to incorporate a modern view of  disability into its eligibility criteria. For this review, we narrowed our scope  to criteria used for initial adult disability determinations that involve the  medical listings at step 3 or a medical-vocational assessment at steps 4  and 5 of SSA’s five-step sequential process.
To determine the status of activities to revise the medical listings and to  develop the OIS, we reviewed our prior reports on the subject and SSA  Office of Inspector General reports, relevant federal laws and regulations,  and program documentation. This documentation included policies, such  as those listed in SSA’s Program Operations Manual System for  examiners; strategic goals for fiscal years 2011 and 2012 presented in  the Agency Performance Plan for 2012; and other guidance. We also  interviewed SSA officials from relevant offices with direct responsibility for  revising disability criteria, as well as those offices that provide support for  these efforts, and key project contractors and stakeholders. To obtain  contextual information on modernization, we reviewed relevant literature,  including studies, position papers, and testimonies from disability groups  and commissions and interviewed disability experts. We assessed the  reliability of data used in this report and found it to be sufficiently reliable  for our purposes.
To determine the extent to which SSA’s efforts to revise its medical  listings and develop a new OIS were anchored in sound project  management practices, we first identified sound project management  practices by reviewing A Guide to the Project Management Body of  Knowledge, our guidance on internal controls, and the Government  Performance and Results Act of 1993. We also identified our recent work  (for fiscal years 2008, 2009, and 2010) that evaluated federal planning  efforts. Through these efforts, we identified six practices for sound project  management which we used as a framework for evaluating SSA’s efforts  to revise the medical listings and develop the OIS. Although there is no  established set of requirements for all plans, we determined that these  practices help implementing parties and decision makers effectively  shape policies, programs, priorities, and resource allocations so that they  can achieve desired results while ensuring accountability. While these  practices may be organized in a variety of ways or use different terms, for  the purposes of this report, we grouped them into six categories, from  plan conception to implementation. Given the differences among SSA’s  efforts to update the medical listings and develop the OIS, we selected  broad, higher-level criteria that may apply to a wide variety of projects or  plans (see table 5).
Additionally, we compared the cost estimate and schedule for completing  SSA’s OIS and related documents with best practices in our Cost  Estimating and Assessment Guide. We compared these practices to the  OIS project because it requires a significant commitment of resources  and time by SSA to complete and will result in an end product. For SSA’s  OIS cost estimate and schedule, we scored each best practice as follows:
Not met—SSA provided no evidence that satisfies any of the criteria.
Minimally met—SSA provided evidence that satisfies a small portion  of the criteria.
Partially met—SSA provided evidence that satisfies about half of the  criteria.
Substantially met—SSA provided evidence that satisfies a large  portion of the criteria.
Met—SSA provided complete evidence that satisfies the entire  criteria.
We provided the results of our schedule and cost analyses to SSA  officials and met with them to confirm the results. Based on the interviews  and additional documentation provided by SSA officials, we updated the  results of our analyses, as needed.
section_title: Appendix II: Progress Updating Six Body Systems since Last Comprehensive Revision
section_title: Appendix III: Description of Cost Estimating Best Practices
subsections: section_title: Characteristic Comprehensive
paragraphs: Characteristic description  The cost estimate should include all costs necessary to achieve agency objectives including government and  contractor labor costs as well as any necessary material or equipment costs. Comprehensive cost estimates  should be structured in sufficient detail to ensure that cost elements are neither omitted nor double counted.  Specifically, the cost estimate should be based on a standardized structure that allows a program to track cost  and schedule consistently over time. Finally, where information is limited and judgments must be made, the cost  estimate should document all cost-influencing ground rules and assumptions.
The cost estimate should be supported by detailed documentation that describes the purpose of the estimate,  the program background and system description, the scope of the estimate, the ground rules and assumptions,  all data sources, estimating methodology and rationale, and the results of the risk analysis. Moreover, this  information should be captured in such a way that the data used to derive the estimate can be traced back to,  and verified against, the sources.
The cost estimate should be based on an assessment of most likely costs (adjusted for inflation), documented  assumptions, historical cost estimates, and actual experiences on other comparable programs. Estimates should  be cross-checked against an independent cost estimate for accuracy, double counting, and omissions. In  addition, the estimate should be updated to reflect any changes.
section_title: Appendix IV: Description of Scheduling Best Practices
paragraphs: Appendix IV: Description of Scheduling Best  Practices Description  A schedule should reflect all activities defined in the program’s work breakdown structure and include  all activities to be performed by the government and contractor.
The schedule should be planned so that all activities are logically sequenced in the order they are to  be carried out.
The schedule should realistically reflect the resources (i.e., labor material and overhead) needed to  do the work, whether all required resources will be available when needed, and whether any funding  or time constraints exist.
The schedule should reflect how long each activity will take to execute.
The schedule should be horizontally and vertically integrated—that is, it should link already  sequenced activities with outcomes while also delineating the relation of supporting tasks and  subtasks to upper-level milestones. Such mapping among levels enables different groups to work to  the same master schedule.
The schedule should identify the critical path, or those activities that, if delayed, will negatively impact  the overall project completion date. The critical path enables analysis of the effect delays may have  on the overall schedule.
The schedule should identify float—the amount of time an activity can slip in the schedule before it  affects other activities—so that flexibility in the schedule can be determined. As a general rule,  activities along the critical path typically have the least amount of float.
The schedule should include a schedule risk analysis that uses statistical techniques to predict the  probability of meeting a completion date. A schedule risk analysis can help management identify and  understand the most important risks and focus on mitigating them.
The schedule should use realistic durations for activities and be monitored to determine when  forecasted completion dates differ from the planned dates. This analysis can be used to assess  whether schedule variances will affect future work.
section_title: Appendix V: Comments from the Social Security Administration
section_title: Appendix VI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Michele Grgich, Assistant  Director; James Bennett; Kate Blumenreich; Tisha Derricotte; Jennifer  Echard; Julie DeVault; Alex Galuten; Sheila McCoy; Patricia M. Owens;  Carol Petersen; Karen Richey; Anjali Tekchandani; Kathleen Van Gelder;  and Walter Vance made key contributions to this report. Also contributing  to this report were Jaime Allentuck, Susan Bernstein, and Amy Frazier.