SSA Disability Decision-Making: Additional Measures Would Enhance Agency's Ability to Determine Whether Racial Bias Exists
Released: 2002-09-09T00:00:00

Letter

Background

DI and SSI are the two largest federal programs providing cash assistance to people with disabilities. Established in 1956, DI provides monthly payments to workers with disabilities (and their dependents or survivors) under the age of 65 who have enough work experience to be qualified for disability benefits. Created in 1972, SSI is a means-tested income assistance program that provides monthly payments to adults or children who are blind or who have other disabilities and whose income and assets fall bellow a certain level. To be considered eligible for either program as an adult, a person must be unable to perform any substantial gainful activity by reason of a medically determinable physical or mental impairment that is expected to result in death or that has lasted or can be expected to last for a continuous period of at least 12 months. Work activity is generally considered substantial and gainful if the person’s earnings exceed a particular level established by statute and regulations.In calendar year 2001, about 6.1 million working age individuals (age 18- 64) received about $59.6 billion in DI benefits, and about 3.8 million working-age individuals received about $19 billion in SSI federal benefits.

To obtain disability benefits, a claimant must file an application at any of SSA’s offices or other designated places. If the claimant meets the nonmedical eligibility criteria, the field office staff forwards the claim to the appropriate state DDS office. DDS staff—generally a team comprised of disability examiners and medical consultants—review medical and other evidence provided by the claimant, obtaining additional evidence as needed to assess whether the claimant satisfies the program requirements, and make the initial disability determination. If the claimant is not satisfied with the DDS determination, the claimant may request a reconsideration within the same DDS. Another DDS team will review the documentation in the case file, as well as any new evidence the claimant may submit, and determine whether the claimant meets SSA’s definition of disability. In 2001, the DDSs made 2.1 million initial disability determinations and over 514,000 reconsiderations.

If the claimant is not satisfied with the reconsideration, the claimant may request a hearing by an ALJ. Within SSA’s OHA, there are approximately 1,100 ALJs who are located in 140 hearing offices across the country. The ALJ conducts a new review of the claimant’s file, including any additional evidence the claimant submitted since the DDS decision. The ALJ may also hear testimony from medical or vocational experts and the claimant regarding the claimant’s medical condition and ability to work. The hearings are recorded, and claimants are usually represented at these hearings. In fiscal year 2001, ALJs made over 347,000 disability decisions.

SSA is required to administer its disability programs in a fair and unbiased manner. However, in our 1992 report, we found that, among ALJ decisions at the hearings level, the racial difference in allowance rates was larger than at the DDS level and did not appear to be related to severity or type of impairment, age or other demographic characteristics, appeal rate, or attorney representation. We recommended, and SSA agreed, to further investigate the reasons for the racial differences at the hearings level and act to correct or prevent any unwarranted disparities.

SSA’s Study of Racial Disparities Was Extensive, but Methodological Weaknesses in Available Documentation Preclude Conclusions

Following our report, SSA undertook an extensive effort to study racial disparities in ALJ decisions at the hearings level, but weaknesses in available documentation preclude conclusions from being drawn. The study involved 4 years of data collection, outside consultants, and many staff who collected and analyzed data from over 15,000 case files. Although the results were not published, SSA officials told us that their statistical analyses of these data revealed no evidence of racial disparities. On the basis of our review of SSA’s internal working papers and other available information, we identified several weaknesses in sampling and statistical methods. Presently, SSA has no further plans to study racial disparities but, if it did, its ability to do so would likely be hampered by data limitations.

SSA’s Effort to Study Racial Disparities Was Extensive, but Results Were Not Published

In response to our 1992 report, SSA initiated a study of racial disparities at the ALJ level that involved several components of the agency. SSA obtained help in designing and conducting the study from staff in its Office of Quality Assurance and Performance Assessment; the Office of Research, Evaluation and Statistics; and the Office of Hearings and Appeals. SSA also created a new division within the Office of Quality Assurance—the Division of Disability Hearings Quality—to spearhead the collection of data needed to study racial disparities and to oversee ongoing quality assurance reviews of ALJ decisions.

Data collection for this study was a large and lengthy effort. In order to construct a representative sample of cases to determine whether race significantly influenced disability decisions, SSA selected a random sample each month from the universe of ALJ decisions, stratifying by race, region, and decisional outcome (allowance or denial). This sample of over 65,000 cases was drawn over a 4-year period—from 1992 to 1996. Then, for each ALJ decision that was selected to be in the sample, SSA requested the case file and a recording of the hearing proceedings from hearing offices and storage facilities across the country. Obtaining this documentation was complicated by the fact that files were stored in different locations, depending on whether the case involved an SSI or DI claim, and whether the ALJ decision was an allowance or denial.

In addition to obtaining files and tapes, the data collection effort included a systematic review of each case—the results of which SSA used, in part, for its analysis of racial disparities. Specifically, each case used in the analysis received three reviews: a peer review by an ALJ, a medical evaluation performed by one or more medical consultants (depending on the number and type of impairments alleged by the claimant), and a general review of the documentation and decisions by a disability examiner. In total, a panel of 10 to 12 ALJs, whose composition changed every 4 months, worked full-time to review cases. In addition, over a 4-year period, 37 to 55 staff, including disability examiners, worked full- time reviewing case files that were used for this study. Ultimately, about 15,000 cases received all three reviews necessary for inclusion in this study.

During and after the 4-year data collection effort, SSA worked with consultants to analyze the data in order to determine the effect of race on ALJ decisions. SSA used descriptive statistics to show that overall application and allowance rates of African Americans differed from whites. In addition, SSA used multivariate analyses to examine the effect of race on ALJ decisions while controlling for other factors that influence decisions. One of SSA’s consultants—a law professor and recognized expert in disability issues—reviewed SSA’s analytical approach and evaluated initial results. In his report to SSA, this consultant expressed overall approval of SSA’s data collection methods, but made several recommendations on how the analysis could be improved—some of which SSA incorporated into later versions of its analysis. SSA subsequently hired two consulting statisticians to review later versions of the analysis. These statisticians expressed concerns about SSA’s methods and offered several suggestions. According to SSA officials, these suggestions were not incorporated into the analysis because they were perceived to be labor intensive and SSA was not sure the effort would result in more definitive conclusions.

According to SSA officials, the agency’s final analysis of the data revealed no evidence of racial disparities, but the results were considered to be not definitive enough to warrant publication. Specifically, SSA officials told us that, by 1998, they found no evidence that race significantly affected ALJ decisions for any of the regions. However, these officials also told us that, due to general limitations of statistical analysis, especially as applied to such complex processes as ALJ decision making, they believed that they could not definitively conclude that no racial bias existed. Given the complexity of the results and the topic’s sensitive nature, SSA officials told us the agency decided not to publish the conclusions of this study.

Available Documentation of Racial Disparities Study Indicated Some Methodological Weaknesses

From our review of SSA’s internal working papers pertaining to the study, and information provided verbally by SSA officials, we identified several weaknesses in SSA’s study of racial disparities. These weaknesses include: using a potentially nonrepresentative final sample of cases in their multivariate analyses, performing only limited analyses to test the representativeness of the final sample, and using certain statistical techniques that could lead to inaccurate or misleading results.

Although SSA started with an appropriate sampling design, its final sample included only a small percentage of the case files in its initial sample in part because staff were unable to obtain many of the associated case files or hearing tapes. SSA was not able to obtain many files and tapes because they were missing (i.e., lost or misplaced) or they were in use and were not made available for the study. For example, according to SSA officials, files for cases involving appeals of ALJ decisions to SSA’s Appeals Council—about half of ALJ denials—were in use and, therefore, excluded from the study. In addition, SSA officials told us that, due to resource constraints, not all of the obtained files underwent all three reviews, which were necessary for inclusion in SSA’s analysis of racial disparities. In the end, less than one-fourth of the cases that were selected to be in the initial sample were actually included in SSA’s final sample.

With less than one-fourth of the sampled cases included in the final sample, SSA took steps to determine whether the final sample of cases was still representative of all ALJ decisions. While the investigation SSA undertook revealed no clear differences between cases that were and were not included in the final sample, we found no evidence that SSA performed certain analyses that could have provided more assurance of the sample’s representativeness. For example, SSA made some basic comparisons between claimants who were included in the final sample and those that were in the initial sample but not the final sample. SSA’s results indicate that these two groups were fairly similar in key characteristics such as racial composition, years of education, and years of work experience. However, we found no indication in the documentation provided to us that SSA tested whether slight differences between the two groups were or were not statistically significant. Further, we found no indication that SSA compared the allowance rates of these two groups. This is an important test because, in order to be statistically representative, claimants in the final sample should not have had significantly different allowance rates from claimants who were not included in the final sample. In addition, although children were not included in SSA’s analysis of racial disparities, SSA’s tests to determine the representativeness of the final sample included children in one group and excluded children from the other. By including children in one of the comparison groups, SSA could not assess whether characteristics of the adults in the two groups were similar.

Another weakness, as documented in internal working papers available for our review, was the inclusion of certain variables in the multivariate analyses of ALJ decisions, which could lead to biased results. SSA guidelines clearly define the information that should be considered in the ALJ decision, and SSA appropriately included many variables that capture this information in its multivariate analysis. However, SSA also included several variables developed during the review process that reflected the reviewer’s evaluation of the hearing proceedings. For example, SSA included a variable that assessed whether the ALJ, in the hearing decision, appropriately documented the basis for his or her decision in the case file. This variable did not influence the ALJ’s decision, but evaluated the ALJ’s compliance with SSA procedures and should not have been included in the multivariate analysis. This and other variables that reflected a posthearing evaluation of ALJ decisions were included in SSA’s multivariate analysis. If these variables are associated with race or somehow reflect racial bias in ALJ decision making, including such variables in multivariate analysis will reduce the explanatory power of race as a variable in that analysis. For example, if a model includes a variable that may reflect racial bias—such as one that indicates the reviewer believed that the original ALJ decision was unfair or not supported—then that variable, rather than the race of the claimant, could show up as a significant factor in the model. The statisticians hired by SSA as outside consultants also expressed concern about the inclusion of these variables in SSA’s analyses.

Finally, in its internal working papers, SSA used a statistical technique— stepwise regression—that was not appropriate given the characteristics of its analysis. Specifically, SSA researchers first identified a set of variables for potential use in their multivariate analysis—variables drawn mostly from data developed during the case file review process. Then, to select the final set of variables, SSA used stepwise regression. Stepwise regression is an iterative computational technique that determines which variables should be included in an analysis by systematically eliminating variables from the starting variable set that are not statistically significant. Using the results from this analysis, SSA constructed a different model for each of SSA’s 10 regions, which were used in SSA’s multivariate analysis to test whether African Americans were treated differently than whites in each region. Stepwise regression may be appropriate to use when there is no existing theory on which to build a model. However, social science standards hold that when there is existing theory, stepwise regression is not an appropriate way to choose variables. In the case of SSA’s study, statutes, regulations, rulings and SSA guidance establish the factors that ALJs should consider in determining eligibility, and thus indicate which variables should be included in a model. By using the results of stepwise regression, SSA’s regional models included variables that were statistically significant but reflected the reviewer’s evaluation of the hearing proceedings—which an ALJ would not consider in a hearing—and therefore were not appropriate. As mentioned earlier, including these variables may have reduced the explanatory power of other variables— such as race; this, in conjunction with the use of stepwise regression, may explain why race did not show up as statistically significant in the regional models. Had SSA chosen the variables for its model on the basis of theory and its own guidelines, race may have been statistically significant. The statisticians hired by SSA as consultants also noted this as a concern.

According to an SSA official, the analysts directly responsible for or involved in the study conducted other analyses that were not reflected in the documentation currently available and provided to us. For example, this SSA official told us that the analysts involved in the study would have tested the statistical significance of slight differences between the cases included and not included in the final sample. This official also said that the analysts used multiple techniques in addition to stepwise regression— and ran the models with and without variables that reflected posthearing evaluations—and still found no evidence of racial bias. However, due to the lack of available documentation, we were unable to review these analyses or corroborate that they were performed.

Future Studies of Racial Disparities Would Be Hampered by Data Limitations

Since the conclusion of its study of racial disparities, SSA no longer analyzes race as part of its ongoing quality review of ALJ decisions, and SSA officials told us they have no plans to do so in the future. SSA still samples and reviews ALJ decisions for quality assurance purposes. However, since 1997, SSA no longer stratifies ALJ decisions by race before identifying a random sample of cases—a practice that had helped to ensure that SSA had a sufficient number of cases in each region to analyze decisions by race. Although the dataset used for SSA’s ongoing quality assurance review of ALJ decisions still includes information on race, SSA no longer analyzes these data to identify patterns of racial disparities.

Even if SSA decided to resume its analysis of racial disparities in ALJ decisions, it would encounter two difficulties. First, SSA collects files for only about 50 percent of sampled cases in its ongoing review of ALJ decisions for quality assurance purposes, such that its final samples may be nonrepresentative of the universe of ALJ decisions. SSA uses this review data to produce annual and biennial reports on ALJ decision making. Data in these reports are also used to calculate the accuracy of ALJ decisions—a key performance indicator used in SSA’s 2000–03 performance plans pursuant to the Government Performance and Results Act. The reasons for obtaining only half of the files are the same, potentially biasing reasons as for the racial disparities study—files are either missing or not made available if the cases are in use for appeals or pending decisions. However, SSA’s annual and biennial reports do not cite the number or percentage of case files not obtained for specific reasons. In addition, SSA officials told us that they do not conduct ongoing analyses to test the representativeness of samples used for quality assurance purposes, and SSA’s annual and biennial reports do not address whether the final sample used for quality assurance purposes and for calculating the performance indicator for ALJ accuracy is representative of the universe of ALJ decisions. In addition to not obtaining about 50 percent of the case files, SSA officials told us that medical consultants and disability examiners only review a portion of cases for which a file was obtained due to limited resources.

Second, future analyses of racial disparities at either the DDS or hearings level is becoming increasingly problematic because, since 1990, SSA no longer systematically collects race data as part of its process in assigning Social Security Numbers (SSN). For many years, SSA has requested information on race and ethnicity from individuals who complete a form to request a Social Security card. Although this process is still in place, since 1990 SSA has been assigning SSNs to newborns through its Enumeration at Birth (EAB) program, and SSA does not collect race data through the EAB program. Under current procedures, SSA is unlikely to subsequently obtain information on race or ethnicity for individuals assigned SSNs at birth unless those individuals apply for a new or replacement SSN (due to change in name or lost card). As of 1998, SSA did not have data on race or ethnicity for 42 percent of SSI beneficiaries under the age of 9. As future generations obtain their SSNs through the EAB program, this number is likely to increase.

SSA Has Taken Limited Steps to Address Possible Racial Bias in Its Hearings Level Decision-Making Process

Concurrent with SSA’s study of racial disparities, SSA’s Office of Hearings and Appeals took several steps to address possible racial bias in disability decision making at the hearings level. These steps included providing diversity training, increasing recruitment efforts for minority ALJs, and administering a new complaint process for the hearings level to help ensure fair and impartial hearings. The complaint process was intended, in part, to help identify patterns of possible racial and ethnic bias and other misconduct; however, this process lacks mechanisms to help OHA easily identify patterns of possible racial or ethnic bias for further investigation or corrective action.

OHA Has Taken Some Steps to Address Possible Racial Bias

SSA’s OHA adopted a mandatory diversity sensitivity program in 1992. All of SSA’s incumbent ALJs were required to attend a 2- or 1-1/2-day course immediately after its development. In addition, the course (now 1 day in length) is included in a 3-week orientation for newly hired ALJs. The course was designed and is conducted by an outside contractor. The course addresses topics such as cultural diversity, geographic diversity, unconscious bias, and gender dynamics through a series of exercises designed to help the ALJs understand how their thought processes, beliefs, and past experiences with people influence their decision-making process.

OHA also increased its efforts to recruit minorities for ALJ and other legal positions, although the impact of these efforts on the racial/ethnic mix of SSA’s ALJ workforce has been limited. According to OHA officials, OHA has attended conferences held by several minority bar organizations, to raise awareness about the opportunities available at SSA to become an ALJ. In addition to having information booths and distributing information on legal careers at OHA, OHA presented a workshop called “How to Become an Administrative Law Judge at OHA,” at each conference. Despite these efforts, there have not been significant changes in the racial/ethnic profile of SSA ALJs.

In addition to these efforts, in 1993 SSA instituted a complaint process under the direction of OHA that provides claimants and their representatives with a new mechanism for voicing complaints specifically about bias or misconduct by ALJs. The ALJ complaint process supplements and is coordinated with the normal appeals process. All SSA claimants have the right to appeal the ALJ decision to the Appeals Council and, in doing so, may allege unfair treatment or misconduct. According to OHA officials, the vast majority of allegations of unfair hearings are submitted by claimants or their representatives in connection with a request for Appeals Council review. Under the 1993 process, claimants or their representatives may also file a complaint at any SSA office, send it by mail, or call it into SSA’s 800 number service. Any complaints where there is a request for Appeals Council review are referred to the Appeals Council for its consideration as part of its normal review. For complaints where the complainant did not request an Appeals Council review, the complaint is reviewed by the appropriate Regional Chief ALJ, and the findings are reported to the Chief ALJ. Regardless of how the complaint was filed or which office reviewed the complaint, OHA’s Special Counsel Staff is notified of all claims and any findings from either the Appeals Council or the Regional Chief ALJ. On the basis of these findings, OHA may decide to take remedial actions against the ALJ, such as a counseling letter, additional training, mentoring or monitoring, an official reprimand, or some other adverse action. OHA’s Special Counsel Staff may also decide to conduct a further investigation. Regardless of which office handles the complaint, OHA acknowledges the receipt of each complaint in writing, notifies the complainant that there will be a review or investigation (unless to do so would disrupt a pending hearing or decision), and notifies the complainant concerning the results of the investigation.

Officials from the Special Counsel Staff told us OHA receives about 700 to 1,000 complaints (out of 400,000 to 500,000 hearings) per year. About 90 percent of these are notifications from the Appeals Council that involve an allegation of bias or misconduct. Officials from the Special Counsel Staff also said that few complaints are related to race. For example, officials noted that, in response to a special request, Special Counsel Staff reviewed all 372 complaints filed during the first 6 months of 2001, and found that only 19 (5.1 percent) were in some way related to race.

ALJ Complaint Process Lacks Mechanisms to Identify Patterns of Racial Bias

While the ALJ complaint process provides a mechanism for claimants to allege discrimination, it lacks useful mechanisms for detecting patterns of possible racial discrimination. In SSA’s public notice on the creation of this process, it was stated that SSA’s Special Counsel would “collect and analyze data concerning the complaints, which will assist in the detection of recurring incidences of bias or misconduct and patterns of improper behavior which may require further review and action.” However, OHA’s methods of collecting, documenting, and filing complaints make this difficult to do. For example, in its instructions to the public, SSA directs complainants to describe, in their own words, how they believe they were treated unfairly. This flexible format for filing complaints may make it difficult for OHA to readily identify a claim alleging racial bias. In contrast, SSA’s Office of General Counsel’s complaint form specifically asks complainants to categorize their claim as being related to such factors as race or sex.

Similarly, OHA does not use a standardized internal cover-sheet to summarize key aspects of the review, such as whether the complaint involved racial or some other type of bias or misconduct, and whether the complaint had merit and what action, if any, was taken. The lack of a cover-sheet makes it difficult to quickly identify patterns of allegations involving race that have merit. In order to determine whether patterns exist, OHA staff would have to reread each complaint.

Additionally, OHA staff do not record key information about complaints— such as the nature of the complaint—in an electronic database so that patterns of bias can be easily identified. OHA’s Special Counsel Staff files complaints and related documents manually, and in chronological order by hearing office. According to OHA officials, this filing system was developed in 1993 when the process was created and complaint workloads were much lower. Today, SSA receives and reviews 700 to 1,000 complaints a year. In order to identify patterns of bias, Special Counsel Staff must not only reread each file, it must tabulate patterns by hand—a time-consuming process that it does not perform on a routine basis.

Finally, OHA does not currently obtain demographic information (such as race, ethnicity, and sex) on complainants, which are important in identifying patterns of bias. These data are important for identifying patterns of possible racial bias because complainants—aware only of their own circumstances and lacking a basis for comparison—may not specifically allege racial bias when they file a complaint about unfair treatment. Without demographic data, it is impossible to discern whether certain types of allegations are disproportionately reported by one race (or sex) and whether further investigative or corrective action is warranted. Although SSA is currently obtaining less race data in its process of assigning SSNs, OHA staff could still obtain data on race and sex for most complainants from the agency’s administrative data.

Conclusions

The steps SSA has taken over the last decade have not appreciably improved the agency’s understanding of whether or not, or to what extent, racial bias exists in its disability decision-making process. SSA’s attempt to study racial disparities was a step in the right direction, but methodological weaknesses evident in SSA’s remaining working papers prevent our concluding, as SSA did, that there is no evidence of racial bias in ALJ decision making. SSA does not have an ongoing effort to demonstrate the race neutrality of its disability programs. Moreover, the continuing methodological weaknesses in SSA’s ongoing quality assurance reviews of ALJ decisions hamper not only its ability to ensure the accuracy of those reviews but also its ability to conduct future studies to help ensure the race neutrality of its programs. Furthermore, in the longer term, SSA’s ability to analyze racial differences in its decision making will diminish due to a lack of data on race and ethnicity. Finally, SSA’s complaint process for ALJs lacks mechanisms—such as summaries of key information on each complaint, an electronic filing system, and information on the race and ethnicity of complainants—that could help identify patterns of possible bias. SSA is not legally required to collect and monitor data to identify patterns of racial disparities, although doing so would help SSA to demonstrate the race neutrality of its programs and, if a pattern of racial bias is detected, develop a plan of action.

Recommendations

To address shortcomings in SSA’s ongoing quality assurance process for ALJs—which would improve SSA’s assessment of ALJ decision-making accuracy—we recommend the agency take the following steps: conduct ongoing analyses to assess the representativeness of the sample used in its quality assurance review of ALJ decisions, including testing the statistical significance of differences in key characteristics of the cases included in the final sample with those that were not obtained; include the results of this analysis in SSA’s annual and biennial reports on ALJ decision making; and use the results to make appropriate changes, if needed, to its data collection or sampling design to ensure a representative sample.

To more readily identify patterns of misconduct, including racial bias, in complaints against ALJs, we also recommend that SSA’s Office of Hearing and Appeals: adopt a form or some other method for summarizing key information on each ALJ complaint, including type of allegation; use internal, administrative data, where available, to identify and document the race and/or ethnicity of complainants; and place the complaint information in an electronic format, periodically analyze this information and report the results to the Commissioner, and develop action plans, if needed.

Agency Comments and Our Response

We provided a draft of this report to SSA for comment. SSA concurred fully with our recommendations and agreed to take steps to implement them. In its general comments, SSA expressed concern that the title of the report might foster the perception that its disability decision making, particularly at the OHA level, is suspect. Although we believe the draft report’s title accurately reflected the report’s content and recommendations, we have modified the title to ensure clarity. SSA also cited a number of reasons for the low percentage of cases included in its final sample, as well as steps it took to ensure the representativeness of its final sample. Nevertheless, we continue to believe that SSA could have performed additional analyses to provide more assurance of the sample’s representativeness.

SSA also provided technical comments and clarifications, which we incorporated in the report, as appropriate. SSA’s general comments and our response are printed in appendix I.

We are sending copies of this report to the Social Security Administration, appropriate congressional committees, and other interested parties. We will also make copies available to others on request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions concerning this report, please call me or Carol Dawn Petersen, Assistant Director, at (202) 512-7215. Staff acknowledgments are listed in appendix II.

Appendix I: Comments from the Social Security Administration

GAO Comments

1. Although we believe the draft report’s title accurately reflected the content of our report and recommendations, we have modified the title to ensure clarity. This report and its recommendations are not restricted to a discussion of only two races. Although we referred to race and ethnicity in the second objective and the conclusion section of the draft reviewed by Social Security Administration (SSA), we added the word “ethnicity” to the recommendations and the body of the report to further clarify this issue. 2. We added language to a note in the report regarding the litigation SSA mentions and that SSA has increased the number of Regional Chief Administrative Law Judges (ALJs) who are members of a racial minority group from 1 to 3 since 1992. 3. We agree with SSA that the low proportion of cases included in the final sample is due to several factors. In our report, we cited several reasons for cases not being included in the final sample that are significant in terms of the number of affected cases and that we believe have the potential for being nonrandom in nature. On the basis of a subsequent discussion with SSA officials, we added a note in our report that a small proportion of cases were excluded because they were later identified as being cases that were not intended to be included in the sample. 4. Although SSA noted that it used “holdout samples and cross modeling” to ensure that the group of cases sampled for this study was essentially free of sampling bias, SSA officials explained to us that these techniques were not used to test for the representativeness of the final sample. 5. We agree with SSA that, with large sample sizes, even small differences generally are statistically significant, and that such statistically significant differences are not always substantively significant. We do not believe, however, that a large sample is sufficient reason to forego significance tests. Moreover, our report cited additional analyses that SSA could have performed to provide more assurance of the sample’s representativeness. Another approach that we do not cite in the report—but which SSA may wish to consider—is multivariate analysis of nonresponse. SSA performed bivariate comparisons of samples to determine whether they contained different proportions of cases with various characteristics. However, two samples can have very similar percentages of, for example, women and African Americans, but be very different with respect to the percentage of African American women. In contrast, multivariate analysis would allow SSA to look systematically and rigorously at different characteristics simultaneously.

Appendix II: GAO Contacts and Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

In addition to those named above, the following individuals made significant contributions to this report: Erin Godtland, Michele Grgich, Stephen Langley, and Ann T. Walker, Education, Workforce and Income Security Issues; Doug Sloane and Shana Wallace, Applied Research and Methods; and Jessica Botsford and Richard Burkard, General Counsel.