Decennial Census: Methods for Collecting and Reporting Data on the Homeless and Others without Conventional Housing Need Refinement
Released: 2003-02-19T00:00:00

Letter

Background

The procedures the Bureau used during the 1990 Census to count people without conventional housing had limitations that resulted in incomplete data. To address these limitations and help improve the quality of the data, the Bureau used a procedure for the 2000 Census called Service-Based Enumeration that attempted to count people where they receive services such as emergency shelters, soup kitchens, and regularly scheduled mobile food vans. Service-Based Enumeration also counted people in targeted nonsheltered outdoor locations such as encampments beneath bridges. The operation occurred from March 27 through March 29, 2000.

According to Bureau officials, Service-Based Enumeration was not designed, and was never intended, to provide a specific count of homeless persons. Instead, the operation was part of a larger effort to count people without conventional housing, including people in “institutional group quarters” such as correctional facilities, nursing homes, and mental hospitals, and “non-institutional group quarters” such as college dormitories, military quarters, and group homes. Service-Based Enumeration counted people in specific categories of noninstitutional group quarters.

To help ensure a complete count of people without conventional housing, the Bureau partnered with local governments and community advocacy groups to obtain lists of service locations and to assist with the enumeration. In some cases, the Bureau hired clients of the advocacy groups and other people trusted by the homeless to conduct Service-Based Enumeration. For example in Atlanta, an advocacy group for homeless veterans helped the Bureau employ homeless veterans to improve the count of this population. Local governments helped the Bureau as well, often investing considerable resources. For example, Los Angeles paid to keep its city-run shelters open on the night they were enumerated so that people using their services could be counted.

Scope and Methodology

To address your concerns about the Bureau’s dissemination of data on persons without conventional housing, we agreed to examine (1) the Bureau’s plans for reporting the results of Service-Based Enumeration and its reasons for changing those plans and (2) the Bureau’s protocols for releasing data. To accomplish these objectives, we interviewed key Bureau officials and reviewed relevant Bureau documents and data such as operational plans, decision memorandums, and the Bureau’s partnership program evaluation.

In order to obtain the perspective of data users, partners, and stakeholders, we conducted in-person and telephone interviews with homeless advocates, local government officials, and representatives of public service agencies in New York City, Los Angeles, Cleveland, Atlanta, and Washington, D.C. These cities had large numbers of people without conventional housing and they were actively involved with the Bureau during the 2000 Census. The organizations we contacted also provided relevant documentation, such as comprehensive file documents relating to partnership activities.

In addition to the above locations, we did our audit work at Bureau headquarters in Suitland, Maryland. Our audit work was conducted from April 2002 through September 2002 in accordance with generally accepted government auditing standards.

We requested comments on a draft of this report from the Secretary of Commerce. On November 21, 2002, the Secretary forwarded the Bureau’s written comments on the draft (see app. I). We address these comments at the end of this report.

The Bureau of the Census Twice Changed Plans for Reporting Service-Based Enumeration Data

Under the Bureau’s original plan for releasing Service-Based Enumeration data in Summary File-1 (SF-1), the emergency and transitional shelter count was one of several categories of noninstitutional group quarters data that were to be reported separately. Other people counted in the Service- Based Enumeration, including people counted at targeted nonsheltered outdoor locations, soup kitchens, and regularly scheduled mobile food vans, were to be combined and reported under the category “other non- institutional group quarters.” This category also included residential care facilities providing protective oversight, shelters against domestic violence, staff dormitories for nurses and interns at military and general hospitals, and living quarters for victims of natural disasters.

This decision was documented in an April 1999 internal memorandum from the Bureau’s Assistant Division Chief for Special Population Statistics to the Assistant Division Chief for Census Programs. The Service-Based Enumeration operation took place a year later, in March 2000. The April 1999 plan was in large part a reaction to the challenges the Bureau faced counting the emergency shelter and street population during the 1990 Census. Although the Bureau disseminated separate counts of people found at emergency shelters, preidentified street locations, and similar sites, the counts proved to be incomplete. Moreover, the Bureau stated in its October 2001 report that despite its warnings to the contrary, the data were sometimes misinterpreted as a “homeless” count.

The October report does not offer an example of this, but the misinterpretation clearly played a role in a lawsuit against the Bureau. As a result, when designing the 2000 census, the Bureau attempted to both improve the count and take precautions to ensure that the Service-Based Enumeration count would not be misconstrued as a count of the homeless.

The Bureau’s data dissemination plans took into account the recommendations of the Commerce Secretary’s 2000 Census Advisory Committee, a panel that included representatives of advocacy and other groups (including representatives from organizations that represent local governments) that met periodically to review the Bureau’s plans. The homeless population was represented by the National Coalition for the Homeless—an advocacy group that coordinates a network of 300 state and local housing and homeless organizations. In its January 1999 final report, the Census 2000 Advisory Committee recommended that special attention be paid to tabulating the results of Service-Based Enumeration and targeted outdoor enumerations so that they could not be aggregated and used as a homeless count.

The Bureau Changed Its Dissemination Plans Because of Data Quality Concerns

In January 2001, 5 months before the SF-1 release, the Bureau reversed its April 1999 decision to release emergency and transitional shelter data separately because of “data quality concerns.” Instead, as shown in figure 1, the Bureau planned to combine the emergency and transitional shelter data with the “other non-institutional group quarters.” This category contained data on a variety of living arrangements including facilities for natural disaster victims. The Bureau’s decision was contained in an internal Bureau memorandum from the Chief of the Population Division to the Chief of the Decennial Systems and Contracts Management Office. Bureau officials told us that the decision to exclude a separate emergency and transitional shelter count in SF-1 was made between December 2000 and January 2001, by the Director of the Decennial Census with input from the Associate Director Decennial Census, the Population Division, the Associate Director for Demographic Programs, the Decennial Management Division, and the Decennial Statistical Studies Division.

According to Bureau officials, their concerns focused on the accuracy of a new statistical procedure called “multiplicity estimation” that adjusted the number counted to better reflect the number of actual shelter users. Because Service-Based Enumeration only counted people who were at these facilities on the day of enumeration, the Bureau intended to use multiplicity estimation to calculate the number of people who used these facilities but were not present during Service-Based Enumeration. The multiplicity estimation procedure was based on information from those who were counted and on the number of times they used the service facilities in the prior week. An estimate of people not counted on the day of enumeration was added to the count of people who were. According to the Bureau, the multiplicity estimates tested well during the 1998 dress rehearsal for the 2000 Census possibly because the three rehearsal sites did not offer large enough sample sizes of the appropriate populations to adequately test this procedure. However, during the 2000 Census the Bureau found that a census question pertaining to facility usage upon which the multiplicity estimates were based had a low response rate. Moreover, the Bureau found that respondents, particularly in shelters, did not answer the question accurately. Due to data quality concerns, the Bureau decided not to use multiplicity estimation to adjust the data and consequently decided not to report the data separately.

Bureau officials said they did not announce the change in plans because they were still evaluating the problems with the data. It was not until June 2000 that the Bureau began recalculating the data and making a final decision on which categories to aggregate. Ultimately, the Bureau did not report any of the Service-Based Enumeration data separately in SF-1. Emergency and transitional shelter data were the only data that were to be released in SF-1 under separate reporting categories that the Bureau decided to combine with another category.

The Bureau Produced a Special Report in October 2001 on the Emergency and Transitional Shelter Population

The release of the SF-1 data in June 2001 produced public discussion in the press, among census partners, and in the Congress about the Bureau’s decision to not separately release Service-Based Enumeration data. In a briefing for staff of the House Committee on Governmental Affairs, the Associate Director of the Decennial Census announced that the Bureau planned to produce a separate report on the emergency and transitional shelter data. In October 2001, the Bureau issued a special report, entitled Emergency and Transitional Shelter Population: 2000. This report separately identified emergency and transitional shelter data for various levels of geography down to the census tract level with 100 or more people in emergency and transitional shelters. The report did not include data for the populations in targeted nonsheltered outdoor locations, soup kitchens, regularly scheduled mobile food vans, and shelters for domestic violence. The 17-page report contains an extensive discussion on the limitations of the data. For example, the Bureau noted that the data in the report should not be construed as a count of people without conventional housing. Moreover, the emergency and transitional shelter data at the census tract level are not in the hard copy, but rather in the Internet version of the report. The Bureau stated that all Census 2000 data at the tract level are available on the Internet and are not available in printed reports.

The October report contains most of the same data that were to be released under the April 1999 dissemination plan for SF-1. The Bureau asserted that the data quality concerns with the emergency and transitional shelter data (cited when it changed the plan to release these data in SF-1) required that the data be presented in a manner that allowed the Bureau to clearly outline the data’s limitations. The October 2001 report contained an extended discussion of these limitations.

The October 2001 report also identified reasons the Bureau did not (and never planned to) separately release data on people counted at targeted nonsheltered outdoor locations, soup kitchens, regularly scheduled mobile food vans, and shelters for victims of domestic violence, including the following.

People without conventional housing who were at outside locations other than the targeted nonsheltered outdoor locations identified for the census were not included in the TNSOL operation.

For the purposes of the TNSOL operation, the definition of “outdoor” excluded both mobile and transient locations used by people experiencing homelessness as well as abandoned buildings.

The option was given to the individuals found at soup kitchens and regularly scheduled mobile food vans to select “usual home elsewhere.” For example, if an individual enumerated at a soup kitchen listed a usual home elsewhere, then that person was tabulated at their usual residence and not at the service location. Therefore, the data on this population would not reflect a true count of the individuals using these services.

Prior to publicly releasing the October report, the Bureau asked two representatives from the National Coalition for the Homeless to review a draft of the portion of the report that described the limitations of the data. The National Coalition for the Homeless commented extensively on the section containing the caveats and limitations in order to strengthen the report. A member of the Board of Directors for the National Coalition for the Homeless told us that he provided this feedback both as an academician and a stakeholder. Bureau officials stated that because of its position on the Bureau’s Census Advisory Committee, the National Coalition for the Homeless was the only advocacy group that reviewed any portion of the October 2001 report prior to its publication.

Meeting Data Users’ Needs Proved Challenging

The controversy surrounding the release of the combined Service-Based Enumeration data highlights the challenges the Bureau faced in 2000 trying to meet the needs of various data users and the work the Bureau still needs to do when planning for the 2010 Census to better reconcile those needs. For example, several organizations we contacted favored the separate release of the Service-Based Enumeration data categories. Indeed, local government officials we talked to in New York City believed that the data would help with grant applications, projections about future service needs, and determining their success in getting people off the streets and into shelters. The Executive Director of the Northeast Ohio Coalition for the Homeless stated that the city of Cleveland does not do its own count of this population and, therefore, the Bureau numbers are the only ones available on this segment of the population. Los Angeles city officials wanted the Service-Based Enumeration data so they could better target their services and, like Cleveland, Los Angeles did not have its own data. Several of these entities stated that the potential misuse of data was not a valid reason for not separately releasing data.

In addition, the majority of the organizations we contacted partnered with the Bureau anticipating that they would be able to use the Service-Based Enumeration data to evaluate whether improvements were made in enumerating local populations without conventional housing in 2000 compared to 1990. The Assistant City Attorney of Los Angeles estimated that Los Angeles spent about $300,000 on the effort to improve the count of Los Angeles’s people without conventional housing. For example, as part of an extensive effort to help the Bureau develop a list of targeted nonsheltered outdoor locations, the city provided senior Bureau staff with a helicopter tour over some outdoor locations where people without conventional housing lived. The Assistant City Attorney of Los Angeles stated that she believed the city would get the targeted nonsheltered outdoor locations data that they helped collect and wanted to review. In addition, because of the Bureau’s focus on counting people at shelters, the city kept shelters open on the night of the enumeration at its own expense even though shelters in Los Angeles typically do not have many people during warm weather. Los Angeles expected to have detailed data to use to evaluate the effectiveness of its resource allocation.

However, the National Coalition for the Homeless and other advocates of the homeless opposed the separate release of any of the Service-Based Enumeration data. They were concerned that these data could be misused as a count of the homeless population and lead to flawed decision-making by policymakers.

Ultimately, the Bureau left a number of data users unsatisfied. Those who wanted the Service-Based Enumeration categories released separately did not feel the Bureau met their expectations with the data released in SF-1 or with the release of the October report. Users who opposed the separate release of the data and were pleased that SF-1 combined the Service-Based Enumeration components with other data were displeased that the October 2001 report was released.

The difficulties the Bureau experienced trying to reconcile the competing needs and interests of data users illustrates the importance of effective communication between the Bureau and its key data users and partners to ensure no expectation gaps develop. More than just a good business practice, federal internal control standards require agencies to have effective external communications with groups that can have a serious impact on programs, projects, operations, and other activities.

However, our conversations with several Bureau partners and our review of Bureau documents suggest that communications were sometimes vague and insufficient. For example, although the April 1999 memorandum that outlined the Bureau’s initial data dissemination plans was written a year before the 2000 Census, this information may not have been effectively communicated to the Bureau’s partners. Indeed, at a Capitol Hill briefing on this topic in June 2001, Bureau officials themselves acknowledged that they did not do a good job of communicating on this issue. Some of the partners we spoke to indicated that had they known earlier about the Bureau’s plans to limit the release of Service-Based Enumeration data they might have focused their resources on different census operations. Further, our review of Bureau documents indicated that the information on the “official plan” for the release of the different Service-Based Enumeration categories of data was limited and inconsistent. Some partners stated that they did not know that the Bureau never intended to report the targeted nonsheltered outdoor location data.

Although the Bureau made numerous presentations on Service-Based Enumeration that emphasized there would be no count of the homeless, the Bureau provided little detail on how components of Service-Based Enumeration would actually be presented. In the absence of clear communication from the Bureau, partners developed their own expectations of what would be released. Several of the local officials and advocates that we spoke to expected that the data would be released in the same detail as it was in 1990, because they were not told otherwise. For example, a Los Angeles government official said that the Bureau stated it would not provide a homeless count in 1990, but it still released the street count data separately. By focusing resources on counting specific categories of the population, the Bureau may have created expectations that there would be a count of that population.

Census Bureau Had Few Documented Guidelines Governing the Release of Census Data

A cause of the Bureau’s shifting position on reporting the components of Service-Based Enumeration appears to be its lack of documented, clear, transparent, and consistently applied guidelines governing the release of data from the 2000 Census. Except for some guidance aimed at protecting the confidentiality of census records, the Bureau had few written guidelines on the level of quality needed to release data to the public. Had these guidelines been in place during the decennial census, they could have informed the Bureau’s decision on whether to release the Service-Based Enumeration data, how to characterize these data, and help defend the decision after it was made. Such guidelines could also provide a basis ahead of time for expectations about the conditions under which data will or will not be released.

Although Bureau officials emphasized that the Bureau has a long tradition of high standards and procedures that yield quality data (to its credit, the Bureau’s quality assurance practices identified the problem with the multiplicity estimator), the officials acknowledged that these standards were primarily part of the agency’s institutional knowledge. The written guidance that did exist appeared to be vague and insufficient for making consistent decisions on the quality thresholds needed for releasing data to the public, and the circumstances under which it might be appropriate to suppress certain data.

According to the Bureau’s Associate Director for Methodology and Standards, a technical paper issued in 1974 and revised in 1987 contained the Bureau’s only written guidelines for discussion and presentation of errors in data. This paper noted that, “stimates for individual cells of a published table should not be suppressed solely because they are subject to large sampling errors or large nonsampling variances, provided users are given adequate caution of the lack of reliability of the data. On the other hand, data known to have very serious bias may be suppressed.”

A Newly Created Bureau Program Could Provide Guidelines for Releasing Data

In 2000, the Bureau initiated a new quality assurance program to document Bureau-wide protocols designed to ensure the quality of data collected and disseminated by the Bureau. The Bureau’s Methodology and Standards Council is charged with setting statistical and survey quality standards and guidelines for Bureau surveys and censuses. In support of this role, the council has established a quality framework in which the demographic, economic, and decennial areas can share and support common principles, standards, and guidelines. The quality framework covers eight unique areas, one of which is dissemination.

Because this Bureau program is in its initial stages, we could not evaluate it. However, Bureau officials believe that the program is a significant first step in addressing the lack of agencywide written guidelines for releasing data. The initiative appears to be consistent with Office of Management and Budget guidelines issued in February 2002 requiring federal agencies to issue their own guidance for ensuring and maximizing the quality, objectivity, utility, and integrity of information disseminated by the agency. As the Bureau develops its guidelines, it will be important that they be well documented, transparent, clearly defined, and consistently applied.

Conclusions

Although Service-Based Enumeration was designed to address the challenges the Bureau encountered during the 1990 Census in obtaining a complete count of people without conventional housing, the Bureau’s experience during the 2000 Census suggests that tallying this population group remains problematic. Moreover, the Bureau’s difficulties were compounded by its shifting position on how to report the data once they were collected. A number of government, community, and advocacy organizations helped the Bureau enumerate this population group. However, the Bureau, by first planning to release the data one way, then changing the decision, and ultimately releasing the data anyway—all for reasons that were not clearly articulated to the Bureau’s stakeholders— raised questions about the Bureau’s decision-making on data quality issues. As noted at the beginning of this report, related questions have also been raised about how the Bureau collected and reported data on Hispanic subgroups. To the extent similar incidents occur in the future, they could undermine public confidence in the accuracy and credibility of Bureau data.

Thus, as the Bureau plans for the 2010 Census, it will be important for it to refine its methods for enumerating people living in unconventional housing and reporting the resulting data, in part by properly testing and evaluating those methods. As noted earlier, the Bureau could not properly test a key statistical technique during the census dress rehearsal because the sample size was too small.

Moreover, while addressing the competing needs and desires of data users will likely remain a considerable challenge, it will be important for the Bureau to more effectively articulate its plans to avoid the expectation gaps that occurred during 2000. The Bureau’s plans for collecting data on persons without conventional housing need to specify how the Bureau plans to separately report these data.

Bureau-wide guidelines on the level of quality needed to release data to the public, on how and when to document data limitations, and on the circumstances under which it is acceptable to suppress data, could help the Bureau be more accountable and consistent in its dealings with data users and stakeholders, and help ensure that the Bureau’s decisions both are, and appear to be, totally objective.

Recommendations for Executive Action

To ensure that the 2010 Census will provide public data users with more complete, accurate, and useful information on the segment of the population without conventional housing, we recommend that the Secretary of Commerce direct the Director of the Bureau of the Census to do the following. 1. Ensure that all procedures for enumerating and estimating segments of the population without conventional housing are properly tested and evaluated under conditions as similar to the census as possible. 2. Develop agencywide guidelines for Bureau decisions on the level of quality needed to release data to the public, how to characterize any limitations in the data, and when it is acceptable to suppress the data for reasons other than protecting the confidentiality of respondents. Ensure that these guidelines are documented, transparent, clearly defined, and consistently applied. 3. Ensure that the Bureau’s plans for releasing data are clearly and consistently communicated with the public.

Agency Comments and Our Evaluation

The Secretary of Commerce forwarded written comments from the Bureau of the Census on a draft of this report (see app. I). The Bureau agreed with each of our recommendations and, as indicated in the letter, is taking steps to implement them. However, it expressed several general concerns about our findings. The Bureau’s principal concerns and our response are presented below. The Bureau also suggested minor wording changes to provide additional context and clarification. We accepted the Bureau’s suggestions and made changes to the text as appropriate.

The Bureau took exception to our findings concerning the adequacy of its data quality guidelines, noting that the Bureau’s decisions regarding the release and characterization of emergency and transitional shelter data were based on established guidelines for data quality. However, the Bureau did not cite any written guidelines to support its position. As noted in our report, Bureau officials, including the Associate Director for Methodology and Standards, told us that the Bureau had few written guidelines, standards, or procedures related to the quality of data released to the public. In this report we acknowledge the Bureau’s tradition of high standards and procedures that yield quality data. However, according to the Bureau, these standards are generally undocumented and part of the agency’s institutional knowledge. To provide a basis for consistent decision-making and clear communication within the Bureau and to the public, guidelines on the quality of data released to the public must be fully documented, transparent, clearly defined, and consistently applied.

Additionally, the Bureau said that when data do not meet an acceptable level of quality, it considers various options for modifying its dissemination plans. The Bureau’s decision to delay the release of the emergency and transitional shelter data may have been entirely appropriate. Our concern is not that the Bureau changed its plans, but that it could not provide us its guidelines for determining an acceptable level of quality or clearly indicate how it determined that the data did not meet minimal quality standards for release in SF-1.

The Bureau also commented that its decisions regarding the distribution of data from SF-1 were well publicized and that the only change in Bureau plans for the release of Service-Based Enumeration data was the decision to delay release of the emergency and transitional shelter data. This report focused on the changing plans for the release of the emergency and transitional shelter data and noted that the Bureau never intended to release any other data from the Service-Based Enumeration. However, we found that the Bureau did not effectively communicate its decisions with its partners or the public. Decisions on the release of the emergency and transitional shelter data were contained in internal decision memoranda. We found that these decisions were not always reflected in new releases of the SF-1 documentation. Although Bureau officials told us that they always intended to produce a separate report on emergency and transitional shelter data, they did not make this intention public when the SF-1 data were released. Some stakeholders did not realize that the Bureau was not releasing emergency and transitional shelter data with SF-1 until they examined the SF-1 data. As we stated in our report, these communication problems can undermine stakeholder and public confidence in the Bureau and its products.

As agreed with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days from its issue date. At that time, we will send copies of this report to the Chairman of the House Committee on Government Reform, the Chairman of the Subcommittee on Civil Service, Census and Agency Organization, the Secretary of Commerce, and the Director of the Bureau of the Census. Copies will be made available to others on request. This report will also be available at no charge on GAO’s home page at http://www.gao.gov.

Please contact me on (202) 512-6806 or by e-mail at daltonp@gao.gov if you have any questions. Other key contributors to this report were Robert Goldenkoff, Timothy Wexler, Elizabeth Powell, Chris Miller, James Whitcomb, Ty Mitchell, Robert Parker, and Michael Volpe.

Related GAO Products on the Results of the 2000 Census and Lessons Learned for a More Effective Census in 2010

2000 Census: Refinements to Full Count Review Program Could Improve Future Data Quality. GAO-02-562. Washington, D.C.: July 3, 2002. 2000 Census: Coverage Evaluation Matching Implemented As Planned, but Census Bureau Should Evaluate Lessons Learned. GAO-02-297. Washington, D.C.: March 14, 2002. 2000 Census: Best Practices and Lessons Learned for a More Cost- Effective Nonresponse Follow-Up. GAO-02-196. Washington, D.C.: February 11, 2002. 2000 Census: Coverage Evaluation Interviewing Overcame Challenges, but Further Research Needed. GAO-02-26. Washington, D.C.: December 31, 2001. 2000 Census: Analysis of Fiscal Year 2000 Budget and Internal Control Weaknesses at the U.S. Census Bureau. GAO-02-30. Washington, D.C.: December 28, 2001. 2000 Census: Significant Increase in Cost Per Housing Unit Compared to 1990 Census. GAO-02-31. Washington, D.C.: December 11, 2001. 2000 Census: Better Productivity Data Needed for Future Planning and Budgeting. GAO-02-4. Washington, D.C.: October 4, 2001. 2000 Census: Review of Partnership Program Highlights Best Practices for Future Operations. GAO-01-579. Washington, D.C.: August 20, 2001.

Decennial Censuses: Historical Data on Enumerator Productivity Are Limited. GAO-01-208R. Washington, D.C.: January 5, 2001. 2000 Census: Information on Short- and Long-Form Response Rates. GAO/GGD-00-127R. Washington, D.C.: June 7, 2000.

GAO’s Mission

The General Accounting Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full- text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select “Subscribe to GAO Mailing Lists” under “Order GAO Products” heading.

Order by Mail or Phone

To Report Fraud, Waste, and Abuse in Federal Programs

Public Affairs