2010 Census: Basic Design Has Potential, but Remaining Challenges Need Prompt Resolution
Released: 2005-02-11T00:00:00

Letter

Background

Congress, GAO, the Department of Commerce Inspector General, and even the Bureau itself have all noted that the 2000 Census was marked by poor planning, which unnecessarily added to the cost, risk, and controversy of the national head count. In January 2003, we named the 2010 Census a major performance and accountability challenge because of our growing concern over the numerous obstacles to a cost-effective enumeration as well as its escalating price tag. More recently, we reported that while the Bureau’s preparations for the 2010 Census appeared to be further along than at a similar point during the planning cycle for the 2000 Census, considerable risks and uncertainties remained. Thus, it is imperative that the Bureau adequately test the various components of its design for the 2010 Census.

A rigorous testing program provides at least four major benefits. First, testing allows the Bureau to refine procedures aimed at addressing problems encountered in past censuses. During the 2000 Census, for example, group quarters were sometimes counted more than once or counted in the wrong location; the wording of the race and ethnicity question confused some respondents, which in some cases resulted in lower quality data; and following up with nonrespondents proved to be costly and labor-intensive. A second benefit is that sound testing can assess the feasibility of new procedures and technologies, such as HHCs (see fig. 1), that have never before been used in a decennial census.

Third, a rigorous testing program helps instill a comfort level among members of Congress and other stakeholders that the Bureau (1) has chosen the optimal design given various trade-offs and constraints and (2) has identified and addressed potential risks and will be able to successfully execute its plan. Such confidence building, developed through regular updates and open lines of communication, is essential for continuing congressional support and funding.

And finally, proper testing early in the decade will help the Bureau to conduct a dress rehearsal in 2008 that fully assesses all aspects of the census design under realistic conditions. Because of various late requirement changes, certain procedures that were added after the 1998 dress rehearsal for the 2000 Census were not properly tested.

Scope and Methodology

As agreed with your offices, our objectives for this report were to (1) assess the soundness of the Bureau’s design for the 2004 census test and whether the Bureau implemented the test consistent with its plans, (2) review the quality of the Bureau’s IT security practices, and (3) identify initial lessons learned from conducting the test and their implications for the 2010 Census.

To assess the soundness of the design we reviewed pertinent documents that described the Bureau's test and evaluation plans. We systematically rated the Bureau’s approach using a checklist of design elements that, based on our review of program evaluation literature, are relevant to a sound study plan. For example, we reviewed the Bureau's approach to determine, among other things, (1) how clearly the Bureau presented research objectives, (2) whether research questions matched the research objectives, and (3) the appropriateness of the data collection strategy for reaching the intended sample population. As part of our assessment of the Bureau’s test design, we also reviewed evaluations of the prior decennial census to determine the degree to which the new operations being tested addressed problematic aspects of the 2000 Census. However, we did not assess the Bureau’s criteria in selecting its objectives for the 2004 census test.

To determine if the Bureau implemented the test consistent with its plans, we made multiple site visits to local census offices in Thomasville, Georgia; and Queens Borough, New York. During these visits, we interviewed local census office mangers and staff, observed various data collection activities, and attended weeklong enumerator training. We observed a total of 20 enumerators as they completed their daily nonresponse follow-up assignments—half of these were in southern Georgia, in the counties of Thomas, Colquitt, and Tift, and half were in Queens (see fig. 2 for maps of the test site areas). The results of these observations are not necessarily representative of the larger universe of enumerators.

To evaluate the quality of the Bureau’s IT security practices, we assessed risk management documentation associated with IT systems and major applications for the 2004 census test. We based our determination on applicable legal requirements, Bureau policy, and leading practices described in our executive guide for information security management.

We also interviewed key Bureau officials associated with computer security.

To identify lessons learned from the 2004 census test, we met with officials from the Bureau’s Decennial Management Division regarding overall test plans and with officials from its Technologies Management Office about using HHCs. Bureau officials and census workers from both test locations also provided suggestions on improving census operations.

We requested comments on a draft of this report from the Secretary of Commerce. On December 20, 2004, the Under Secretary for Economic Affairs, Department of Commerce, forwarded written comments from the Bureau (see app. I). We address these comments in the “Agency Comments and Our Evaluation” section at the end of this report.

The Census Test Was Generally Sound, but Refinements Could Produce Better Cost and Performance Data

The Bureau designed a sound census test and generally implemented it as planned. However, in looking ahead, the Bureau’s planning and investment decisions could benefit from analyzing (1) the degree to which HHCs contributed to the Bureau’s cost containment goal and (2) the results of the targeted second mailing, an operation designed to increase participation by sending a follow-up questionnaire to nonresponding households. Future tests could also be more informative if the Bureau developed quantifiable productivity and other performance requirements for the HHCs and then used the 2006 test to determine whether the devices are capable of meeting those requirements. Collectively, these refinements could provide Bureau officials with better information to guide its IT and other design decisions, as well as refine future census tests.

The Bureau Developed a Sound Test Design

The design of the 2004 census test contained many components of a sound study (see table 1). For example, the Bureau identified test objectives, designed related research questions, and described a data collection strategy appropriate for a field test. The Bureau also developed evaluation plans for each of the test’s 11 research questions, and explained how stakeholders were involved with the design, as well as how lessons learned from past studies were incorporated.

Additional Analysis Would Provide Better Data on the Impact of Key Test Components

Although the Bureau plans to evaluate various aspects of the 2004 test, it does not currently plan to assess the impact that the HHCs and targeted second mailing had on cost savings and productivity. According to the Bureau, the census test was focused more on determining the feasibility of using the HHCs and less on the devices’ ability to save money. Likewise, the Bureau said it is not assessing the impact of the targeted second mailing because the operation is not one of its four test objectives for improving (1) field data collection using the HHC, (2) the coverage of undercounted groups, (3) questions about race and ethnicity, and (4) methods for defining special places and group quarters.

These decisions might be shortsighted, however, in that the Bureau included the HHCs and targeted second mailing in the 2010 Census design, in part, to reduce staff, improve productivity, and control costs. For example, Bureau studies have shown that sending out replacement questionnaires could yield a gain in overall response of 7 to 10 percent from households that do not respond to the initial census mailing, and thus generate significant cost savings by eliminating the need for census workers to obtain those responses via personal visits. Thus, information on the degree to which the HHCs and second mailing contribute to these key goals could help inform future budget estimates, investment and design decisions, as well as help refine future census tests.

Moreover, the feasibility of a targeted second mailing is an open question, as the Bureau has never before included this operation as part of a decennial census. Although a second mailing was part of the original design for the 2000 Census, the Bureau had abandoned it because it was found to be logistically unworkable. A Bureau official said that the second mailing was included in the 2004 test only to facilitate the enumeration process, and it would be better tested in a larger scale operation such as the 2008 dress rehearsal. However, we believe that it would be more prudent to assess the second mailing earlier in the census cycle, such as, during the 2006 test so that its basic feasibility could be assessed, any refinements could be evaluated in subsequent tests, and the impact on savings could be estimated more accurately.

Future Tests Could Be Improved

While the design of the 2004 test was generally sound, refinements could strengthen the next field test in 2006. Opportunities for improvement exist in at least two areas: ensuring that (1) the HHCs can meet the demanding requirements of field data collection and (2) management of the local census offices mirrors an actual enumeration as much as possible.

With respect to the HHCs, because they replace the paper version of the nonresponse follow-up questionnaire the devices must function effectively. Further, this test was the first time the Bureau used the HHCs under census-like conditions so their functionality in an operational environment was unknown. Bureau officials have acknowledged that for the 2004 test they had no predefined indicators of success or failure other than if there was a complete breakdown the test would be halted. This is a very low standard. Now that the Bureau has demonstrated the basic functionality of the computers, it should next focus on determining the specific performance requirements for the HHCs and assess whether the devices are capable of meeting them. For example, the Bureau needs productivity benchmarks for the number of interviews per hour and per day that is expected per census worker. Durability measures, such as how many devices were repaired or replaced, should be considered as well. Assessing whether the HHCs can meet the requirements of nonresponse follow-up will help inform future design and investment decisions for whether or not to include the devices in the 2010 design.

Ensuring that key positions in the local census offices are filled from the same labor pool as they would be in an actual decennial census could also enhance future census tests. Such was not the case during the 2004 test when, according to the Bureau, because of difficulties finding qualified applicants, it used an experienced career census employee to manage the overall day-to-day operations of the local census office at the Queens test site. Another career employee occupied the office’s regional technician slot, whose responsibilities included providing technical and administrative guidance to the local census office manager. In the actual census, the Bureau would fill these and other positions with temporary employees recruited from local labor markets. However, because the Bureau staffed these positions with individuals already familiar with census operations and who had ties to personnel at the Bureau’s headquarters, the Queens test may not have been realistic and the test results could be somewhat skewed.

The Bureau Needs to Implement Better IT Security Practices

The Bureau operated a number of IT systems in order to transmit, manage, and process data for the test. The equipment was located at various places including the Bureau’s headquarters in Suitland, Maryland; its National Processing Center in Jeffersonville, Indiana; a computer facility in Bowie, Maryland; as well as the New York and Georgia test sites.

Under Title 13 of the U.S. Code, the Bureau must protect from disclosure the data it collects about individuals and establishments. Thus, the Bureau’s IT network must support both the test’s telecommunications and data processing requirements, as well as safeguard the confidentiality and integrity of respondents’ information.

The Federal Information Security Management Act of 2002 (FISMA) requires each agency to develop, document, and implement an agency-wide information security program for the IT systems that supports its operations. Although the Bureau took a number of steps to implement IT security over the systems used for the test, based on available information, the Bureau did not meet several of FISMA’s key requirements. As a result, the Bureau could not ensure that the systems supporting the test were properly protected against intrusion or unauthorized disclosure of sensitive information. For example: IT inventory was not complete. FISMA requires an inventory of major information systems and interfaces. The Bureau did not have a complete inventory that showed all applications and general support IT systems associated with the test. Without such information, the Bureau could not ensure that security was effectively implemented for all of its systems used in the test, including proper risk assessments, adequate security plans, and effectively designed security controls.

There was not sufficient evidence that the Bureau assessed all of the devices used in the test for vulnerabilities, or that it corrected previously identified problems. FISMA requires that agencies test and evaluate the effectiveness of information security policies, procedures, and practices for each system at least annually and that agencies have a process for remediating any identified security weaknesses. Since the Bureau could not provide us with a complete inventory of all network components used in the test, we could not determine if the Bureau’s tests and evaluations were complete. Moreover, there was not always evidence about whether the Bureau had corrected past problems or documented reasons for not correcting them. As a result, the Bureau did not have adequate assurance that the security of systems used in the 2004 census test was adequately tested and evaluated or that identified weaknesses were corrected on a timely basis.

Assessments were not consistent. FISMA requires agencies to assess the risks that could result from the unauthorized access, use, disclosure, disruption, modification, or destruction of information or information systems. Although the Bureau performed risk assessments for some of the IT components used in the 2004 census test, the documentation was not consistent. For example, documentation of information sensitivity risks (high, medium, and low) for confidentiality, integrity, and availability of information were not consistent and did not always follow Bureau policy. In addition, documents showed different numbers of file servers, firewalls, and even different names of devices. Without complete and consistent risk assessment documentation, the Bureau had limited assurance that it properly understood the security risks associated with the test.

The Bureau did not always follow its own risk policies. FISMA requires the implementation of policies and procedures to prevent and/or mitigate security risks. Although Bureau policies allowed for the waiver of security policies, if appropriate, we noted that such policies were not always followed. For example, a waiver for the test of certain password policies was not properly documented and other system documents were not properly updated to reflect the waiver. As a result, the risk assessment for the 2004 census test did not properly identify the related risks and did not identify any compensating controls to reduce the risk to an acceptable level.

As the Bureau plans future tests and the census itself, it will be important for it to strengthen its IT security risk management practices, ensuring they fully adhere to FISMA requirements and its own IT security policies.

Test Reveals Technical, Training, and Other Challenges in Need of Prompt Resolution

The 2004 test suggests that while certain census initiatives have potential, formidable challenges remain. For example, the HHCs show promise in that enumerators were successful in using them to collect data from nonrespondents and remove late mail returns. Still, they were not street ready as they experienced transmission and memory overload problems. Likewise, automated maps were difficult to use, certain questionnaire items confused respondents, and enumerators did not always follow interview protocols. These problems shed light on issues in need of the Bureau’s attention as it develops solutions and incorporates refinements for additional testing in the years ahead.

HHCs Were Effective for Conducting Interviews and Removing Late Mail Returns

The Bureau purchased 1,212 HHCs for the test at a total cost of about $1.5 million. The devices were sent directly to the two test sites packaged in kits that included a battery, AC adaptor, and modem card for transmitting data via the telephone. The HHCs were also equipped with a Global Positioning System (GPS), a satellite-based navigational system to help enumerators locate street addresses. The Bureau anticipates the HHCs will allow it to eliminate the millions of paper questionnaires and maps that enumerators need when following up with nonrespondents, thereby improving their efficiency and reducing overall costs.

Because the Bureau had never used HHCs in the decennial census, an important goal of the test was to see whether enumerators could use them for interviewing nonrespondents (see fig. 3). Most workers we observed had little trouble using the device to complete the interviews. In fact, most said they were pleased with the HHC’s overall functionality, durability, screen clarity, and the ability to toggle between the questionnaire and taking GPS coordinates.

Another important function of the HHC was removing late mail returns from each enumerator’s assignment area(s). Between the Georgia and Queens test sites, over 7,000 late mail returns were removed, reducing the total nonresponse follow-up workload by nearly 6 percent.

The ability to remove late mail returns from the Bureau’s nonresponse follow-up workload could help save money in that it could eliminate the need for enumerators to make expensive follow-up visits to households that return their questionnaires after the mail-back deadline. Had the Bureau possessed this capability during the 2000 Census, it could have eliminated the need to visit nearly 773,000 late-responding households and saved an estimated $22 million (based on our estimate that a 1-percentage point increase in workload could add at least $34 million in direct salary, benefits, and travel costs to the price tag of nonresponse follow-up). Because of the Bureau’s experience in 2000, in our 2002 report on best practices for more cost-effective nonresponse follow-up, we recommended, and the Bureau agreed, that it should develop options that could purge late mail returns from its nonresponse follow-up workload.

Technical and Training Difficulties Caused HHC Transmission Problems

Each day, enumerators were to transmit completed nonresponse follow-up cases to headquarters and receive assignments, software uploads, or both via a telephone modem (see fig. 4 for a flowchart describing the file transmission process). However, the majority of workers we interviewed had problems doing so, in large part because of technical reasons or because the Bureau’s training did not adequately prepare them for the complexity of the transmission procedure, which was a multistep process involving the connection of a battery pack, cables, and other components. As reliable transmissions are crucial to the success of nonresponse follow- up, it will be important for the Bureau to resolve these issues so that the HHCs can be reevaluated in 2006.

Difficulties began during training when the first transmission was supposed to occur and continued through the remainder of the test. During that first transmission, the Bureau needed to upload a number of software upgrades along with each census worker’s first assignment. Many of these transmissions failed because of the volume of data involved. Thus, without cases, the trainees could not complete an important section of on-the-job training. The Bureau acknowledged that these initial problems could have been avoided if the final version of software had been installed on the devices prior to their distribution at training.

Transmission problems persisted throughout nonresponse follow-up. According to the Bureau, during the first 2 weeks of this operation, successful data transmission occurred 80 percent of the time once a connection was made. However, a number of enumerators never even established a connection because of bad phone lines, incorrect passwords, and improper setup of their modems. Other transmission problems were due to the local telecommunication infrastructure at both test sites. For example, in Georgia, older phone lines could not always handle transmissions, while in Queens, apartment intercoms that used phone lines sometimes interrupted connections.

Further, while the transmission rate ultimately increased to 95 percent-– roughly the maximum allowed by the technology—that level is still short of the performance level needed for 2010. During the 2000 Census, a 95 percent success rate would have resulted in the failure to transmit around 30,000 completed questionnaires each day.

During the test, the Bureau also had to contend with census workers who were “living off the grid”; that is, they only used cellular phones and lacked landlines to transmit and receive data from their homes. While individuals could make alternative arrangements, such as using a neighbor’s telephone, an increasing number of people nationwide in the coming years might give up their landline service to rely on cellular phones, which could be problematic for the Bureau. Bureau officials have noted that all these transmission problems need to be addressed before 2010.

HHCs experienced memory overloads if too many assignment areas were loaded onto them. An assignment area typically contains 40 housing units or cases that are assigned to an enumerator for nonresponse follow-up. The design was to have an entire assignment area transmitted to the HHC even when as few as one case needed follow-up. However, some enumerators’ HHCs became overloaded with too much data, as cases had to be reassigned due to staff turnover, a larger-than-expected number of refusals, and reassignments resulting from language problems. As such, when HHCs became overloaded they would crash and enumerators had to reconfigure them at the local census office, which made them less productive. To the Bureau’s credit, during the test, it was able to work out a solution to avoid overloads by assigning individual cases instead of the entire assignment area to a census worker’s HHC.

Another problem that surfaced during the test was that the HHC’s mapping feature was difficult to use. To contain costs and increase efficiency, the Bureau expects to replace paper maps with the electronic maps loaded on the HHCs for 2010. However, during the test, enumerators reported that they did not always use the mapping function because it ran slowly and did not provide sufficient information. Instead, they relied on local maps or city directories, and one worker explained that she found it easier to use an Internet mapping service on her home computer to prepare for her route.

Without the Bureau’s maps, enumerators might not properly determine whether a housing unit was located in the Bureau’s geographic database. This verification is important for ensuring that housing units and the people who reside in them are in the correct census block, as local and state jurisdictions use census population figures for congressional redistricting and allocating federal funds.

Enumerators were also unable to use the HHCs’ “go back” function to edit questionnaires beyond a certain point in the interview. In some cases, this led to the collection of incorrect data. For example, we observed one worker complete half an interview, and then discover that the respondent was providing information on a different residence. After the census worker entered the number of residents and their names, the “go back” function was no longer available and as a result that data could not be deleted or edited. Instead, the worker added information in the “notes section” to explain that the interview had taken place at the wrong household. However, Bureau officials told us that they had not planned to review or evaluate these notes and were not aware that such address mix- ups had been documented in the notes section.

To the extent address mix-ups and other inconsistencies occur and are not considered during data processing, accuracy could be compromised. In earlier censuses when the Bureau used paper questionnaires, if workers made mistakes, they could simply erase them or record the information on new forms. As mistakes are inevitable, it will be important for the Bureau to ensure that the HHCs allow enumerators to edit information, while still maintaining the integrity of the data.

Bureau Needs to Review Format of Coverage Improvement and Race/Ethnicity Questions

We found that questions designed to improve coverage and better determine race and ethnicity were awkward for enumerators to ask and confusing for respondents to answer. Consequently, enumerators sometimes did not read the questions exactly as worded, which could adversely affect the reliability of the data collected for these items, as well as the Bureau’s ability to evaluate the impact of the revised questions. Our observations also highlight the importance of ensuring that workers are trained to follow interview protocols; this issue will be discussed later in this report.

Coverage Improvement

While the Bureau attempts to count everyone during a census, inevitably some people are missed and others are counted more than once. To help ensure that the Bureau properly counts people where they live, the Bureau revised and assessed its residency rules for the 2004 census test. For example, under the residence rules, college students should be counted at their campus addresses if they live and stay there most of the time. The Bureau also added two new coverage questions aimed at identifying household residents who might have been missed or counted in error (see fig. 5 for coverage questions).

Enumerators were to show respondents flashcards with the residence rules to obtain the number of people living or staying in the housing unit and to read the two coverage questions. However, during our field visits we noted that they did not consistently use the flashcards, preferring to summarize them instead. Likewise, enumerators did not always ask the new coverage questions as written, sometimes abbreviating or skipping them altogether. A frequent comment from the workers we spoke with was that the two new coverage questions were awkward because the questions seemed redundant. Indeed, one census worker said that he asked the overcount and undercount questions more times than not, but if people were in a hurry, he did not ask the questions. During one of these hurried interviews, we observed that the census worker did not ask the questions and simply marked “no” for the response.

Race and Ethnicity Questions

Collecting reliable race and ethnicity data is an extremely difficult task. Both characteristics are subjective, which makes accurate measurement problematic. In 2003, the Bureau tested seven different options for formatting the race and ethnic questions, and selected what it thought was the optimal approach to field test in 2004. The Bureau planned to examine respondent reaction to the new race and Hispanic origin questions by comparing responses collected using the paper questionnaire to responses recorded on the HHCs during nonresponse follow-up.

One change the Bureau planned to analyze was the removal of the “some other race” write-in option from the questionnaire. In 2000, the Bureau found that when given this option, respondents would check off “some other race,” but did not always write in what their race was. Thus, in the 2004 test, the Bureau wanted to assess respondents’ reaction to the removal of the “some other race” write-in option. Specifically, the Bureau wanted to see whether respondents would skip the item or select from one of the other options given.

However, we found that the Bureau formatted the race question on the paper questionnaire differently from the question on the HHC. As shown in figure 6, on the paper version, there is not a category for another race other than those categories listed, thus forcing respondents to select a category or skip the question entirely.

This contrasts with the HHCs where, if respondents do not fit into one of the five race categories, the questionnaire format allows them to provide an “other” response and enumerators can record their answers. In fact, the HHC requires enumerators to record a response to the race question and will not allow the interview to continue until a response is entered. As a result, the data recorded by the two questionnaire formats are not comparable as they could produce different data depending on the data collection mode.

According to the Bureau, it formatted the paper version of the race question differently from the HHC version because it considered the “other” response option on the HHC a respondent comment and not a write-in response. Nevertheless, if the Bureau’s purpose is to measure respondent reaction to eliminating the write-in option, it is uncertain what conclusions the Bureau will be able to draw given that this option, even though in the form of a comment, is still available to the respondent during the nonresponse follow-up interview.

As was the case with the coverage measurement question, enumerators at both test locations did not always follow proper interview procedures because they felt the questions were awkward to ask and confused respondents. For example, some workers did not use the flashcards designed to guide respondents in selecting categories for their race and ethnicity and to ensure data consistency. One census worker said that rather than use the flashcards or ask the questions, he might “eyeball” the race and ethnicity. Another worker said that most people laughed at the Spanish, Hispanic, or Latino origin question and she had complaints about the wording of this question. A third census worker noted that he was “loose with the questions” because he could pose them better. Like lapses to the coverage improvement procedures for the 2004 census test, deviating from the interview procedures for the new race and ethnicity questions may affect the reliability of the data and the validity of the Bureau’s conclusions concerning respondent reaction to these questions.

Since the 2004 census test, the 2005 Consolidated Appropriations Act required that the Bureau include “some other race” as a category when collecting census data on race identification. Consequently, the Bureau said it will include this category on all future census tests and the 2010 Census itself. Thus, while research into eliminating the “some other race” category is now moot, it will still be important for the Bureau to have similar formats for the HHCs and paper questionnaires so that similar data can be captured across modes. Likewise, it will be important for the wording of those questions to be clear and for enumerators to follow proper procedures during interviews.

New Procedures Should Help Reduce Duplicate Enumerations of Group Quarter Residents, but Other Challenges Remain

As noted previously, under its residence rules, the Bureau enumerates people where they live and stay most of the time. To facilitate the count, the Bureau divides residential dwellings into two types: housing units, such as single-family homes and apartments, and group quarters, which include dormitories, prisons, and nursing homes.

The Bureau tested new group quarters procedures in 2004 that were designed to address the difficulties the Bureau had in trying to identify and count this population group during the 2000 Census. For example, communities reported instances where prison inmates were counted in the wrong county and residents of college dormitories were counted twice.

One refinement the Bureau made was integrating its housing unit and group quarter address lists in an effort to avoid counting them once as group quarters and again as housing units, a common source of error during the 2000 Census. Census workers were then sent out to verify whether the dwellings were in fact group quarters and, if so, to classify the type of group quarter using a revised “other living quarters facility” questionnaire.

A single address list could, in concept, help reduce the duplicate counting that previously occurred when the lists were separate. Likewise, we observed that census workers had no problems using the revised facility questionnaire and accompanying flashcard that allowed the respondent to select the appropriate type of living facility. This new procedure addresses some of the definitional problems by shifting the responsibility for defining the group quarter type from the Bureau to the respondent, who is in a better position to know about the dwelling.

Another change tested in 2004 was the classification of group homes, which in 2000 was a part of the group quarter inventory. Group homes are sometimes difficult for census workers to spot because they often look the same as conventional housing units (see fig. 7). As a result, they were sometimes counted twice during the 2000 Census—once as a group quarter, and once as a housing unit. For the 2004 test, the Bureau decided to treat group homes as housing units and include them in the housing unit list.

Early indications from the Bureau suggest that including group homes as housing units, whereby they receive a short-form questionnaire in the mail, may not work. According to the Bureau, the format of the short form is not well suited to group home residents. For example, the questionnaire asks for the “name of one of the people living or staying here who owns or rents this place.” Since the state or an agency typically owns group homes, these instructions do not apply. The Bureau stated that it plans to reassess how it will identify and count people living in group homes.

We identified other problems with the Bureau’s group quarters validation operation during the 2004 census test. For example, we were told that census workers were provided maps of the areas they were assigned but needed maps for adjoining areas so that they could more accurately locate the physical location of the group quarters. In Georgia, where workers used address data from the 2000 Census, the crew leader explained that approximately one-third of all the addresses provided were incorrectly spotted on maps and had to be redone. They also lacked up-to-date instructions—for example, they did not know that they were to correct addresses rather than just delete them if the addresses were wrong. Further, census workers said that scenarios in the manual and classroom training were based on perfect situations; thus, they did not provide adequate training for atypical settings or when problems arose.

The Bureau Should Rethink Its Approach to Training Enumerators

The success of the census is directly linked to the Bureau’s ability to train enumerators to do their jobs effectively. This is a tremendous task given the hundreds of thousands of enumerators the Bureau needs to hire and train in just a few weeks. Further, enumerators are temporary employees, often with little or no prior census experience, and are expected, after just a few days of training, to do their jobs with minimal supervision, under sometimes difficult and dangerous conditions. Moreover, the individuals who train enumerators—crew leaders—are often recent hires themselves, with little, if any, experience as instructors. Overall, few, if any, organizations face the training challenges that confront the Bureau with each decennial population count.

To train the 1,100 enumerators who conducted nonresponse follow-up for the 2004 test, the Bureau employed essentially the same approach it has used since the 1970 Census: crew leaders read material word-for-word from a training manual to a class of 15 to 20 students. The notable exception was that in transitioning from a paper questionnaire to the HHCs, the Bureau lengthened the training time from 3 days to 5 days. However, given the demographic and technological changes that have taken place since 1970, the Bureau might want to explore alternatives to this rigid approach.

As noted earlier, during nonresponse follow-up, enumerators experienced a variety of problems that could be mitigated through improved training. The problems included difficulties setting up equipment to transmit and download data; failure to read the coverage and race/ethnicity questions exactly as worded; and not properly using the flashcards, which were designed to help respondents answer specific questions.

Most of the shortcomings related to training that we observed during the test were not new. In fact, the Bureau had identified these and a number of other training weaknesses in its evaluation of the 2000 Census, but it is clear they have not been fully resolved. Thus, as the Bureau plans for the 2010 Census, it will be important for it to resolve long-standing training problems as well as address new training issues, such as how best to teach enumerators to use the HHCs and their associated automated processes. Our observations of the test point to specific options the Bureau might want to explore. They include (1) placing greater emphasis on the importance of following prescribed interview procedures and reading questions exactly as worded; (2) supplementing verbatim, uniform training with modules geared toward addressing the particular enumeration challenges that census workers are likely to encounter at specific locales; and (3) training on how to deal with atypical situations or respondent reluctance.

To help evaluate its future training needs, the Bureau hired a contractor to review the training for the 2004 test and recommend actions for improving it. From GAO’s work on assessing agencies’ training and development efforts, we have developed a framework that can also help in this regard. Though too detailed to discuss at length in this report, highlights of the framework, and how they could be applied to census training, include: 1. performing proper front-end analysis to help ensure that the Bureau’s enumerator training is aligned with the skill and competencies needed to meet its field data collection requirements and work processes and that the Bureau leverages best practices and lessons learned from training enumerators and from past experience; 2. identifying specific training initiatives that in conjunction with other strategies, improve enumerators’ performance and help the Bureau meet its goal of collecting high-quality data from nonrespondents; 3. ensuring effective and efficient delivery of training that reinforces new and needed competencies, skills, and behaviors without being wedded to past, and perhaps outmoded, methods; and 4. evaluating the training to ensure it is addressing known skill and competency weaknesses through such measures as assessing participant reactions and changes in enumerators’ skill levels and behaviors.

Readiness Will Be Critical for Future Tests

Several key features of the 2004 test were not test ready; that is, they were not fully functional or mature when they were employed at the test sites. This is a serious shortcoming because it hampered the Bureau from fully evaluating and refining the various census-taking procedures that will be used in subsequent tests and the actual census in 2010. Further, to the extent these features were integrated with other operations, it impeded the Bureau from fully assessing those associated activities as well.

Our work, and that of the Department of Commerce Inspector General, identified the following areas where the Bureau needed to be more prepared going into the test: The HHCs crashed, in part, because earlier testing did not identify software defects that caused the download of more data to the HHCs than their memory cards could hold.

Transmission failures occurred during enumerator training, in part, because the HHCs were shipped without the latest version of needed software. Although the Bureau ultimately provided the latest software after several weeks, the upgraded version was unavailable for training field operations supervisors and crew leaders and for the initial enumerator training.

According to the Department of Commerce Inspector General, the Bureau finalized the requirements for the new group quarter definitions too late for inclusion in group quarters training manuals. Consequently, the training lacked certain key instructions, such as how to categorize group homes.

The Bureau experienced other glitches during the test that with better preliminary testing or on-site dry runs, might have been detected and possibly addressed before the test started. These included the slow start- up of the HHC’s mapping function, and the tendency for apartment house intercoms to interrupt transmissions.

An important objective of any type of test is to identify what is working and where improvements are needed. Thus, it should not be surprising, and, in fact, should be expected and commended, that shortcomings were found with some of the various activities and systems assessed during the 2004 test. We believe that the deficiency is not the existence of problems; rather it is the fact that several components were incomplete or still under development going into the test, which made it difficult for the Bureau to gauge their full potential. The Bureau had a similar experience in the dress rehearsal for the 2000 Census, when, because a number of new features were not test ready, the Bureau said it could not fully test them with any degree of assurance as to how they would affect the head count.

Because of the tight time frames and deadlines of the census, the Bureau needs to make the most of its limited testing opportunities. Thus, as the Bureau plans for the next field test in 2006 and the 2008 dress rehearsal, it will be important for the Bureau to ensure the various census operations are fully functional at the time of the test so they can be properly evaluated.

Conclusions

The Bureau is well aware that a successful enumeration hinges on early research, development, testing, and evaluation of all aspects of the census design. This is particularly true for the 2010 Census for which, under its current plan, the Bureau will be relying on HHCs and other methods and technologies that (1) have never been used in earlier censuses and (2) are mission critical. Consequently, the 2004 test was an important milestone in the 2010 life cycle because it demonstrated the fundamental feasibility of the Bureau’s basic design and allows the Bureau to advance to the next and more mature phase of planning and development.

Nevertheless, while the test revealed no fatal flaws in the Bureau’s approach, the results highlighted serious technical, training, methodological, and procedural difficulties that the Bureau will need to resolve. Since one of the purposes of testing is to determine the operational feasibility of the census design, it is not surprising that problems surfaced. However, looking toward the future, it will be critical for the Bureau to diagnose the source of these challenges, devise cost- effective solutions, and integrate refinements and fixes in time to be assessed during the next field test scheduled for 2006. It will also be important for Congress to monitor the Bureau’s progress as it works to resolve these issues.

Recommendations for Executive Action

To facilitate effective census planning and development, and to help the Bureau achieve its key goals for the census—reduce risks, improve accuracy, and contain costs, we recommend that the Secretary of Commerce direct the Bureau to take the following eight actions: Analyze the impact that HHCs and the targeted second mailing had on cost savings and other Bureau objectives.

Ensure the Bureau’s IT security practices are in full compliance with applicable requirements, such as the FISMA, as well as its own internal policies.

Enhance the reliability and functionality of HHCs by, among other actions, (1) improving the dependability of transmissions, (2) exploring the ability to speed up the mapping feature, (3) eliminating the causes of crashes, and (4) making it easier for enumerators to edit questionnaires.

Define specific, measurable performance requirements for the HHCs and other census-taking activities that address such important measures as productivity, cost savings, reliability, durability, and test their ability to meet those requirements in 2006.

Review and test the wording and formatting of the coverage and race/ethnicity questions to make them less confusing to respondents and thus help ensure the collection of better quality data, and ensure they are formatted the same way on both the HHC and paper versions of the census form.

Develop a more strategic approach to training by ensuring the curriculum and instructional techniques (1) are aligned with the skills and competencies needed to meet the Bureau’s data collection requirements and methodology and (2) address challenges identified in the 2004 test and previous censuses.

Revisit group quarter procedures to ensure they allow the Bureau to best locate and count this population group.

Ensure that all systems and other census-taking functions are as mature as possible and test ready prior to their deployment for the 2006 test, in part by conducting small-scale, interim tests under the various conditions and environments the Bureau is likely to encounter during the test and actual enumeration.

Further, to ensure the transparency of the census-planning process and facilitate Congressional monitoring, we also recommend that the Secretary of Commerce direct the Bureau to regularly update Congress on the progress it is making in addressing these and any other challenges, as well as the extent to which the Bureau is on track for meeting the overall goals of the 2010 Census.

Agency Comments and Our Evaluation

The Under Secretary for Economic Affairs at the Department of Commerce forwarded us written comments from the Census Bureau on a draft of this report on December 20, 2004, which are reprinted in appendix I. The Bureau noted that the 2004 test was its first opportunity to assess a number of the new methods and technologies under development for 2010, and emphasized the importance of a sustained, multiyear planning, testing, and development program to its census modernization effort.

The Bureau generally agreed with seven of our nine recommendations, and described the steps it was taking to address our concerns. The Bureau also provided additional context and clarifying language and we have added this information to the report where appropriate.

Specifically, the Bureau generally agreed with our recommendations relating to improving IT security practices, the reliability of the HHCs, training, testing, and enumeration procedures—and reported it was already taking a number of steps to address our concerns. We commend the Bureau for recognizing the risks and challenges that lie ahead and taking action to address them. We will continue to monitor the Bureau’s progress in resolving these issues and update Congress on a regular basis.

At the same time, the Bureau took exception to our recommendations to (1) analyze the impact that HHCs and targeted second mailings had on cost savings and other Bureau objectives, and (2) define specific, measurable performance requirements for the HHCs and other census-taking activities and test their ability to meet those requirements in 2006. With respect to the first recommendation, the Bureau noted that it did not establish cost- savings and other impacts as test objectives, in part, because the Bureau believes that the national sample mail test that it conducted in 2003 provided a better method for determining the boost in response rates that could accrue from a second mailing. The Bureau maintains that analyzing the impact of the second mailing would provide it with no more information beyond what it has already established from the 2003 test and would be of little value.

We believe this recommendation still applies because it will be important for the Bureau to assess the impact of the targeted second mailing on other Bureau objectives. As we noted in the report, the Bureau included the HHCs and targeted second mailing in the 2010 Census design, in part, to reduce staff, improve productivity, and control costs. Further, as we also note in the report, the feasibility of a targeted second mailing is an open question. Thus, information on the degree to which the HHCs and second mailing contribute to these key goals could help inform future budget estimates, investment and design decisions, as well as help refine future census tests. In short, the purpose of the analysis we recommend would not be to see whether these features of the 2010 Census will produce cost- savings, but the extent of those savings and the impact on other Bureau objectives.

With respect to the second recommendation, the Bureau noted that it had “baseline assumptions” about productivity, cost-savings, and other measures for the 2004 Census test and that a key objective of the test was to gather information to help refine these assumptions. According to the Bureau, this will also be a key objective of the 2006 Census Test, although its performance goal will not be whether it meets specific measures. Instead, the Bureau intends to focus on successfully collecting information to further refine those assumptions. As a result, the Bureau believes the 2006 test will not be a failure if HHC productivity is not achieved, but that it will be a failure if productivity data are not collected.

The Bureau’s position is inconsistent with our recommendation which we believe still applies. As noted in the report, we call on the Bureau to define measurable performance requirements for the HHCs as well as take the next step and assess whether the HHCs can meet those requirements as part of the 2006 test. This information is essential because it will help the Bureau gauge whether HHCs can meet its field data collection needs in 2010. Should the HHCs fail to meet these pre-specified performance requirements during the 2006 test, the Bureau would need to rethink how it employs these devices in 2010.

As agreed with your offices, unless you release its contents earlier, we plan no further distribution of this report until 30 days from its date. At that time, we will send copies of this report to the Secretary of Commerce and the Director of the U.S. Census Bureau. Copies will be made available to others on request. This report will also be available at no charge on GAO’s home page at http://gao.gov. Please contact me at (202) 512-6806 or daltonp@gao.gov or Robert Goldenkoff, Assistant Director, at (202) 512- 2757 or goldenkoffr@gao.gov if you have any questions. Key contributors to this report were Tom Beall, David Bobruff, Betty Clark, Robert Dacey, Richard Donaldson, Elena Lipson, Ronald La Due Lake, Robert Parker, Lisa Pearson, and William Wadsworth.

Comments from the Department of Commerce

GAO’s Mission

The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to www.gao.gov and select “Subscribe to Updates.”

Order by Mail or Phone

To Report Fraud, Waste, and Abuse in Federal Programs

Congressional Relations

Public Affairs