2010 Census: Basic Design Has Potential, but Remaining Challenges Need Prompt Resolution
Published: 2005-01-12T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: A rigorous testing and evaluation program is a critical component of the census planning process because it helps the U.S. Census Bureau (Bureau) assess activities that show promise for a more cost-effective head count. The Bureau conducted a field test in 2004, and we were asked to (1) assess the soundness of the test design and the extent to which the Bureau implemented it consistent with its plans, (2) review the quality of the Bureau's information technology (IT) security practices, and (3) identify initial lessons learned from conducting the test and their implications for future tests and the 2010 Census.
section_title: What GAO Found
paragraphs: The Bureau's design for the 2004 census test addressed important components of a sound study, and the Bureau generally implemented the test as planned. For example, the Bureau clearly identified its research objectives, developed research questions that supported those objectives, and developed evaluation plans for each of the test's 11 research questions. The initial results of the test suggest that while certain new procedures show promise for improving the cost-effectiveness of the census, the Bureau will have to first address a number of problems that could jeopardize a successful head count. For example, enumerators had little trouble using hand held computers (HHC) to collect household data and remove late mail returns. The computers could reduce the Bureau's reliance on paper questionnaires and maps and thus save money. The test results also suggest that certain refinements the Bureau made to its procedures for counting dormitories, nursing homes, and other "group quarters" could help prevent the miscounting of this population group. Other aspects of the test did not go as smoothly. For example, security practices for the Bureau's IT systems had weaknesses; the HHCs had problems transmitting data; questionnaire items designed to improve coverage and better capture race/ethnicity confused respondents; enumerators sometimes deviated from prescribed enumeration procedures; and certain features of the test were not fully operational at the time of the test, which hampered the Bureau from fully gauging their performance. With few testing opportunities remaining, it will be important for (1) the Bureau to find the source of these problems, devise cost-effective solutions, and integrate refinements before the next field test scheduled for 2006, and (2) Congress to monitor the Bureau's progress in resolving these issues.

section_title: Letter
paragraphs: The consequences of a poorly planned census are high given the billions of  dollars spent on the enumeration and the importance of collecting quality  data.  Therefore, a rigorous testing and evaluation program is a critical  component of the census planning process because it helps the U.S. Census  Bureau (Bureau) assess activities and related information technology (IT)  systems that show promise for a more cost-effective head count.  In  preparing for the 2010 Census, the Bureau conducted a field test in 2004  and plans additional tests for 2005 and 2006.  It is important that these early  assessments lead to a design that is sufficiently mature so that the dress  rehearsal for the 2010 Census, now planned for 2008, will demonstrate the  feasibility of the various operations and technologies planned for the  decennial under conditions that are as close as possible to the actual  census.
The Bureau designed the 2004 census test to examine the feasibility of  using (1) handheld computers (HHC) for field data collection; (2) new  methods for improving coverage; (3) redesigned race and ethnicity  (Hispanic origin) questions; and (4) improved methods for defining and  identifying nursing homes, prisons, college dormitories, and similar  facilities known collectively as “group quarters.”  The Bureau established  these test objectives as part of a broader effort to modernize and re- engineer the 2010 Census.  Major goals of this initiative are to improve the  accuracy, reduce the risk, and contain the cost of the 2010 Census,  estimates of which now exceed $11 billion.  A rigorous planning and testing  program is critical to this effort.
As agreed with your offices, our objectives for this review were to assess  the soundness of the design of the 2004 census test and the extent to which  the Bureau implemented the test consistent with its plans.  We also agreed  to review the quality of its IT security practices, and identify initial lessons  learned from the test and the implications they have for the Bureau’s future  plans.
To address these objectives, we reviewed applicable planning, IT, and other  documents, and interviewed knowledgeable Bureau officials responsible  for key operations and computer security.  We also made several visits to  the two test sites—an urban location in the northwestern portion of  Queens Borough, New York, and a rural location in south central Georgia.   We conducted our work from November 2003 through November 2004 in  accordance with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: Congress, GAO, the Department of Commerce Inspector General, and even  the Bureau itself have all noted that the 2000 Census was marked by poor  planning, which unnecessarily added to the cost, risk, and controversy of  the national head count.  In January 2003, we named the 2010 Census a  major performance and accountability challenge because of our growing  concern over the numerous obstacles to a cost-effective enumeration as  well as its escalating price tag.  More recently, we reported that while the  Bureau’s preparations for the 2010 Census appeared to be further along  than at a similar point during the planning cycle for the 2000 Census,  considerable risks and uncertainties remained.  Thus, it is imperative that  the Bureau adequately test the various components of its design for the  2010 Census.
A rigorous testing program provides at least four major benefits.  First,  testing allows the Bureau to refine procedures aimed at addressing  problems encountered in past censuses.  During the 2000 Census, for  example, group quarters were sometimes counted more than once or  counted in the wrong location; the wording of the race and ethnicity  question confused some respondents, which in some cases resulted in  lower quality data; and following up with nonrespondents proved to be  costly and labor-intensive.  A second benefit is that sound testing can  assess the feasibility of new procedures and technologies, such as HHCs  (see fig. 1), that have never before been used in a decennial census.
Third, a rigorous testing program helps instill a comfort level among  members of Congress and other stakeholders that the Bureau (1) has  chosen the optimal design given various trade-offs and constraints and   (2) has identified and addressed potential risks and will be able to  successfully execute its plan.  Such confidence building, developed through  regular updates and open lines of communication, is essential for  continuing congressional support and funding.
And finally, proper testing early in the decade will help the Bureau to  conduct a dress rehearsal in 2008 that fully assesses all aspects of the  census design under realistic conditions.  Because of various late  requirement changes, certain procedures that were added after the 1998  dress rehearsal for the 2000 Census were not properly tested.
section_title: Scope and Methodology
paragraphs: As agreed with your offices, our objectives for this report were to   (1) assess the soundness of the Bureau’s design for the 2004 census test and  whether the Bureau implemented the test consistent with its plans,   (2) review the quality of the Bureau’s IT security practices, and (3) identify  initial lessons learned from conducting the test and their implications for  the 2010 Census.
To assess the soundness of the design we reviewed pertinent documents  that described the Bureau's test and evaluation plans.  We systematically  rated the Bureau’s approach using a checklist of design elements that,  based on our review of program evaluation literature, are relevant to a  sound study plan.  For example, we reviewed the Bureau's approach to  determine, among other things, (1) how clearly the Bureau presented  research objectives, (2) whether research questions matched the research  objectives, and (3) the appropriateness of the data collection strategy for  reaching the intended sample population.  As part of our assessment of the  Bureau’s test design, we also reviewed evaluations of the prior decennial  census to determine the degree to which the new operations being tested  addressed problematic aspects of the 2000 Census.  However, we did not  assess the Bureau’s criteria in selecting its objectives for the 2004 census  test.
To determine if the Bureau implemented the test consistent with its plans,  we made multiple site visits to local census offices in Thomasville, Georgia;  and Queens Borough, New York.  During these visits, we interviewed local  census office mangers and staff, observed various data collection activities,  and attended weeklong enumerator training.  We observed a total of 20  enumerators as they completed their daily nonresponse follow-up  assignments—half of these were in southern Georgia, in the counties of  Thomas, Colquitt, and Tift, and half were in Queens (see fig. 2 for maps of  the test site areas).  The results of these observations are not necessarily  representative of the larger universe of enumerators.
To evaluate the quality of the Bureau’s IT security practices, we assessed  risk management documentation associated with IT systems and major  applications for the 2004 census test.  We based our determination on  applicable legal requirements, Bureau policy, and leading practices  described in our executive guide for information security management.
We also interviewed key Bureau officials associated with computer  security.
To identify lessons learned from the 2004 census test, we met with officials  from the Bureau’s Decennial Management Division regarding overall test  plans and with officials from its Technologies Management Office about  using HHCs.  Bureau officials and census workers from both test locations  also provided suggestions on improving census operations.
We requested comments on a draft of this report from the Secretary of  Commerce.  On December 20, 2004, the Under Secretary for Economic  Affairs, Department of Commerce, forwarded written comments from the  Bureau (see app. I).  We address these comments in the “Agency Comments  and Our Evaluation” section at the end of this report.
section_title: The Census Test Was Generally Sound, but Refinements Could Produce Better Cost and Performance Data
paragraphs: The Bureau designed a sound census test and generally implemented it as  planned.  However, in looking ahead, the Bureau’s planning and investment  decisions could benefit from analyzing (1) the degree to which HHCs  contributed to the Bureau’s cost containment goal and (2) the results of the  targeted second mailing, an operation designed to increase participation by  sending a follow-up questionnaire to nonresponding households.  Future  tests could also be more informative if the Bureau developed quantifiable  productivity and other performance requirements for the HHCs and then  used the 2006 test to determine whether the devices are capable of meeting  those requirements.  Collectively, these refinements could provide Bureau  officials with better information to guide its IT and other design decisions,  as well as refine future census tests.
subsections: section_title: The Bureau Developed a Sound Test Design
paragraphs: The design of the 2004 census test contained many components of a sound  study (see table 1).  For example, the Bureau identified test objectives,  designed related research questions, and described a data collection  strategy appropriate for a field test.  The Bureau also developed evaluation  plans for each of the test’s 11 research questions, and explained how  stakeholders were involved with the design, as well as how lessons learned  from past studies were incorporated.
section_title: Additional Analysis Would Provide Better Data on the Impact of Key Test Components
paragraphs: Although the Bureau plans to evaluate various aspects of the 2004 test, it  does not currently plan to assess the impact that the HHCs and targeted  second mailing had on cost savings and productivity.  According to the  Bureau, the census test was focused more on determining the feasibility of  using the HHCs and less on the devices’ ability to save money.  Likewise,  the Bureau said it is not assessing the impact of the targeted second mailing  because the operation is not one of its four test objectives for improving   (1) field data collection using the HHC, (2) the coverage of undercounted  groups, (3) questions about race and ethnicity, and (4) methods for defining  special places and group quarters.
These decisions might be shortsighted, however, in that the Bureau  included the HHCs and targeted second mailing in the 2010 Census design,  in part, to reduce staff, improve productivity, and control costs.  For  example, Bureau studies have shown that sending out replacement  questionnaires could yield a gain in overall response of 7 to 10 percent from  households that do not respond to the initial census mailing, and thus  generate significant cost savings by eliminating the need for census  workers to obtain those responses via personal visits.  Thus, information  on the degree to which the HHCs and second mailing contribute to these  key goals could help inform future budget estimates, investment and design  decisions, as well as help refine future census tests.
Moreover, the feasibility of a targeted second mailing is an open question,  as the Bureau has never before included this operation as part of a  decennial census.  Although a second mailing was part of the original  design for the 2000 Census, the Bureau had abandoned it because it was  found to be logistically unworkable.  A Bureau official said that the second  mailing was included in the 2004 test only to facilitate the enumeration  process, and it would be better tested in a larger scale operation such as  the 2008 dress rehearsal.  However, we believe that it would be more  prudent to assess the second mailing earlier in the census cycle, such as,  during the 2006 test so that its basic feasibility could be assessed, any  refinements could be evaluated in subsequent tests, and the impact on  savings could be estimated more accurately.
section_title: Future Tests Could Be Improved
paragraphs: While the design of the 2004 test was generally sound, refinements could  strengthen the next field test in 2006.  Opportunities for improvement exist  in at least two areas:  ensuring that (1) the HHCs can meet the demanding  requirements of field data collection and (2) management of the local  census offices mirrors an actual enumeration as much as possible.
With respect to the HHCs, because they replace the paper version of the  nonresponse follow-up questionnaire the devices must function effectively.   Further, this test was the first time the Bureau used the HHCs under  census-like conditions so their functionality in an operational environment  was unknown.  Bureau officials have acknowledged that for the 2004 test  they had no predefined indicators of success or failure other than if there  was a complete breakdown the test would be halted.  This is a very low  standard.  Now that the Bureau has demonstrated the basic functionality of  the computers, it should next focus on determining the specific  performance requirements for the HHCs and assess whether the devices  are capable of meeting them.  For example, the Bureau needs productivity  benchmarks for the number of interviews per hour and per day that is  expected per census worker.  Durability measures, such as how many  devices were repaired or replaced, should be considered as well.  Assessing  whether the HHCs can meet the requirements of nonresponse follow-up  will help inform future design and investment decisions for whether or not  to include the devices in the 2010 design.
Ensuring that key positions in the local census offices are filled from the  same labor pool as they would be in an actual decennial census could also  enhance future census tests.  Such was not the case during the 2004 test  when, according to the Bureau, because of difficulties finding qualified  applicants, it used an experienced career census employee to manage the  overall day-to-day operations of the local census office at the Queens test  site.  Another career employee occupied the office’s regional technician  slot, whose responsibilities included providing technical and  administrative guidance to the local census office manager.  In the actual  census, the Bureau would fill these and other positions with temporary  employees recruited from local labor markets.  However, because the  Bureau staffed these positions with individuals already familiar with  census operations and who had ties to personnel at the Bureau’s  headquarters, the Queens test may not have been realistic and the test  results could be somewhat skewed.
section_title: The Bureau Needs to Implement Better IT Security Practices
paragraphs: The Bureau operated a number of IT systems in order to transmit, manage,  and process data for the test.  The equipment was located at various places  including the Bureau’s headquarters in Suitland, Maryland; its National  Processing Center in Jeffersonville, Indiana; a computer facility in Bowie,  Maryland; as well as the New York and Georgia test sites.
Under Title 13 of the U.S. Code, the Bureau must protect from disclosure  the data it collects about individuals and establishments.  Thus, the  Bureau’s IT network must support both the test’s telecommunications and  data processing requirements, as well as safeguard the confidentiality and  integrity of respondents’ information.
The Federal Information Security Management Act of 2002 (FISMA)  requires each agency to develop, document, and implement an agency-wide  information security program for the IT systems that supports its  operations.  Although the Bureau took a number of steps to implement IT  security over the systems used for the test, based on available information,  the Bureau did not meet several of FISMA’s key requirements.  As a result,  the Bureau could not ensure that the systems supporting the test were  properly protected against intrusion or unauthorized disclosure of sensitive  information.  For example:  IT inventory was not complete.  FISMA requires an inventory of  major information systems and interfaces.  The Bureau did not have a  complete inventory that showed all applications and general support IT  systems associated with the test.  Without such information, the Bureau  could not ensure that security was effectively implemented for all of its  systems used in the test, including proper risk assessments, adequate  security plans, and effectively designed security controls.
There was not sufficient evidence that the Bureau assessed all of  the devices used in the test for vulnerabilities, or that it  corrected previously identified problems.  FISMA requires that  agencies test and evaluate the effectiveness of information security  policies, procedures, and practices for each system at least annually and  that agencies have a process for remediating any identified security  weaknesses.  Since the Bureau could not provide us with a complete  inventory of all network components used in the test, we could not  determine if the Bureau’s tests and evaluations were complete.   Moreover, there was not always evidence about whether the Bureau had  corrected past problems or documented reasons for not correcting  them.  As a result, the Bureau did not have adequate assurance that the  security of systems used in the 2004 census test was adequately tested  and evaluated or that identified weaknesses were corrected on a timely  basis.
Assessments were not consistent.  FISMA requires agencies to  assess the risks that could result from the unauthorized access, use,  disclosure, disruption, modification, or destruction of information or  information systems.  Although the Bureau performed risk assessments  for some of the IT components used in the 2004 census test, the  documentation was not consistent.  For example, documentation of  information sensitivity risks (high, medium, and low) for confidentiality,  integrity, and availability of information were not consistent and did not  always follow Bureau policy.  In addition, documents showed different  numbers of file servers, firewalls, and even different names of devices.    Without complete and consistent risk assessment documentation, the  Bureau had limited assurance that it properly understood the security  risks associated with the test.
The Bureau did not always follow its own risk policies.  FISMA  requires the implementation of policies and procedures to prevent  and/or mitigate security risks.  Although Bureau policies allowed for the  waiver of security policies, if appropriate, we noted that such policies  were not always followed.  For example, a waiver for the test of certain  password policies was not properly documented and other system  documents were not properly updated to reflect the waiver.  As a result,  the risk assessment for the 2004 census test did not properly identify the  related risks and did not identify any compensating controls to reduce  the risk to an acceptable level.
As the Bureau plans future tests and the census itself, it will be important  for it to strengthen its IT security risk management practices, ensuring they  fully adhere to FISMA requirements and its own IT security policies.
section_title: Test Reveals Technical, Training, and Other Challenges in Need of Prompt Resolution
paragraphs: The 2004 test suggests that while certain census initiatives have potential,  formidable challenges remain.  For example, the HHCs show promise in  that enumerators were successful in using them to collect data from  nonrespondents and remove late mail returns.  Still, they were not street  ready as they experienced transmission and memory overload problems.   Likewise, automated maps were difficult to use, certain questionnaire  items confused respondents, and enumerators did not always follow  interview protocols.  These problems shed light on issues in need of the  Bureau’s attention as it develops solutions and incorporates refinements  for additional testing in the years ahead.
subsections: section_title: HHCs Were Effective for Conducting Interviews and Removing Late Mail Returns
paragraphs: The Bureau purchased 1,212 HHCs for the test at a total cost of about $1.5  million.  The devices were sent directly to the two test sites packaged in  kits that included a battery, AC adaptor, and modem card for transmitting  data via the telephone.  The HHCs were also equipped with a Global  Positioning System (GPS), a satellite-based navigational system to help  enumerators locate street addresses.  The Bureau anticipates the HHCs will  allow it to eliminate the millions of paper questionnaires and maps that  enumerators need when following up with nonrespondents, thereby  improving their efficiency and reducing overall costs.
Because the Bureau had never used HHCs in the decennial census, an  important goal of the test was to see whether enumerators could use them  for interviewing nonrespondents (see fig. 3).  Most workers we observed  had little trouble using the device to complete the interviews.  In fact, most  said they were pleased with the HHC’s overall functionality, durability,  screen clarity, and the ability to toggle between the questionnaire and  taking GPS coordinates.
Another important function of the HHC was removing late mail returns  from each enumerator’s assignment area(s).  Between the Georgia and  Queens test sites, over 7,000 late mail returns were removed, reducing the  total nonresponse follow-up workload by nearly 6 percent.
The ability to remove late mail returns from the Bureau’s nonresponse  follow-up workload could help save money in that it could eliminate the  need for enumerators to make expensive follow-up visits to households  that return their questionnaires after the mail-back deadline.  Had the  Bureau possessed this capability during the 2000 Census, it could have  eliminated the need to visit nearly 773,000 late-responding households and  saved an estimated $22 million (based on our estimate that a 1-percentage  point increase in workload could add at least $34 million in direct salary,  benefits, and travel costs to the price tag of nonresponse follow-up).   Because of the Bureau’s experience in 2000, in our 2002 report on best  practices for more cost-effective nonresponse follow-up, we  recommended, and the Bureau agreed, that it should develop options that  could purge late mail returns from its nonresponse follow-up workload.
section_title: Technical and Training Difficulties Caused HHC Transmission Problems
paragraphs: Each day, enumerators were to transmit completed nonresponse follow-up  cases to headquarters and receive assignments, software uploads, or both  via a telephone modem (see fig. 4 for a flowchart describing the file  transmission process).  However, the majority of workers we interviewed  had problems doing so, in large part because of technical reasons or  because the Bureau’s training did not adequately prepare them for the  complexity of the transmission procedure, which was a multistep process  involving the connection of a battery pack, cables, and other components.   As reliable transmissions are crucial to the success of nonresponse follow- up, it will be important for the Bureau to resolve these issues so that the  HHCs can be reevaluated in 2006.
Difficulties began during training when the first transmission was supposed  to occur and continued through the remainder of the test.  During that first  transmission, the Bureau needed to upload a number of software upgrades  along with each census worker’s first assignment.  Many of these  transmissions failed because of the volume of data involved.  Thus, without  cases, the trainees could not complete an important section of on-the-job  training.  The Bureau acknowledged that these initial problems could have  been avoided if the final version of software had been installed on the  devices prior to their distribution at training.
Transmission problems persisted throughout nonresponse follow-up.   According to the Bureau, during the first 2 weeks of this operation,  successful data transmission occurred 80 percent of the time once a  connection was made.  However, a number of enumerators never even  established a connection because of bad phone lines, incorrect passwords,  and improper setup of their modems.  Other transmission problems were  due to the local telecommunication infrastructure at both test sites.  For  example, in Georgia, older phone lines could not always handle  transmissions, while in Queens, apartment intercoms that used phone lines  sometimes interrupted connections.
Further, while the transmission rate ultimately increased to 95 percent-– roughly the maximum allowed by the technology—that level is still short of  the performance level needed for 2010.  During the 2000 Census, a 95  percent success rate would have resulted in the failure to transmit around  30,000 completed questionnaires each day.
During the test, the Bureau also had to contend with census workers who  were “living off the grid”; that is, they only used cellular phones and lacked  landlines to transmit and receive data from their homes.  While individuals  could make alternative arrangements, such as using a neighbor’s telephone,  an increasing number of people nationwide in the coming years might give  up their landline service to rely on cellular phones, which could be  problematic for the Bureau.  Bureau officials have noted that all these  transmission problems need to be addressed before 2010.
HHCs experienced memory overloads if too many assignment areas were  loaded onto them.  An assignment area typically contains 40 housing units  or cases that are assigned to an enumerator for nonresponse follow-up.   The design was to have an entire assignment area transmitted to the HHC  even when as few as one case needed follow-up.  However, some  enumerators’ HHCs became overloaded with too much data, as cases had  to be reassigned due to staff turnover, a larger-than-expected number of  refusals, and reassignments resulting from language problems.  As such,  when HHCs became overloaded they would crash and enumerators had to  reconfigure them at the local census office, which made them less  productive.  To the Bureau’s credit, during the test, it was able to work out  a solution to avoid overloads by assigning individual cases instead of the  entire assignment area to a census worker’s HHC.
Another problem that surfaced during the test was that the HHC’s mapping  feature was difficult to use.  To contain costs and increase efficiency, the  Bureau expects to replace paper maps with the electronic maps loaded on  the HHCs for 2010.  However, during the test, enumerators reported that  they did not always use the mapping function because it ran slowly and did  not provide sufficient information.  Instead, they relied on local maps or  city directories, and one worker explained that she found it easier to use an  Internet mapping service on her home computer to prepare for her route.
Without the Bureau’s maps, enumerators might not properly determine  whether a housing unit was located in the Bureau’s geographic database.   This verification is important for ensuring that housing units and the people  who reside in them are in the correct census block, as local and state  jurisdictions use census population figures for congressional redistricting  and allocating federal funds.
Enumerators were also unable to use the HHCs’ “go back” function to edit  questionnaires beyond a certain point in the interview.  In some cases, this  led to the collection of incorrect data.  For example, we observed one  worker complete half an interview, and then discover that the respondent  was providing information on a different residence.  After the census  worker entered the number of residents and their names, the “go back”  function was no longer available and as a result that data could not be  deleted or edited.  Instead, the worker added information in the “notes  section” to explain that the interview had taken place at the wrong  household.  However, Bureau officials told us that they had not planned to  review or evaluate these notes and were not aware that such address mix- ups had been documented in the notes section.
To the extent address mix-ups and other inconsistencies occur and are not  considered during data processing, accuracy could be compromised.  In  earlier censuses when the Bureau used paper questionnaires, if workers  made mistakes, they could simply erase them or record the information on  new forms.  As mistakes are inevitable, it will be important for the Bureau  to ensure that the HHCs allow enumerators to edit information, while still  maintaining the integrity of the data.
section_title: Bureau Needs to Review Format of Coverage Improvement and Race/Ethnicity Questions
paragraphs: We found that questions designed to improve coverage and better  determine race and ethnicity were awkward for enumerators to ask and  confusing for respondents to answer.  Consequently, enumerators  sometimes did not read the questions exactly as worded, which could  adversely affect the reliability of the data collected for these items, as well  as the Bureau’s ability to evaluate the impact of the revised questions.  Our  observations also highlight the importance of ensuring that workers are  trained to follow interview protocols; this issue will be discussed later in  this report.
subsections: section_title: Coverage Improvement
paragraphs: While the Bureau attempts to count everyone during a census, inevitably  some people are missed and others are counted more than once.  To help  ensure that the Bureau properly counts people where they live, the Bureau  revised and assessed its residency rules for the 2004 census test.  For  example, under the residence rules, college students should be counted at  their campus addresses if they live and stay there most of the time.  The  Bureau also added two new coverage questions aimed at identifying  household residents who might have been missed or counted in error (see  fig. 5 for coverage questions).
Enumerators were to show respondents flashcards with the residence rules  to obtain the number of people living or staying in the housing unit and to  read the two coverage questions.  However, during our field visits we noted  that they did not consistently use the flashcards, preferring to summarize  them instead.  Likewise, enumerators did not always ask the new coverage  questions as written, sometimes abbreviating or skipping them altogether.   A frequent comment from the workers we spoke with was that the two new  coverage questions were awkward because the questions seemed  redundant.  Indeed, one census worker said that he asked the overcount  and undercount questions more times than not, but if people were in a  hurry, he did not ask the questions.  During one of these hurried interviews,  we observed that the census worker did not ask the questions and simply  marked “no” for the response.
section_title: Race and Ethnicity Questions
paragraphs: Collecting reliable race and ethnicity data is an extremely difficult task.   Both characteristics are subjective, which makes accurate measurement  problematic.  In 2003, the Bureau tested seven different options for  formatting the race and ethnic questions, and selected what it thought was  the optimal approach to field test in 2004.  The Bureau planned to examine  respondent reaction to the new race and Hispanic origin questions by  comparing responses collected using the paper questionnaire to responses  recorded on the HHCs during nonresponse follow-up.
One change the Bureau planned to analyze was the removal of the “some  other race” write-in option from the questionnaire.  In 2000, the Bureau  found that when given this option, respondents would check off “some  other race,” but did not always write in what their race was.  Thus, in the  2004 test, the Bureau wanted to assess respondents’ reaction to the  removal of the “some other race” write-in option.  Specifically, the Bureau  wanted to see whether respondents would skip the item or select from one  of the other options given.
However, we found that the Bureau formatted the race question on the  paper questionnaire differently from the question on the HHC.  As shown in  figure 6, on the paper version, there is not a category for another race other  than those categories listed, thus forcing respondents to select a category  or skip the question entirely.
This contrasts with the HHCs where, if respondents do not fit into one of  the five race categories, the questionnaire format allows them to provide an  “other” response and enumerators can record their answers.  In fact, the  HHC requires enumerators to record a response to the race question and  will not allow the interview to continue until a response is entered.  As a  result, the data recorded by the two questionnaire formats are not  comparable as they could produce different data depending on the data  collection mode.
According to the Bureau, it formatted the paper version of the race  question differently from the HHC version because it considered the  “other” response option on the HHC a respondent comment and not a  write-in response.  Nevertheless, if the Bureau’s purpose is to measure  respondent reaction to eliminating the write-in option, it is uncertain what  conclusions the Bureau will be able to draw given that this option, even  though in the form of a comment, is still available to the respondent during  the nonresponse follow-up interview.
As was the case with the coverage measurement question, enumerators at  both test locations did not always follow proper interview procedures  because they felt the questions were awkward to ask and confused  respondents.  For example, some workers did not use the flashcards  designed to guide respondents in selecting categories for their race and  ethnicity and to ensure data consistency.  One census worker said that  rather than use the flashcards or ask the questions, he might “eyeball” the  race and ethnicity.  Another worker said that most people laughed at the  Spanish, Hispanic, or Latino origin question and she had complaints about  the wording of this question.  A third census worker noted that he was  “loose with the questions” because he could pose them better.  Like lapses  to the coverage improvement procedures for the 2004 census test,  deviating from the interview procedures for the new race and ethnicity  questions may affect the reliability of the data and the validity of the  Bureau’s conclusions concerning respondent reaction to these questions.
Since the 2004 census test, the 2005 Consolidated Appropriations Act  required that the Bureau include “some other race” as a category when  collecting census data on race identification.  Consequently, the Bureau  said it will include this category on all future census tests and the 2010  Census itself.  Thus, while research into eliminating the “some other race”  category is now moot, it will still be important for the Bureau to have  similar formats for the HHCs and paper questionnaires so that similar data  can be captured across modes.   Likewise, it will be important for the  wording of those questions to be clear and for enumerators to follow  proper procedures during interviews.
section_title: New Procedures Should Help Reduce Duplicate Enumerations of Group Quarter Residents, but Other Challenges Remain
paragraphs: As noted previously, under its residence rules, the Bureau enumerates  people where they live and stay most of the time.  To facilitate the count,  the Bureau divides residential dwellings into two types:  housing units,  such as single-family homes and apartments, and group quarters, which  include dormitories, prisons, and nursing homes.
The Bureau tested new group quarters procedures in 2004 that were  designed to address the difficulties the Bureau had in trying to identify and  count this population group during the 2000 Census.  For example,  communities reported instances where prison inmates were counted in the  wrong county and residents of college dormitories were counted twice.
One refinement the Bureau made was integrating its housing unit and  group quarter address lists in an effort to avoid counting them once as  group quarters and again as housing units, a common source of error  during the 2000 Census.  Census workers were then sent out to verify  whether the dwellings were in fact group quarters and, if so, to classify the  type of group quarter using a revised “other living quarters facility”  questionnaire.
A single address list could, in concept, help reduce the duplicate counting  that previously occurred when the lists were separate.  Likewise, we  observed that census workers had no problems using the revised facility  questionnaire and accompanying flashcard that allowed the respondent to  select the appropriate type of living facility.  This new procedure addresses  some of the definitional problems by shifting the responsibility for defining  the group quarter type from the Bureau to the respondent, who is in a  better position to know about the dwelling.
Another change tested in 2004 was the classification of group homes,  which in 2000 was a part of the group quarter inventory.  Group homes are  sometimes difficult for census workers to spot because they often look the  same as conventional housing units (see fig. 7).  As a result, they were  sometimes counted twice during the 2000 Census—once as a group  quarter, and once as a housing unit.  For the 2004 test, the Bureau decided  to treat group homes as housing units and include them in the housing unit  list.
Early indications from the Bureau suggest that including group homes as  housing units, whereby they receive a short-form questionnaire in the mail,  may not work.  According to the Bureau, the format of the short form is not  well suited to group home residents.  For example, the questionnaire asks  for the “name of one of the people living or staying here who owns or rents  this place.”  Since the state or an agency typically owns group homes, these  instructions do not apply.  The Bureau stated that it plans to reassess how it  will identify and count people living in group homes.
We identified other problems with the Bureau’s group quarters validation  operation during the 2004 census test.  For example, we were told that  census workers were provided maps of the areas they were assigned but  needed maps for adjoining areas so that they could more accurately locate  the physical location of the group quarters.  In Georgia, where workers  used address data from the 2000 Census, the crew leader explained that  approximately one-third of all the addresses provided were incorrectly  spotted on maps and had to be redone.  They also lacked up-to-date  instructions—for example, they did not know that they were to correct  addresses rather than just delete them if the addresses were wrong.   Further, census workers said that scenarios in the manual and classroom  training were based on perfect situations; thus, they did not provide  adequate training for atypical settings or when problems arose.
section_title: The Bureau Should Rethink Its Approach to Training Enumerators
paragraphs: The success of the census is directly linked to the Bureau’s ability to train  enumerators to do their jobs effectively.  This is a tremendous task given  the hundreds of thousands of enumerators the Bureau needs to hire and  train in just a few weeks.  Further, enumerators are temporary employees,  often with little or no prior census experience, and are expected, after just  a few days of training, to do their jobs with minimal supervision, under  sometimes difficult and dangerous conditions.  Moreover, the individuals  who train enumerators—crew leaders—are often recent hires themselves,  with little, if any, experience as instructors.  Overall, few, if any,  organizations face the training challenges that confront the Bureau with  each decennial population count.
To train the 1,100 enumerators who conducted nonresponse follow-up for  the 2004 test, the Bureau employed essentially the same approach it has  used since the 1970 Census:  crew leaders read material word-for-word  from a training manual to a class of 15 to 20 students.  The notable  exception was that in transitioning from a paper questionnaire to the  HHCs, the Bureau lengthened the training time from 3 days to 5 days.   However, given the demographic and technological changes that have  taken place since 1970, the Bureau might want to explore alternatives to  this rigid approach.
As noted earlier, during nonresponse follow-up, enumerators experienced a  variety of problems that could be mitigated through improved training.  The  problems included difficulties setting up equipment to transmit and  download data; failure to read the coverage and race/ethnicity questions  exactly as worded; and not properly using the flashcards, which were  designed to help respondents answer specific questions.
Most of the shortcomings related to training that we observed during the  test were not new.  In fact, the Bureau had identified these and a number of  other training weaknesses in its evaluation of the 2000 Census, but it is  clear they have not been fully resolved.  Thus, as the Bureau plans for the  2010 Census, it will be important for it to resolve long-standing training  problems as well as address new training issues, such as how best to teach  enumerators to use the HHCs and their associated automated processes.   Our observations of the test point to specific options the Bureau might  want to explore.  They include (1) placing greater emphasis on the  importance of following prescribed interview procedures and reading  questions exactly as worded; (2) supplementing verbatim, uniform training  with modules geared toward addressing the particular enumeration  challenges that census workers are likely to encounter at specific locales;  and (3) training on how to deal with atypical situations or respondent  reluctance.
To help evaluate its future training needs, the Bureau hired a contractor to  review the training for the 2004 test and recommend actions for improving  it.  From GAO’s work on assessing agencies’ training and development  efforts, we have developed a framework that can also help in this regard.   Though too detailed to discuss at length in this report, highlights of the  framework, and how they could be applied to census training, include: 1. performing proper front-end analysis to help ensure that the Bureau’s  enumerator training is aligned with the skill and competencies needed  to meet its field data collection requirements and work processes and  that the Bureau leverages best practices and lessons learned from  training enumerators and from past experience; 2. identifying specific training initiatives that in conjunction with other  strategies, improve enumerators’ performance and help the Bureau  meet its goal of collecting high-quality data from nonrespondents;  3. ensuring effective and efficient delivery of training that reinforces new  and needed competencies, skills, and behaviors without being wedded  to past, and perhaps outmoded, methods; and  4. evaluating the training to ensure it is addressing known skill and  competency weaknesses through such measures as assessing  participant reactions and changes in enumerators’ skill levels and  behaviors.
section_title: Readiness Will Be Critical for Future Tests
paragraphs: Several key features of the 2004 test were not test ready; that is, they were  not fully functional or mature when they were employed at the test sites.   This is a serious shortcoming because it hampered the Bureau from fully  evaluating and refining the various census-taking procedures that will be  used in subsequent tests and the actual census in 2010.  Further, to the  extent these features were integrated with other operations, it impeded the  Bureau from fully assessing those associated activities as well.
Our work, and that of the Department of Commerce Inspector General,  identified the following areas where the Bureau needed to be more  prepared going into the test:  The HHCs crashed, in part, because earlier testing did not identify  software defects that caused the download of more data to the HHCs  than their memory cards could hold.
Transmission failures occurred during enumerator training, in part,  because the HHCs were shipped without the latest version of needed  software.  Although the Bureau ultimately provided the latest software  after several weeks, the upgraded version was unavailable for training  field operations supervisors and crew leaders and for the initial  enumerator training.
According to the Department of Commerce Inspector General, the  Bureau finalized the requirements for the new group quarter definitions  too late for inclusion in group quarters training manuals.  Consequently,  the training lacked certain key instructions, such as how to categorize  group homes.
The Bureau experienced other glitches during the test that with better  preliminary testing or on-site dry runs, might have been detected and  possibly addressed before the test started.  These included the slow start- up of the HHC’s mapping function, and the tendency for apartment house  intercoms to interrupt transmissions.
An important objective of any type of test is to identify what is working and  where improvements are needed.  Thus, it should not be surprising, and, in  fact, should be expected and commended, that shortcomings were found  with some of the various activities and systems assessed during the 2004  test.  We believe that the deficiency is not the existence of problems; rather  it is the fact that several components were incomplete or still under  development going into the test, which made it difficult for the Bureau to  gauge their full potential.  The Bureau had a similar experience in the dress  rehearsal for the 2000 Census, when, because a number of new features  were not test ready, the Bureau said it could not fully test them with any  degree of assurance as to how they would affect the head count.
Because of the tight time frames and deadlines of the census, the Bureau  needs to make the most of its limited testing opportunities.  Thus, as the  Bureau plans for the next field test in 2006 and the 2008 dress rehearsal, it  will be important for the Bureau to ensure the various census operations  are fully functional at the time of the test so they can be properly evaluated.
section_title: Conclusions
paragraphs: The Bureau is well aware that a successful enumeration hinges on early  research, development, testing, and evaluation of all aspects of the census  design.  This is particularly true for the 2010 Census for which, under its  current plan, the Bureau will be relying on HHCs and other methods and  technologies that (1) have never been used in earlier censuses and (2) are  mission critical.  Consequently, the 2004 test was an important milestone in  the 2010 life cycle because it demonstrated the fundamental feasibility of  the Bureau’s basic design and allows the Bureau to advance to the next and  more mature phase of planning and development.
Nevertheless, while the test revealed no fatal flaws in the Bureau’s  approach, the results highlighted serious technical, training,  methodological, and procedural difficulties that the Bureau will need to  resolve.  Since one of the purposes of testing is to determine the  operational feasibility of the census design, it is not surprising that  problems surfaced.  However, looking toward the future, it will be critical  for the Bureau to diagnose the source of these challenges, devise cost- effective solutions, and integrate refinements and fixes in time to be  assessed during the next field test scheduled for 2006.  It will also be  important for Congress to monitor the Bureau’s progress as it works to  resolve these issues.
section_title: Recommendations for Executive Action
paragraphs: To facilitate effective census planning and development, and to help the  Bureau achieve its key goals for the census—reduce risks, improve  accuracy, and contain costs, we recommend that the Secretary of  Commerce direct the Bureau to take the following eight actions:    Analyze the impact that HHCs and the targeted second mailing had on  cost savings and other Bureau objectives.
Ensure the Bureau’s IT security practices are in full compliance with  applicable requirements, such as the FISMA, as well as its own internal  policies.
Enhance the reliability and functionality of HHCs by, among other  actions, (1) improving the dependability of transmissions, (2) exploring  the ability to speed up the mapping feature, (3) eliminating the causes of  crashes, and (4) making it easier for enumerators to edit questionnaires.
Define specific, measurable performance requirements for the HHCs  and other census-taking activities that address such important measures  as productivity, cost savings, reliability, durability, and test their ability  to meet those requirements in 2006.
Review and test the wording and formatting of the coverage and  race/ethnicity questions to make them less confusing to respondents  and thus help ensure the collection of better quality data, and ensure  they are formatted the same way on both the HHC and paper versions of  the census form.
Develop a more strategic approach to training by ensuring the  curriculum and instructional techniques (1) are aligned with the skills  and competencies needed to meet the Bureau’s data collection  requirements and methodology and (2) address challenges identified in  the 2004 test and previous censuses.
Revisit group quarter procedures to ensure they allow the Bureau to  best locate and count this population group.
Ensure that all systems and other census-taking functions are as mature  as possible and test ready prior to their deployment for the 2006 test, in  part by conducting small-scale, interim tests under the various  conditions and environments the Bureau is likely to encounter during  the test and actual enumeration.
Further, to ensure the transparency of the census-planning process and  facilitate Congressional monitoring, we also recommend that the Secretary  of Commerce direct the Bureau to regularly update Congress on the  progress it is making in addressing these and any other challenges, as well  as the extent to which the Bureau is on track for meeting the overall goals  of the 2010 Census.
section_title: Agency Comments and Our Evaluation
paragraphs: The Under Secretary for Economic Affairs at the Department of Commerce  forwarded us written comments from the Census Bureau on a draft of this  report on December 20, 2004, which are reprinted in appendix I.  The  Bureau noted that the 2004 test was its first opportunity to assess a number  of the new methods and technologies under development for 2010, and  emphasized the importance of a sustained, multiyear planning, testing, and  development program to its census modernization effort.
The Bureau generally agreed with seven of our nine recommendations, and  described the steps it was taking to address our concerns.  The Bureau also  provided additional context and clarifying language and we have added this  information to the report where appropriate.
Specifically, the Bureau generally agreed with our recommendations  relating to improving IT security practices, the reliability of the HHCs,  training, testing, and enumeration procedures—and reported it was already  taking a number of steps to address our concerns.  We commend the  Bureau for recognizing the risks and challenges that lie ahead and taking  action to address them.  We will continue to monitor the Bureau’s progress  in resolving these issues and update Congress on a regular basis.
At the same time, the Bureau took exception to our recommendations to  (1) analyze the impact that HHCs and targeted second mailings had on cost  savings and other Bureau objectives, and (2) define specific, measurable  performance requirements for the HHCs and other census-taking activities  and test their ability to meet those requirements in 2006.  With respect to  the first recommendation, the Bureau noted that it did not establish cost- savings and other impacts as test objectives, in part, because the Bureau  believes that the national sample mail test that it conducted in 2003  provided a better method for determining the boost in response rates that  could accrue from a second mailing.  The Bureau maintains that analyzing  the impact of the second mailing would provide it with no more  information beyond what it has already established from the 2003 test and  would be of little value.
We believe this recommendation still applies because it will be important  for the Bureau to assess the impact of the targeted second mailing on other  Bureau objectives.  As we noted in the report, the Bureau included the  HHCs and targeted second mailing in the 2010 Census design, in part, to  reduce staff, improve productivity, and control costs.  Further, as we also  note in the report, the feasibility of a targeted second mailing is an open  question.  Thus, information on the degree to which the HHCs and second  mailing contribute to these key goals could help inform future budget  estimates, investment and design decisions, as well as help refine future  census tests.  In short, the purpose of the analysis we recommend would  not be to see whether these features of the 2010 Census will produce cost- savings, but the extent of those savings and the impact on other Bureau  objectives.
With respect to the second recommendation, the Bureau noted that it had  “baseline assumptions” about productivity, cost-savings, and other  measures for the 2004 Census test and that a key objective of the test was  to gather information to help refine these assumptions.  According to the  Bureau, this will also be a key objective of the 2006 Census Test, although  its performance goal will not be whether it meets specific measures.   Instead, the Bureau intends to focus on successfully collecting information  to further refine those assumptions.  As a result, the Bureau believes the  2006 test will not be a failure if HHC productivity is not achieved, but that it  will be a failure if productivity data are not collected.
The Bureau’s position is inconsistent with our recommendation which we  believe still applies.  As noted in the report, we call on the Bureau to define  measurable performance requirements for the HHCs as well as take the  next step and assess whether the HHCs can meet those requirements as  part of the 2006 test.  This information is essential because it will help the  Bureau gauge whether HHCs can meet its field data collection needs in  2010.  Should the HHCs fail to meet these pre-specified performance  requirements during the 2006 test, the Bureau would need to rethink how it  employs these devices in 2010.
As agreed with your offices, unless you release its contents earlier, we plan  no further distribution of this report until 30 days from its date.  At that  time, we will send copies of this report to the Secretary of Commerce and  the Director of the U.S. Census Bureau.  Copies will be made available to  others on request.  This report will also be available at no charge on GAO’s  home page at http://gao.gov.  Please contact me at (202) 512-6806 or  daltonp@gao.gov or Robert Goldenkoff, Assistant Director, at (202) 512- 2757 or goldenkoffr@gao.gov if you have any questions.  Key contributors  to this report were Tom Beall, David Bobruff, Betty Clark, Robert Dacey,  Richard Donaldson, Elena Lipson, Ronald La Due Lake, Robert Parker, Lisa  Pearson, and William Wadsworth.
section_title: Comments from the Department of Commerce
subsections: section_title: GAO’s Mission
paragraphs: The Government Accountability Office, the audit, evaluation and  investigative arm of Congress, exists to support Congress in meeting its  constitutional responsibilities and to help improve the performance and  accountability of the federal government for the American people. GAO  examines the use of public funds; evaluates federal programs and policies;  and provides analyses, recommendations, and other assistance to help  Congress make informed oversight, policy, and funding decisions. GAO’s  commitment to good government is reflected in its core values of  accountability, integrity, and reliability.
section_title: Obtaining Copies of GAO Reports and Testimony
paragraphs: The fastest and easiest way to obtain copies of GAO documents at no cost  is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts  newly released reports, testimony, and correspondence on its Web site. To  have GAO e-mail you a list of newly posted products every afternoon, go to  www.gao.gov and select “Subscribe to Updates.”
subsections: section_title: Order by Mail or Phone
section_title: To Report Fraud, Waste, and Abuse in Federal Programs
section_title: Congressional Relations
section_title: Public Affairs