2010 Census: Census Bureau Continues to Make Progress in Mitigating Risks to a Successful Enumeration, but Still Faces Various Challenges
Released: 2009-10-07T00:00:00

Letter

Background

As you know, Mr. Chairman, the decennial census is a constitutionally mandated enterprise critical to our nation. Census data are used to apportion seats and redraw Congressional districts, and to help allocate over $400 billion in federal aid to state and local governments each year.

We added the 2010 Census to our list of high-risk areas in March 2008 because improvements were needed in the Bureau’s management of IT systems, the reliability of the HHCs, and the quality of the Bureau’s cost estimates. Compounding the risk was that the Bureau canceled a full dress rehearsal of the census that was scheduled in 2008, in part, because of the HHC’s performance problem, which included freeze-ups and unreliable data transmissions. Although the Bureau had planned to use the HHCs to collect data for both address canvassing and in going door to door following up with nonrespondents, the Bureau ultimately decided to use the HHCs for address canvassing and revert to collecting nonresponse follow-up data using paper. As a result of this decision, the Bureau had to redesign components of its field data collection system to accommodate the new approach, thus introducing new risks.

Among other actions, in response to our findings and recommendations, the Bureau strengthened its risk management efforts, including the development of a high-risk improvement plan that described the Bureau’s strategy for managing risk and key actions to address our concerns.

Still, in March 2009, in testimony before this Subcommittee, we continued to question the Bureau’s readiness. Specifically, we noted that with little more than a year remaining until Census Day, uncertainties surrounded critical operations and support systems, and the Bureau lacked sufficient policies, procedures, and trained staff to develop high-quality cost estimates. Moving forward, we said that it will be essential for the Bureau to develop plans for testing systems and procedures not included in the dress rehearsal, and for Congress to monitor the Bureau’s progress.

The Bureau Has Made Progress on the Management and Testing of Key IT Systems, but Little Time Remains to Address Outstanding Issues

Since 2005, we have reported on weaknesses in the Bureau’s management of its IT acquisitions, and issues continue concerning the Bureau’s IT management and testing of key 2010 Census systems. In March 2009, we reported and testified that while the Bureau took initial steps to enhance its program-wide oversight of testing activities, those steps had not been sufficient. Furthermore, while the Bureau had made progress in testing key decennial systems, critical testing activities remained to be performed before they would be ready to support the 2010 Census. At that time we recommended that the Bureau improve its oversight of the completion of testing activities for key systems.

In response to our findings and recommendations, the Bureau has taken several steps to improve its management of IT for the 2010 Census. For example, the Bureau named a Decennial Census Testing Officer whose primary responsibilities include monitoring testing for decennial census activities. In order to help improve the rigor and quality of test planning and documentation, this official leads a bimonthly process to consolidate and evaluate test planning and status across all key decennial census operations, resulting in a decennial census testing overview document.

With respect to system testing, progress is being made, but much testing remains to be completed as shown in the following table.

The Bureau has also made progress in end-to-end testing, but substantial work remains to be completed. For example, the Bureau has completed limited end-to-end tests for nonresponse follow-up and group-quarters enumeration on the Paper-Based Operations Control System (PBOCS), a work flow management system the Bureau developed late in the census cycle when it moved from the HHCs to a paper-based approach to nonresponse follow-up and other field operations. However, Bureau officials stated that, although they were satisfied with the results of the tests, significant additional testing will be needed. For example, several critical issues were identified during these tests that will need to be resolved and retested. In addition, the test was not designed to evaluate the level of system performance needed while processing the estimated 48 million housing units that will be in the nonresponse-follow-up workload. According to the Bureau, a performance test is being designed for the first major release; however, detailed plans for this test have not yet been completed. Finally, the test was performed with experienced census employees, while the system will be used by newer, temporary employees.

Given the importance of IT systems to the decennial census, it is critical that the Bureau ensure these systems are thoroughly tested. Bureau officials have repeatedly stated that the limited amount of time remaining will make completing all testing activities challenging.

The Bureau Needs to Prioritize Remaining Needs for Systems Used to Manage Field Data Collection

The Bureau faces significant challenges finalizing PBOCS. Most notably, the Bureau needs to determine the remaining detailed requirements for the system to be developed. As of early September 2009, the Bureau had established high-level requirements for its PBOCS but had not yet finalized the detailed requirements. High-level requirements describe in general terms what functions the system will accomplish, such as producing specific management reports on the progress of specific paper-based operations or checking-out and checking-in groups of census forms for shipping or processing. Detailed requirements describe more specifically what needs to be done in order to accomplish such functions. For PBOCS, such detailed requirements might include, for example, which data from which data source should be printed where on a specific management report. According to Bureau officials, in the absence of such specificity in the requirements for the 2008 dress rehearsal, contract programmers with little decennial census experience made erroneous assumptions about which data to use when preparing some quality control reports. As a result, quality assurance managers were unable to rely on the reports for tracking progress.

In recognition of the serious implications that shortcomings in PBOCS would have for the conduct of the 2010 Census and to see whether there were additional steps that could be taken to mitigate the outstanding risks to successful PBOCS development and testing, in June 2009, the Bureau chartered an assessment of PBOCS, chaired by the Bureau’s chief information officer (CIO). The assessment team reported initially in late July 2009 and provided an update the following month. The review stated that the PBOCS developers had made a strong effort to involve the system stakeholders in the development process. However, the review also identified several concerns with PBOCS development. For example, the review found and we confirmed that the Bureau could improve its requirements management for PBOCS. According to the CIO, the Bureau has taken steps to address some of these findings, such as providing additional resources for testing and development; however, resolving problems found during testing before the systems need to be deployed will be a challenge.

At the end of our review, the Bureau presented evidence of the steps it had taken to document and prioritize requirements. We did not assess the effectiveness of these steps. Until the Bureau completes the detailed requirements for PBOCS, it will not have reasonable assurance that PBOCS will meet the program’s needs. The Bureau is continuing to examine how improvements will be made.

The Bureau Generally Completed Address Canvassing Ahead of Schedule But Went Over Budget

A successful census relies on an accurate list of all addresses where people live in the country, because it identifies all households that are to receive a census questionnaire and serves as a control mechanism for following up with households that fail to respond. If the address list is inaccurate, people can be missed, counted more than once, or included in the wrong location.

Address canvassing is one of several procedures the Bureau uses to help ensure an accurate address list and, because it is based on on-site verification, it is particularly important for identifying the locations of nontraditional or “hidden” housing units such as converted attics and basements. Although these types of dwellings have always existed, the large number of foreclosures the nation has recently experienced, as well as the natural disasters that have hit the Gulf Coast and other regions, have likely increased the number of people doubling-up, living in motels, cars, tent cities, and other less conventional living arrangements. Such individuals are at greater risk of being missed in the census.

The Bureau conducted address canvassing from March to July 2009. During that time, about 135,000 address listers went door to door across the country, comparing the housing units they saw on the ground to what was listed in the database of their HHCs. Depending on what they observed, listers could add, delete, or update the location of housing units.

Although the projected length of the field operation ranged from nine to fourteen weeks, most early local census offices completed the effort in less than 10 weeks. Moreover, the few areas that did not finish early were delayed by unusual circumstances such as access issues created by flooding. The completion rate is a remarkable accomplishment given the HHC’s troubled history. The testing and improvements the Bureau made to the reliability of the HHCs prior to the start of address canvassing, including a final field test that was added to the Bureau’s preparations in December 2008, played a key role in the pace of the operation, but other factors, once address canvassing was launched, were important as well, including the (1) prompt resolution of problems with the HHCs as they occurred and (2) lower than expected employee turnover.

With respect to the prompt resolution of problems, although the December 2008 field test indicated that the more significant problems affecting the HHCs had been resolved, various glitches continued to affect the HHCs in the first month of the operation. For example, we were informed by listers or crew leaders in 14 early local census offices that they had encountered problems with transmissions, freeze-ups, and other problems. Moreover, in 10 early local census offices we visited, listers said they had problems using the Global Positioning System function on their HHCs to precisely locate housing units. When such problems occurred, listers called their crew leaders and the Bureau’s help desk troubleshooted the problems. When the issues were more systemic in nature, such as a software issue, the Bureau was able to quickly fix them using software patches.

Moreover, to obtain an early warning of trouble, the Bureau monitored key indicators of the performance of the HHCs such as the number of successful and failed HHC transmissions. This approach proved useful as Bureau quality control staff were alerted to the existence of a software problem when they noticed that the devices were taking a long time to close out completed assignment areas.

The Bureau also took steps to address procedural issues. For example, in the course of our field observations, we noticed that in several locations listers were not always adhering to training for identifying hidden housing units. Specifically, listers were instructed to knock on every door and ask, “Are there any additional places in this building where people live or could live?” However, we found that listers did not always ask this question. On April 28, 2009, we discussed this issue with senior Bureau officials. The Bureau, in turn, transmitted a message to listers’ HHCs emphasizing the importance of following training and querying residents if possible.

Lower than expected attrition rates and listers’ availability to work more hours than expected also contributed to the Bureau’s ability to complete the address canvassing operation ahead of schedule. For example, the Bureau had planned for 25 percent of new hires to quit before, during, or soon after training; however, the national average was 16 percent. Bureau officials said that not having to replace listers with inexperienced staff accelerated the pace of the operation. Additionally, the Bureau assumed that employees would be available 18.5 hours a week. Instead, they averaged 22.3 hours a week.

The Bureau’s address list at the start of address canvassing consisted of 141.8 million housing units. Listers added around 17 million addresses and marked about 21 million for deletion because, for example, the address did not have a structure. All told, listers identified about 4.5 million duplicate addresses, 1.2 million nonresidential addresses, and about 690,000 addresses that were uninhabitable structures. Importantly, these preliminary results represent actions taken during the production phase of address canvassing and do not reflect actual changes made to the Bureau’s master address list as the actions are first subject to a quality control check and then processed by the Bureau’s Geography Division.

The preliminary analysis of addresses flagged for add and delete shows that the results of the operation (prior to quality control) were generally consistent with the results of address canvassing for the 2008 dress rehearsal. Table 2 compares the add and delete actions for the two operations.

Address Canvassing Costs Exceeded Budget Because of Unanticipated Workload and Hiring

According to the Bureau’s preliminary analysis, the estimated cost for address canvassing field operations was $444 million, or $88 million (25 percent) more than its initial budget of $356 million. As shown in table 3, according to the Bureau, the cost overruns were because of several factors.

One such factor was that the address canvassing cost estimate was not comprehensive, which resulted in a cost increase of $41 million. The Bureau inadvertently excluded 11 million addresses identified in address file updates from the initial address canvassing workload and fiscal year 2009 budget. Further, the additional 11 million addresses increased the Bureau’s quality control workload, where the Bureau verifies certain actions taken to correct the address list. Specifically, the Bureau failed to anticipate the impact these addresses would have on the quality control workload and therefore did not revise its cost estimate accordingly. Moreover, under the Bureau’s procedures, addresses that failed quality control would need to be recanvassed, but the Bureau’s cost model did not account for the extra cost of recanvassing of any addresses. As a result, the Bureau underestimated its quality control workload by 26 million addresses which resulted in $34 million in additional costs, according to the Bureau.

Bringing aboard more staff than was needed also contributed to the cost overruns. For example, according to the Bureau’s preliminary analysis, training additional staff accounted for about $7 million in additional costs. Bureau officials attributed the additional training cost to inviting additional candidates to initial training because of concerns that recruiting and hiring staff would be problematic, even though (1) the Bureau’s staffing goals already accounted for the possibility of high turnover and (2) the additional employees were not included in the cost estimate or budget.

The largest field operation will be nonresponse follow-up, when the Bureau is to go door to door in an effort to collect data from households that did not mail back their census questionnaire. Over 570,000 enumerators will need to be hired for that operation. To better manage the risk of staffing difficulties while simultaneously controlling costs, several potential lessons learned can be drawn from the Bureau’s experience during address canvassing. For example, we found that the staffing authorization and guidance provided to some local census managers were unclear and did not specify that there was already a cushion in the hiring goals for local census offices to account for potential turnover. Also, basing the number of people invited to initial training on factors likely to affect worker hiring and retention, such as the local employment rate, could help the Bureau better manage costs.

According to Bureau officials, they are reviewing the results from address canvassing to determine whether they need to revisit the staffing strategy for nonresponse follow-up and have already made some changes. For example, in recruiting candidates, when a local census office reaches 90 percent of its qualified applicant goal, it is to stop blanket recruiting and instead focus its efforts on areas that need more help, such as tribal lands. However, in hiring candidates, the officials pointed out that they are cautious not to underestimate resource needs for nonresponse follow-up based on address canvassing results because they face different operational challenges in that operation than for address canvassing. For example, for nonresponse follow-up, the Bureau needs to hire enumerators who can work in the evenings when people are more likely to be at home and who can effectively deal with reluctant respondents, whereas with address canvassing, there was less interaction with households and the operation could be completed during the day.

Address Canvassing Cost Overruns Are Symptomatic of Weaknesses with Cost Estimation Efforts

Problems with accurately estimating the cost of address canvassing are indicative of long-standing weaknesses in the Bureau’s ability to develop credible and accurate cost estimates for the 2010 Census. Accurate cost estimates are essential to a successful census because they help ensure that the Bureau has adequate funds and that Congress, the administration, and the Bureau itself can have reliable information on which to base decisions. However, in our past work, we noted that the Bureau’s estimate lacked detailed documentation on data sources and significant assumptions, and was not comprehensive because it did not include all costs. Following best practices from our Cost Estimating and Assessment Guide, such as defining necessary resources and tasks, could have helped the Bureau recognize the need to update address canvassing workload and other operational assumptions, resulting in a more reliable cost estimate.

Given the Bureau’s past difficulties in developing credible and accurate cost estimates, we are concerned about the reliability of the figures that were used to support the 2010 budget, especially the costs of nonresponse follow-up, which is estimated to cost $2.7 billion. We have discussed the cost estimate for nonresponse follow-up with Bureau officials, and they have said they are looking to see how foreclosures and vacant housing units might affect the nonresponse follow-up workload. In addition, Bureau officials said they will analyze address canvassing data and determine if there are any implications for future operations.

Nevertheless, there still remains a great deal of uncertainty around the final cost of the 2010 Census. In part, this is because of changes made to the census design after April 2008, when the Bureau reverted to a paper- based data collection method for nonresponse follow-up in response to the performance problems with the HHCs. The uncertainty also stems from the fact that the assumptions used to develop the revised cost estimate were not tested during the 2008 dress rehearsal. According to budget documents, after the decision to return to a paper-based nonresponse follow-up, the life cycle cost estimate increased by over $3 billion dollars.

Moving forward, it will be important for the Bureau to ensure the reliability of the 2020 cost estimate, and the Bureau has already taken several actions in that regard. For example, based on recommendations from our June 2008 report, the Bureau plans to train its staff on cost estimation skills, including conducting uncertainty analysis. In addition, the Bureau is developing the Decennial Budget Integration Tool (DBiT), which according to the Bureau, should consolidate budget information and enable the Bureau to better document its cost estimates. Officials said that DBiT is capturing actual fiscal year 2009 costs, which will be used to estimate the life cycle cost for the 2020 census. However, officials also said that DBiT needs further testing, and may not be fully used until the 2012 budget.

Bureau Needs to Improve Its Policies and Procedures for Fingerprinting Temporary Employees

To better screen its workforce of hundreds of thousands of temporary census workers, the Bureau plans to fingerprint its temporary workforce for the first time in the 2010 Census. In past censuses, temporary workers were only subject to a name background check that was completed at the time of recruitment. The Federal Bureau of Investigation (FBI) is to provide the results of a name background check when temporary workers are first recruited. At the end of the workers’ first day of training, Bureau employees who have received around 2 hours of fingerprinting instruction are to capture two sets of ink fingerprint cards. The cards are then sent to the Bureau’s National Processing Center in Jeffersonville, Indiana, to be scanned and electronically submitted to the FBI. If the results show a criminal record that makes an employee unsuitable for employment, the Bureau is to either terminate the person immediately or place the individual in nonworking status until the matter is resolved. If the first set of prints are unclassifiable, the National Processing Center is to send the FBI the second set of prints.

However, fingerprinting during address canvassing was problematic. Of the over 162,000 employees hired for the operation, 22 percent—or approximately 35,700 workers—had unclassifiable prints that the FBI could not process. The FBI determined that the unclassifiable prints were generally the result of errors that occurred when the prints were first made. Factors affecting the quality of the prints included difficulty in first learning how to effectively capture the prints and the adequacy of the Bureau’s training. Further, the workspace and environment for taking fingerprints was unpredictable, and factors such as the height of the workspace on which the prints were taken could affect the legibility of the prints.

Consistent with FBI guidance, the Bureau relied solely on the results of the name background check for the nearly 36,000 employees with unclassifiable prints. However, it is possible that more than 200 people with unclassifiable prints had disqualifying criminal records but still worked, and had contact with the public during address canvassing. Indeed, of the prints that could be processed, fingerprint results identified approximately 1,800 temporary workers (1.1 percent of total hires) with criminal records that name check alone failed to identify. Of the 1,800 workers with criminal records, approximately 750 (42 percent) were terminated or were further reviewed because the Bureau determined their criminal records—which included crimes such as rape, manslaughter, and child abuse—disqualified them from census employment.

Projecting these percentages to the 35,700 temporary employees with unclassifiable prints, it is possible that more than 200 temporary census employees might have had criminal records that would have made them ineligible for census employment. Applying these same percentages to the approximately 600,000 people the Bureau plans to fingerprint for nonresponse follow-up, unless the problems with fingerprinting are addressed, we estimate that approximately 785 employees with unclassifiable prints could have disqualifying criminal records but still end up working for the Bureau.

Aside from public safety concerns, there are cost issues as well. The FBI charged the Bureau $17.25 per person for each background check, whether or not the fingerprints were classifiable.

The Bureau stated that it has taken steps to improve image quality for fingerprints captured in future operations by refining instruction manuals and providing remediation training on proper procedures. In addition, the Bureau is considering activating a feature on the National Processing Center’s scanners that can check the legibility of the image and thus prevent poor quality prints from reaching the FBI. These are steps in the right direction. As a further contingency, it might also be important for the Bureau to develop a policy for re-fingerprinting employees to the extent that both cards cannot be read.

The Bureau Used Enhanced Training and Guidance for Canvassing Hurricane Affected Areas

The scale of the destruction in those areas affected Hurricanes Katrina, Rita, and Ike made address canvassing in parts of Mississippi, Louisiana, and Texas, especially challenging (see fig. 1). Hurricane Katrina alone destroyed or made uninhabitable an estimated 300,000 homes. Recognizing the difficulties associated with address canvassing in these areas because of shifting and hidden populations and changes to the housing stock, the Bureau, partly in response to recommendations made in our June 2007 report, developed supplemental training materials for natural disaster areas to help listers identify addresses where people are, or may be, living when census questionnaires are distributed. For example, the materials noted the various situations listers might encounter, such as people living in trailers, homes marked for demolition, converted buses and recreational vehicles, and nonresidential space such as storage areas above restaurants. The training material also described the clues that could alert listers to the presence of non-traditional places where people are living and provided a script they should follow when interviewing residents on the possible presence of hidden housing units.

Additional steps taken by the city of New Orleans also helped the Bureau overcome the challenge of canvassing neighborhoods devastated by Hurricane Katrina. As depicted in fig. 2 below, city officials replaced the street signs even in abandoned neighborhoods. This assisted listers in locating the blocks they were assigned to canvass and expedited the canvassing process in these deserted blocks.

To further ensure a quality count in the hurricane affected areas, the Bureau plans to hand-deliver an estimated 1.2 million questionnaires (and simultaneously update the address list) to housing units in much of southeast Louisiana and south Mississippi that appear inhabitable, even if they do not appear on the address list updated by listers during address canvassing. Finally, the Bureau stated that it must count people where they are living on Census Day and emphasized that if a housing unit gets rebuilt and people move back, then that is where those people will be counted. However, if they are living someplace else, then they will be counted where they are living on Census Day.

Concluding Observations

The Bureau has made remarkable progress in improving its overall readiness for 2010, with substantial strides being made in the management of its IT systems and other areas. That said, as I noted throughout this statement, considerable challenges and uncertainties lie ahead. While the decennial is clearly back on track, many things can happen over the next few months, and keeping the entire enterprise on plan continues to be a daunting challenge fraught with risks.

Mr. Chairman and members of this Subcommittee, this concludes my statement. I would be happy to respond to any questions that you might have at this time.

GAO Contacts

If you have any questions on matters discussed in this statement, please contact Robert N. Goldenkoff at (202) 512-2757 or by e-mail at goldenkoffr@gao.gov. Other key contributors to this testimony include Steven Berke, Virginia Chanley, Benjamin Crawford, Jeffrey DeMarco, Dewi Djunaidy, Vijay D’Souza, Elizabeth Fan, Ronald Fecso, Amy Higgins, Richard Hung, Kirsten Lauber, Jason Lee, Andrea Levine, Signora May, Ty Mitchell, Naomi Mosser, Catherine Myrick, Lisa Pearson, David Powner, David Reed, Jessica Thomsen, Jonathan Ticehurst, Shaunyce Wallace, Timothy Wexler, and Katherine Wulff.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.