Census 2010: Census at Critical Juncture for Implementing Risk Reduction Strategies
Released: 2008-04-09T00:00:00

Letter

Redesign Implications for Decennial Census Operations

The Decennial Census is at a critical stage in the 2008 Dress Rehearsal, in which the Bureau has its last opportunity to test its plans for 2010 under census-like conditions. The dress rehearsal features a mock Census Day, now set for May 1, 2008. Last year at this time, the Bureau carried out a major dress rehearsal operation—address canvassing—in which the Bureau updated address lists and collected global positioning coordinates for mapspots. The largest field operation of the dress rehearsal was to have begun this month. In this operation (nonresponse follow-up), field staff were to conduct face-to-face interviews with households that did not mail back their questionnaires.

Prior to the redesigning effort, the Bureau had already changed its plans for the dress rehearsal, in part, to focus greater attention on the testing of technology. In a November 20, 2007, decision memo, the Bureau announced that it would delay Census Day for the dress rehearsal by 1 month, to May 1, 2008. The Bureau also listed a number of operations it no longer planned to rehearse, including group quarters enumeration and census coverage measurement. Also in February 2008, the Bureau announced that it would remove from the scope of the FDCA program contract the development of all systems and software associated with the census coverage measurement operation.

The redesign approach selected by the Secretary will require that the Bureau quickly develop and test a paper-based nonresponse follow-up operation. Any paper-based option has its own set of unique issues, such as setting up operations to support paper field data collection centers and seeking printing solutions for enumerator forms. Among other issues, decisions on a printing solution will need to be made soon. Although the Bureau has carried out paper-based operations before, in some cases they now involve new procedures and system interfaces that as a result of their exclusion from the dress rehearsal, will not be tested under census-like conditions. For nonresponse follow-up in 2010 the Bureau will be using newly developed systems for integrating responses and controlling workload. For example, the Bureau will need to rely on a newly developed system called the Decennial Response Integration System to identify households that have not returned census forms and for collecting the results of enumerators conducting nonresponse follow-up person interviews. Dropping the use of the HHCs for nonresponse follow-up and reverting to paper for that operation this late in the decade also precludes nonresponse follow-up from being fully tested in the dress rehearsal. Under the delayed dress rehearsal this operation was to begin next month, soon after households in dress rehearsal locations were to return their census forms. A paper operation requires different training, maps, and other material to be prepared prior to the operation. The Bureau has announced no specific plans for conducting field testing of certain key operations, such as nonresponse follow-up. Without sufficient testing, operational problems can go undiscovered and the opportunity to improve operations will be lost.

The redesign’s move from the use of HHCs to a paper-based nonresponse follow-up operation may limit the Bureau’s ability to reduce follow-up with persons who are late in returning their census questionnaires. One of the primary advantages the Bureau cited for using HHCs was the ability, as late mail returns came in, to remove those addresses from enumerators’ assignments—preventing enumerators from doing unnecessary work. According to the Bureau, in 2000 enumerators visited over 4 million households that had returned their census form late. In 2004, the Bureau tested the capability of an earlier prototype of the HHC to adjust workloads by identifying late mail returns. We reported in 2007 that based on these tests it appears that if the Bureau had possessed this capability during the 2000 Census, it could have eliminated the need to visit nearly 773,000 late-responding households and saved an estimated $22 million (based on our estimate that a 1 percentage point increase in workload could add at least $34 million in direct salary, benefits, and travel costs to the price tag of nonresponse follow-up). The Director of the Census Bureau stated that he believes that the Bureau can still partially adjust enumerator workload to recognize late mail returns without the use of HHCs. To achieve this objective, the Bureau will need to specify the process it will use and conduct appropriate tests.

The redesign will also affect the 2010 Census address canvassing operation. The Secretary’s decision to use the HHCs for the 2010 address canvassing operation means that certain performance issues with the handheld technology must be addressed promptly. Field staff experienced difficulties using the technology during the address canvassing dress rehearsal. For example, workers reported problems with HHCs when working in large assignment areas during address canvassing. The devices could not accommodate more than 720 addresses—3 percent of dress rehearsal assignment areas were larger than that. The amount of data transmitted and used slowed down the HHCs significantly. Identification of these problems caused the contractor to create a task team to examine the issues, and the team recommended improving the end-to-end performance of the mobile solution by controlling the size of assignment area data delivered to the HHC both for address canvassing and nonresponse follow-up operations. One specific recommendation was limiting the size of assignment areas to 200 total addresses. However, the redesign effort took another approach that uses laptops and the software application used for the American Community Survey to collect information in large assignment areas. It is not yet clear how this work-around will be carried out. Furthermore, the Bureau will need to define specific and measurable performance requirements for the HHCs as we recommended in January 2005.

Another operational issue is the ability of the contractor to accept changes to its address files after it completes address canvassing updates. This could preclude the Bureau from conducting “restart/redo” operations for an area where the address file is discovered to be incorrect. This function is critical in developing an accurate and complete address list. Without the ability to update the mailing list for “restart/redo” operations, the Bureau would consider not mailing census questionnaires to addresses in that area and instead deliver census forms by hand. This has the potential to significantly increase costs.

Redesign Implications for IT Acquisitions

The Bureau still needs to agree upon and finalize requirements for the FDCA program. In March 2006, we reported that the FDCA project office had not implemented the full set of acquisition management capabilities (such as project and acquisition planning and requirements development and management) that were needed to effectively manage the program. For example, although the project office had developed baseline functional requirements for the acquisition, the Bureau had not yet validated and approved them. Subsequently, in October 2007, we reported that changes to requirements had been a contributing factor to both cost increases and schedule delays experienced by the FDCA program. In June 2007, an assessment by an independent contractor of the FDCA program reported on requirements management problems—much like those we reported in March 2006. Similar to our recommendation, the independent assessors recommended that the Bureau immediately stabilize requirements by defining and refining them. The Bureau has recently made efforts to further define requirements for the FDCA program, and it has estimated that the revised requirements will result in significant cost increases. On January 16, 2008, the Bureau provided the FDCA contractor with a list of over 400 requirements for the FDCA program to reconcile. Although some of these new requirements will be dropped based on the Secretary’s recent decision, many will still need to be addressed to ensure that FDCA will perform as needed.

Commerce and Bureau officials need to address critical weaknesses in risk management practices. In October 2007, we reported that the FDCA project had weaknesses in identifying risks, establishing adequate mitigation plans, and reporting risk status to executive-level officials. For example, the FDCA project team had not developed mitigation plans that were timely or complete nor did it provide regular briefings on risks to senior executives. The FDCA project team’s failure to report a project’s risks to executive-level officials reduces the visibility of risks to executives who should be playing a role in mitigating them.

As of October 2007, in response to the cost and schedule changes, the Bureau decided to delay certain system functionality for FDCA. As a result, the operational testing that was to occur during the dress rehearsal period around May 1, 2008, would not include tests of the full complement of Decennial Census systems and their functionality. Operational testing helps verify that systems function as intended in an operational environment. In late 2007, according to Bureau officials, testing plans for IT systems were to be finalized in February 2008. Therefore, we recommended that the Bureau plan and conduct critical testing, including end-to-end testing of the Decennial Census systems. As of March 2008, the Bureau still had not developed these test plans. In the recent program redesign, the Bureau included conducting end-to-end testing. The inability to perform comprehensive operational testing of all interrelated systems increases the risk that further cost overruns will occur, that decennial systems will experience performance shortfalls, or both.

Given the redesigning effort, implementing our recommendations associated with managing the IT acquisitions is as critical as ever. Specifically, the Bureau needs to strengthen its acquisition management capabilities, including finalizing FDCA requirements. Further, it also needs to strengthen its risk management activities, including developing adequate risk mitigation plans for significant risks and improving its executive-level governance of these acquisitions. The Bureau also needs to plan and conduct key tests, including end-to-end testing, to help ensure that decennial systems perform as expected.

Redesign Implications for Decennial Census Life Cycle Costs

Even without considering the recent expected cost increases announced by the Bureau to accommodate the redesign of the FDCA program, the Bureau’s cost projections for the 2010 Census revealed an escalating trend from the 1970 Census. As shown in figure 1, the estimated $11.8 billion cost (expressed in constant 2010 dollars) of the 2010 Census, before the FDCA program redesign, represented a more than tenfold increase over the $1 billion spent on the 1970 Census. The 1970 Census was the first Census to rely on mailing census forms to households and asking for a mail return—a major part of the data collection. Although some of the cost increase could be expected because the number of housing units—and hence the Bureau’s workload—has gotten larger, the cost growth has far exceeded the increase in the number of housing units. The Bureau estimated that the number of housing units for the 2010 Census would increase by almost 14 percent over Census 2000 levels.

As figure 2 shows, before the FDCA program redesign, the Bureau estimated that the average cost per housing unit for the 2010 Census was expected to increase by approximately 26 percent over 2000 levels, from $69.79 per housing unit to $88.19 per housing unit in constant 2010 dollars. When the projected cost increase that accompanies the FDCA program redesign is considered, the average cost per housing unit will increase by an even greater percentage.

Given the projected increase in spending, it will be imperative that the Bureau effectively manage the 2010 Census, as the risk exists that the actual, final cost of the census could be considerably higher than anticipated. Indeed, this was the case for the 2000 Census, when the Bureau’s initial cost projections proved to be too low because of such factors as unforeseen operational problems and changes to the fundamental design. The Bureau estimated that the 2000 Census would cost around $5 billion. However, the final price tag for the 2000 Census was more than $6.5 billion, a 30 percent increase in cost. Large federal deficits and other fiscal challenges underscore the importance of managing the cost of the census, while promoting an accurate, timely census.

We have repeatedly reported that the Bureau would be challenged to control the cost of the 2010 Census. In January 2004, we reported that under the Bureau’s approach for reengineering the 2010 Census, the Bureau might find it difficult to reduce operational risk because reengineering introduces new risks. To manage the 2010 Census and contain costs, we recommended that the Bureau develop a comprehensive, integrated project plan for the 2010 Census that should include the itemized estimated costs of each component, including a sensitivity analysis and an explanation of significant changes in the assumptions on which these costs were based. In response, the Bureau provided us with the 2010 Census Operations and Systems Plan, dated August 2007. This plan represented an important step forward at the time. It included inputs and outputs and described linkages among operations and systems. However, it did not yet include sensitivity analysis, risk mitigation plans, a detailed 2010 Census timeline, or itemized estimated costs of each component. Going forward, it will be important for the Bureau to update its operations plan.

The assumptions in the fiscal year 2009 President’s Budget life cycle cost estimate of $11.5 billion may not have included recent productivity data from last year’s address canvassing dress rehearsal. According to the Bureau, initially, the cost model assumed productivity for address canvassing to be 25.6 addresses per hour for urban/suburban areas. However, results from the address canvassing dress rehearsal showed productivity of 13.4 addresses per hour for urban/suburban areas. While the life cycle cost estimate increased slightly to $11.5 billion in the fiscal year 2009 President’s Budget, these increases were attributed to other factors and not to lower-than-expected canvassing productivity. Best practices call for cost model assumptions to be updated as new information becomes available. We previously reported that the life cycle cost estimate has not been updated to reflect changes in assumptions. In July 2006, we testified that the estimate had not been updated to reflect the results of testing conducted in 2004. As the Bureau updates its estimate of the life cycle cost annually and as part of the redesigning effort, it will be important that it reflect changing assumptions for productivity and hours worked.

Concluding Observations

Given its size and complexity, carrying out the Decennial Census presents significant challenges under any circumstances. Late changes in census plans and operations, long-standing weaknesses in IT acquisition and contract management, limited capacity for undertaking these critical management functions, scaling back of dress rehearsal activities, and uncertainty as to the ultimate cost of the 2010 Census puts the success of this effort in jeopardy. Managing these risks is critical to the timely completion of a reliable and cost-effective census. Implementing our recommendations would help the Bureau effectively manage the myriad of interrelated operations needed to ensure an accurate and complete count in 2010 (Bureau officials have agreed with many of our recommendations, but have not fully implemented them).

The dress rehearsal represents a critical stage in preparing for the 2010 Census. This is the time when the Congress and others should have the information they need to know how well the design for 2010 is likely to work, what risks remain, and how those risks will be mitigated. We have highlighted some of the risks today. Going forward, it will be important for the Bureau to specify how it will ensure that planned dress rehearsal operations will be successfully carried out, and how it will provide assurance that the largest operation—nonresponse follow-up—will be tested in the absence of a full dress rehearsal. Likewise, the Bureau will need to establish plans for working around limitations in the technology to be used in address canvassing operations. It is critical that the Bureau ensure that the technology for conducting address canvassing is a success.

The Bureau should implement prior recommendations in moving forward. Contractor-developed IT systems and deliverables need to be closely monitored to ensure that contractors are performing within budget. As we have stressed throughout this testimony and in our prior recommendations, the Bureau needs to practice aggressive project management and governance over both the IT and non-IT components. Further, it is essential that the Bureau implement our recommendations related to information technology. The Bureau must solidify the FDCA program requirements, strengthen risk management activities, and plan and conduct critical testing of the Decennial Census systems.

Mr. Chairmen, Census Day is less than 2 years away and address canvassing is 1 year away. The challenges we highlighted today call for effective risk mitigation by the U.S. Census Bureau, and careful monitoring and oversight by the Department of Commerce, the Office of Management and Budget, the Congress, GAO, and other key stakeholders. As in the past, we look forward to supporting the committee and subcommittee’s oversight efforts to promote an accurate and cost-effective census.

Mr. Chairmen, this concludes our statement. We would be glad to answer any questions you and the committee and subcommittee members may have.

Contacts and Acknowledgments

If you have any questions on matters discussed in this testimony, please contact Mathew Scirè at (202) 512-6806 or sciremj@gao.gov or David A. Powner at (202) 512-9286 or pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this testimony. Other key contributors to this testimony include Signora May, Assistant Director; Mathew Bader; Thomas Beall; Jeffrey DeMarco; Elizabeth Hosler; Richard Hung; Anne Inserra; Andrea Levine; Lisa Pearson; Sonya Phillips; Cynthia Scott; Niti Tandon; Jonathan Ticehurst; Timothy Wexler; and Katherine Wulff.

This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.