Information Technology: Census Bureau Needs to Strengthen Testing of 2010 Decennial Systems
Released: 2009-03-05T00:00:00

Letter

Background

The Bureau’s mission is to provide comprehensive data about the nation’s people and economy. The 2010 census enumerates the number and location of people on Census Day, which is April 1, 2010. However, census operations begin long before Census Day and continue afterward. For example, address canvassing for the 2010 census will begin in April 2009, while the Secretary of Commerce must report tabulated census data to the President by December 31, 2010, and to state governors and legislatures by March 31, 2011.

The decennial census is a major undertaking for the Bureau that includes the following major activities: Establishing where to count. This includes identifying and correcting addresses for all known living quarters in the United States (address canvassing) and validating addresses identified as potential group quarters, such as college residence halls and group homes (group quarters validation).

Collecting and integrating respondent information. This includes delivering questionnaires to housing units by mail and other methods, processing the returned questionnaires, and following up with nonrespondents through personal interviews (nonresponse follow-up). It also includes enumerating residents of group quarters (group quarters enumeration) and occupied transitional living quarters (enumeration of transitory locations), such as recreational vehicle parks, campgrounds, and hotels. It also includes a final check of housing unit status (field verification) where Bureau workers verify potential duplicate housing units identified during response processing.

Providing census results. This includes tabulating and summarizing census data and disseminating the results to the public.

Role of IT in the Decennial Census

Automation and IT are to play a critical role in the success of the 2010 census by supporting data collection, analysis, and dissemination. Several systems will play a key role in the 2010 census. For example, enumeration “universes,” which serve as the basis for enumeration operations and response data collection, are organized by the Universe Control and Management (UC&M) system, and response data are received and edited to help eliminate duplicate responses using the Response Processing System (RPS). Both UC&M and RPS are legacy systems that are collectively called the Headquarters Processing System.

Geographic information and support to aid the Bureau in establishing where to count U.S. citizens are provided by the Master Address File/Topologically Integrated Geographic Encoding and Referencing (MAF/TIGER) system. The Decennial Response Integration System (DRIS) is to provide a system for collecting and integrating census responses from all sources, including forms and telephone interviews. The Field Data Collection Automation (FDCA) program includes the development of handheld computers for the address canvassing operation and the systems, equipment, and infrastructure that field staff will use to collect data. Paper-Based Operations (PBO) was established in August 2008 primarily to handle certain operations that were originally part of FDCA. PBO includes IT systems and infrastructure needed to support the use of paper forms for operations such as group quarters enumeration activities, nonresponse follow-up activities, enumeration at transitory locations activities, and field verification activities. These activities were originally to be conducted using IT systems and infrastructure developed by the FDCA program. Finally, the Data Access and Dissemination System II (DADS II) is to replace legacy systems for tabulating and publicly disseminating data.

Comprehensive Testing Improves Chances of a Successful Decennial Census

As stated in our testing guide and the Institute of Electrical and Electronics Engineers (IEEE) standards, complete and thorough testing is essential for providing reasonable assurance that new or modified IT systems will perform as intended. To be effective, testing should be planned and conducted in a structured and disciplined fashion that includes processes to control each incremental level of testing, including testing of individual systems, the integration of those systems, and testing to address all interrelated systems and functionality in an operational environment.

Further, this testing should be planned and scheduled in a structured and disciplined fashion. Comprehensive testing that is effectively planned and scheduled can provide the basis for identifying key tasks and requirements and better ensure that a system meets these specified requirements and functions as intended in an operational environment.

Dress Rehearsal Includes Testing of Certain Systems and Operations

In preparation for the 2010 census, the Bureau planned what it refers to as the Dress Rehearsal. The Dress Rehearsal includes systems and integration testing, as well as end-to-end testing of key operations in a census-like environment. During the Dress Rehearsal period, running from February 2006 through June 2009, the Bureau is developing and testing systems and operations, and it held a mock Census Day on May 1, 2008. The Dress Rehearsal activities, which are still under way, are a subset of the activities planned for the actual 2010 census and include testing of both IT and non-IT related functions, such as opening offices and hiring staff.

The Dress Rehearsal identified significant technical problems during the address canvassing and group quarters validation operations. For example, during the Dress Rehearsal address canvassing operation, the Bureau encountered problems with the handheld computers, including slow and inconsistent data transmissions, the devices freezing up, and difficulties collecting mapping coordinates. As a result of the problems observed during the Dress Rehearsal, cost overruns and schedule slippage in the FDCA program, and other issues, the Bureau removed the planned testing of several key operations from the Dress Rehearsal and switched key operations, such as nonresponse follow-up, to paper-based processes instead of using the handheld computers as originally planned.

Bureau Is Making Progress in Key System Testing, but Lacks Plans and Schedules

Through the Dress Rehearsal and other testing activities, the Bureau has completed key system tests, but significant testing has yet to be done, and planning for this is not complete. Table 1 summarizes the status and plans for system testing.

Bureau Has Conducted Limited Integration Testing, but Has Not Developed 2010 Test Plans and Schedules for Integration Testing

Effective integration testing ensures that external interfaces work correctly and that the integrated systems meet specified requirements. This testing should be planned and scheduled in a disciplined fashion according to defined priorities.

For the 2010 census, each program office is responsible for and has made progress in defining system interfaces and conducting integration testing, which includes testing of these interfaces. However, significant activities remain to be completed. For example, for systems such as PBO, interfaces have not been fully defined, and other interfaces have been defined but have not been tested. In addition, the Bureau has not established a master list of interfaces between key systems, or plans and schedules for integration testing of these interfaces. A master list of system interfaces is an important tool for ensuring that all interfaces are tested appropriately and that the priorities for testing are set correctly. As of October 2008, the Bureau had begun efforts to update a master list it had developed in 2007, but it has not provided a date when this list will be completed.

Without a completed master list, the Bureau cannot develop comprehensive plans and schedules for conducting systems integration testing that indicate how the testing of these interfaces will be prioritized. With the limited amount of time remaining before systems are needed for 2010 operations, the lack of comprehensive plans and schedules increases the risk that the Bureau may not be able to adequately test system interfaces, and that interfaced systems may not work together as intended.

Bureau Has Conducted Limited End-to-End Testing as Part of the Dress Rehearsal, but Has Not Developed Testing Plans for Critical Operations

Although several critical operations underwent end-to-end testing in the Dress Rehearsal, others did not. As of December 2008, the Bureau had not established testing plans or schedules for end-to-end testing of the key operations that were removed from the Dress Rehearsal, nor has it determined when these plans will be completed. These operations include enumeration of transitory locations, group quarters enumeration, and field verification.

The decreasing time available for completing end-to-end testing increases the risk that testing of key operations will not take place before the required deadline. Bureau officials have acknowledged this risk in briefings to the Office of Management and Budget. However, as of January 2009, the Bureau had not completed mitigation plans for this risk. According to the Bureau, the plans are still being reviewed by senior management. Without plans to mitigate the risks associated with limited end-to-end testing, the Bureau may not be able to respond effectively if systems do not perform as intended.

Bureau Lacks Sufficient Executive-Level Oversight and Guidance for Testing

As stated in our testing guide and IEEE standards, oversight of testing activities includes both planning and ongoing monitoring of testing activities. Ongoing monitoring entails collecting and assessing status and progress reports to determine, for example, whether specific test activities are on schedule. In addition, comprehensive guidance should describe each level of testing and the types of test products expected.

In response to prior recommendations, the Bureau took initial steps to enhance its programwide oversight; however, these steps have not been sufficient. For example, in June 2008, the Bureau established an inventory of all testing activities specific to all key decennial operations. However, the inventory has not been updated since May 2008, and officials have no plans for further updates.

In another effort to improve executive-level oversight, the Decennial Management Division began producing (as of July 2008) a weekly executive alert report and has established (as of October 2008) a dashboard and monthly reporting indicators. However, these products do not provide comprehensive status information on the progress of testing key systems and interfaces. Further, the assessment of testing progress has not been based on quantitative and specific metrics. The lack of quantitative and specific metrics to track progress limits the Bureau’s ability to accurately assess the status and progress of testing activities. In commenting on our draft report, the Bureau provided selected examples where they had begun to use more detailed metrics to track the progress of end-to-end testing activities.

The Bureau also has weaknesses in its testing guidance. According to the Associate Director for the 2010 census, the Bureau did establish a policy strongly encouraging offices responsible for decennial systems to use best practices in software development and testing, as specified in level 2 of Carnegie Mellon’s Capability Maturity Model® Integration. However, beyond this general guidance, there is no mandatory or specific guidance on key testing activities such as criteria for each level or the type of test products expected. The lack of guidance has led to an ad hoc—and, at times—less than desirable approach to testing.

Implementation of Recommendations Could Help Ensure Key Testing Activities are Completed

In our report, we are making ten recommendations for improvements to the Bureau’s testing activities. Our recommendations include finalizing system requirements and completing development of test plans and schedules, establishing a master list of system interfaces, prioritizing and developing plans to test these interfaces, and establishing plans to test operations removed from the Dress Rehearsal. In addition, we are recommending that the Bureau improve its monitoring of testing progress and improve executive-level oversight of testing activities.

In written comments on the report, the department had no significant disagreements with our recommendations. The department stated that its focus is on testing new software and systems, not legacy systems and operations used in previous censuses. However, the systems in place to conduct these operations have changed substantially and have not yet been fully tested in a census-like environment. Consistent with our recommendations, finalizing test plans and schedules and testing all systems as thoroughly as possible will help to ensure that decennial systems will work as intended.

In summary, while the Bureau’s program offices have made progress in testing key decennial systems, much work remains to ensure that systems operate as intended for conducting an accurate and timely 2010 census. This work includes system, integration, and end-to-end testing activities. Given the rapidly approaching deadlines of the 2010 census, completing testing and establishing stronger executive-level oversight are critical to ensuring that systems perform as intended when they are needed.

Mr. Chairman and members of the subcommittee, this concludes our statement. We would be pleased to respond to any questions that you or other members of the subcommittee may have at this time.

Contacts and Staff Acknowledgements

If you have any questions about matters discussed in this testimony, please contact David A. Powner at (202) 512-9286 or pownerd@gao.gov or Robert Goldenkoff at (202) 512-2757 or goldenkoffr@gao.gov. Other key contributors to this testimony include Sher`rie Bacon, Barbara Collier, Neil Doherty, Vijay D’Souza, Elizabeth Fan, Nancy Glover, Signora May, Lee McCracken, Ty Mitchell, Lisa Pearson, Crystal Robinson, Melissa Schermerhorn, Cynthia Scott, Karl Seifert, Jonathan Ticehurst, Timothy Wexler, and Katherine Wulff.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.