Information Technology: Census Bureau Testing of 2010 Decennial Systems Can Be Strengthened
Released: 2009-03-05T00:00:00

Letter

Background

The Bureau’s mission is to provide comprehensive data about the nation’s people and economy. Its core activities include conducting decennial, economic, and government censuses; conducting demographic and economic surveys; managing international demographic and socioeconomic databases; providing technical advisory services to foreign governments; and performing other activities such as producing official population estimates and projections.

Conducting the decennial census is a major undertaking that includes the following major activities: Establishing where to count. This includes identifying and correcting addresses for all known living quarters in the United States (address canvassing) and validating addresses identified as potential group quarters, such as college residence halls and group homes (group quarters validation).

Collecting and integrating respondent information. This includes delivering questionnaires to housing units by mail and other methods, processing the returned questionnaires, and following up with nonrespondents through personal interviews (nonresponse follow-up). It also includes enumerating residents of group quarters (group quarters enumeration) and occupied transitional living quarters (enumeration of transitory locations), such as recreational vehicle parks, campgrounds, and hotels. It also includes a final check of housing unit status (field verification) where Bureau workers verify potential duplicate housing units identified during response processing.

Providing census results. This includes processes to tabulate and summarize census data and disseminate the results to the public.

Figure 1 illustrates key decennial activities.

The 2010 census enumerates the number and location of people on Census Day, which is April 1, 2010. However, census operations begin long before Census Day and continue afterward. For example, address canvassing for the 2010 census will begin in April 2009, while tabulated census data must be distributed to the President by December 31, 2010, and to state legislatures by March 31, 2011. Figure 2 presents a timeline of key decennial operations.

Role of IT in the Decennial Census

Automation and IT are to play a critical role in the success of the 2010 census by supporting data collection, analysis, and dissemination. Several systems will be used in the 2010 census. For example, enumeration “universes,” which serve as the basis for enumeration operations and response data collection, are organized by the Universe Control and Management (UC&M) system, and response data are received and edited to help eliminate duplicate responses using the Response Processing System (RPS). Both UC&M and RPS are legacy systems that are collectively called the Headquarters Processing Systems.

Geographic information and support to aid the Bureau in establishing where to count U.S. citizens are provided by the Master Address File/Topologically Integrated Geographic Encoding and Referencing (MAF/TIGER) system. The Decennial Response Integration System (DRIS) is to provide a system for collecting and integrating census responses from all sources, including forms and telephone interviews. The Field Data Collection Automation (FDCA) program includes the development of handheld computers for the address canvassing operation and the systems, equipment, and infrastructure that field staff will use to collect the data. Paper-Based Operations (PBO) was established in August 2008, primarily to handle some of the operations that were originally part of FDCA. PBO includes IT systems and infrastructure needed to support the use of paper forms for operations such as group quarters enumeration activities, nonresponse follow-up activities, enumeration at transitory locations activities, and field verification activities. These activities were originally to be conducted using IT systems and infrastructure developed by the FDCA program. Finally, the Data Access and Dissemination System II (DADS II) is to replace legacy systems for tabulating and publicly disseminating data.

Table 1 describes the key systems supporting the 2010 census, as well as the offices responsible for their development.

Figure 3 shows the timeframes when each of the systems for the 2010 census are to be operational, according to the Census Bureau.

We Have Previously Reported on Weaknesses in Management of Census IT Systems

We have reported long-standing weaknesses in the Bureau’s management of its IT systems. For example, in October 2007, we reported on the status and plans of key 2010 census IT acquisitions and whether the Bureau was adequately managing associated risks. We identified critical weaknesses in the Bureau’s risk management practices, including those associated with risk identification, mitigation, and executive-level oversight. Further, operational testing planned during the census Dress Rehearsal would take place without the full complement of systems and functionality that was originally planned, the Bureau had not finalized its plans for testing all the systems, and it was unclear whether the plans would include testing to address all interrelated systems and functionality. We recommended that the Bureau develop a comprehensive plan to conduct an end-to-end test of its systems under census-like conditions.

In March 2008, we designated the 2010 census as a high-risk area, citing several long-standing and emerging challenges. These challenges included, among other things, weaknesses in risk management and system testing, elimination of several operations from the 2008 Dress Rehearsal, and questions surrounding the performance of handheld computers developed for the 2010 census. We have also testified on significant risks facing the 2010 census. For example, in March 2008, we testified that the FDCA program was experiencing significant problems, including schedule delays and cost increases resulting from changes to system requirements, which required additional work and staffing. Shortly thereafter, in April 2008, we testified on the Bureau’s efforts to implement risk reduction strategies, including the decision to drop the use of handheld computers during the nonresponse follow-up operation and revert to a paper-based operation. Further, in June 2008, we testified that the Bureau had taken important steps to plan for a paper-based nonresponse follow-up operation, but several aspects remained uncertain. We concluded that it was critical to test capabilities for supporting the nonresponse follow-up operation.

In July 2008, we reported that continued planning and testing of the handheld computers would be critical to the address canvassing operation. Specifically, the Bureau had developed a testing plan that included a limited operational field test, but the plan did not specify the basis for determining whether the FDCA solution was ready for address canvassing and when and how this determination would occur.

In response to our findings and recommendations, the Bureau has taken several steps to improve its management of the 2010 Decennial Census. For example, the Bureau has sought external assessments of its activities from independent research organizations, implemented a new management structure and management processes, brought in experienced personnel in key positions, and established improved reporting processes and metrics.

Comprehensive Testing Improves Chances of a Successful Decennial Census

As stated in our testing guide and the Institute of Electrical and Electronics Engineers (IEEE) standards, complete and thorough testing is essential for providing reasonable assurance that new or modified IT systems will perform as intended. To be effective, testing should be planned and conducted in a structured and disciplined fashion that includes processes to control each incremental level of testing, including testing of individual systems, the integration of those systems, and testing to address all interrelated systems and functionality in an operational environment.

System testing: verifies that the complete system (i.e., the full complement of application software running on the target hardware and systems software infrastructure) meets specified requirements. It allows for the identification and correction of potential problems within an individual system, prior to integration with other systems.

Integration testing: verifies that systems, when combined, work together as intended. Effective integration testing ensures that external interfaces work correctly and that the integrated systems meet specified requirements.

End-to-end testing: verifies that a defined set of interrelated systems, which collectively support an organization’s core business area or function, interoperate as intended in an operational environment. The interrelated systems include not only those owned and managed by the organization, but also the external systems with which they interface.

To be effective, this testing should be planned and scheduled in a structured and disciplined fashion. Comprehensive testing that is effectively planned and scheduled can provide the basis for identifying key tasks and requirements and better ensure that a system meets these specified requirements and functions as intended in an operational environment.

Dress Rehearsal Includes Testing of Certain Systems and Operations

In preparation for the 2010 census, the Bureau planned what it refers to as the Dress Rehearsal. The Dress Rehearsal is managed by the Bureau’s Decennial Management Division, in collaboration with other Bureau divisions (including the program offices, shown in table 1, which are responsible for developing and testing each of the systems). The Dress Rehearsal includes systems and integration testing, as well as end-to-end testing of key operations in a census-like environment. During the Dress Rehearsal period, running from February 2006 through June 2009, the Bureau is developing and testing systems and operations, and it held a mock Census Day on May 1, 2008. The Dress Rehearsal activities, which are still under way, are a subset of the activities planned for the actual 2010 census and include testing of both IT and non-IT related functions, such as opening offices and hiring staff.

Dress Rehearsal Testing of Key Systems and Activities Identified Problems with Technologies

The Dress Rehearsal tested several activities involving key systems. For example, the Bureau tested key systems with address canvassing and group quarters validation operations, including FDCA handheld computers and the MAF/TIGER system. In addition, the Bureau used the UC&M system and MAF/TIGER to provide an initial list of housing unit addresses for the Dress Rehearsal test sites. Questionnaires were mailed to these housing units in April 2008. Subsequently, a mock Census Day was held on May 1, 2008—1 month later than originally planned. The mock Census Day was delayed, in part, to focus greater attention on testing the technology being used.

The Dress Rehearsal identified significant technical problems during the address canvassing operations. For example, the Bureau had originally planned to use handheld computers, developed under the FDCA program, for operations such as address canvassing and non-response followup. However, from May 2007 to June 2007, the Bureau tested the handhelds under census-like conditions for the first time during the Dress Rehearsal address canvassing operation. Bureau officials observed a number of performance problems with the handheld computers, such as slow and inconsistent data transmissions. In addition, help desk logs revealed that users had frequently reported problems, such as the devices freezing up or users having difficulties collecting mapping coordinates and working with large blocks (geographic areas with large numbers of housing units, more often found in urban areas).

The Bureau also found system problems during testing of the group quarters validation operation, in which field staff validate addresses as group quarters and collect information required for their later enumeration. As part of this operation, the Bureau tested the operations control system—designed to manage field operations that rely on paper, as well as those that rely on the handheld computers—and the system was found to be unreliable. As a result, the workload for these operations had to be supplemented with additional paper-based efforts by local census office staff, instead of being performed electronically, as intended.

Results of Dress Rehearsal Testing Led to Decisions to Remove Testing of Certain Operations and to Revert to Some Paper-Based Processes for Key Operations

As a result of the problems observed with the handheld computers and operations control system, cost overruns and schedule slippage in the FDCA program, and other issues, the Bureau removed the planned testing of key operations from the Dress Rehearsal as follows: update/leave (that is, after enumerators update addresses, they leave questionnaires at housing units; this occurs mainly in rural areas lacking street names, house numbers, or both), enumeration of transitory locations, group quarters enumeration, and field verification.

Furthermore, in April 2008, the Secretary of Commerce announced a redesign of the 2010 Decennial Census, including the FDCA program. Specifically, the Bureau would no longer use handheld computers for nonresponse follow-up (its largest field operation), but would conduct paper-based nonresponse follow-up, as in previous censuses. It would, however, continue to use the handheld computers for the address canvassing operations. In May 2008, the Bureau issued a plan that detailed key components of the paper-based operation and described processes for managing it and other operations. It later established the PBO office to manage designing, developing, and testing paper-based operations, as well as to prepare related training materials.

Additional Testing Is Planned to Supplement the Dress Rehearsal

In addition to the planned Dress Rehearsal testing, the Bureau is planning supplementary testing to prepare for the 2010 Decennial Census. This testing includes system, integration, and end-to-end testing of changes resulting from the Dress Rehearsal, operations or features that were not tested during the Dress Rehearsal, and additional features or enhancements that are to be added after the Dress Rehearsal.

Bureau Is Making Progress in Conducting Key Decennial System Testing, but Lacks Plans and Schedules to Guide Remaining Efforts

The Bureau has made progress in conducting system, integration, and end- to-end testing for the 2010 census, but much remains to be done. Significant testing remains to be done, and many plans for the remaining testing activities have not been developed. The weaknesses in the Bureau’s IT testing can be attributed, in part, to a lack of sufficient executive-level oversight and guidance on testing. Without comprehensive oversight and guidance, the Bureau cannot ensure that it is thoroughly testing its systems before the 2010 Decennial Census.

Bureau Has Performed Many System Testing Activities, but Much Remains to be Done

Through the Dress Rehearsal and other testing activities, the Bureau has completed key system tests, but significant testing has yet to be performed, and planning for this is not complete. For example, the Headquarters Processing systems (UC&M and RPS) are still completing system testing related to the Dress Rehearsal, and the program office is planning for further testing. For DRIS, on the other hand, system testing related to the Dress Rehearsal is complete, and additional 2010 system testing is under way. Table 2 summarizes the status and plans for system testing.

For both Headquarters Processing Systems (UC&M and RPS), system testing for the Dress Rehearsal has been partially completed, as shown in table 3.

For UC&M, Dress Rehearsal system testing is divided into three phases, as shown. These phases include a total of 19 products (used to control and track Dress Rehearsal enumeration activities). The completed phases (1 and 2) included the development and testing of 14 products. Program officials had planned to complete testing of the remaining 5 products for UC&M by October 2008, but as of December 2008, the program had not yet completed this testing.

For RPS, Dress Rehearsal system testing is being done by component— eight components perform functions for key activities such as data integration—where response data are integrated before processing. According to program officials, development and testing of four components are complete, and the remaining four components are planned to be completed by March 2009.

In addition to ongoing Dress Rehearsal system testing, the program office intends to perform system testing for 2010 census operations, but plans for this testing have not yet been developed. According to program officials, they have not developed testing plans and schedules for additional testing for the 2010 census because Bureau management has not yet finalized the requirements for 2010 operations. Finalizing these requirements may involve both changes to existing requirements and new requirements. Program officials stated that they do not anticipate substantial changes in UC&M and RPS system requirements for 2010 census operations and plan to have them finalized by May 2009. In commenting on a draft of this report, the Bureau provided an initial test plan and schedule, but did not provide the finalized baselined requirements for the Headquarters Processing Systems for 2010 operations.

Until the baseline requirements are established, it is unclear whether the amount of additional testing necessary for 2010 census operations will be significant. According to industry best practices, defining requirements for a system is important because they provide a baseline for development and testing activities and are used to establish test plans, which define schedule activities, roles and responsibilities, resources, and system testing priorities. The absence of finalized requirements increases the risk that there may not be sufficient time and resources to adequately test the systems, which are critical to ensuring that address files are accurately organized into enumeration universes and that duplicate responses are eliminated.

MAF/TIGER Program Has Partially Completed Testing, but Test Plans and Schedules Are Incomplete and Ability to Track Progress Is Unclear

System testing has been partially completed for MAF/TIGER products (that is, extracts from the MAF/TIGER system) required for the 2010 census. For MAF/TIGER, testing activities are defined by products needed for key activities, such as address canvassing. During Dress Rehearsal system testing, the program office completed testing for a subset of MAF/TIGER products for address canvassing, group quarters validation, and other activities.

Additional system testing is planned for the 2010 census. According to program officials, as of December 2008, the Bureau had defined requirements and completed testing for 6 of approximately 60 products needed for 2010 operations (these 6 products are related to address canvassing). The program office has also developed detailed test plans and schedules through April 2009, but these do not cover all of the remaining products needed to support the 2010 census. Table 4 is a summary of the status of MAF/TIGER 2010 testing and plans.

According to program officials, the detailed test plans for the remaining products will be developed after the requirements for each are finalized. As mentioned, establishing defined requirements is important because these provide a baseline for development and testing activities and define the basic functions of a product. The officials stated that they were estimating the number of products needed, but would only know the exact number when the requirements for the 2010 census operations are determined. The officials added that, by January 2009, they plan to have a detailed schedule and a list of the products needed through December 2009.

Without knowing the total number of products, related requirements, and when the products are needed for operations, the Bureau risks both not sufficiently defining what each product needs to do and not being able to effectively measure the progress of MAF/TIGER testing activities, which therefore increases the risk that there may not be sufficient time and resources to adequately test the system and that the system may not perform as intended.

DRIS Testing Is Under Way; Plans Have Been Established but Will Be Revised

System testing has been partially completed for DRIS components, including paper, workflow control and management, and telephony. Portions of the functionality in each DRIS component are being developed and tested across five increments for 2010 operations. As of November 2008, the program had planned and completed testing of increments 1, 2, and 3. Testing of increment 4 is currently ongoing. Table 5 is a summary of the status of DRIS testing for 2010 operations. (In addition, system testing of a subset of DRIS functionality, including the integration of certain response data, took place during the Dress Rehearsal.)

The DRIS program has developed a detailed testing plan and schedule, including the remaining testing for increment 5. For example, detailed testing plans have been developed for all 558 functional requirements for DRIS. According to program officials, most of the 558 functional requirements will be fully tested during increments 4 and 5. As of November 2008, 22 of the 558 requirements had been tested.

Although plans and schedules were completed, the change from handheld computers to paper processes for nonresponse follow-up has caused changes to DRIS processing requirements. For example, DRIS program officials stated that they now need to process an additional 40 million paper forms generated as a result of this switch. Although DRIS program officials stated that they are prepared to adjust their test schedule and plan to accommodate this change, they cannot do so until the requirements have been finalized for the switch to paper processes. (This responsibility is primarily that of the PBO program office.) Furthermore, based on the switch to paper, DRIS may not be able to conduct a test using these operational systems and live data. This increases the risk that the Bureau could experience problems with these systems and the processing of paper forms during the 2010 census. The DRIS program office is addressing this risk by developing alternative strategies for testing and providing additional resources as contingencies for activities that may not be fully tested before 2010 operations begin.

FDCA System Testing Is Proceeding on an Aggressive Schedule, but Key Test Plans Have Not Been Developed

FDCA testing has been partially completed, but much more work remains. System testing for FDCA took place during the Dress Rehearsal, but problems encountered during this testing led to the removal of key operations from the Dress Rehearsal and the April 2008 redesign, as described earlier. Going forward, FDCA development and testing for 2010 operations are being organized based on key census activities. For example, FDCA testing for the address canvassing and group quarters validation operations was completed in December 2008. The FDCA contractor is currently developing and testing a backup system (known as the continuity of operations system) for address canvassing and group quarters validation, and is currently testing another system (known as map printing) to provide printed maps for paper-based field operations, such as nonresponse follow-up. Table 6 summarizes the FDCA test status.

Although system testing for address canvassing and group quarters validation was recently completed, program officials have not demonstrated that all current system requirements have been fully tested. As part of a contract revision required by the April 2008 redesign, the FDCA program established a revised baseline for system requirements on November 20, 2008. According to program officials, this revision included both modifications to previous requirements and removal of requirements that were part of activities transferred to PBO. As of December 2008, program officials stated that detailed testing plans for many of the requirements exist, but need to be revised to address the newly baselined requirements.

Furthermore, as of December 2008, the FDCA program had not finalized detailed testing plans and schedules for the continuity of operations and map printing systems. According to program officials, they had not yet developed detailed testing plans and schedules for testing systems’ requirements because their focus has been on testing the system for address canvassing and group quarters validation. Officials added that they plan to begin testing the requirements for the continuity of operations system in January 2009, and for the map printing system in February 2009. However, without having established testing plans and schedules for these systems, it is unclear what amount of testing will be needed and whether sufficient time has been allocated in the schedule to fully test these systems before they are needed for operations.

Paper-Based Operations Are in Early Development, but Detailed 2010 Testing Plans and Schedules Have Not Yet Been Developed

Testing has only recently started for PBO because it is still in the preliminary phase of program planning and initial system development as the Bureau shifts responsibility for certain operations from the FDCA program office to PBO. Because this office has only recently been created, it is currently hiring staff, developing a schedule for several iterations of development, establishing a means to store and trace requirements, developing testing plans, and establishing a configuration management process.

According to program officials, development will occur in five releases, numbered 0 through 4. The first release, Release 0, is planned to contain functionality for the nonresponse follow-up and group quarters enumeration activities. Table 7 provides the current status of PBO Release 0 test activities.

However, the Bureau still has not yet determined when detailed testing plans and schedules for PBO systems will be developed. Officials stated that a more detailed schedule for Release 0 and development schedules for the remaining releases are under development, and that they plan to have the majority of the schedules developed by the end of January 2009. In commenting on a draft of this report, the Bureau provided a partial schedule for PBO test activities.

Furthermore, officials stated they had not yet fully defined which requirements PBO would be accountable for, and which of these requirements will be addressed in each iteration of development. The officials did state that the requirements will be based on those requirements transferred from FDCA as part of the reorganization. Bureau officials stated they had not yet completed these activities because responsibility for the requirements was only formally transferred as of October 2008. The program office expects to have its first iteration of requirements traceable to test cases by March 2009. However, officials did not know what percentage of program requirements will be included in this first iteration.

Although progress has been made in establishing the PBO program office, numerous critical system development activities need to be planned and executed in a limited amount of time. Because of the compressed schedule and the large amount of planning that remains, PBO risks not having its systems developed and tested in time for the 2010 Decennial Census. Testing is critical to ensure that the paper forms used to enumerate residents of households who do not mail back their questionnaires, group quarters, and transitional living quarters are processed accurately.

DADS II Is in Early Stages of Development, and Testing Plans and Schedules Are Being Developed

The DADS system (which had been used in the 2000 census) is currently being tested during the Dress Rehearsal, which is scheduled to be completed in March 2009. However, the Bureau intends to replace DADS with DADS II, which is currently being developed and tested for 2010 operations. DADS II is still in the early part of its life cycle, and the program office has only recently started system testing activities. The two main DADS II components, the Replacement Tabulation System (RTS) and Replacement Dissemination System (RDS), are being developed and tested across a series of three iterations. As of December 2008, the program had begun iterations 1 and 2 for RTS, and iteration 1 for RDS. Table 8 summarizes the RTS and RDS testing status.

The DADS II program office has developed a high-level test plan for RTS and RDS system testing, but has not yet defined detailed testing plans and a schedule for testing system requirements. System requirements for the new system have been baselined, with 202 requirements for RTS and 318 requirements for RDS. According to program officials, the program office is planning to develop detailed testing plans for the system requirements for both RTS and RDS.

Bureau Has Conducted Limited Integration Testing, but Has Not Developed 2010 Test Plans and Schedules for Integration Testing

Effective integration testing ensures that external interfaces work correctly and that the integrated systems meet specified requirements. This testing should be planned and scheduled in a disciplined fashion according to defined priorities.

For the 2010 census, each program office is responsible for and has made progress in defining system interfaces and conducting integration testing, which includes testing of these interfaces. However, significant activities remain in order for comprehensive integration testing to be completed by the date that the systems are needed for 2010 operations. For example, DRIS has conducted integration testing with some systems, such as FDCA, UC&M, and RPS, and is scheduled to complete integration testing by February 2010. The FDCA program office has also tested interfaces related to the address canvassing operation scheduled to begin in April 2009. However, for many other systems, such as PBO, interfaces have not been fully defined, and other interfaces have been defined but have not been tested. Table 9 provides the status of integration testing among key systems.

In addition, the Bureau has not established a master list of interfaces between key systems, or plans and schedules for integration testing of these interfaces. A master list of system interfaces is an important tool for ensuring that all interfaces are tested appropriately and that the priorities for testing are set correctly. Although the Bureau had established a list of interfaces in 2007, according to Bureau officials, it was not updated because of resource limitations at the time and other management priorities. As of October 2008, the Bureau had begun efforts to update this list, but it has not provided a date when this list will be completed.

Without a completed master list, the Bureau cannot develop comprehensive plans and schedules for conducting systems integration testing that indicate how the testing of these interfaces will be prioritized. This is important because a prioritized master list of system interfaces, combined with comprehensive plans and schedules to test the interfaces, would allow for tracking the progress of this testing. With the limited amount of time remaining before systems are needed for 2010 operations, the lack of comprehensive plans and schedules increases the risk that the Bureau may not be able to adequately test system interfaces, and that interfaced systems may not work together as intended.

Bureau Has Conducted Limited End-to-End Testing as Part of the Dress Rehearsal, but Has Not Developed Testing Plans for Critical Operations

The Dress Rehearsal was originally conceived to provide a comprehensive end-to-end test of key 2010 census operations; however, as mentioned earlier, because of the problems encountered with the handheld devices, among other things, testing was curtailed. As a result, although several critical operations underwent end-to-end testing in the Dress Rehearsal, others did not. According to the Associate Director for the 2010 census, the Bureau tested approximately 23 of 44 key operations during the Dress Rehearsal. Examples of key operations that underwent end-to-end testing during the Dress Rehearsal are address canvassing and group quarters validation. An example of a key operation that was not tested is the largest field operation—nonresponse follow-up.

Although the Bureau recently conducted additional testing of the handhelds, this test was not a robust end-to-end test. In December 2008, after additional development and improvements to the handheld computers, the Bureau conducted a limited field test for address canvassing, intended to assess software functionality in an operational environment. We observed this test and determined that users were generally satisfied with the performance of the handhelds. According to Bureau officials, the performance of the handheld computers has substantially improved from previous tests. However, the test was not designed to test all the functionality of the handhelds in a robust end-to-end test—rather, it included only a limited subset of functionality to be used during the 2009 address canvassing operations. Further, the field test did not validate that the FDCA system fully met specified requirements for the address canvassing operation. Bureau officials stated that additional testing of the FDCA system, such as performance testing, mitigated the limitations of this field test.

Nonetheless, the lack of robustness of the field test poses several risks for 2010 operations. Specifically, without testing all the FDCA system’s requirements in a robust operational environment, it is unclear whether the system can perform as intended when the address canvassing operation begins in April 2009.

Furthermore, as of December 2008, the Bureau has neither established testing plans nor schedules to perform end-to-end testing of the key operations that were removed from the Dress Rehearsal, nor has it determined when these plans will be completed. As previously mentioned, these operations include enumeration of transitory locations, group quarters enumeration, and field verification.

Although the Bureau has established a high-level strategy for testing these operations, which provides details about the operations to be tested, Bureau officials stated that they have not developed testing plans and schedules because they are giving priority to tests for operations that are needed in early 2009. In addition, key systems needed to test these operations are not ready to be tested because they are either still in development or have not completed system testing. Until system and integration testing activities are complete, the Bureau cannot effectively plan and schedule end-to-end testing activities. Without sufficient end-to- end testing, operational problems can go undiscovered, and the opportunity to improve these operations will be lost.

The decreasing time available for completing end-to-end testing increases the risk that testing of key operations will not take place before the required deadline. Bureau officials have acknowledged this risk in briefings to the Office of Management and Budget. The Bureau is in the process of identifying risks associated with incomplete testing and developing mitigation plans, which it had planned to have completed by November 2008. However, as of January 2009, the Bureau had not completed these mitigation plans. According to the Bureau, the plans are still being reviewed by senior management. Without plans to mitigate the risks associated with limited end-to-end testing, the Bureau may not be able to respond effectively if systems do not perform as intended.

Bureau Lacks Sufficient Executive-Level Oversight and Guidance for Testing

As stated in our testing guide and IEEE standards, oversight of testing activities includes both planning and ongoing monitoring of testing activities. Ongoing monitoring entails collecting and assessing status and progress reports to determine, for example, whether specific test activities are on schedule. Using this information, management can effectively determine whether corrective action is needed and, if so, what action should be taken. In addition, comprehensive guidance should describe each level of testing (for example, system, integration, or end-to-end), criteria for each level, and the type of test products expected. The guidance should also address test preparation and oversight activities.

Although the 2010 Decennial Census is managed by the Decennial Management Division, the oversight and management of key census IT systems is performed on a decentralized basis. DRIS, FDCA, and DADS II each have a separate program office within the Decennial Automation Contracts Management Office; Headquarters Processing and PBO are managed within the Decennial System and Processing Office; and MAF/TIGER is managed within the Geography Division. Each program has its own program management reviews and develops plans and tracks metrics related to testing. These offices and divisions collectively report to the Associate Director for the 2010 Census. According to the Bureau, the associate director chairs biweekly meetings where the officials responsible for these systems meet to review the status of key systems development and testing efforts.

In addition, in response to prior recommendations, the Bureau took initial steps to enhance its programwide oversight; however, these steps have not been sufficient to establish adequate executive-level oversight. In June 2008, the Bureau established an inventory of all testing activities specific to all key decennial operations. This inventory showed that, as of May 2008, 18 percent of about 1049 system testing activities had not been planned. (See fig. 4.) In addition, approximately 67 percent of about 836 operational testing activities had not been planned.

Although officials from the Decennial System and Processing Office described the inventory effort as a means of improving executive-level oversight, the inventory has not been updated since May 2008, and officials have no plans for further updates. Instead, officials stated that they plan to track testing progress as part of the Bureau’s detailed master schedule of census activities. However, this schedule does not provide comprehensive status information on testing.

In another effort to improve executive-level oversight, the Decennial Management Division began producing (as of July 2008) a weekly executive alert report and has established (as of October 2008) monthly dashboard and reporting indicators. However, these products do not provide comprehensive status information on the testing progress of key systems and interfaces. For example, the executive alert report does not include the progress of testing activities, and although the dashboard provides a high-level, qualitative assessment of testing for key operations and selected systems, it does not provide information on the testing progress of all key systems and interfaces.

Further, the assessment of testing progress has not been based on quantitative and specific metrics. For example, the status of testing key operations removed from the Dress Rehearsal was marked as acceptable, or “green,” although the Bureau does not yet have plans for testing these activities. Bureau officials stated that they marked these activities as acceptable because, based on past experience, they felt comfortable that a plan would be developed in time to adequately test these operations. The lack of quantitative and specific metrics to track progress limits the Bureau’s ability to accurately assess the status and progress of testing activities. In commenting on a draft of this report, the Bureau provided selected examples in which they had begun to use more detailed metrics to track the progress of end-to-end testing activities.

Finally, although the Bureau announced in August 2008 that it was planning to hire a senior manager who would have primary responsibility for monitoring testing across all decennial systems and programs, the position had not been filled as of January 2009. Instead, agency officials stated that the role is being filled by another manager from the Decennial Statistical Studies Division, who has numerous other responsibilities.

The Bureau also has weaknesses in its testing guidance; it has not established comprehensive guidance for system testing. According to the Associate Director for the 2010 Census, the Bureau did establish a policy strongly encouraging offices responsible for decennial systems to use best practices in software development and testing, as specified in level 2 of Carnegie Mellon’s Capability Maturity Model® Integration. However, beyond this general guidance, there is no additional guidance on key testing activities such as criteria for each level or the type of test products expected. Standardized policies and procedures help to ensure comprehensive processes across an organization and allow for effective executive-level oversight. The lack of guidance has led to an ad hoc—and, at times—less than desirable approach to testing.

Conclusions

While the Bureau’s program offices have made progress in testing key decennial systems, much work remains to ensure that systems operate as intended for conducting an accurate and timely 2010 census. Several program offices have yet to prepare and execute system test plans and schedules and ensure that system requirements are fully tested. In addition, the Bureau has not developed a master list of interfaces, which is necessary to prioritize testing and to develop comprehensive integration test plans and schedules. Additionally, end-to-end testing plans for key operations have not been finalized or executed based on established priorities to help ensure that systems will support census operations.

Weaknesses in the Bureau’s IT testing can be attributed, in part, to a lack of sufficient executive-level oversight and guidance. More detailed metrics and status reports would help the Bureau to better monitor testing progress and identify and address problems. Giving accountability for testing to a senior-level official would also provide the focus and attention needed to complete critical testing. Also, completing risk mitigation plans will help ensure that actions are in place to address potential problems with systems. Given the rapidly approaching deadlines of the 2010 census, completing these important tests and establishing stronger executive-level oversight and guidance are critical to ensuring that systems perform as intended when they are needed.

Recommendations for Executive Action

To ensure that testing activities for key systems for the 2010 census are completed, we are making 10 recommendations. We recommend that the Secretary of Commerce require the Director of the Census Bureau to expeditiously implement the following recommendations: For the Headquarters UC&M and RPS, finalize requirements for 2010 census operations and complete testing plans and schedules for 2010 operations that trace to baselined system requirements.

For MAF/TIGER, establish the number of products required, define related requirements, and establish a testing plan and schedule for 2010 operations.

For FDCA, establish testing plans for the continuity of operations and map printing systems that trace to baselined system requirements.

For PBO, develop baseline requirements and complete testing plans and schedules for 2010 operations.

Establish a master list of system interfaces; prioritize the list, based on system criticality and need date; define all interfaces; and develop integration testing plans and schedules for tracking the progress of testing these interfaces.

Establish a date for completing testing plans for the operations removed from the Dress Rehearsal operations and prioritize testing activities for these operations.

Finalize risk mitigation plans detailing actions to address system problems that are identified during testing.

Establish specific testing metrics and detailed status reports to monitor testing progress and better determine whether corrective action is needed for all key testing activities.

Designate a senior manager with primary responsibility for monitoring testing and overseeing testing across the Bureau.

In addition, after the 2010 census, we recommend that the Bureau establish comprehensive systems and integration testing guidance to guide future testing of systems.

Agency Comments and Our Evaluation

The Associate Under Secretary for Management of the Department of Commerce provided written comments on a draft of this report. The department’s letter and general comments are reprinted in appendix II.

In the comments, the department and Bureau stated they had no significant disagreements with our recommendations. However, the department and Bureau added that since the FDCA replan last year, their testing strategy has been to focus on those things they have not done before, and to demonstrate to their own satisfaction that new software and systems will work in production. The department added that it has successfully conducted Census operations before, and was focusing “on testing the new things for 2010—not things that have worked before.”

While we acknowledge that the Bureau has conducted key census operations before, the systems and infrastructure in place to conduct these operations have changed substantially since the 2000 census. For example, while the Bureau has conducted paper-based nonresponse followup during previous censuses, it will be using newly developed systems which have not yet been fully tested in a census-like environment to integrate responses and manage the nonresponse followup work load. In addition, new procedures, such as one to remove questionnaires that were mailed in late from the nonresponse followup operation, have not been tested with these systems. Any significant change to an existing IT system introduces the risk that the system may not work as intended; therefore, testing all systems after changes have been made to ensure the systems work as intended is critical to the success of the 2010 census.

In addition, the department and Bureau provided technical comments, such as noting draft plans that had been developed after the conclusion of our work, that we have incorporated where appropriate.

We are sending copies of this report to the Secretary of Commerce, the Director of the U.S. Census Bureau, and other appropriate congressional committees. The report also is available at no charge on the GAO Web site at http://www.gao.gov. If you have any questions about this report, please contact David Powner at (202) 512-9286 or pownerd@gao.gov. GAO staff who made contributions to this report are listed in appendix III. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report.

Appendix I: Scope and Methodology

To determine the status of and plans for testing key decennial systems, we analyzed documentation related to system, integration, and end-to-end testing. For system testing, we analyzed documentation related to each key decennial system, including system test plans, schedules, requirements, results, and other test-related documents. We then compared the Bureau’s practices with those identified in our testing guide and Institute of Electrical and Electronics Engineers (IEEE) standards to determine the extent to which the Bureau had incorporated best practices in testing. We also interviewed program officials and contractors of key decennial systems to obtain information on the current status of and plans for testing activities.

For integration testing, we analyzed interface control documents, interface testing plans, and schedules. We also analyzed documentation of the Census Bureau’s (Bureau) oversight of integration testing activities, including efforts to issue integration testing guidance and monitor the progress of integration testing activities. We interviewed program officials at each key decennial system program office to obtain information on the current status of and plans for integration testing and interviewed program officials at the Decennial Systems Processing Office to obtain information on the executive-level oversight of integration testing activities. We compared the Bureau’s practices with those identified in our testing guide and IEEE guidance.

For end-to-end testing, we analyzed documentation related to the testing of key census operations during the Bureau’s Dress Rehearsal, additional testing conducted for the address canvassing operation, and efforts to establish testing plans and schedules for operations removed from the Dress Rehearsal. We also observed the Bureau’s operational field test, held in December 2008 in Fayetteville, North Carolina. We interviewed program officials at the Decennial Systems Processing Office to obtain information on the current status and plans for end-to-end testing activities. We compared the Bureau’s practices with those identified in our testing guide and IEEE guidance.

We also analyzed documentation of the Bureau’s overall oversight of testing, including its executive alert reports and monthly dashboard reports. In addition, we assessed system testing guidance and interviewed the Associate Director for the Decennial Census to obtain information on the overall oversight of testing activities.

We conducted this performance audit from June 2008 to February 2009 in the Washington, D.C., and Fayetteville, North Carolina, areas, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objective. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objective.

Appendix II: Comments from the Department of Commerce

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact name above, individuals making contributions to this report included Cynthia Scott (Assistant Director), Sher`rie Bacon, Barbara Collier, Neil Doherty, Vijay D’Souza, Nancy Glover, Lee McCracken, Jonathan Ticehurst, Melissa Schermerhorn, and Karl Seifert.