Information Technology: Census Bureau Needs to Strengthen Testing of 2010 Decennial Systems
Published: 2009-03-05T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Decennial Census is mandated by the U.S. Constitution and provides vital data that are used, among other things, to reapportion and redistrict congressional seats and allocate federal financial assistance. In March 2008, GAO designated the 2010 Decennial Census a high-risk area, citing a number of long-standing and emerging challenges, including weaknesses in the U.S. Census Bureau's (Bureau) management of its information technology (IT) systems and operations. In conducting the 2010 census, the Bureau is relying on both the acquisition of new IT systems and the enhancement of existing systems. Thoroughly testing these systems before their actual use is critical to the success of the census. GAO was asked to testify on its report, being released today, on the status and plans of testing of key 2010 decennial IT systems.
section_title: What GAO Found
paragraphs: Although the Bureau has made progress in testing key decennial systems, critical testing activities remain to be performed before systems will be ready to support the 2010 census. Bureau program offices have completed some testing of individual systems, but significant work still remains to be done, and many plans have not yet been developed (see table below). In its testing of system integration, the Bureau has not completed critical activities; it also lacks a master list of interfaces between systems and has not developed testing plans and schedules. Although the Bureau had originally planned what it refers to as a Dress Rehearsal, starting in 2006, to serve as a comprehensive end-to-end test of key operations and systems, significant problems were identified during testing. As a result, several key operations were removed from the Dress Rehearsal and did not undergo end-to-end testing. The Bureau has neither developed testing plans for these key operations, nor has it determined when such plans will be completed. Weaknesses in the Bureau's testing progress and plans can be attributed in part to a lack of sufficient executive-level oversight and guidance. Bureau management does provide oversight of system testing activities, but the oversight activities are not sufficient. For example, Bureau reports do not provide comprehensive status information on progress in testing key systems and interfaces, and assessments of the overall status of testing for key operations are not based on quantitative metrics. Further, although the Bureau has issued general testing guidance, it is neither mandatory nor specific enough to ensure consistency in conducting system testing. Without adequate oversight and more comprehensive guidance, the Bureau cannot ensure that it is thoroughly testing its systems and properly prioritizing testing activities before the 2010 Decennial Census, posing the risk that these systems may not perform as planned.

section_title: Letter
paragraphs: Thank you for the opportunity to participate in today’s hearing on the 2010  census. The U.S. Census Bureau (Bureau) is relying on both the  acquisition of new systems and the enhancement of existing legacy  systems for conducting operations for the 2010 Decennial Census. As you  know, the census is mandated by the U.S. Constitution and provides data  that are vital to the nation. These data are used, for example, to  reapportion and redistrict the seats of the U.S. House of Representatives,  realign the boundaries of the legislative districts of each state, and allocate  federal financial assistance. Carrying out the census is the responsibility of  the Department of Commerce’s Census Bureau, which is relying on  automation and technology to improve the coverage, accuracy, and  efficiency of the 2010 census. Because the accuracy of the 2010 census  depends in part on the proper functioning of these systems, both  individually and when integrated, thorough testing of these systems before  their actual use is critical to the success of the census.
As you know, in March 2008, we designated the 2010 Decennial Census as  a high-risk area, citing a number of long-standing and emerging  challenges, including weaknesses in the Bureau’s management of its  information technology (IT) systems and operations. The 2010 Decennial  Census remained as one of our high-risk areas in our recent high-risk  update issued in January 2009. This statement summarizes the findings in  our report, being released by the subcommittee today, on the status and  plans of testing of key 2010 decennial IT systems.
Our work for this report was performed in accordance with generally  accepted government auditing standards. Those standards require that we  plan and perform the audit to obtain sufficient, appropriate evidence to  provide a reasonable basis for our findings and conclusions based on our  audit objective. We believe that the evidence obtained provides a  reasonable basis for our findings and conclusions based on our audit  objective.
subsections: section_title: Background
paragraphs: The Bureau’s mission is to provide comprehensive data about the nation’s  people and economy. The 2010 census enumerates the number and  location of people on Census Day, which is April 1, 2010. However, census  operations begin long before Census Day and continue afterward. For  example, address canvassing for the 2010 census will begin in April 2009,  while the Secretary of Commerce must report tabulated census data to the  President by December 31, 2010, and to state governors and legislatures by  March 31, 2011.
The decennial census is a major undertaking for the Bureau that includes  the following major activities:    Establishing where to count. This includes identifying and correcting  addresses for all known living quarters in the United States (address  canvassing) and validating addresses identified as potential group  quarters, such as college residence halls and group homes (group quarters  validation).
Collecting and integrating respondent information. This includes  delivering questionnaires to housing units by mail and other methods,  processing the returned questionnaires, and following up with  nonrespondents through personal interviews (nonresponse follow-up). It  also includes enumerating residents of group quarters (group quarters  enumeration) and occupied transitional living quarters (enumeration of  transitory locations), such as recreational vehicle parks, campgrounds,  and hotels. It also includes a final check of housing unit status (field  verification) where Bureau workers verify potential duplicate housing  units identified during response processing.
Providing census results. This includes tabulating and summarizing  census data and disseminating the results to the public.
subsections: section_title: Role of IT in the Decennial Census
paragraphs: Automation and IT are to play a critical role in the success of the 2010  census by supporting data collection, analysis, and dissemination. Several  systems will play a key role in the 2010 census. For example, enumeration  “universes,” which serve as the basis for enumeration operations and  response data collection, are organized by the Universe Control and  Management (UC&M) system, and response data are received and edited  to help eliminate duplicate responses using the Response Processing  System (RPS). Both UC&M and RPS are legacy systems that are  collectively called the Headquarters Processing System.
Geographic information and support to aid the Bureau in establishing  where to count U.S. citizens are provided by the Master Address  File/Topologically Integrated Geographic Encoding and Referencing  (MAF/TIGER) system. The Decennial Response Integration System (DRIS)  is to provide a system for collecting and integrating census responses from  all sources, including forms and telephone interviews. The Field Data  Collection Automation (FDCA) program includes the development of  handheld computers for the address canvassing operation and the  systems, equipment, and infrastructure that field staff will use to collect  data. Paper-Based Operations (PBO) was established in August 2008  primarily to handle certain operations that were originally part of FDCA.  PBO includes IT systems and infrastructure needed to support the use of  paper forms for operations such as group quarters enumeration activities,  nonresponse follow-up activities, enumeration at transitory locations  activities, and field verification activities. These activities were originally  to be conducted using IT systems and infrastructure developed by the  FDCA program. Finally, the Data Access and Dissemination System II  (DADS II) is to replace legacy systems for tabulating and publicly  disseminating data.
section_title: Comprehensive Testing Improves Chances of a Successful Decennial Census
paragraphs: As stated in our testing guide and the Institute of Electrical and  Electronics Engineers (IEEE) standards, complete and thorough testing is  essential for providing reasonable assurance that new or modified IT  systems will perform as intended. To be effective, testing should be  planned and conducted in a structured and disciplined fashion that  includes processes to control each incremental level of testing, including  testing of individual systems, the integration of those systems, and testing  to address all interrelated systems and functionality in an operational  environment.
Further, this testing should be planned and scheduled in a structured and  disciplined fashion. Comprehensive testing that is effectively planned and  scheduled can provide the basis for identifying key tasks and requirements  and better ensure that a system meets these specified requirements and  functions as intended in an operational environment.
section_title: Dress Rehearsal Includes Testing of Certain Systems and Operations
paragraphs: In preparation for the 2010 census, the Bureau planned what it refers to as  the Dress Rehearsal. The Dress Rehearsal includes systems and  integration testing, as well as end-to-end testing of key operations in a  census-like environment. During the Dress Rehearsal period, running from  February 2006 through June 2009, the Bureau is developing and testing  systems and operations, and it held a mock Census Day on May 1, 2008.  The Dress Rehearsal activities, which are still under way, are a subset of  the activities planned for the actual 2010 census and include testing of  both IT and non-IT related functions, such as opening offices and hiring  staff.
The Dress Rehearsal identified significant technical problems during the  address canvassing and group quarters validation operations. For  example, during the Dress Rehearsal address canvassing operation, the  Bureau encountered problems with the handheld computers, including  slow and inconsistent data transmissions, the devices freezing up, and  difficulties collecting mapping coordinates. As a result of the problems  observed during the Dress Rehearsal, cost overruns and schedule slippage  in the FDCA program, and other issues, the Bureau removed the planned  testing of several key operations from the Dress Rehearsal and switched  key operations, such as nonresponse follow-up, to paper-based processes  instead of using the handheld computers as originally planned.
section_title: Bureau Is Making Progress in Key System Testing, but Lacks Plans and Schedules
paragraphs: Through the Dress Rehearsal and other testing activities, the Bureau has  completed key system tests, but significant testing has yet to be done, and  planning for this is not complete. Table 1 summarizes the status and plans  for system testing.
subsections: section_title: Bureau Has Conducted Limited Integration Testing, but Has Not Developed 2010 Test Plans and Schedules for Integration Testing
paragraphs: Effective integration testing ensures that external interfaces work  correctly and that the integrated systems meet specified requirements.  This testing should be planned and scheduled in a disciplined fashion  according to defined priorities.
For the 2010 census, each program office is responsible for and has made  progress in defining system interfaces and conducting integration testing,  which includes testing of these interfaces. However, significant activities  remain to be completed. For example, for systems such as PBO, interfaces  have not been fully defined, and other interfaces have been defined but  have not been tested. In addition, the Bureau has not established a master  list of interfaces between key systems, or plans and schedules for  integration testing of these interfaces. A master list of system interfaces is  an important tool for ensuring that all interfaces are tested appropriately  and that the priorities for testing are set correctly. As of October 2008, the  Bureau had begun efforts to update a master list it had developed in 2007,  but it has not provided a date when this list will be completed.
Without a completed master list, the Bureau cannot develop  comprehensive plans and schedules for conducting systems integration  testing that indicate how the testing of these interfaces will be prioritized.  With the limited amount of time remaining before systems are needed for  2010 operations, the lack of comprehensive plans and schedules increases  the risk that the Bureau may not be able to adequately test system  interfaces, and that interfaced systems may not work together as intended.
section_title: Bureau Has Conducted Limited End-to-End Testing as Part of the Dress Rehearsal, but Has Not Developed Testing Plans for Critical Operations
paragraphs: Although several critical operations underwent end-to-end testing in the  Dress Rehearsal, others did not. As of December 2008, the Bureau had not  established testing plans or schedules for end-to-end testing of the key  operations that were removed from the Dress Rehearsal, nor has it  determined when these plans will be completed. These operations include    enumeration of transitory locations,    group quarters enumeration, and  field verification.
The decreasing time available for completing end-to-end testing increases  the risk that testing of key operations will not take place before the  required deadline. Bureau officials have acknowledged this risk in  briefings to the Office of Management and Budget. However, as of January  2009, the Bureau had not completed mitigation plans for this risk.  According to the Bureau, the plans are still being reviewed by senior  management. Without plans to mitigate the risks associated with limited  end-to-end testing, the Bureau may not be able to respond effectively if  systems do not perform as intended.
section_title: Bureau Lacks Sufficient Executive-Level Oversight and Guidance for Testing
paragraphs: As stated in our testing guide and IEEE standards, oversight of testing  activities includes both planning and ongoing monitoring of testing  activities. Ongoing monitoring entails collecting and assessing status and  progress reports to determine, for example, whether specific test activities  are on schedule. In addition, comprehensive guidance should describe  each level of testing and the types of test products expected.
In response to prior recommendations, the Bureau took initial steps to  enhance its programwide oversight; however, these steps have not been  sufficient. For example, in June 2008, the Bureau established an inventory  of all testing activities specific to all key decennial operations. However,  the inventory has not been updated since May 2008, and officials have no  plans for further updates.
In another effort to improve executive-level oversight, the Decennial  Management Division began producing (as of July 2008) a weekly  executive alert report and has established (as of October 2008) a  dashboard and monthly reporting indicators. However, these products do  not provide comprehensive status information on the progress of testing  key systems and interfaces. Further, the assessment of testing progress  has not been based on quantitative and specific metrics. The lack of  quantitative and specific metrics to track progress limits the Bureau’s  ability to accurately assess the status and progress of testing activities. In  commenting on our draft report, the Bureau provided selected examples  where they had begun to use more detailed metrics to track the progress  of end-to-end testing activities.
The Bureau also has weaknesses in its testing guidance. According to the  Associate Director for the 2010 census, the Bureau did establish a policy  strongly encouraging offices responsible for decennial systems to use best  practices in software development and testing, as specified in level 2 of  Carnegie Mellon’s Capability Maturity Model® Integration. However,  beyond this general guidance, there is no mandatory or specific guidance  on key testing activities such as criteria for each level or the type of test  products expected. The lack of guidance has led to an ad hoc—and, at  times—less than desirable approach to testing.
section_title: Implementation of Recommendations Could Help Ensure Key Testing Activities are Completed
paragraphs: In our report, we are making ten recommendations for improvements to  the Bureau’s testing activities. Our recommendations include finalizing  system requirements and completing development of test plans and  schedules, establishing a master list of system interfaces, prioritizing and  developing plans to test these interfaces, and establishing plans to test  operations removed from the Dress Rehearsal. In addition, we are  recommending that the Bureau improve its monitoring of testing progress  and improve executive-level oversight of testing activities.
In written comments on the report, the department had no significant  disagreements with our recommendations. The department stated that its  focus is on testing new software and systems, not legacy systems and  operations used in previous censuses. However, the systems in place to  conduct these operations have changed substantially and have not yet  been fully tested in a census-like environment. Consistent with our  recommendations, finalizing test plans and schedules and testing all  systems as thoroughly as possible will help to ensure that decennial  systems will work as intended.
In summary, while the Bureau’s program offices have made progress in  testing key decennial systems, much work remains to ensure that systems  operate as intended for conducting an accurate and timely 2010 census.  This work includes system, integration, and end-to-end testing activities.  Given the rapidly approaching deadlines of the 2010 census, completing  testing and establishing stronger executive-level oversight are critical to  ensuring that systems perform as intended when they are needed.
Mr. Chairman and members of the subcommittee, this concludes our  statement. We would be pleased to respond to any questions that you or  other members of the subcommittee may have at this time.
section_title: Contacts and Staff Acknowledgements
paragraphs: If you have any questions about matters discussed in this testimony, please  contact David A. Powner at (202) 512-9286 or pownerd@gao.gov or Robert  Goldenkoff at (202) 512-2757 or goldenkoffr@gao.gov. Other key  contributors to this testimony include Sher`rie Bacon, Barbara Collier, Neil  Doherty, Vijay D’Souza, Elizabeth Fan, Nancy Glover, Signora May, Lee  McCracken, Ty Mitchell, Lisa Pearson, Crystal Robinson, Melissa  Schermerhorn, Cynthia Scott, Karl Seifert, Jonathan Ticehurst, Timothy  Wexler, and Katherine Wulff.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.