Information Technology: Census Bureau Needs to Improve Its Risk Management of Decennial Systems
Released: 2007-10-05T00:00:00

Letter

Background

The Census Bureau’s mission is to serve as the leading source of high- quality data about the nation’s people and economy. The Bureau’s core activities include conducting decennial, economic, and government censuses, conducting demographic and economic surveys, managing international demographic and socioeconomic databases, providing technical advisory services to foreign governments, and performing such other activities as producing official population estimates and projections.

Conducting the decennial census is a major undertaking involving considerable preparation, which is currently under way. A decennial census involves identifying and correcting addresses for all known living quarters in the United States (known as “address canvassing”); sending questionnaires to housing units; following up with nonrespondents through personal interviews; identifying people with nontraditional living arrangements; managing a voluminous workforce responsible for follow-up activities; collecting census data by means of questionnaires, calls, and personal tabulating and summarizing census data; and disseminating census analytical results to the public.

Role of IT in the Decennial Census

The Bureau estimates that it will spend about $3 billion on automation and IT for the 2010 Census, including four major systems acquisitions that are expected to play a critical role in improving its coverage, accuracy, and efficiency. Figure 1 shows the key systems and interfaces supporting the 2010 Census; the four major IT systems involved in the acquisitions are highlighted. As the figure shows, these four systems are to play important roles with regard to different aspects of the process.

To establish where to count (as shown in the top row of fig. 1), the Bureau will depend heavily on a database that provides address lists, maps, and other geographic support services. The Bureau’s address list, known as the Master Address File (MAF), is associated with a geographic information system containing street maps; this system is called the Topologically Integrated Geographic Encoding and Referencing (TIGER) database. The ® MAF/TIGER database, highlighted in fig. 1, is the object of the first major IT acquisition—the MAF/TIGER Accuracy Improvement Project (MTAIP).

The project is to provide corrected coordinates on a county-by-county basis for all current features in the TIGER database. The vital role of this database in the census operations is the reason that MTAIP is a key acquisition, even though it is relatively small in scale (compared with the other three key IT acquisitions) and will not result in new systems.

To collect respondent information (see the middle row of fig. 1), the Bureau is pursuing two initiatives. First, the Field Data Collection Automation (FDCA) program is expected to provide automation support for field data collection operations as well as reduce costs and improve data quality and operational efficiency. This acquisition includes the systems, equipment, and infrastructure that field staff will use to collect census data, such as mobile computing devices.

Second, the Decennial Response Integration System (DRIS) is to provide a system for collecting and integrating census responses from all sources, including forms, telephone interviews, and mobile computing devices in the field. DRIS is expected to improve accuracy and timeliness by standardizing the response data and providing it to other Bureau systems for analysis and processing.

To provide results, the Data Access and Dissemination System II (DADS II) acquisition (see the bottom row of fig. 1) is to replace legacy systems for tabulating and publicly disseminating data. The DADS II program is expected to provide comprehensive support to DADS. Replacement of the legacy systems is expected to maximize the efficiency, timeliness, and accuracy of tabulation and dissemination products and services; minimize the cost of tabulation and dissemination; and increase user satisfaction with related services.

Table 1 provides a brief overview of the four acquisitions.

Responsibility for these acquisitions lies with the Bureau’s Decennial Management Division and the Geography Division. Each of the four acquisitions is managed by an individual project team staffed by Bureau personnel. Additional information on the contracts for these four systems is provided in appendix II.

In preparation for the 2010 Census, the Bureau plans a series of tests of its operations and systems (new and existing) in different environments, as well as to conduct what it refers to as the Dress Rehearsal. During the Dress Rehearsal period, which runs from February 2006 through June 2009, the Bureau plans to conduct development and testing of systems, run a mock Census Day, and prepare for Census 2010, which will include opening offices and hiring staff.

As part of the Dress Rehearsal activities, the Bureau began address canvassing in April 2007 and plans to distribute questionnaires in February 2008 in preparation for the mock Census Day on April 1, 2008. It plans to begin performing nonresponse follow-up activities immediately afterwards. These Dress Rehearsal activities are to provide an operational test of the available system functionalities, in a census-like environment, as well as other operational and procedural activities.

Prior IT Management Reviews of Census Activities

We have previously reported on weaknesses in the Bureau’s IT acquisition management. In June 2005, we reported on the Bureau’s progress in five IT areas—investment management, systems development and management, enterprise architecture management, information security, and human capital. These areas are important because they have substantial influence on the effectiveness of organizational operations and, if implemented effectively, can reduce the risk of cost and schedule overruns and performance shortfalls. We reported that while the Bureau had many practices in place, much remained to be done to fully implement effective IT management capabilities. To improve the Bureau’s IT management, we made several recommendations. The Bureau agreed with the recommendations but is still in the process of implementing them.

In March 2006, we presented testimony on the Bureau’s progress in implementing acquisition and management capabilities for two key IT system acquisitions for the 2010 Census—FDCA and DRIS. We testified that although the project offices responsible for these two contracts had carried out initial acquisition management activities, neither office had the full set of capabilities needed to effectively manage the acquisitions, including a full risk management process. Effective management of major IT programs requires that organizations use sound acquisition and management processes, including project and acquisition planning, solicitation, requirements development and management, and risk management. We recommended that the Bureau implement key activities needed to effectively manage acquisitions. For example, we recommended that the Bureau establish and enforce a system acquisition management policy that incorporates best practices, including those for risk management. The Bureau agreed with our recommendations and is in the process of implementing them.

Decennial IT Acquisitions Are at Various Stages of Development and Show Mixed Progress against Schedule and Cost Baselines

Three key systems acquisitions for the 2010 Census are in process, and a fourth contract was recently awarded. The ongoing acquisitions are showing mixed progress in providing deliverables while adhering to planned schedules and cost estimates. Currently, two of the three projects have experienced schedule delays, and the date for awarding the fourth contract was postponed several times. In addition, we estimate that one of the three ongoing projects (FDCA) will incur about $18 million in cost overruns. In response to schedule delays as well as other factors, including cost, the Bureau has made schedule adjustments and plans to delay certain system functionality. As a result, Dress Rehearsal operational testing will not address the full complement of systems and functionality that was originally planned, and the Bureau has not yet finalized its plans for further system tests. Delaying functionality increases the importance of system testing after the Dress Rehearsal operational testing to ensure that the decennial systems work as intended.

MTAIP Is Completing Improvements on Schedule and at Estimated Cost

MTAIP is a project to improve the accuracy of the MAF/TIGER database, which contains information on street locations, housing units, rivers, railroads, and other geographic features. MTAIP is to provide corrected coordinates on a county-by-county basis for all current features in the TIGER database. Features not now in TIGER are to be added with accurate coordinates and required attributes.

Currently, the acquisition is in the second and final phase of its life cycle. During Phase I, from June 2002 through December 2002, the contractor identified technical requirements and established the production approach for Phase II activities. In Phase II, which began in January 2003 and is ongoing, the contractor is developing improved maps for all 3,037 counties in the United States; to date, it has delivered more than 75 percent of these maps, which are due by September 2008. Beginning in fiscal year 2008, maintenance for the contract will begin. The contract closeout activities are scheduled for fiscal year 2009.

MTAIP is on schedule to complete improvements by the end of fiscal year 2008 and is meeting cost estimates. The following is the status of MTAIP’s schedule and cost estimates: The MTAIP acquisition is on schedule for the deliverables for Phases I and II. According to Bureau documents, as of September 2006, the contractor (Harris Corporation) had delivered (as required) 2,000 improved county maps out of the 3,037. As of March 2007, Bureau documents showed that the contractor had completed 338 of the 694 counties expected to be complete by the end of fiscal year 2007. The contractor is scheduled to complete the remaining 356 counties by the end of fiscal year 2007.

Cost estimates for Phase I and Phase II are $4.8 million and $205.2 million, respectively, for a total contract value of $210 million. The contract met cost estimates for Phase I, and based on cost performance reports, we project no cost overruns by September 2008. As of June 2007, the Bureau had obligated $178 million through September 2010.

FDCA Has Provided Deliverables, but It Has Delayed Functionality and Is Experiencing Cost Increases

FDCA is to provide the systems, equipment, and infrastructure that field staff will use to collect census data. It is to establish office automation for the 12 regional census centers, the Puerto Rico area office, and approximately 450 temporary local census offices. It is to provide the telecommunications infrastructure for headquarters, regional and local offices, and mobile computing devices for field workers. FDCA also is to facilitate integration with other 2010 Census systems and to provide development, deployment, technical support, de-installation, and disposal services. At the peak of the 2010 Census, about 4,000 field operations supervisors, 40,000 crew leaders, 500,000 enumerators and address listers, and several thousand office employees are expected to use or access FDCA components.

The FDCA acquisition is currently in the first phase of execution, since it has completed its baseline planning period in June 2006. The contractor is currently in the process of developing and testing FDCA software for the Dress Rehearsal Census Day. In future phases, the project will continue development, deploy systems and hardware, support census operations, and perform operational and contract closeout activities.

However, as shown in table 2, according to the Bureau it revised its original schedule and delayed or eliminated some key functionality that was expected to be ready during Execution Period 1. The Bureau, said it revised the schedule because it realized it had underestimated the costs for the early stages of the contract, and that it could not meet the level of first-year funding because the fiscal year 2006 budget was already in place. According to the Bureau, this initial underestimation led to schedule changes and overall cost increases.

In the revised schedule, the Bureau delayed or eliminated some key functionality from the Dress Rehearsal, including the automated software distribution system. Further, the revised software development schedule stretches from two to seven increments over a longer period of time. Delivery of these increments ranges from December 2006 through December 2008. As of May 2007, the contractor reported that the increment development schedule continues to be aggressive.

The project is meeting all planned milestones on the revised schedule. The contractor has delivered 1,388 mobile computing devices to be used in address canvassing for the Dress Rehearsal. Also, key FDCA support infrastructure has been installed, including the Network Operations Center, Security Operation Center, and the Data Processing Centers. According to the department, all Regional Census Centers and Puerto Rico area offices have been identified and are on schedule to open in January 2008.

The project life-cycle costs have already increased. At contract award in March 2006, the total cost of FDCA was estimated not to exceed $596 million. However, in September 2006, the project life-cycle cost was increased to about $624 million. In May 2007, the life-cycle cost rose by a further $23 million because of increasing system requirements, which resulted in an estimated life-cycle cost of about $647 million. Table 3 shows the current life-cycle cost estimates for FDCA.

In addition, the FDCA project has already experienced $6 million in cost overruns, and more are expected. Both our analysis and the contractor’s analysis expect FDCA to experience additional cost overruns. Based on our analysis of cost performance reports (from July 2006 to May 2007), we project that the FDCA project will experience further cost overruns by December 2008. The FDCA cost overrun is estimated between $15 million and $19 million, with the most likely overrun to be about $18 million. Harris, in contrast, estimates about a $6 million overrun by December 2008.

According to Harris, the major cause of projected cost overruns is the system requirements definition process. For example, in December 2006, Harris indicated that the requirements for the Dress Rehearsal Paper Based Operations in Execution Period 1 had increased significantly. According to the cost performance reports, this increase has meant that more work must be conducted and more staffing assigned to meet the Dress Rehearsal schedule.

The schedule changes to FDCA have increased the likelihood that the systems testing at the Dress Rehearsal will not be as comprehensive as planned. The inability to perform comprehensive operational testing of all interrelated systems increases the risk that further cost overruns will occur and that decennial systems will experience performance shortfalls.

After a Schedule Revision, DRIS Is Delivering Reduced Functionality at Projected Cost

DRIS is to provide a system for collecting and integrating census responses, standardizing the response data, and providing it to other systems for analysis and processing. The DRIS functionality is critical for providing assistance to the public via telephone and for monitoring the quality and status of data capture operations.

The DRIS acquisition is currently in the first of three overlapping project phases. In Phase I, which extends from March 2006 to September 2008, the project is performing software development and testing of DRIS. By December 2007, it is to provide an initial system to be used for the Dress Rehearsal Census Day, during which DRIS will process 14 census forms (out of 84 possible forms). In October 2007, the project is to begin Phase II, in which it is to deploy the completed system and perform other activities to support census operations. The final phase is to be devoted to data archiving and equipment disposal.

Although DRIS is currently on schedule to meet its December 2007 milestone, the Bureau revised the original DRIS schedule after the contract was awarded in October 2005. Under the revised schedule (see table 4), the Bureau delayed or eliminated some functionality that was expected to be ready for the Dress Rehearsal Census Day.

According to Bureau officials, they delayed the schedule and eliminated functionality for DRIS when they realized they had underestimated the fiscal year 2006 through 2008 costs for development. As shown in table 5, the government’s funding estimates for DRIS Phase I were significantly lower than the contractor’s.

Originally, the DRIS solution was to include paper, telephone, Internet, and field data collection processing; selection of data capture sites; and preparation and processing of 2010 Census forms. However, the Bureau reduced the scope of the solution by eliminating the Internet functionality. In addition, the Bureau has stated that it will not have a robust telephone questionnaire assistance system in place for the Dress Rehearsal. The Bureau is also delaying selecting sites for data capture centers, preparing data capture facilities, and recruiting and hiring data capture staff.

Although Bureau officials told us that the revisions to the schedule should not affect meeting milestones for the 2010 Census, the delays mean that more systems development and testing will need to be accomplished later. Given the immovable deadline of the decennial census, the Bureau is at risk of reducing functionality or increasing costs to meet its schedule.

The government’s estimate for the DRIS project was $553 million through the end of fiscal year 2010. In October 2005, at contract award, the Phase I and Phase II value was $484 million.

The DRIS project is not experiencing cost overruns, and our analysis of cost performance reports from April 2006 to May 2007 projects no cost overruns by December 2008. As of May 2007, the Bureau had obligated $37 million, and the project was 44 percent completed. As of May 2007, the DRIS contract value had not increased.

DADS II Contract Was Recently Awarded after a Delay

The DADS II acquisition is to replace the legacy DADS systems, which tabulate and publicly disseminate data from the decennial census and other Bureau surveys. The DADS II contractor is also expected to provide comprehensive support to the Census 2000 legacy DADS systems.

In January 2007, the Bureau released the DADS II request for proposal. The contract was awarded in September 2007.

However, the Bureau had delayed the DADS II contract award date multiple times. The award date was originally planned for the fourth quarter of 2005, but the date was changed to August 2006. On March 8, 2006, the Bureau estimated it would delay the award of the DADS II contract from August to October 2006 to gain a clearer sense of budget priorities before initiating the request for proposal process. The Bureau then delayed the contract award again by about another year. Because of these delays, DADS II will not be developed in time for the Dress Rehearsal. Instead, the Bureau will use the legacy DADS system for tabulation during the Dress Rehearsal. However, the Bureau’s plan is to have the DADS II system available for the 2010 Census.

No cost information on the DADS II contract was available because it was recently awarded.

Delayed Functionality Increases the Importance of Further Operational Testing

Operational testing helps verify that systems function as intended in an operational environment. For system testing to be comprehensive, system functionality must be completed. Further, for multiple interrelated systems, end-to-end testing is performed to verify that all interrelated systems, including any external systems with which they interface, are tested in an operational environment.

However, as described above, two of the projects have delayed planned functionality to later phases, and one project contract was recently awarded (September 2007). As a result, the operational testing that is to occur during the Dress Rehearsal period around April 1, 2008, will not include tests of the full complement of decennial census systems and their functionality. According to Bureau officials, they have not yet finalized their plans for system tests. If further delays occur, the importance of these system tests will increase. Delaying functionality and not testing the full complement of systems increase the risk that costs will rise further, that decennial systems will not perform as expected, or both.

The Bureau Is Making Progress in Risk Management Activities, but Critical Weaknesses Remain

The project teams varied in the extent to which they followed disciplined risk management practices. For example, three of the four project teams had developed strategies to identify the scope of the risk management effort. However, three project teams had weaknesses in identifying risks, establishing adequate mitigation plans, and reporting risk status to executive-level officials. These weaknesses in completing key risk management activities can be attributed in part to the absence of Bureau policies for managing major acquisitions, as we described in our earlier report. Without effective risk management practices, the likelihood of project success is decreased.

According to the Software Engineering Institute (SEI), the purpose of risk management is to identify potential problems before they occur. When problems are identified, risk-handling activities can be planned and invoked as needed across the life of a project in order to mitigate adverse impacts on objectives. Effective risk management involves early and aggressive risk identification through the collaboration and involvement of relevant stakeholders. Based on SEI’s Capability Maturity Model® Integration (CMMI), risk management activities can be divided into four ® key areas (see fig. 2): preparing for risk management, identifying and analyzing risks, executive oversight.

The discipline of risk management is important to help ensure that projects are delivered on time, within budget, and with the promised functionality. It is especially important for the 2010 Census, given the immovable deadline.

Project Teams Usually Established Risk Preparation Activities, but Improvements Are Possible

Risk preparation involves establishing and maintaining a strategy for identifying, analyzing, and mitigating risks. The risk management strategy addresses the specific actions and management approach used to perform and control the risk management program. It also includes identifying and involving relevant stakeholders in the risk management process. Table 6 shows the status of the four project teams’ implementation of key risk preparation activities.

As the table shows, three project teams have established most of the risk management preparation activities. However, the MTAIP project team implemented the fewest practices. The team did not adequately determine risk sources and categories, or adequately develop a strategy for risk management. As a result, the project’s risk management strategy is not comprehensive and does not fully address the scope of the risk management effort, including discussing techniques for risk mitigation and defining adequate risk sources and categories.

In addition, three project teams (MTAIP, FDCA, and DADS II) had weaknesses regarding stakeholder involvement. The three teams did not provide sufficient evidence that the relevant stakeholders were involved in risk identification, analysis, and mitigation activities; reviewing the risk management strategy and risk mitigation plans; or communicating and reporting risk management status. In addition, the FDCA project team had not identified relevant stakeholders. These weaknesses can be attributed in part to the absence of Bureau policies for managing major acquisitions, as we described in our earlier reports. Without adequate preparation for risk management, including establishing an effective risk management strategy and identifying and involving relevant stakeholders, project teams cannot properly control the risk management process.

The Project Teams Identified and Analyzed Risks, but Not All Key Risks Were Identified

Risks must be identified and described in an understandable way before they can be analyzed and managed properly. This includes identifying risks from both internal and external sources and evaluating each risk to determine its likelihood and consequences. Analyzing risks includes risk evaluation, categorization, and prioritization; this analysis is used to determine when appropriate management attention is required. Table 7 shows the status of the four project teams’ implementation of key risk identification and evaluation activities.

As of July 2007, the MTAIP and DRIS project teams were adequately identifying and documenting risks, including system interface risks. For example, these teams were able to identify the following: The MTAIP project identified significant risks regarding potential changes in funding and the turnover of contractor personnel as the program nears maturity.

The DRIS project identified significant risks regarding new system security regulations, changes or increases to Phase II baseline requirements, and new interfaces after Dress Rehearsal.

However, the FDCA and DADS II project teams did not identify all risks, including specific system interface risks. For example: The FDCA project had not identified any significant risks related to the handheld mobile computing devices, for the project office to monitor and track, despite problems arising during the recent address canvassing component of the Dress Rehearsal. However, it did identify significant risks for the contractor to manage; these risks were associated with using the handheld mobile computing devices including usability and failure rates. Responsibility for mitigating these risks was transferred to the contractor.

The FDCA and DADS II projects did not provide evidence that specific system interface risks are being adequately identified to ensure that risk handling activities will be invoked should the systems fail during 2010 Census. For example, although the DADS II will not be available for the Dress Rehearsal, the project team did not identify any significant interface risks associated with this system.

One reason for these weaknesses, as mentioned earlier, is the absence of Bureau policies for managing major acquisitions. Failure to adequately identify and analyze risks could prevent management from taking the appropriate actions to mitigate those risks; this increases the probability that the risks will materialize and magnifies the extent of damage incurred in such an event.

Three of Four Project Teams’ Risk Mitigation Plans and Monitoring Activities Were Incomplete

Risk mitigation involves developing alternative courses of action, workarounds, and fallback positions, with a recommended course of action for the most important risks to the project. Mitigation includes techniques and methods used to avoid, reduce, and control the probability of occurrence of the risk; the extent of damage incurred should the risk occur; or both. Examples of activities for mitigating risks include documented handling options for each identified risk; risk mitigation plans; contingency plans; a list of persons responsible for tracking and addressing each risk; and updated assessments of risk likelihood, consequence, and thresholds. Table 8 shows the status of the four project teams’ implementation of key risk mitigation activities.

Three project teams (MTAIP, FDCA, and DADS II) developed mitigation plans that were often untimely or included incomplete activities and milestones for addressing the risks. Some of these untimely and incomplete activities and milestones included the following: Although the MTAIP project team developed mitigation plans, the plans were not comprehensive and did not include thresholds defining when risk becomes unacceptable and should trigger the execution of the mitigation plan.

The FDCA project team had developed mitigation plans for the most significant risks, but the plans did not always identify milestones for implementing mitigation activities. Moreover, the plans did not identify any commitment of resources, several did not establish a period of performance, and the team did not always update the plans with the latest information on the status of the risk. In addition, the FDCA project team did not provide evidence of developing mitigation plans to handle the other significant risks as described in their risk mitigation strategy. (These risks included a lack of consistency in requirements definition and insufficient FDCA project office staffing levels.)

The mitigation plans for DADS II were incomplete, with no associated future milestones and no evidence of continual progress in working towards mitigating a risk. In several instances, DADS II mitigation plans were listed as “To Be Determined.”

With regard to the second practice in the table (periodically monitoring risk status and implementing mitigation plans), the MTAIP, FDCA, and DADS II project teams were not always implementing the mitigation plans as appropriate. For example, although the MTAIP project team has periodically monitored the status of risks, its mitigation plans do not include detailed action items with start dates and anticipated completion dates; thus, the plans do not ensure that mitigation activities are implemented appropriately and tracked to closure. The FDCA and DADS II project teams did not identify system interface risks nor prepare adequate mitigation plans to ensure that systems will operate as intended. In addition, the DADS II risk reviews showed no evidence of developing risk- handling action items, tracking any existing open risk-handling action items, or regularly discussing mitigation steps with other risk review team members.

Because they did not develop complete mitigation plans, the MTAIP, FDCA, and DADS II project teams cannot ensure that for a given risk, techniques and methods will be invoked to avoid, reduce, and control the probability of occurrence.

Project Teams Are Inconsistent in Reporting Risk Status to Executive- Level Management

Reviews of the project teams’ risk management activities, status, and results should be held on a periodic and event-driven basis. The reviews should include appropriate levels of management, such as key Bureau executives, who can provide visibility into the potential for project risk exposure and appropriate corrective actions. Table 9 shows the status of the four project teams’ implementation of activities for senior-level risk oversight.

The project teams were inconsistent in reporting the status of risks to executive-level officials. DRIS and DADS II did regularly report risks; however, the FDCA and MTAIP projects did not provide sufficient evidence to document that these discussions occurred or what they covered. Although presentations were made on the status of the FDCA and MTAIP projects to executive-level officials, presentation documents did not include evidence of discussions of risks and mitigation plans. Failure to report a project’s risks to executive-level officials reduces the visibility of risks to executives who should be playing a role in mitigating them.

Conclusions

The IT acquisitions planned for 2010 Census will require continued oversight to ensure that they are achieved on schedule and at planned cost levels. Although the MTAIP and DRIS acquisitions are currently meeting cost estimates, FDCA is not. In addition, while the Bureau is making progress developing systems for the Dress Rehearsal, it is deferring certain functionality, with the result that the Dress Rehearsal operational testing will address less than a full complement of systems. Delaying functionality increases the importance of later development and testing activities, which will have to occur closer to the census date. It also raises the risk of cost increases, given the immovable deadline for conducting the 2010 Census.

The Bureau’s project teams for each of the four acquisitions have implemented many practices associated with establishing sound and capable risk management processes, but they are not always consistent: the teams have not always identified risks, developed complete risk mitigation plans, or briefed senior-level officials on risks and mitigation plans. Among risks that were not identified are those associated with the FDCA mobile computing devices and systems testing. Also, mitigation plans were often untimely or incomplete. Further, no evidence was available of senior-level briefings to discuss risks and mitigation plans.

One reason for these weaknesses is the absence of Bureau policies for managing major acquisitions, as we pointed out in earlier work. Until the project teams and the Decennial Management Division implement appropriate risk management activities, they face an increased probability that decennial systems will not be delivered on schedule and within budget or perform as expected.

Recommendations for Executive Action

To ensure that the Bureau’s four key acquisitions for the 2010 Census operate as intended, we are making four recommendations. First, to ensure that the Bureau’s decennial systems are fully tested, we recommend that the Secretary of Commerce require the Director of the Census Bureau to direct the Decennial Management Division and Geography Division to plan for and perform end-to-end testing so that the full complement of systems is tested in a census-like environment.

To strengthen risk management activities for the decennial census acquisitions, the Secretary should also direct the Director of the Census Bureau to ensure that project teams identify and develop a comprehensive list of risks for the acquisitions, particularly those for system interfaces and mobile computing devices, and analyze them to determine probability of occurrence and appropriate mitigating actions; develop risk mitigation plans for the significant risks, including defining the mitigating actions, milestones, thresholds, and resources; and provide regular briefings on significant risks to senior executives, so that they can play a role in mitigating these risks.

We are not making recommendations at this time regarding the Bureau’s policies for managing major acquisitions, as we have already done so in previous reports.

Agency Comments and Our Evaluation

In response to a draft of this report, the Under Secretary for Economic Affairs of Commerce provided written comments from the department. These comments are reproduced in appendix III.

The department disagreed with our conclusion about operational testing during the 2008 Dress Rehearsal. According to the department, although some minimal functionalities are not a part of the Dress Rehearsal, all critical systems and interfaces would be tested during the 2008 Dress Rehearsal. It planned to conduct additional fully integrated testing of all systems and interfaces after the Dress Rehearsal, including the functionalities not included in the Dress Rehearsal itself. It also planned to incorporate lessons learned from the Dress Rehearsal in this later testing. Nonetheless, the Bureau’s test plans have not been finalized. Further, the Dress Rehearsal will not include two critical systems (the DRIS telephone system and the DADS II tabulation system). Thus, it remains unclear whether testing will in fact address all interrelated systems and functionality in a census-like environment. Consistent with our recommendation, following up with documented test plans to do end-to- end testing would help ensure that decennial systems will work as intended.

With regard to risk management, the department said it plans to examine additional ways to manage risks and will prepare a formal action plan in response to our final report. However, it disagreed with our assessment with regard to risk identification, pointing out that one project identified risks associated with handheld mobile computing devices and assigned responsibility for these to the contractor. In addition, the project identified systems interfaces as a risk. However, the project did not identify significant risks for the project office to monitor and track related to problems arising during the address canvassing component of the Dress Rehearsal. Also, although this project identified a general risk related to system interfaces, it did not identify specific risks related to particular interfaces.

The department also provided technical comments that we incorporated where appropriate.

We are sending copies of this report to the Chairman and Ranking Member of the Committee on Homeland Security and Governmental Affairs. We are also sending copies to the Secretary of Commerce, the Director of the U.S. Census Bureau, and other appropriate congressional committees. We will make copies available to others on request. In addition, this report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you have any questions about this report, please contact David A. Powner at (202) 512-9286 or pownerd@gao.gov or Madhav S. Panwar at (202) 512-6228 or panwarm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) determine the status and plans, including schedule and costs, for four key information technology (IT) acquisitions, and (2) assess whether the Census Bureau is adequately managing the risks facing these key system acquisitions.

To determine the status and plans, we reviewed documents related to the major 2010 Census acquisitions, including requests for proposals, acquisition contracts, project plans, schedules, cost estimates, program review reports, earned value management data, test plans, and other acquisition-related documents. We analyzed earned value management data obtained from the contractors to assess the contractor’s cost and schedule performance. We also interviewed program officials to determine the current status of the acquisitions’ schedules and cost estimates.

To assess the status of risk management, we evaluated the practices for key areas (establishing a risk strategy, risk identification, mitigation, and reporting) and compared these to industry standards—specifically, the Capability Maturity Model). The CMMI model was developed by Carnegie Mellon University’s Software Engineering Institute (SEI) and includes criteria to evaluate risk management for development and maintenance activities. We adapted these CMMI criteria and performed a Class B Standard CMMI Appraisal Method for Process Improvement to evaluate the risk management of program teams and contractors involved in the decennial system acquisitions and development initiatives. In doing so, we selected leading practices within the areas of preparing for risk management, identifying and analyzing risks, mitigating risks, and executive oversight. We evaluated the practices as fully implemented, partially implemented, or not implemented. Specifically, a blank circle indicates that practices are not performed at all or are performed on a predominantly ad hoc basis; a half circle indicates that the while selected key practices have been performed, others remain to be implemented; and a solid circle indicates that practices adhere to industry standards.

To evaluate the extent to which the Bureau and contractors followed these leading practices, we reviewed relevant documents such as risk management plans, risk reports, mitigation plans, meeting minutes from risk review meetings; we also interviewed knowledgeable officials about their risk management activities. Specifically, we met with project team officials for the four key decennial system acquisitions and their primary contractors (Harris Corporation and Lockheed Martin), as applicable. We also reviewed the lists of risks identified by each of the project teams and their primary contractors and assessed their accuracy and completeness, including whether the risks were associated with the acquisition’s development plans.

We conducted our work from December 2006 through August 2007 in the Washington, D.C., metropolitan area in accordance with generally accepted government auditing standards.

Appendix II: Key 2010 Census Information Technology Acquisitions

Harris Corporation

Appendix III: Comments from the Department of Commerce

The following are GAO’s comments on the department’s letter dated September 25, 2007.

GAO Comments

1. Although the department states that it plans to test all critical systems and interfaces either during or after the Dress Rehearsal, we are aware of two critical systems (the DRIS telephone system and the DADS II tabulation system) that are not to be included in the Dress Rehearsal, and the Bureau’s plans are not yet finalized. As a result, we stand by our characterization that operational testing would take place during the Dress Rehearsal without the full complement of systems and functionality originally planned. Consistent with our recommendation, following up with documented test plans to do end-to-end testing would help ensure that systems work as intended. 2. The department said that our statement could be interpreted that cost increases resulted from an increase in the number of system requirements. It said this is not entirely accurate because although some requirements were added (generally related to security), other cost increases were due to the process of developing detailed requirements from high-level functional requirements. However, it is our view that the process of developing detailed requirements from high-level functional requirement does not inevitably lead to cost increases if the functional requirements were initially well-defined. 3. See comment 1. 4. We have modified our report to reflect this additional information.

However, although our discussion of schedule and cost changes preceded our discussion of risk management, we did not intend to imply that risk management weaknesses had contributed to these changes. We revised our report to help clarify this. 5. We have revised our report to clarify the use of automation for data collection for all FDCA components. 6. See comment 1. 7. We agree that this statement is referring to all FDCA equipment, infrastructure, and systems. 8. We have revised our report to update the status of the systems. 9. We have revised our report to reflect the status of the office site selections. 10. See comment 2. 11. We disagree with the department’s comment that “cost overrun” refers to a contractor originally underestimating costs. We use “cost overrun” to refer to any increase in costs from original estimates. 12. We have revised our report to reflect this information. 13. We have revised our report to add this information. 14. We agree that the FDCA project identified certain risks, as the department describes. However, although it identified risks associated with handheld mobile computing devices and assigned responsibility for these to the contractor, it did not identify significant risks for the project office to monitor and track related to problems arising during the address canvassing component of the Dress Rehearsal. In addition, although this project identified interface management as a risk, it did not identify specific risks related to other systems. Accordingly, although we modified our report to reflect this information, we did not change our overall evaluation. 15. The department stated that for the FDCA and MTAIP projects, risk status is regularly discussed with executive-level officials at Commerce and the Bureau, and that it provided us with briefing slides to support this statement. It said that it also uses other communication channels to report project issues and risks. However, the evidence provided did not show that FDCA and MTAIP risks were regularly discussed with executive-level officials. For example, while the FDCA project provided two presentations in October 2006 and March 2007, these presentations did not have discussions of risk and mitigation plans. Similarly, our review of the MTAIP project teams’ presentations during quarterly reviews did not show that risk status was discussed. Therefore, we still conclude that these projects, unlike the other two, did not have sufficient evidence that executive-level officials were being regularly briefed on risk status. 16. See comment 2. 17. We have revised our report to reflect this information.

Appendix IV: GAO Contacts and Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

In addition to the contacts named above, individuals making contributions to this report included Cynthia Scott (Assistant Director), Mathew Bader, Carol Cha, Barbara Collier, Neil Doherty, Karl Seifert, Niti Tandon, Amos Tevelow, and Jonathan Ticehurst.