2010 Census: Plans for Decennial Census Operations and Technology Have Progressed, But Much Uncertainty Remains
Released: 2008-06-11T00:00:00

Letter

Background

The Bureau has less than two years until Census Day. To ensure a successful census, sound risk management will be crucial, particularly given its scope, magnitude, and immutable deadlines of the census. The size of the decennial operation means that small problems can magnify quickly, and big problems could be overwhelming. For example, 60 seconds might seem like an inconsequential amount of time, but in 2000, if enumerators had spent just 1 minute more at each household during nonresponse follow-up, almost $10 million would have been added to the cost of the census. Further, sound risk management is important to a successful census because many risks are interrelated, and a shortcoming in one operation could cause other operations to spiral downward. For instance, a low mail response rate would drive up the follow-up workload, which in turn would increase staffing needs and costs. Of course, the reverse is also true, where a success in one operation could positively affect downstream operations. Nevertheless, rigorous up-front planning and testing, as well as risk mitigation plans, are the best ways to stave off problems. Finally, the census is conducted against a backdrop of immutable deadlines; the census’ elaborate chain of interrelated pre- and post-Census Day activities is predicated upon those dates. To meet legally mandated reporting requirements, including delivery of population counts to the President on December 31, 2010, census activities need to take place at specific times and in the proper sequence.

Bureau Has Taken Important Steps in Planning for a Paper- based Nonresponse Follow-up Operation, But Much Remains Uncertain

On May 8, 2008 the Bureau issued its plans for conducting the 2010 Census paper-based nonresponse follow-up operation outlining key operational decisions. Among these is the need to develop an information system to manage the workload for a paper-based nonresponse follow-up operation and for additional field infrastructure, such as more telephones and computers to support this operation, to restructure the replacement mailing and the removal of late mail returns from the nonresponse follow- up workload, as well as the need for cognitive testing of the enumerator questionnaire used to collect data from nonrespondents.

The contractor carrying out the FDCA program will develop the operations control system, which is designed to manage field operations that rely on paper as well as those that rely upon the handheld computers. The Bureau is particularly concerned about this system because when it was tested as part of earlier dress rehearsal operations—for example, during group quarters validation—it was found to be unreliable. As a result, the workload for these operations had to be supplemented with additional paper-based efforts by local census office staff, instead of electronically as intended. The operations control system is critical because it is intended to provide managers with essential real-time information such as enumerator productivity and the status of workload such as interviews conducted and remaining. Bureau officials said that the manual workaround was manageable for the dress rehearsal with just two local census offices; however, such a manual workaround would be nearly impossible to do when operations are carried out nationwide next year. Officials said that they expect to review computer screen shots of the operations control system reports it will use to manage the nonresponse follow-up operation in January 2009; however, the Bureau has not yet determined when and how testing of the operations control system before nonresponse follow-up, which begins in April 2010, will occur.

The Bureau will be using newly developed systems for integrating responses and managing nonresponse follow-up workload that have not yet been fully tested in a census-like environment. The Bureau’s contract for the Decennial Response Integration System, designed to help identify households that have not yet returned census forms and to collect the results from enumerators conducting nonresponse follow-up interviews, will process each mail return and enumerator questionnaire and transmit to the FDCA program the number of questionnaires received. In turn, FDCA will manage the nonresponse follow-up workload, in part by removing initial late mail returns from the list of housing units requiring follow-up visits. Consequently, depending on time and cost considerations, Bureau officials believe that the Bureau must conduct, at a minimum, a small scale simulation of the integration and communication between the Decennial Response Integration System and FDCA for such aspects as load testing for a paper-based operation, and interfaces such as when the paper is processed by the Decennial Response Integration System and when the check-in status is transmitted to individual local census offices through management reports processed by the FDCA program. When or how these tests will be completed is not clear.

The Bureau’s plans for nonresponse follow-up will also require changes in local census office infrastructure. The Bureau expects it will need additional hardware, including printing and scanning equipment, computers, and telephones. Further, the Bureau expects to scale the FDCA network to support a system for keying in large volumes of data related to hiring and payroll for over 700,000 field workers it plans to hire for the nonresponse follow-up operation. Previously, the Bureau expected to maintain field worker time reporting using the handheld computer. Also, the Bureau expected to hire fewer field workers.

The Bureau’s redesign has also changed the replacement mailing strategy which will be used in 2010. The replacement mailing is a second mailing sent to nonresponding households. Testing has shown that a second mailing increases the overall response rate and reduces costs by increasing the number of returns that come in by mail, decreasing the need for census field workers to collect census data in person. Prior to the redesign, the Bureau planned to send second mailings to all nonresponding households that initially received the census form in the mail. However these plans changed, in part because, according to the Bureau, without using handheld computers for nonresponse follow-up, it would not be able to dynamically remove late mail returns—including those resulting from the replacement mailing—from the enumerator assignments on a daily basis. The Bureau had to devise a way to balance the time available to print replacement questionnaires with the time available to remove late mail returns from the paper-based nonresponse follow-up workload.

The Bureau now plans a multi-part approach. First, it will send approximately 25-30 million blanket replacement mailings to census tracts with low response rates, based on historical response rate data from 1990 and 2000 Census and the American Community Survey. As a result, all housing units in these selected census tracts would receive a second census form, regardless of whether or not they returned the initial form. Similarly, the Bureau plans to target a second mailing to an additional 15 million households in census tracts that are in the middle-range of mail response rates. Finally, the Bureau will not send a replacement mailing to households located in census tracts that previously had high mail response rates. This combination “blanket” and “targeted” mailing strategy is a new approach that will not be tested prior to the 2010 Census. If the replacement mailing does not function as planned, this strategy could confuse respondents in the blanket mailing areas and result in multiple responses from the same household that return both forms. It is instructive to consider that the Bureau’s previous experience with a blanket second “replacement” questionnaire sent to all housing units located in the 1998 dress rehearsal sites caused a significant number of households with multiple responses. As a result, the replacement mailing was dropped from the 2000 Census design because the Bureau was concerned that it would have been overwhelming to process multiple census responses during the actual census.

Moreover, without the benefit of implementing nonresponse follow-up during the dress rehearsal, the Bureau will not know how well its new system for removal of late mail returns will work. While the Bureau encourages respondents to mail back their census forms quickly, some are not returned until the middle of April or later, after the nonresponse follow-up operation has begun. To reduce the cost of nonresponse follow- up and to minimize respondent burden, it is beneficial to the Bureau to remove these late mail returns from the nonresponse follow-up universe. Because nonresponse follow-up will be paper-based rather than conducted with handheld computers, the Bureau will remove late mail returns with the FDCA program prior to April 20 and manually thereafter; however, the recent Bureau plans provide only timelines for removing late mail returns and the Bureau has not yet finalized the workload estimates or how it will manage this work. Not having an opportunity to rehearse its strategy for removing late mail returns makes difficult any estimate of resulting workload.

In addition, Bureau officials said that it will be important to conduct cognitive testing of the questionnaire used by enumerators for nonresponse follow-up. With the change from using handheld computers, a paper questionnaire will be used by census enumerators in the 2010 nonresponse follow-up when making personal visits to housing units to collect census data. When developing this questionnaire, the Bureau plans to draw upon its extensive research and testing of interviewer-conducted questionnaires developed for other censuses and surveys as well as lessons learned in Census 2000. According to its May 8, 2008 plans for conducting the paper-based nonresponse follow-up, the Bureau will conduct this cognitive and usability testing in early summer 2008 and the testing will address both respondent interactions and ease of use for the census enumerators. The Bureau expects the questionnaire will have space for up to six people as in Census 2000 and will link other household members to the address via a continuation form; include coverage questions; meet the Decennial Response Integration System data capture specifications; and collect data on the outcome of the enumeration.

Not being able to test the paper-based nonresponse follow-up in the 2008 Dress Rehearsal introduces risk because the dress rehearsal will no longer be a dry-run of the decennial census. While the Bureau has carried out a paper-based follow-up operation in the past, there are now new procedures and system interfaces that, as a result of its exclusion from the dress rehearsal, will not be tested under census-like conditions. We discussed the nonresponse follow-up plan with Bureau officials and they acknowledge the importance of testing new and changed activities of nonresponse follow-up as well as system interfaces to reduce risk. However, because plans have changed for many aspects of the nonresponse follow-up operation, Bureau officials are uncertain about testing and are still trying to determine which activities and interfaces will be tested and when that testing will occur.

It is important to note that the Bureau has taken some important initial steps to manage the replannning effort. For example, the Bureau has added temporary “action officers” to its 2010 governance structure. As of April 17, 2008, six action officers had been identified to achieve the six objectives in its Recovery Plan—nonresponse follow-up replan, reduce FDCA risk, improve communications, document decennial program testing, improve program management, and baseline an integrated schedule. Each action officer is assigned to one of the objectives. These action officers are intended to be catalysts, liaisons, and facilitators responsible for ensuring that the tasks and milestones for each objective are met. Also, the action officers meet with the Associate and Assistant Directors to facilitate quick decision-making and on a regular basis provide updates on the status of plans. Weekly, the Bureau’s Director meets with the Department of Commerce’s Deputy Secretary to discuss the status of the replan for the 2010 Census.

The Bureau has also issued documents that describe actions it will take to identify and manage risk. The Bureau’s 2010 Census Program Management Plan, issued May 5, 2008, contains information about the risk management process and notes that 24 program-level or high level-risks have been identified, were currently being validated, and that each of these 24 risks would have either mitigation or contingency plans associated with them. However, according to Bureau officials, these 24 risks were associated with an automated operation and the Bureau had not yet developed risks related to the paper-based nonresponse follow-up operation. We requested information on these 24 risks, and on June 4, 2008, the Bureau provided us with an updated program-level risk document. The update now includes 25 program-level risks and identifies several risks related to the redesign including late design changes and testing. However, the Bureau has not updated project-level risks—which are risks specific to an operation or system—for nonresponse follow-up since the change to paper was announced. Once the Bureau provides project-level risk documents, we will assess the Bureau’s actions to identify, prioritize, and manage risk for the replanned nonresponse follow-up operation.

Bureau Has Improved Program Management and Oversight, but Cost Estimates Need Timely Reconciling

The Bureau has taken steps to strengthen the FDCA program office leadership and expertise. The Bureau has recently assigned an experienced Bureau manager to manage the FDCA program office. According to the Bureau, the manager has extensive experience in directing major IT projects. The Bureau has also hired an outside IT expert, to provide advice and guidance to the FDCA program office. The Bureau has also implemented key activities to help improve management and transparency of contractor activities. Bureau officials have established a schedule for daily assessment meetings with contractor personnel; are conducting weekly status assessment and resolution meetings with the Deputy Director and Director; and are holding regular meetings with the Department of Commerce.

The Bureau has obtained cost estimates for FDCA from both Harris and MITRE, based on the recent changes to the scope of the program. In particular, these cost estimates include the January 16, 2008 requirements and the decision for a paper-based nonresponse follow-up operation. Harris is estimating that the revised FDCA program will cost roughly $1.3 billion; however, this cost estimate is preliminary and expected to be further refined.

At the direction of the Bureau, MITRE developed an independent government cost estimate in April 2008. MITRE’s estimate is about $726 million, which is nearly $600 million less than the contractor’s rough order of magnitude estimate. A comparison of the two estimates reveals significant differences in two areas: software development and common support. In particular, Harris is estimating that software development will be about $200 million greater than MITRE’s independent estimate; and that common support will be about $300 million greater than MITRE’s estimate.

Software development ($200 million difference): MITRE officials noted that these differences could be attributed to different assumptions based on abnormal software development (such as starts and stops due to budget instability), labor rates used, amount of additional staff needed in order to maintain the schedule and to address quality and testing issues, as well as cost contingency reserves.

Common support ($300 million difference): Although this program element contains the largest cost difference, MITRE officials noted that they could not identify the primary cost drivers that caused the gap. However, possible explanations could be cost contingency reserves that may have been built into the Harris estimate, labor rates used, unexpected high level of change management personnel resulting from budget and requirements changes, and other potential impacts on management resulting from program instability.

Harris had originally planned to deliver the cost estimate by August 20, 2008. However, the Bureau requested that this estimate be delivered sooner and Harris recently agreed to deliver this cost estimate by July 15, 2008. The Bureau and contractor plan to reconcile and agree to a final estimate by August 15, 2008. We plan to analyze the independent cost estimate and the Harris final estimate for the program. As part of this analysis, we intend to evaluate the methodology, as well as underlying assumptions, used to develop each estimate.

The Bureau needs to act swiftly to finalize the FDCA program’s cost estimate and renegotiate the contract. In particular, it will need to have a final cost estimate from Harris in mid-July, and will need to reconcile this estimate with MITRE’s independent estimate thoroughly and quickly to have a final cost estimate by August 15, 2008. Our body of work on the lessons learned on other major IT acquisitions, highlights the importance of establishing realistic cost estimates (through reconciliation of program and independent cost estimates), using fixed price contract techniques for low risk procurement areas, where appropriate, and establishing management reserve funds for unexpected costs. In moving forward, it is important that the Bureau exercise diligence in finalizing the contract terms to ensure that the FDCA program is conducted in a timely and efficient manner for the 2010 decennial.

Bureau’s Integrated Schedule Identified Activities and Associated Milestones but Did Not Address Risks and Costs

The Bureau designed its 2010 Census Integrated Schedule, dated May 22, 2008, to provide information on its schedule framework and activity-level design as well as to describe the program complexity and methods that the Bureau will use to manage the 44 interdependent operations, incorporating over 11,000 unique activities, to conduct the 2010 Census. The Bureau briefed committee staff and us on this final integrated schedule last week. Based on this briefing and our preliminary review of the schedule, we can offer some observations.

The integrated schedule does identify activities that need to be accomplished for the decennial and the Bureau establishes milestones for completing tasks. However, the schedule does not link those activities with associated risks nor does it capture the cost of operations. We previously recommended to manage the 2010 Census and contain costs, the Bureau develop a comprehensive, integrated project plan for the 2010 Census that should include risk and mitigation plans, updated cost estimates, and detailed milestones that identify all significant relationships. We also observed that testing the handheld computer that will be used in the address canvassing operation—an activity we have previously identified as important in mitigating risks associated with use of new technology—overlapped with its deployment. Specifically, in describing the testing and integrating of handheld computers for the address canvassing operation, the schedule indicates that this activity will begin in December 2008 and be completed in late March 2009; however, the deployment of the handheld device for address canvassing will actually start in February 2009, before the completion of testing and integration. It would appear uncertain that the testing and integration milestones would permit modification to technology or operations prior to the onset of operations. Further, the Bureau’s integrated schedule does not specifically define testing (e.g., system, integration, and end-to-end). Separately, the Bureau on June 6, 200 produced a testing plan for the address canvassing operation.

On May 22, 2008, the Bureau also issued the 2010 Census Key Operational Milestone Schedule. This represents a higher level summary of key operations and is linked to the more exhaustive integrated schedule. The Bureau identified about 175 activities that it considers key and that are used by senior management to oversee the 2010 Census. However, there are several notable exceptions to this schedule of key operational milestones. For example, there is no key milestone for identification of program and project risks in light of the significant change in planned operations, nor for developing necessary mitigation or contingency plans. Including key milestones for risk identification and mitigation in its high- level schedule will enable the Bureau to stay focused on activities which can directly impact the quality or cost of the 2010 Census. Nor does the schedule include a milestone for when testing of key activities related to nonresponse follow-up will take place. This is despite the fact that this represents the single largest field operation and will not be part of a dress rehearsal. The Bureau does recognize that it could include in its high-level summary schedule a key milestone for nonresponse follow-up testing activities. Further testing schedules for address canvassing and the operations control system also do not appear as key milestones, though they do appear in the detailed integrated schedule. Including these critical activities as part of the list of key milestones could ensure greater management attention, as well as help in focus oversight. We are currently reviewing in greater detail the summary and integrated schedule of milestones and the recently revised program-level risk document provided on June 4, 2008.

In summary, the Bureau has taken some important steps toward managing the changes it plans for conducting the 2010 Census. Yet much remains uncertain and in the absence of a full dress rehearsal, the risks to a successful decennial census are substantial. Risks are especially high for the 2010 Census nonresponse follow-up operation both because the Bureau will not reap the benefits of having a dress rehearsal for this key operation but also because it is changing its approach late in the decade. These make even more compelling the need for the Bureau to specify what tests it plans to conduct in the absence of a dress rehearsal and when such testing will take place.

The Bureau will also need to take several next steps to finalize the FDCA program’s cost estimate. In particular, it will need to have a final cost estimate from Harris, as soon as possible, in order to have a sufficient amount of time to complete modifications to the contract by the end of the fiscal year. Our body of work on the lessons learned on other major IT acquisitions, highlights the importance of establishing realistic cost estimates (through reconciliation of program and independent cost estimates), using fixed price contract techniques for low risk procurement areas, such as hardware, and establishing management reserve funds for unexpected costs. In moving forward, it is important that the Bureau exercise diligence in finalizing the contract terms to ensure that the FDCA program can be conducted in a timely and efficient manner.

Finally, the Bureau has developed a detailed integrated schedule of activities that need to be conducted during the 2010 Census and established milestones for completing them. It will be important for the Bureau to ensure that among the key milestones and activities that are highlighted for management and oversight are those that represent the greatest impact on the ultimate cost and quality of the 2010 Census.

Mr. Chairmen and members of the committee and subcommittee, this concludes our statement. We would be happy to respond to any questions that you or members of the subcommittee may have at this time.

If you have any questions on matters discussed in this testimony, please contact Mathew J. Scirè at (202) 512-6806 or David A. Powner at (202) 512- 9286 or by email at sciremj@gao.gov or pownerd@gao.gov. Other key contributors to this testimony include Carol Cha, Betty Clark, Vijay D’Souza, Sarah Farkas, Richard Hung, Andrea Levine, Catherine Myrick, Lisa Pearson, Cynthia Scott, and Niti Tandon.

This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.