2010 Census: Census Bureau Has Made Progress on Schedule and Operational Control Tools, but Needs to Prioritize Remaining System Requirements
Published: 2009-11-13T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: To carry out the decennial census, the U.S. Census Bureau (Bureau) conducts a sequence of thousands of activities and numerous operations. As requested, The Government Accountability Office (GAO) examined (1) the Bureau's use of scheduling tools to maintain and monitor progress and (2) the status of two systems key to field data collection: the control system the Bureau will use to manage the work flow for paper-based operations, including nonresponse follow-up, and the system used to manage quality control of two major field operations. GAO applied schedule analysis tools; reviewed Bureau evaluations, planning documents, and other documents on work flow management; and interviewed Bureau officials.
section_title: What GAO Found
paragraphs: The Bureau's master schedule provides a useful tool to gauge progress, identify and address potential problems, and promote accountability as the Bureau carries out the census. GAO found that the Bureau's use of its master schedule generally follows leading scheduling practices that enable such high-level oversight. However, errors GAO found in the Bureau's schedule hinder the Bureau's ability to identify the effects of activity delays and to plan for the unexpected. The Bureau has recently begun taking systematic steps to identify and correct remaining errors. However, within its schedule, the Bureau does not identify the resources needed to complete activities, making it difficult for the Bureau to evaluate the costs of schedule changes or the resource constraints that may occur at peak levels of activity. Leveraging the 2010 scheduling experience and including resource needs in the 2020 schedule should facilitate planning for the 2020 Census, already underway. The automated control system that the Bureau plans to use to help manage major field data collection operations has significant development and testing milestones remaining, with some scheduled to finish shortly before the system needs to be deployed. This aggressive schedule leaves little time for resolving problems that may arise, and without prioritized and final software specifications and reliable progress measures, the Bureau may not get what it needs from the system to conduct the operations. Additionally, development of quality control software for two major field operations faces delays, although detailed specifications and test plans are final.

section_title: Letter
paragraphs: The Constitution mandates that the federal government count the U.S.  population every 10 years—a large and complex undertaking known as the  decennial census. To carry out the census, the U.S. Census Bureau  (Bureau) conducts numerous operations and coordinates a sequence of  thousands of interdependent activities. Given mandated deadlines that the  Bureau faces for delivering census tabulations, the Bureau uses a master  activity schedule to help managers stay on track and alert them to  potential delays. With less than 5 months until Census Day, April 1, 2010,  there is little time remaining for the Bureau to deal with any unexpected  problems that may arise, so early recognition of potential delays is  essential.
The largest census field operation is Nonresponse Follow-up (NRFU),  where field staff visit households that have not mailed back their census  forms. The Bureau will rely on a Paper-Based Operations Control System  (PBOCS) to manage this follow-up operation and other activities  scheduled to be completed in the coming months. Field office managers  will use PBOCS to assign work to roughly 1.2 million temporary  employees in about 500 offices nationwide. Among other activities, field  office managers will also track response data collected by field staff from  an estimated 47 million census forms from households that did not return  their forms by mail.
Another key part of managing upcoming census field operations is  ensuring the quality of the data collected. The quality control operation  helps the Bureau determine whether the first interviews held to collect  census data were done correctly, look for workers who may need  additional training, and identify workers who are falsifying census data.  For some operations, the quality control workload will be determined by  matching and coding software that will identify cases that need follow-up  and provide that case selection information to PBOCS. While the Bureau  developed a similar operations control system for the 2000 Census and has  used the core functionality of the matching software in census-like tests  before, neither PBOCS nor the matching software was used in a “dress  rehearsal” for the decennial held in 2008. PBOCS and its integration with  the matching software has not been fully tested in a census-like  environment.
Because of the importance of staying on schedule to meet mandated  deadlines, effectively managing field operations, and ensuring the quality  of the data collected, as you requested, we examined (1) the Bureau’s use  of scheduling tools to maintain and monitor progress and (2) the status of  two systems critical to conducting field data collection: the control system  the Bureau will use to manage the workflow for paper-based operations  including NRFU, and the system used to manage quality control of two  major field operations.
To meet these objectives, we reviewed Bureau planning documents,  evaluations, schedules, scheduling best practices, and operational and  software specifications. We also interviewed Bureau and contractor staff  regarding the planning, scheduling, and status of selected areas related to  field office workflow management. To address the first objective, we  examined the schedule, comparing its estimates to relevant best practices  and testing its data for reliability using methods described in detail in  appendix I. To address the second objective, we compared the schedule of  Bureau operations to the testing schedule of systems needed to conduct  those operations, examined previous evaluations of those systems, and  evaluated the status of operational and software specifications. We also  reviewed past GAO reports as appropriate to meet our objectives.
We conducted this performance audit from October 2008 to November  2009 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: The Secretary of Commerce is legally required to (1) conduct the census  on April 1 of the decennial year, (2) report the state population counts to  the President for purposes of congressional apportionment by December  31 of the decennial year, and (3) send population tabulations to the states  for purposes of redistricting no later than April 1 of the year following  Census Day. The Bureau has defined over 40 different operations in its  high-level requirements document, describing all of the planned operations  and systems needed to meet these mandates.
For the 2010 Census, the Bureau is using a comprehensive master  schedule to integrate the work to be carried out in the dozens of  operations. The schedule provides a high-level roadmap for Bureau  executives and is used to alert executives to activities that are behind  schedule or experiencing issues, allowing problems to be addressed so the  census can continue to proceed on track. Staying on schedule is crucial to  accomplishing all of the tasks involved in conducting the census. In fact,  scheduling and planning are so important that the Bureau has already  established a high-level schedule for planning the 2020 Census.
While the schedule can be used to manage census operations at a high  level and dictates major time allocations and deadlines, local census  offices across the nation require more detailed plans to conduct  enumeration that exceed the detail included in the master schedule. A  successful census depends, in large part, on the field work carried out in  these local census offices where employees on the ground in local  communities build a list of where to count people and count people who  do not return their census forms. As we have previously reported, the  Bureau had initially planned to carry out major field data collection  activities using hand held computing devices. Development and  performance problems with the hand held device led the Secretary of  Commerce in April 2008 to abandon using the device for most of its  intended operations and resulted in the Bureau removing the NRFU  operation from the 2008 Dress Rehearsal. As a result, the Bureau was not  able to use the dress rehearsal as a comprehensive end-to-end test of the  interoperability of all of its planned systems, and the Bureau has had to  develop plans to support and conduct the affected operations on paper as  it did for the 2000 Census.
For the 2010 Census, the Bureau will manage remaining fieldwork  activities with PBOCS. This system is intended to provide managers with  essential real-time information, such as worker productivity and  completion rates for field operations. It also allows managers in the field  to assign or reassign cases among workers. If the system does not work as  intended, it could hinder or delay field operations and introduce errors  into files containing collected data.
Another responsibility of field offices is implementing the quality control  process established by the Bureau to ensure that correct information is  collected by field staff and data are not falsified. To ensure data quality  and consistency of quality control procedures, the Bureau will manage,  track, match, and review answers provided during re-interview operations  using its Census Matching Review and Coding System (Census MaRCS).  This system is also to designate quality control assignments where  selected households will be re-interviewed in order to determine that the  original enumerator correctly conducted the interview. Census MaRCS is  also to assist in identifying interviews where the data from the re-interview  do not match the data from the original interview, indicating that a  mistake has been made.
Both PBOCS and Census MaRCS are key systems that have not been fully  tested. Since 2005, we have reported concerns with the Bureau’s  management and testing of key information technology systems. In March  2009, we reviewed the status of and plans for the testing of key 2010  Census systems. We reported that while the Bureau has made progress in  conducting systems, integration, and end-to-end testing, critical testing  against baseline requirements still remained to be performed before  systems would be ready to support the 2010 Census and the planning for  the testing needed much improvement. While the Bureau has made  noteworthy progress in gearing up for the enumeration, with less than a  year remaining until Census Day, uncertainties surround the Bureau’s  overall readiness for 2010.
section_title: The Bureau Is Successfully Using Schedule Tools to Track the Implementation of the Census, but Opportunities Exist for Improvement in 2010 and Beyond
paragraphs: The Bureau has implemented processes around its master schedule that  comply with a number of scheduling process criteria that are important to  maintaining a schedule that is a useful management tool. Such a schedule  can provide a road map for systematic execution of a program and the  means by which to gauge progress, identify and address potential  problems, and promote accountability. We have documented the  importance of adhering to these criteria and of implementing associated  best practices in our GAO Cost Estimating and Assessment Guide.  According to these criteria, a schedule should be  comprehensive, with logically sequenced activities spanning the scope  of work to be performed so that the full picture is available to  managers;  current, with the progress on ongoing activities updated regularly so  that managers can readily know the status of the project; and  controlled, with a documented process for changes to the schedule so  that the integrity of the schedule is assured.
The Bureau’s master schedule represents all 44 of the operations  described by the broad requirements for the census in its 2010 Census  Operational Plan. While the Bureau continues to add activities to its  central schedule, by including at least all the activities described in these  broad requirements, the Bureau is ensuring that it has a comprehensive  schedule that will be less likely to miss critical interactions between  operations. The Bureau ensured a complete scope of the schedule with  input from stakeholders throughout the agency, with reviews of previous  schedules, and building on a number of census tests during the decade. As  a result, the schedule is the primary source for senior managers, on a  weekly basis, to determine what census activity is ahead of or behind  schedule and provides a resource for determining any impact to the  overall project of delays in major activities.
The Bureau has documented and implemented a formal process for  keeping the data in the schedule current. Staff within each Bureau division  are responsible for ensuring that schedule activities within their division  have their status updated on a weekly basis. Staff update the actual start  and finish dates, the percentage of an activity completed so far, and  estimates of the time remaining to complete each activity in progress. The  Bureau is recording status information on an average of more than 1,300  activities in the schedule ongoing during any given week, generating  historical data that could provide valuable input to future schedule  estimates.
Finally, the Bureau has implemented a formal change control process that  preserves a baseline of the schedule so that progress can be meaningfully  measured. The Bureau’s criteria for justifying changes are clearly  documented and require approval by a team of senior managers and  acknowledgment of the impact by each affected team within the Bureau.  Since the master schedule was baselined in May 2008, about 300 changes  have been approved. Even corrections to the schedule for known errors,  such as incorrect links between activities, must be approved through the  change control process, helping to ensure the integrity of the schedule.
In addition to these practices, the Bureau has positioned itself to monitor  the schedule regularly to help ensure that the census is progressing and  that work is being completed as planned. A central team of staff working  with the schedule implements a process that begins with the weekly  updates of the schedule status and involves subject matter experts from  multiple divisions, and monitors and resolves schedule-related issues,  resulting in a weekly briefing to the Deputy Director and the Director of  the Census, which includes documented explanations for critical activities  scheduled to start late. For example, the central team began reporting  from the master schedule in March 2009 that the printing of questionnaires  for a field operation to validate locations of group quarters in September  and October 2009 might be running late. The schedule showed that late  printing of the questionnaires would trigger their late delivery and the late  assembly of job assistance kits needed to support the operation, and thus  put the timeliness of the operation in danger. According to a Bureau  official, the Bureau then addressed the issue by deciding to unlink the kit  assembly from the questionnaire printing, allowing kits to begin assembly  on time and having questionnaires delivered directly to field offices when  they were ready, letting the operation begin on time.
subsections: section_title: Bureau Is Removing Logic Errors from Its Master Schedule to Improve Its Reliability
paragraphs: When we began analyzing the Bureau’s master schedule, we discovered a  significant number of activities in the schedule that had either missing or  inaccurate information describing their relationships with other activities  in the schedule. We brought these to the Bureau’s attention, and the  Bureau has begun systematically identifying such activities and correcting  their information in the schedule.
In accordance with scheduling best practices, activities in the schedule  should be linked logically with relationships to other activities that  precede or follow them, and they should be linked in the correct order.  Since reports that the Bureau uses to manage the census depend on the  schedule having been built properly, inconsistent adherence to these  scheduling practices has occasionally created false alarms about the  schedule and created unnecessary work for those who have had to resolve  them.
In our analysis of the Bureau’s schedule, we found that nearly all  relationships between activities are generally in place in the schedule and  some activities in the schedule do not need relationships. However, many  activities appeared in the schedule missing one of their logical  relationships. From January 2009 through August, an average of more than  1,200 of the more than 11,000 activities in the entire schedule were missing  relationships to other activities from either their start or end dates. Each  month, on average, over 1,100 of the over 6,100 as yet not completed  activities were missing relationships. For example, within the Bureau’s  master schedule, an activity listed for receiving finished materials from the  NRFU re-interview operation appeared in the schedule with no  relationship to subsequent activities, making it appear that any delays in  its completion would have no impact on subsequent census activities.  While the absence of such a relationship in the schedule does not imply  that the Bureau would miss the potential impact of any delays in the  completion of this activity, the incidence of a large number of such  missing relationships can confound attempts to trace the chain of impacts  that any delays may have throughout the schedule.
Similarly, we found a small number of activities in the schedule that had  been linked together in the wrong order, so that one activity might appear  to finish before a necessary prior activity had been completed. Such an  incorrect relationship can unnecessarily complicate the use of the  schedule to guide work or measure progress. The number of such  apparent out-of-sequence activities in the entire schedule has decreased  from on average more than 100 each month in January through March to  60 in August.
Since June 2009, Bureau staff have been running structured queries on the  data supporting the master schedule to identify activities with missing or  incorrect data; researching each activity to determine what, if any,  corrections to the data are needed; forwarding proposed changes to  affected activities, operation by operation, to program officials for review;  and submitting changes to the Bureau’s formal change control process.  The Bureau reports that since this concerted effort began to correct such  errors that affect activities in 37 different census operations, the Bureau  had completed research for activities in 15 of the operations and approved  changes in 12 of them by early October 2009. The Bureau also informed us  that this review process involving the program officials responsible for the  logic errors has provided an educational opportunity for the officials to  see how their programs can directly affect others, and as a result has  heightened awareness about the importance of getting schedule  information keyed in correctly.
section_title: Improvements to the Schedule Could Help the Bureau Manage the 2020 Census
paragraphs: A schedule provides an estimate of how long a given work plan will take to  complete. Since the duration of the work described by the activities listed  in a schedule is generally uncertain, a schedule can be analyzed for the  amount of risk that its underlying work plan is exposed to. Schedule risk  analysis—the systematic analysis of the impact of a variety of “what if”  scenarios—is an established best practice to help identify areas of a  schedule that need additional management attention. Conducting a  schedule risk analysis helps establish the level of confidence in meeting  scheduled completion dates and the amount of contingency time needed  for given levels of confidence, and helps identify high-priority risks to a  schedule. The Bureau is tracking risks to the census and managing those  risks on a regular basis, as documented in the 2010 Census Risk  Management Plan, but these data are not being mapped into the schedule  at a level that can be used for a systematic schedule risk analysis.
A well-defined schedule should help identify the amount of human capital  and financial resources that are needed to execute the programs within  the scope of the schedule, providing a real-time link between time and cost  and helping to reduce uncertainty in cost estimates and the risk of cost  overruns. However, the Bureau does not link within its schedule estimates  of resource requirements—such as labor hours and materials—to  respective activities. Having this information linked in a schedule  enhances an organization’s capability to monitor, manage, and understand  resource productivity; plan for the availability of required resources; and  understand and report cost and staffing requirements. For example, if the  Bureau were to find itself behind schedule with major operations to be  completed, and resource requirements were linked in the schedule, the  Bureau could then better assess the trade-offs between either adding more  resources or reducing the scope of the operations. In addition, when  resources are linked to activities in the schedule, scheduling tools can  identify periods of their peak usage and assist managers with reordering  activities to level out demands on potentially scarce or costly resources.  When we met with Bureau officials and discussed this, they pointed out  that incorporating this schedule best practice would be difficult to do late  in the preparations for 2010, but they expressed interest in incorporating  this schedule best practice as a step forward in the Bureau’s use of the  schedule to manage decennial censuses.
Finally, the Bureau’s use of a master schedule in 2010 that is, according to  the Bureau, more highly integrated into the management of the decennial  census provides an opportunity to draw many potential lessons for 2020.  The Bureau learned lessons from its use of the 2000 master schedule as  documented in a 2003 Bureau management evaluation, in particular with  its adoption of the formal change control process implemented for 2010.  Yet as noted in the evaluation, there were questions about the quality of  the data maintained in the schedule. Without a reliable change control  process, the schedule did not provide a reliable baseline, making  evaluation of schedule and activity duration estimates difficult, if not  impossible. The Bureau is generating a large amount of data—and  experience—with its efforts in developing, maintaining, and using the 2010  master schedule. Unless the Bureau prioritizes the need for documenting  lessons learned from the current experience—as it did for the 2000  Census—and formally puts in place an effort to capture and analyze  schedule data, changes to baselines, and variances between estimated and  actual durations, it runs the risk of missing out on another opportunity for  using additional lessons some of its staff may already be learning.
section_title: Bureau Plans to Deliver Two Key Systems on a Tight Schedule
paragraphs: The automated control system that the Bureau plans to use to help manage  the data collection operations of the decennial census still faces significant  development and testing milestones, some of which are scheduled to be  completed just before the system needs to be deployed before respective  field operations begin. As a result, should the Bureau encounter any  significant problems during final testing, there will be little time to make  changes before systems are needed to support field operations. PBOCS  will help manage both paper, and people, and it needs to exchange data  successfully with several other Bureau systems, such as one used for  processing payroll. The Bureau plans to complete development and testing  of PBOCS in three major releases, grouping the releases of parts of PBOCS  together loosely by the timing of the field operations those parts are  needed to support. The Bureau already completed a preliminary release of  PBOCS with limited functionality in June 2009 to support some initial  testing.
Figure 1 shows the development, testing, and operation periods for the  three remaining releases and the operations that PBOCS supports.  According to the baseline of the Bureau’s master schedule, PBOCS should  be deployed and operational anywhere from 1 to 6 weeks before each  operation begins for operations leading up to and including NRFU.  According to the Bureau, the system should ideally be ready for use during  training periods so that managers can familiarize themselves with the  system they will have to use and can begin using the system to assign work  to new staff. This also requires that PBOCS be ready in time so that  production data can be loaded into the system, as well as information  about the employees to whom work will be assigned. For example, PBOCS  for NRFU is to finish its final testing in March 2010, about 9 weeks before  NRFU is scheduled to begin on May 1, 2010, and deployment is scheduled  to take place about 6 weeks before NRFU starts, leaving 3 weeks of  contingency time in the event that unexpected problems arise during  PBOCS development. This means that if any significant problems are  identified during the testing phases of PBOCS, there is generally little time  to resolve the problems before the system needs to be deployed. In  addition, it will be more difficult for the Bureau to integrate into PBOCS  training for users on any late changes in the PBOCS software. While the  Bureau relies on last-minute additions to training and procedures  documents to communicate late changes to workers, Bureau officials  agreed that it can be difficult to incorporate such last-minute additions  into training sessions and for users to learn them, and doing so should be  avoided if possible.
The Bureau also faces the significant challenge of developing the detailed  specifications for the software to be developed. As of early-September  2009, the Bureau had established high-level requirements for PBOCS and  has reported completing development of release 1 of PBOCS. The Bureau  reports that as of late-October its requirements development, system  development, and system testing for phase 1 is largely completed.
However, the Bureau has not yet finalized the detailed requirements for  this release or for later releases. High-level requirements describe in  general terms what functions the system will accomplish, such as  producing specific management reports on the progress of specific paper- based operations. Detailed requirements describe more specifically what  needs to be done in order to accomplish such functions. For example, a  detailed requirement would specify the specific data that should be pulled  from the specific data set to produce specific columns of a specific report.  While high-level requirements provide software programmers with general  guidelines on, for example, what types of reports should be produced,  without a clear understanding of the detailed requirements, the  programmers cannot be sure that they are identifying the correct source of  information for producing such reports, and reports can thus be  inaccurate. According to Bureau officials, previous contract programmers  with little decennial census experience and no involvement with current  development efforts made erroneous assumptions about which data to use  when preparing some quality control reports that became problematic in  the dress rehearsal.
Without detailed requirements, the Bureau also cannot be sure how  frequently such reports should be updated or which staff should have  access to which reports. Further, software developers may not have the  required information to meet the Bureau’s needs. Also, as we have  previously reported, detailed operational requirements determine system  development, and without well-defined requirements, systems are at risk  of cost increases, schedule delays, or performance shortfalls. As we have  reported and testified numerous times, the Bureau experienced this with  an earlier contract to automate the support of its field data collection  activity, which included the failed handheld computing device.
The Bureau’s PBOCS development managers have told us that they are  working closely with stakeholders in an iterative process of short  development cycles to help mitigate PBOCS development risks caused by  not having detailed requirements written in advance. Embedding subject  matter experts within the software development process can help mitigate  risk inherent in the short time frame the Bureau has remaining to develop  and test PBOCS. Yet the absence of well-documented and prioritized  detailed requirements for PBOCS, which still need to be developed and  tested, remains among the most significant risks to getting PBOCS ready  on time. Furthermore, the Bureau lacks reliable development progress  measures that permit estimating which requirements may not get  addressed and that are important to ensuring the visibility of the  development program to Bureau leadership. Aggressive monitoring of  system development and testing progress and of the effort remaining will  help ensure that program officials who will rely on these systems can  anticipate what risks they face and what mitigation activities they may  need for shortfalls in the final systems.
In recognition of the serious implications that a failed PBOCS would have  for the conduct of the 2010 Census, and to see whether there were  additional steps that could be taken to mitigate the outstanding risks to  successful PBOCS development and testing, in June 2009 the Bureau  chartered an assessment of PBOCS. The assessment team, chaired by the  Bureau’s Chief Information Officer (CIO), reported initially in late July  2009 and provided an update report in late August 2009. According to the  August update and our discussion of it with the CIO, the team increased its  risk rating in two areas of PBOCS that it is monitoring in part because of  the absence of fully documented requirements, testing plans, progress  measures, and deployment plans.
In its comments on the draft of this report, the Department of Commerce  provided information describing several steps the Bureau was taking to  monitor the progress of PBOCS development. According to Commerce,  the Bureau already does the following:    Daily Project Management Standup meetings, which cover action item  management, calendar review, activity sequencing, and any threats or  action-blocking issues.
Daily Architecture Review Board and team leads meetings.    Weekly Program Management Review Board meetings, and thrice- weekly Product Architecture Review Board meetings.
At least weekly review of progress by the PBOCS Internal Assessment  Team—chaired by the CIO. The team briefs the Bureau Director at  least monthly.
Monthly Quality Assurance Board meetings and twice-monthly Risk  Review Board meetings.
At the end of our review, the PBOCS development team demonstrated two  software tools it said it was using to help manage its iterative process of  short development cycles. We did not fully assess their use of the tools.
The progress measures they demonstrated with one of the tools predicted  that the completion dates would be missed. It also showed that the  development team was underestimating the development effort required to  achieve its iterative development goals. When we noted this, the  presenters told us that the information in their management system was  not current. Until the Bureau completes the detailed requirements for  PBOCS and prioritizes them and its PBOCS development monitoring is  relying on current and reliable progress measures, such as those the  development team attempted to demonstrate to us for estimating the effort  needed to complete remaining development, it will not be able to fully  gauge the PBOCS development progress and have reasonable assurance  that PBOCS will meet the program’s needs. The Bureau is continuing to  examine how improvements will be made.
subsections: section_title: Quality Control Matching and Coding Software for Nonreponse Follow-up and Update/Enumerate Faces Testing Delays and Revisions to Requirements
paragraphs: The Bureau has experienced delays in the development and testing of  software that will play a key role along with PBOCS in controlling and  managing field data collection activity for the quality assurance programs  of NRFU and Update/Enumerate. Census MaRCS will help manage the  process of identifying systematic or regular violations in the door-to-door  data collection procedures. In particular, Census MaRCS will be a tool to  help target additional households needing reinterview as part of the  quality assurance program for these two major census data collection  operations in the field. Therefore, fully developing and testing Census  MaRCS will be important to the successful conduct of the census.
Like other systems at the Bureau, Census MaRCS had to undergo design  changes when the Bureau made the April 2008 decision to switch to paper- based operations. Detailed performance requirements—such as the  information to be included on reports and its sources, including  performance metrics, such as the number of users the system is designed  to handle—are documented and were baselined in May 2009. However,  specifications have been added or clarified as software development has  progressed and improvements have been suggested.
Software development has at times been slower than expected, leading to  delays in some testing. Test plans for Census MaRCS software and  interfaces are in place, having been documented in May 2009. According to  those plans, the Bureau is in the second of three stages of testing Census  MaRCS and is scheduled to complete its final stage in December 2009,  almost 2 months before its first deployment for the Update/Enumerate  operation. Slower-than-expected development has delayed some parts of  the second phase of testing, which will thus finish late according to the  Bureau, but Bureau officials have indicated that they believe that delay  can be absorbed into the schedule, and that the system will be delivered as  scheduled in February 2010.
The compressed testing schedule leaves little time for additional delays in  writing software or conducting tests. The Bureau is working on additional  plans to test the interfaces between systems like PBOCS and Census  MaRCS to ensure that they work together, but those test plans have not yet  been finalized. Since Census MaRCS was not used in the dress rehearsal,  and a full end-to-end system test—that is, a test of whether all interrelated  systems collectively work together as intended in an operational  environment—is not planned for in the time remaining before the system  is required to be deployed, successful testing of the interfaces with other  systems is critical to the system’s readiness. If development or testing  delays persist, it will be more important than ever that system  requirements be prioritized so that effort is spent on the “must haves”  necessary for system operation.
section_title: Conclusions
paragraphs: Our review of the Bureau’s master schedule for conducting the 2010  Census and the processes the Bureau uses to manage it suggests that it is  doing a commendable job conducting such a large and complex  undertaking consistent with leading scheduling practices. Furthermore,  the Bureau’s systematic effort to correct errors that we have identified in  the schedule will further improve the ability of the master schedule to  support senior management oversight and decision making as 2010  approaches. Other improvements, such as embedding estimates of  resource needs into the schedule, may take more time to implement. Yet,  the Bureau’s generally well-defined and integrated schedule provides an  essential road map for the systematic execution of the census and the  means by which to gauge progress, identify and address potential  problems, and promote accountability. Leveraging the Bureau’s  experience with scheduling for 2010 by documenting it should provide  lessons learned for similar efforts in 2020 as well.
Moreover, since we testified on the status of the 2010 Census in March  2009, the Bureau has made progress on a number of key elements needed  to manage the work flow in field operations. In particular, the Bureau has  made progress in developing and testing systems to support paper-based  operations in the wake of the Secretary of Commerce’s April 2008 decision  to switch to paper-based operations for most field data collection activity.  That said, some delays are also occurring, and since so much still remains  to be done in the months leading up to Census Day, the Bureau has limited  time to fix any potential problems that arise in systems that are not  thoroughly tested.
While the Bureau has made significant progress in developing test  schedules for key systems, careful monitoring of the progress in  addressing, and setting priorities among, the remaining detailed  requirements for the control system supporting paper-based operations is  critical for the Bureau to anticipate what risks it faces and mitigations it  may need for shortfalls in the final system. Based on the challenges faced  in the earlier program implementing handheld computing devices, the  Bureau has already experienced the ill effect of having to change its plans  when a system does not fully meet planned program needs. With limited  time before implementation, it is uncertain whether the Bureau may be  able to complete development and fully test all key aspects of its systems,  like PBOCS and Census MaRCS, which are still under development.  Continued aggressive monitoring by the Bureau and improvements in the  progress measures on system development and testing, of effort  remaining, and of the risks relating to these efforts is needed. Such effort  will help ensure that Bureau leadership, as well as Bureau program  officials who will rely on these systems, have early warning on what, if  any, desired system features will be unavailable in the final systems,  maximizing time available to implement mitigation strategies as needed.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Commerce require the Director of  the U.S. Census Bureau to take the following three actions:  To improve the Bureau’s use of its master schedule to manage the 2020  decennial census:  Include estimates of the resources, such as labor, materials, and  overhead costs, in the 2020 integrated schedule for each activity as the  schedule is built, and prepare to carry out other steps as necessary to  conduct systematic schedule risk analyses on the 2020 schedule.
Take steps necessary to evaluate the accuracy of the Bureau’s  baselined schedule and determine what improvements to the Bureau’s  schedule development and management processes can be made for  2020.
To improve the Bureau’s ability to manage paper-based field operations in  the 2010 Decennial Census,  finalize and prioritize detailed requirements and implement reliable  progress reporting on the development of the paper-based operations  control system, including estimates of effort needed to complete  remaining development.
section_title: Agency Comments and Our Evaluation
paragraphs: The Secretary of Commerce provided written comments on a draft of this  report on November 3, 2009. The comments are reprinted in appendix II.  Commerce did not comment on the first two recommendations, but  provided additional information on steps it had already been taking to  monitor progress of PBOCS development related to the third  recommendation. Commerce commented on how we characterized the  status of system development and testing, and provided additional  statements about the status. Commerce also made some suggestions  where additional context or clarification was needed, and where  appropriate we made those changes.
With respect to our third recommendation to finalize and prioritize  detailed requirements and implement reliable progress reporting on  PBOCS, including estimates of effort needed to complete remaining  development, Commerce described numerous regular meetings that the  development team holds, as well as efforts by others in the Bureau to  monitor and report on PBOCS development progress. We added  references to these monitoring efforts within the report. We agree that the  monitoring efforts the department describes can help assess the progress  being made; however, the monitoring efforts can only assess the progress  being reported to them. First, a complete set of requirements is needed to  understand the work that has been accomplished and the work remaining.  As we have noted, the Bureau has not yet developed a full set of  requirements. Second, the management tool’s measurement of the  development activity successfully addressing the requirements is directly  related to the effectiveness of test cases developed for those requirements.  However, as we have previously noted, if a requirement has not been  adequately defined, it is unlikely that a test will discover a defect.  Accordingly, until the Bureau completes the detailed requirements for  PBOCS and prioritizes them it is unable to use these tools to fully gauge its  progress toward meeting the overall project’s goals and objectives of  system development. We clarified our discussion of this in the report and  reworded the recommendation to better focus on the need for reliable  information.
Commerce maintained that our draft report implied that no PBOCS  development had begun and that no testing would be completed until  March 2010. The draft report described development and testing dates that  clearly illustrate that both development and testing have been occurring  over several months leading up to their conclusion. We have included  additional language in the text to clarify that development and testing take  place over a period of time. We have also included a statement from  Commerce that much of this activity has largely been completed for the  first of three phases.
Commerce commented on the accuracy of the dates used in the figures in  the draft report. We verified that the dates we used were the correct dates  that the Bureau had provided to us earlier. We made minor adjustments to  some of the gridlines in the graphic for presentation purposes.
The Bureau also provided additional information on the prior testing of  matching software, the context for NRFU being dropped from the dress  rehearsal, how PBOCS errors could potentially introduce errors into  census files, and contract programmers we reported being involved in  dress rehearsal PBOCS not being the same ones helping the Bureau with  PBOCS development now. We revised the report as appropriate in  response.
Commerce commented on our discussion of the unreliability of  information that the Bureau PBOCS development team provided us during  a demonstration of two tools that it uses to help manage its iterative  process of short development cycles. Commerce described the use of one  of the two tools but not the one whose progress tracking measures were  demonstrated to us and for which the team told us that data were not  current. We have revised the text to more clearly state that more than one  tool was used to demonstrate how development and testing was being  managed by the PBOCS development team, and we have added additional  language to describe the progress information that should have been  current but that was not.
Finally, as we had noted in the draft report, the Bureau has already begun  taking action to address errors we had identified in its master schedule.  Since we sent a draft of this report to Commerce, the Bureau provided  additional information on the status of that effort, and we have updated  this report accordingly.
We are sending copies of this report to the Secretary of Commerce, the  Director of the U.S. Census Bureau, and interested congressional  committees. The report also is available at no charge on GAO’s Web site at  http://www.gao.gov.
If you have any questions about this report please contact me at (202) 512- 2757 or goldenkoffr@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made major contributions to this report are  listed in appendix III.
section_title: Appendix I: Scope and Methodology for Reviewing the Bureau’s Schedule
paragraphs: We reviewed the U.S. Census Bureau (Bureau) program’s schedule  estimates and compared them with relevant best practices in GAO  Cost Estimating and Assessment Guide: Best Practices for  Developing and Managing Capital Program Cost to determine the  extent to which they reflect key practices that are fundamental to  having a reliable schedule. These practices address whether the  schedule is  comprehensive, with logically sequenced activities spanning the scope  of work to be performed so that the full picture is available to  managers;  current, with progress on ongoing activities updated regularly so that  managers can readily know the status of the project; and  controlled, with a documented process for changes to the schedule so  that the integrity of the schedule is ensured.
In doing so, we independently assessed a copy of the program’s integrated  master schedule and its underlying schedules against our best practices.  We also interviewed knowledgeable program officials to discuss their use  of best practices in creating the program’s current schedule and we  attended a schedule walk-through to better understand how the schedule  was constructed and maintained.
We tested the Bureau’s schedule data for reliability by  running a schedule check report in Pertmaster which is a scheduling  analysis software tool that identifies missing logic, constraints, and so  forth;    using the schedule information from Pertmaster, copying the schedule  data into Excel, and checking for specific problems that could hinder  the schedule’s ability to dynamically respond to changes;  examining whether there were any open-ended activities (i.e., activities  with no predecessors, successors, or both);  searching for activities with poor logic;  identifying whether there were any lags or leads that should only be  used to show how two tasks interact and not to represent work;    determining if activities were resource loaded, which helps to cost out  the schedule and examine whether resources are overstretched or not  available when needed;  examining whether the schedule was baselined, when it had its status  updated, and what deviations there were from the plan; and  examining if there were any actual start or finish dates recorded in the  future and whether there was any broken logic.
section_title: Appendix II: Comments from the Department of Commerce
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: In addition to the contact named above, Ty Mitchell, Assistant Director;  Virginia Chanley; Vijay D’Souza; Jason Lee; Andrea Levine; Donna Miller;  Crystal Robinson; Jessica Thomsen; Jonathon Ticehurst; and Katherine  Wulff made key contributions to this report.
section_title: Related GAO Products
paragraphs: 2010 Census: Census Bureau Continues to Make Progress in Mitigating  Risks to a Successful Enumeration, but Still Faces Various Challenges.  GAO-10-132T. Washington, D.C.: October 7, 2009.  2010 Census: Fundamental Building Blocks of a Successful  Enumeration Face Challenges. GAO-09-430T. Washington, D.C.: March 5,  2009.
Information Technology: Census Bureau Testing of 2010 Decennial  Systems Can Be Strengthened. GAO-09-262. Washington, D.C.: March 5,  2009.
Census 2010: Census Bureau’s Decision to Continue with Handheld  Computers for Address Canvassing Makes Planning and Testing  Critical. GAO-08-936. Washington, D.C.: July 31, 2008.
Census 2010: Census at Critical Juncture for Implementing Risk  Reduction Strategies. GAO-08-659T. Washington, D.C.: April 9, 2008.
Information Technology: Census Bureau Needs to Improve Its Risk  Management of Decennial Systems. GAO-08-79. Washington, D.C.:  October 5, 2007.  2010 Census: Basic Design Has Potential, but Remaining Challenges  Need Prompt Resolution. GAO-05-9. Washington, D.C.: January 12, 2005.