2020 Census: Initial Research Milestones Generally Met but Plans Needed to Mitigate Highest Risks
Released: 2012-12-07T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: GAO's prior work has shown that it will be important for the Bureau to reexamine the design of the census in order to ensure a cost effective census in 2020. As requested, this report evaluates the Bureau's efforts to improve the cost-effectiveness of the enumeration, paying particular attention to the following three key efforts: (1) leveraging the Internet to increase self-response; (2) improving how the Bureau builds its address and mapping databases, including a possible move to targeted address canvassing, and use of private-sector geographic data; (3) and using administrative records to reduce nonresponse follow-up costs. This report (1) identifies what opportunities and risks, if any, the Bureau might need to consider for these efforts going forward and (2) examines to what extent these three efforts are on track with respect to scheduling, resources, and other performance metrics. To meet these objectives, GAO reviewed Bureau documents and interviewed officials.

section_title: What GAO Found
paragraphs: According to U.S. Census Bureau (Bureau) officials, to inform timely design decisions, research on new methods to improve the cost effectiveness of the 2020 Census must be accomplished early enough in the decade to confirm their likely impact on both cost and quality. Three key efforts--(1) the use of the Internet as a response option, (2) a potential move towards targeted address canvassing, and (3) the possible use of administrative records to replace data collected during census field operations--present the Bureau with potential opportunities to reduce costs while maintaining quality. The Bureau's 2020 Research and Testing Program has 14 fiscal year 2012 projects focused on informing design decisions related to the three key efforts. Bureau officials are also aware that the changes they are testing come with many risks, and for each project the Bureau has identified a number of risks and prioritized them from high to low. However, the Bureau has not developed mitigation or contingency plans for these project risks. For example, there are several risks, including tight time frames and accurate cost information, without mitigation and contingency plans. Additionally, GAO found that the Bureau had not developed cost estimates for any of its 2020 research and testing projects as required in guidance provided to project teams. Unreliable cost estimates was one of the reasons the 2010 Census was placed on GAO's high-risk list. Without timely cost estimates, it will be difficult for the Bureau to ensure that resources are adequate to support the research and testing program.

The Bureau met its internal deadline for submitting each of its 14 research project's plans and charters. However, not all the project plans were complete. For example, some project teams did not fully document the types of skills needed or perform a skills gap assessment to determine the resources needed to carry out their respective projects as required in the Bureau's planning template. According to Bureau officials, they are working to document these skills sets, as well as any gaps in skills. Completing this analysis is important to ensuring sufficient resources are available for conducting the research and testing projects. Additionally, performance metric documentation for several projects was incomplete. According to Bureau guidance, project teams were to provide performance metrics for measuring progress and for determining the project's final outcome. However, one team did not provide either of these required performance metrics, while six other teams did not include performance metrics that could be used to monitor research and testing progress. Absent these metrics, the Bureau does not have the assurance that it will be able to avoid potential schedule slips or the certainty as to whether project outcomes will be adequate for making decisions.

section_title: What GAO Recommends
paragraphs: GAO recommends that the Acting Census Director take a number of actions to improve the Bureau's research and testing for the 2020 Census, such as developing risk mitigation plans, contingency plans and cost estimates for each project, and performance metrics and skill sets for those projects that do not have them. The Department of Commerce concurred with GAO's findings and recommendations and provided one clarification, which was included in the final report. Also, the Bureau in its comments noted that it has begun to address GAO's recommendations.

section_title: Letter
paragraphs: The 2010 Census, at around $13 billion, was the most expensive  headcount in our nation’s history. Meanwhile, the cost of conducting the  census has, on average, nearly doubled each decade since 1970 in  constant 2010 dollars. If that growth rate continues, the 2020 Census  could potentially cost around $25 billion. The U.S. Census Bureau  (Bureau) has recognized the need for more cost-effective approaches to  conducting the enumeration and is committed to identifying and  implementing innovations and improvements as necessary to conduct the  2020 Census at a lower cost per housing unit than the approximately  $100 per housing unit cost of the 2010 Census (in constant 2010 dollars)  while still maintaining high quality. Between fiscal years 2012 and 2014,  the Bureau will conduct its research and testing phase for the 2020  Census in order to develop an initial design by September 2014. The  decisions made in the decade’s first years shape the design and set the  framework for later decisions that will affect how the census will be  ultimately conducted.

As requested, this report evaluates the Bureau’s research efforts to  improve the cost-effectiveness of the 2020 Census enumeration, paying  particular attention to the following three key efforts that past Bureau  research has shown could result in substantial cost savings:  leveraging the Internet to increase self-response;  improving how the Bureau builds its address and mapping databases,  including a possible move to targeted address canvassing (as  opposed to the full address canvassing used in earlier censuses,  where field staff walked nearly every street in the country to verify the  address list), and use of private-sector geographic data; and using administrative records,nonresponse follow-up costs. such as tax data, to reduce  Our objectives were to: (1) identify what opportunities and risks, if any,  the Bureau might need to consider for these research efforts going  forward and (2) examine to what extent these three research efforts are  on track with respect to scheduling, resources, and other performance  metrics.

To meet these objectives, we reviewed documentary evidence, including  2020 Census research and testing plans, strategic framework documents,  and our prior reports on the subject; and conducted interviews with  cognizant Bureau officials, as well as knowledgeable stakeholders, such  as those at the National Academy of Sciences, to determine opportunities  and risks related to the Internet, administrative records and the address  and mapping databases. We also obtained and compared Bureau- developed research planning requirements to research deliverables in  order to identify gaps in the Bureau’s research and testing efforts as it  relates to schedule, resources, and performance metrics. Additionally, we  interviewed agency officials to identify the Bureau’s 2020 research and  testing projects, timelines, and benchmarks. We obtained and reviewed  contract-related documents related to geographic data and services, as  well as interviewed Bureau officials to determine the Bureau’s plan for  using the private sector to assist with the Bureau’s address and mapping  needs. For more information on our scope and methodology see  appendix I.

We conducted this performance audit from October 2011 to September  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: The Bureau’s 2020 Census early research agenda is focused on  conducting the 2020 Census at a lower cost than the 2010 Census while  maintaining high quality results. In order to do this the Bureau has  developed a range of design alternatives for the 2020 Census. The  Bureau’s 2020 design alternatives have potential for containing costs but  also have the potential to increase risks. The design alternatives focus on  options to use the Internet and other social media to increase response  rate, target address canvassing, and expand the use of administrative  records. Figure 1 shows the current range of six 2020 Census design  alternatives currently being considered and how the three key efforts fit  into them. According to the Bureau, the final 2020 design is likely to  incorporate both existing approaches as well as design alternatives that  have never been used in the decennial census. Greater changes to the  overall design will result in greater potential for cost savings. However,  greater design changes also could result in greater risk, and testing will  be needed to identify the risks, costs, and benefits of any new  approaches. According to the Bureau, alternative one has the lowest risk  because it most closely mirrors the 2010 Census design (mailout of  census forms, and in-person follow-up of non respondents) and is not  dependent on implementing innovations such as increased use of  administrative records and targeted address canvassing.

The Bureau formally kicked off initial planning for the 2020 Census on  October 3, 2008, with the 2020 Census Planning Summit. At that  meeting, decennial-census managers and subject matter experts  discussed challenges, strengths, and weaknesses that would need to be  considered during the 2020 Census strategic decision-making process.  From 2009 to 2011, during the options analysis phase of 2020 Census  planning, Bureau officials went through a research topic winnowing  process where they: identified cost and quality drivers, conducted  brainstorming sessions, received expert input (e.g., National Academy of  Sciences), cataloged over 600 ideas, clustered those ideas into over 100  projects with research questions, and then prioritized them against the   for the 2020 Census. The Bureau identified  Bureau’s Guiding Principles26 research projects to inform preliminary design decisions with the goal  of conducting the 2020 Census at a lower cost than the 2010 Census,  while maintaining high quality.

Using the 26 research projects, Bureau officials plan to validate the  feasibility of the various strategies for reducing costs between fiscal years  2012 and 2014. Bureau officials stated that they are planning to narrow  down the design alternatives by September 2014, and that research  would continue into the next phase. Bureau officials believe that  conducting research earlier in the decade will better position them to  make more effective changes to the 2020 Census design. Figure 2 shows  the five phases of the life cycle for the 2020 Census.

The Bureau is now in the “Early Research and Testing” phase and is  currently refining and developing its 26 project plans. These plans  include, among other requirements, project goals, objectives, scope,  limitations, and skills required, and had a set of required iterative research  and testing deliverables due 30, 60, 90, and 120 days after project kickoff.  According to Bureau documentation, this process encouraged the  development and proper consideration of ideas. The Bureau selected 17  of the 26 projects to start in fiscal year 2012. The remaining projects are  scheduled to begin in fiscal years 2013 and 2014. Of the 17 research  projects the Bureau is conducting in 2012, 14 projects are to inform  design decisions related to the Internet, administrative records, and   targeted address canvassing. See figure 3 for a list of these 14 research  projects and their objectives. According to the Bureau’s timeline, project  teams are to complete their work on these 14 projects by September  2014.

Move mouse over each activity to see more information afterwards CLICK          to close the open pop-up information display panel.

To print text version of this graphic, go to appendix II.

section_title: Key Efforts Present the Bureau with Both Opportunities and Risks
paragraphs: Research on new methods likely to result in a more cost-effective 2020  Census must be accomplished early enough in the decade to confirm  their likely impact on both cost and quality and to inform timely design  decisions. The use of the Internet as a response option, a potential move  towards targeted address canvassing, and the possible use of  administrative records to replace data collected during census field  operations present the Bureau with opportunities to reduce costs while  maintaining a high quality census. As part of the 2020 Census planning  process the Bureau has identified both program-level and project-level  risks associated with these options. According to the Bureau, program- level risks span the entire 2020 research and testing phase and if not  managed properly could jeopardize the phase’s goals and objectives. In  contrast, project-level risks pertain to the successful completion of  specific projects. However, the Bureau has not developed mitigation and  contingency plans for the project-level risks. Also, the Bureau has not  developed cost estimates for any of its 2020 research and testing projects  as required in guidance provided to project teams. Finally, some of the  project plans we reviewed were incomplete. For example, not all project  plans identified the necessary staff resources, while other project plans  failed to provide performance metrics for measuring progress and for  determining the project’s final outcome.
subsections: section_title: All Three Key Efforts Present the Bureau with Cost Saving Opportunities
subsections: section_title: Internet
paragraphs: During the 2000 Census, the Bureau piloted an Internet response option  that had a limited number of respondents. They considered building on  this 2000 experience for the 2010 Census, but the Bureau decided in July  2006 not to include the Internet option in the design for the 2010 Census  because test results indicated that the Internet option did not increase the  overall response rate, including response among hard-to-count population  groups, and they had underestimated the costs of the contract that  included developing the Internet option.

However, more recent tests conducted in 2011 showed that adding an  Internet response option increased the overall response rate. The 2011  test results, coupled with the increased prevalence and accessibility of  the Internet, led Bureau officials to commit to providing an Internet  response option for the 2020 Census. With regard to the cost for the 2020  Census, if this option can help achieve an overall increase in the  response rate it can save money, since Bureau field staff would need to  visit fewer households during nonresponse follow-up (NRFU), which is  the largest and most costly census field operation. Furthermore, testing  has shown that the cost of an Internet survey is low compared to a mail  survey, which incurs printing and postage costs. Moreover, web survey  responses are generally available more quickly and are of better quality  than responses from a mail survey because there is no lag time as the  responses are captured in real time and there are reminders to prompt  the respondent if a question is unanswered. Quicker and complete  responses can also help reduce the amount of time and money spent on  following up on late or incomplete census forms.

In April and November 2011, the Bureau tested an Internet response  option in its American Community Survey (ACS), and beginning in January 2013, the Bureau plans to offer it as a response option. Bureau  officials have stated that they intend to build on this existing information  technology infrastructure in order to reduce the cost of implementing an  Internet option for the 2020 Census.

Given the Bureau’s decision to move forward with an Internet design  option, six research and testing projects are focused on how best to  implement an Internet response option in concert with other self-response  options. The project team with primary responsibility for coordinating the  design of the Internet response option is tasked with answering the  following research question:

How does the Bureau leverage technology and new response modes  (including the Internet) to increase self-response, improve  nonresponse follow-up data collection strategies, while maintaining  overall quality?

One objective of this project is to determine what contact strategy should  be used to inform the public that census responses can be filed via the  Internet in order to increase self-response (i.e., Bureau use of email,  social networks or phone for initial contact or for reminders). Contact  strategy testing will examine the impact of various Bureau instructions to  respondents, as well as the timing of the delivery of the mailing pieces, on  such factors as response rates, return rates, percentage of Internet  returns, and speed of returns. This project will also determine what mode  (paper, Internet, or phone) is best for each demographic, geographic, and  language group, as well as determine the effect a lack of access to  technology has on each demographic group. Additionally this project will  refine and test security features. The other five projects will cover such  areas as types of languages that will be offered by response mode, public  perception and potential concerns about an Internet response, and  management of the workload for multiple modes of response.

In the 2010 and earlier censuses, the Bureau mounted a full address  canvassing operation, where field staff verified every housing unit in the  nation to update the Master Address File (MAF) and the associated  mapping system called TIGER® (Topologically Integrated Geographic  Encoding and Referencing). This labor-intensive effort was one of the  more expensive components of the 2010 Census: the 2010 Address  Canvassing Operation required 140,000 temporary workers to verify 145  million addresses (by going door-to-door) at a cost of $444 million, or 3  percent of the $13 billion total cost of the 2010 Census. One of the  reasons the Bureau conducted a full address canvassing for the 2010  Census was because it wanted to capture hidden and hard-to-capture  addresses, such as housing units in converted garages and large storage  sheds. For the 2020 Census the Bureau would like to reduce the amount  of field work and cost of address and map updates by targeting the  address canvassing operation. According to Bureau planning documents,  through early research and testing the Bureau plans to attempt to  determine the quality and stability of its address file in terms of the  number of changes (adds, deletes, duplicates, and corrected addresses)  that took place to each census block in 2010. Depending on that  outcome, the Bureau would target for address canvassing only those  blocks where the quality of the addresses was determined to be  inadequate.

To determine whether to target address canvassing for the 2020 Census,  the Bureau has two ongoing efforts to improve the Bureau’s map and  address databases. First, the Bureau’s Geography Division is working  with the United States Postal Service (USPS) and other federal agencies  on a new program called the Geographic Support Systems (GSS)  Initiative, which allows government agencies at all levels to regularly  share and continuously update their address lists with the Bureau.  According to current plans, USPS will continue providing the Bureau with  address updates. Likewise, tribal, state, and local governments, which  maintain address lists for purposes such as emergency response and  property assessment, will also have the opportunity to share addresses  with the Bureau throughout the decade, rather than solely 2 years prior to  the decennial, as had been the case in prior decennial censuses.  Additionally, the GSS Initiative is working to develop, and in some cases  improve, methods for identifying and capturing hidden and hard-to- capture addresses.

As part of this first effort, the Bureau is investigating the role and possible  contributions the private sector can make in improving its address and  mapping databases. For the 2010 Census the Bureau relied heavily on  the private sector to update addresses and maps. Specifically, in June  2002, the Bureau awarded an 8-year contract of about $200 million to  improve the accuracy of its mapping database and investigate options  and methods of updating its address database. The contract, among  other tasks, corrected in TIGER® the location of streets, boundaries, and  other map features so that coordinates would be aligned with their true  geographic locations. According to Bureau officials, continued reliance on  the private sector is necessary in order to maintain the major upgrades  that were made to its address and mapping databases last decennial.  The Geography Division has nine contracts in place, totaling about $90  million dollars, to help it manage its mapping and address database (see  app. III for a list of those contracts). Some of the work being done by the  private sector includes a service contract to provide system upgrades and  maintenance to the MAF/TIGER® database and a data entry services  contract to update the MAF/TIGER® database with address information  provided by state and local governments.

Moreover, to ensure the most current address and mapping data are  available for the 2020 Census, the Bureau, in April 2012, began  conducting market research with three private-sector firms to assess how  those firms might assist the Bureau with its address and mapping needs.  Possibilities to be explored include how to best maintain address and  map data, understand private-sector innovation and technology, and  options for address and map data-sharing methods. According to Bureau  officials, negotiations concerning the scope, tasks, and activities to be  completed with the first contractor have taken place. However,  negotiations with the remaining two firms have not begun. Use of the  private sector for mapping is an area we have previously reported needs  to be explored. Specifically, in December 2010, we listed a number of  reexamination areas which have particular implications for controlling  costs and improving accuracy and raised the question: To what extent  can private-sector and other sources of information such as maps,  address lists, and geographic databases be employed to help support the  census?

The second effort to improve the Bureau’s address and mapping  database is centered on the 2020 Research and Testing Program. There  are three research projects designed to inform the Bureau’s decision  about the extent to which it might be able to conduct a targeted address  canvassing operation. Two projects will use modeling to predict where  coverage errors (e.g., missed or duplicate addresses) occur in the MAF.  These models will be used to assess the quality of data sources, types of  errors, and what information, such as a missing map location for an  address, is not in the MAF. If the Bureau decides to conduct targeted  address canvassing, this information will be used to determine which  areas meet acceptable quality standards, as well as where targeted  address canvassing would be effective. According to the Bureau,  collectively the two projects are to answer the following question:

As the related GSS Initiative proceeds, how will the Bureau determine  the required level of quality needed in the address frame to conduct  an accurate census and then measure the quality of the continually  updated MAF for that purpose?

The third research project in this effort, the Local Update of Census  Addresses (LUCA) Program Improvement project, will examine how to  modify the Bureau’s existing partnership program in consultation with the  GSS Initiative. The LUCA program is a partnership program that provides  an opportunity for local and tribal governments to review and update  individual address information or block-by-block address counts from the  MAF and associated geographic information in the TIGER® database just  prior to the decennial. The goal of LUCA is to improve the completeness  and accuracy of address information. This research project’s objective is  to determine the best way to modify and improve LUCA procedures to  ensure compatibility with continual updating of address information  throughout the decade. The project will also determine whether it is  necessary for local and tribal governments to actively participate in LUCA,  a once-a-decade operation, if they have been submitting addresses and  mapping data continuously throughout the decade as part of the GSS  Initiative.

Administrative records are a growing source of information on individuals  and households. For purposes of the decennial census, the Bureau is  considering administrative records from government agencies, including  tax data and Medicare records, as well as commercial sources to identify  persons associated with a particular household address. During the  2010 Census, the Bureau made limited use of administrative records. For  example, the Bureau used USPS files to update its address list, and  federal agency records (such as those from the Department of Defense)  were used to count military and federal civilian employees stationed  outside of the United States.

The amount of and quality of administrative records the Bureau is able to collect will  impact the amount of cost savings the Bureau is able to realize. could allow the Bureau to reduce the scale of its NRFU operation and  reduce the need for field office space and staff. The 2012 research and  testing projects are also focused on identifying the best available  administrative records to use for address frame building. Administrative  data from other sources would be combined with USPS data, thus  allowing the Bureau to continuously update its MAF throughout the  decade. Finally, administrative records could reduce the cost of quality  control, according to Bureau officials. In order to ensure the accuracy of  the field work conducted during census field operations, such as NRFU,  the Bureau sends quality control staff into the field to verify the work  completed by a random sample of enumerators. Bureau officials are  determining for 2020 whether administrative records can be used as a  quality control check, rather than sending staff into the field.

There are nine research and testing projects that will inform the Bureau’s  decision making on the extent to which the Bureau can expand the use of  administrative records in order to reduce costs. These research and  testing efforts are focused on determining whether the quality of  administrative records is sufficient to be used for this purpose. Initial  testing is to be completed and decisions on these matters are to be made  by September 2014. Of the nine, there are two primary administrative  records research projects. They are tasked with answering the following  research questions:

How can the Bureau best develop and maintain an independent  collection of administrative records and assess the quality of those  records (best sources and methods)?

How can the Bureau leverage administrative records (including  commercial files) to significantly reduce decennial census cost while  maintaining quality?

The objective of one of these projects is to acquire, process, and analyze  administrative records from federal, state, and commercial sources to  assess their utility for the 2020 Census. The other project will research  and test methods to enhance NRFU operations with administrative  records, such as replacing phone or in-person collected data with  administrative data.

section_title: The Bureau Has Identified 2020 Census Program- Level and Project-Level Risks but Has Not Developed Project-Level Mitigation and Contingency Plans
paragraphs: Bureau officials are aware that the changes they want to make to the  decennial census come with many risks. The Bureau has identified and  prioritized from high to low both program-level and project-level risks.  According to the Bureau, program-level risks span the entire 2020  research and testing phase and if not managed properly could jeopardize  the phase’s goals and objectives. In contrast, project-level risks pertain to  the successful completion of projects. In accordance with best risk  management practices, the Bureau has identified and drafted mitigation  plans for all of the 14 program-level risks it has identified. In May 2012 the  Bureau designated two of the 14 program-level risks as high risk:

Timely research and testing results: The 2020 research and testing  strategy involves “many small field tests to research design  alternatives with an accelerated, agile, and informed decision-making  process for incorporating changes to the 2020 Census design.”  According to the Bureau, if the research and testing results are late,  then decisions will not be made on time and the program may not be  ready to move out of the research and testing phase on schedule.  According to the Bureau’s 2020 Census Risk Management Mitigation  Plans, officials have begun implementing a mitigation strategy for this  risk, including documenting clear roles and responsibilities and  communicating them to staff; establishing a schedule that contains  decision milestones and tracking interdependencies of research  results and testing projects; and reviewing ongoing project status  including updates on dependencies, risks, and metrics.

Significant budget cuts: The Bureau reported that if its funding for  2020 Census planning is significantly reduced for fiscal year 2013,  key projects will have to be delayed and as a result the Bureau could  have major technical and operational difficulties making preliminary  design decisions. According to Bureau documents, to mitigate this  risk, Bureau officials have briefed staff from House of Representatives  Subcommittee for Commerce, Justice, Science and Related  Agencies, Committee on Appropriations, and the Office of  Management and Budget on the implications of reduced funding and  plan to brief oversight committee staff. In addition, Bureau officials   reported they have analyzed budgetary needs for fiscal year 2013 and  prioritized projects to manage limited resources.

Related to these risks, a June 2012 congressional hearing held by the  Joint Economic Committee discussed the impact of eliminating or  reducing funding to the ACS. According to Bureau officials, such action  would have a significant impact on the 2020 Census research and testing  program because ACS is one of its testing platforms. This risk is not listed  as a program-level risk, or part of the current risk mitigation plans, but  Bureau managers told us they are strategizing on what they will do if ACS  is not funded and will update the risk mitigation plan accordingly.

The Bureau has identified lack of support from external stakeholders as a  medium program-level risk. Bureau officials are concerned that if external  stakeholders, such as Congress, the Office of Management and Budget,  and the National Academy of Sciences, do not support research on  alternative design options, then the research and testing agenda will have  to be redesigned. To manage this risk, Bureau officials have  communicated the alternative design options to a wide range of external  stakeholders, but are concerned that support may diminish as time  passes and stakeholders change. To mitigate this risk, the Bureau is  engaging external stakeholders in communications regarding 2020  research and testing efforts and seeking feedback on Bureau plans.  According to Bureau documents, the Bureau has met with external  stakeholders, including congressional staff on its House of  Representatives and Senate oversight committees, to discuss 2020  Census planning, and is working on a strategy for keeping stakeholders  informed as plans progress.

Unlike the program-level risks, risk mitigation and contingency plans have  not been drafted to address the project-specific risks. Specifically, each of   that  the Bureau’s research and testing projects has its own risk registeridentifies and prioritizes the risks associated with that project, but no risk  mitigation or contingency plans have been developed for these project- level risks. Several risks appear in multiple project team risk registers,  including tight time frames and accurate cost information. For example,  one Internet-related project listed that there is a risk that if tests are not  timed properly, and if time frames are too tight then they may not be able  to apply what they have learned from one test to the next. Similarly, one  of the administrative records project teams was concerned that one year  to integrate data, build a model, and conduct adequate analysis of  administrative records was “not realistic.” In another example concerning  cost data, two teams stated that without accurate cost data at the project  level, research results will not be sufficient to inform the design of the  2020 Census.

Project teams also identified project-level risks directly related to the use  of administrative records in place of data collected by enumerators during  field visits. Among these risks is access to administrative records to build  a high quality compilation of administrative records for each household  across the country. To accomplish this task, the Bureau needs timely  access to data collected by the federal government and, as previously  mentioned, is currently compiling administrative records from various  federal agencies including the Departments of Housing and Urban  Development and Health and Human Services. However, not all agencies  are authorized to provide their data to the Bureau thus limiting, and in  some cases preventing, the Bureau’s use of that data. Bureau officials  provided us with two examples of agency administrative record sources  that they are not authorized to access: records maintained under the  Family Educational Rights and Privacy Act and the National Directory of  New Hires. Bureau officials said they would like to have access to these  and other data sources for statistical purposes in order to further increase  the number and improve the quality of available administrative records.  According to Bureau officials, reviewing and determining whether they  have access to administrative records with every agency is a time  consuming process and the Bureau would like to have the ability to fully  access and use agency administrative data, especially since the  confidentiality of census data are protected by Title 13.forward, according to a senior Bureau official, legislation might need to be  enacted that would compel federal agencies to provide the Bureau with  access to administrative data for the decennial census. However, the   To move this  enactment of such legislation is likely to be a time consuming process  that raises serious policy concerns including whether the Bureau has  made a sufficiently strong case that it needs access to all federally  collected data and the resulting impact on personal privacy protections.

According to Bureau officials, the project teams did not submit risk  mitigation and contingency plans for the project-level risks as required at  the 90-day checkpoint because after reviewing all of the project-level risk  registers the project leaders first wanted to unduplicate similar risks  across projects in order to prevent redundancies. Bureau officials were  not able to provide a date for when risk mitigation and contingency plans  would be available. However, they stated they are committed to  managing risk in order to avoid the missteps encountered in 2010 when  the Bureau was on our high-risk list due to weaknesses in managing IT,  operational planning, and cost estimating. In our prior work, we reported  that risk mitigation involves identifying, analyzing, prioritizing, and  documenting risks, and ideally more than one alternative should be  assessed. The Department of Commerce’s Inspector General has  recommended that risk management activities begin from the outset of  the current decennial census life cycle, rather than just before field  operations. Without mitigation plans the 2020 Census planning team may  not be able to fully manage risks associated with these projects.

section_title: Project-Level Cost Estimates Have Not Been Developed
paragraphs: The Bureau did not develop cost estimates for any of its 2020 research  and testing projects by the 90-day checkpoint as required in guidance  provided to project teams. The importance of reliable cost estimates is  underscored by the Bureau’s experience leading up to the 2010 Census,  where we found that the Bureau’s cost estimates lacked detailed  documentation on data sources and significant assumptions and was not  comprehensive because it did not include all costs. Partly as a result,  some operations had substantial variances between their initial cost  estimates and their actual costs. For example, the 2010 address  canvassing operation cost $88 million more than the original estimate of  $356 million, an overrun of about 25 percent; and the Bureau’s 2010  NRFU operation cost $1.6 billion, about $660 million, or 29 percent, less  than the Bureau initially estimated.

According to 2020 Census planning documents, at the 90-day checkpoint  project teams were to: (1) prepare and submit cost estimate requests  needed for the project; (2) document estimates for completing the work  for the project; (3) identify and document funds needed for additional  resources and requirements, and (4) prepare and submit requests for  additional funds. According to a senior Bureau official, cost estimates  have not yet been completed because more work needs to be done to  determine project costs and they are in the process of formulating cost  estimates for each project.

section_title: Research and Testing Project Plans Were Delivered on Schedule, but Not All Project-Level Resources and Performance Metrics Have Been Identified
paragraphs: The Bureau’s early research and testing is currently refining the project  plans associated with the14 projects related to the Internet, administrative  records, and targeted address canvassing. Each of the research and  testing projects has a set of required deliverables due 30, 60, 90, and 120  days after project kickoff. We determined that each project team had  provided the 120-day required deliverables on schedule to Bureau  managers in June 2012. The research and testing documents required by  the 2020 Census planning team and delivered by the 14 project teams  describe management and technical plans for each project, and include a  team charter, project plan, resources, preliminary schedule, and field  tests. These documents are important because they serve as the  baseline for each of the research and testing projects.

However, not all resource requirements were documented. For example,  some project teams did not fully document the types of skills needed, or  perform a skills gap assessment to determine the resources needed to  carry out their respective projects as required in the Bureau’s planning  template (the template lays out in detail what is to be included in the  deliverables). Bureau officials plan to use this skills information to allocate  staff to ensure teams have the skills necessary to complete their projects.  While 11 project teams provided all the required information, 2 project  teams conducting research on issues that will inform decisions on  targeted address canvassing and expanded use of administrative  records, respectively, did not provide documentation that a skills gap  analysis was conducted and another project researching expanded use of  administrative records did not provide the necessary skill sets.  According to Bureau officials, they are working to document these skills  sets, as well as any gaps. Completing this analysis is important to  ensuring sufficient resources are available for conducting the research  and testing projects.

We also found incomplete performance metric documentation for several  projects. According to Bureau guidance, project teams were to provide  performance metrics (including the methodology for monitoring and  evaluating project performance), as well as exit criteria for determining  each project’s final outcome. However, the “Enhancing Demographic  Analysis” team that is researching the use of administrative records to  assess the accuracy of census counts at the national level did not provide  any of the required performance metrics, while six teams failed to  include performance metrics that could be used to monitor research and  testing progress. According to Bureau officials, not all teams followed the  project templates, and those teams have been asked to adhere to the  requirements and provide all necessary information. Project performance  metrics are a key element of effective project planning. As we have noted  in prior work on project planning, measurable performance goals should  be identified and performance data should be gathered to determine  progress and whether desired outcomes have been achieved. Absent  such metrics, the Bureau does not have the assurance that it will be able  to avoid potential slips in the research and testing schedule, or certainty  as to whether project outcomes will be adequate for making decisions.

section_title: Conclusion
paragraphs: Just over 7 years remain until Census Day 2020. While this might seem  like an ample amount of time to shore up the Bureau’s planning process  and take steps to control costs, past experience has shown that the chain  of interrelated preparations that need to occur at specific times and in the  right sequence leave little room for delay or missteps. According to the  Bureau, if 2010 operations are repeated in 2020, continued growth in  population size and complexity will likely lead to an unsustainable  increase in census cost. Further, traditional enumeration methods may no  longer effectively produce a high quality census. To contain costs and  maintain quality, bold innovations in both planning for and conducting the  2020 Census will be required.

The early research and testing phase represents a critical stage in  preparing for the 2020 Census. At this time Bureau management is  shaping the next decennial census as it determines what new operations  will be a part of the 2020 Census design, which operations need to be  revised, what risks remain, and how those risks will be mitigated. We  have highlighted some of the cost savings that can be attained through  three new operational changes being considered—the use of the Internet  as a self-response option, replacing enumerator collected data with  administrative records, and targeting only certain addresses for field  verification.

However, these innovations also come with risks that need to be  managed. Specifically, while the Bureau has identified project-level risks  for the 14 projects we reviewed, it has not developed risk mitigation plans  for them. Given the number of changes the Bureau would like to make for  2020 it is imperative that risks are sufficiently managed and that  mitigation plans be in place. For example, the Bureau needs to manage  the risk for accessing federally collected administrative data that it will use  for the 2020 Census. While it currently has access to some federally  collected data, the Bureau would like access to all federally collected  data, which according to the Bureau would require a statutory change.  However, such a change raises serious policy concerns including the  impact on personal privacy protections, and most likely would be a time  consuming process. Therefore, it is imperative that risk mitigation and  contingency plans be in place if the Bureau is unable to successfully gain  access to all federally collected data. Other areas that need attention  include ensuring that all research and testing projects have reliable cost  estimates, so as to avoid research and funding shortfalls, that  performance metrics are identified to ensure project goals are monitored  and met, and that skills are identified to ensure the selection and  assignment of appropriate staff to each project.

section_title: Recommendations for Executive Action
paragraphs: We recommend that the Acting Secretary of Commerce require the Under  Secretary for Economic Affairs who oversees the Economics and  Statistics Administration, as well as the Acting Director of the U.S.  Census Bureau, to take the following three actions to improve the  Bureau’s Research and Testing for the 2020 Census, and thus better  position the Bureau to carry out a cost-effective decennial census:

Develop risk mitigation and contingency plans for all projects to  ensure that risks are adequately managed to minimize their effect on  the project.

Develop cost estimates for each project.

Ensure documentation for projects are complete, including specifying  the performance metrics that will be the basis for determining that  each of the projects has completed its work and identifying skills  needed to inform the selection and assignment of appropriate staff to  each research project.

section_title: Agency Comments and Our Evaluation
paragraphs: The Acting Secretary of Commerce provided written comments on a draft  of this report on October 22, 2012. The comments are reprinted in  appendix IV. The Department of Commerce agreed with the assessments  and recommendations of the report. In addition, the Acting Secretary of  Commerce provided a technical comment and suggestion where  additional context might be needed, and we revised the report to reflect  this comment.

The Bureau in its comments stated that our report accurately represented  the extensive work that has been completed during the early research  and testing phase for the 2020 Census. To address our concerns, the  Bureau noted that it is now focusing on managing risks at the project level  and will begin obtaining more specific cost estimation details for each  project. The Bureau also agrees with our recommendations to ensure that  performance metrics and skill sets are identified for all projects and  teams. Finally, the Bureau stated that these efforts are either already  underway or are planned as a major focus during fiscal year 2013.

As arranged with your offices, unless you publicly announce the contents  of this report earlier, we plan no further distribution until 30 days after the  date of this report. At that time, we will send copies of this report to the  Acting Secretary of Commerce, the Under Secretary of Economic Affairs,  the Acting Director of the U.S. Census Bureau, and interested  congressional committees. The report also is available at no charge on  the GAO website at http://www.gao.gov.

If you have any questions on matters discussed in this report, please  contact me at (202) 512-2757 or by e-mail at goldenkoffr@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. Key contributors to  this report are located in appendix V.

section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report evaluates the U.S. Census Bureau’s (Bureau) efforts to  improve the cost-effectiveness of the enumeration, paying particular  attention to the following three key efforts that past Bureau research has  shown could result in substantial cost savings: (1) leveraging the Internet  to increase self-response; (2) improving how the Bureau builds its  address and mapping databases, including a possible move to targeted  address canvassing, and use of private-sector geographic data; and (3)  using administrative records, such as tax data, to reduce nonresponse  follow-up costs. To meet our objectives of identifying what opportunities  and risks, if any, the Bureau might need to consider for these efforts  going forward, and examining to what extent these three efforts are on  track with respect to scheduling, resources, and other performance  metrics, we reviewed Bureau documents pertaining to the early planning  of the 2020 Census. These generally consisted of the Bureau’s  operational plans, strategic framework documents, strategies and  planning memorandums, timelines, and benchmarks. Many of these  documents were considered draft, but Bureau officials said they were  sufficiently developed for purposes of our review. To describe the  Bureau’s efforts to use private-sector firms to assist with its address and  mapping needs, we spoke to agency officials and reviewed the Bureau’s  current mapping and address contracts and its request for information to  solicit input from private-sector firms. We interviewed Bureau officials  responsible for the early planning of the 2020 Census and inquired about  their process for designing their research and testing program, as well as  opportunities and risks presented by these three key efforts.

We reviewed our prior reports on and other reports on leading practices  to identify gaps in the Bureau’s research and testing efforts as it relates to  schedule, resources, and performance metrics. We also reviewed reports  from and spoke to knowledgeable stakeholders, such as officials at the  National Academy of Sciences to determine opportunities and risks  related to the Internet, administrative records, and the address and  mapping databases. In addition, we spoke with Bureau officials and  reviewed our prior work to identify obstacles the Bureau may need to  address moving forward; for example, legal barriers to the Bureau’s  access to administrative records.

To assess the Bureau’s progress in meeting its stated goals for the  research and testing program, we evaluated whether each of the 14  research projects the Bureau initiated in 2012 had provided the required  deliverables: the 120 day project plan, the project schedule, the project  charter, and the project risk register. We also performed a content  analysis of Bureau planning documents. For this analysis we examined  the deliverables for each of the 14 research projects. We developed a  data collection instrument to record the presence or absence of 11 pieces  of information, or elements, in the deliverables for each project. The 11  elements were identified through a review of the Bureau’s project plan  template which outlines information that each project was expected to  report by the end of the 120-day initiation phase. A coding dictionary was  created to provide a uniform set of coding criteria for each element coded.  Each element could be coded as present, partially present, or absent.  During the coding process, two analysts independently coded documents  for each project using the coding dictionary as a guide. In the next phase,  the two coders met to reconcile any discrepancies between their coding  decisions and then made a final determination for that element. In cases  of disagreement, the engagement methodologist was consulted as a  tiebreaker. Following the coding process, the team analyzed the number  of present and absent elements to identify patterns for each element  across the 14 projects, as well as the completeness of documentation for  each project.

This engagement did not examine the Bureau’s information technology  policies, procedures, and information security in depth because our  information technology team is working on two engagements that cover  these issues.

section_title: Appendix II: Fourteen Research Projects Will Inform Design Decisions on Key Efforts (Text for Interactive Figure 3)
paragraphs: Appendix II: Fourteen Research Projects Will  Inform Design Decisions on Key Efforts (Text  for Interactive Figure 3)

Objectives in Internet development  Develop requirements for the Internet response option and coordinate the relationship  between different response modes (i.e., Internet, paper, telephone)

section_title: Appendix III: Census Mapping and Address Contracts
paragraphs: Appendix III: Census Mapping and Address  Contracts Description of services  Develop, implement, and support a system which allows the Census Bureau  (Bureau) to update its Master Address File (MAF) with TIGER® (Topologically  Integrated Geographic Encoding and Referencing) data, the U.S. mapping data, as  the data is received.

Analyze mapping data as it is provided by state and local governments and other  agencies, and upload changes into TIGER®.

Customize and maintain software so that over 40,000 state, local, and tribal  governments can update and verify the Bureau’s address and mapping databases.

Supply digital map files and digital aerial imagery to maintain and update the MAF/  TIGER® database.

Customize, install, and provide technical support for software to process and edit  files to outline geographic areas, and support mapping applications, including the  use of handheld computers to collect and edit data.

Provide software to match addresses from surveys and other agencies to the MAF/  TIGER® database.

Develop and maintain software which automatically places text on Bureau maps so  all map features are clearly identified.

Update and consolidate the infrastructure, servers and database, of the MAF/  TIGER® system.
subsections: section_title: Total
paragraphs: Provide services for the modernization of the Bureau’s MAF/TIGER® system,  including expansion, design, programming, operation and maintenance of the  infrastructure supporting the MAF/ TIGER® system.

Cost of the most recent purchase.

section_title: Appendix IV: Comments from the Department of Commerce

section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: Other key contributors to this report include Lisa Pearson, Assistant  Director; David Bobruff; Robert Gebhart; Richard Hung; Kirsten B.  Lauber; Andrea Levine; and Timothy Wexler.

section_title: Related GAO Products
paragraphs: 2020 Census: Sustaining Current Reform Efforts Will Be Key to a More  Cost-Effective Enumeration. GAO-12-905T. Washington, D.C.: July 18,  2012.  2020 Census: Additional Steps Are Needed to Build on Early Planning.  GAO-12-626. Washington, D.C.: May 17, 2012.

Decennial Census: Additional Actions Could Improve the Census  Bureau’s Ability to Control Costs for the 2020 Census. GAO-12-80.  Washington, D.C.: January 24, 2012.  2010 Census: Preliminary Lessons Learned Highlight the Need for  Fundamental Reforms. GAO-11-496T. Washington, D.C.: April 6, 2011.  2010 Census: Data Collection Operations Were Generally Completed as  Planned, but Long-standing Challenges Suggest Need for Fundamental  Reforms. GAO-11-193. Washington, D.C.: December 14, 2010.

GAO Cost Estimating and Assessment Guide: Best Practices for  Developing and Managing Capital Program Costs. GAO-09-3SP.  Washington D.C.: March 2, 2009.  2010 Census: Census Bureau Should Take Action to Improve the  Credibility and Accuracy of Its Cost Estimate for the Decennial Census.  GAO-08-554. Washington, D.C.: June 16, 2008.

Information Technology: Significant Problems of Critical Automation  Program Contribute to Risks Facing 2010 Census. GAO-08-550T.  Washington, D.C.: March 5, 2008.  2010 Census: Cost and Design Issues Need to Be Addressed Soon.  GAO-04-37. Washington, D.C.: January 15, 2004.

High-Risk Series: Quick Reference Guide. GAO/HR-97-2. Washington,  D.C.: February 1997.


Question-summary annotations: questions: question: How can the Census Bureau reduce costs while maintaining quality?
answer: Three key efforts--(1) the use of the Internet as a response option, (2) a potential move towards targeted address canvassing, and (3) the possible use of administrative records to replace data collected during census field operations--present the Bureau with potential opportunities to reduce costs while maintaining quality.
child_questions: question: What is the focus of the 2020 Research and Testing Program?
answer: The Bureau's 2020 Research and Testing Program has 14 fiscal year 2012 projects focused on informing design decisions related to the three key efforts.
child_questions: question: To what extent does the Bureau assess the risks of these projects?
answer: Bureau officials are also aware that the changes they are testing come with many risks, and for each project the Bureau has identified a number of risks and prioritized them from high to low. However, the Bureau has not developed mitigation or contingency plans for these project risks. For example, there are several risks, including tight time frames and accurate cost information, without mitigation and contingency plans.

question: To what extent has the Bureau followed the guidance issues to project teams?
answer: Additionally, GAO found that the Bureau had not developed cost estimates for any of its 2020 research and testing projects as required in guidance provided to project teams.
child_questions: question: Why are reliable cost estimates critical?
answer: Unreliable cost estimates was one of the reasons the 2010 Census was placed on GAO's high-risk list. Without timely cost estimates, it will be difficult for the Bureau to ensure that resources are adequate to support the research and testing program.

questions: question: To what extent did the Bureau meet its goals for the research projects?
answer: The Bureau met its internal deadline for submitting each of its 14 research project's plans and charters. However, not all the project plans were complete.
child_questions: question: What flaws existed in the planning?
answer: For example, some project teams did not fully document the types of skills needed or perform a skills gap assessment to determine the resources needed to carry out their respective projects as required in the Bureau's planning template.
child_questions: question: How is the Bureau working to remedy this?
answer: According to Bureau officials, they are working to document these skills sets, as well as any gaps in skills.

question: Why is this important?
answer: Completing this analysis is important to ensuring sufficient resources are available for conducting the research and testing projects.

question: To what extent did project teams provide full performance metric documentation?
answer: However, one team did not provide either of these required performance metrics, while six other teams did not include performance metrics that could be used to monitor research and testing progress.
child_questions: question: Why are these metrics critical?
answer: Absent these metrics, the Bureau does not have the assurance that it will be able to avoid potential schedule slips or the certainty as to whether project outcomes will be adequate for making decisions.

questions: question: What has GAO previously found regarding the Bureau?
answer: GAO's prior work has shown that it will be important for the Bureau to reexamine the design of the census in order to ensure a cost effective census in 2020.

question: What does this report examine?
answer: As requested, this report evaluates the Bureau's efforts to improve the cost-effectiveness of the enumeration, paying particular attention to the following three key efforts: (1) leveraging the Internet to increase self-response; (2) improving how the Bureau builds its address and mapping databases, including a possible move to targeted address canvassing, and use of private-sector geographic data; (3) and using administrative records to reduce nonresponse follow-up costs.
child_questions: question: What aspects of these efforts did GAO assess?
answer: This report (1) identifies what opportunities and risks, if any, the Bureau might need to consider for these efforts going forward and (2) examines to what extent these three efforts are on track with respect to scheduling, resources, and other performance metrics.

question: How did GAO collect information for this report?
answer: To meet these objectives, GAO reviewed Bureau documents and interviewed officials.

questions: question: What did GAO recommend?
answer: GAO recommends that the Acting Census Director take a number of actions to improve the Bureau's research and testing for the 2020 Census, such as developing risk mitigation plans, contingency plans and cost estimates for each project, and performance metrics and skill sets for those projects that do not have them.
child_questions: question: How did the DOC respond to GAO's findings?
answer: The Department of Commerce concurred with GAO's findings and recommendations and provided one clarification, which was included in the final report.

question: How has the Bureau acted on the recommendations?
answer: Also, the Bureau in its comments noted that it has begun to address GAO's recommendations.