2020 Census: Key Challenges Need to Be Addressed to Successfully Enable Internet Response
Published: 2015-02-05T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The U.S. Census Bureau plans to significantly change the methods and technology it uses to count the population with the 2020 Decennial Census, such as offering an option for households to respond to the survey via the Internet. This involves developing and acquiring IT systems and infrastructure to support the collection and processing of Internet response data.
GAO was asked to review the Bureau's efforts to deliver an Internet response option for the 2020 census. GAO's objectives were to (1) describe the Bureau's efforts to identify demographic groups likely to use Internet response and how they compare to historically hard-to-count populations, (2) assess the reliability of estimated costs and savings for Internet response, and (3) determine key challenges associated with delivering an Internet response option. To do this, GAO reviewed Bureau studies, cost estimates, project plans, schedules, and other documentation and compared them against relevant guidance. GAO also interviewed Bureau officials and experts.
section_title: What GAO Found
paragraphs: The U.S. Census Bureau (Bureau) has taken preliminary steps and plans to further examine the impact of introducing an Internet response option on historically hard-to-count segments of the population (these include, but are not limited to, minorities, renters, children, low-income households, and low-education households). For example, the Bureau is applying lessons learned from its implementation of an Internet response option for another household survey, called the American Community Survey, which is conducted on a smaller scale than the decennial census. Additionally, the Bureau is planning two 2020 census field tests in 2015 that are expected to provide data on Internet response rates among different demographic groups, including the historically hard-to-count populations.
The Bureau's preliminary estimated costs of about $73 million for the Internet response option are not reliable because its estimate did not conform to best practices. For example, the estimate has not been updated to reflect significant changes related to the Internet response option that have occurred since it was developed in 2011. Additionally, the unreliability of the Bureau's cost estimate for the Internet response option cast doubt on the reliability of associated potential cost savings estimates. Officials have recognized weaknesses in the Bureau's cost estimate and stated that they plan to update it based on a preliminary decision for the overall design of the 2020 census.
While efforts to deliver an Internet response option are under way, the Bureau faces several scheduling, task, and capability challenges in developing such an option for the 2020 census, including:
Key questions related to estimating the Internet self-response rate and determining the information technology (IT) infrastructure needed to support it may not be answered in time for the preliminary design decision, scheduled for September 2015. Specifically, the Bureau has not developed project plans and research methodologies for answering these questions. In November 2014, officials stated that they had recently begun working on establishing methodologies for answering these questions. However, Bureau officials do not know when the methodologies will be established or when project plans will be updated or created to reflect this new work. Until such plans and methodologies are established, concerns will persist as to whether these two critical questions will be answered in time to inform the design decision in September 2015.
High-level time frames for making decisions related to implementing cloud computing (i.e., a means for enabling on-demand access to shared and scalable pools of computing resources), such as selecting, testing, and implementing a cloud environment that meets the Bureau's scalability, budget, security, and privacy needs, have not been established. While Bureau officials estimated that such time frames will be established around June 2015, until they are established the Bureau will lack assurance that it has enough time to successfully implement a cloud environment prior to system testing, which is to begin in 2018.
section_title: What GAO Recommends
paragraphs: GAO recommends that the Department of Commerce's Census Bureau update estimated costs for the Internet response option and ensure they are reliable, develop methodologies for answering key research questions, and establish high-level time frames for cloud computing decisions. The department neither agreed nor disagreed with the recommendations.

section_title: Letter
paragraphs: One of the U.S. Census Bureau’s (Bureau) most important functions is  conducting the decennial census, which is mandated by the Constitution  and provides data that are vital to the nation. The information collected is  used to apportion seats in the House of Representatives, realign the  boundaries of legislative districts, and allocate billions of dollars in federal  financial assistance. Conducting the decennial census is a major  undertaking, and the Bureau has recognized the need to develop a cost- effective design for the 2020 census.
For the 2020 Decennial Census, the Bureau plans to significantly change  the methods and technology it uses to count the population, such as  offering an option for households to respond to the survey via the  Internet—referred to as the Internet response option. To deliver an  Internet response option, the Bureau needs to, among other things,  design and develop an Internet response application, develop and  acquire the information technology (IT) infrastructure to support the large  volume of data processing and storage, and plan communication and  outreach strategies to motivate households to respond via the Internet.
In light of the Bureau’s decision to use the Internet response option to  collect self-responses for the 2020 Decennial Census, you asked us to  review the Bureau’s efforts to deliver this option. The specific objectives  of our review were to (1) describe the Bureau’s efforts to identify  demographic groups likely to use Internet response and how they  compare to historically hard-to-count populations, (2) assess the reliability  of estimated costs and savings for the Internet response option, and (3)  determine any key challenges associated with delivering an Internet  response option for the 2020 census.
To address the first objective, we reviewed relevant studies conducted by  the Bureau on demographic groups likely to use the Internet response  option and how they compare to historically hard-to-count populations,  and interviewed Bureau officials on their plans to further assess the  impact of an Internet response option on historically hard-to-count  populations. To address the second objective, we analyzed  documentation on the Internet response option portion of the Bureau’s  2020 census rough-order-of-magnitude life-cycle cost estimate. We also  interviewed Bureau officials on its approach for estimating costs and  potential savings for the 2020 census. We compared the Bureau’s  estimating methodology and documentation against best practices for  developing reliable cost estimates identified in GAO’s Cost Estimating  and Assessment Guide. To address the third objective, we interviewed  experts from key Census Bureau advisory committees to obtain their  perspectives on the challenges the Bureau faces in implementing an  Internet response option. We also obtained and reviewed documentation  on the Bureau’s projects supporting the Internet response option, such as  project plans, schedules, risk registers, monthly status reports, and  program management review briefings and compared these documents  against each other and against relevant guidance, such as the Office of  Management and Budget (OMB) and National Institute of Standards and  Technology (NIST) cloud computing guidance, to determine if there were  any inconsistencies. We also interviewed Bureau officials to obtain  information on progress made on the Internet response option. We then  aggregated the key challenges. Appendix I contains further details on our  objectives, scope, and methodology.
We conducted this performance audit from May 2014 to February 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: The Bureau’s mission is to collect and provide comprehensive data about  the nation’s people and economy. Its core activities include conducting  decennial, economic, and government censuses; conducting  demographic and economic surveys; managing international  demographic and socioeconomic databases; providing technical advisory  services to foreign governments; and performing other activities such as  producing official population estimates and projections.
The Bureau is part of the Department of Commerce and is in the  department’s Economics and Statistics Administration, led by the Under  Secretary for Economic Affairs. The Bureau is headed by a Director and  is organized into directorates corresponding to key programmatic and  administrative functions, as depicted in figure 1. Two of these directorates  are responsible for the projects that support the Internet response option  for the 2020 census: (1) Associate Director for 2020 Census and (2)  Associate Director for Information Technology and Chief Information  Officer.
subsections: section_title: 2020 Decennial Census Research and Testing
paragraphs: Bureau officials have established a goal of conducting a high-quality 2020  census at a lower cost than the 2010 census. In order to achieve cost  savings and quality targets for the 2020 Decennial Census, the Bureau  must make fundamental changes to the design, implementation, and  management of the decennial census. Accordingly, the Bureau identified  four key design areas that are intended to enable the Bureau to achieve  its goal: (1) reengineering address canvassing to eliminate a nationwide  field address canvassing effort in 2019, (2) utilizing existing administrative  records to reduce non-response follow-up workload,field operations to more efficiently and effectively manage the 2020  census fieldwork, and (4) optimizing self-response to generate the largest  possible self-response rate, thus eliminating the need to follow-up with  those households. The optimizing self-response design area includes  examining contact strategies and self-response modes to identify which  are best for each demographic, geographic, and language-based group.  This includes giving households the option of responding to the census  through an Internet-based survey—referred to as the Internet response  option. Other self-response modes include mailing paper questionnaires  to households and asking them to complete and mail back the forms, as  well as providing a phone number for households to call and provide their  responses via telephone.
In fiscal year 2012, the Bureau began research and testing alternatives in  the design areas. Key research and testing is expected to continue until  the end of fiscal year 2015 (the end of September 2015), which is when  the Bureau plans to decide how it will design the operations for the 2020  Decennial Census—referred to as the preliminary design decision—and  will produce an updated total life-cycle cost estimate. The Bureau  completed a major field test in 2014, and as of October 2014, was in the  process of finalizing the design of an Optimizing Self-Response test that  is planned for the spring of 2015, and was beginning to design a National  Content Test, which is planned for the fall of 2015. These tests are  intended, in part, to test components of the Internet response option. Following the preliminary design decision, the Bureau plans to conduct  additional research and testing and further refine its design decisions  through 2018. By September 2018, the Bureau plans to have fully  implemented the 2020 design so that it can begin operational readiness  testing. Figure 2 provides the timeline for planned 2020 Decennial  Census research and testing.
The Bureau is planning two additional field tests in 2015 that are not related to the  Internet response option—Address Validation Test and 2015 Census Test.
section_title: Bureau’s Past Efforts to Adopt an Internet Response Option
paragraphs: The 2020 census is intended to be the first time that the Bureau  implements an Internet response option on a wide scale for the decennial  census. During the 2000 Decennial Census, the Bureau tested the use of  an Internet response option. The Internet option had few respondents  (approximately 63,000 households representing about 169,000 persons),  in part because the Bureau did not advertise this response option.  However, the test demonstrated that the Internet response option worked  operationally.
The Bureau considered building on the 2000 experience for the 2010  Decennial Census by including the Internet response option in the scope  of the Decennial Response Integration System contract that was awarded  in October 2005. This contract included requirements to provide  functionality for the public to respond to the 2010 census via paper,  telephone, and the Internet. In July 2006, the Bureau decided not to  include the Internet response option in the design for the 2010 census  and eliminated it from the scope of the contract. This was because testing  indicated that the Internet response option did not increase the overall  response rate enough to justify the costs of building and securing this  option, which the Bureau had underestimated. The Bureau also had  continued concerns about the ability to sufficiently secure respondents’  data.
Additionally, in January 2013, the Bureau implemented an Internet  response option for its American Community Survey (ACS), which is  another household survey conducted by the 2020 Census Directorate on  a smaller scale than the decennial census. The ACS continuously collects  data on social, demographic, economic, and housing characteristics that  help determine how federal funds are allocated to states and localities  and provide information to communities to aid in planning investments  and services. The Bureau collects ACS data on a monthly sample of  households and aggregates the results into 1-, 3-, and 5-year estimates,  depending on the population size of the area. Bureau officials stated that  they intend to build on the IT infrastructure and lessons learned from the  ACS in order to implement the Internet response option for the 2020  census.
section_title: Bureau’s Plans for an Internet Response Option for the 2020 Census
paragraphs: The Bureau determined that the Internet response option offers several  benefits for the 2020 census, including the added convenience for  households in an increasingly Internet-enabled population to respond to  the survey; better quality data, which could reduce the amount of follow- up that is needed for surveys with incomplete or inconsistent data; and  less printing, postage, and processing of paper questionnaires. Bureau  officials have also stated that the Internet response option would provide  opportunities to administer the survey in multiple languages more easily  than with paper questionnaires.
The Bureau’s efforts to deliver an Internet response option for the 2020  census include several key components:  Internet response application: Design and develop an online survey  instrument that allows respondents to enter and submit their  information to the Bureau. To enhance Internet response participation,  the Bureau is researching an option for respondents to submit their  information without having the traditional Bureau-issued ID number  (referred to as “non-ID processing”). To achieve this, the Bureau will  need to develop the capability to validate respondent-provided  addresses either automatically against its master address file in real- time or in batches offline.
IT infrastructure: Develop and acquire the IT infrastructure (e.g.,  servers, hardware, and network capacity) needed to support the data  processing, storage, and transactions from Internet responses. The  Bureau has stated that it plans to use cloud computing solutions,  which is a means for establishing on-demand access to shared and  scalable pools of computing resources, to help support the large  volume of data processing, storage, and transactions needed for  Internet responses.
Communication and outreach: Planning for how the Bureau can  make use of partnership efforts, advertising, and outreach methods  like social media to maximize the use of an Internet response option  and motivate households to self-respond via Internet.
Additionally, the Bureau is exploring other potential design features  related to Internet response during the research and testing phase, which  may or may not be included in the final design of the 2020 census. For  example, the Bureau is testing whether there is value in asking  households to pre-register via a separate online portal prior to the census  with a preference on how they would like to receive reminders (e.g., e- mail or text message) from the Bureau on completing the survey. The  Bureau is also testing whether there is value in using e-mail addresses  purchased from commercial sources to contact respondents and ask  them to complete the survey. According to the Bureau, it will also need to  conduct research to help address privacy and information security  concerns the public may have regarding its use of the Internet to contact  respondents and collect their personal information, as well as to  determine which non-English languages will be offered with the Internet  response option.
The Bureau currently has 16 projects, planned or under way, that are  related to the 2020 census Internet response option.being managed by two different directorates—the 2020 Census   These projects are  Directorate is responsible for many of the research and testing projects,  while the IT Directorate is responsible for developing and acquiring the IT  systems and infrastructure needed to support the 2020 census. The IT  Directorate’s projects fall under an enterprise-wide program initiated in  fiscal year 2015 called the Census Enterprise Data Collection and  Processing (CEDCAP) program. CEDCAP is intended to integrate  disparate, program-specific survey data collection and processing  systems that the Bureau uses to conduct its many surveys. Table 1  summarizes the Bureau’s projects related to the Internet response option.
section_title: Enumerating Historically Hard-to-Count Populations
paragraphs: While the Bureau plans to offer multiple self-response modes in the 2020  census, including the paper and telephone options that have been offered  in prior censuses, the effect of an Internet response on population groups  that are already typically undercounted or missed during the census is  unclear. The Bureau has identified segments of the population that are  more difficult to enumerate based on prior censuses, such as minorities,  renters, children, low-income households, and low-education households.  To help identify hard-to-count populations, the Bureau segmented the  population into eight unique groups based on census demographic,  socioeconomic, housing, and mail response data. Each group contained  housing units with similar characteristics such as housing vacancy,  tenure, marital status, education, poverty, and unemployment level. The  groups that targeted the hard-to-count population included single  unattached mobile renter, economically disadvantaged homeowner,  economically disadvantaged renter, ethnic enclave homeowner, and  ethnic enclave renter.
To help reduce the undercount for the 2010 census, the Bureau  embarked on a number of outreach and enumeration activities aimed at  getting the hard-to-count populations to participate in the census. For  example, the Bureau used actual participation data from the 2000 census  as well as market and attitudinal research that identified different  mindsets people have about the census, such as those who are less  likely to participate because they doubt the census provides tangible  benefits or are concerned that the census is an invasion of privacy and  that the information collected will be misused. The Bureau used this  information to tailor its paid media efforts, such as buying additional  media in areas with low participation rates.   with a population that is growing larger,  As we have previously reported,more diverse, and more reluctant to participate, a complete and accurate  census has become an increasingly daunting task. While the Bureau  invested more resources in reaching out to and enumerating the hard-to- count population groups in the 2010 census, it achieved the same overall  participation rate as in the 2000 census. This trend is likely to continue as  the nation’s population gets larger, more diverse, and more difficult to  count.
section_title: Internet Use in the United States
paragraphs: According to data from the Pew Research Center, Internet use has  increased significantly since 2000, from 50 percent to 87 percent of adults  in the United States (see fig. 3).
Also according to data from the Pew Research Center, as of 2014,  Internet use varied among different demographic groups, such as  race/ethnicity, age group, education level, and household income (see  table 2).
Another population group to consider when introducing the Internet  response option is adults age 65 or older. Although they have historically  been a high self-response group using the paper-based method, Internet  use among this group was only 57 percent, compared to 97 percent  among adults under age 30.
In addition, ownership of mobile computing devices such as smartphones  and tablets has increased significantly in the past several years, as has  the use of cell phones to access the Internet, send and receive e-mails,  download software applications, and send and receive text messages  (see figs. 4 and 5).
section_title: Prior Related GAO Reports
paragraphs: Our prior work has identified the importance of having sound  management processes in place to help the Bureau as it manages the  multimillion dollar investments needed for its decennial census. For the  last decennial, we issued multiple reports and testimonies from 2005  through 2010 on weaknesses in the Bureau’s acquisition, management,  and testing of key 2010 census IT systems. Since the 2010 census, we  have issued additional reports and testimonies on weaknesses in the  Bureau’s efforts to institutionalize IT and program management controls  for the 2020 census. Relevant reports include the following:  In June 2008, we reported that the 2010 census life-cycle cost  estimate was not reliable and the Bureau had insufficient policies and  procedures and inadequately trained staff for high-quality cost  estimation.was not reliable, annual budget requests based on that estimate were  not fully informed. We recommended that the Bureau, among other  things, thoroughly document and update the estimate and for future  estimates, establish policies and procedures for cost estimation. The  Bureau has partially implemented these recommendations.
We also stated that because the life-cycle cost estimate  In January 2012, we reported that the Bureau was taking steps to  strengthen its life-cycle cost estimates but had not yet established   We also guidance for developing the 2020 life-cycle cost estimate. reported that the Bureau had not identified decision points at which  executives would review progress and decide whether the Bureau is  prepared to move from one project phase to another. We  recommended that the Bureau, among other things, identify decision  points and finalize guidance for the 2020 life-cycle cost estimate. The  Bureau has not yet implemented these recommendations.
In May 2012, we reported that the Bureau was taking steps consistent  with leading practices for long-term project planning for the 2020  census, such as creating a high-level schedule of program  management activities. However, the Bureau’s schedule did not  include milestones or deadlines for key decisions needed to support  transition between planning phases, which could result in later  planning activity not being based on evidence from early research and  testing. We also reported that the Bureau was taking steps in strategic  workforce planning, but it had not yet identified the goals that should  guide workforce planning or how to monitor, report, and evaluate its  progress toward achieving them. Accordingly, we made several  recommendations aimed at addressing these issues. The Bureau has  taken steps to implement these recommendations, but has not fully  implemented them.
In September 2012, we reported that the Bureau had drafted a new  investment management plan, system development methodology, and  requirements development and management processes to improve its  ability to manage IT investments and systems development.However, additional work was needed to ensure that these processes  were effective and successfully implemented across the Bureau, such  as finalizing plans for implementing its new investment management  and systems development processes across the Bureau. We also  reported that the Bureau had not fully put in place key practices for  effective IT workforce planning, including conducting an IT skills  assessment and gap analysis and establishing a process for  directorates to coordinate on IT workforce planning. To address these  weaknesses, we made a number of recommendations to the Bureau.  The Bureau has taken steps to address the recommendations, such  as finalizing its investment management process, conducting an  enterprise-wide IT competency assessment and gap analysis, and  developing action plans to address the identified gaps.
In November 2012, we evaluated the Bureau’s efforts to improve the  cost-effectiveness of enumeration in the 2020 census, paying  particular attention to three key efforts, one of which included  leveraging the Internet to increase self-response. We reported weaknesses in developing mitigation or contingency plans for several  project risks, including those related to tight time frames and accurate  cost information; weaknesses in developing cost estimates for  research and testing projects; incomplete project plans; and  incomplete performance metric documentation. We made a number of  recommendations to address these weaknesses, and the Bureau has  partially implemented them.
In January 2013, we reported on the Bureau’s implementation of  information security controls to protect the confidentiality, integrity,  and availability of the information and systems that support its  mission. We concluded that the Bureau had a number of weaknesses in controls intended to limit access to its systems and  information, as well as those related to managing system  configurations and unplanned events. We attributed these  weaknesses to the fact that the Bureau had not fully implemented a  comprehensive information security program, and made 13 public  recommendations and over 100 other recommendations that were for  limited distribution to address these deficiencies. The Bureau has  partially implemented the recommendations.
In September 2013, we testified on progress the Bureau had made in  its efforts to contain enumeration costs, including its efforts to  strengthen IT management and security practices. We noted that  the Bureau was exploring technology options for 2020 census  operations that collectively represent a dramatic leap from 2010,  including the “bring your own device” model for field data collection  and the Internet response option. We stressed the importance of the  Bureau strengthening its ability to manage its IT investments as well  as its practices for securing the information it collects and  disseminates.
In November 2013, we reported that the Bureau was not producing  reliable schedules for two efforts related to the 2020 census: (1)  building a master address file and (2) 2020 census research and  testing. We reported, for example, that the Bureau did not include all  activities and required resources in its schedules, or logically link a  number of the activities in a sequence. We recommended that the  Bureau take actions to improve the reliability of its schedules,  including ensuring that all relevant activities are included in the  schedules, complete scheduling logic is in place, and a quantitative  risk assessment is conducted. We also recommended that the Bureau  undertake a robust workforce planning effort to identify and address  gaps in scheduling skills for staff that work on schedules. The Bureau  has taken steps to implement these recommendations, but has not  fully implemented them.
In April 2014, we reported on the Bureau’s IT-related efforts for the  2020 census. We found that several of the IT-related projects lacked schedules and plans, and that it was uncertain whether the work  would be completed in time to inform the operational design decision  for the 2020 census, planned in September 2015. We also reported  that the Bureau had not prioritized its projects to determine which  were the most important to complete before the decision. We  recommended that the Bureau prioritize the research and testing that  it needed to complete in order to support the operational design  decision, and ensure that project plans and schedules were  developed consistent with the new prioritized approach. The Bureau  has made significant progress toward addressing the  recommendations. For example, it developed a document to guide the  Bureau’s path to making the preliminary design decision in September  2015, and Bureau officials identified projects that could be deferred to  after the preliminary design decision.
section_title: Census Bureau Has Taken Preliminary Steps to Examine the Impact of the Internet Option on Hard-to- Count Populations and Has Further Evaluation Plans
paragraphs: The Bureau has taken preliminary steps to identify demographic groups  likely to use the Internet response option in the 2020 census and how  they compare to historically hard-to-count populations by examining  existing ACS studies in this area and applying the lessons learned to the  decennial census. For example, the Bureau issued a report in May 2014  on the effects of adding an Internet response option to the ACS among  different segments of the population. The study found that the total self- response rate was statistically significantly higher after introducing the  Internet response option and the Internet response rate was about 55  percent. The study also found that, while none of the groups had  dramatically low rates of Internet participation, the addition of an Internet  response option had a positive effect on certain groups (e.g., advantaged  homeowner, single unattached mobile) and a negative effect on others  (e.g., ethnic enclave homeowner, economically disadvantaged  homeowner). The study suggested that, by pushing households to  respond by Internet (i.e., mailing an Internet response invitation and only  later mailing a paper questionnaire if the household did not first respond  by Internet), the Bureau may be discouraging some households from self- responding at all and that this may be happening in certain hard-to-count  groups. The 2020 Census Directorate officials told us that they are  incorporating the results of the ACS studies into future decennial census  field tests. For example, officials said that they plan to include both the  Internet response and paper questionnaire option in the first mailing for  hard-to-count population groups in the 2015 National Content Test.
In addition to using the ACS studies, the Bureau plans to conduct future  tests and research specific to the 2020 Decennial Census, which is  expected to produce more information in this area. For example, the  Bureau is planning two tests in 2015 that are expected to provide data on  Internet response rates among different demographic groups, including  historically hard-to-count populations. Specifically, the April 2015  Optimizing Self-Response Test site location was selected based on,  among other things, lower than average 2010 Decennial Census  response rates and ACS Internet response rates, Internet penetration at  least as high as the national average, and the ability to segment by hard- to-count populations. Additionally, 2020 Census Directorate officials  stated that the Bureau plans to begin coverage improvement research  with the September 2015 National Content Test, which is expected to  provide a nationally representative demographic sampling and national  Internet self-response rates. Bureau officials stated that additional  research in this area in 2016 and beyond is yet to be determined.
section_title: Preliminary 2020 Census Internet Response Option Cost Estimate Was Not Reliable
paragraphs: GAO’s Cost Estimating and Assessment Guide identifies a number of  best practices that are the basis of effective program cost estimating and  should result in reliable and valid cost estimates that management can  use for making informed decisions. Specifically, a reliable cost estimate  should be comprehensive (costs are neither omitted nor double counted);  well-documented (the estimate is thoroughly documented, including  source data and significance, clearly detailed calculations and results,  and explanations for choosing a particular method or reference); accurate  (the estimate is unbiased, not overly conservative or overly optimistic, and  based on an assessment of most likely costs); and credible (the estimate  discusses any limitations of the analysis from uncertainty or biases  surrounding data or assumptions).
A rough-order-of-magnitude estimate is a less-rigorous cost estimate developed from  limited data and in a short time. of the estimate in 2014, with a resulting total of approximately $12.7  billion. This included, among other things, about $73 million for the  Internet response option. However, the estimate for the Internet response  option did not meet the characteristics of a reliable estimate. Specifically:
The Internet response option cost estimate was not comprehensive.  This is because the Internet response option cost estimate included  costs from 2010 to 2020 and provided a subset of assumptions for  researching, testing, and deploying an Internet response option. While  the estimate was structured around these high-level cost elements,  these elements were not defined, and therefore it is not clear whether  all costs associated with the Internet response option were included.  Bureau officials stated that the estimate was not developed based on  a work breakdown structure with defined elements because the  2020 census program was not mature enough to have such a  structure at the time the initial estimate was developed. They stated  that the estimate will be updated to reflect the program’s work  breakdown structure once the preliminary design decision is made in  September 2015. However, a work breakdown structure should have  been initially set up when the program was established and  successively updated with more detail over time as more information  became known about the program.
The Internet response estimate was not well-documented. We found  that the documentation included selected source data, assumptions,  and calculations, but it did not provide a clear and traceable view of  the estimated costs and all the assumptions that were included in the  Internet response option cost estimate. Additionally, the Internet  response option cost estimate was developed in a separate file and  rolled up into a single line item labeled “Internet” within the 2020  census life-cycle cost estimate, without including source information to  show where the estimate originally came from or details of what the  estimate was based on. This was problematic because Bureau  officials were initially unable to locate the relevant files and were  unable to provide us with an explanation for how the Internet  response option estimates were developed. After several weeks,  officials finally located the separate file that contained the high-level  Internet response cost elements.
The estimate was not accurate. Specifically, while accurate estimates  are to be based on an assessment of most likely costs and historical  data for similar programs, the Internet response option cost estimate  was based on subject matter expert opinion and analogous data from  the ACS, which is similar in function to the decennial census but not in  scale. The scale of the decennial census is significantly larger than  ACS, and will require IT systems and infrastructure to be sized  accordingly. However, it is not clear in the documentation how the  Bureau applied the ACS cost data in estimating 2020 census costs,  nor could Bureau officials explain this. For example, 2020 census IT  infrastructure costs were estimated to be higher than ACS IT  infrastructure costs, but Bureau officials could not explain what  assumptions and methodology were used to account for the much  larger scale of IT infrastructure needed for the 2020 census compared  to the ACS. Without insight into how historical data were used to  estimate future costs, we cannot determine whether the cost models  are accurate.
Additionally, in 2014, the Bureau revised selected components of the  2020 census life-cycle cost estimate to inform the fiscal year 2015  budget request, but did not revise the Internet response option cost  estimate as part of this update even though there had been significant  changes to the program since 2011. For example, projects related to  the Internet response option have evolved, and thus not all projects  are represented in the 2011 $73 million estimate, such as non-ID  processing. Also, the Bureau determined in 2013 that its existing IT  infrastructure was not sufficient to support the scale-up needed for  Internet response data processing and storage in the 2020 census,  and is planning to address this as part of the enterprisewide IT data  collection and processing program, known as CEDCAP. According to  Bureau officials, they are in the process of developing estimated costs  for relevant CEDCAP projects, and these will be incorporated into the  Internet response option cost estimate. Additionally, the Internet  response option cost estimate was not updated with more current  data from the analogous ACS program, as a result of its  implementation of an Internet response option in January 2013.
The estimate was not credible. Sensitivity and risk and uncertainty  analyses were conducted on the 2020 census life-cycle cost estimate,  but they were not completed properly. For example, a risk and  uncertainty analysis was only applied to fiscal years 2018 to 2020  rather than on the total cost estimate. According to Bureau officials,  the Bureau focused on fiscal years 2018 to 2020 because 80 percent  of all costs for the decennial census are expected to be incurred  during this period. However, accounting for all risks throughout the life  of the program is necessary to adequately capture the uncertainty  associated with a program’s estimate. The Bureau also did not  perform risk and uncertainty analyses on the Internet response option  cost estimate to reflect the uncertainty introduced by the significant  program changes discussed previously, such as the uncertainty  associated with the initial estimate for IT infrastructure costs ($11.4  million).
The 2020 Census Directorate officials have recognized that the estimate  for the Internet response option had weaknesses and stated that the  preliminary 2020 census rough-order-of-magnitude life-cycle cost  estimate was not developed to be an official cost estimate, but rather a  “top-down” approach to estimating costs and potential savings focused  around the major design categories. Nevertheless, the Bureau considered  the estimate to be a budget-quality estimate and used it to inform the  fiscal year 2015 budget request. Bureau officials stated that the  preliminary 2020 census life-cycle cost estimate (including the Internet  response option estimate) will be updated once the preliminary design  decision is made in September 2015.
Additionally, Bureau officials identified several ongoing efforts that the  Bureau has planned to improve its institutional cost estimating practices.  Specifically, Bureau officials stated that the Bureau established a new  centralized cost estimating office in August 2013 that is expected to,  among other things, issue guidance and policies for developing reliable  cost estimates and establish a standardized work breakdown structure for  censuses and surveys by the third quarter of fiscal year 2015, create a  cost estimating certificate program by the fourth quarter of fiscal year  2015, and conduct independent cost estimates, as needed. Bureau  officials also stated that they have certified four staff in cost estimation  techniques, moved the 2020 census estimate to a more robust cost  estimating tool, and planned actions to address competency gaps in cost  estimating. While these are important steps to institutionalizing good cost  estimating practices, until the Bureau updates the Internet response  option cost estimate to ensure that it conforms with best practices, the  estimate will continue to be unreliable.
In addition to the $12.7 billion 2020 census rough-order-of-magnitude life- cycle cost estimate, the Bureau estimated potential cost savings for the  major design options, such as optimizing self-response, which includes  the Internet response option. The Bureau estimated that implementing all  design options could result in potential savings of up to $5 billion.  Potential cost savings were estimated by adjusting inputs for different  2020 census design scenarios and calculating the difference between  those estimated 2020 census costs and the cost of repeating the 2010  census design. For example, to estimate potential cost savings for the  optimizing self-response design option, the Bureau reduced the amount  of printing, postage, and infrastructure needed for processing paper  questionnaires based on the percent of responses expected via Internet.  The Bureau estimated that the optimizing self-response option could  result in potential savings of about $550 million to $1 billion. However,  the unreliability of estimated costs for the Internet response option  discussed above casts doubt on the reliability of estimated cost savings  associated with the optimizing self-response design option.
section_title: Census Bureau Faces Scheduling, Task, and Capability Challenges in Planning for an Internet Response Option
paragraphs: While efforts are under way to deliver an Internet response option for the  2020 census, significant challenges remain. We identified four major  challenges the Bureau faces in implementing an Internet response option  for the 2020 census: project schedules associated with the Internet  response option are not fully integrated; key questions may not be  answered in time for the preliminary design decision; high-level time  frames for cloud computing decisions and implementation for the 2020  census have not been determined; and gaps in IT skill sets continue to  exist. According to 2020 Census Directorate and IT Directorate officials,  the Bureau has recently begun to develop methodologies for answering  key research questions, using a contractor to assist in assessing cloud  computing technologies for the 2020 census, and implementing actions to  close critical competency gaps. However, without established  methodologies and time frames for completing these efforts, the Bureau  will be limited in its ability to make informed design decisions, which could  impact implementation of the 2020 Census.
subsections: section_title: Internet Response Option Project Schedules Have Not Been Fully Integrated
paragraphs: According to best practices identified in GAO’s Schedule Assessment  Guide, a detailed schedule should be horizontally traceable, meaning  that it should link products and outcomes associated with other  sequenced activities. These links are commonly referred to as “hand-offs”  and serve to verify that activities are arranged in the right order for  achieving aggregated products or outcomes. Such mapping or alignment  of levels enables different groups to work to the same master schedule.
As previously mentioned, the Bureau has several different projects  related to the Internet response option, planned or currently under way  (see table 1 for the complete list of projects). Certain projects, such as IT  Infrastructure 2020 Decennial Scale-Up and E-correspondence, were  initiated in October 2014 and thus did not yet have detailed schedules.  For selected ongoing projects, such as the Optimizing Self Response,  Non-ID Processing, and 2014 Census Test projects, the Bureau had  integrated the schedules into the 2020 census program’s integrated  master schedule. However, for other projects, which have been under  way for 5 months, the Bureau has not yet integrated their schedules with  the 2020 census program’s integrated master schedule, so the  dependencies among the different projects (e.g., hand-offs between  teams) are not linked together. For example:
The Bureau established a schedule for developing the Centurion  Internet response option application for the 2015 tests, which included  activities starting as early as July 2014. However, as of November  2014, this schedule was not yet linked to the overarching 2020 census  program schedule. According to Bureau officials, the Centurion  schedule was recently developed and is eventually to be integrated  with the 2020 census program schedule. However, Bureau officials  did not know when this would be complete—despite ongoing activities  with the Centurion project and the 2020 census program.
Activities for the 2015 Optimizing Self-Response Test and 2015  National Content Test were not included in the 2020 census program  schedule. According to 2020 Census Directorate officials, detailed  schedules for the 2015 tests were still under development and will be  incorporated into the 2020 census program schedule once they are  complete. However, as of November 2014, Bureau officials did not  have a time frame for when this would be done, and preparations for  the 2015 Optimizing Self-Response Test are well under way, with the  test planned to begin in less than 5 months.
As stated earlier, we have previously recommended that the Bureau take  actions to improve the reliability of its schedules, including steps to  ensure that all relevant activities and associated resources are included in  the schedules and complete scheduling logic is in place. Bureau officials  have recognized the need to fully integrate project schedules with the  2020 census program’s integrated master schedule, but stated that they  are currently working to identify required resources for activities in the  2020 census program integrated master schedule. They stated that, in  the meantime, the project teams are conducting work according to their  project schedules, which are maintained separately. However, until the  Bureau fully implements our recommendation to integrate dependent  activities in the program schedule, the Bureau is unable to generate an  accurate “critical path”—the sequence of steps needed to achieve the end  goal that, if they slip, could negatively affect the overall project completion  date. This limits the Bureau’s ability to use the 2020 census program  schedule to estimate reliable dates, determine the impact of any schedule  slippages on major milestones, and identify opportunities for increased  efficiency.
section_title: Key Questions May Not Be Answered In Time for Preliminary Design Decision
paragraphs: In September 2014, the Bureau released a planning document titled The  Path to the 2020 Census Design Decision, which identified inputs, such  as research questions, design components, and testing, needed to inform  the preliminary design decision planned for September 2015. This  included the following key research questions related to the Internet  response option that were determined to be critical inputs into the  preliminary design decision:
What are the best methods for communicating the importance of  responding to the 2020 census, including methods for promoting the  use of Internet response?
What percentage of the population has access to the Internet?
What is the estimated Internet self-response rate?
What IT infrastructure for security and scalability is necessary to  support the Internet as the primary mechanism for self-response?
Is there value in asking households to pre-register online for the 2020  census?
Is it necessary to provide households with an identification code to  respond via the Internet?
While the Bureau has documented in relevant project plans its methods  for answering the questions related to communication strategies, online  pre-registration, and processing household responses without  identification codes, it has not yet established how it will determine  answers to the questions related to self-response rate and IT  infrastructure for the preliminary design decision. As stated in 2020  census program management guidance, detailed project plans and  specific research questions, as well as study plans describing the  management and technical approach should be established for research  and testing projects.
According to Bureau officials in November 2014, they began to establish  a new project team that was intended to be responsible for estimating the  Internet self-response rate. However, Bureau officials did not have a time  frame for when a project plan or study plan would be developed that  would document the methodology for how the Internet self-response rate  will be estimated to inform the preliminary design decision.
Additionally, Bureau officials told us that they had established a new  project—the 2020 Census Architecture and IT Roadmap—in June 2014  that is intended to help determine the IT infrastructure needed to support  the Internet response option. This project is to deliver initial 2020 census  architecture and IT roadmap documents by preliminary design. However,  the Bureau had not yet established a project plan or study plan, and had  not developed the methodology—as part of this project or another—for  determining scalability and security infrastructure needs for the Internet  response by September 2015.
As previously stated, the Bureau is committed to producing the  preliminary design decision and developing the life-cycle cost estimate for  the 2020 census by September 2015. With about 8 months remaining  until the preliminary design decision is to be made, and major tests  already designed or completed (i.e., the 2014 Census test and the April  2015 optimizing self-response test), the Bureau has limited time to  determine how these critical questions will be answered. Accordingly,  until the Bureau establishes and implements clear plans for answering the  Internet response rate and IT infrastructure questions, the Bureau will  have limited information for beginning its development and  implementation of systems and infrastructure. Also, as previously  discussed, it is uncertain how complete or reliable the Bureau’s Internet  response option and IT infrastructure cost estimates will be to inform key  design decisions by this time.
section_title: High-Level Time Frames for Cloud Computing Decisions and Implementation Have Not Been Established
paragraphs: The OMB Federal Cloud Computing Strategy recognizes the importance  of organizations planning effectively when selecting services to move to a  cloud environment. The strategy recommends that organizations create  roadmaps for cloud deployment and migration in order to prioritize  services that minimize risks to the organization. NIST also recognizes that  moving to a cloud environment is a business decision, where the  organization’s business case should consider relevant factors such as  transition and life-cycle costs and security and privacy requirements.
As previously mentioned, the Bureau’s existing IT infrastructure is not  sufficient to process and store the large volume of anticipated electronic  Internet survey responses for the 2020 census. Accordingly, Bureau  officials stated that the Bureau plans to use a cloud environment to  provide the needed capability. The Bureau has taken steps to research  the use of the cloud to deliver the level of scalability needed to support  the Internet response option for the 2020 census. Specifically, the Bureau  has identified and prioritized 2020 census capabilities to potentially move  into the cloud environment, such as Internet data collection. Additionally,  the Bureau plans to conduct volume testing of the Centurion application in  a cloud environment beginning in November 2014, in order to collect  additional data to determine if it can meet the expected capacity in 2020.  IT Directorate officials also said that they are working closely with federal  agencies such as NIST and the Federal Aviation Administration to  determine requirements for moving services to a cloud environment.
Although the Bureau expects cloud decisions to lay the groundwork for  accommodating the volume of users expected for the 2017 Economic  Census and 2020 Decennial Census, the Bureau has not established  time frames to determine when key cloud computing decisions need to be  made and actions need to be taken for the 2020 census, such as  selecting, testing, and deploying a cloud environment that meets its  needs for scalability, budget, security, and privacy protection of personally  identifiable information. IT Directorate officials stated that by the third  quarter of fiscal year 2015 the Bureau will develop a schedule for the IT  Infrastructure 2020 Decennial Scale-Up project, which will include high- level time frames for a subset of relevant activities, such as conducting an  analysis of alternatives to determine whether a cloud solution is the best  alternative for addressing the scale-up. Additionally, IT Directorate  officials told us that the Bureau plans to develop a strategy in 2016 to  outline the overall approach for acquiring cloud solutions for the 2020  census, among other things.
The Bureau is planning to begin systems readiness testing in October  2018, which is when the Bureau has determined that the systems and  processes for the 2020 census must be developed and ready for end-to- end system testing (approximately 3.7 years away). As we have  previously reported, the federal government procurement process can  require a significant amount of time, and agencies can face challenges  in implementing cloud computing, such as meeting federal security  requirements, obtaining guidance, acquiring knowledge and expertise to  implement cloud services, certifying and accrediting vendors, and  ensuring data portability and interoperability. Without established high- level time frames, Bureau officials will not know whether there is enough  time to effectively implement a cloud environment. The Bureau has not  yet established such time frames for the 2020 cloud implementation  approach due to a lack of internal cloud computing expertise.
In an effort to offset this lack of internal expertise, the Bureau issued a  task order in September 2014 to a contractor for assistance in assessing,  analyzing, and recommending cloud computing technologies for the 2020  census. While this assistance may be helpful, until the Bureau, at a  minimum, documents time frames for selecting, testing, and deploying its  cloud environment, it will not know whether there is enough time to  successfully implement a cloud solution that meets scalability, budget,  security, and privacy protection of personally identifiable information  needs for the 2020 census.
section_title: Gaps in IT Skills Continue to Exist
paragraphs: As our prior work and leading guidance recognize, having the right  knowledge and skills is critical to the success of a program. In response  to prior GAO recommendations on developing strategic workforce  planning capabilities, the Bureau completed an enterprise-wide  competency assessment in 2013 and identified several mission-critical  gaps in technical competencies within the IT and 2020 census workforce  that would be needed to support the Internet response option. In August  2014, the Bureau completed action plans and targets aimed at  addressing the IT and 2020 census workforce competency gaps. The  gaps related to the Internet response option and planned actions to  address them are summarized in table 3.
Moving forward, the Bureau plans to monitor quarterly status reports on  the implementation of these actions and closing competency gaps,  beginning around December 2014. Fully implementing actions to close  these competency gaps will be critical to ensuring the Bureau has the  skills it needs to effectively develop and implement the Internet response  option.
section_title: Conclusions
paragraphs: The introduction of an Internet response option for the 2020 census has  the potential to offer numerous benefits, such as added convenience for  households in an increasingly Internet-enabled population to respond to  the survey, better quality data which can result in less follow-up work, and  reduced costs associated with processing paper questionnaires.  Identifying ways to increase participation via the Internet will help  increase the benefits of this response mechanism.
While the Bureau’s initial estimate of approximately $73 million for the  Internet response option was included in the fiscal year 2015 budget  request, this estimate lacks reliability, which in turn, calls into question the  reliability of the potential cost savings estimate of about $550 million to $1  billion for the optimizing self-response design category (which includes  the Internet response option). Although the Bureau has several important  cost estimating improvements under way, until the Bureau updates the  Internet response option cost estimate to ensure that it meets best  practices, the estimate will continue to be unreliable.
Additionally, the Bureau’s ability to effectively manage the scheduling,  task, and capability challenges it faces in planning for an Internet  response option will be critical to the success of the 2020 census.  Specifically, without fully implementing our prior recommendations on  integrating dependent activities into the schedule, the Bureau continues  to be unable to estimate reliable dates and make informed decisions  based on an accurate critical path.
Further, with about 8 months remaining before key design decisions need  to be made, the Bureau has not established the methodologies for  answering two Internet response option research questions that were  deemed critical for the preliminary design decision—estimating the  Internet self-response rate and determining the IT infrastructure for  security and scalability needed. Accordingly, until clear plans for  answering these questions are established and implemented, the Bureau  will have limited information for beginning its development and  implementation of systems and infrastructure.
Additionally, even though the Bureau acknowledges the need to acquire a  cloud solution to compensate for the fact that its existing IT infrastructure  is not sufficient to support a wide-scale Internet response option for the  2020 census, it has not defined the high-level time frames for when key  cloud computing decisions need to be made and implemented. As a  result, with systems readiness testing for the complete 2020 census  design planned to begin in October 2018, it is uncertain how the Bureau  can ensure that there is sufficient time to accomplish this objective.
Finally, the Bureau faces continuing gaps in mission-critical technical  competencies, including cloud computing, security integration and  engineering, and requirements development. Fully implementing the  Bureau’s planned actions to close these gaps will be critical to ensuring  that it has the skills needed to effectively deliver the 2020 census Internet  response option.
section_title: Recommendations for Executive Action
paragraphs: To ensure that the Bureau is better positioned to deliver an Internet  response option for the 2020 Decennial Census, we are recommending  that the Secretary of Commerce direct the Under Secretary for Economic  Affairs to direct the Director of the Census Bureau to take the following  three actions: ensure that the estimated costs associated with the Internet response  option are updated to reflect significant changes in the program and to  fully meet the characteristics of a reliable cost estimate; ensure that the methodologies for answering the Internet response  rate and IT infrastructure research questions are determined and  documented in existing or future project plans in time to inform key  design decisions; and develop high-level time frames for selecting, testing, and deploying a  cloud environment to guide the Bureau’s approach to enabling  scalability for the 2020 census.
section_title: Agency Comments and Our Evaluation
paragraphs: We received written comments on a draft of this report from the  Department of Commerce, which are reprinted in appendix II. The  department neither agreed nor disagreed with our recommendations, but  provided comments that are discussed in detail below.
In its comments, the department stated that our conclusion regarding the  unreliability of estimated costs and savings for the Internet response  option seemed to be based on the fact that the Bureau did not build a  bottom-up estimate using documented parameters from previous  censuses or research. The department also stated that it needed to  conduct critical tests and research to inform key cost parameters and that  it planned to revise the estimated costs and savings as research and  testing efforts are completed. The department added that it believed its  top-down estimates were sufficient for the purpose of identifying major  cost drivers for the decennial census.
However, as stated in the report, our assessment took into account the  Bureau’s top-down approach and that it had developed a less-rigorous,  “rough-order-of-magnitude” estimate; we therefore performed a high-level  analysis of the cost estimate and methodology. We also described what is  expected of a rough-order-of-magnitude cost estimate and recognized  that such an estimate is developed from limited data and in a short time.  For example, regarding the lack of a work breakdown structure, our report  notes that this should be initially set up when a program is established  and updated with more detail over time as more information becomes  known about the program.
Nevertheless, even when taking these factors into consideration, the  Bureau’s rough-order-of-magnitude estimate for the Internet response  option was not reliable when assessed against best practices that are  applicable to such estimates. For example, as we state in our report, the  Internet response option estimate was not accurate, in part, because  when the Bureau revised selected components of its estimate in 2014, it  did not revise the Internet response option portion, despite significant  changes to the program since the original 2011 estimate. These changes  included the realization that additional IT infrastructure, beyond the  Bureau’s existing infrastructure, would be needed in order to support the  scale-up needed for Internet response data processing and storage.  Given the weaknesses identified in our report, we continue to maintain  that the Bureau cannot be assured that its top-down estimate was  sufficient for the purpose of identifying major cost drivers for the decennial  census.
The department also recognized the need to improve its capabilities and  skill sets in cost estimation and identified several planned and ongoing  actions, such as standardizing a work breakdown structure, establishing  enterprise guidance and policies for cost reporting, and continuing to hire  and train certified cost estimators. As stated in our report, while these are  important steps to institutionalize good estimating practices, until the  Bureau updates the Internet response option cost estimate to ensure that  it conforms to best practices, the estimate will continue to be unreliable.  Accordingly, we maintain that actions to address our first  recommendation are still needed.
Regarding our conclusions on key challenges associated with delivering  an Internet response option, the department stated that it believed it had  developed project plans, methodologies, and time frames for making  decisions related to IT infrastructure needs, including the use of cloud  computing to support 2020 census requirements. The department further  stated that the wording or descriptions of these activities in its  documentation may use internal jargon that may make them difficult to  find for those who do not regularly work on these activities, and that it  would work to mitigate this issue.
We disagree that a misunderstanding of the plans, methodologies, and  time frames exists. During our review, we took appropriate measures to  ensure that we had collected and analyzed the most current, complete,  and accurate information available, including meetings with  knowledgeable officials. Furthermore, at the conclusion of our review, we  met with key officials from the 2020 Census and IT Directorates and  confirmed the facts of our analysis, including the fact that project plans  and methodologies for making decisions related to the IT infrastructure  needs for the 2020 census did not exist.
We disagree that a misunderstanding of the plans, methodologies, and  time frames exists. During our review, we took appropriate measures to  ensure that we had collected and analyzed the most current, complete,  and accurate information available, including meetings with  knowledgeable officials. Furthermore, at the conclusion of our review, we  met with key officials from the 2020 Census and IT Directorates and  confirmed the facts of our analysis, including the fact that project plans  and methodologies for making decisions related to the IT infrastructure  needs for the 2020 census did not exist.
The department also commented that the documentation produced for the  2020 census design decision would include key dates for making the  decisions (e.g., the need for and likely extent of cloud computing use),  and revised estimates of key cost parameters (e.g., Internet response  rates). The department stated that it had established a plan and an  implementation team to produce this documentation, referred to as the  2020 Census Concept of Operations. While we recognize that the design  decision would help inform some of the time frames related to cloud  computing, our concern is that high-level time frames for when key  decisions need to be made do not yet exist. Therefore, the Bureau does  not have assurance that there will be enough time to overcome potential  challenges in acquiring and effectively implementing cloud computing  services, such as meeting federal security requirements, certifying and  accrediting vendors, and ensuring data portability and interoperability.
Further, while we support the Bureau’s efforts to ensure that the outcome  of the design decision is appropriately documented, our report highlights  the need for documenting the research methodologies that will be used to  make those design decisions. As stated in our report, with about 8  months remaining until the design decision and with major tests already  designed or completed, the Bureau has limited time to determine how  critical research questions will be answered. Consequently, we maintain  that actions are still needed to respond to our recommendations to  develop key methodologies and high-level time frames for cloud  computing decisions.
The department provided two additional comments related to the  background section of our report:
First, the department noted concern that discussions of differential  Internet availability among historically hard-to-count populations imply  that this could lead to reduced coverage of these populations in the  2020 census and stated that the Bureau believed that this was  unlikely to result in less coverage of these populations. The  department explained that knowledge of the differentials would help  the Bureau to effectively deploy outreach, partnership, social media,  and other advertising investments in 2020. It further stated that the  Internet would not be the only response option offered in 2020  because individuals without access to, or desire to use, the Internet  will still be able to respond on paper, by phone, or in person with a  census interviewer. We agree that the differential Internet availability  among historically hard-to-count populations should not imply that the  introduction of an Internet response option would lead to reduced  coverage. To clarify that point, we modified the report to state that the  Internet response option’s effect on historically hard-to-count  populations is unclear. We intentionally did not draw conclusions  about what the potential impact would be. The discussion of varying  Internet availability is included to illustrate why the impact of an  Internet response option on historically hard-to-count populations is a  relevant topic to examine, and the report also notes that the Bureau  plans to offer multiple self-response modes for the 2020 census,  including paper and telephone. This information is introduced to  provide context to the discussion of the Bureau’s completed steps and  further plans to examine the impact of introducing an Internet  response option on hard-to-count populations.
Second, the department stated that it was uncertain whether the  background section discussing prior related GAO reports took into  account that some of the Bureau’s planned actions were not yet  scheduled to be completed and expressed concern that this  discussion implied that the Bureau had plans to complete all actions  by this point in time. However, the section in question is intended only  to summarize relevant prior GAO reports and recommendations and  provide a brief status update on progress made to address these  recommendations. As it is background information, we did not refer to  or imply time frames for when the recommendations should be  implemented.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the Secretary of  Commerce, the Under Secretary for Economic Affairs, the Director of the  U.S. Census Bureau, and interested congressional committees. In  addition, the report is available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-4456 or chac@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made key contributions to this report are  listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to (1) describe the Census Bureau’s (Bureau) efforts  to identify demographic groups likely to use Internet response and how  they compare to historically hard-to-count populations, (2) assess the  reliability of estimated costs and savings for the Internet response option,  and (3) determine key challenges associated with delivering an Internet  response option for the 2020 census.
To describe the Bureau’s efforts to examine the impact of an Internet  response option on the historically hard-to-count populations, we  identified and reviewed studies conducted by the Bureau on demographic  groups likely to use the Internet response option and how they compare  to historically hard-to-count populations. We also conducted a literature  review to identify other relevant studies on hard-to-count populations,  Internet usage among demographic groups, and the impact of an Internet  response survey on hard-to-count populations. We interviewed Bureau  officials on their plans to further assess the impact of an Internet  response option on historically hard-to-count populations.
To assess the reliability of estimated costs and savings for the Internet  response option, we obtained and analyzed the Bureau’s documentation  supporting the 2020 census rough-order-of-magnitude life-cycle cost  estimate, the Internet response option portion of the cost estimate, and  the potential cost savings estimate. We compared the cost estimating  methodology and documentation against best practices for developing  reliable cost estimates identified in GAO’s Cost Estimating and  Assessment Guide.account that the Bureau developed the 2020 census cost estimate as a  less-rigorous, “rough-order-of-magnitude” cost estimate, and therefore  performed a high-level analysis of the Bureau’s cost estimate and  methodology. We also interviewed Bureau officials to verify that our  findings were accurate and discussed their approach to estimating costs  and potential savings. Our report notes the instances where reliability  impacts the quality of the cost estimate.
When applying the best practices, we took into  To identify key challenges to delivering an Internet response option for  the 2020 census, we identified relevant experts within the Bureau’s key  advisory groups: the National Academy of Sciences, National Advisory  Committee, and Census Scientific Advisory Committee. These advisory  groups consist of academic and industry experts from various fields,  including information technology and Internet survey design, and they  meet with the Bureau regularly to provide feedback on various areas,  including the 2020 census program. We interviewed relevant experts from  these advisory groups to discuss their perspectives on key challenges the  Bureau faces in implementing an Internet response option.
We also analyzed documentation on the Bureau’s projects related to the  Internet response option, such as project plans, schedules, risk registers,  and monthly status reports; program-level documentation, such as 2020  census and IT strategy documents; and workforce action plans. We also  interviewed Bureau officials from the 2020 census program and the IT  Directorate to obtain information on progress made on the Internet  response option. We compared the Bureau’s efforts against relevant  guidance and best practices, such as those identified in GAO’s Schedule  Assessment Guide, prior GAO work, the National Institute of Standards  and Technology’s guidance on cloud computing, the Office of  Management and Budget’s Federal Cloud Computing Strategy, and the  Software Engineering Institute’s Capability Maturity Model® Integration, to  determine if there were any gaps.
We aggregated the results to identify the key challenges the Bureau faces  in delivering an Internet response option for the 2020 census and  presented the preliminary challenges that we identified to experts and  Bureau officials to obtain their feedback on the challenges. We also  observed Bureau activities related to the Internet response option,  including 2020 census program management reviews, advisory group  meetings, demonstration of the Internet response application and pre- registration portal used for the 2014 census test, and operations of the  2014 census site test.
We conducted this performance audit from May 2014 to February 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Comments from the U.S. Department of Commerce
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
paragraphs: Carol R. Cha at (202) 512-4456 or chac@gao.gov.
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, the following staff made key  contributions to this report: Shannin G. O’Neill (Assistant Director), Jason  Lee, Jennifer Leotta, Lee McCracken, Dana Pon, and Jeanne Sung.