2020 Census: Additional Actions Could Strengthen Future Census Test Designs
Released: 2013-11-21T00:00:00

Letter

Background

The Bureau’s testing program for the 2010 Census relied principally on a small number of large-scale census-like tests. Specifically, the 2010 testing program included two national tests on the content of questionnaires in 2003 and 2005, two site tests focused on data- collection methods and systems in 2004 and 2006, and a final “dress rehearsal” at two sites in 2008. The dress rehearsal, considered to be the final step of a decade of research and testing, had the primary focus of testing automated field operations and their interfaces. The Bureau previously reported that implementing the census tests, including the dress rehearsal, cost about $108 million.

As part of the Bureau’s effort to conduct the 2020 Census at a cost lower than the 2010 Census, it plans to invest in early research and conduct smaller more frequent tests to inform its 2020 Census design decisions. The lifecycle for 2020 Census preparation is divided into five phases, as illustrated in figure 1. The Bureau is attempting to frontload critical research and testing to an earlier part of the decade than it had in prior decennials. It intends to use the early research and testing phase through fiscal year 2015 to develop a preliminary design and to evaluate the possible impact that changes would have on the census’ cost and quality. By the end of the early research and testing phase, the Bureau plans to decide on preliminary operational designs.

In August 2012, as part of the 2020 Census testing program, the Bureau issued a research and testing management plan. The plan defines eight phases of the life cycle for a census field test, as shown in figure 2. According to the plan, the first three phases culminate in the approval of a field test design by a group of senior Bureau managers that provides decision-making support to the 2020 Census program.

To facilitate the field test design process, the Bureau developed templates as guidance for developing test designs. The Bureau also developed management plans in specific functional areas. One, a communications and stakeholder engagement plan, identified stakeholder groups involved in 2020 Census planning. Another, a governance plan, identified decision-making bodies for the 2020 Research and Testing program. The plans outline processes the Bureau will implement as it prepares for the 2020 Census. Prior to this, the Bureau used 2010 Census program guidance and standards to govern some of its earliest design discussions.

The Bureau plans to conduct 10 field tests in preparation for the 2020 Census during the early research and testing phase. According to the Bureau, not all field tests are alike, and they vary in scope and capability. Some will be designed to encompass more exploratory questions. Others will be designed to more rigorously test the implementation of specific operations. When we initiated this review in early 2013, the Bureau had designed its initial three tests as summarized in table 1. Other planned field tests will cover topics such as building the address list and narrowing possible approaches for self-response, non-response follow-up, and workload management. The field tests will culminate in a larger test late in fiscal year 2015 to further narrow possible design options.

The Bureau Generally Followed Most Key Practices for Designing Tests and Has Taken Steps to Strengthen Its Design Process

Based on our prior work, we identified 25 key practices for a sound study plan. Following these practices before test designs are completed can help ensure that test designs are appropriate, feasible, and produce useful results. We organized the 25 practices into the following six themes. general research design, data collection plan, data analysis plan, design process management. sample and survey, stakeholders and resources, and As demonstrated in figure 3, the Bureau generally followed most of the 25 key practices for two of the three field test designs and at least partially for the third field test design.

Research questions frame the scope of a test, drive the design, and help ensure that findings are useful and answer the research objectives. The objectives should be relevant, creating a clear line of sight to the Bureau’s goals for the 2020 Census. In addition, clearly articulating the test design in advance of conducting a test aids researchers in discussing methodological choices with stakeholders. Across the three field tests, the Bureau generally followed three of the general research design practices, followed one practice to a varying degree, but did not follow the practice of identifying potential biases (see table 2).

The Bureau defined concepts, considered relevant prior research in each test, and included objectives that were relevant. However, the Bureau omitted research questions from the design of the 2012 National Census Test (2012 NCT). Identifying specific research questions linked to the research objectives helps ensure that answers resulting from the field test will address the needs of the 2020 Census research program reflected in the research objectives. For example, the 2013 Quality Control Test (2013 QCT) design includes an objective to investigate how the Bureau can modernize and increase the efficiency and utility of its field infrastructure. The corresponding research question states that the test will research the feasibility of using Global Positioning System (GPS) data. The test would determine, among other things, if field staff appropriately visited housing units for address listing and enumeration and if GPS data can be used to reduce or eliminate field quality control checks. None of the three test designs addressed potential biases, such as cultural bias. If a test design does not address potential biases, systematic errors could be introduced. Such errors could affect the accuracy of the test and thus potential design decisions for the 2020 Census.

Data Collection Plan

Identifying data sources and data collection procedures is key to obtaining relevant and credible information. The Bureau generally followed two of the data collection practices for all three tests and followed the others to varying degrees (see table 3).

In its initial three field test designs, the Bureau generally followed the practices of clearly presenting how data will be collected, and describing a method to encourage responses, as applicable. The three other practices were followed less consistently. These practices help researchers ensure that the data collected for a test will be sufficient and appropriate. First, only the 2013 National Census Contact Test (2013 NCCT) design included a plan for administering and monitoring data collection. The test designs should include data collection procedures that will obtain relevant and credible information to ensure that the resulting information meets the decision maker’s needs. In its design documents, Bureau officials explained they would collect data using telephone interviews from Census Bureau contact centers, outbound interviewing, and telephone questionnaire assistance. The Bureau further explained that survey data and the information on the outcomes of calls would be provided to the survey team. Second, the Bureau discussed the level of difficulty in two of the test designs, but did not explain why it would be difficult to obtain the data. Bureau officials stated that with limited resources and based on the importance of the objectives of a given test, they will not be able to apply all the practices equally to every test design. For example, the 2013 QCT relied on tests of software by Bureau staff— not a traditional field test involving contact with households. So, although the Bureau identified that it may be difficult to reliably identify deviations in procedures using GPS, it did not include an explanation or mitigation of the possible difficulty. Third, the Bureau only identified factors that may interfere with data collection in the 2013 NCCT design.

Data Analysis Plan

Pre-specifying a data analysis plan as part of a test design can help researchers select the most appropriate data to measure and the most accurate and reliable ways to collect them. Although the Bureau generally followed two of the four practices related to a data analysis plan in the test designs that we reviewed, its discussion of the possible limitations of findings or test results varied (see table 4).

For all of the test designs, the Bureau generally identified a basis for comparing the test results and included a proposed design or research plan that was directly related to the objectives and/or questions. In addition, analytical techniques were proposed for two of three of the test designs. While in the 2012 NCT design the Bureau documented that the information from re-interviewing respondents will be used to validate their initial responses, officials did not discuss how they would match this information. Lastly, only the 2013 NCCT design included a discussion of possible limitations. For example, in its design the Bureau noted that one of the data sources used in the test might not be representative. Discussing possible limitations is important so that it is clear what the test design can and cannot address, and so that test results are not overly generalized.

Sample and Survey

A survey with an appropriate sample design can provide descriptive information about a population and its subgroups, as well as information about relationships among variables being measured. In addition, researchers should consult prior relevant research and test any new questions so that the survey questions will elicit appropriate information from respondents to address the Bureau’s data needs. For all three tests, the Bureau generally followed two of the sample and survey practices, while use of the third practice varied across the field tests (see table 5).

Across the test designs, the Bureau included both a discussion of how to reach the intended sample and the status of the survey instrument or questionnaire, as applicable. While all of the test designs included a rationale for the type of sample, the 2013 NCCT design also included a rationale for the size of its sample. Bureau documents showed this sample size was selected due to the absence of any documentation of a prior study and the test team’s conservative estimation of the response rate. Further, the Bureau noted the estimated response rate and selected sample size needed to enable the team to determine the quality and comprehensiveness of the data in its analysis. Test designs that explain how their sampling methodology will yield information of sufficient quality for its intended purpose provide a better justification for their cost.

Stakeholders and Resources

Managing stakeholders, identifying team member responsibilities, and identifying resources are key to a test’s success, as people are the primary resource of a high-performing organization. For the 2013 test designs, the Bureau followed all of the stakeholder and resource practices. The 2012 test design did not follow three of the four practices (see table 6).

The Bureau included a timeline and required resources in each field test design, including how much each field test would cost. The Bureau also generally followed the other practices related to stakeholders and resources for two of the test designs. However, for the 2012 NCT, officials did not identify stakeholders, their respective roles in the test, or their involvement in developing the test design. The latter two tests were designed with management plans for communication, stakeholder engagement, and governance, which for example, states that stakeholders’ roles should be defined and that their feedback should be gathered. The 2013 Quality Census Test design documented the role that a stakeholder had in outlining how the Bureau can increase data accuracy. By including these practices in guidance, the Bureau has better ensured that its people and resources are being effectively and efficiently leveraged during the development of future 2020 Census tests.

Design Process Management

Good management of the design process can help managers identify factors that can affect test quality, such as potential risk of delays and challenges to the lines of communication. Across the three tests, the Bureau’s governance process for developing test designs varied in following the four practices (see table 7).

First, the Bureau identified clear reporting relationships for only the 2013 QCT. It partially followed this practice for the 2013 NCCT design, and did not follow it for the 2012 NCT design. For the two latter tests, the Bureau utilized membership lists and responsibilities matrices to identify test and project teams, and the assigned tasks and deliverables. Second, the Bureau identified review and approval roles for two of the test designs, but not for the 2012 NCT. For example, for the 2013 QCT design, the Bureau identified which individuals were supposed to review and approve certain design documents. When authority is clearly assigned and communicated, individuals can be held responsible accordingly.

Third, the Bureau’s documentation of performance measures and timelines with associated milestones for all three test designs did not identify how the measures would be monitored. For example, for the 2013 QCT design the Bureau included a list of deliverables with associated dates, such as sending an initial study plan to senior Bureau officials for their review. However, it did not indicate how the Bureau would know whether these deliverables were implemented by the indicated dates. Measuring and monitoring performance allows Bureau managers to track progress toward their goals and have crucial information on which to base their organizational and management decisions.

Fourth, the Bureau did not follow its guidance for approving its 2013 NCCT test design and only partially followed it for the 2013 QCT design. According to the Bureau’s August 2012 research and testing management plan, the test designs should be approved at four different stages. The test design phase is complete after the fourth approval. The Bureau partially followed this practice for the 2013 QCT by documenting approval of its design at only one of the stages. According to Bureau meeting records, senior Bureau officials discussed the 2013 NCCT design after its implementation. Further, Bureau records indicate that senior Bureau officials discussed the 2012 NCT design, before the August 2012 management plan was issued, but did not document approval. In July 2013, the Bureau began using a table that includes test- design approval dates. This practice helps ensure that management’s approval of a plan maintains its relevance and value to management in control over operations. Further, documenting that management has approved a design provides accountability and offers transparency as to when decisions were made.

Recent Changes to a Design Template Could Help the Bureau Better Follow Key Practices in Future Test Designs

The Bureau’s design templates outline the information that should be included in two of its key design documents, the field test overview and the field test plan. The templates list topics to be discussed in the overview and plan, and, in some cases, provide examples of what staff should include for a topic. We found that the templates did not address some of the practices we identified for a sound study plan. For example, the templates did not require a test design to include (1) discussion of potential biases, (2) identification of factors that could interfere with obtaining data, (3) identification of difficulties in collecting data, and (4) specification of stakeholder’s roles. In response to this audit, the Bureau subsequently revised its field test template to include these four practices as topics to be discussed.

The Bureau Has Already Begun Incorporating Lessons Learned from Initial Test Designs

As the Bureau works to develop field tests to inform decisions about the 2020 Census, Bureau officials are learning lessons that can strengthen the design of future tests. According to Bureau officials, our audit helped to reinforce the Bureau’s need to draw on early lessons learned from initial tests. These lessons were derived from examining where the Bureau did not follow best practices for study designs and identifying corrective actions. The Bureau has adopted some of these test design lessons and is taking steps to adopt others. Table 8 lists six lessons learned from the initial three field tests.

One lesson the Bureau identified is the importance of obtaining buy-in from management early in the test development process. While designing the three initial tests, Bureau field test designers did not brief senior Bureau management on the development of the designs or involve them in the planning or review of data collection methods. In addition, managers of various Bureau divisions responsible for methodology and other subject matter areas requested to be involved in the process earlier. According to Bureau officials, without early involvement, it may be difficult to obtain upper management approval of test designs quickly, which can lead to unexpected late changes or delays in testing. Early managerial involvement can help ensure early agreement on goals, objectives, timing, and capabilities needed to support a test. This lesson complements the practices of identifying stakeholders benefiting from the field test as well as stakeholders involved in the preparation of the design.

To involve management earlier, Bureau officials began briefing upper management about the planning of tests during other regularly scheduled agency-wide executive meetings early in the test planning stages. Officials also started conducting one-day planning sessions beginning with tests planned for fiscal year 2014. Since beginning these sessions, Bureau officials said they have improved at communicating input from external experts at the National Academy of the Sciences to upper Bureau management. Further, officials said they have found the sessions to be effective in identifying issues early. Bureau officials said they now intend to hold these planning sessions for each test.

Obtain Internal Expert Review

In January 2013, the Bureau began convening a Bureau-wide test strategy review body. This panel of experts first met after the 2013 National Census Contact Test was implemented. In February 2013, Bureau officials decided that prior to conducting future tests, design teams would present the plans, sample design, and objectives to the panel. According to the Bureau, the panel will now look at the Bureau’s research strategy and goals, design decisions, and how the field tests will affect design decisions in fiscal years 2014 and 2015, and clarify operational milestones. The first pre-implementation presentation was conducted in February 2013 for the 2013 Quality Control Test. This allowed the test team to clarify the 2013 QCT’s purpose and verify its testing methodology with Bureau-wide experts. Bureau research managers believe the test’s design is better because of these meetings, and expect future test designs to benefit similarly.

Conduct Post-Test Reviews

During our review, we discussed with Bureau officials whether the Bureau took steps to evaluate the test development process after the three initial tests. The officials told us they recently started conducting staff reviews of tests they have implemented. Such post-test reviews allow the Bureau the opportunity to identify any further lessons learned from developing tests to improve either the design or management of remaining tests for the 2020 Census. The Bureau conducted its first post-test review following the 2013 National Census Contact Test. The review documented, for example, that involving stakeholders such as methodologists, in test planning and identifying their roles and responsibilities helps improve communication during the design process. Further, the review documented that test designs should not only identify responsible parties, but have information on what deliverables are expected of these parties. In addition, the Bureau also conducted a review of the 2013 Quality Control Test. Going forward, these reviews will provide the Bureau with additional opportunities to build its knowledge base on conducting small, targeted field tests.

Track the Status of Field Test Design Deliverables Across All Tests

The Bureau has taken steps to improve how it monitors the status of field test design deliverables. Bureau officials said that they previously reported on the status of some test deliverables in a biweekly report. However, these reports did not track the status of all deliverables across all the tests. As a result, senior decennial managers had to contact individual test team leaders to obtain the status for each of the initial test designs. Bureau officials acknowledged that our review led them to realize that with additional field test designs being created, monitoring across all field tests would improve their test status reporting process, and increase their efficiency in collating status information for managers. In July 2013, a Bureau official informed us that they began using a new tracking sheet to monitor the progress of field test deliverables. The new tracking tool provides a more comprehensive and global perspective on the status of deliverables across all tests. Bureau officials described this as an evolving process and said that they plan to take additional steps to develop a process for monitoring the status of field tests as well.

Ensure Team Leaders Are Informed of Key Design Elements

The Bureau has also recognized the importance of keeping team leaders informed about key design elements. According to the Bureau, design teams are required to submit certain documents for field test design reviews and approvals. Testing guidance is available electronically. Newly assigned team leaders are individually emailed links to baseline documents. New team leaders are also provided a team leader handbook. However, the handbook does not identify which documents are required for field test design development, nor does it indicate which documents were required for submission for test design reviews or approvals. Without having a listing of required documents, the Bureau risks duplicating its efforts to keep team leaders informed of key design elements. To ensure that team leaders are consistently informed of field test development guidance and the documents that should be prepared to support test design reviews and approvals, Bureau officials said they plan to include a listing of such documents in the team leader handbook. Bureau officials acknowledged that our work offered a way of improving how some information is disseminated to team leaders. The effort to revise the team leader handbook is in progress, but Bureau officials could not provide a timeline for completing it. Achieving a consistent understanding among team leaders of documents required for field test design approvals could help reduce possible delays in the test design review process.

Maintain Reporting Structures That Support Clear Communication

In response to this audit and as part of its effort to adapt its management structures to oversee multiple census field tests being developed concurrently, Bureau officials say that they are realigning field test governance processes to improve communication and accountability. Further, they said that the Bureau has already taken steps to identify one point of contact for each future test. Previously, a field test coordinator had to track input from various project team leaders involved with a particular census field test. This lesson complements the practice of identifying clear internal reporting relationships, including who reports to whom and points of contact for different types of information for a sound research plan. In addition to identifying reporting relationships, the Bureau acknowledged that taking steps like establishing one point of contact will help it to more effectively maintain clear lines of communication, and establish accountability when it develops a field test.

While the Bureau has taken some initial steps to implement its proposed restructuring, such as conducting a field test management group meeting to further integrate the 2020 field tests across projects, it has not formalized other proposed field test management restructuring and guidance revisions. For example, to improve the coordination of field test planning, the Bureau has proposed creating a field test management team that would provide centralized coordination and streamline the test processes. Further, Bureau officials said that the research and testing plan is under review and being updated to reflect the current process for approving field test designs and plans. Internal controls require that agencies complete, within established time frames, all actions that correct or otherwise resolve the matters brought to management’s attention. The controls also require management to periodically evaluate the organizational structure and make changes as necessary in response to changing conditions. But, the proposed restructuring and guidance revisions have not yet been formalized. The Bureau has many other competing priorities that may need attention more urgently and officials could not provide us a timeline or milestones for formalizing the changes. Meanwhile, the Bureau is continuing work on the design of tests. Without a timeline and milestones for this restructuring, the Bureau risks uncoordinated management for its field tests. This puts the field test’s effectiveness and efficiency regarding possible overlap and duplication at risk.

Lastly, it is important for the Bureau to document the lessons it has learned from designing its initial tests. In conducting post-test reviews, the Bureau documented some lessons learned from the 2013 National Census Contact Test and 2013 Quality Control Test. However, Bureau officials acknowledged that the Bureau did not consistently document the lessons learned from the design phases for the initial tests. Internal controls require that managers use control activities, such as the production of records or documentation, to aid in managing risk. Documenting these lessons can help reduce the risk of repeating prior deficiencies or inconsistencies that may lead to test development delays. Given the long time frames involved in planning the census, documentation is essential to ensure lessons are incorporated into future tests.

Conclusion

In its effort to design smaller and more targeted tests for the 2020 Census, the Bureau has taken important steps that could make its testing strategy more effective. The Bureau’s investments in early testing are intended to validate the use of strategies and innovations geared toward reducing cost. The three test designs we reviewed generally or partially followed most key practices for designing a sound research plan. The Bureau has already begun taking corrective actions on others, in part by adding additional requirements for designs to its standard guidance. Finalizing planned revisions that focus on field test management in the team leader handbook can help improve how team leaders learn about test design elements. Formalizing its proposed field test management restructuring and guidance revisions will enable the Bureau to ensure that there is improved accountability, communication, and monitoring of its test management design processes. Further, documenting lessons learned while designing the initial field tests can increase the Bureau’s ability to take advantage of the prior experiences. By ensuring these practices are consistently used in field tests, the Bureau will increase the soundness of the tests in areas such as design management and stakeholder involvement. This, in turn, will enhance the likelihood that the Bureau achieves its goal of conducting a cost effective census.

Recommendation for Executive Action

We recommend that the Secretary of Commerce require the Under Secretary for Economic Affairs who oversees the Economics and Statistics Administration, as well as the Director of the U.S. Census Bureau, to take the following three steps to improve the Bureau’s process of designing its census field tests for the 2020 Census:

Finalize planned revisions that focus on field test management in the team leader handbook.

Set a timeline and milestones for formalizing proposed field test management restructuring and guidance revisions.

Document lessons learned from designing initial field tests.

Agency Comments and Our Evaluation

We provided a draft of this report to the Department of Commerce for comment. In its written comments, reproduced in appendix II, the Department of Commerce concurred with our recommendations. The Department of Commerce also provided minor technical comments that were incorporated, as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of Commerce, the Under Secretary of Economic Affairs, the Director of the U.S. Census Bureau, and interested congressional committees. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff have any questions about this report please contact me at (202) 512-2757 or goldenkoffr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. The GAO staff that made major contributions to this report is listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

The objectives of our review were to determine to what extent the Bureau followed key practices for a sound study plan in designing the earliest 2020 Decennial Census field tests, and to identify any lessons learned from the design process that may help improve future tests. To identify key practices for a sound study plan, we reviewed program evaluation literature, including our design evaluation guide, our 2004 review of Census Bureau overseas field tests, our 2012 review of the planning of the 2020 Census, and our guide to internal controls.

We selected 25 key practices from these sources. We shared these practices with the Bureau and it agreed that they were reasonable. Using these criteria, we evaluated whether the Bureau’s three initial field test designs followed key practices for a sound research plan. Additionally, since the program evaluation literature noted the importance of program management to developing a sound study plan, we also interviewed Bureau officials on the management processes used for developing the designs. We did not evaluate the outcomes of the field tests.

To determine to what extent the designs for the initial 2020 Decennial Census field tests were consistent with key practices for a sound study plan, we reviewed Bureau design documents and interviewed Bureau officials about the field test design process. We compared each of the 25 practices to the Bureau’s field test design documents for the three initial tests to answer the question of whether the respective practice was followed. Our determinations provide a measure of the general rigor of the test designs, although they do not recognize the extent to which the Bureau may have considered the key practices later in the life cycle of the designs.

After comparing documents provided by the Bureau for each field test to the key practices and determining the extent each practice was followed for each test, we verified the determinations by having different auditors independently determine the extent practices were followed for 25 percent of each others’ initial determinations for each test. We rated each practice as being either “generally followed,” “partially followed,” or “not followed.” We also discussed our preliminary findings with Bureau officials to learn of additional context or documents that we might have missed. Table 9 describes how we made our determinations.

For each test, we limited our scope to the “design” of the tests, which is the first three of eight phases of the field test life cycle and includes initiation, concept development, and planning, analysis and design.

Our reviews of test designs were designed as snapshots “as of” the approval of the designs by senior management, or at an equivalent stage of their life cycle, intended to benchmark or baseline the preparation of future test designs. As such, our determination that a given test design did not follow a given key practice does not mean that the Bureau did not consider that key practice later in the test’s life cycle.

To identify lessons learned from how the tests were designed, we examined where the Bureau had not followed key practices and identified corrective actions needed. We determined the extent to which the key practice criteria were followed and then considered whether there was a pattern or an underlying cause, such as a lack of guidance. We then discussed with Bureau officials what lessons they had learned and what lessons they could implement for future field tests.

We conducted this performance audit from January 2013 to October 2013 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Commerce

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Ty Mitchell, Assistant Director; Maya Chakko; Robert Gebhart; Ellen Grady, Wilfred Holloway; Andrea Levine; Donna Miller; and Aku Pappoe made key contributions to this report.