2010 Census: Census Bureau Needs Procedures for Estimating the Response Rate and Selecting for Testing Methods to Increase Response Rate
Released: 2008-10-30T00:00:00

Letter

Background

In 1970, the Bureau moved away from conducting the census door-to-door and began mailing census questionnaires to households to be filled out and returned by mail. Since the 1970 Census, the Bureau has used mail as the primary method for collecting census data. For Census 2000, the mailout/mailback method was used for more than 80 percent of the population. Households that fail to mail back the census questionnaires are included in nonresponse follow-up workload, where enumerators follow up with door-to-door visits and telephone calls or solicit census data from knowledgeable people, such as neighbors.

The census response rate declined dramatically from 1970 through 1990. The 1970 response rate was 78 percent. The rate decreased to 75 percent for the 1980 Census and then decreased again to 65 percent for the 1990 Census. Although the Bureau estimated that the 2000 Census response rate would continue to decline to 61 percent, actual response exceeded the estimate, reaching 65 percent for the mailout/mailback universe prior to nonresponse follow-up. The response rate is defined as the percentage of census forms completed and returned for all housing units that were on the Bureau’s address file eligible to receive a census questionnaire delivered by mail or by a census enumerator. The denominator used in calculating the response rate includes vacant housing units and other addresses that were determined to be undeliverable or deleted through other census operations.

Since 1970, per household census costs have increased (in constant 2010 dollars) from about $14 in 1970 to an estimated $88 in 2010—a figure that does not include the recent, major redesign of the Field Data Collection Automation program. According to the Bureau, factors contributing to the increased costs include an effort to accommodate more complex households, busier lifestyles, more languages and greater cultural diversity, and increased privacy concerns. In addition, the number of housing units—and hence, the Bureau’s workload—has continued to increase. The Bureau estimated that the number of housing units for the 2010 Census will increase by almost 14 percent over 2000 Census levels (from 117.5 million to 133.8 million housing units). As a result of multiple factors, the total inflation-adjusted life cycle cost for the decennial census has increased more than tenfold since 1970, as shown in figure 1.

The Bureau conducts its testing, evaluation, and experimentation program for the decennial census primarily through the Decennial Statistical Studies Division. The division develops and coordinates the application of statistical techniques in the design and conduct of decennial census programs. The Bureau conducts various tests throughout the decade to determine the effect and feasibility of form or operational changes. For example, in the 1990s, the Bureau tested whether adding a mandatory response message and implementing a multiple mailing strategy, such as sending an advance letter and reminder postcard, as shown in figures 2 and 3, would increase the response rate. Based on positive test results, these methods were added to the Census 2000 design.

The Decennial Statistical Studies Division also conducts evaluations of census operations typically using data collected as part of the decennial process to determine whether individual steps in the census operated as expected. It is also responsible for conducting experiments during the census that may be instructive for future censuses. These experiments usually involve using alternative processes for a subset of the population. The Bureau is working with internal and external stakeholders on defining and developing its program for experiments and evaluations for the 2010 Census, known as Census Program for Evaluations and Experiments, with design work and implementation starting in 2008 and continuing through 2011. The final set of activities would include analysis, documentation, and presentation of the research, and these activities would start in 2009 and be completed by the end of fiscal year 2013.

The Response Rate Estimate Lacks Support, Is Not Systematically Reevaluated, and Is Not Clearly Incorporated into Planning Efforts

2010 Census Response Rate Estimate Is Not Fully Supported

The Bureau developed the response rate estimate for the 2010 Census in 2001 but could not demonstrate support for one of the three components underpinning the estimate. To establish the 2010 estimate, Bureau officials told us that they used Census 2000 mail response data as the baseline and then incorporated three other components to arrive at a response rate estimate of 69 percent: an increase in response related to eliminating the long form and moving to a short-form-only census, an increase in response related to sending a replacement questionnaire to nonresponding households, and a general decline in mail response due to decreasing public participation in surveys. The components of the estimate are outlined in figure 4, and a detailed explanation of the baseline and each component follows.

The baseline repone rte i the rte chieved in milot/milback reas in Census 2000 when the nonrepone follow-p worklod was identified.

Tritioning to hort-form- only census expected to increase reponecause the long form, ent to one-ixth of the poption in the past, has lower repone rte hitoriclly.

Miling replcement qtionnire to ll household tht hve not reponded certin dte has een hown to increase mil repone in te.

Prticiption i expected to decline asrt of trend of decreasing public prticiption in survey.

Baseline Rate

Included in the baseline national response rate of 65 percent are the effects of methods, such as the multiple mailing strategy and the communications campaign (called the Partnership and Marketing Program) that were implemented in Census 2000 and are planned to be implemented in the 2010 Census as well. Bureau officials stated that the communications campaign, which included paid advertising, had an impact on the response rate achieved in 2000, but were unable to quantify that effect and did not project the campaign’s effect for 2010.

Short-Form-Only Census

The Bureau estimated an increase of 1 percentage point from eliminating the long form and moving to a short-form-only census. The Bureau conducted an analysis comparing Census 2000 response rates for the short form and the long form. Even though the difference in response rates for the two form types was rather large—the short form had a response rate that was more than 10 percentage points higher than the long form—the overall effect on the response rate estimate was small because the long form was sent to only approximately 17 percent of housing units in 2000.

Replacement Mailing

In 2001, the Bureau expected that sending a replacement questionnaire to households that had not responded by a certain date would increase the response rate by 7 percentage points. The magnitude of this component was based on test results from the 1990s, which showed an increase in the response rate related to the replacement mailing of at least 7 percentage points, and possibly 10 percentage points or more. Bureau officials stated that they estimated this effect conservatively because they assumed that a higher response rate from a replacement questionnaire is more likely to occur in tests than in the decennial census. Also, testing showed that not all the respondents who return replacement questionnaires would be removed from the nonresponse follow-up workload prior to being enumerated because of the timing of the enumerator’s visit.

General Decline

The Bureau estimated a decrease of 4 percentage points from the baseline level due to what it believes is a decline over time in public survey participation. However, this assumption is not supported by quantitative analysis or research studies but rather based on the opinion of subject matter experts and senior officials. Further, the Bureau could not demonstrate who was consulted, when they were consulted, or how they decided on the amount of the general decline.

Best practices for cost estimation from our Cost Assessment Guide call for documenting assumptions and data sources used to develop cost estimates, such as the general decline component of the response rate estimate. In addition, several experts we interviewed agreed that the Bureau should have used quantitative analysis of mail response data from other surveys to support the general decline component when it developed the estimate in 2001. To help support the general decline component, we suggested that the Bureau use changes in participation in the American Community Survey, which uses a similar mailing strategy to the decennial census and for which response is also required by law, for comparison purposes. In May 2008, the Bureau completed an analysis of American Community Survey response rate data from 2000 to 2007, which demonstrated a decline of 6.6 percentage points in cooperation rates to the initial questionnaire. Although the decline in response to the American Community Survey may not be directly comparable to response behavior for the decennial census, it provides some support for the Bureau’s assumption that mail survey participation may be declining.

The Bureau does not have procedures for developing the response rate estimate. Specifically, the Bureau has no established policies for documenting deliberations related to or data sources used in developing the estimate, including no 2010 Census decision memorandum to document the original estimate prepared in 2001. Several experts also suggested that the Bureau should develop a model for developing the estimate and incorporate response rate data and demographic data from other surveys.

Bureau officials, however, noted that they have not determined how to apply data from other surveys to estimate the census response rate because the census differs from other surveys, nor do they plan to develop a model of the response rate estimate. Instead, Bureau officials stated that they tried to conservatively estimate the response rate to ensure that they are adequately prepared during nonresponse follow-up and to avoid repeating what happened in 1990, when the Bureau overestimated the response rate, requiring a supplemental appropriation of $110 million and forcing it to extend nonresponse follow-up by up to 8 weeks for some areas. In contrast, in 2000 the response rate exceeded the Bureau’s estimate. Conservatively estimating the response rate may be a reasonable approach. However, having clear, detailed documentation about decisions related to or data sources used in developing the estimate would enable the Bureau and others to better assess whether the estimate is reliable, such as whether its assumptions are supported and realistic. An unreliable response rate estimate can produce an inaccurate cost estimate and can increase risk and uncertainty to operational plans that are based on the response rate estimate.

The Bureau Does Not Systematically Reevaluate the Response Rate Estimate

Although the Bureau updated the response rate estimate as a result of a major redesign of census operations undertaken in April 2008, the Bureau still lacks procedures for establishing when and how to reevaluate the response rate estimate. From 2001 through April 2008, the Bureau did not reevaluate the estimate to determine whether it should be updated, even though, after establishing the initial estimate in 2001, the Bureau completed several evaluations from Census 2000 and conducted several census tests that could have informed the estimate. During the first few years of the decade, the Bureau completed 12 evaluations that discussed aspects of the Census 2000 response rate. In addition, the Bureau conducted five tests from 2003 through 2007 that were designed, in part, to test methods to increase response. For example, in 2005 and 2007, the Bureau tested a two-column bilingual form, which includes identical questions and response options in English and Spanish. The 2005 test demonstrated a significant positive effect on mail response rates of 2.2 percentage points overall, and a 3.2 percentage point increase in areas with high concentrations of Hispanic and non-White populations, and the 2007 test revealed a similar impact. However, Bureau officials stated that they were concerned that the two-column bilingual form has not been tested in a census environment. In addition, they noted that the bilingual form will be sent to only a portion of the country—approximately 10 percent of census tracts—with high concentrations of Spanish-speaking populations, based on American Community Survey data.

Best practices state that assumptions, such as the response rate estimate, should be revised as new information becomes available. Our Cost Assessment Guide recommends that preliminary information and assumptions be monitored to determine relevance and accuracy and be updated as changes occur to reflect the best information available. Further, according to several experts, the Bureau could have updated the estimate based on response rate data and demographic data from other surveys, and another expert suggested that the Bureau use triggering events, such as after tests, for reviewing the response rate estimate. Based on this expert’s experience, the Bureau could establish a change control board chaired by senior Bureau officials to determine whether the response rate estimate should be revised. However, Bureau officials explained that they do not update the estimate based on results from tests because the tests cannot replicate the decennial census environment. Overall, the Bureau has not specified when or how it is to reevaluate its response rate estimate. Establishing these procedures would help the Bureau ensure that the estimate is current and reliable in order to better inform planning efforts.

The Bureau revised the estimate down to 64 percent after announcing that nonresponse follow-up would be changed from an automated to a paper- based operation. Prior to the redesign, the Bureau had planned to reduce the nonresponse follow-up workload by using handheld computers to remove households that returned their forms late—including many replacement mailing returns—from the enumerator assignments on a daily basis. According to Bureau officials, they revised the response rate estimate based on the timing of mail returns in 2000 and replacement questionnaire returns in the 2003 test. However, the revised estimate does not fully reflect recently designed procedures to remove late mail returns in 2010. Specifically, for 2010 the Bureau now plans to mail the replacement questionnaire earlier, send a blanket replacement questionnaire to some areas and a targeted replacement questionnaire to others, and conduct three clerical removals of late mail returns immediately prior to and during the nonresponse follow-up operation. These operational plans are still being finalized pending further analysis. Bureau officials said that although they hope that these revised operations will increase response, they did not update the response rate estimate to reflect these current operational plans because they have not tested this approach under decennial census conditions and therefore have no basis for estimating the potential effect of these operational changes on response.

Local Response Rate Estimates Used in Planning Efforts Lack Support

To estimate costs and to plan for activities such as nonresponse follow-up, questionnaire printing, and postage needs, the Bureau uses a life cycle cost model that relies on response rate estimates for four primary LCO types and not the national estimate. Using LCO-type estimates helps to account for expected differences in response patterns across geographic settings. For example, response rates for inner-city areas are estimated to be lower than in suburban areas. These estimates range from 55 percent to 72 percent, but lack support for how they were developed. In determining the 2010 response rate estimates for the four LCO types, the Bureau said that it equally applied two components from the national estimate—the replacement mailing component and the short-form-only component—to the LCO types in the cost model. However, the Bureau did not conduct quantitative analysis to determine whether the replacement mailing component in the national estimate should be applied equally for each LCO type. By applying an equal percentage point increase across all LCO types for the replacement mailing, the Bureau has, in effect, assumed that the LCO type with the lowest baseline—48 percent—would experience a higher relative increase in response—15 percent—due to the replacement mailing than the LCO type with the highest baseline—65 percent—which would be expected to experience an 11 percent increase in response. The Bureau has not demonstrated support for assuming that the LCO types will experience these different relative increases in response. Further, the Bureau could not demonstrate whether or how the short-form-only component and the general decline component of the national estimate were reflected in the life cycle cost model.

Best practices for cost estimation from our Cost Assessment Guide call for assumptions to be thoroughly documented, data to be normalized, and each cost element to be supported by auditable and traceable data sources. Following these best practices could allow the Bureau to enhance its use of the response rate estimate in planning for the decennial census and better inform stakeholders about the reliability of the estimate. It is unclear why officials could only explain how one component of the national response rate estimate was applied to LCO-type estimates in the life cycle cost model. However, according to the Bureau, it is currently documenting how these estimates are used to calculate the costs for operations and how the estimates have changed over time. Having support for the LCO-type estimates would better inform the Bureau’s planning efforts for operations that directly rely on response, such as determining workload and hiring enumerators for nonresponse follow-up.

The Bureau Plans to Incorporate Additional Methods to Increase Mail Response in 2010

Two of Nine Methods Tested to Increase Response Rate Are Planned to Be Included in the 2010 Census

We determined from reviewing Bureau testing documents that through various national and field tests and experiments, the Bureau tested nine methods to increase self-response to the 2010 Census, as shown in table 1. The Bureau currently plans to implement two of these methods—the replacement questionnaire and two-column bilingual form (see fig. 5). Three other methods for increasing mail response and four methods for increasing response through an electronic data collection system were tested but are not planned for implementation in 2010. An additional method the Bureau plans to include in the 2010 Census to increase response, the Integrated Communications Campaign, has not been tested, although the Bureau conducted a similar campaign in 2000 and plans to test campaign messages with audiences in early 2009. Additional details regarding the testing of these methods can be found in appendix II.

For the 2010 Census, the Bureau does not plan to include three methods it tested to increase mail response, as shown in figures 6, 7, and 8. Two of these methods—a telephone reminder call and messaging to distinguish the initial and replacement questionnaires—were not found to significantly improve response and therefore will not be implemented as part of the 2010 Census. The third method—including a due date on the initial mailing package—was found to generate faster responses when tested in 2003, and increase overall response by 2 percentage points when tested in conjunction with a compressed mailing schedule in 2006. However, the Bureau also believes that the use of a due date alone could cause a lower response rate because some people may not send back the census form after the due date. According to Bureau officials, they are not including this method in the decennial census design because they would like to further test it—both with and without a compressed mailing schedule—in a census environment. It will be important for the Bureau to optimize this testing opportunity by designing the test to determine the extent to which the faster and higher response is due to a compressed schedule versus a due date, as well as exploring other test treatments the Bureau has recommended in the past, such as including the due date on multiple mailing pieces. Bureau officials said that additional testing on the use of a due date will be conducted as part of the 2010 Census Program for Evaluations and Experiments, which the National Academy of Sciences recommended in its December 2007 interim report. The Bureau did not provide further details on plans for the Census Program for Evaluations and Experiments, but officials have said that they will consider costs and staffing needs in deciding what to evaluate. The academy’s final report is due in September 2009.

In addition to testing methods to increase mail response, the Bureau also tested four methods to increase self-response through electronic data collection systems: (1) providing for response via an interactive voice response system;( 2) sending a letter to encourage either responding via Internet or returning the initial questionnaire, instead of sending a replacement questionnaire; (3) using computer-assisted telephone interviewing; and (4) providing capability for responding via the Internet. The Bureau found that the first two methods did not increase response in tests and does not plan to include them in the 2010 Census. Computer- assisted telephone interviewing allows respondents to use the telephone to connect with operators who record interview responses electronically. The Bureau found that this method increased the overall response rate in a Census 2000 experiment. However, in a March 2004 report, the Bureau also stated that the method would likely be too costly in terms of hardware, software, and staffing resources, compared to the increase in response it might generate. Computer-assisted telephone interviewing was not tested after the 2000 Census.

Although the Internet option was found to increase overall response during the Census 2000 experiment, it did not increase overall response when tested again in 2003, when a replacement mailing was also tested. According to a July 2006 Bureau decision memo titled, “Rationale for the Decision to Eliminate the Internet Option from the DRIS Contract,” the Bureau decided not to include the Internet option in the design of the 2010 Census largely because it had underestimated the costs of the contract that included developing the Internet response option. In responding to a draft of this report, the Bureau stated that they made the decision because test results showed that offering the Internet option did not increase overall response and would not offer any cost savings. According to the 2006 memo, the Bureau also determined that the operational risks and costs to develop this option outweighed the potential benefits. In terms of benefits, the Bureau found improvements in data completeness and data timeliness from Internet responses in tests conducted in 2000/2001 for the American Community Survey and in 2003 and 2005 for the National Census Tests. The Bureau noted that these benefits could translate into reduced costs since less follow-up is required to improve data accuracy, and earlier receipt of responses could result in fewer replacement questionnaires that need to be mailed and fewer households that need to be enumerated during nonresponse follow-up. However, with only 6 to 7 percent of the test population using the Internet option, the Bureau concluded that no cost savings could be realized from reducing the number or size of data capture centers (facilities that process returned questionnaires) planned for 2010. Finally, the Bureau stated that the inability to fully test the Internet option and growing concerns about Internet security made it unfeasible for the Bureau to implement the Internet as a response option for 2010. Despite its July 2006 decision, the Bureau recently announced that it is considering including the Internet option as part of the 2010 Census design; however, the Bureau has not developed plans for further testing this option.

Testing and Evaluation Plans for the 2010 Communications Campaign Are Not Yet Fully Developed

Through its contractor and subcontractors, the Bureau has taken a number of steps to inform the planning of the communications campaign but has not yet fully developed the campaign’s testing and evaluation plans. From late 2006 through early 2008, focus groups were conducted with various ethnic groups and hard-to-count populations, such as unattached singles, to identify potential barriers and motivators to participation, to better understand methods of communication that work for different groups, and to develop the campaign’s overall creative expression. The Bureau also developed an audience segmentation model using Census 2000 response data, updated through 2006 using American Community Survey data, to provide a more detailed understanding of the characteristics, such as home ownership and unemployment level, of those more or less likely to respond, as well as where they live, in order to better target communications to encourage census participation. In addition, in 2008, a national phone and in-person survey is being conducted to further explore barriers and motivators to response, particularly in hard-to-count populations, which would inform campaign message development.

According to agency officials, beginning in early 2009, campaign messages are to be tested by showing storyboards to audiences that will use electronic devices to vote on the messages. These messages will be tested in 14 languages and for other populations, such as Spanish speakers in Puerto Rico. According to the draft plan, testing of events, partnership toolkits, promotional items, and public relations ideas will also be conducted. However, the Bureau has not yet developed detailed plans for this testing because, according to one official, the Bureau intends to further develop testing plans as future fiscal year funding amounts become available. An example of a Census 2000 communications campaign poster is shown in figure 9.

In addition, although the Bureau expects to award a contract by the end of fiscal year 2008 for an independent evaluation measuring the campaign’s performance against its goals of increasing mail response, improving accuracy, and reducing the differential undercount, and improving cooperation with enumerators, it has not yet done so. In the past, the Bureau has said that although evaluations have shown that the Census 2000 communications campaign increased awareness for that census, it was difficult to link increased awareness to changes in respondent behavior. Bureau officials said that they have attempted to analyze Census 2000 data to identify factors that influence behavior, but their research results were inconclusive. Going forward, it will be important for the Bureau to determine its plans for evaluating the 2010 communications campaign so that it does not miss opportunities to collect data in the census environment to inform future campaigns.

Bureau Lacks Procedures for Selecting for Testing Methods to Increase Response Rate

For 2010, the Bureau developed a strategy to identify various methods to test for increasing response. Specifically, the Bureau established test objectives and research questions, such as identifying the optimal mix of response options for the public to respond to the 2010 Census and determining security and confidentiality issues surrounding technology. The Bureau also developed the 2010 Census Integrated Program Plan and 2010 Census Operations and Systems Plan to better document its planning. However, Bureau officials did not document the methods that they considered but decided not to test in the 2010 testing cycle or the rationale behind those decisions. Further, for methods that the Bureau decided to test, officials could not provide support for how they selected or prioritized these methods. Although officials said that they considered cost, complexity of the test, and compatibility of experiments in their decisions, they did not specify how they defined or weighed these factors to select and prioritize the nine methods they chose to test.

To ensure thorough and comprehensive planning of the decennial census, our past work on lessons learned from Census 2000 highlighted the importance of documentation to support research, testing, and evaluation, and a comprehensive and prioritized plan of goals, objectives, and projects. While the Bureau has developed the 2010 Census Integrated Program Plan and 2010 Census Operations and Systems Plan to better document its planning, these and other planning documents we reviewed did not provide support for how the Bureau selected and prioritized methods to test. It is unclear why the Bureau lacks procedures for documenting decisions concerning how it selected for testing methods to increase response. Documenting decisions about methods that were not selected for testing would help the Bureau more effectively build capacity and institutional knowledge about changes in methods to consider in the future.

According to Bureau officials, they consider the experience of other survey organizations when identifying methods for increasing response rate. For example, they said that they attend research conferences to learn about experiences of other organizations that conduct national surveys to identify potential methods to increase response rate. Both Bureau officials and some survey experts noted that methods used to increase response in other surveys may only be indirectly applicable to the decennial census. For example, for the Economic Census, officials said that the Bureau sends nonresponding businesses multiple reminder letters; the final letter to large businesses informs them that the Department of Justice will be pursuing nonrespondents. Bureau officials said that these methods are less feasible for the decennial census to implement because of the shorter time frame for obtaining responses and concerns about being respondent- friendly to households. In addition, one survey expert noted that methods applicable to small-scale surveys, such as personalizing a cover letter, may be less feasible to implement for the decennial census.

Further, survey experts we interviewed generally said that they were unaware of additional methods from censuses undertaken by other countries or private sector surveys that the Bureau could consider to increase mail response. Some experts noted differences between how the United States and other countries conduct their censuses, which may make it difficult to directly transfer other countries’ practices to the U.S. census. For example, some European censuses use a population register to collect names, unlike the United States, which builds its survey frame from phone and address lists. In addition, past research has provided evidence that government-sponsored self-administered surveys, such as the decennial census, tend to achieve higher response rates than nongovernmental surveys.

Nonetheless, testing modifications to methods that the Bureau has previously considered or tested in earlier studies may yield additional opportunities for increasing response. For example, the Bureau could test various telephone reminder messages stating that response is mandatory by law or providing instructions for obtaining a new census form. Although Bureau officials said that they have previously used the American Community Survey to inform the census design, analyzing respondent behavior to the American Community Survey, because of its similar mailing strategy to the decennial census, could help the Bureau regularly refine its survey methodology for increasing census response. Although the survey forms are different, many concepts, such as targeting the second mailing, modifying the appearance of the mailing, and varying telephone and Internet messages to prompt nonrespondents, could be tested with reasonable inference to the census.

Conclusions

Nonresponse follow-up is the largest field operation, and the Bureau estimates that it will cost more the $2 billion. To control the cost of nonresponse follow-up, it will be important for the Bureau to devise a strategy for getting people to return their census forms. A reliable response rate estimate is a critical element necessary for determining the resources needed to carry out nonresponse follow-up. The Bureau did not have support for one of the components of the 2010 response rate estimate—a general decline in responsiveness—and the Bureau does not have procedures for reviewing the estimate after testing to determine whether it should be revised. Establishing procedures for developing the response rate estimate, including documenting data sources and decisions, would enable the Bureau and others to better assess the estimate’s reliability. Also, establishing procedures for when, such as after tests or other triggering events, and how to reevaluate the estimate would help the Bureau ensure that it is providing the most current response rate estimate for planning nonresponse follow-up and other activities, such as questionnaire printing. The Bureau’s strategy of estimating the response rate conservatively may be prudent given past difficulties with conducting nonresponse follow-up after overestimating the response rate in 1990. Nonetheless, establishing and following procedures for developing, documenting, and reevaluating the estimate are important steps for understanding differences between the estimate and the actual response rate for 2010 and for evaluating the components and underlying assumptions when developing the estimate for the next census. Successful enumeration depends on early research, testing, and evaluation of census methods to increase response. Establishing procedures for selecting and prioritizing the testing of methods—such as the Internet or reminder telephone call—including documenting methods considered but not tested, would help the Bureau demonstrate that it has chosen an optimal research strategy for the decennial census, more effectively build capacity and institutional knowledge about changes in methods to consider in the future, and enable it to more efficiently begin testing for the next census.

Recommendations for Executive Action

To enhance credibility of the response rate for determining cost and planning for future census activities, to inform assumptions underlying the 2020 response rate estimate, and to improve the planning and transparency of the Bureau’s research and testing, we are recommending that the Secretary of Commerce direct the Bureau to take the following three actions: Establish procedures for developing the 2020 response rate estimate, including documenting the data sources supporting the estimate's components and decisions that are made in establishing the components and analyzing 2010 data to assess the reasonableness of assumptions used in applying the national estimate’s components to the LCO-type estimates.

Establish procedures for reevaluating and updating the 2020 estimate, including identifying events or changes in related operations that should trigger a review and documenting the results of such reviews.

Establish procedures for selecting methods for increasing response rate that will be the subject of research and testing, including requirements for documenting how the Bureau defines and weighs factors used to select methods and documentation on methods considered but not tested.

Agency Comments and Our Evaluation

The Secretary of Commerce provided written comments on a draft of this report on September 22, 2008. The comments are reprinted in appendix III. Commerce generally agreed with our conclusions and recommendations and stated that the Bureau is committed to developing and implementing a documented, systematic methodology for establishing response rate estimates for future censuses and reevaluating the estimates throughout the decade. Because the recommendations in our draft report focused on costs and planning for the 2010 Census, we revised our recommendations to reflect actions to be taken to support future census planning, including analyzing 2010 data to assess the reasonableness of assumptions used in applying the national estimate’s components to the LCO-type estimates.

Commerce also provided technical corrections, which we incorporated as appropriate. In its comments, Commerce disagreed with our statement that the Internet response option increased response in the Census 2000 experiment. Commerce cited a summary statement from a Bureau report that concluded that the use of multiple response modes tested in 2000 does not increase response. However, Bureau analyses from October 25, 2002, and March 2004 on the Census 2000 experiment stated that the overall response rate increased when the Internet was offered as an alternative response mode. We therefore made no changes in the report.

Commerce strongly disagreed with our statement about the reason why the Bureau decided not to include the Internet option in the design of the 2010 Census. In the report, we state that underestimating contract costs was the primary reason the Bureau eliminated the Internet option, which we attribute to a July 19, 2006, memo documenting the Bureau’s rationale for eliminating the Internet response option from the Decennial Response Integration System (DRIS) contract. This memo states that the decision “was due largely to the fact that the Census Bureau underestimated the FY 2006-2008 contractor costs proposed to develop DRIS.” We therefore left this statement unchanged. However, we added the Bureau’s explanation provided in its agency comment letter that the decision was based on test results, which showed that offering this option did not increase overall response and would not offer any cost savings.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days after its issue date. At that time, we will send copies of this report to the Secretary of Commerce, the Department of Commerce’s Inspector General, the Director of the U.S. Census Bureau, and interested congressional committees. We will make copies available to others upon request. This report will also be available at no charge on GAO’s Web site at http://www.gao.gov.

If you or your staff have any questions concerning this report, please contact Mathew J. Scirè at (202) 512-6806 or sciremj@gao.gov or Ronald S. Fecso at (202) 512-2700 or fecsor@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

The objectives of this report were to (1) assess how the U.S. Census Bureau (Bureau) develops, supports, and updates the response rate estimate, and the extent to which the Bureau uses the response rate estimate to inform its 2010 planning efforts; (2) describe the methods the Bureau considered for increasing response rates in 2010 and what it did to test these methods; and (3) assess how the Bureau identifies and selects for testing methods to increase response rates, including considering other surveys’ methods for increasing response. The scope of our review was limited to the census mailout/mailback universe, which covered more than 80 percent of households in Census 2000. The majority of households in this category have standard city-style addresses, allowing them to receive their census questionnaires in the mail in addition to being expected to return their questionnaires by mail. We excluded from our review operations and methods aimed at enumerating those not included in the mailout/mailback universe, such as the Be Counted initiative and Enumeration at Transitory Locations; those initiatives related to, but not primarily focused on, increasing self-response, such as improving the quality of the Master Address File and providing assistance through Questionnaire Assistance Centers; methods tested prior to 2000 and already implemented in previous censuses; and those primarily intended to improve operational efficiency, such as postal tracking.

To determine how the Bureau develops and uses the response rate estimate, we reviewed documentation to support the components of the response rate estimate and research literature on efforts to estimate survey response rate. We interviewed Bureau officials in the Decennial Management Division about the process used to develop and update the estimate, the assumptions on which the estimate is based, and how the estimate is used. We also interviewed experts on survey methodology from Statistics Canada, the U.S. Department of Labor’s Bureau of Labor Statistics, and various academic institutions, as well as former Census Bureau officials and survey methodologists to obtain their views on the strengths and weaknesses of the Bureau’s process for developing and updating the estimate. We compared responses to a common set of questions we asked the experts in order to identify themes. We also reviewed Bureau strategic planning documents—in particular, the 2010 Census life cycle cost model with supporting documentation, as well as the 2010 Census Integrated Program Plan and the 2010 Census Operations and Systems Plan—to understand the Bureau’s use of the estimate, and evaluated the Bureau’s practices for using the response rate estimate for generating the life cycle cost estimate against best practices criteria in our Cost Assessment Guide and other relevant GAO products.

To describe the methods the Bureau considered for increasing response rates in 2010 and how the Bureau tested these methods, we reviewed Bureau analyses and evaluations of tests on methods to increase mail response and various Bureau planning documents, such as the 2010 Census Operations and Systems Plan. In addition, we reviewed documentation on the Bureau’s communications campaign, such as evaluations of the 2000 Partnership and Marketing Program and the 2010 Census Integrated Communications Campaign Plan. We interviewed Bureau officials in the Decennial Management Division to obtain additional context about the methods they considered and tests they conducted.

To assess how the Bureau identifies and selects methods to test for increasing response rates, we reviewed various Bureau planning documents, such as research and development planning group action plans, for factors that the Bureau considers in selecting methods to test. We interviewed Bureau officials in the Decennial Management Division and other divisions about the process for identifying and selecting methods for increasing response. We also reviewed our past work on lessons learned from Census 2000 on the importance of documentation and planning in order to evaluate the Bureau’s process for selecting and prioritizing methods to test. We interviewed experts on survey methodology from Statistics Canada, the U.S. Department of Labor’s Bureau of Labor Statistics, and various academic institutions, as well as former Census Bureau officials and researchers, about additional methods for increasing response that the Bureau could consider and to obtain their perspectives on the Bureau’s process for identifying, testing, and implementing methods. We compared responses to a common set of questions we asked the experts in order to identify recurring themes. We also reviewed research literature on methodologies to increase response to mail surveys to identify additional methods that the Bureau could consider.

We interviewed the following experts in survey methodology: Anil Arora, Director General, Census Program Branch, Statistics Paul Biemer, Distinguished Fellow in statistics, Research Triangle Don A. Dillman, Regents Professor, Thomas S. Foley Distinguished Professor of Government and Public Policy, and Deputy Director for Research and Development in the Social and Economic Sciences Research Center, Washington State University, and former Senior Survey Methodologist, U.S. Census Bureau Elizabeth Martin, former Senior Survey Methodologist, U.S. Census Colm O’Muircheartaigh, Vice President for Statistics and Methodology, National Opinion Research Center and Professor, Irving B. Harris Graduate School of Public Policy Studies, University of Chicago Kenneth Prewitt, Carnegie Professor of Public Affairs, School of International and Public Affairs, Columbia University and former Director, U.S. Census Bureau Clyde Tucker, Senior Survey Methodologist, Bureau of Labor Statistics We conducted our review from July 2007 through September 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Testing of Methods to Increase Response to 2010 Census

Replacement mailing. The Bureau plans to include the replacement mailing in the 2010 Census design. The replacement mailing involves sending a new questionnaire to households after households have received the initial census questionnaire in order to increase the likelihood that households will respond. The Bureau tested a targeted replacement mailing—where households that have not returned their initial questionnaires by a cutoff date receive the replacement form—in the early 1990s and found that this method resulted in a 10 to 12 percent increase in the mail response rate. A blanket replacement mailing—where all households received a replacement questionnaire, including those that had already responded—was planned for Census 2000, but the Bureau dropped it from the design because of operational concerns that became apparent in the 1998 dress rehearsal. A targeted replacement mailing was incorporated into the 2010 design, based on the test results from the 1990s and the Bureau’s plans to move to a short-form-only census. In 2003, the Bureau tested the targeted replacement mailing and was able to confirm the results of the tests from the early 1990s. Subsequent testing of the replacement mailing focused on ensuring that the Bureau could implement it successfully, including processing and removing the returns from the nonresponse follow-up workload using the handheld computers.

The decision to eliminate the handheld computers from nonresponse follow-up and make nonresponse follow-up a paper-based operation diminished the Bureau’s ability to remove late mail returns (including replacement questionnaires that are returned) from the nonresponse follow-up workload. The Bureau then examined options to get a better effect from the replacement mailing and decided to try to get the replacement mailings out more quickly. In order to do this, the Bureau decided that it would send a blanket replacement mailing to census tracts identified as low response areas, based on Census 2000 response data and updated demographic data from the American Community Survey. The replacement mailing packets for these tracts can be printed and labeled in advance—approximately 25 million total. Households not in these areas will receive a replacement questionnaire on a targeted basis—that is, only if they are in census tracts identified as medium response areas and if their initial questionnaires have not been received by the cutoff date. The Bureau estimates that it can print and label 15 million of these targeted replacement mailing packages in 4 days. Under this schedule, the Bureau will finish mailing out the replacement questionnaires on April 8, whereas under the original plan the replacement questionnaire mail out would not be completed until April 19. As a result, more households should be able to return the replacement questionnaires in time to be removed from the nonresponse follow-up workload.

Two-column bilingual form. The Bureau plans to use a two-column bilingual form in certain locations in 2010. The two-column bilingual form provides two response columns, one in English and one in Spanish. Each column contains the same questions and response options, and respondents are instructed to choose the language that is most comfortable for them. The Bureau tested the bilingual form in 2005 to determine whether it would (1) increase overall response to the census and (2) lower item nonresponse (when a household returns the form but does not respond to a particular question) when compared to the standard English form. In the 2005 test, the two-column bilingual form panel demonstrated a significant positive effect on mail response rates of 2.2 percentage points overall, and a 3.2 percentage point increase in areas with high concentrations of Hispanic and non-White populations, but did not achieve lower rates of item nonresponse. The Bureau conducted another test of the two-column bilingual form in 2007, which produced response rate results similar to the 2005 results. In 2010, the Bureau plans to mail the two-column bilingual form to households in communities with heavy concentrations of Spanish speakers and areas with low English proficiency.

Telephone reminder call. The Bureau does not plan to use a telephone reminder call in 2010. The multiple contact strategy that the Bureau used in 2000 included a reminder postcard mailed 1 week after the initial questionnaire packets were mailed. As part of the 2003 National Census Test, the Bureau tested the effect of an automated telephone reminder call in place of a reminder postcard on increasing the mail response rate. Out of a sample of 20,000 households, the Bureau was able to obtain phone numbers for 6,208 (31 percent), and these households received telephone calls to remind them to return their questionnaires if they had not already done so. The initial results indicated a significantly higher cooperation rate for the telephone reminder call panel when compared to the control panel, which received reminder postcards.

The Bureau later conducted a supplementary analysis to determine if the higher cooperation rate was related to an underlying higher propensity to cooperate among households with listed telephone numbers. This analysis compared cooperation rates of two groups of households: those for which telephone numbers were available and those for which telephone numbers were not available. Among households for which a telephone number was available, the Bureau observed no significant difference in cooperation rates for housing units that received a reminder postcard compared to those that received a reminder telephone call. The Bureau does not plan to use a telephone reminder call in place of a reminder postcard in the 2010 Census.

Due date on initial questionnaire mailing package. The Bureau will not place a due date on the initial questionnaire mailing package as part of the 2010 Census design. The value of placing a due date on the mailing package with the initial questionnaire is that it might evoke a sense of urgency or importance in respondents, leading to increased response to the census. In 2003, when the Bureau tested this method, it found that the inclusion of a due date did not significantly affect the overall response rate. However, it did result in a significantly higher response rate at an earlier point in the test, which can decrease the size of the replacement questionnaire mailing.

The Bureau tested a due date again in 2006, this time combined with a “compressed schedule,” in which questionnaires were mailed 14 days (as opposed to 21 days, as in the control group) before Census Day. A due date for mailing the questionnaire back was given in the advance letter, on the outgoing envelope and cover letter for the questionnaire mailing, and on the reminder postcard. The final report of the test concluded that the inclusion of the due date and compressed schedule resulted in a significantly higher mail response rate, by 2.0 percentage points. In addition, several measures of data completeness and coverage showed significant improvement. Bureau officials decided not to include the use of a due date in the 2010 Census, noting that they would like to conduct further testing to better understand the effects of a due date versus a compressed schedule. The Bureau is considering whether to test both the use of a due date and a compressed schedule in its 2010 Census Program for Evaluations and Experiments.

Messaging to distinguish the replacement from the initial questionnaire. The Bureau does not plan to use a message to distinguish the replacement questionnaire from the initial questionnaire in 2010. The inclusion of a message on the replacement questionnaire informing respondents that they did not need to return the replacement questionnaire if they had already provided a response from the previous mailing was thought to be a way to reduce the number of multiple returns, and possibly improve overall response. The Bureau tested this method in 2005, and while the results showed a significant decline in the submission of multiple returns, they also showed a significant decrease in the response rate of 1.2 percentage points. Including a message to distinguish the replacement from the initial questionnaire was not recommended for, and will not be included in, the 2010 Census.

Methods to Increase Self-Response Using Electronic Data Collection Systems

Interactive voice response. The Bureau does not plan to use interactive voice response in 2010. Interactive voice response allows respondents to use the telephone to respond verbally to digitized voice files that contain census questions and instructions. Speech recognition software is used to determine and record responses. Interactive voice response was tested in 2000, with households given the choice of providing their census data via interactive voice response or a paper questionnaire. The response rate for the interactive voice response panel was not statistically different from that of the control (mail only) group. However, the results are difficult to interpret because a portion of the sample in the interactive voice response panel either received the census form late or did not receive it at all.

In 2003, the Bureau tested two different strategies for getting respondents to provide their census data using interactive voice response. One strategy, known as a push strategy, did not include a paper questionnaire in the initial mailing but rather pushed respondents to reply using interactive voice response. The other strategy, known as a choice strategy, included an initial paper questionnaire with the interactive voice response information on it, allowing respondents to choose their response mode. All nonrespondents received a paper replacement questionnaire. Households that had a choice of responding via paper or interactive voice response had overall response rates similar to households that only received paper, with about 7 percent of respondents given the choice using interactive voice response. Households in the push strategy had a significantly lower response rate—4.9 percentage points lower than households that only received a paper questionnaire. The Bureau does not plan to give respondents the option of providing their census data using interactive voice response in 2010.

Letter encouraging Internet response or returning initial questionnaire instead of sending replacement questionnaire. The Bureau will not send a letter encouraging Internet response or returning the initial questionnaire instead of sending a replacement questionnaire in 2010. Instead of mailing a replacement paper questionnaire to nonrespondents, a letter could be sent to households encouraging them either to complete the paper questionnaire that they previously received or to use the Internet to submit their responses. The Bureau tested this method in 2005, sending a replacement mailing that contained a letter with an Internet address and an identification number needed to access the Internet questionnaire in place of a paper replacement questionnaire. Compared to sending out a paper replacement questionnaire, this method resulted in significantly fewer responses overall, decreasing the response rate by 3.7 percentage points. The Bureau will not include a letter encouraging the use of the Internet instead of a replacement questionnaire in 2010, in part because the Internet option was dropped from the 2010 Census design.

Computer-assisted telephone interviewing. The Bureau will not use computer-assisted telephone interviewing for increasing response in 2010. Computer-assisted telephone interviewing allows respondents to use the telephone to connect with operators, who conduct interviews and record responses electronically. In a 2000 experiment, households were given the option of returning a paper form or providing their census data via computer-assisted telephone interviewing. When compared to households that were only given paper questionnaires, computer-assisted telephone interviewing brought about a 2.06 percentage point improvement in the overall response rate and also had a low item nonresponse rate. However, it entailed substantial costs for hardware, software, and programmer and interviewer time. The Bureau has not tested computer-assisted telephone interviewing since 2000 and does not plan to use this option in 2010.

Internet. The Internet response option allows respondents to use an Internet-based questionnaire—with screens designed to resemble the paper questionnaire—to respond to the census. Respondents answer multiple-choice questions by clicking the appropriate buttons and checkboxes and text-entry questions by typing their answers into response fields. The Bureau gave respondents the option to respond by Internet in a 2000 experiment. Some households in the test were given the choice of providing their census data via Internet or a paper questionnaire. The experiment found that the Internet response option resulted in a 2.46 percentage point increase in response.

In 2003, the Bureau again tested allowing respondents the option of providing their answers via the Internet by sending a paper questionnaire along with instructions for responding via the Internet in the initial mailing. Households that had a choice to respond by paper or the Internet had a similar overall response rate to households that were provided only paper, with about 10 percent of respondents choosing to respond by Internet.

The Bureau decided not to include the Internet in the 2010 Census, despite including it in the scope of the contract awarded in 2005 for the Decennial Response Integration System. The Bureau noted that this decision was based on a number of factors, including the Bureau’s underestimation of the contractor costs for the first 3 years of the contract, as well as test results that indicated that the Internet would not increase the response rate and concerns about the security of respondents’ data prior to the Bureau receiving it.

Communications campaign. Census 2000 included a greatly expanded outreach and promotion campaign—including, for the first time, paid advertising—in an attempt to increase public awareness of and promote positive attitudes about the census. This program, called the Partnership and Marketing Program, was considered a success after the Bureau reversed the trend of declining mail response rates, and the Bureau made plans to continue the program for 2010. Bureau officials stated that the 2010 campaign consolidates all census communications under a single communications contract. This Integrated Communications Campaign aims to motivate the entire populations of the 50 states, the District of Columbia, Puerto Rico, and other U.S. territories to participate in the census through partnerships (with community groups, businesses, colleges, faith-based organizations, and other targeted groups); public relations; events; Census in Schools; and paid advertisements in broadcast, print, and online media.

Bureau officials noted that the advertising campaign would not be included in the dress rehearsal because the Bureau’s experience including the advertising campaign in the 1998 Dress Rehearsal did not provide the feedback needed to revise the creative aspects of the campaign. To refine the communications campaign, the Bureau conducted an audience segmentation analysis to identify how to best reach people with the paid advertising campaign. In addition, the Bureau conducted focus groups in 2006 and 2007 to provide information on what motivates individuals to respond.

The communications campaign is scheduled to run from mid-2008 through June 2010. In 2008 and 2009, most of the activities are focused on preparing and mobilizing partnerships. Further development of specific messages for various audiences and communication channels will take place from November 2008 through April 2009. Starting in mid-2009, the partnership program will begin outreach to certain hard-to-count populations. Activities and events targeting all audiences will begin in January 2010 and peak in March 2010.

Appendix III: Comments from the Department of Commerce

Appendix IV: GAO Contacts and Staff Acknowledgments

Acknowledgments

In addition to contacts named above, Lisa Pearson, Assistant Director; David Bobruff; Don Brown; and Elizabeth Fan made key contributions to this report. Tom Beall, Susan Etzel, Andrea Levine, Donna Miller, Ellen Rominger, and Elizabeth Wood provided significant technical support.