Federal Information Collection: A Reexamination of the Portfolio of Major Federal Household Surveys Is Needed
Released: 2006-12-15T00:00:00

Letter

Background

The purpose of the PRA is to (1) minimize the federal paperwork burden for individuals, small businesses, state and local governments, and other persons; (2) minimize the cost to the federal government of collecting, maintaining, using, and disseminating information; and (3) maximize the usefulness of information collected by the federal government. The PRA also aims to provide for timely and equitable dissemination of federal information; improve the quality and use of information to increase government accountability at a minimized cost; and manage information technology to improve performance and reduce burden, while improving the responsibility and accountability of OMB and the federal agencies to Congress and the public.

To achieve these purposes, the PRA prohibits federal agencies from conducting or sponsoring an information collection unless they have prior approval from OMB. The PRA requires that information collections be approved by OMB when facts or opinions are solicited from 10 or more people. Under the law, OMB is required to determine that an agency information collection is necessary for the proper performance of the functions of the agency, including whether the information will have practical utility.

The PRA requires every agency to establish a process for its chief information officer (CIO) to review program offices’ proposed information collections, such as certifying that each proposed collection complies with the PRA, including ensuring that it is not unnecessarily duplicative. The agency is to provide two public notice periods—an initial 60-day notice period and a 30-day notice period after the information collection is submitted to OMB for approval. Agencies are responsible for consulting with members of the public and other affected agencies to solicit comments on, among other things, ways to minimize the burden on respondents, including through the use of automated collection techniques or other forms of information technology. According to an OMB official, this could include asking for comments on a proposal to use administrative data instead of survey data.

Following satisfaction of these requirements, an agency is to submit its proposed information collection for OMB review, whether for new information collections or re-approval of existing information collections. Before an agency submits a proposed information collection for approval, an agency may invest substantial resources to prepare to conduct an information collection. An agency may undertake, among other things, designing the information collection, testing, and consulting with users. For example, over the last 8 years, BLS has led an interagency effort designed to develop a measure of the employment rate of adults with disabilities pursuant to Executive Order 13078 signed by President Clinton in 1998. This effort has entailed planning, developing, and testing disability questions to add to the CPS. OMB is responsible for determining whether each information collection is necessary for the proper performance of the agency’s functions. According to the Statistical Programs of the United States Government: Fiscal Year 2006, an estimated $5.4 billion in fiscal year 2006 was requested for statistical activities.

The PRA also requires the establishment of the Interagency Council on Statistical Policy (ICSP). According to the Statistical Programs of the United States Government: Fiscal Year 2006, the ICSP is a vehicle for coordinating statistical work, particularly when activities and issues cut across agencies; for exchanging information about agency programs and activities; and for providing advice and counsel to OMB on statistical matters.

The PRA also requires OMB to annually report on the paperwork burden imposed on the public by the federal government and efforts to reduce this burden, which is reported in Managing Information Collection: Information Collection Budget of the United States Government. For example, the 2006 Information Collection Budget reported on agency initiatives to reduce paperwork, such as HHS’s assessment of its information collections with a large number of burden hours, which resulted in reducing the department’s overall burden hours by over 36 million in fiscal year 2005.

OMB produces the annual Statistical Programs of the United States Government report to fulfill its responsibility under the PRA to prepare an annual report on statistical program funding. This document outlines the effects of congressional actions and the funding for statistics proposed in the President’s current fiscal year budget, and highlights proposed program changes for federal statistical activities. It also describes a number of long-range planning initiatives to improve federal statistical programs, including making better use of existing data collections while protecting the confidentiality of statistical information.

More Than 500 Statistical or Research Surveys Have Been Approved

At the time of our review, OMB had approved 584 new and ongoing statistical and research surveys as recorded in the database of OMB- approved information collections. OMB uses the database for tracking purposes, as it provides the only centralized information available on the characteristics of the surveys that OMB has approved. The database contains information on some, but not all, of the characteristics of the information collections. The information that agencies provide in the packages they submit to OMB for approval includes additional data, such as the estimated cost.

Statistical and research surveys represent about 7 percent of the total universe of 8,463 OMB-approved information collections, the majority of which, as shown in figure 1, are for regulatory or compliance and application for benefits purposes. Although there are certain surveys funded through grants and contracts that are not approved by OMB under the PRA, OMB stated that there is no comprehensive list of these surveys.

Forty percent of OMB-approved statistical and research surveys were administered to individuals and households, as shown in figure 2.

Annual estimated burden hours are defined as the amount of time for the average respondent to fill out a survey times the number of respondents. Figure 3 shows the range of burden hours, for general purpose research and statistics information collections, with about 35 percent of the surveys each accounting for 1,000 or fewer total burden hours.

According to an OMB official, the electronic system, Regulatory Information Service Center Office of Information and Regulatory Affairs Consolidated Information System, has automated the agency submission and OMB review process. This new system, which was implemented in July of 2006, is intended to allow OMB and agency officials to search information collection titles and abstracts for major survey topics and key words.

Table 2 provides information from agency officials and documents for the selected surveys that we reviewed in more depth. For these seven surveys, the sample sizes ranged from 5,000 individuals for the NHANES to 55,000 housing units for the AHS. The NHANES has a much smaller sample size and greater cost (as compared to the other surveys with similar burden hours) because it includes both an interview and a physical examination in a mobile exam center. The physical examination can include body measurements and tests and procedures, such as a blood sample and dental screening, to assess various aspects of respondents’ health. Other differences among the surveys we reviewed included their specific purposes (e.g., to obtain health information or demographics data); the time period considered (some of the surveys provide data as of a certain point in time while others are longitudinal and follow the same respondents over a period of time); and the frequency with which the surveys were conducted.

In addition, many of these surveys have been in existence for decades. Of the seven surveys we reviewed, five are defined by the Statistical Programs of the United States Government Fiscal Year 2006 as major household surveys (ACS, AHS, CPS, NHIS, and SIPP), and in addition MEPS’s household sample is a sub-set of NHIS’s sample. The ACS, unlike the other surveys, is mandatory and will replace the decennial census long-form. In addition to the surveys that we reviewed, two other surveys, the Consumer Expenditure Surveys and the National Crime Victimization Survey, are also defined by the Statistical Programs of the United States Government of 2006 as major household surveys.

Agencies and OMB Have Procedures Intended to Identify and Prevent Unnecessary Duplication

Agencies and OMB have procedures intended to identify and prevent unnecessary duplication in information collections. Agencies are responsible for certifying that an information collection is not unnecessarily duplicative of existing information as part of complying with OMB’s approval process for information collections. OMB has developed guidance that agencies can use in complying with the approval process. Once an agency submits a proposed information collection to OMB, OMB is required to review the agency’s paperwork, which includes the agency’s formal certification that the proposed information collection is not unnecessarily duplicative.

Agencies are Responsible for Identifying and Preventing Unnecessary Duplication

“For example, unnecessary duplication exists if the need for the proposed collection can be served by information already collected for another purpose - such as administrative records, other federal agencies and programs, or other public and private sources. If specific information is needed for identification, classification, or categorization of respondents; or analysis in conjunction with other data elements provided by the respondent, and is not otherwise available in the detail necessary to satisfy the purpose and need for which the collection is undertaken; and if the information is considered essential to the purpose and need of the collection, and/or to the collection methodology or analysis of results, then the information is generally deemed to be necessary, and therefore not duplicative within the meaning of the PRA and OMB regulation.”

When an agency is ready to submit a proposed information collection to OMB, the agency’s CIO is responsible for certifying that the information collection satisfies the PRA standards, including a certification that the information collection is not unnecessarily duplicative of existing information sources. We have previously reported that agency CIOs across the government generally reviewed information collections and certified that they met the standards in the act. However, our analysis of 12 case studies at the Internal Revenue Service (IRS) and the Department of Veterans Affairs, HUD, and DOL, showed that the CIOs certified collections even though support was often missing or incomplete. For example, seven of the cases had no information and two included only partial information on whether the information collection avoided unnecessary duplication. Further, although the PRA requires that agencies publish public notices in the Federal Register and otherwise consult with the public, agencies governmentwide generally limited consultation to the publication of the notices, which generated little public comment. Without appropriate support and public consultation, agencies have reduced assurance that collections satisfy the standards in the act. We recommended that the Director of OMB alter OMB’s current guidance to clarify the kinds of support that it asks agency CIOs to provide for certifications and to direct agencies to consult with potential respondents beyond the publication of Federal Register notices. OMB has not implemented these recommendations.

OMB Is Responsible for Reviewing Agencies’ Efforts to Identify and Prevent Unnecessary Duplication

OMB has three different guidance publications that agencies can consult in the process of developing information collection submissions, according to OMB officials. The three guidance publications address unnecessary duplication to varying degrees. The draft, Implementing Guidance for OMB Review of Agency Information Collection, provides, among other things, instructions to agencies about how to identify unnecessary duplication of proposed information collections with existing available information sources.

OMB’s Questions and Answers When Designing Surveys for Information Collections discusses when it is acceptable to duplicate questions used in other surveys. The publication also encourages agencies to consult with OMB when they are proposing new surveys, major revisions, or large-scale experiments or tests, before an information collection is submitted. For example, when BLS was developing its disability questions for the CPS, BLS officials stated that they consulted OMB on numerous occasions. OMB officials also said that when they are involved early in the process, it is easier to modify an agency’s plan for an information collection.

OMB officials told us that an agency consultation with OMB before an information collection is developed can provide opportunities to identify and prevent unnecessary duplication. For example, according to an OMB official, while OMB was working with the Federal Emergency Management Agency (FEMA) to meet the need for information on the impact of Hurricane Katrina, OMB identified a survey partially funded by the National Institute of Mental Health (NIMH) that was in the final stages of design and would be conducted by Harvard University—the Hurricane Katrina Advisory Group Initiative. OMB learned that this survey, which was funded through a grant (and was not subject to review and approval under the PRA), planned to collect data on many of the topics that FEMA was interested in. OMB facilitated collaboration between FEMA and HHS and ultimately, FEMA was able to avoid launching a new survey by enhancing the Harvard study.

OMB’s draft of the Proposed Standards and Guidelines for Statistical Surveys, which focuses on statistical surveys and their design and methodology, did not require that agencies assess potential duplication with other available sources of information as part of survey planning. We suggested that OMB require that when agencies are initiating new surveys or major revisions of existing surveys they include in their written plans the steps they take to ensure that a survey is not unnecessary duplicative with available information sources. OMB has incorporated this suggestion.

Under the PRA, OMB is responsible for reviewing proposed information collections to determine whether a proposed information collection meets the PRA criteria, which include a requirement that it not unnecessarily duplicate available information. According to an OMB official responsible for reviewing information collections, OMB’s review process consists of several steps. She said that once an agency has submitted the proposed information collection package to OMB, the package is sent to the appropriate OMB official for review. When there is a need for clarification or questions exist, this OMB official told us that OMB communicates with the agency either through telephone conferences or via e-mail. After approval, OMB is required to assign a number to each approved information collection, which the agencies are then to include on their information collection (e.g., survey) forms.

In addition to its responsibilities for reviewing proposed information collections, OMB also contributes to or leads a wide range of interagency efforts that address federal statistics. For example, OMB chairs the ICSP. The ICSP is a vehicle for coordinating statistical work, exchanging information about agency programs and activities, and providing advice and counsel to OMB on statistical matters. The council consists of the heads of the principal statistical agencies, plus the heads of the statistical units in the Environmental Protection Agency, IRS, National Science Foundation, and Social Security Administration (SSA). According to an OMB official, the ICSP can expand its membership for working groups to address specific topics. For example, the ICSP established an employment-related health benefits subcommittee and included non-ICSP agencies, such as HHS’s AHRQ (which co-chaired the subcommittee). The ICSP member agencies exchange experiences and solutions with respect to numerous topics of mutual interest and concern. For example, in the past year, the council discussed topics such as the revision of core standards for statistical surveys opportunities for interagency collaboration on information technology development and investment and sample redesign for the major household surveys with the advent of the ACS.

Duplicative Content in Selected Surveys Exists, but Survey Purposes and Scope Differ

On the basis of OMB’s definition of unnecessary duplication, the surveys we reviewed could be considered to contain necessary duplication. To examine selected surveys to assess the extent of unnecessary duplication in areas with similar subject matter, we looked at surveys that addressed three areas: (1) people without health insurance (CPS, NHIS, MEPS, and SIPP), (2) people with disabilities (NHIS, NHANES, MEPS, SIPP, and ACS), and (3) the housing questions on the AHS and ACS. We found that the selected surveys had duplicative content and asked similar questions in some cases. However, the agencies and OMB judged that this was not unnecessary duplication given the differences among the surveys. In some instances, the duplication among these surveys yielded richer data, allowing fuller descriptions of specific topics and providing additional perspectives on a topic, such as by focusing on the different sources and effects of disabilities. The seven surveys we reviewed originated at different times and differ in many aspects, including the samples drawn, the time periods measured, the types of information collected, and level of detail requested. These factors can affect costs and burden hours associated with the surveys. In addition, the differences can create confusion in some cases because they produce differing estimates and use different definitions.

Surveys That Measure People without Health Insurance Produce Differing Estimates

Although the CPS, NHIS, MEPS, and SIPP all measure people who do not have health insurance, the surveys originated at different times and differ in several ways, including the combinations of information collected that relate to health insurance, questions used to determine health insurance status, and time frames. Health insurance status is not the primary purpose of any of these surveys, but rather one of the subject areas in each survey. In addition, because each survey has a different purpose, each survey produces a different combination of information related to people’s health insurance.

The CPS originated in 1948 and provides data on the population’s employment status. Estimates from the CPS include employment, unemployment, earnings, hours of work, and other indicators. Supplements also provide information on a variety of subjects, including information about employer-provided benefits like health insurance. CPS also provides information on health insurance coverage rates for sociodemographic subgroups of the population. The time frame within which data is released varies; for example, CPS employment estimates are released 2-3 weeks after collection while supplement estimates are released in 2-9 months after collection.

The NHIS originated in 1957 and collects information on reasons for lack of health insurance, type of coverage, and health care utilization. The NHIS also collects data on illnesses, injuries, activity limitations, chronic conditions, health behaviors, and other health topics, which can be linked to health insurance status. HHS stated that although health insurance data are covered on other surveys, NHIS’s data on health insurance is key to conducting analysis of the impact of health insurance coverage on access to care, which is generally not collected on other surveys.

The MEPS originated in 1977 and provides data on health insurance dynamics, including changes in coverage and periods without coverage. The MEPS augments the NHIS by selecting a sample of NHIS respondents and collecting additional information on the respondents. The MEPS also links data on health services spending and health insurance status to other demographic characteristics of survey respondents. The MEPS data can also be used to analyze the relationship between insurance status and a variety of individual and household characteristics, including use of and expenditures for health care services.

The SIPP originated in 1983 in order to provide data on income, labor force, and government program participation. The information collected in the SIPP, such as the utilization of health care services, child well-being, and disability, can be linked to health insurance status. The SIPP also measures the duration of periods without health insurance.

Because the surveys use different methods to determine health insurance status, they can elicit different kinds of responses and consequently differing estimates within the same population. To determine if a person is uninsured, surveys use one of two methods: they ask respondents directly if they lack insurance coverage or they classify individuals as uninsured if they do not affirmatively indicate that they have coverage. The CPS and the NHIS directly ask respondents whether they lack insurance coverage. While the difference between these approaches may seem subtle, using a verification question prompts some people who did not indicate any insurance coverage to rethink their status and indicate coverage that they had previously forgotten to mention.

The surveys also differ both in the time period respondents are asked to recall and in the time periods measured when respondents did not have health insurance. Hence, the surveys produce estimates that do not rely upon standardized time or recall periods and as a result are not directly comparable. The ASEC to the CPS is conducted in February, March, and April and asks questions about the prior calendar year. An interviewer asks the respondent to remember back for the previous calendar year which can be as long as 16 months in the April interview. The other three surveys, in contrast, asked about coverage at the time of the interview. Because a respondent’s ability to recall information generally degrades over time, most survey methodologists believe that the longer the recall period, the less accurate the answers will be to questions about the past, such as exactly when health insurance coverage started or stopped, or when it changed because of job changes. Another difference is the time period used to frame the question. The CPS asked whether the respondent was uninsured for an entire year, while NHIS, MEPS, and SIPP asked whether the individual was ever insured, or was uninsured at the time of the interview, for the entire last year, and at any time during the year.

Table 3 illustrates the differing estimates obtained using data from the four selected surveys. While these differences can be explained, the wide differences in the estimates are of concern and have created some confusion. For example, the 2004 CPS estimate for people who were uninsured for a full year is over 50 percent higher than the NHIS estimate for that year. HHS has sponsored several interagency meetings on health insurance data, which involved various agencies within HHS and the Census Bureau. The meetings focused on improving estimates of health insurance coverage and included, among other things, examining how income data are used, exploring potential collaboration between HHS and the Census Bureau on whether the CPS undercounts Medicaid recipients, examining health insurance coverage rates, and discussing a potential project to provide administrative data for use in the CPS. As a result, HHS created a Web site with reports and data on relevant surveys and HHS’s office of the Assistant Secretary for Planning and Evaluation (ASPE) produced the report Understanding Estimates of the Uninsured: Putting the Differences in Context with input from the Census Bureau in an effort to explain the differing estimates.

Surveys that Measure Disability Status Differ in Definitions, Purposes, and Methodologies Used

Similarly, although the NHIS, NHANES, MEPS, SIPP, and ACS all estimate the percentage of the population with disabilities, the surveys define disability differently and have different purposes and methodologies. In addition to these five surveys, which measure aspects of disability, BLS is also currently developing questions to measure the employment levels of the disabled population. HHS also stated that disability is included on multiple surveys so that disability status can be analyzed in conjunction with other information that an agency needs. For example, disability information is used by health departments to describe the health of the population, by departments of transportation to assess access to transportation systems, and departments of education in the education attainment of people with disabilities. The lack of consistent definitions is not unique to surveys; there are over 20 different federal agencies that administer almost 200 different disability programs for purposes of entitlement to public support programs, medical care, and government services.

Although each of the surveys asks about people’s impairments or functionality in order to gauge a respondent’s disability status, there are some differences in how disability is characterized. For example, the NHIS asks respondents if they are limited in their ability to perform age- dependent life and other activities. The NHIS also asks about the respondent needing assistance with performing activities of daily living and instrumental activities of daily living. The NHANES measures the prevalence of physical and functional disability for a wide range of activities in children and adults. Extensive interview information on self- reported physical abilities and limitations is collected to assess the capacity of the individual to do various activities without the use of aids, and the level of difficulty in performing the task. The MEPS provides information on days of work or school missed due to disability. The SIPP queries whether the respondent has limitations of sensory, physical, or mental functioning and limitations on activities due to health conditions or impairments. The ACS asks about vision or hearing impairment, difficulty with physical and cognitive tasks, and difficulty with self-care and independent living.

Because surveys produce different types of information on disability, they can provide additional perspectives on the sources and effects of disabilities, but they can also cause confusion because of the differences in the way disability is being measured. The NHIS contains a broad set of data on disability-related topics, including the limitation of functional activities, mental health questions used to measure psychological distress, limitations in sensory ability, and limitations in work ability. Moreover, the NHIS provides data, for those persons who indicated a limitation performing a functional activity, about the source or condition of their functional limitation. The NHANES links medical examination information to disability. The MEPS measures how much individuals spend on medical care for a person with disabilities and can illustrate changes in health status and health care expenses. The SIPP provides information on the use of assistive devices, such as wheelchairs and canes. Finally, the ACS provides information on many social and economic characteristics, such as school enrollment for people with disabilities as well as the poverty and employment status of people with different types of disabilities.

However, the estimates of disability in the population that these surveys produce can vary widely. A Cornell University study compared disability estimates among the NHIS, SIPP, and ACS. A number of categories of disability were very similar, such as the nondisabled population, while others, such as the disabled population or people with sensory disabilities, had widely varying estimates, as shown in table 4. For example, according to data presented in a Cornell University study that used survey questions to define and subsequently compare different disability measures across surveys, the SIPP 2002 estimate of people with sensory disabilities for ages 18-24 was more than six times the NHIS estimate for that year for ages 18-24. In commenting on this report, the DOC and HHS acknowledged that comparing the NHIS and SIPP with respect to sensory disabilities is problematic. HHS officials noted that the confusion caused by these different estimates derives mostly from the lack of a single definition of disability, which leads to data collections that use different questions and combinations of information to define disability status.

Because the concept of disability varies, with no clear consensus on terminology or definition, and there are differing estimates, several federal and international groups are examining how the associated measures of disability could be improved. HHS’s Disability Workgroup, which includes officials from HHS and the Department of Education, examines how disability is measured and used across surveys. The task of another federal group, the Subcommittee on Disability Statistics of the Interagency Committee on Disability Research, is to define and standardize the disability definition. The Washington Group on Disability Statistics (WGDS), an international workgroup sponsored by the United Nations in which OMB and NCHS participate, is working to facilitate the comparison of data on disability internationally. The WGDS aims to guide the development of a short set or sets of disability measures that are suitable for use in censuses, sample-based national surveys, or other statistical formats, for the primary purpose of informing policy on equalization of opportunities. The WGDS is also working to develop one or more extended sets of survey items to measure disability, or guidelines for their design, to be used as components of population surveys or as supplements to specialty surveys. HHS added that the interest in standardizing the measurement of disability status is also driven by the desire to add a standard question set to a range of studies so that the status of persons with disabilities can be described across studies.

The AHS and ACS Ask Some Similar Questions on Housing, but Their Purposes and Scope Differ

In 2002, we reported that the AHS and ACS both covered the subject of housing. Of the 66 questions on the 2003 ACS, 25 were in the section on housing characteristics, and all but one of these questions were the same as or similar to the questions on the AHS. For example, both the AHS and the ACS ask how many bedrooms a housing unit has. However, the two surveys differ in purposes and scope.

The purpose of the AHS is to collect detailed housing information on the size, composition, and state of housing in the United States, and to track changes in the housing stock over time, according to a HUD official. To that end, the AHS includes about 1,000 variables, according to a HUD official, such as the size of housing unit, housing costs, different building types, plumbing and electrical issues, housing and neighborhood quality, mortgage financing, and household characteristics. The AHS produces estimates at the national level, metropolitan level for certain areas, and homogenous zones of households with fewer than 100,000 households. The AHS is conducted every 2 years nationally and every 6 years in major metropolitan areas, except for six areas, which are surveyed every 4 years.

In contrast, the level of housing data in the ACS is much less extensive. The ACS is designed to replace the decennial Census 2010 long-form and covers a wide range of subjects, such as income, commute time to work, and home values. The ACS provides national and county data and, in the future, will provide data down to the Census tract level, according to a Census Bureau official. The ACS is designed to provide communities with information on how they are changing, with housing being one of the main topic areas along with a broad range of household demographic and economic characteristics.

The AHS and ACS also have different historical and trend data and data collection methods. The AHS returns to the same housing units year after year to gather data; therefore, it produces data on trends that illustrate the flow of households through the housing stock, according to a HUD official, while the ACS samples new households every month. Historical data are also available from the AHS from the 1970s onward, according to a HUD official.

Analysts can use AHS data to monitor the interaction among housing needs, demand, and supply, as well as changes in housing conditions and costs. In addition, analysts can also use AHS data to support the development of housing policies and the design of housing programs appropriate for different groups. HUD uses the AHS data, for example, to analyze changes affecting housing conditions of particular subgroups, such as the elderly. The AHS also plays an important role in HUD’s monitoring of the lending activities of the government-sponsored enterprises, Fannie Mae and Freddie Mac, in meeting their numeric goals for mortgage purchases serving minorities, low-income households, and underserved areas. AHS’s characteristic of returning to the same housing units year after year provides the basis for HUD’s Components of Inventory Change (CINCH) and Rental Dynamics analyses. The CINCH reports examine changes in housing stock over time by comparing the status and characteristics of housing units in successive surveys. The Rental Dynamics program, which is a specialized form of CINCH, looks at rental housing stock changes, with an emphasis on changes in affordability. Another use of AHS data has been for calculating certain fair market rents (FMR), which HUD uses to determine the amount of rental assistance subsidies for major metropolitan areas between the decennial censuses. However, HUD plans to begin using ACS data for fiscal year 2006 FMRs. As we previously reported, this could improve the accuracy of FMRs because the ACS provides more recent data that closely matches the boundaries of HUD’s FMR areas than the AHS.

In our 2002 report, which was published before the ACS was fully implemented, we also identified substantial overlap for questions on place of birth and citizenship, education, labor force characteristics, transportation to work, income, and, in particular, housing characteristics. We recommended that the Census Bureau review proposed ACS questions for possible elimination that were asked on the AHS to more completely address the possibility of reducing the reporting burden in existing surveys. The Census Bureau responded that they are always looking for opportunities to streamline, clarify, and reduce respondent burden, but that substantial testing would be required before changes can be made in surveys that provide key national social indicators.

The Advent of the ACS and the Proposed Reengineering of the SIPP Are Changes to the Portfolio of Major Household Surveys

In addition to efforts underway to try to reconcile inconsistencies among surveys that address the same subject areas, a number of major changes have occurred or are planned to occur that will affect the overall portfolio of major household surveys. As previously discussed, the ACS was fully implemented in 2005 and provides considerable information that is also provided in many other major household surveys. The ACS is the cornerstone of the government’s effort to keep pace with the nation’s changing population and ever-increasing demands for timely and relevant data about population and housing characteristics. The new survey will provide current demographic, socioeconomic, and housing information about America’s communities every year, information that until now was only available once a decade. Starting in 2010, the ACS will replace the long-form census. As with the long-form, information from the ACS will be used to administer federal and state programs and distribute more than $200 billion a year. Detailed data from national household surveys can be combined with data from the ACS to create reliable estimates for small geographic areas using area estimation models.

Partly in response to potential reductions in funding for fiscal year 2007, the Census Bureau is planning to reengineer the SIPP with the intent of ultimately providing better information at lower cost. SIPP has been used to estimate future costs of certain government programs. For example, HUD used SIPP’s longitudinal capacity to follow families over time to determine that households with high-rent burdens in one year move in and out of high-rent burden status over subsequent years. Therefore, although the overall size of the population with worst-case housing needs is fairly stable, the households comprising this population change with considerable frequency—an issue that HUD told us is potentially important in the design of housing assistance programs.

Although the SIPP has had problems with sample attrition and releasing data in a timely manner, which the reengineering is intended to ameliorate, there has been disagreement about this proposal among some users of SIPP data. Census Bureau officials said they are meeting with internal and external stakeholders and are considering using administrative records. Census Bureau officials told us that they could develop a greater quality survey for less money, with a final survey to be implemented in 2009. They also said that they may consider using the ACS or CPS sampling frame.

Agencies Have Undertaken Efforts to Improve the Efficiency and Relevance of Surveys

In addition to the seven surveys discussed previously, we also identified examples of how, over the years, agencies have undertaken efforts to enhance their surveys’ relevance and efficiency through steps such as using administrative data in conjunction with survey data, reexamining and combining or eliminating surveys, and redesigning existing surveys.

Agencies Have Used Administrative Data in Conjunction with Surveys

The Census Bureau and BLS have used administrative data collected for the administration of various government programs in conjunction with survey data. The Census Bureau and BLS have used the administrative data to target specific populations to survey and to obtain information without burdening survey respondents.

The Census Bureau uses administrative data in combination with survey data to produce its Economic Census business statistics, which, every 5 years, profile the U.S. economy from the national to the local level. The Economic Census relies on the centralized Business Register, which is compiled from administrative records from IRS, SSA, and BLS, along with lists of multi-establishment businesses that the Census Bureau maintains. The Business Register contains basic economic information for over 8 million employer businesses and over 21 million self-employed businesses. The Economic Census uses the Business Register as the sampling frame to identify sets of businesses with specific characteristics, such as size, location, and industry sector.

BLS also uses a combination of administrative and survey data to produce its quarterly series of statistics on gross job gains and losses. BLS uses administrative data provided by state workforce agencies that compile and forward quarterly state unemployment insurance (UI) records to BLS. These state agencies also submit employment and wage data to BLS. The data states provide to BLS include establishments subject to state UI laws and federal agencies subject to the Unemployment Compensation for Federal Employees program, covering approximately 98 percent of U.S. jobs. These administrative data enable BLS to obtain information on many businesses without having to impose a burden on respondents. BLS augments the administrative data with two BLS-funded surveys conducted by the states. The Annual Refiling Survey updates businesses’ industry codes and contact information, and the Multiple Worksite Report survey provides information on multiple work sites for a single business, data that are not provided by the UI records, enabling BLS to report on business statistics by geographic location. Combining the data from these surveys with administrative data helps BLS increase accuracy, update information, and include additional details on establishment openings and closings.

However, because of restrictions on information sharing, BLS is not able to access most of the information that the Census Bureau uses for its business statistics because much of this information is commingled with IRS data. The Confidential Information Protection and Statistical Efficiency Act of 2002 (CIPSEA, 44 U.S.C. § 3501 note) authorized identifiable business records to be shared among the Bureau Economic Analysis (BEA), BLS, and the Census Bureau for statistical purposes. CIPSEA, however, did not change the provisions of the Internal Revenue Code that preclude these agencies from sharing tax return information for statistical purposes. OMB officials stated that there is continued interest in examining appropriate CIPSEA companion legislation on granting greater access for the Census Bureau, BLS, and BEA to IRS data.

Reexamination Has Led to Modification or Elimination of Surveys

Several agencies have reexamined some of their surveys, which has led to their elimination or modification. The Census Bureau, for example, reviewed its portfolio of Current Industrial Reports (CIR) program surveys of manufacturing establishments, which resulted in the elimination and modification of some surveys. Census Bureau officials said they decided to undertake this reexamination in response to requests for additional data that could not be addressed within existing budgets without eliminating current surveys. They were also concerned that the character of manufacturing, including many of the industries surveyed by the CIR program, had changed since the last reexamination of the CIR programs, which had been over 10 years earlier. Using criteria developed with key data users, Census Bureau officials developed criteria and used them to rank 54 CIR program surveys. The criteria included 11 elements, such as whether the survey results were important to federal agencies or other users, and the extent to which the subject matter represented a growing economic activity in the United States. The recommendations the Census Bureau developed from this review were then published in the Federal Register and after considering public comments, the Census Bureau eliminated 11 surveys, including ones on knit fabric production and industrial gases. The Census Bureau also redesigned 7 surveys, scaling back the information required to some extent and updating specific product lists. As a result of this reexamination, the Census Bureau was able to add a new survey on “analytical and biomedical instrumentation,” and it is considering whether another new CIR program survey is needed to keep pace with manufacturing industry developments. Census Bureau officials told us that they plan on periodically reexamining the CIR surveys in the future.

HHS has also reexamined surveys to identify improvements, in part by integrating a Department of Agriculture (USDA) survey which covered similar content into HHS’s NHANES. For about three decades, HHS and USDA conducted surveys that each contained questions on food intake and health status (NHANES and the Continuing Survey of Food Intakes by Individuals, respectively). HHS officials stated that HHS and USDA officials considered how the two surveys could be merged for several years before taking action. According to HHS officials, several factors led to the merger of the two surveys, including USDA funding constraints, the direct involvement of senior-level leadership on both sides to work through the issues, and HHS officials’ realization that the merger would enable them to add an extra day of information gathering to the NHANES. Integrating the two surveys into the NHANES made it more comprehensive by adding a follow-up health assessment. According to HHS officials, adding this component to the original in-person assessment allows agency officials to better link dietary and nutrition information with health status.

Another mechanism HHS has established is a Data Council, which, in addition to other activities, assesses proposed information collections. The Data Council oversees the entire department’s data collections to ensure that the department relies, where possible, on existing core statistical systems for new data collections rather than on the creation of new systems. The Data Council implements this strategy through communicating and sharing plans, conducting annual reviews of proposed data collections, and reviewing major survey modifications and any new survey proposals. According to HHS officials, in several instances, proposals for new surveys and statistical systems have been redirected and coordinated with current systems. For example, HHS officials stated that when the Centers for Disease Control and Prevention (CDC) proposed a new survey on youth tobacco use, the Data Council directed it to the Substance Abuse and Mental Health Services Administration’s National Survey of Drug Use and Health. The Data Council stated that by adding questions on brand names, CDC was able to avoid creating a new survey to measure youths’ tobacco use.

OMB recognizes that the federal government should build upon agencies’ practice of reexamining individual surveys to conduct a comprehensive reexamination of the portfolio of major federal household surveys, in light of the advent of the ACS. OMB officials acknowledged that this effort would be difficult and complex and would take time. According to OMB, integrating or redesigning the portfolio of major household surveys could be enhanced if, in the future, there is some flexibility to modify the ACS design and methods. For example, an OMB official stated that using supplements or flexible modules periodically within the ACS might enable agencies to integrate or modify portions of other major household surveys. OMB officials indicated that such an effort would likely not happen until after the 2010 decennial census, a critical stage for ACS when ACS data can be compared to 2010 Census data. OMB officials said and their long- range plans have already indicated their expectation that there will be improved integration of the portfolio of related major household surveys with the advent of the ACS. For example, the Statistical Programs of the United States Government: Fiscal Year 2006 describes plans for redesigning the samples for demographic surveys, scheduled for initial implementation after 2010, when the ACS may become the primary data source.

Conclusions

In light of continuing budgetary constraints, as well as major changes planned and underway within the U.S. statistical system, the portfolio of major federal household surveys could benefit from a holistic reexamination. Many of the surveys have been in place for several decades, and their content and design may not have kept pace with changing information needs. The duplication in content in some surveys, while considered necessary, may be a reflection of incremental attempts over time to address information gaps as needs changed. OMB and the statistical agencies have attempted to address some of the more troublesome aspects of this duplication by providing explanations of the differences in health insurance estimates and with efforts to develop more consistent definitions of disability. These efforts, however, while helpful, address symptoms of the duplication without tackling the larger issues of need and purpose. In many cases, the government is still trying to do business in ways that are based on conditions, priorities, and approaches that existed decades ago and are not well suited to addressing today’s challenges. Thus, while the duplicative content of the surveys can be explained, there may be opportunities to modify long-standing household surveys, both to take advantage of changes in the statistical system, as well as to meet new information needs in the face of ever-growing constraints on budgetary resources.

Some agencies have begun to take steps to reevaluate their surveys in response to budget constraints and changing information needs. Agencies have reexamined their surveys and used administrative data in conjunction with survey data to enhance their data collection efforts. These actions, however, focused on individual agency and user perspectives. By building upon these approaches and taking a more comprehensive focus, a governmentwide reexamination could help reduce costs in an environment of constrained resources and help prioritize information needs in light of current and emerging demands.

Given the upcoming changes in the statistical system, OMB should lead the development of a new vision of how the major federal household surveys can best fit together. OMB officials told us they are beginning to think about a broader effort to better integrate the portfolio of major household surveys once the ACS has been successfully implemented. Providing greater coherence among the surveys, particularly in definitions and time frames, could help reduce costs to the federal government and associated burden hours. The Interagency Council on Statistical Policy (ICSP) could be used to bring together relevant federal agencies, including those that are not currently part of the ICSP. The ICSP has the leadership authority, and in light of the comprehensive scope of a reexamination initiative, could draw on leaders from the agencies that collect or are major users of federal household survey data. While OMB officials have stated that the ACS may not have demonstrated its success until after 2010, the complexity and time needed to reexamine the portfolio of major federal household surveys means that it is important to start planning for that reexamination.

Recommendation for Executive Action

To deal with the longer term considerations crucial in making federally funded surveys more effective and efficient, GAO recommends that the Director of OMB work with the Interagency Council on Statistical Policy to plan for a comprehensive reexamination to identify opportunities for redesigning or reprioritizing the portfolio of major federal household surveys.

Agency Comments

We requested comments on a draft of this report from the Director of OMB and the Secretaries of Commerce, HHS, HUD, and Labor or their designees. We obtained oral and technical comments on a draft of this report from the Chief Statistician of the United States and her staff at OMB, as well as written comments from the Acting Deputy Under Secretary for Economic Affairs at Commerce; the Assistant Secretary for Legislation at HHS; and the Assistant Secretary for Policy Development and Research at HUD; and technical comments from the Acting Commissioner of BLS at Labor, which we incorporated in the report as appropriate. In commenting on a draft of the report, OMB officials stated that the draft report presented an interesting study that addresses an issue worth looking at. OMB officials generally agreed with our recommendation, although they expressed concerns about the range of participants that might be involved in such a reexamination. We revised the recommendation to provide clarification that OMB should work with the Interagency Council on Statistical Policy rather than with all relevant stakeholders and decision makers. OMB officials also expressed concerns about moving from examining selected surveys in three subject areas to the conclusion that the entire portfolio of household surveys should be reexamined. In response we clarified that we were recommending a comprehensive reexamination of the seven surveys that comprise the portfolio of major federal household surveys, most of which were included in our review. OMB officials also provided clarification on how we characterized their statements on reexamining the portfolio of major household surveys, which we incorporated into the report.

Each of the four departments provided technical clarifications that we incorporated into the report, as appropriate. In addition, HHS and HUD officials offered written comments on our findings and recommendation, which are reprinted in appendix II. HHS stated that a reexamination was not warranted without evidence of unnecessary duplication and also highlighted a number of examples of agency efforts to try to clarify varying estimates. However we did not rely on evidence of duplication, but rather based our recommendation on other factors, including a need to provide greater coherence among the surveys and to take advantage of changes in the statistical system to reprioritize information needs and possibly help reduce costs to the federal government and associated burden hours. Further, in light of the major upcoming changes involving the ACS and SIPP, and in conjunction with constrained resources and changing information needs, we believe that the major household surveys should be considered from a broader perspective, not simply in terms of unnecessary duplication.

HHS also provided a number of general comments. We incorporated additional information to reflect HHS’s comments on the different uses of disability information, a standard set of disability questions, NHIS’s coverage of access to care, and the fact that MEP’s sample is a subset of the NHIS sample. HHS’s comments on differences in estimates and the lack of a single definition of disability were already addressed in the report. HHS also stated that NCHS works through various mechanisms to ensure that surveys are efficient. We support efforts to enhance efficiency and believe that our recommendation builds upon such efforts.

HUD officials were very supportive of our recommendation, stating that such a reexamination is especially important as the ACS approaches full- scale data availability. In response to HUD’s comments suggesting adding more information on SIPP and AHS, we expanded the report’s discussion of the longitudinal dimension of SIPP and AHS.

As agreed with your office, unless you publicly announce the contents of the report earlier, we plan no further distribution of it until 30 days from the date of the report. We will then send copies of this report to the appropriate congressional committees and to the Director of OMB, and the Secretaries of Commerce, HHS, HUD, and Labor, as well as to other appropriate officials in these agencies. We will also make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http:/www.gao.gov.

If you or your staff have any questions regarding this report, please contact me at (202) 512-6543 or steinhardtb@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix II.

Appendix I: Scope and Methodology

To answer our first objective of identifying the number and characteristics of Office of Management and Budget (OMB)-approved federally funded statistical and research surveys, we obtained the database of information collections that had been approved by OMB as of August 7, 2006. The information in the database is obtained from Form 83-I which is part of an agency’s submission for OMB approval of an information collection. As the approval is in effect for up to 3 years, this database reflects all those collections with OMB approval for their use as of that date, and is thus a snapshot in time.

Although OMB Form 83-I requires agencies to identify various types of information about an information collection, including whether the information collection will involve statistical methods, the form does not require agencies to identify which information collections involve surveys consequently the database of OMB-approved information collections does not identify which information collections are surveys. Furthermore, the definition of information collections contained in the Paperwork Reduction Act (PRA) of 1980 is written in general terms and contains very few limits in scope or coverage. On the form, agencies can select from seven categories when designating the purpose of an information collection, which are (1) application for benefits, (2) program evaluation, (3) general purpose statistics, (4) audit, (5) program planning or management, (6) research, and (7) regulatory or compliance. When completing the form, agencies are asked to mark all categories that apply, denoting the primary purpose with a “P” and all others that apply with an “X.” Since OMB does not further define these categories, the agency submitting the request determines which categories best describe the purpose(s) of the proposed collection. The choices made may reflect differing understandings of these purposes from agency to agency or among individuals in the same agency.

The list of surveys contained in this report was derived from the database of OMB-approved information collections and therefore contains all information collections that an agency designated as either “general purpose statistics” or “research” in the primary purpose category that we used as a proxy for the universe of surveys. The directions to agencies completing the forms call for agencies to mark “general purpose statistics” when the data are collected chiefly for use by the public or for general government use without primary reference to the policy or program operations of the agency collecting the data. Agencies are directed to mark “research” when the purpose is to further the course of research, rather than for a specific program purpose. We did not determine how accurately or reliably agencies designated the purpose(s) of their information collections. It is also possible that the database may contain other federally funded surveys that the agency did not identify under the primary purpose we used to “identify” surveys, and these would not be included in our list of surveys.

We have taken several steps to ensure that the database of OMB-approved information collections correctly recorded agency-submitted data and contained records of all Forms 83-I submitted to OMB. Our report, entitled Paperwork Reduction Act: New Approach May Be Needed to Reduce Burden on Public, GAO-05-424 (Washington, D.C.: May 20, 2005), examined the reliability of the database of OMB-approved information collections and concluded that the data were accurate and complete for the purposes of that report. Because this assessment was recent, we decided that we would not repeat this assessment. We did, however, compare a sample of the surveys from the Inventory of Approved Information Collection on OMB’s Web site to our copy of the database of OMB-approved collections. We found that all of the surveys in the Inventory of Approved Information Collection were contained in the database.

Not all information collections require OMB approval under the PRA. OMB’s draft Implementing Guidance for OMB Review of Agency Information Collection explains that in general, collections of information conducted by recipients of federal grants do not require OMB approval unless the collection meets one or both of the following two conditions: (1) the grant recipient is collecting information at the specific request of the sponsoring agency or (2) the terms and conditions of the grant require that the sponsoring agency specifically approve the information collection or collection procedures. As also stated in the OMB draft, information collections that are federally funded by contracts do not require OMB approval unless the information collection meets one or both of the following two conditions: (1) if the agency reviews and comments upon the text of the privately developed survey to the extent that it exercises control over and tacitly approves it or (2) if there is the appearance of sponsorship, for example, public endorsement by an agency, the use of an agency seal in the survey, or statements in the instructions of the survey indicating that the survey is being conducted to meet the needs of a federal agency. Although there are additional surveys funded through grants and contracts that are not approved by OMB under the PRA, OMB stated that there is no comprehensive list. In addition, the draft guidance states that the PRA does not apply to current employees of the federal government, military personnel, military reservists, and members of the National Guard with respect to all inquiries within the scope of their employment and for purposes of obtaining information about their duty status.

For the second objective describing current agency and OMB roles in identifying and preventing unnecessary duplication, we took several different steps. We reviewed the PRA requirements for agencies and OMB. We also interviewed agency clearance officers at the Departments of Commerce, Health and Human Services, and Labor about their processes for submitting information collection packages to OMB. These agencies are the top three agencies in terms of funding for statistical activities in fiscal year 2006. We also interviewed OMB officials about their role in approving proposed information collections.

For the third objective, through reviewing our reports and literature and by interviewing agency officials, we identified surveys with duplicative content. We identified duplication by looking for areas of potential duplication when several surveys contained questions on the same subject. This duplication was strictly based on similar content in the surveys on the same subject, specifically people without health insurance and those with disabilities. We also looked at the duplication in the subject area of housing between the American Community Survey and American Housing Survey, which had been identified by our previous work. We also looked at environmental surveys, but determined that there was not duplicative content with our major surveys. Once we had identified the three subject areas, we used literature and interviews to identify the current federally funded surveys that were cited as the major surveys in each theme. We did not focus on any particular type of survey, but rather chose the surveys that were cited as the major surveys in each theme. To learn more about the duplicative content between surveys related to these three themes, we reviewed relevant literature and agency documents. We also interviewed officials from OMB, and the Departments of Commerce, Labor, Health and Human Services, and Housing and Urban Development. In addition, we interviewed experts from organizations that focus on federal statistics, such as at the Council of Professional Associations on Statistics and the Committee on National Statistics, National Academies of Science.

Although we have included the Census Bureau’s Survey of Income and Program Participants as part of our assessment of potential duplication, the fiscal year 2007 President’s budget proposed to cut Census Bureau funding by $9.2 million, to which the Census Bureau responded by stating that it would reengineer the SIPP. Therefore, the fate of the SIPP is uncertain, and reengineering has not been completed.

For the fourth objective, we also interviewed OMB officials, agency officials, and organizations that focus on federal statistics. Through the combination of agency and OMB interviews, expert interviews, and research, we identified selected agency efforts to improve the efficiency and relevance of surveys.

Appendix II: Comments from the Department of Housing and Urban Development

Appendix III: Comments from the Department of Health & Human Services

Appendix IV: GAO Contact and Staff Acknowledgments

Staff Acknowledgments

In addition to the contact named above, key contributors to this report were Susan Ragland, Assistant Director; Maya Chakko; Kisha Clark; Ellen Grady; Elizabeth M. Hosler; Andrea Levine; Jean McSween; Elizabeth Powell; and Greg Wilmoth.