Small Area Estimation: New Developments and Directions for 

Health and Human Services Data 

John L. Czajka,* Amang Sukasih,* and Susan Queen** 

*Mathematica Policy Research 

1100 First Street, NE, 12th Floor 

Washington, DC 20002-4221

jczajka@mathematica-mpr.com

asukasih@mathematica-mpr.com

**Office of the Assistant Secretary for Planning and Evaluation 

Department of Health and Human Services 

200 Independence Ave., SW 

Washington, DC 20201 

susan.queen@hhs.gov

Proceedings of the 2013 Federal Committee on Statistical Methodology (FCSM) Research Conference 

Introduction 

Household surveys provide timely estimates of a wide range of population characteristics and play a critical role in 

policy analysis and development in the federal government, but the sample designs of most household surveys do 

not support estimates below the national level. Program administrative data avoid the limits of sample size when 

they include the entire population of participants in a program, but their contents are often narrowly focused, and the 

universes that they cover may be narrow as well. 

 

Small area estimation (SAE) methods were developed to address the need for estimates of characteristics for 

geographic areas or other domains (population subgroups, for example) that cannot be obtained directly from survey 

or administrative data. Many of the techniques used in small area estimation combine data from multiple sources 

and often include both a survey component and an administrative component. With advances in data processing 

capacity and computational algorithms, it has become feasible to apply estimation methods that were impractical not 

that long ago. Applications of SAE by federal agencies have multiplied in the past two decades. 

Greater sharing of information about methods, data sources, and how to present small area estimates would be useful 

to all who are involved in preparing such estimates—but especially to those who are new to this endeavor, have 

more limited resources than the major statistical agencies, or continue to use less effective methods.

Seeing the opportunity for a significant step forward, the Office of the Assistant Secretary for Planning and 

Evaluation (ASPE) in the Department of Health and Human Services (HHS) undertook the planning of an expert 

panel meeting to bring together practitioners of SAE to share their insights and jointly explore possibilities for 

improving their methods and outcomes. ASPE contracted with Mathematica Policy Research to identify prospective 

panelists and other attendees and develop an agenda; prepare background materials for the meeting participants; 

facilitate the meeting and prepare a summary of the panel discussion; and synthesize the background materials and 

the panel discussion into a final report. The broadly written objective of the project was to assist agencies within 

HHS and the broader federal statistical community in making more effective use of SAE to address data gaps for 

small geographic areas and populations. 

The panel meeting was held on June 4, 2013, at the Hubert H. Humphrey Building and had 37 attendees, including 

the ASPE and Mathematica project staff. Eight operating divisions of HHS were represented along with seven other 

federal agencies and the Committee on National Statistics. Panelists included faculty from three universities and 
members of four research organizations. This paper summarizes the principal findings and conclusions from the 

panel meeting and the broader project. 

Overview of Findings and Observations 

Based on our review of federal government programs producing small area estimates coupled with the discussion at 

the expert panel meeting, we report the following observations: 

Within HHS use of SAE methods to produce small area estimates on a regular basis is limited, but 

these applications show a high level of sophistication and cutting edge methods 

The development of a program of small area estimates requires significant time and resources, but 

these costs are small in comparison to new or expanded data collection 

Software limitations are an especially challenging part of the development and implementation of 

small area methods 

The identification and use of suitable auxiliary variables present a number of challenges 

Comparative evaluations of alternative approaches are not as common as prospective users might 

expect 

Understanding how estimates will be used and how they will perform under those circumstances are 

important to both development and evaluation 

Validation of small area estimates remains a significant challenge, and varied approaches have been 

used 

Interpretation and communication of small area estimates require careful attention 

The American Community Survey (ACS) changes the landscape for small area estimates in a number 

of ways 

Collaboration presents both opportunities and challenges, yet the sharing of technical resources across 

agency divisions and even across agencies may be necessary to more fully exploit the potential of 

small area methods 

Below we develop each of these points. 

Applications within HHS 

We identified several production applications of SAE within HHS. These include: 

State estimates of age-group-specific prevalence estimates of 25 substance use behaviors measured in 

the National Survey of Drug Use and Health; Substance Abuse and Mental Health Services 

Administration (SAMHSA) 

County estimates of diabetes prevalence, incidence and risk factors; Centers for Disease Control and 

Prevention (CDC) 

State and county estimates of cancer risk factors and screening; National Cancer Institute (NCI) 

State estimates of household substitution of wireless for landline telephones; National Center for 

Health Statistics (NCHS) 

Model-based estimates of vital statistics for geographic and demographic small domains; NCHS 

Most of these applications are relatively new. The CDC and NCI estimates and the NCHS estimates of wireless 

substitution were produced within the last decade. By contrast, the SAMHSA estimates began in the early 1990s. 

Reflecting the skills of those involved, the newer approaches and the SAMHSA estimates all employ empirical or 

hierarchical Bayes methods. 

In addition to these production applications, small area estimates have been developed as part of methodological 

research at NCHS, and both CDC and NCI are working to extend their methods to other measures while the Agency 
for Healthcare Research and Quality (AHRQ) is developing experimental county-level estimates for its quality 

indicators program and exploring the production of additional state estimates from the Medical Expenditure Panel 

Survey (MEPS). 

Requirements for Establishing an SAE Program 

The development of a program of small area estimates requires significant expertise, time and resources. Requisite 

skills include statistics and/or econometrics with an emphasis on modeling; high-level programming; and strong 

communication ability. Meeting these needs may require hiring new staff or obtaining contractor support. 

Development and implementation can take two or more years. Nevertheless, these costs are dwarfed by the expense 

of new or expanded data collection. 

Software 

Most of the formal SAE methods used by federal agencies and their contractors are computationally intensive. As 

such, they require specialized software or specialized routines within larger software packages. Computational speed 

is a particular issue for users, as is the extent of diagnostic information that the software can provide. Some of the 

most sophisticated users of SAE methods have written their own software to address these limitations. 

Auxiliary Variables 

Auxiliary variables play a critical role in the production of small area estimates. Administrative records are a 

valuable source of auxiliary variables because they can provide very precise estimates for small domains. In 

addition, the quality of measurement in administrative records can exceed that of comparable survey estimates, but 

quality may vary widely across items, across areas, and over time. Moreover, anomalies can occur as the result of 

administrative actions, natural disasters, or other factors. This underscores the importance of consulting with agency 

staff when using an agency’s administrative data. 

Comparative Evaluations 

Our literature search and our review of federal programs identified very few examples of comparative evaluations of 

alternative small area methods. The primary reason, we infer, is that the effort required to implement a method fully 

enough to conduct an empirical evaluation is substantial. Because of this, agency staff may research the alternatives 

through the literature or by talking to other users, but the choice of a method, typically, is made before significant 

development occurs. In refining the application of the chosen method, agency researchers may conduct comparative 

evaluations of selected options within the overall methodology, but even this type of methodological research sees 

limited use. Consequently, the federal statistical community has learned little about how different SAE approaches 

compare when applied to the same estimation problem. Thus, agencies that are developing new applications and 

seeking to choose the most suitable method cannot draw on external research to determine what methods are likely 

to work best for their particular applications. What they can learn is what methods have been applied successfully to 

similar problems and, if they choose one of these methods, what issues they can expect to encounter and how they 

can address these issues. 

Anticipating Uses 

How a set of small area estimates is likely to be used may have implications for how they are created and how they 

are evaluated. A number of the estimates produced by federal agencies are used for funding allocation, and in those 

circumstances the quality of the estimates has direct implications for how well federal funds are targeted and how 

efficiently they are distributed. Stability, on the one hand, and sensitivity to change in underlying needs are 

important properties for an estimator that will produce estimates for funding allocation—although they may appear 

to be at odds. The desired properties of an estimator should affect design choices at the outset and determine what 

criteria are given the most weight in evaluating the estimates. If the estimates prove to be inadequate in some 

respects, the response of administrators or legislators may be to establish hold-harmless provisions or other 

administrative remedies that undermine the use of the estimates. Statistical solutions would be preferable but may be 

difficult to implement once the estimates have begun to be used. 
Validation 

Evaluating small area estimates is perhaps the most challenging aspect of their production. That small area methods 

had to be used in the first place implies that direct estimates of sufficient precision are not available to use in an 

evaluation. When the census long form was still in place, small area estimates of census-like variables could be 

evaluated by applying the methodology to a census year and comparing the results to the census estimates. But this 

is no longer an option except for the basic demographic characteristics measured on the decennial census short form. 

Users have developed creative solutions to evaluate their estimates. Simulations using an artificial population are 

popular. Subsamples are drawn from the simulated population, estimates are created from the subsamples, and the 

estimates are compared to “truth” as defined by the simulated population. Comparisons using the most precise direct 

estimates (for example, the largest states) are sometimes used, but the largest areas may model differently than 

smaller areas, and the findings may be more limited than the users would prefer. Another approach used by a federal 

agency is to assess how well the small area estimates preserve known correlations. Cross-validation has also been 

applied to evaluate small area estimates; a subset of areas is removed, the model re-estimated, and the results applied 

to the areas that were removed. Finally, given the critical role of modeling in the development of small area 

estimates, careful examination of model diagnostics—that is, internal validation—may be the most common form of 

validation. 

Interpreting and Communicating Results 

No less than with direct estimates, producers of small area estimates need to consider how best to communicate the 

uncertainty associated with their estimates. They may also have to explain why the estimates are better than direct 

estimates. Meeting with stakeholders can be exceedingly useful in explaining the notion of borrowing strength and 

how it is reflected in the small area estimates and in learning and addressing the concerns and questions that the 

stakeholders may have regarding the estimates. 

Some producers of small area estimates have developed informative graphics to assist users in making comparisons 

across areas. State estimates, for example, lend themselves to visual displays showing which state estimates are 

significantly different from others. In addition, maps can be useful not only in validation, but also as an aid to 

interpretation and communication of findings. 

Users who are interested only in point estimates present a particularly difficult challenge because they profess not to 

care about the lower variance that differentiates small area estimates from direct estimates. Persuading them to 

accept the small bias in model-based estimates in return for lower variance is a hard sell. Sometimes the best way to 

explain uncertainty is to allow those who use the statistics to observe it first hand—by showing them estimates for 

multiple areas for consecutive years. 

Impact of the ACS 

The replacement of the census long form by the ACS affects small area estimation in a number of ways. First, it 

eliminates long-form variables previously used in models and validation. Second, it provides direct estimates for 

many characteristics, geographic areas, and subpopulations that would have required SAE previously. Third, it 

provides a wide range of auxiliary variables that are contemporaneous with the direct estimates used in model 

development. Fourth, it eliminates the need to deal with the variable time lag that characterizes the use of auxiliary 

variables from the last decennial census. Fifth, it expands opportunities for SAE through pairing the ACS with other 

surveys that capture the relevant content but lack the ACS’s sample size. At the same time the ACS also shows the 

limits of even very large-scale surveys. For most substate areas, only three- or five-year averages are published, and 

they can be very imprecise. For such areas SAE can provide more timely and more precise estimates. 

Cross-agency Collaboration 

Two important benefits of collaboration are providing access to broader staff expertise and providing access to data. 

Nevertheless, the challenges to collaboration can be significant—particularly across departments. Common interest 
in a successful collaboration is critical to establishing inter-agency agreements that will foster a successful 

partnership. Models of successful collaboration—such as that between NCI, NCHS, and two universities—can be 

studied for ideas on how to make such ventures effective. 

In addition to formal collaboration, interagency working groups can be helpful for sharing ideas and keeping 

members up to date on activities that are ongoing or under consideration. Panelists responded enthusiastically to an 

OMB offer to create a group under the FCSM that would allow those working on small area projects or interested in 

the topic to communicate on a regular basis. 

Considerations 

One of the goals of this project was to help ASPE in communicating to others in the department what small area 

methods can and cannot do to address data gaps involving states, substate areas, and other small domains. The 

successful SAE programs at CDC, NCHS, NCI, and SAMHSA and the ongoing work at the Agency for Healthcare

Research and Quality (AHRQ) to develop county estimates for its quality indicators program provide examples that 

illustrate the potential of these methods to fill such data gaps, the amount of effort that this may entail, and the types 

of data that must be assembled. It would be useful to draw on these examples to convey a realistic appreciation of 

what SAE can do and what its application entails. 

Other agencies in HHS that routinely use small area statistics in their work recognize the value of such data but may 

not have the requisite staff or resources to develop their own small area estimates. We suspect that each of these 

agencies may be able to identify needs for expanded small area data in the future. If that should occur, agencies 

should explore potential collaboration with NCHS, which has the deepest staff with significant experience with SAE 

techniques. 

Administrative data collected by CMS could provide a potential source of new auxiliary variables for applications of 

SAE within HHS. It will be important to monitor applications within the operating divisions to identify needs that 

could be addressed by CMS data and, where such needs exist, encourage collaboration. There might be interest in 

state estimates from the Medicare Current Beneficiary Survey (MCBS) and, if such interest is identified, CMS might 

work with NCHS staff that could be helpful in setting up a collaborative program to produce such estimates. 

The Division of Program Statistics in the Office of Public Health Support in the Indian Health Service (HIS) 

produces a major report that relies heavily on decennial census data: Regional Differences in Indian Health. With 

the elimination of the census long form, many of the estimates in this report will have to be obtained from the ACS. 

This opens up the possibility of producing estimates more often than once a decade to better document trends, but 

sample size becomes an issue. It may be valuable for IHS to assess the merits of using SAE methods to produce 

more frequent statistics on the AIAN population. 

While HRSA does not require SAE methods to identify shortage areas as they have been defined traditionally, using 

actual counts of medical professionals along with Census Bureau population estimates, the application of small area 

methods could enable HRSA to broaden the way it defines shortage areas. HRSA may want to explore whether there 

might be value in this approach. 

Finally, some mechanism for sharing experiences with SAE would serve a broad array of current users and 

prospective users within the federal government. We encourage OMB to pursue the establishment of a working 

group on small area methods and applications, and we anticipate that ASPE would participate in this effort and 

inform the operating divisions of HHS of developments that are communicated through this group.