THERE AND BACK AGAIN: 

DEMOGRAPHIC SURVEY SAMPL

IN THE 21ST CENTURY

Colm O’Muircheartaigh

NORC, University of Chicago

Page 1
OVERVIEW

• History of Demographic Survey Sam• 20th Century Sample Design

• New Directions• Evaluation of Lists, GIS, and Maps• Implications• New National Sample Designs• Swiss Cheese• Tailored Samples vs Master Sample• Conclusion 

Page 2
HISTORY 1

• A N Kiaer (1895) 

– ISI Berne Representative Enumerations

– Miniature of the population– Multi-stage design – places, towns, street

– Stratified

• US implementation

– Cressy L Wilbur (1896-7) – [vital statist

• small representative areas 

– Carroll D Wright (1875 et seq) – [labor 

• representative statistics

– Non-probability samples

Page 3
HISTORY 2–DEVELOPMENT

• Bowley (1906) 

– Theory for simple random sampling

• Neyman (1934) 

– Superiority of probability sampling– Theory for unequal cluster sampling

• Hansen Hurwitz Madow 1940s 

– PPS at higher stages

• Adequate “representation” of importan• Leads to identification of certainty PSU

– Equal workloads at final stage (HUs)

• Efficiency of field allocation and estima

• 1950s: national master samples ISR, 

Page 4
THE BASIC NATIONAL DEMOGRAPHIC 

• Multi-stage

– Costs– Feasibility

• Some self-representing PSUs• Stratified

– Incorporating knowledge of population a

Page 5
20th CENTURY DEMOGRAPHIC SURVEY

SAMPLE DESIGN ELSEWHERE

• Scandinavia

– Register-based

• China

– Register-based– Late 1980s, registers deteriorated

• UK

– Electoral registers, updated annually– 1980s, registers deteriorated– Postcode address file (PAF), centrally a– Periodic redesign

Page 6
20th CENTURY DEMOGRAPHIC SURVEY

SAMPLE DESIGN IN USA

• Decennial update of frame, and• Absence of a current list of populatio

– Selection of a MASTER SAMPLE of PSUs 

– Listing of the frame for the master samp– Use as reservoir for the decade– Updating in the field for the sample only

Page 7
NEW DIRECTIONS IN THE USA

• Availability of current administrativ• Matching and pre-classification of ge• GIS and GPS• Tailored samples vs master samples

Page 8
WHY LISTS WOULD MAKE A DIFFERE

• Cost parameters would change• Nature of PSU might change• Subsampling fraction might change• Timing of revisions could change

Page 9
THE (NON-CENSUS) ADMINISTRATIVE 

• USPS delivery sequence file 

– Ordered within ZIP by carrier route– Within carrier route by walk sequence

• Available through licensees

– Primarily purchased by direct-mail organizatio

• Usability

– Basis for MAF in urban areas– Addresses in standard format– Operational incentives for updating– Can be geocoded and mapped– Contains PO boxes and rural route boxes (not m

Page 10
USING/EVALUATING THE LIST

• 1 Direct/non-evaluative use, single city surv• 2 Evaluation against traditional listing, 200• 3 Inner-city evaluation and use, 2002-3 NOR• 4 Direct/non-evaluative use as national fram• 5 “Rural” evaluation, 2003 NORC• 6 Basis for national design template, 2003-4 • 7 National comparison with traditional listi

Page 11
DIRECT USE (RTI-2001)

• Iannachione, Staab, Redden 

– Houston, TX– Geocoded > 99% of addresses – Selected sample from list– 97% of selected addresses yielded HUs– Order of magnitude of list and census co– No direct coverage check

Page 12
VALIDATING THE LIST (NORC 2001-

• O’Muircheartaigh, Eckman, and We• NORC GSS Field Test 2001: 14 segm

– First, traditional listing (T)– Then, geocoded USPS list for the areas – Finally, independent enhanced list (E) b

• Comparison of coverage

– T – Traditional– U – USPS addresses geocoded inside seg– E – U enhanced in the field– USPS – full USPS list whether geocoded 

Page 13
Page 14
Page 15
ISSUES ENCOUNTERED IN ENHANCED L

• Issues with USPS list

– missing apartment numbers– addresses removed at request of residen– PO boxes, rural route boxes unusable– includes hard to find HUs missed by fie

• Geocoding issues

– block boundaries– side-of-street errors

• Matching geographies

– ZIPs vs blocks, block groups, tracts

Page 16
Page 17
COST COMPARISON T VS E

• Travel costs, etc.

– Equal

• Listing costs

– T approximately twice as expensive as E

Page 18
COMPARISON OF T, U, AND E

• U in E • E in U • T in U • E in T • E in USPS 

Page 19
INNER CITY EVALUATIONS (NORC 20

• O’Muircheartaigh, Eckman, English• The Making Connections Project

– Funded by Annie E. Casey Foundation

• 10 Deprived Inner-City Communitie

– Denver, Des Moines, Indianapolis, San – Milwaukee, Hartford, Providence, Oak

Page 20
INNER CITY EVALUATIONS

• Purchased USPS lists for ZIPs surro

community

– Geocoded all– With U as base: 

• Produced E with in-person listing• Compared U and E for coverage

– Compared U and E coverage during fie

Page 21
INNER CITY EVALUATIONS

• Two key measures: 

– How much of E is in U (the geocoded part of US– How much of E is in USPS as a whole

Page 22
INNER CITY EVALUATIONS

• Overall results

– 90% of E in U– 94% of E in USPS

• Difference due to geocoding/map inaccu

• Range across cities:

– 82% - 95% of E in U– 83% - 99% of E in USPS

• Characteristics of missed HUs

– In most severe cases, many vacant HUs

• MHU

– Only moderately successful

Page 23
DIRECT USE NATIONAL FRAME (RTI 

• Staab, Iannachione• Used postal frame exclusively for Eu• Used postal geographies• Ignored ZIPs with no residential add• Ignored residents without street add

Page 24
NATIONAL LIST EVALUATION (NORC/

• O’Muircheartaigh, Lepkowski, Heer

– HRS and NSHAP– National listing of 549 segments by ISR– Purchase of USPS lists for 100 segment– Comparison of T and U– Nationally representative

Page 25
USE FOR NORC NATIONAL SAMPLE D

• Geographic units• Preclassification of list quality• Stratification• Optimal design

Page 26
THE POPULATION

• 8.2 million census blocks• 66,275 tracts• 3219 counties• 281 (C)MSAs in Census 2000

– Now 362 MSAs and 565 Micropolitan S

• Variable population density• Variable list quality

Page 27
PRECLASSIFICATION OF GEOGRAPHIES

• Census classification of blocks [TEA 

enumeration area]

– Available for all blocks– Indicator of feasibility of using USPS lis

• Whether suitable for mail-out • Address type

• Type A tracts

– 95% of HUs in tract are in blocks classif

• Type B tracts

– All other tracts

Page 28
THE DESIGN – 1

Categorize MSAs/counties according to 

density and list qualityLarge MSAs (likely certainty areas) density population dominated by Ty[category 1]Small counties with less than 30% of type A tracts or less than 15,000 pop[category 3]All other counties/MSAs [category 2

Page 29
THE DESIGN – 2

• Category 1

– 45% of population in 4.5% of the area

• Category 2

– 40% of population in 25% of the area

• Category 3

– 15% of population in 70% of the area

Page 30
Map of Categories 1, 2, and 3

Page 31
THE DESIGN – 3

• Different designs are appropriate for 

categories

• A major problem:

– Even in the high density urban MSAs r

style address) areas are interspersed wistyle address) areas

Page 32
24 Category 1 Areas Showing Type A and 

Page 33
Chicago Category 1 MSA Showing Type A 

Page 34
Los Angeles Category 1 MSA Showing Ty

Page 35
Category 2 Areas Showing Type A an

Page 36
Type A and B Tracts In Worcester, MA [a categ

Page 37
Type A and B Tracts In Champaign/Urbana, IL [a 

Page 38
Type A and B Tracts In Billings, MT [a categor

Page 39
THE DESIGN SOLUTION

• The Swiss cheese frame

– Stratum 1 contains all type A tracts in c

• In this stratum, the tract is the PSU

– Stratum 2 contains all type A tracts in c

• In this stratum the MSA/county is the PS

– All remaining tracts (category 1B, categ

category 3) 

• In this stratum, the MSA/county is the P• Supplementary tracts from category 1B

Page 40
Type 1B Segment in Riverside CA, showin

Census Count, and USPS Address L

Page 41
Stratum 1 – All Type A Tracts in Catego

Page 42
STRATUM 1

• 42% of population, 2% of area, 24 ce• Direct selection of tracts as PSUs• Contemporaneous USPS list with MH

for HU selection

Page 43
Stratum 2 – All Type A Tracts in Categ

Page 44
STRATUM 2

• 30% of population, 3% of area, 607 

(or parts thereof)

• 60 MSAs/counties (or parts thereof) 

selections

• Selection of tracts as SSUs• Contemporaneous USPS list with MH

for HU selection

Page 45
Stratum 3- 1B, 2B, and 3 Tracts

Page 46
STRATUM 3 [composite of categories 3, 2B

• 28% of population, 93% of area, 3074 

(or parts thereof)

• Selected of 28 MSAs/counties (or part

PSUs

• Constructed segments (blocks or grou

SSUs

• Listed master sample of HUs within se

– Collect geocode during listing (GPS devic– Reservoir for decade

Page 47
Map Showing Strata 1, 2, and 3

Page 48
IMPLICATIONS OF LISTS FOR SAMPLE D

• Tailored samples vs Master sample• Rural – no change from previous d

• Definition of rural?

• Non-rural 

– For timeliness, coverage, and cost, E – Is U superior to T? – Not desirable to construct very much 

• Non-rural can be extended as qual

Page 49
FEATURES OF NEW DESIGNS

• Flexibility for tailored designs

– Accommodates modified stratification w

and 2 using ACS and/or other informatidecade

– Permits updates to HU frame using USP– Allows different definition and number 

stratum depending on size of sample and requirements

• Timeliness

– Can take advantage of any list upgrades 

Page 50
THERE ...

• 19th Century

– Multi-stage cluster sample of HUs– Stratified by urbanicity– Use of lists where possible– Selection from street addresses or regis– Designs tailored to specific projects

• Mid-20th Century

– Area sampling as conceptual framewor– Decennial listing/master samples– Re-design decennially

Page 51
... AND BACK AGAIN

• 21st Century

– Lists as frames– GIS/location as unique identifier– Designs differentiated by cost/feasibilit

• The Mechanisms

– Available (high) quality lists– GIS – identification and tracking– Pre-classification of geographies– Computer power

Page 52
• The Result

– Tailored samples – Cheaper, better samples– Unnecessary uniformity minimized – Subject matter can inform sample desig– Database linkages for analysis

Page 53
CHALLENGES

• For designers:

– Matching list geographies and census g– Better map data bases– Unique identifiers for addresses– Confidentiality/anonymity concerns

• For users: 

– Taking advantage of the potential

• Overall, most exciting time for samp

Neyman in 1934 and the subsequent 

Page 54