Quality of Geospatial Data 

Integrated from Multiple Sources

John L. Eltinge John.L.Eltinge@census.gov

FCSM Workshop on Quality 

of Integrated Geospatial Data

October 26, 2018

. 

1
Acknowledgements and Disclaimer

The author thanks many colleagues in the federal 

statistical system, academia and the private sector for 

very valuable discussions of the topics considered in 

this paper. 

The views expressed in this paper are those of the 

author, and do not necessarily represent the policies of 

the United States Census Bureau, nor the Federal 

Committee on Statistical Methodology.

2
Overview: Dimensions of Data Quality

I. FCSM Workshops on Integration 

of Multiple Data Sources: Transparent 

Reporting & Practical Improvement

II. “Relevance” Dimension of Quality

III. “Accuracy” Dimension of Quality

3
4

Columns: Operational Goals

Rows: Quality 

Dimensions 

Transparent

Reporting

Practical

Improvement

Relevance

Accuracy

Many others, 

plus risk & 

cost
I. FCSM Workshops on Data Quality - 1

A. Prospective Integration of Multiple Data 

Sources: Wonderful Opportunity to 

1. Improve quality/risk/cost profiles of 

current statistical information products 

and services

2. Expand statistical portfolios

5
II.B. Data Quality – Previous Workshops

- Input data quality (December 1, 2017)

- Processing quality (January 25, 2018)

- Output data quality (February 26, 2018)

- Metadata (September 14, 2018)

6
I.C. Data Quality – Multiple 

Dimensions - 1

Quantitative features: “accuracy”

Extend traditional “Total Survey Error” models 

(Biemer et al, 2017)

- Population coverage

- Linkage errors

- Definitional errors and inconsistencies

- Incomplete data

- Estimation errors (Lohr and Raghunathan, 2017; 

Elliott and Valliant, 2017)

7
I.C. Data Quality – Multiple 

Dimensions - 2

Qualitative features:

relevance, timeliness, comparability, 

coherence, accessibility

This talk: “relevance” and “accuracy” 

dimensions for geospatial data

8
II. “Relevance” Dimension - 1

A. General issue: Do our

- Formal conceptual and statistical framework

- Measurement and estimation methods

- Interpretation of results (and limitations)

align well with primary inferential questions 

of key stakeholders (and value conveyed)? 

9
II.A. “Relevance” Dimension - 2

In other words: Spell out clearly

- What questions are we asking?

- Why (and when) are the questions 

(and answers) important for specified groups 

of data users?

- Consider both “use value” and “option value”

10





II.D. Relevance – Applications - 1

Five Applications, with Prospective 

Interpretation (cf. CEP, 2017)

1. Purely descriptive reports (means or totals)

and related ranks 

- Per “triple goal” estimation

(Shen and Louis, 1998)

16


II.D. Relevance – Applications - 4

For any of “description,” “association,” 

“prediction,” “causality” or “control”:

1. Level of aggregation (e.g., geography) for: 

- Practical distinctions among areas, groups

- Inform realistic decisions on prospective 

intervention?

19
II. Relevance – Applications – 5

2. Quality of information at the specified level of 

aggregation (section III)?

3. Stakeholder risks incurred through poor 

quality or break in series?

4. Value conveyed, accounting for (1)-(3)?

- Both “use value” and “option value”

20
III. “Accuracy” Dimension of Quality

A. Assessment of estimation (prediction) 

accuracy 

– Accounting for which components of 

variability?

21



III.C. “Accuracy” Measures - 2

2. Record linkage effects

3. Adjust for exploratory-analysis and modelselection effects

a. Contrast between formal inference and 

exploratory analyses (cf. Tukey, 1962, others)

b. Nuances among multiple inferential goals

(Shen and Louis, 1998, “triple goal” SAE)

25
III.C. “Accuracy” Measures - 3

3. Reporting summaries

a. For specific estimands and point estimators

b. Summaries across estimands

- Of special interest for “unified” decision 

on spatial estimation methods

26
III.D. “Accuracy” – Implications

1. Solid inferences: What do (can) we know fairly 

well from current data, accounting for errors?

2. Response to abovementioned limitations: 

 Find better data sources (more admin records; 

calibration/bridge surveys)? Cost-effective?

 Improve linkage, imputation, analysis methods?

27
III.E. Quality and Risk

1. Loss of, or major changes in, data sources

2. Production system changes (w/related costs)

3. Disclosure issues

Tools for identification and management of risks?

Implications for management and integration of 

regional data sources? 

28
III.F. Quality and Cost

For many resource dimensions

1. Incorporate both fixed and variable cost 

components

2. Additional costs incurred through 

integration of multiple data sources 

- cf. “complex supply chain management” 

29
III.G. Quality, Risk and Cost

Empirical Information on Dominant Factors for 

Quality, Risk and Cost? Vary Across Sources?

1. Observational data (e.g., paradata)

2. Formal experiments – factorial designs or 

evolutionary operation?

3. Modeling diagnostics (Rao & Molina, 2015; Elliott 

and Valliant, 2017; Lohr & Raghunathan, 2017)

30
IV. Closing Remarks

A. Quality of Geospatial Data Based on Integration of 

Multiple Data Sources

1. Multiple dimensions of quality (plus cost and risk)

2. Today’s talk: 

“relevance” and “accuracy” dimensions 

31


References (1)

Biemer, Paul P., Edith de Leeuw, Stephanie Eckman, Brad Edwards, Frauke

Kreuter, Lars E. Lyberg, N. Clyde Tucker, Brady T. West (Editors) (2017). Total 

Survey Error in Practice. New York: Wiley. 

Biemer, Paul P., Dennis Trewin, Heather Bergdahl and Lilli Japec (2014). A 

System for Managing the Quality of Official Statistics. Journal of Official 

Statistics 30, 381–415. https://doi.org/10.2478/jos-2014-0022

Cochran, W.G. Sampling Techniques, Third Edition. New York: Wiley.

Commission on Evidence-Based Policymaking (2017). The Promise of 

Evidence-Based Policymaking: Report of the Commission on Evidence-Based 

Policymaking. Available through:

https://www.cep.gov/content/dam/cep/report/cep-final-report.pdf

Elliott, Michael R. and Richard Valliant (2017). Inference for Nonprobability 

Samples. Statistical Science 32, 249-26434
References (2)

Imbens, Guido W. and Donald B. Rubin (2015). Causal Inference for Statistics, 

Social and Biomedical Sciences: An Introduction. New York: Cambridge University 

Press. 

Japec, Lilli, Frauke Kreuter, Marcus Berg, Paul Biemer, Paul Decker, Cliff Lampe, 

Julia Lane, Cathy O’Neil, Abe Usher (2015). AAPOR Task Force Report on Big Data. 

Available through: 

https://www.aapor.org/Education-Resources/Reports/Big-Data.aspx

Lohr, Sharon L and Trivellore E. Raghunathan (2017). Combining Survey Data with 

Other Data Sources. Statistical Science 32, 293-312

National Academies of Sciences, Engineering, and Medicine (2017). Federal 

Statistics, Multiple Data Sources, and Privacy Protection: Next Steps. Washington, 

DC: The National Academies Press. https://doi.org/10.17226/24893.

35
References (3)

National Research Council (2013) . Principles and Practices for a Federal Statistical Agency, 

Fifth Edition. Committee on National Statistics. Constance F. Citro and Miron L. Straf, 

Editors. Division of Behavioral and Social Sciences and Education. Washington, DC: The 

National Academies Press. 

Rao, J.N.K. and I. Molina (2015). Small Area Estimation, Second Edition. New York: Wiley.

Shen, W. and T.A. Louis (1998). “Triple-Goal Estimates in Two-Stage Hierarchical Models.” 

Journal of the Royal Statistical Society, Series B 60, 455-471 https://doi.org/10.1111/1467-

9868.00135

Simon, H. A. (1956). "Rational Choice and the Structure of the Environment." Psychological 

Review 63 (2), 129–138. 

Tourangeau, Roger, Brad Edwards, Timothy P. Johnson, Kirk M. Wolter and Nancy Bates 

(eds.) (2014). Hard-to-Survey Populations. New York: Wiley. 

Tukey, John W. (1962). The Future of Data Analysis. Ann. Math. Statist. 33, 1--67. 

doi:10.1214/aoms/1177704711 

Wolter, K.M. (2007). Introduction to Variance Estimation, Second Edition. New York: Springer.

36