NATIONAL CENTER FOR SCIENCE AND ENGINEERING STATISTICS

NATIONAL SCIENCE FOUNDATION

A Data Quality Scorecard to Assess 

a Data Source’s Fitness for Use

Elizabeth Mannshardt

FCSM

10/25/2023

John Finamore; NCSES

Julie Banks, F. Jay Breidt, Zachary H. Seeskin, Kiegan Rice; NORC at the University of Chicago
Overview

2

Motivation

Assessment

Design Features

Discussion
Motivation

3
N A T I O N A L C E N T E R F O R S C I E N C E A N D E N G I N E E R I N G S T A T I S T I C S 

https://ncses.nsf.gov

Motivation

4

The FCSM Framework for Data Quality provides valuable guidance to assess data 

quality. Tools are needed to support application of the Framework to data files.

Federal Committee on Statistical Methodology. 2020. A Framework for Data Quality. FCSM 20-04. 
N A T I O N A L C E N T E R F O R S C I E N C E A N D E N G I N E E R I N G S T A T I S T I C S 

https://ncses.nsf.gov

Motivation

5

Data quality assessment should be user-friendly, objective, and reproducible. 
N A T I O N A L C E N T E R F O R S C I E N C E A N D E N G I N E E R I N G S T A T I S T I C S 

https://ncses.nsf.gov

Motivation – NCSES’s federal clearinghouse role 

6

A Variety of Data Sources

• NCSES official statistics

• Universities 

• Government agencies

• Businesses 

• Individuals

• Census, BLS, NCES, etc.

• International, OECD data

• Bibliometric data

• Patent data information
N A T I O N A L C E N T E R F O R S C I E N C E A N D E N G I N E E R I N G S T A T I S T I C S 

https://ncses.nsf.gov

Motivation

7

• Quality assessment is critical to evidence-building 

• Challenges 

o Complex policy and research questions 

o Additional/evolving data sources and data types 

• Needs 

o Consistent terminology 

o A user-friendly tool to assess data capabilities and limitations

o Method to operationalize the FCSM Data Quality Framework 
Assessment

8
N A T I O N A L C E N T E R F O R S C I E N C E A N D E N G I N E E R I N G S T A T I S T I C S 

https://ncses.nsf.gov

Assessment

9

Fitness for use and Framing Questions

• Use case

• Data reference 

period

• Temporal scale

• Assessment 

reference period
N A T I O N A L C E N T E R F O R S C I E N C E A N D E N G I N E E R I N G S T A T I S T I C S 

https://ncses.nsf.gov

Assessment

10

Operationalizing the FCSM Data Quality Framework
N A T I O N A L C E N T E R F O R S C I E N C E A N D E N G I N E E R I N G S T A T I S T I C S 

https://ncses.nsf.gov

Assessment

11

Example Scorecard Questions

▪ Relevance (Utility Domain):

Does data documentation clearly state appropriate uses for the data?

Are the units included in the data relevant to the use case?

▪ Coherence (Objectivity Domain):

Are key variables defined in the same way for each record in data, including across subgroups?

Are key variables measured using same techniques as similar constructs from external data sources?

▪ Credibility (Integrity Domain):

Is the data producer a non-partisan organization?

Is there documentation on all of the following topics: 

(a) Data collection methods or data generation processes, 

(b) Methodological assumptions

(c) Data error or mitigations to reduce error

(d) Data limitations.
Design Features

12
N A T I O N A L C E N T E R F O R S C I E N C E A N D E N G I N E E R I N G S T A T I S T I C S 

https://ncses.nsf.gov

Design Features

13

After completing the evaluation, the users can generate an HTML or PDF report that can be 

shared as part of the supporting documentation for a project.
N A T I O N A L C E N T E R F O R S C I E N C E A N D E N G I N E E R I N G S T A T I S T I C S 

https://ncses.nsf.gov

Design Features

14

With the R Shiny and R Markdown setup, the tool readily supports edits

o Dashboard content maintained in Excel metadata spreadsheet

▪ Domain and dimension details 

▪ Question text

▪ Question details and information

o Updates to metadata in spreadsheet automatically reflected in dashboard 

o User interface elements generated programmatically using functions that 

pull text from Excel metadata
Discussion

15
N A T I O N A L C E N T E R F O R S C I E N C E A N D E N G I N E E R I N G S T A T I S T I C S 

https://ncses.nsf.gov

Discussion

16

We recommend that users with different areas of expertise collaborate 

to complete each scorecard. Areas of expertise needed include:

1. Familiarity with or investigation of data source

2. Knowledge of subject matter applications

3. Statistical expertise
N A T I O N A L C E N T E R F O R S C I E N C E A N D E N G I N E E R I N G S T A T I S T I C S 

https://ncses.nsf.gov

Discussion

17

Score is use-case specific
N A T I O N A L C E N T E R F O R S C I E N C E A N D E N G I N E E R I N G S T A T I S T I C S 

https://ncses.nsf.gov

Discussion

18

Data documentation should inform responses to scoring items and may be 

more complete for some data sources than others (survey vs non-survey)
N A T I O N A L C E N T E R F O R S C I E N C E A N D E N G I N E E R I N G S T A T I S T I C S 

https://ncses.nsf.gov

Discussion

19

Comparing data quality scores can reveal strengths and limitations of 

different data sources, including between survey and non-survey data. 
N A T I O N A L C E N T E R F O R S C I E N C E A N D E N G I N E E R I N G S T A T I S T I C S 

https://ncses.nsf.gov

Discussion

20

Because quality of a data file is based on a specific use case, it can be 

valuable to evaluate data quality with alternate use cases.

• Use cases may vary in needs for timeliness, granularity, or accuracy

• Aspects of ‘relevance’ dimension may vary among use cases

• The scorecard can load a prior use case scorecard for the same data 

source as a starting point
N A T I O N A L C E N T E R F O R S C I E N C E A N D E N G I N E E R I N G S T A T I S T I C S 

https://ncses.nsf.gov

Discussion

21

Instances where non-survey data sources have advantages
N A T I O N A L C E N T E R F O R S C I E N C E A N D E N G I N E E R I N G S T A T I S T I C S 

https://ncses.nsf.gov

Discussion

22

Consider use case: Some data sources have advantages with accuracy
N A T I O N A L C E N T E R F O R S C I E N C E A N D E N G I N E E R I N G S T A T I S T I C S 

https://ncses.nsf.gov

Discussion

23

Consider use case: Some data sources have advantages with accuracy, 

timeliness
N A T I O N A L C E N T E R F O R S C I E N C E A N D E N G I N E E R I N G S T A T I S T I C S 

https://ncses.nsf.gov

Discussion

24

Consider use case: Some data sources have advantages with accuracy, 

timeliness, and granularity
N A T I O N A L C E N T E R F O R S C I E N C E A N D E N G I N E E R I N G S T A T I S T I C S 

https://ncses.nsf.gov

Discussion

25

Security documentation may be sparse
N A T I O N A L C E N T E R F O R S C I E N C E A N D E N G I N E E R I N G S T A T I S T I C S 

https://ncses.nsf.gov

Discussion

26

The scorecard allows for ready and systematic data quality assessment 

of a range of data files applying the FCSM Data Quality Framework.

▪ Yes/no questions support scoring by data quality 

dimensions and comparing data files

▪ Tool incorporates technological advantages of R Shiny and 

R Markdown implementation and report generation

▪ Provides a way to quantify qualitative measures
@NCSESgov

https://ncses.nsf.gov

27

Elizabeth Mannshardt

emannsha@nsf.gov