THE PROMISE OF

EVIDENCE-BASED POLICYMAKING

Report of the Commission on Evidence-Based Policymaking

Appendices E-H
This page intentionally left blank
The Promise of Evidence-Based Policymaking iii

The Promise of

Evidence-Based Policymaking

Report of the Commission on Evidence-Based Policymaking

Appendices E-H

September 2017
This page intentionally left blank
The Promise of Evidence-Based Policymaking v

Contents

Appendix E: Report on the CEP Survey of Federal Offices..............................................1

Purpose and Development of the CEP Survey of Federal Offices ......................................1

Response Rates..................................................................................................................................3

Data Analysis.......................................................................................................................................3

Highlights of Survey Results............................................................................................................4

Attachment A. List of Federal Offices Eligible for the 

CEP Survey of Federal Offices, by Agency, 2017 ................................................................10

Attachment B. Condensed Version of Survey Items from the 

CEP Survey of Federal Offices, 2017........................................................................................17

Attachment C. Detailed Survey Items and Skip Patterns from the 

CEP Survey of Federal Offices, 2017.......................................................................................23

Attachment D. Federal Office Reponses to Open-Ended 

Questions on the CEP Survey of Federal Offices, 2017 ...................................................115

Appendix F: CEP Public Meeting Materials and Presentations..................................191

July 22, 2016 Meeting .................................................................................................................192

September 9, 2016 Meeting .....................................................................................................224

November 4, 2016 Meeting....................................................................................................... 257

December 12, 2016 Meeting ......................................................................................................314

January 13, 2017 Meeting.......................................................................................................... 372

February 24, 2017 Meeting .......................................................................................................435

March 13, 2017 Meeting .............................................................................................................485

Appendix G: CEP Public Input–Hearing Testimony and 

Other Public Comments..................................................................................................505

Washington, DC Public Input–Hearing Testimony.............................................................506

Chicago Public Input–Hearing Testimony ............................................................................ 572

San Francisco Public Input–Hearing Testimony.................................................................. 657

Other Public Input–Hearing Testimony ................................................................................. 725

Comments Submitted to the Federal Register ................................................................. 908

Appendix H: Prior Commissions Related to Evidence Building ...............................2319
This page intentionally left blank
The Promise of Evidence-Based Policymaking 1

Appendix E: Report on the 

CEP Survey of Federal Off ices

Beginning in January 2017, the Commission on Evidence-Based Policymaking (CEP) extended invitations to over 200 Federal offices to 

complete a survey entitled, “Building Evidence 

from Data: Capacity and Needs in the Federal 

Government.” This report describes the purpose 

and development of the survey, methods used in 

data collection, and observed response patterns. 

It then presents highlighted findings based on a 

limited analysis of selected survey items.

Purpose and Development 

of the CEP Survey of Federal 

Offices

The CEP Survey of Federal Offices provided an 

opportunity for eligible offices to share information with the Commission about how they collect, 

use, and make data available for evidence-building. The survey also collected detailed information about the barriers Federal offices face in accessing data for evidence building. The survey was 

designed to be applicable to individual Federal offices involved in evidence building and use; therefore multiple relevant offices within each Federal 

department received an invitation to complete 

the survey. The Commission sought detailed information directly from individual offices so it 

could learn about variations in capacity and the 

prevalence of key barriers within Federal departments and government wide.

List of Eligible Offices

The Commission developed the initial list of 

offices eligible to participate in the CEP Survey of 

Federal Offices in early November 2016. Eligible 

offices were defined as those Federal units generating data, building evidence, and using evidence 

for policymaking. The initial list was developed 

from Federal budget accounts for Chief Financial 

Officers (CFO) Act Agencies at the bureau level. 

Additional offices were added, including at the 

sub-bureau level,1 to ensure coverage of known 

Federal offices involved in evidence building and 

use. The added offices included units identified in 

Statistical Programs of the United States Government: 

Fiscal Year 20162 and offices that are members of 

the Interagency Council on Evaluation Policy. The 

list excluded offices that primarily provide operational support or administrative functions, such 

as Offices of the General Counsel and procurement offices. 

During data collection, discussions with individual offices and department officials led to several list adjustments. Some Federal offices were 

added to the list to ensure complete coverage. For 

example, the Office of Advocacy in the U.S. Small 

Business Administration was added and the National Center for Special Education Research in 

the U.S. Department of Education was made an 

eligible office separate from the National Center 

for Education Research. The National Institutes of 

Health elected to submit a single coordinated response for its Institutes and Centers; the list was 

adjusted accordingly. 

The final list of eligible offices consisted of 209 

offices from 20 agencies with at least two operating units, plus an additional 11 single bureau 

agencies and commissions (e.g., Corporation for 

National and Community Service, Institute of 

Museum and Library Services). See Attachment A 

for the final list of eligible and responding offices. 

Each office on the list was coded by type of 

office (Principal Statistical Agency, Evaluation 

Office, or Other) and by priority for nonresponse 

follow-up (High, Medium, or Low). Type of office 

designation was derived from a variety of sources. 

Principal Statistical Agencies were identified from 

1. Budget account data obtained from the FY 2017 President’s 

Budget provided by the Office of Management and Budget.

2. Office of Management and Budget, Statistical Programs of the 

United States Government (Washington: D.C.: 2017); https://

obamawhitehouse.archives.gov/sites/default/f iles/omb/assets/

information_and_regulatory_affairs/statistical-programs-2017.pdf

(accessed August 8, 2017).
2 The Promise of Evidence-Based Policymaking

Statistical Programs of the United States Government: 

Fiscal Year 2016. Evaluation Offices were identified 

through the Interagency Council on Evaluation 

Policy and through Commission expertise. “Other 

Offices” comprised the remainder and included a 

variety of statistical programs, program agencies, 

and policy offices.

The nonresponse priority designation was 

based on OMB’s annual report of statistical activities and Commissioner and CEP staff expertise.3

Offices designated as high priority for nonresponse 

follow-up included statistical and evaluation offices plus other offices that are either large or play a 

significant role in Federal evidence building. Offices designated as low priority for nonresponse 

follow-up included small offices and offices with 

a limited role in Federal evidence building. Medium offices comprised the remainder. The priority designation was used during data collection to 

determine the intensity of nonresponse follow-up 

efforts.

Questionnaire Development and 

Testing

CEP staff developed the survey content based 

on Commissioner input about key concepts they 

wished to measure. The draft survey was programmed into the MAX Survey platform managed 

by the Office of Management and Budget (OMB) 

during November and December of 2016. Commissioners reviewed the survey in early December of 2016 and OMB officials offered suggestions 

which were incorporated into a testing version. In 

late December of 2016, staff sent the testing version of the survey to nine Federal colleagues with 

relevant expertise and availability during the testing timeframe. All of the survey testers were from 

offices eligible for the survey but none of the testers was their office’s designated respondent for 

the actual survey data collection.

Survey testers provided substantial comments 

on the survey’s content, flow, and item wording. 

3. This determination was also informed by the Office of Management and Budget’s (OMB) paper, “Overview of Federal Evidence-Building Efforts,” White Paper for the Commission on Evidence-Based Policymaking (Washington, D.C.: OMB, Executive 

Office of the President, 2016); https://obamawhitehouse.archives.

gov/sites/default/f iles/omb/mgmt-gpra/overview_of_federal_evidence_building_efforts.pdf (accessed August 8, 2017).

CEP staff debriefed by telephone with most of the 

testers after they had filled out the survey and 

provided written comments. One tester agreed to 

fill out the survey and concurrently “think aloud” 

while CEP staff listened, took notes, and asked 

questions about item clarity and construction. All 

of this input helped to ensure that the intention 

and wording of the survey items would be clear to 

respondents. Final revisions based on testing were 

made in early January 2017. 

See Attachment B for a summary of survey 

items and Attachment C for detailed survey content including skip patterns. The survey closes with a series of open-ended questions about 

barriers to evidence building and examples of 

evidence-building activities. Responses to the 

open-ended questions are shown in Attachment 

D.

Data Collection Procedures

CEP staff identified a specific senior Federal official—generally a Director, Deputy Director, Chief 

of Staff, or Chief of Policy—as the primary contact and respondent for each eligible office. Initial invitations to complete the survey were sent 

to respondents by email from the MAX Survey 

platform on Monday, January 9, 2017. Reminder 

emails were sent to all nonrespondents on each 

of the three following Tuesdays in January. On 

February 1, 2017, the CEP Executive Director sent 

a follow-up email to all nonrespondents directly 

from her CEP email account rather than through 

the MAX Survey system. The email from her CEP 

account resulted in a large number of undeliverable email returns that had not previously been 

detected by the MAX system. The inaccurate email 

addresses generally fell into two categories: emails 

for officials who had left Federal service during 

the Presidential transition in late January 2017 

and emails for incumbent Federal officials that 

contained errors. CEP staff investigated all of the 

undeliverable emails, found replacements for officials who had left, and corrected email address 

errors. New invitations were sent to replacement 

respondents and to corrected email addresses in 

early February. Beginning on February 7th, CEP 

staff began intensive email and telephone nonresponse follow-up of eligible offices designated on 

the final list of eligible offices as “high priority;” 
The Promise of Evidence-Based Policymaking 3

follow-up efforts continued until the close of data 

collection. There was no additional follow-up conducted for “medium priority” and “low priority” 

offices. Data collection remained open with late 

responses accepted until the end of April 2017.

Response Rates

At the close of data collection, 90 of the 209 

eligible offices had submitted survey responses for 

an overall response rate of 43 percent as shown 

in Table A. Response rates varied considerably by 

type of office and priority designation. All 13 of 

the Principal Statistical Agencies, 83 percent of 

Evaluation Offices, and 79 percent of the offices 

designated as “high” priority responded to the 

survey. In contrast, 36 percent of the Other offices 

and 29 and 30 percent of “medium” and “low” offices, respectively, submitted responses.

Table B shows the number of eligible and responding offices for each agency. An agency had to 

have at least two eligible offices to be listed separately in this table. Single office agencies and commissions are aggregated in Table B. Agency-level 

response rates ranged from 100 percent for the 

National Science Foundation and the U.S. Small 

Business Administration to 0 percent for the U.S. 

Agency for International Development.

Data Analysis

CEP staff downloaded a .csv file from the Max 

Survey platform containing all of the survey responses for responding offices. The analysis file 

excluded responses to the open-ended questions 

at the end of the survey. The .csv file was converted to an Excel file for analysis with pivot tables. CEP staff edited the data in creating the final 

analysis file, including concatenating the names 

of responding Federal offices into one variable 

and recoding “yes” to 1 and “no” to 0 for certain 

variables used in the limited analysis presented in 

this Appendix. The final survey analysis file used 

to create the tables presented in this Appendix is 

available as an Excel spreadsheet at www.cep.gov/

library/Final-CEP-Survey-Analysis-File.xlsx.

The results presented in Tables 1 through 4 

represent only a small portion of the potential for 

analysis from this survey. For example, the survey goes into considerable detail about the various data services that agencies provide (e.g., do 

you have staff dedicated to conducting disclosure 

review?) and the specifics of data linkages (e.g., 

What linkage keys do you use? How do you protect them?). The final survey analysis file may be 

a resource for those seeking more information 

about capacity for and barriers to evidence generation and use among Federal agencies. 

Table A.—Survey Response Rates by Type of Office and Priority Category

Number 

of Eligible 

Offices

Number 

Responding

Response 

Rate

Type of Office

Principal Statistical Agency 13 13 100%

Evaluation Office 12 10 83%

Other 184 67 36%

Priority Category

High 57 45 79%

Medium 63 18 29%

Low 89 27 30%

Total 209 90 43%
4 The Promise of Evidence-Based Policymaking

Highlights of Survey Results

Tables 1 and 2 are based on the responses provided by the 79 responding Federal offices that 

reported that they collect or analyze data to produce statistics about the economy, society, or environment, to evaluate a program to understand 

its implementation or impact on participants; to 

support research on a population, group of entities, policy, or program, or to conduct analysis 

of existing policies or programs to assist decision 

making or who spend 1 to 100 percent of their total annual budget for these statistics, evaluation, 

research, or policy analysis (SERA) purposes. 

Table 1 summarizes barriers to using data for 

evidence building, by showing the number of Federal offices that reported that each of nine factors 

is a “major” or “moderate” barrier to its ability to 

use data for statistical, evaluation, research, or 

policy analysis purposes.

• Over half of responding Federal offices reported that “lack of ability to hire appropriate staff ”, “inadequate funding”, and “legal 

limitations” were major or moderate barriers. 

Ten out of 13 Principal Statistical Agencies 

and 8 out 10 Evaluation Offices cited “legal 

limitations” as a barrier. About three-quarters 

of offices that spend 91 to 100 percent of their 

budget for SERA purposes reported that “legal 

limitations” was a major or moderate barrier.

Table B.—Survey Response Rates by Agency

Agency

Number 

of Eligible 

Offices

Number 

Responding

Response 

Rate

U.S. Department of Agriculture 19 4 21%

U.S. Department of Commerce 19 5 26%

U.S. Department of Defense 4 2 50%

U.S. Department of Education 13 6 46%

U.S. Department of Energy 10 4 40%

U.S. Department of Health and Human Services 39 24 62%

U.S. Department of Homeland Security 9 1 11%

U. S. Department of Housing and Urban Development 6 1 17%

U.S. Department of Justice 12 5 42%

U.S. Department of Labor 8 4 50%

U.S. Department of State 3 1 33%

U.S. Department of the Interior 10 5 50%

U.S. Department of the Treasury 8 4 50%

U.S. Department of Transportation 11 2 18%

U.S. Department of Veterans Affairs 5 3 60%

U.S. Environmental Protection Agency 8 2 25%

National Science Foundation 3 3 100%

U.S. Small Business Administration 2 2 100%

U.S. Social Security Administration 7 5 71%

U.S. Agency for International Development 2 0 0%

Single Bureaus and Commissions 11 7 64%

Total 209 90 43%
The Promise of Evidence-Based Policymaking 5

Table 2 summarizes the most prevalent barriers to accessing specific kinds of data, showing 

the number of Federal offices reporting that they 

considered using 13 different kinds of data for 

statistical, evaluation, research, or policy analysis purposes, but ultimately were not able to use 

the data because it was too hard to get access to 

the specific dataset they needed. For each kind 

of data, the table then shows how many offices 

reported certain barriers to using those data and 

whether those barriers occurred in their own office, in their own agency, in another agency, or at 

the state level. State level is only applicable to certain kinds of data.

• Of the 79 offices that collect or analyze data 

for SERA purposes or spend a portion of their 

budget for SERA purposes, 22 found Federal 

tax data hard to access. Twelve offices found 

criminal justice data and Medicare/Medicaid 

claims and utilization data hard to access and 

11 found education data (Federal or state), 

Federal social security data, or state unemployment insurance wage record data hard to 

access. 

• Among the 22 offices that found Federal tax 

data hard to access, 13 reported that regulations or policies in another agency that make 

it difficult to link data was a barrier to access 

and 11 reported that statutes prohibiting 

data sharing in another agency was a barrier 

to access. Seven offices indicated that lack of 

staff, policies, and procedures to establish data 

sharing agreements in their own offices was a 

barrier to accessing Federal tax data. 

Tables 3 and 4 are based on the 70 responding 

Federal offices reporting that they collect data 

about individual units (e.g., people, businesses, 

nonprofit organizations, schools) either directly 

or through an intermediary source (e.g., state or 

local governments, private entities) routinely (as 

a regular part of their programs or processes), or 

occasionally (not as a regular part of their programs or processes).

Table 3 shows the number and percent of Federal offices that reported that any other agency, or 

its representatives, access their confidential data 

for SERA purposes and the number and percent 

of agencies that reported that researchers outside 

the Federal government access their confidential 

data. 

• Three-quarters of the Principal Statistical 

Agencies and 5 out of 8 Evaluation offices 

report that other agencies access their confidential data for SERA purposes. Offices with 

higher percentages of their budget allocated 

for SERA purposes are more likely to report 

that other agencies access their data. 

• All of the Principal Statistical Agencies and 5 

out of 8 Evaluation Offices report that outside 

researchers access their confidential data, 

while only 22% (13 out of 60) of Other offices 

report that outside researchers access their 

confidential data. 

Table 4 shows the number and percent of 

Federal offices that typically create metadata or 

technical documentation for the data the office 

produces, including variable definitions, units of 

measurement, and response ranges; changes to 

definitions, scope, or data collection processes 

over time; and imputation procedures or other 

edits to the data.

• Overall, two-thirds of responding Federal offices that collect data routinely or occasionally 

report that they produce metadata that includes variable definitions, units of measurement, and response ranges, compared to 61 

percent that produce metadata that includes 

changes to definitions, scope, or data definitions over time and 53 percent that produce 

metadata that include imputation procedures 

or other edits to the data. 

• A higher proportion of Principal Statistical 

Agencies and Evaluation Offices than Other 

offices produced metadata with each of the 

three elements.
6 The Promise of Evidence-Based Policymaking

Table 1. Federal offices that report each factor is a "major" or "moderate" barriers to 

using data for evidence building, by selected office characteristics: 2017

Factors

Number of 

Federal 

offices

Lack of 

ability 

to hire 

appropriate 

staff

Lack of 

ability to 

execute 

and 

manage 

contracts

Inadequate 

funding

Legal 

limitations

Information 

Technology (IT) 

systems 

or requirments

Privacy 

and 

data 

security 

requirements

Political 

processes (e.g. 

transitions 

in agency/

office leadership, congressional 

oversight)

Information 

collection 

requirements 

(e.g. PRA 

and IRB)

Other

Total 79* 43 13 48 41 35 37 21 36 7

Type of Office

Principal 

Statistical 

Agency

13 8 4 7 10 5 5 6 2 1

Evaluation 

office 10 5 1 4 8 5 9 2 6 1

Other 56 30 8 37 23 25 23 13 28 5

Percent of Budget for SERA Purposes

0 percent 1 0 0 1 0 0 0 0 0 0

1 to 10 

percent 27 14 4 13 7 13 12 4 13 3

11 to 50 

percent 11 8 2 9 6 6 3 5 6 1

51 to 90 percent 8 3 1 2 4 3 4 2 2 1

91 to 100 

percent 25 14 5 17 19 10 15 7 11 1

No Response 7 4 1 6 5 3 3 3 4 1

*The number of responding Federal offices varies across factors due to item nonresponse. To calculate percentages, use a base of 79 responding offices for 

each factor except for "lack of ability to execute and hire contracts" and "privacy and data security requirements" (77), "legal limitations" (75), and "political 

processes" (71).

NOTE: Table based on the 79 responding Federal offices that reported on the survey that they collect or analyze data for statistical, evaluation, research, 

or policy analysis (SERA) purposes or spend a portion of their budget for SERA purposes."Type of Office" was determined in consultation with OMB when 

developing the list of eligible offices. "Percent of Budget for SERA Purposes" is the response to an item on the survey.

SOURCE: "Building Evidence from Data: Capacity and Needs in the Federal Government," U.S. Commission on Evidence-Based Policymaking Survey of 

Federal Offices, 2017.
The Promise of Evidence-Based Policymaking 7

Table 2. Kinds of data Federal offices found hard to access, by barriers in their own office, another office, another agency, or in states: 2017

Barriers

Lack of staff, policies, and 

procedures to establish 

data sharing agreements

Statutes prohibiting 

data sharing

Regulations or policies that 

make it difficult to link data

Lack of capacity in Information 

Technology systems

Lack of staff or contractors 

with specific technical 

expertise needed to link data

Concerns about keeping 

the linked data secure 

and confidential

Number 

of offices 

reporting 

data are 

hard to 

access

Own 

Office

Other 

Office

Other 

AgencyState Own 

Office

Other 

Office

Other 

AgencyState Own 

Office

Other 

Office

Other 

AgencyState Own 

Office

Other 

Office

Other 

AgencyState Own 

Office

Other 

Office

Other 

AgencyState Own 

Office

Other 

Office

Other 

AgencyState

Total 79 23 9 9 7 3 7 40 9 5 11 55 14 10 3 3 2 16 5 6 2 5 12 10 2

Kind of Data

Criminal justice data 12 4 3 2 1 1 2 3 1 2 0 5 3 1 1 1 2 5 1 2 2 0 1 2 0

Decennial Census/

American Community 

Survey (ACS)

7 0 1 1 NA 0 0 3 NA 0 0 4 NA 0 0 0 NA 0 0 0 NA 0 0 0 NA

Education data (Federal) 4 0 0 1 NA 0 0 2 NA 0 0 2 NA 0 0 0 NA 0 0 0 NA 0 0 1 NA

Education data (state or 

private) 11 0 0 0 0 0 1 3 3 0 1 4 2 0 0 0 0 0 0 0 0 0 0 0 2

Federal data on business 

program participation 6 0 0 0 NA 0 1 1 NA 0 1 2 NA 2 0 0 NA 2 0 1 NA 0 0 0 NA

Federal establishment 

survey data (e.g., 

economic and 

agricultural censuses)

8 1 0 0 NA 0 0 4 NA 0 0 3 NA 0 0 0 NA 0 0 0 NA 0 0 0 NA

Federal tax data 22 7 1 1 NA 2 1 11 NA 1 2 13 NA 3 1 1 NA 2 1 1 NA 2 3 3 NA

Federal social security 

data 11 3 1 1 NA 0 0 3 NA 0 1 3 NA 1 0 0 NA 2 1 2 NA 0 1 1 NA

Human services program 

data (e.g., SNAP and 

TANF)

9 2 0 0 0 0 1 2 1 0 1 5 2 1 0 0 0 1 0 0 0 0 1 1 0

Medicaid/Medicare 

claims and utilization 

data

12 2 1 0 1 0 1 2 0 1 2 3 1 0 0 0 0 2 1 0 0 1 2 0 0

Military service and 

veteran benefits records 7 1 1 2 NA 0 0 0 NA 1 1 3 NA 1 1 1 NA 1 1 0 NA 0 0 1 NA

State unemployment 

insurance wage record 

data

11 1 1 1 3 0 0 4 3 0 0 4 4 1 0 0 0 0 0 0 0 1 1 0 0

NOTE: Table based on the 79 responding Federal offices that reported on the survey that they collect or analyze data for statistical, evaluation, research, or policy analysis (SERA) purposes or spend a portion of their budget for SERA purposes. Responding offices could report 

experiencing the same barrier from multiple sources. For example, some of the the 23 offices reporting that “lack of staff, policies, and procedures to establish data sharing agreements” in their own office was a barrier may also have reported that barrier as occurring in another 

office, another agency, or at the state level.

NA: Barriers at state level not applicable for this kind of data.

SOURCE: “Building Evidence from Data: Capacity and Needs in the Federal Government,” U.S. Commission on Evidence-Based Policymaking Survey of Federal Offices, 2017.
8 The Promise of Evidence-Based Policymaking

Table 3. Federal offices that provide other agencies and researchers access 

to the data they collect, by selected office characteristics: 2017

Number 

of Federal 

offices

Other Federal agencies 

access data

Researchers external to 

the Federal government 

access data

Number Percent Number Percent

Total 70 34 48.6 32 45.7

Type of Office

 Principal Statistical Agency 13 10 76.9 13 100.0

 Evaluation office 8 5 62.5 5 62.5

 Other 49 19 38.8 14 28.6

Percent of Budget for SERA Purposes

 0 percent 3 1 33.3 1 33.3

 1 to 10 percent 24 9 37.5 4 16.7

 11 to 50 percent 10 4 40.0 3 30.0

 51 to 90 percent 6 3 50.0 6 100.0

 91 to 100 percent 19 12 63.2 14 73.7

 No Response 8 5 62.5 4 50.0

NOTE: Table based on the 70 responding Federal offices that collect data about individual units either directly or through an intermediary. “Type of Office” was determined in consultation with OMB in developing the list of eligible offices. “Percent of Budget 

for SERA Purposes” is the response to an item on the survey. SERA stands for statistics, evaluation, research or policy analysis.

SOURCE: “Building Evidence from Data: Capacity and Needs in the Federal Government,” U.S. Commission on Evidence-Based 

Policymaking Survey of Federal Offices, 2017.
The Promise of Evidence-Based Policymaking 9

Table 4. Federal offices that typically create metadata or documentation for data 

they collect that includes various elements, by selected office characteristics: 2017

Elements

Number of 

Federal offices

Variable definitions, 

units of measurement, 

and response ranges

Changes to definitions, 

scope, or data collection 

processes over time

Imputation procedures or 

other edits to the data

Number Percent Number Percent Number Percent

Total 70 47 67.1 43 61.4 37 52.9

Type of Office

Principal Statistical 

Agency 13 12 92.3 11 84.6 11 84.6

Evaluation office 8 8 100.0 5 62.5 7 87.5

Other 49 28 57.1 27 55.1 19 38.8

Percent of Budget for SERA Purposes

0 percent 3 1 33.3 2 66.7 0 0.0

1 to 10 percent 24 14 58.3 12 50.0 9 37.5

11 to 50 percent 10 6 60.0 6 60.0 5 50.0

51 to 90 percent 6 5 83.3 4 66.7 4 66.7

91 to 100 percent 19 17 89.5 15 78.9 15 78.9

No Response 8 5 62.5 4 50.0 4 50.0

NOTE: Table based on the 70 responding Federal offices that collect data about individual units either directly or through an intermediary. “Type of Office” was determined in consultation with OMB in developing the list of eligible offices. “Percent of Budget for SERA 

Purposes” is the response to an item on the survey. SERA stands for statistics, evaluation, research or policy analysis.

SOURCE: “Building Evidence from Data: Capacity and Needs in the Federal Government,” U.S. Commission on Evidence-Based Policymaking Survey of Federal Offices, 2017.
10 The Promise of Evidence-Based Policymaking

Attachment A. List of Federal Offices Eligible for the CEP Survey of 

Federal Offices, by Agency, 2017

NOTE: Responding offices are in italics.

Agency Office

Broadcasting Board 

of Governors Broadcasting Board of Governors

U.S. Consumer 

Product Safety 

Commission

Consumer Product Safety Commission

Corporation for 

National and 

Community Service

Corporation for National and Community Service

U.S. Department of 

Agriculture

Agricultural Marketing Service

Agricultural Research Service

Animal and Plant Health Inspection Service

Chief Economist

Economic Research Service

Farm Service Agency

Food and Nutrition Service

Food Safety and Inspection Service

Foreign Agricultural Service

Forest Service

Grain Inspection, Packers and Stockyards Administration

National Agricultural Statistics Service

National Institute of Food and Agriculture

Natural Resources Conservation Service

Office of Civil Rights

Risk Management Agency

Rural Development - Rural Housing Service

Rural Development - Rural Utilities Service

World Agricultural Outlook Board

U.S. Department of 

Commerce

Bureau of Industry and Security

Bureau of the Census

Bureau of Economic Analysis

Economic Development Administration

Economics and Statistics Administration (ex BEA and Census)

International Trade Administration

Minority Business Development Agency

National Institute of Standards and Technology
The Promise of Evidence-Based Policymaking 11

Agency Office

U.S. Department of 

Commerce

(continued)

National Oceanic and Atmospheric Administration — National 

Environmental Satellite Data and Information Service

National Oceanic and Atmospheric Administration — National Marine 

Fisheries Service

National Oceanic and Atmospheric Administration — National Ocean 

Service

National Oceanic and Atmospheric Administration — National Weather 

Service

National Oceanic and Atmospheric Administration — Office of Marine 

and Aviation Operations

National Oceanic and Atmospheric Administration — Office of Oceanic 

and Atmospheric Research

National Technical Information Service

National Telecommunications and Information Administration

Office of Performance Evaluation and Risk Management

U.S. Patent and Trademark Office

United States Travel and Tourism Administration

U.S. Department of 

Defense

Army Corps of Engineers

Defense Health Agency

Defense Manpower Data Center

Office of the Assistant Secretary of Defense for Research and Engineering

U.S. Department of 

Education

National Center for Education Evaluation and Regional Assistance

National Center for Education Research

National Center for Education Statistics

National Center for Special Education Research

Office for Civil Rights

Office of Career, Technical, and Adult Education

Office of Elementary and Secondary Education

Office of English Language Acquisition

Office of Federal Student Aid

Office of Innovation and Improvement

Office of Planning, Evaluation, and Policy Development

Office of Postsecondary Education

Office of Special Education and Rehabilitative Services

U.S. Department of 

Energy

Advanced Research Projects Agency for Energy

Chief Information Officer

Energy Information Administration

National Nuclear Security Administration

Office of Electricity Delivery and Energy Reliability

Office of Energy Efficiency and Renewable Energy
12 The Promise of Evidence-Based Policymaking

Agency Office

U.S. Department of 

Energy

(continued)

Office of Environment, Health, Safety and Security

Office of Fossil Energy

Office of Nuclear Energy

Office of Science

U.S. Department of 

Health and Human 

Services

Administration for Children and Families - Administration on Children, 

Youth, and Families

Administration for Children and Families - Office of Child Care

Administration for Children and Families - Office of Child Support 

Enforcement

Administration for Children and Families - Office of Community Services

Administration for Children and Families - Office of Family Assistance

Administration for Children and Families - Office of Head Start

Administration for Children and Families - Office of Planning, Research, 

and Evaluation

Administration for Children and Families - Office of Refugee 

Resettlement

Administration for Community Living

Agency for Healthcare Research and Quality

Centers for Disease Control and Prevention

Centers for Disease Control and Prevention - Center for Global Health

Centers for Disease Control and Prevention - National Center for Chronic 

Disease Prevention and Health Promotion

Centers for Disease Control and Prevention - National Center for 

Emerging Zoonotic and Infectious Diseases

Centers for Disease Control and Prevention - National Center for 

Environmental Health

Centers for Disease Control and Prevention - National Center for HIV/

AIDS, Viral Hepatitis, Sexually Transmitted Disease, and 

Tuberculosis Prevention

Centers for Disease Control and Prevention - National Center for 

Immunization and Respiratory Diseases

Centers for Disease Control and Prevention - National Center for Injury 

Prevention and Control

Centers for Disease Control and Prevention - National Center on Birth 

Defects and Developmental Disabilities

Centers for Disease Control and Prevention - National Institute for 

Occupational Safety and Health

Centers for Disease Control and Prevention - National Center for Health 

Statistics

Centers for Disease Control and Prevention - Office of the Associate 

Director for Policy

Centers for Disease Control and Prevention - Office of the Associate 

Director for Science
The Promise of Evidence-Based Policymaking 13

Agency Office

U.S. Department of 

Health and Human 

Services

(continued)

Centers for Disease Control and Prevention - Program Performance and 

Evaluation Office, Office of the Director

Centers for Medicare and Medicaid Services - Center for Medicare and 

Medicaid Innovation

Centers for Medicare and Medicaid Services - Office of Enterprise Data 

and Analytics

Centers for Medicare and Medicaid Services - Office of the Director

Food and Drug Administration

Food and Drug Administration - Center for Drug Evaluation and Research

Health Resources and Services Administration

Indian Health Service

National Institutes of Health - NIH Office of Science Policy

Office of Adolescent Health

Office of Minority Health

Office of Population Affairs

Office of the Assistant Secretary for Planning and Evaluation

Office of the National Coordinator

Substance Abuse and Mental Health Services Administration - Center for 

Behavioral Health Statistics and Quality

Substance Abuse and Mental Health Services Administration - Office of 

Policy, Planning, and Innovation

U.S. Department of 

Homeland Security

Citizenship and Immigration Services

Federal Emergency Management Agency

National Protection and Programs Directorate

Office of Policy

Office of Policy - Office of Immigration Statistics

Science and Technology Directorate

Transportation Security Administration

U.S. Customs and Border Protection

U.S. Immigration and Customs Enforcement

U.S. Department of 

Housing and Urban 

Development

Asst. Sec. for Community Planning and Development

Asst. Sec. for Housing Programs

Asst. Sec. for Public and Indian Housing Programs

Fair Housing and Equal Opportunity

Office of Lead Hazard Control and Healthy Homes

Policy Development and Research

U.S. Department of 

Justice

Bureau of Alcohol, Tobacco, Firearms, and Explosives

Bureau of Prisons

Drug Enforcement Administration

Federal Bureau of Investigation - Criminal Justice Information Service
14 The Promise of Evidence-Based Policymaking

Agency Office

U.S. Department of 

Justice

(continued)

Federal Bureau of Investigation - Office of the director

Office of Community Oriented Policing Services

Office of Justice Programs - Bureau of Justice Assistance

Office of Justice Programs - Bureau of Justice Statistics

Office of Justice Programs - National Institute of Justice

Office of Justice Programs - Office of Juvenile Justice and Delinquency 

Prevention

Office of Justice Programs - Office of Victims of Crime

Office of Justice Programs - Office of Violence Against Women

U.S. Department of 

Labor

Bureau of Labor Statistics

Employee Benefits Security Administration

Employment and Training Administration

Mine Safety and Health Administration

Occupational Safety and Health Administration

Office of the Assistant Secretary for Policy - Chief Evaluation Office

Office of Workers' Compensation Programs

Wage and Hour Division

U.S. Department of 

State

Evaluation and Aid Effectiveness Branch, Office of US Foreign Assistance 

Resources

Office of Policy and Planning

Office of the U.S. Global AIDS Coordinator

U.S. Department of 

the Interior

Bureau of Indian Affairs and Bureau of Indian Education

Bureau of Land Management

Bureau of Ocean Energy Management

Bureau of Reclamation

Bureau of Safety and Environmental Enforcement

National Park Service

Office of Natural Resources Revenue

Office of Surface Mining Reclamation and Enforcement

United States Fish and Wildlife Service

United States Geological Survey

U.S. Department of 

the Treasury

Alcohol and Tobacco Tax and Trade Bureau

Bureau of Engraving and Printing

Bureau of Fiscal Service

Federal Financing Bank

Financial Crimes Enforcement Network

Internal Revenue Service - Research, Analysis, and Statistics

Internal Revenue Service - Statistics of Income Division

United States Mint
The Promise of Evidence-Based Policymaking 15

Agency Office

U.S. Department of 

Transportation

Bureau of Transportation Statistics

Federal Aviation Administration

Federal Highway Administration

Federal Motor Carrier Safety Administration

Federal Railroad Administration

Federal Transit Administration

Maritime Administration

National Highway Traffic Safety Administration

Pipeline and Hazardous Materials Safety Administration

Saint Lawrence Seaway Development Corporation

Surface Transportation Board

U.S. Department of 

Veterans Affairs

National Cemetery Administration

National Center for Veterans Analysis and Statistics

Office of Policy and Planning

Veterans Benefits Administration

Veterans Health Administration

U.S. Environmental 

Protection Agency

Office of Air and Radiation

Office of Chemical Safety and Pollution Prevention

Office of Enforcement and Compliance Assurance

Office of Environmental Information

Office of Land and Emergency Management

Office of Policy - National Center for Environmental Economics

Office of Policy - Office of Strategic Environmental Management 

Office of Water

U.S. Equal 

Employment 

Opportunity 

Commission

Equal Employment Opportunity Commission

U.S. General Services 

Administration General Services Administration

Institute of Museum 

and Library Services Institute of Museum and Library Services

Millennium 

Challenge 

Corporation

Millennium Challenge Corporation

National Aeronautics 

and Space 

Administration

National Aeronautics and Space Administration

U.S. National 

Archives 

and Records 

Administration

National Archives and Records Administration
16 The Promise of Evidence-Based Policymaking

Agency Office

National Science 

Foundation

National Center for Science and Engineering Statistics

Office of Integrative Activities

Office of the Director

Nuclear Regulatory 

Commission Nuclear Regulatory Commission

U.S. Office 

of Personnel 

Management

Office of Personnel Management

U.S. Small Business 

Administration

Office of Advocacy

Office of Performance Management

U.S. Social Security 

Administration

Office of Data Exchange and Policy Publications

Office of Disability policy

Office of Income Security Programs

Office of Research, Demonstration, and Employment Support

Office of Retirement Policy

Office of the Actuary

Office of Research, Evaluation, and Statistics

U.S. Agency for 

International 

Development

Office of Learning, Evaluation and Research, Bureau for Policy, Planning 

and Learning

US Agency for International Development
The Promise of Evidence-Based Policymaking 17

Attachment B. Condensed Version of Survey Items from the CEP 

Survey of Federal Offices, 2017

Approximately what percentage of your office's total annual budget is for statistical, evaluation, research, 

or policy analysis purposes?

0 percent

1 to 10 percent

11 to 50 percent

51 to 90 percent

91 to 100 percent

Collecting Data

Does your office, or its representatives, collect data about individual units (e.g., people, businesses, 

nonprofit organizations, schools) either directly or through an intermediary source (e.g., state or local 

governments, private entities)?

Yes, routinely (as a regular part of our programs or processes)

Yes, occasionally (not as a regular part of our programs or processes)

No

Approximately what percentage of your office's total annual budget is for collecting data from individual units?

0 percent

1 to 10 percent

11 to 50 percent

51 to 90 percent

91 to 100 percent

For which of the following purposes does your office, or its representatives, collect data from individual 

units? 

To produce statistics about the economy, society, or environment

To evaluate a program to understand its implementation or impact on participants 

To support research on a population, group of entities, policy, or program

To conduct analysis of existing policies or programs to assist decision making 

To conduct or support program operations

To monitor program or grant performance

Are any of the data collected by your office, or its representatives, considered confidential?

Who analyzes your office's confidential data? 

Staff physically present in my office

Staff working in a location other than my office (including teleworking) 

Contractors physically present in my office

Contractors working in a location other than my office 

Staff or contractors in another office in my agency 

Staff or contractors in another agency

My office's grantees

Researchers outside of the Federal government

Analyzing Micro-Level Data From Another Agency

Does your office, or its representatives, analyze micro-level data collected by any other Federal office 

or agency?

Which of the following kinds of micro-level data collected by any other office or agency does your 

office, or its representatives, analyze?
18 The Promise of Evidence-Based Policymaking

Kinds of Data:

Criminal Justice Data

Decennial Census/American Community Survey (ACS) 

Education data (Federal)

Education data (State or Private)

Federal data on business program participation

Federal establishment survey data (e.g., Economic and Agricultural Censuses) 

Federal tax data

Federal social security data

Human services data (e.g. SNAP and TANF) 

Medicaid/Medicare claims and utilizations data 

Military service and veteran benefits records 

State unemployment insurance wage record data 

Vital records (birth and death) data

For which of the following purposes does your office, or its representatives, analyze micro-level data 

collected by any other Federal office or agency? 

To produce statistics about the economy, society, or environment

To evaluate a program to understand its implementation or impact on participants 

To support research on a population, group of entities, policy, or program

To conduct analysis of existing policies or programs to assist decision making 

To conduct or support program operations

To monitor program or grant performance

Linking Data

Does your office, or its representatives, link the confidential data it collects with data from another 

agency?

Which of the following kinds of data does your office, or its representatives, link with your confidential 

data collected from individual units? [Kinds of Data]

Which of the following best describes your process for linking to [each Kind of Data]? 

Our office, or its representatives, does the linkage.

Another office or agency, or its representatives, does the linkage. 

A third party, or its representatives, does the linkage.

Other

Does your office have staff or contractors whose primary job duties involve the technical process of 

data linkage?

How many staff or contractors does your office have whose primary job duties involve the technical 

process of data linkage?

1 to 2

3 to 5

6 to 10

More than 10

Which of the following statements best describes the regularity of data linkage in your office?

All of our data linkages are for ad hoc projects.

Most of our data linkages are for ad hoc projects; a few are a part of our regular processes.

Our data linkages are about evenly split between ad hoc projects and regular processes.

Most of our data linkages are a part of our regular process; a few are ad hoc projects.

All of our data linkages are part of our regular processes.

How does your office pay for data linkage?

My office has a specific budget allocation that covers data linkage.

My office chooses to set aside a portion of the budget to help pay for data linkage. 

Programs within my office that do data linkage absorb the cost in their budgets.

Other offices provide funds for data linkages. 

User fees help pay for data linkage.
The Promise of Evidence-Based Policymaking 19

Which of the following does your office, or its representatives, use for data linkage? 

A probabilistic match of key characteristics.

A linking key (personal identifier).

A combination of probabilistic matching and linking keys

Which of the following keys does your office, or its representatives, use for data linkage? 

Social Security Number (SSN)

Masked SSN

Employer Identification Number (EIN) Name

Address Data of Birth

Biometrics (e.g. fingerprints)

How does your office, or its representatives, secure data linking keys? 

By restricting access to the linkage key or linkage key crosswalk.

By generating a pseudo-anonymous key (e.g., hash, unique ID, etc) for linking. 

By removing the linkage key from the file(s) once linkage is complete.

Does your office, or its representatives, keep micro-level data that are linked for statistical, research, 

evaluation, or policy analysis purposes so they can be analyzed again at a later time?

Yes, routinely (as a regular part of our programs or processes)

Yes, occasionally (not as a regular part of our programs or processes)

No

Accessing Data

Does another agency, or its representatives, ever access your office's confidential data for its own 

statistical, evaluation, research, or policy analysis purposes?

How does another agency, or its representatives, access your office's confidential data? 

Staff or contractors come onsite to my office or agency to access the data.

Through a Federal Statistical Research Data Center (FSRDC)

Through a distributed thin client Citrix or VPN network

Through a Memorandum of Understanding (MOU), license, or contract agreement 

Through a data archive (e.g., Interuniversity Consortium for Political and Social Research)

Approximately how many other agencies, or their representatives, access your office's confidential 

data each year?

1 to 2

3 to 5

6 to 10

More than 10

Which of the following statements best describes how regularly another agency, or its representatives, access your office's confidential data?

Many other agencies routinely (e.g., annually, quarterly, monthly) access my office's confidential data.

A few other agencies routinely (e.g., annually, quarterly, monthly) access my office's confidential data.

Many other agencies occasionally (i.e. on an ad hoc basis) access my office's confidential data.

A few other agencies occasionally (i.e. on an ad hoc basis) access my office's confidential data.

Some agencies routinely (e.g., annually, quarterly, monthly) and some agencies occasionally (i.e. on 

an ad hoc basis) access my office's confidential data.

Do researchers outside of the Federal government ever access your office's confidential data?

Approximately how many outside researchers access your office's confidential data each year?

1 to 10

11 to 50

51 to 100

More than 100

How do researchers outside of the Federal government access your office's confidential data?

The researcher comes onsite to my office or agency to access the data

Through a Federal Statistical Research Data Center (FSRDC) 

Through a distributed thin client Citrix or VPN network

Through a Memorandum of Understanding (MOU), license, or contract agreement 

Through a data archive (e.g., Interuniversity Consortium for Political and Social Research)
20 The Promise of Evidence-Based Policymaking

Does your office receive and review requests from outside researchers for access to confidential data?

How often does your office receive new requests from outside researchers for access to confidential data?

Daily

Weekly

Monthly

Yearly

Sporadically

Which of the following statements best describes how regularly outside researchers access your office's 

confidential data?

Many outside researchers routinely (e.g. annually, quarterly, monthly) access my office's confidential data.

A few outside researchers routinely (e.g. annually, quarterly, monthly) access my office's confidential data.

Many outside researchers occasionally (i.e. on an ad hoc basis) access my office's confidential data.

A few outside researchers occasionally (i.e. on an ad hoc basis) access my office's confidential data.

Some outside researchers routinely (e.g. annually, quarterly, monthly) and some outside researchers occasionally (i.e. on an ad hoc basis) access my office's confidential data.

Does your office have staff or contractors whose primary job duties involve reviewing and processing 

outside researcher requests for access to confidential data?

How many staff or contractors does your office have whose primary job duties involve reviewing and 

processing outside researcher requests for access to confidential data?

1 to 2

3 to 5

6 to 10

More than 10

Does your office have a system for tracking requests from outside researchers for access to confidential data?

Does your office have a process for reviewing output produced by outside researchers to prevent disclosure of the identities of individual units?

How often does your office receive new requests for disclosure review of outside researcher analyses?

Daily

Weekly

Monthly

Yearly

Sporadically

Does your office have a process for documenting the results of disclosure review of outside researcher 

analyses?

Does your office have staff or contractors whose primary job duties involve disclosure review of outside 

researcher analyses?

How many staff or contractors does your office have whose primary job duties involve disclosure review of outside researcher analyses?

1 to 2

3 to 5

6 to 10

More than 10

Does your office, or its representatives, typically create metadata or technical documentation for the 

data it produces that includes the following? 

Variable definitions, units of measurement, and response ranges

Changes to definitions, scope, or data collection processes over time

Imputation procedures or other edits to the data

Managing Historical Data

Does your office, or its representatives, have a process for archiving, managing, and retrieving your 

historical micro-level data?

Does your office have staff or contractors whose primary job duties involve archiving, managing, and 

retrieving your historical micro-level data?
The Promise of Evidence-Based Policymaking 21

How many staff or contractors does your office have whose primary job duties involve archiving, managing, and retrieving your historical micro-level data?

1 to 2

3 to 5

6 to 10

More than 10

Does your office, or its representatives, continue to manage or clean your historical micro-level data?

How easy or difficult is it for your office, or its representatives, to retrieve and use your historical micro-level data?

very easy

somewhat easy

somewhat difficult

very difficult

When does your office, or its representatives, destroy your historical micro-level data? 

We archive all historical micro-level data.

We destroy historical micro-level data when they are no longer being used for the purpose for which they 

were collected. 

We destroy historical micro-level data on a schedule established by statute, regulation, or policy.

We destroy historical micro-level data when they are no longer relevant for research or policy analysis. 

We destroy historical micro-level data when they are no longer relevant for research or policy analysis.

Using Data for Statistical, Evaluation, Research, or Policy Analysis Purposes

Which of the following kinds of data has your office, or its representatives, considered using for statistical, evaluation, research, or policy analysis purposes but did not because it was too hard to get access 

to the specific dataset you needed?

[Kinds of Data]

Which of the following factors made it hard for your office to get access to [each Kind of Data]? 

Lack of staff, policies, and procedures to establish data sharing agreements

Statutes prohibiting data sharing

Regulations or policies that make it difficult to link data 

Lack of capacity in information technology systems

Lack of staff or contractors with specific technical expertise needed to link data 

Concerns about keeping the data secure and confidential

[For each Kind of Data] Was/were the [factor from the previous question]:

In your office?

In another office in your agency

In another agency

At the state or local level [only applied to appropriate Kinds of Data]

To what extent is each of the following a barrier to your office's ability to use data for statistical, evaluation, research, or policy analysis purposes? 

Lack of ability to hire appropriate staff

Lack of ability to execute and manage contracts 

Inadequate funding

Legal limitations

Information Technology (IT) systems or requirements 

Privacy and data security requirements

Political processes (e.g., transitions in agency/office leadership, Congressional oversight)

Information collection requirements (e.g. Paperwork Reduction Act and Institutional Review Boards)
22 The Promise of Evidence-Based Policymaking

Which ONE of the following is the most critical barrier to your office's ability to use data for statistical, 

evaluation, research, or policy analysis purposes?

Lack of ability to hire appropriate staff

Lack of ability to execute and manage contracts 

Inadequate funding

Legal limitations

Information Technology (IT) systems or requirements 

Privacy and data security requirements

Political processes (e.g., transitions in agency/office leadership, Congressional oversight)

Information collection requirements (e.g. Paperwork Reduction Act and Institutional Review Boards)

If applicable, please provide additional information about this most critical barrier.

If applicable, please list up to five statistical, evaluation, research, or policy analysis activities your office, 

or its representatives, have not been able to complete because it was too hard to access the right data.

If applicable, please provide an example of a micro-level data linkage project or process in your office 

that works well.

If applicable, please provide an example of how your office has used data to build evidence for policy 

making.
The Promise of Evidence-Based Policymaking 23

Attachment C. Detailed Survey Items and Skip Patterns from the 

CEP Survey of Federal Offices, 2017

Building Evidence from Data--Capacity and 

Needs in the Federal Government

There are 178 questions in this survey

U.S. Commission on Evidence-Based Policymaking Survey of 

Federal Offices

The U. S. Commission on Evidence-Based Policymaking, established by law in March 2016, is composed of 15 members 

appointed by the President and Congress. Its mission is to identify ways to strengthen the government’s evidence-building 

efforts. 

This survey of Federal offices is a critical part of the Commission's fact-finding. It is designed to help the Commission 

understand how Federal offices collect, analyze, and provide access to data and how their needs for evidence are 

addressed. 

The section below defines a number of terms that recur in the survey. In many cases, these definitions are also repeated at 

the bottom of questions for your convenience.      

Thank you for completing this survey on behalf of your Federal office.  

Definitions 

Confidential data: identifiable micro-level data that cannot be made public due to law, policy, or practice 

Data linkage: the merging of two or more micro-level datasets originally collected for different purposes 

Entity: any public, private, or nonprofit organization

Individual unit: the lowest level unit at which data are collected, such as people, households, businesses, nonprofit 

organizations, schools 

Intermediary source: any public or private organization that collects data from individual units and sends them to a 

Federal office, such as state or local government agencies, postsecondary institutions, grantees, or law enforcement 

agencies 

Micro-level data: a data file with an entry (record) for each individual unit 

Primary job duties: duties performed more than 50% of full-time equivalent work hours 

Statistical, evaluation, research, or policy analysis purposes: activities intended to support the production, analysis, 

and dissemination of information, including the creation and analysis of federal statistics; production or use of evaluation for 

federally-supported or -directed programs; research on a population, group of entities, policy, or program; or analysis of 

policies or proposals to inform decision making. These purposes do NOT include program operations (e.g., compliance 

monitoring, eligibility determinations, enforcement actions). 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Govern... Page 1 of 92
24 The Promise of Evidence-Based Policymaking

[]Which Federal agency are you responding for? *

Please choose only one of the following:

 Broadcasting Board of Governors 

 Consumer Product Safety Commission 

 Department of Agriculture 

 Department of Commerce 

 Department of Defense 

 Department of Education 

 Department of Energy 

 Department of Health and Human Services 

 Department of Homeland Security 

 Department of Housing and Urban Development 

 Department of Justice 

 Department of Labor 

 Department of State 

 Department of the Interior 

 Department of the Treasury 

 Department of Transportation 

 Department of Veterans Affairs 

 Environmental Protection Agency 

 Equal Employment Opportunity Commission 

 General Services Administration 

 Institute of Museum and Library Services 

 National Aeronautics and Space Administration 

 National Archives and Records Administration 

 National Science Foundation 

 Nuclear Regulatory Commission 

 Office of Personnel Management 

 Small Business Administration 

 Social Security Administration 

 US Agency for International Development 

 Corporation for National and Community Service 

 Millennium Challenge Corporation 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Govern... Page 2 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 25

[]Which office in the Department of Agriculture are you responding for? *

Only answer this question if the following conditions are met:

Answer was 'Department of Agriculture' at question '1 [agency]' (Which Federal agency are you responding for?)

Please choose only one of the following:

 Food and Nutrition Service 

 Farm Service Agency 

 Risk Management Agency 

 Forest Service 

 Natural Resources Conservation Service 

 Foreign Agricultural Service 

 National Institute of Food and Agriculture 

 Agricultural Research Service 

 Agricultural Marketing Service 

 Animal and Plant Health Inspection Service 

 Food Safety and Inspection Service 

 Rural Development - Rural Housing Service 

 Rural Development - Rural Utilities Service 

 National Agricultural Statistics Service 

 Economic Research Service 

 Grain Inspection, Packers and Stockyards Administration 

 Office of Civil Rights 

 World Agricultural Outlook Board 

 Chief Economist 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Govern... Page 3 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
26 The Promise of Evidence-Based Policymaking

[]Which office in the Department of Commerce are you responding for? *

Only answer this question if the following conditions are met:

Answer was 'Department of Commerce' at question '1 [agency]' (Which Federal agency are you responding for?)

Please choose only one of the following:

 National Oceanic and Atmospheric Administration - National Environmental Satellite Data and 

Information Service 

 National Oceanic and Atmospheric Administration - National Marine Fisheries Service 

 National Oceanic and Atmospheric Administration - National Ocean Service 

 National Oceanic and Atmospheric Administration - National Weather Service 

 National Oceanic and Atmospheric Administration - Office of Marine and Aviation Operations 

 National Oceanic and Atmospheric Administration - Office of Oceanic and Atmospheric Research 

 Bureau of the Census 

 National Institute of Standards and Technology 

 National Telecommunications and Information Administration 

 International Trade Administration 

 Economic Development Administration 

 Bureau of Industry and Security 

 Economics and Statistics Administration (ex BEA and Census) 

 Minority Business Development Agency 

 National Technical Information Service 

 United States Travel and Tourism Administration 

 U.S. Patent and Trademark Office 

 Bureau of Economic Analysis 

 Office of Performance Evaluation and Risk Management 

[]Which office in the Department of Defense are you responding for? *

Only answer this question if the following conditions are met:

Answer was 'Department of Defense' at question '1 [agency]' (Which Federal agency are you responding for?)

Please choose only one of the following:

 Office of the Assistant Secretary of Defense for Research and Engineering 

 Army Corps of Engineers 

 Defense Health Agency 

 Defense Manpower Data Center 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Govern... Page 4 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 27

[]Which office in the Department of Education are you responding for? *

Only answer this question if the following conditions are met:

Answer was 'Department of Education' at question '1 [agency]' (Which Federal agency are you responding for?)

Please choose only one of the following:

 Office of Federal Student Aid 

 Office of Elementary and Secondary Education 

 Office of Special Education and Rehabilitative Services 

 Office of Postsecondary Education 

 Office of Career, Technical, and Adult Education 

 Office of Innovation and Improvement 

 Office of English Language Acquisition 

 National Center for Education Research and National Center for Special Education Research 

 National Center for Education Statistics 

 National Center for Educational Evaluation 

 Office for Civil Rights 

 Office of Planning, Evaluation, and Policy Development 

[]

Which office in the Department of Energy are you responding for?

*

Only answer this question if the following conditions are met:

Answer was 'Department of Energy' at question '1 [agency]' (Which Federal agency are you responding for?)

Please choose only one of the following:

 National Nuclear Security Administration 

 Office of Science 

 Office of Energy Efficiency and Renewable Energy 

 Office of Fossil Energy 

 Office of Nuclear Energy 

 Office of Electricity Delivery and Energy Reliability 

 Advanced Research Projects Agency for Energy 

 Energy Information Administration 

 Office of Environment, Health, Safety and Security 

 Chief Information Officer 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Govern... Page 5 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
28 The Promise of Evidence-Based Policymaking

[]

Which office in the Department of Health and Human Services are you 

responding for?

*

Only answer this question if the following conditions are met:

Answer was 'Department of Health and Human Services' at question '1 [agency]' (Which Federal agency are you 

responding for?)

Please choose only one of the following:

 Centers for Medicare and Medicaid Services - Office of Enterprise Data and Analytics 

 Centers for Medicare and Medicaid Services - Center for Medicare and Medicaid Innovation 

 Centers for Medicare and Medicaid Services - Office of the Director 

 Office of Minority Health 

 Administration for Children and Families - Office of Planning, Research, and Evaluation 

 Administration for Children and Families - Office of Child Support Enforcement 

 Administration for Children and Families - Office of Family Assistance 

 Administration for Children and Families - Office of Community Services 

 Administration for Children and Families - Administration on Children, Youth, and Families 

 Administration for Children and Families - Office of Refugee Resettlement 

 Administration for Children and Families - Office of Child Care 

 Administration for Children and Families - Office of Head Start 

 National Institutes of Health - National Institute on Aging 

 National Institutes of Health - National Institute on Alcohol Abuse and Alcoholism 

 National Institutes of Health - National Institute of Allergy and Infectious Diseases 

 National Institutes of Health - National Institute of Biomedical Imaging and Bioengineering 

 National Institutes of Health - National Institute of Child Health and Human Development 

 National Institutes of Health - National Institute on Deafness and Other Communication Disorders 

 National Institutes of Health - National Institute of Dental and Craniofacial Research 

 National Institutes of Health - National Institute of Diabetes and Digestive and Kidney Diseases 

 National Institutes of Health - National Institute on Drug Abuse 

 National Institutes of Health - National Institute of Environmental Health Sciences 

 National Institutes of Health - National Institute of General Medical Sciences 

 National Institutes of Health - National Institute of Mental Health 

 National Institutes of Health - National Cancer Institute 

 National Institutes of Health - National Eye Institute 

 National Institutes of Health - National Heart, Lung, and Blood Institute 

 National Institutes of Health - National Human Genome Research Institute 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Govern... Page 6 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 29

 National Institutes of Health - National Center for Complementary and Integrative Health 

 National Institutes of Health - NIH Clinical Center 

 National Institutes of Health - NIH Center for Scientific Review 

 National Institutes of Health - Fogarty International Center 

 National Institutes of Health - National Center for Advancing Translational Sciences 

 National Institutes of Health - National Institute of Arthritis and Musculoskeletal and Skin Diseases 

 National Institutes of Health - National Institute on Minority Health and Health Disparities 

 National Institutes of Health - National Institute of Neurological Disorders and Stroke 

 National Institutes of Health - National Institute of Nursing Research 

 National Institutes of Health - National Library of Medicine 

 National Institutes of Health - NIH Office of Behavioral Health and Social Sciences Research 

 National Institutes of Health - NIH Office of Disease Prevention 

 National Institutes of Health - NIH Office of Program Evaluation and Performance 

 National Institutes of Health - NIH Office of Research Infrastructure Programs 

 National Institutes of Health - NIH Office of Research on Women's Health 

 National Institutes of Health - NIH Office of Strategic Coordination 

 National Institutes of Health - NIH Division of Program Coordination, Planning, and Strategic 

Initiatives 

 National Institutes of Health - NIH Office of AIDS Research 

 National Institutes of Health - NIH Office of Communications & Public Liaison 

 National Institutes of Health - NIH Office of Extramural Research 

 National Institutes of Health - NIH Office of Management 

 National Institutes of Health - Office of Legislative Policy and Analysis 

 National Institutes of Health - NIH Office of Science Policy 

 Health Resources and Services Administration 

 Centers for Disease Control and Prevention 

 Centers for Disease Control and Prevention - Office of the Associate Director for Science 

 Centers for Disease Control and Prevention - Office of the Associate Director for Policy 

 Centers for Disease Control and Prevention - Program Performance and Evaluation Office, Office of 

the Director 

 Centers for Disease Control and Prevention - National Center on Birth Defects and Developmental 

Disabilities 

 Centers for Disease Control and Prevention - National Center for Chronic Disease Prevention and 

Health Promotion 

 Centers for Disease Control and Prevention - National Center for Emerging Zoonotic and Infectious 

Diseases 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Govern... Page 7 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
30 The Promise of Evidence-Based Policymaking

 Centers for Disease Control and Prevention - National Center for Environmental Health 

 Centers for Disease Control and Prevention - National Center for Health Statistics 

 Centers for Disease Control and Prevention - National Center for HIV/AIDS, Viral Hepatitis, Sexually 

Transmitted Disease, and Tuberculosis Prevention 

 Centers for Disease Control and Prevention - National Center for Immunization and Respiratory 

Diseases 

 Centers for Disease Control and Prevention - National Center for Injury Prevention and Control 

 Centers for Disease Control and Prevention - National Institute for Occupational Safety and Health 

 Centers for Disease Control and Prevention - Center for Global Health 

 Indian Health Service 

 Substance Abuse and Mental Health Services Administration - Center for Behavioral Health Statistics 

and Quality 

 Substance Abuse and Mental Health Services Administration - Office of Policy, Planning, and 

Innovation 

 Food and Drug Administration 

 Food and Drug Administration - Center for Drug Evaluation and Research 

 Administration for Community Living 

 Agency for Healthcare Research and Quality 

 Office of the Assistant Secretary for Planning and Evaluation 

 Office of Population Affairs 

 Office of Adolescent Health 

 Office of the National Coordinator 

 National Institutes of Health - NIH ODP Office of Dietary Supplements 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Govern... Page 8 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 31

[]

Which office in the Department of Homeland Security are you responding for?

*

Only answer this question if the following conditions are met:

Answer was 'Department of Homeland Security' at question '1 [agency]' (Which Federal agency are you responding 

for?)

Please choose only one of the following:

 U.S. Customs and Border Protection 

 Federal Emergency Management Agency 

 U.S. Immigration and Customs Enforcement 

 Transportation Security Administration 

 Citizenship and Immigration Services 

 National Protection and Programs Directorate 

 Science and Technology Directorate 

 Office of Policy 

 Office of Policy - Office of Immigration Statistics 

[]Which office in the Department of Housing and Urban Development do you 

represent? *

Only answer this question if the following conditions are met:

Answer was 'Department of Housing and Urban Development' at question '1 [agency]' (Which Federal agency are 

you responding for?)

Please choose only one of the following:

 Asst. Sec. for Public and Indian Housing Programs 

 Asst. Sec. for Housing Programs 

 Asst. Sec. for Community Planning and Development 

 Office of Lead Hazard Control and Healthy Homes 

 Policy Development and Research 

 Fair Housing and Equal Opportunity 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Govern... Page 9 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
32 The Promise of Evidence-Based Policymaking

[]

Which office in the Department of Justice are you responding for?

*

Only answer this question if the following conditions are met:

Answer was 'Department of Justice' at question '1 [agency]' (Which Federal agency are you responding for?)

Please choose only one of the following:

 Federal Bureau of Investigation - Criminal Justice Information Service 

 Federal Bureau of Investigation - Office of the director 

 Bureau of Prisons 

 Office of Justice Programs - Bureau of Justice Statistics 

 Office of Justice Programs - National Institute of Justice 

 Office of Justice Programs - Office of Juvenile Justice and Delinquency Prevention 

 Office of Justice Programs - Bureau of Justice Assistance 

 Office of Justice Programs - Office of Victims of Crime 

 Office of Justice Programs - Office of Violence Against Women 

 Office of Community Oriented Policing Services 

 Drug Enforcement Administration 

 Bureau of Alcohol, Tobacco, Firearms, and Explosives 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 10 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 33

[]

Which office in the Department of Labor are you responding for?

*

Only answer this question if the following conditions are met:

Answer was 'Department of Labor' at question '1 [agency]' (Which Federal agency are you responding for?)

Please choose only one of the following:

 Employment and Training Administration 

 Office of Workers' Compensation Programs 

 Occupational Safety and Health Administration 

 Bureau of Labor Statistics 

 Office of the Assistant Secretary for Policy - Chief Evaluation Office 

 Mine Safety and Health Administration 

 Wage and Hour Division 

 Employee Benefits Security Administration 

[]

Which office in the Department of State are you responding for?

*

Only answer this question if the following conditions are met:

Answer was 'Department of State' at question '1 [agency]' (Which Federal agency are you responding for?)

Please choose only one of the following:

 Evaluation and Aid Effectiveness Branch, Office of US Foreign Assistance Resources 

 Office of Policy and Planning 

 Office of the U.S. Global AIDS Coordinator 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 11 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
34 The Promise of Evidence-Based Policymaking

[]

Which office in the Department of the Interior are you responding for?

*

Only answer this question if the following conditions are met:

Answer was 'Department of the Interior' at question '1 [agency]' (Which Federal agency are you responding for?)

Please choose only one of the following:

 National Park Service 

 Bureau of Indian Affairs and Bureau of Indian Education 

 United States Fish and Wildlife Service 

 Bureau of Land Management 

 Bureau of Reclamation 

 United States Geological Survey 

 Office of Surface Mining Reclamation and Enforcement 

 Bureau of Safety and Environmental Enforcement 

 Bureau of Ocean Energy Management 

 Office of Natural Resources Revenue 

[]

Which office in the Department of the Treasury are you responding for?

*

Only answer this question if the following conditions are met:

Answer was 'Department of the Treasury' at question '1 [agency]' (Which Federal agency are you responding for?)

Please choose only one of the following:

 Internal Revenue Service - Research, Analysis, and Statistics 

 Internal Revenue Service - Statistics of Income Division 

 Bureau of Fiscal Service 

 Alcohol and Tobacco Tax and Trade Bureau 

 Financial Crimes Enforcement Network 

 Bureau of Engraving and Printing 

 United States Mint 

 Federal Financing Bank 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 12 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 35

[]

Which office in the Department of the Transportation are you responding for?

*

Only answer this question if the following conditions are met:

Answer was 'Department of Transportation' at question '1 [agency]' (Which Federal agency are you responding for?)

Please choose only one of the following:

 Federal Highway Administration 

 Federal Aviation Administration 

 Federal Transit Administration 

 Federal Railroad Administration 

 National Highway Traffic Safety Administration 

 Federal Motor Carrier Safety Administration 

 Maritime Administration 

 Pipeline and Hazardous Materials Safety Administration 

 Saint Lawrence Seaway Development Corporation 

 Bureau of Transportation Statistics 

 Surface Transportation Board 

[]

Which office in the Department of Veterans Affairs are you responding for?

*

Only answer this question if the following conditions are met:

Answer was 'Department of Veterans Affairs' at question '1 [agency]' (Which Federal agency are you responding 

for?)

Please choose only one of the following:

 Veterans Health Administration 

 National Cemetery Administration 

 National Center for Veterans Analysis and Statistics 

 Office of Policy and Planning 

 Veterans Benefits Administration 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 13 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
36 The Promise of Evidence-Based Policymaking

[]

Which office in the Environmental Protection Agency are you responding for?

*

Only answer this question if the following conditions are met:

Answer was 'Environmental Protection Agency' at question '1 [agency]' (Which Federal agency are you responding 

for?)

Please choose only one of the following:

 Office of Policy - Office of Strategic Environmental Management (OSEM) 

 Office of Policy - National Center for Environmental Economics 

 Office of Water 

 Office of Air and Radiation 

 Office of Land and Emergency Management 

 Office of Chemical Safety and Pollution Prevention 

 Office of Enforcement and Compliance Assurance 

 Office of Environmental Information 

[]

Which office in the National Science Foundation are you responding for?

*

Only answer this question if the following conditions are met:

Answer was 'National Science Foundation' at question '1 [agency]' (Which Federal agency are you responding for?)

Please choose only one of the following:

 Office of the Director 

 National Center for Science and Engineering Statistics 

 Office of Integrative Activities 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 14 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 37

[]

Which office in the Small Business Administration are you responding for?

*

Only answer this question if the following conditions are met:

Answer was 'Small Business Administration' at question '1 [agency]' (Which Federal agency are you responding 

for?)

Please choose only one of the following:

 Office of Performance Management 

 Office of Advocacy 

[]

Which office in the Social Security Administration are you responding for?

*

Only answer this question if the following conditions are met:

Answer was 'Social Security Administration' at question '1 [agency]' (Which Federal agency are you responding for?)

Please choose only one of the following:

 Office of Research, Demonstration, and Employment Support 

 Office of Research, Evaluation, and Statistics 

 Office of Retirement Policy 

 Office of Income Security Programs 

 Office of Data Exchange and Policy Publications 

 Office of Disability policy 

 Office of the Actuary 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 15 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
38 The Promise of Evidence-Based Policymaking

[]

Which office in the US Agency for International Development are you 

responding for?

*

Only answer this question if the following conditions are met:

Answer was 'Corporation for National and Community Service' at question '1 [agency]' (Which Federal agency are 

you responding for?)

Please choose only one of the following:

 Office of Learning, Evaluation and Research, Bureau for Policy, Planning and Learning 

[] In case we need to follow up with you, please provide your contact 

information: *

Please write your answer(s) here:

Name

E-mail Address

Phone Number

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 16 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 39

[]

For the remainder of the survey, please note the following additional 

definitions:

My/your agency: the Federal Agency or Department you marked in the first 

question 

Another agency: any Federal Agency or Department other than the one you 

marked in the first question 

My/your office: the Federal office you marked in the second question 

Another office in my/your agency:another Federal office within the Agency or 

Department you marked in the first question

Representatives: individuals who receive direction from, but are not directly 

employed by, your office or agency, including contractors, grantees, and 

those on interagency personnel agreements (IPAs)

Approximately what percentage of your office's total annual budget is for 

statistical, evaluation, research, or policy analysis purposes?

Please choose only one of the following:

 0 percent 

 1 to 10 percent 

 11 to 50 percent 

 51 to 90 percent 

 91 to 100 percent 

Statistical, evaluation, research, or policy analysis purposes: activities intended to support the production, 

analysis, and dissemination of information, including the creation and analysis of federal statistics; production or 

use of evaluation for federally-supported or -directed programs; research on a population, group of entities, 

policy, or program; or analysis of policies or proposals to inform decision making. These purposes do NOT 

include program operations (e.g., compliance monitoring, eligibility determinations, enforcement actions). 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 17 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
40 The Promise of Evidence-Based Policymaking

[]

Collecting Data

Does your office, or its representatives, collect data about individual units 

(e.g., people, businesses, nonprofit organizations, schools) either directly or 

through an intermediary source (e.g., state or local governments, private 

entities)?

*

Please choose only one of the following:

 Yes, routinely (as a regular part of our programs or processes) 

 Yes, occasionally (not as a regular part of our programs or processes) 

 No 

Individual unit: the lowest level unit at which data are collected, such as people, households, businesses, 

nonprofit organizations, schools. Intermediary source: any public or private organization that collects data 

from individual units and sends them to a Federal office, such as state or local government agencies, 

postsecondary institutions, grantees, or law enforcement agencies.

[] Approximately what percentage of your office's total annual budget is for 

collecting data from individual units? 

Only answer this question if the following conditions are met:

Answer was 'Yes, occasionally (not as a regular part of our programs or processes) ' or 'Yes, routinely (as a regular 

part of our programs or processes) ' at question '24 [collect3]' ( Collecting Data Does your office, or its 

representatives, collect data about individual units (e.g., people, businesses, nonprofit organizations, schools) either 

directly or through an intermediary source (e.g., state or local governments, private entities)? )

Please choose only one of the following:

 0 percent 

 1 to 10 percent 

 11 to 50 percent 

 51 to 90 percent 

 91 to 100 percent 

Individual unit: the lowest level unit at which data are collected, such as people, households, businesses, 

nonprofit organizations, schools.

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 18 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 41

[]For which of the following purposes does your office, or its 

representatives, collect data from individual units? *

Only answer this question if the following conditions are met:

Answer was 'Yes, occasionally (not as a regular part of our programs or processes) ' or 'Yes, routinely (as a regular 

part of our programs or processes) ' at question '24 [collect3]' ( Collecting Data Does your office, or its 

representatives, collect data about individual units (e.g., people, businesses, nonprofit organizations, schools) either 

directly or through an intermediary source (e.g., state or local governments, private entities)? )

Please choose all that apply:

 To produce statistics about the economy, society, or environment 

 To evaluate a program to understand its implementation or impact on participants 

 To support research on a population, group of entities, policy, or program 

 To conduct analysis of existing policies or programs to assist decision making 

 To conduct or support program operations 

 To monitor program or grant performance 

Other: 

Individual unit: the lowest level unit at which data are collected, such as people, households, businesses, 

nonprofit organizations, schools.

[]Are any of the data collected by your office, or its representatives, 

considered confidential? *

Only answer this question if the following conditions are met:

Answer was 'Yes, routinely (as a regular part of our programs or processes) ' or 'Yes, occasionally (not as a regular 

part of our programs or processes) ' at question '24 [collect3]' ( Collecting Data Does your office, or its 

representatives, collect data about individual units (e.g., people, businesses, nonprofit organizations, schools) either 

directly or through an intermediary source (e.g., state or local governments, private entities)? )

Please choose only one of the following:

 Yes 

 No 

Confidential data: identifiable micro-level data that cannot be made public due to law, policy, or practice. 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 19 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
42 The Promise of Evidence-Based Policymaking

[]Who analyzes your office's confidential data? 

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '27 [confidential6]' (Are any of the data collected by your office, or its representatives, 

considered confidential? )

Please choose all that apply:

 Staff physically present in my office 

 Staff working in a location other than my office (including teleworking) 

 Contractors physically present in my office 

 Contractors working in a location other than my office 

 Staff or contractors in another office in my agency 

 Staff or contractors in another agency 

 My office's grantees 

 Researchers outside of the Federal government 

Other: 

Confidential data: identifiable micro-level data that cannot be made public due to law, policy, or practice. 

[]

Analyzing Micro-Level Data From Another Agency

Does your office, or its representatives, analyze micro-level data collected by 

any other Federal office or agency?

*

Only answer this question if the following conditions are met:

Answer was 'No' or 'Yes, occasionally (not as a regular part of our programs or processes) ' or 'Yes, routinely (as a 

regular part of our programs or processes) ' at question '24 [collect3]' ( Collecting Data Does your office, or its 

representatives, collect data about individual units (e.g., people, businesses, nonprofit organizations, schools) either 

directly or through an intermediary source (e.g., state or local governments, private entities)? )

Please choose only one of the following:

 Yes 

 No 

Micro-level data: a data file with an entry (record) for each individual unit.

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 20 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 43

[]Which of the following kinds of micro-level data collected by any other 

office or agency does your office, or its representatives, analyze? 

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '29 [micro8]' (Analyzing Micro-Level Data From Another Agency Does your office, or 

its representatives, analyze micro-level data collected by any other Federal office or agency? )

Please choose all that apply:

 Criminal justice data 

 Decennial Census/American Community Survey (ACS) 

 Education data (Federal) 

 Education data (State or Private) 

 Federal data on business program participation 

 Federal establishment survey data (e.g., Economic and Agricultural Censuses) 

 Federal tax data 

 Federal social security data 

 Human services program data (e.g., SNAP and TANF) 

 Medicaid/Medicare claims and utilization data 

 Military service and veteran benefits records 

 State unemployment insurance wage record data 

 Vital records (birth and death) data 

 None of the above 

Other: 

Micro-level data: a data file with an entry (record) for each individual unit. 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 21 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
44 The Promise of Evidence-Based Policymaking

[]For which of the following purposes does your office, or its representatives, 

analyze micro-level data collected by any other Federal office or agency? *

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '29 [micro8]' (Analyzing Micro-Level Data From Another Agency Does your office, or 

its representatives, analyze micro-level data collected by any other Federal office or agency? )

Please choose all that apply:

 To produce statistics about the economy, society, or environment 

 To evaluate a program to understand its implementation or impact on participants 

 To support research on a population, group of entities, policy, or program 

 To conduct analysis of existing policies or programs to assist decision making 

 To conduct or support program operations 

 To monitor program or grant performance 

Other: 

Micro-level data: a data file with an entry (record) for each individual unit.

[]

Linking Data

Does your office, or its representatives, link the confidential data it collects 

with data from another agency? 

*

Only answer this question if the following conditions are met:

Answer was 'Yes, occasionally (not as a regular part of our programs or processes) ' or 'Yes, routinely (as a regular 

part of our programs or processes) ' at question '24 [collect3]' ( Collecting Data Does your office, or its 

representatives, collect data about individual units (e.g., people, businesses, nonprofit organizations, schools) either 

directly or through an intermediary source (e.g., state or local governments, private entities)? ) and Answer was 

'Yes' at question '27 [confidential6]' (Are any of the data collected by your office, or its representatives, 

considered confidential? )

Please choose only one of the following:

 Yes 

 No 

Confidential data: identifiable micro-level data that cannot be made public due to law, policy, or practice. 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 22 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 45

[]Which of the following kinds of data does your office, or its representatives, 

link with your confidential data collected from individual units? 

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '32 [link12]' (Linking Data Does your office, or its representatives, link the confidential 

data it collects with data from another agency? )

Please choose all that apply:

 Criminal justice data 

 Decennial Census/American Community Survey (ACS) 

 Education data (Federal) 

 Education data (State or Private) 

 Federal data on business program participation 

 Federal establishment survey data (e.g., Economic and Agricultural Censuses) 

 Federal tax data 

 Federal social security data 

 Human services program data (e.g., SNAP and TANF) 

 Medicaid/Medicare claims and utilization data 

 Military service and veteran benefits records 

 State unemployment insurance wage record data 

 Vital records (birth and death) data 

 None of the above 

Other: 

Confidential data: identifiable micro-level data that cannot be made public due to law, policy, or practice. 

Individual unit: the lowest level unit at which data are collected, such as people, households, businesses, 

nonprofit organizations, schools.

[]Which of the following best describes your process for linking to Criminal 

justice data? 

Only answer this question if the following conditions are met:

Answer was 'Criminal justice data' at question '33 [link13]' (Which of the following kinds of data does your office, or 

its representatives, link with your confidential data collected from individual units?)

Please choose all that apply:

 Our office, or its representatives, does the linkage. 

 Another office or agency, or its representatives, does the linkage. 

 A third party, or its representatives, does the linkage. 

Other: 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 23 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
46 The Promise of Evidence-Based Policymaking

[]

Which of the following best describes your process for linking to Decennial 

Census/American Community Survey (ACS) data?

Only answer this question if the following conditions are met:

Answer was 'Decennial Census/American Community Survey (ACS)' at question '33 [link13]' (Which of the following 

kinds of data does your office, or its representatives, link with your confidential data collected from individual units?)

Please choose all that apply:

 Our office, or its representatives, does the linkage. 

 Another office or agency, or its representatives, does the linkage. 

 A third party, or its representatives, does the linkage. 

Other: 

[]

Which of the following best describes your process for linking to Education 

data (Federal)?

Only answer this question if the following conditions are met:

Answer was 'Education data (Federal)' at question '33 [link13]' (Which of the following kinds of data does your office, 

or its representatives, link with your confidential data collected from individual units?)

Please choose all that apply:

 Our office, or its representatives, does the linkage. 

 Another office or agency, or its representatives, does the linkage. 

 A third party, or its representatives, does the linkage. 

Other: 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 24 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 47

[]

Which of the following best describes your process for linking to Education 

data (State)?

Only answer this question if the following conditions are met:

Answer was 'Education data (State or Private)' at question '33 [link13]' (Which of the following kinds of data does 

your office, or its representatives, link with your confidential data collected from individual units?)

Please choose all that apply:

 Our office, or its representatives, does the linkage. 

 Another office or agency, or its representatives, does the linkage. 

 A third party, or its representatives, does the linkage. 

Other: 

[]

Which of the following best describes your process for linking to Federal data 

on business program participation?

Only answer this question if the following conditions are met:

Answer was 'Federal data on business program participation' at question '33 [link13]' (Which of the following kinds of 

data does your office, or its representatives, link with your confidential data collected from individual units?)

Please choose all that apply:

 Our office, or its representatives, does the linkage. 

 Another office or agency, or its representatives, does the linkage. 

 A third party, or its representatives, does the linkage. 

Other: 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 25 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
48 The Promise of Evidence-Based Policymaking

[]

Which of the following best describes your process for linking to Federal 

establishment survey data (e.g. Economic and Agricultural Censuses)?

Only answer this question if the following conditions are met:

Answer was 'Federal establishment survey data (e.g., Economic and Agricultural Censuses)' at question '33 

[link13]' (Which of the following kinds of data does your office, or its representatives, link with your confidential data 

collected from individual units?)

Please choose all that apply:

 Our office, or its representatives, does the linkage. 

 Another office or agency, or its representatives, does the linkage. 

 A third party, or its representatives, does the linkage. 

Other: 

[]

Which of the following best describes your process for linking to Federal tax 

data?

Only answer this question if the following conditions are met:

Answer was 'Federal tax data' at question '33 [link13]' (Which of the following kinds of data does your office, or its 

representatives, link with your confidential data collected from individual units?)

Please choose all that apply:

 Our office, or its representatives, does the linkage. 

 Another office or agency, or its representatives, does the linkage. 

 A third party, or its representatives, does the linkage. 

Other: 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 26 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 49

[]

Which of the following best describes your process for linking to Federal 

social security data?

Only answer this question if the following conditions are met:

Answer was 'Federal social security data' at question '33 [link13]' (Which of the following kinds of data does your 

office, or its representatives, link with your confidential data collected from individual units?)

Please choose all that apply:

 Our office, or its representatives, does the linkage. 

 Another office or agency, or its representatives, does the linkage. 

 A third party, or its representatives, does the linkage. 

Other: 

[]

Which of the following best describes your process for linking to Human 

services program data (e.g. SNAP and TANF)?

Only answer this question if the following conditions are met:

Answer was 'Human services program data (e.g., SNAP and TANF)' at question '33 [link13]' (Which of the following 

kinds of data does your office, or its representatives, link with your confidential data collected from individual units?)

Please choose all that apply:

 Our office, or its representatives, does the linkage. 

 Another office or agency, or its representatives, does the linkage. 

 A third party, or its representatives, does the linkage. 

Other: 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 27 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
50 The Promise of Evidence-Based Policymaking

[]

Which of the following best describes your process for linking to 

Medicaid/Medicare claims and utilization data?

Only answer this question if the following conditions are met:

Answer was 'Medicaid/Medicare claims and utilization data' at question '33 [link13]' (Which of the following kinds of 

data does your office, or its representatives, link with your confidential data collected from individual units?)

Please choose all that apply:

 Our office, or its representatives, does the linkage. 

 Another office or agency, or its representatives, does the linkage. 

 A third party, or its representatives, does the linkage. 

Other: 

[]

Which of the following best describes your process for linking to Military 

service and veteran benefits records?

Only answer this question if the following conditions are met:

Answer was 'Military service and veteran benefits records' at question '33 [link13]' (Which of the following kinds of 

data does your office, or its representatives, link with your confidential data collected from individual units?)

Please choose all that apply:

 Our office, or its representatives, does the linkage. 

 Another office or agency, or its representatives, does the linkage. 

 A third party, or its representatives, does the linkage. 

Other: 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 28 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 51

[]

Which of the following best describes your process for linking to State 

unemployment insurance wage record data?

Only answer this question if the following conditions are met:

Answer was 'State unemployment insurance wage record data' at question '33 [link13]' (Which of the following kinds 

of data does your office, or its representatives, link with your confidential data collected from individual units?)

Please choose all that apply:

 Our office, or its representatives, does the linkage. 

 Another office or agency, or its representatives, does the linkage. 

 A third party, or its representatives, does the linkage. 

Other: 

[]

Which of the following best describes your process for linking to Vital records 

(birth and death) data?

Only answer this question if the following conditions are met:

Answer was 'Vital records (birth and death) data' at question '33 [link13]' (Which of the following kinds of data does 

your office, or its representatives, link with your confidential data collected from individual units?)

Please choose all that apply:

 Our office, or its representatives, does the linkage. 

 Another office or agency, or its representatives, does the linkage. 

 A third party, or its representatives, does the linkage. 

Other: 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 29 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
52 The Promise of Evidence-Based Policymaking

[]Does your office have staff or contractors whose primary job duties involve 

the technical process of data linkage? 

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '32 [link12]' (Linking Data Does your office, or its representatives, link the confidential 

data it collects with data from another agency? )

Please choose only one of the following:

 Yes 

 No 

Primary job duties: duties performed more than 50% of full-time equivalent work hours. Data linkage: the 

merging of two or more micro-level datasets originally collected for different purposes.

[]How many staff or contractors does your office have whose primary job 

duties involve the technical process of data linkage? 

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '47 [linkstaff18]' (Does your office have staff or contractors whose primary job duties 

involve the technical process of data linkage?)

Please choose only one of the following:

 1 to 2 

 3 to 5 

 6 to 10 

 More than 10 

Primary job duties: duties performed more than 50% of full-time equivalent work hours. Data linkage: the 

merging of two or more micro-level datasets originally collected for different purposes.

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 30 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 53

[]Which of the following statements best describes the regularity of data 

linkage in your office? 

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '32 [link12]' (Linking Data Does your office, or its representatives, link the confidential 

data it collects with data from another agency? )

Please choose only one of the following:

 All of our data linkages are for ad hoc projects. 

 Most of our data linkages are for ad hoc projects; a few are a part of our regular processes. 

 Our data linkages are about evenly split between ad hoc projects and regular processes. 

 Most of our data linkages are a part of our regular process; a few are ad hoc projects 

 All of our data linkages are part of our regular processes. 

Data linkage: the merging of two or more micro-level datasets originally collected for different purposes.

[]How does your office pay for data linkage? 

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '32 [link12]' (Linking Data Does your office, or its representatives, link the confidential 

data it collects with data from another agency? )

Please choose all that apply:

 My office has a specific budget allocation that covers data linkage. 

 My office chooses to set aside a portion of the budget to help pay for data linkage. 

 Programs within my office that do data linkage absorb the cost in their budgets. 

 Other offices provide funds for data linkages. 

 User fees help pay for data linkage. 

Other: 

Data linkage: the merging of two or more micro-level datasets originally collected for different purposes.

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 31 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
54 The Promise of Evidence-Based Policymaking

[]Which of the following does your office, or its representatives, use for data 

linkage?

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '32 [link12]' (Linking Data Does your office, or its representatives, link the confidential 

data it collects with data from another agency? )

Please choose all that apply:

 A probabilistic match of key characteristics. 

 A linking key (personal identifier). 

 A combination of probabilistic matching and linking keys. 

 Don't know 

Other: 

Data linkage: the merging of two or more micro-level datasets originally collected for different purposes.

[]Which of the following keys does your office, or its representatives, use for 

data linkage? 

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '32 [link12]' (Linking Data Does your office, or its representatives, link the confidential 

data it collects with data from another agency? )

Please choose all that apply:

 Social Security Number (SSN) 

 Masked SSN 

 Employer Identification Number (EIN) 

 Name 

 Address 

 Date of Birth 

 Biometrics (e.g., fingerprints) 

 Don't know 

Other: 

Data linkage:the merging of two or more micro-level datasets originally collected for different purposes.

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 32 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 55

[]How does your office, or its representatives, secure data linking keys? 

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '32 [link12]' (Linking Data Does your office, or its representatives, link the confidential 

data it collects with data from another agency? )

Please choose all that apply:

 By restricting access to the linkage key or linkage key crosswalk. 

 By generating a pseudo-anonymous key (e.g., hash, unique ID, etc) for linking. 

 By generating a different pseudo-anonymous key (e.g., hash, unique ID, etc) for each linkage or for 

each data source. 

 By removing the linkage key from the file(s) once linkage is complete. 

 Don't know 

Other: 

[]Does your office, or its representatives, keep micro-level data that are 

linked for statistical, research, evaluation, or policy analysis purposes so they 

can be analyzed again at a later time? 

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '32 [link12]' (Linking Data Does your office, or its representatives, link the confidential 

data it collects with data from another agency? )

Please choose only one of the following:

 Yes, routinely (as a regular part of our programs or processes) 

 Yes, occasionally (not as a regular part of our programs or processes) 

 No 

Micro-level data: a data file with an entry (record) for each individual unit. Statistical, evaluation, research, 

or policy analysis purposes: activities intended to support the production, analysis, and dissemination of 

information, including the creation and analysis of federal statistics; production or use of evaluation for federallysupported or -directed programs; research on a population, group of entities, policy, or program; or analysis of 

policies or proposals to inform decision making. These purposes do NOT include program operations (e.g., 

compliance monitoring, eligibility determinations, enforcement actions). 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 33 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
56 The Promise of Evidence-Based Policymaking

[]

Accessing Data

Does another agency, or its representatives,ever access your office's 

confidential data for its own statistical, evaluation, research, or policy 

analysis purposes?

*

Only answer this question if the following conditions are met:

Answer was 'Yes, occasionally (not as a regular part of our programs or processes) ' or 'Yes, routinely (as a regular 

part of our programs or processes) ' at question '24 [collect3]' ( Collecting Data Does your office, or its 

representatives, collect data about individual units (e.g., people, businesses, nonprofit organizations, schools) either 

directly or through an intermediary source (e.g., state or local governments, private entities)? ) and Answer was 

'Yes' at question '27 [confidential6]' (Are any of the data collected by your office, or its representatives, 

considered confidential? )

Please choose only one of the following:

 Yes 

 No 

Confidential data: identifiable micro-level data that cannot be made public due to law, policy, or practice. 

Statistical, evaluation, research, or policy analysis purposes: activities intended to support the production, 

analysis, and dissemination of information, including the creation and analysis of federal statistics; production or 

use of evaluation for federally-supported or -directed programs; research on a population, group of entities, 

policy, or program; or analysis of policies or proposals to inform decision making. These purposes do NOT 

include program operations (e.g., compliance monitoring, eligibility determinations, enforcement actions). 

[]How does another agency, or its representatives, access your office's 

confidential data? 

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '55 [access27]' (Accessing Data Does another agency, or its representatives, ever 

access your office's confidential data for its own statistical, evaluation, research, or policy analysis purposes? )

Please choose all that apply:

 Staff or contractors come onsite to my office or agency to access the data. 

 Through a Federal Statistical Research Data Center (FSRDC) 

 Through a distributed thin client Citrix or VPN network 

 Through a Memorandum of Understanding (MOU), license, or contract agreement 

 Through a data archive (e.g., Interuniversity Consortium for Political and Social Research) 

Other: 

Confidential data: identifiable micro-level data that cannot be made public due to law, policy, or practice. 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 34 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 57

[]Approximately how many other agencies, or their representatives, access 

your office's confidential data each year?

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '55 [access27]' (Accessing Data Does another agency, or its representatives, ever 

access your office's confidential data for its own statistical, evaluation, research, or policy analysis purposes? )

Please choose only one of the following:

 1 to 2 

 3 to 5 

 6 to 10 

 More than 10 

Confidential data: identifiable micro-level data that cannot be made public due to law, policy, or practice. 

[]Which of the following statements best describes how regularly another 

agency, or its representatives, access your office's confidential data? 

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '55 [access27]' (Accessing Data Does another agency, or its representatives, ever 

access your office's confidential data for its own statistical, evaluation, research, or policy analysis purposes? )

Please choose only one of the following:

 Many other agencies routinely (e.g., annually, quarterly, monthly) access my office's confidential 

data. 

 A few other agencies routinely (e.g., annually, quarterly, monthly) access my office's confidential 

data. 

 Many other agencies occasionally (i.e. on an ad hoc basis) access my office's confidential data. 

 A few other agencies occasionally (i.e. on an ad hoc basis) access my office's confidential data. 

 Some agencies routinely (e.g., annually, quarterly, monthly) and some agencies occasionally (i.e. on 

an ad hoc basis) access my office's confidential data. 

Confidential data: identifiable micro-level data that cannot be made public due to law, policy, or practice. 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 35 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
58 The Promise of Evidence-Based Policymaking

[]Do researchers outside of the Federal government ever access your office's 

confidential data? *

Only answer this question if the following conditions are met:

Answer was 'Yes, occasionally (not as a regular part of our programs or processes) ' or 'Yes, routinely (as a regular 

part of our programs or processes) ' at question '24 [collect3]' ( Collecting Data Does your office, or its 

representatives, collect data about individual units (e.g., people, businesses, nonprofit organizations, schools) either 

directly or through an intermediary source (e.g., state or local governments, private entities)? ) and Answer was 

'Yes' at question '27 [confidential6]' (Are any of the data collected by your office, or its representatives, 

considered confidential? )

Please choose only one of the following:

 Yes 

 No 

Confidential data: identifiable micro-level data that cannot be made public due to law, policy, or practice. 

[]Approximately how many outside researchers access your office's 

confidential dataeach year?

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '59 [outside30]' (Do researchers outside of the Federal government ever access your 

office's confidential data?)

Please choose only one of the following:

 1 to 10 

 11 to 50 

 51 to 100 

 More than 100 

Confidential data: micro-level data deemed sensitive because it could potentially identify individual units.

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 36 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 59

[]How do researchers outside of the Federal government access your office's

confidential data? 

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '59 [outside30]' (Do researchers outside of the Federal government ever access your 

office's confidential data?)

Please choose all that apply:

 The researcher comes onsite to my office or agency to access the data. 

 Through a Federal Statistical Research Data Center (FSRDC) 

 Through a distributed thin client Citrix or VPN network 

 Through a license or contract agreement with researchers and their institutions 

 Through a data archive (e.g., Interuniversity Consortium for Political and Social Research) 

Other: 

Confidential data: identifiable micro-level data that cannot be made public due to law, policy, or practice. 

[]Does your office receive and review requests from outside researchers for 

access to confidential data? *

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '59 [outside30]' (Do researchers outside of the Federal government ever access your 

office's confidential data?)

Please choose only one of the following:

 Yes 

 No 

Confidential data: identifiable micro-level data that cannot be made public due to law, policy, or practice. 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 37 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
60 The Promise of Evidence-Based Policymaking

[]How often does your office receive new requests from outside researchers 

for access to confidential data? 

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '62 [review32]' (Does your office receive and review requests from outside researchers 

for access to confidential data?)

Please choose only one of the following:

 Daily 

 Weekly 

 Monthly 

 Yearly 

 Sporadically 

Confidential data: identifiable micro-level data that cannot be made public due to law, policy, or practice. 

[]Which of the following statements best describes how regularly outside 

researchers access your office's confidential data? 

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '59 [outside30]' (Do researchers outside of the Federal government ever access your 

office's confidential data?)

Please choose only one of the following:

 Many outside researchers routinely (e.g. annually, quarterly, monthly) access my office's confidential 

data. 

 A few outside reseachers routinely (e.g. annually, quarterly, monthly) access my office's confidential 

data. 

 Many outside researchers occasionally (i.e. on an ad hoc basis) access my office's confidential data. 

 A few outside researchers occasionally (i.e. on an ad hoc basis) access my office's confidential data. 

 Some outside researchers routinely (e.g. annually, quarterly, monthly) and some outside researchers 

occasionally (i.e. on an ad hoc basis) access my office's confidential data. 

Confidential data: identifiable micro-level data that cannot be made public due to law, policy, or practice. 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 38 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 61

[]Does your office have staff or contractors whose primary job duties involve 

reviewing and processing outside researcher requests for access to 

confidential data? 

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '62 [review32]' (Does your office receive and review requests from outside researchers 

for access to confidential data?)

Please choose only one of the following:

 Yes 

 No 

Primary job duties: duties performed more than 50% of full-time equivalent work hours. Confidential data:

identifiable micro-level data that cannot be made public due to law, policy, or practice. 

[]How many staff or contractors does your office have whose primary job 

duties involve reviewing and processing outside researcher requests for 

access to confidential data? 

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '65 [prim35]' (Does your office have staff or contractors whose primary job duties 

involve reviewing and processing outside researcher requests for access to confidential data?)

Please choose only one of the following:

 1 to 2 

 3 to 5 

 6 to 10 

 More than 10 

Primary job duties: duties performed more than 50% of full-time equivalent work hours. Confidential data:

identifiable micro-level data that cannot be made public due to law, policy, or practice. 

[]Does your office have a system for tracking requests from outside 

researchers for access to confidential data? 

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '62 [review32]' (Does your office receive and review requests from outside researchers 

for access to confidential data?)

Please choose only one of the following:

 Yes 

 No 

Confidential data: identifiable micro-level data that cannot be made public due to law, policy, or practice. 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 39 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
62 The Promise of Evidence-Based Policymaking

[]Does your office have a process for reviewing output produced by outside 

researchers to prevent disclosure of the identities of individual units? *

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '62 [review32]' (Does your office receive and review requests from outside researchers 

for access to confidential data?)

Please choose only one of the following:

 Yes 

 No 

Individual unit: the lowest level unit at which data are collected, such as people, households, businesses, 

nonprofit organizations, schools.

[]How often does your office receive new requests for disclosure review of 

outside researcher analyses? 

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '68 [outrev37]' (Does your office have a process for reviewing output produced by 

outside researchers to prevent disclosure of the identities of individual units?)

Please choose only one of the following:

 Daily 

 Weekly 

 Monthly 

 Yearly 

 Sporadically 

[]Does your office have a process for documenting the results of disclosure 

review of outside researcher analyses? 

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '68 [outrev37]' (Does your office have a process for reviewing output produced by 

outside researchers to prevent disclosure of the identities of individual units?)

Please choose only one of the following:

 Yes 

 No 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 40 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 63

[]Does your office have staff or contractors whose primary job duties involve 

disclosure review of outside researcher analyses? 

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '68 [outrev37]' (Does your office have a process for reviewing output produced by 

outside researchers to prevent disclosure of the identities of individual units?)

Please choose only one of the following:

 Yes 

 No 

Primary job duties: duties performed more than 50% of full-time equivalent work hours.

[]How many staff or contractors does your office have whose primary job 

duties involve disclosure review of outside researcher analyses? 

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '71 [staff39]' (Does your office have staff or contractors whose primary job duties 

involve disclosure review of outside researcher analyses?)

Please choose only one of the following:

 1 to 2 

 3 to 5 

 6 to 10 

 More than 10 

 Other 

Primary job duties: duties performed more than 50% of full-time equivalent work hours. 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 41 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
64 The Promise of Evidence-Based Policymaking

[]Does your office, or its representatives, typically create metadata or 

technical documentation for the data it produces that includes the following? 

Only answer this question if the following conditions are met:

Answer was 'Yes, occasionally (not as a regular part of our programs or processes) ' or 'Yes, routinely (as a regular 

part of our programs or processes) ' at question '24 [collect3]' ( Collecting Data Does your office, or its 

representatives, collect data about individual units (e.g., people, businesses, nonprofit organizations, schools) either 

directly or through an intermediary source (e.g., state or local governments, private entities)? )

Please choose the appropriate response for each item:

Yes No

Variable definitions, 

units of 

measurement, and 

response ranges

Changes to 

definitions, scope, 

or data collection 

processes over time 

Imputation 

procedures or other 

edits to the data

[]

Managing Historical Data

Does your office, or its representatives, have a process for archiving, 

managing, and retrieving your historical micro-level data?

*

Only answer this question if the following conditions are met:

Answer was 'Yes, occasionally (not as a regular part of our programs or processes) ' or 'Yes, routinely (as a regular 

part of our programs or processes) ' at question '24 [collect3]' ( Collecting Data Does your office, or its 

representatives, collect data about individual units (e.g., people, businesses, nonprofit organizations, schools) either 

directly or through an intermediary source (e.g., state or local governments, private entities)? )

Please choose only one of the following:

 Yes 

 No 

Micro-level data: a data file with an entry (record) for each individual unit.

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 42 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 65

[]Does your office have staff or contractors whose primary job duties 

involve archiving, managing, and retrieving your historical micro-level data? 

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '74 [arch41]' (Managing Historical Data Does your office, or its representatives, have a 

process for archiving, managing, and retrieving your historical micro-level data? )

Please choose only one of the following:

 Yes 

 No 

Primary job duties: duties performed more than 50% of full-time equivalent work hours. Micro-level data: a 

data file with an entry (record) for each individual unit.

[]How many staff or contractors does your office have whose primary job 

duties involve archiving, managing, and retrieving your historical micro-level 

data? 

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '75 [hist42]' (Does your office have staff or contractors whose primary job duties 

involve archiving, managing, and retrieving your historical micro-level data? )

Please choose only one of the following:

 1 to 2 

 3 to 5 

 6 to 10 

 More than 10 

Micro-level data: a data file with an entry (record) for each individual unit. Primary job duties: duties 

performed more than 50% of full-time equivalent work hours.

[] Does your office, or its representatives, continue to manage or clean 

your historical micro-level data?  

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '74 [arch41]' (Managing Historical Data Does your office, or its representatives, have a 

process for archiving, managing, and retrieving your historical micro-level data? )

Please choose only one of the following:

 Yes 

 No 

Micro-level data: a data file with an entry (record) for each individual unit. 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 43 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
66 The Promise of Evidence-Based Policymaking

[]How easy or difficult is it for your office, or its representatives, to retrieve 

and use your historical micro-level data? 

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '74 [arch41]' (Managing Historical Data Does your office, or its representatives, have a 

process for archiving, managing, and retrieving your historical micro-level data? )

Please choose only one of the following:

 very easy 

 somewhat easy 

 somewhat difficult 

 very difficult 

Micro-level data: a data file with an entry (record) for each individual unit.

[]When does your office, or its representatives, destroy your historical microlevel data? 

Only answer this question if the following conditions are met:

Answer was 'Yes' at question '74 [arch41]' (Managing Historical Data Does your office, or its representatives, have a 

process for archiving, managing, and retrieving your historical micro-level data? )

Please choose all that apply:

 We archive all historical micro-level data. 

 We destroy historical micro-level data when they are no longer being used for the purpose for which 

they were collected. 

 We destroy historical micro-level data on a schedule established by statute, regulation, or policy. 

 We destroy historical micro-level data when they are no longer relevant for research or policy 

analysis. 

 We destroy historical micro-level data when we can no longer support their antiquated data structure. 

Other: 

Micro-level data: a data file with an entry (record) for each individual unit.

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 44 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 67

[]

Using Data for Statistical, Evaluation, Research, or Policy Analysis Purposes

Which of the following kinds ofdata has your office, or its 

representatives, considered using for statistical, evaluation, research, or 

policy analysis purposes but did not because it was too hard to get access to 

the specific dataset you needed? 

Only answer this question if the following conditions are met:

-------- Scenario 1 --------

Answer was 'To evaluate a program to understand its implementation or impact on participants ' or 'To produce 

statistics about the economy, society, or environment ' or 'To support research on a population, group of entities, 

policy, or program' or 'To conduct analysis of existing policies or programs to assist decision making ' at question '26 

[purpose5]' (For which of the following purposes does your office, or its representatives, collect data from individual 

units? )

-------- or Scenario 2 --------

Answer was 'To produce statistics about the economy, society, or environment ' or 'To evaluate a program to 

understand its implementation or impact on participants ' or 'To support research on a population, group of entities, 

policy, or program' or 'To conduct analysis of existing policies or programs to assist decision making ' at question '31 

[micropurpose10]' (For which of the following purposes does your office, or its representatives, analyze micro-level 

data collected by any other Federal office or agency?)

-------- or Scenario 3 --------

Answer was '91 to 100 percent ' or '51 to 90 percent' or '11 to 50 percent' or '1 to 10 percent' at question '23 

[SERA46]' ( For the remainder of the survey, please note the following additional definitions: My/your agency: the 

Federal Agency or Department you marked in the first question Another agency: any Federal Agency or 

Department other than the one you marked in the first question My/your office: the Federal office you marked in the 

second question Another office in my/your agency: another Federal office within the Agency or Department you 

marked in the first question Representatives: individuals who receive direction from, but are not directly employed 

by, your office or agency, including contractors, grantees, and those on interagency personnel agreements (IPAs) 

Approximately what percentage of your office's total annual budget is for statistical, evaluation, research, or policy 

analysis purposes? )

Please choose all that apply:

 Criminal justice data 

 Decennial Census/American Community Survey (ACS) 

 Education data (Federal) 

 Education data (State or Private) 

 Federal data on business program participation 

 Federal establishment survey data (e.g., Economic and Agricultural Censuses) 

 Federal tax data 

 Federal social security data 

 Human services program data (e.g., SNAP and TANF) 

 Medicaid/Medicare claims and utilization data 

 Military service and veteran benefits records 

 State unemployment insurance wage record data 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 45 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
68 The Promise of Evidence-Based Policymaking

 Vital records (birth and death) data 

 None of the above 

Statistical, evaluation, research, or policy analysis purposes: activities intended to support the production, 

analysis, and dissemination of information, including the creation and analysis of federal statistics; production or 

use of evaluation for federally-supported or -directed programs; research on a population, group of entities, 

policy, or program; or analysis of policies or proposals to inform decision making. These purposes do NOT 

include program operations (e.g., compliance monitoring, eligibility determinations, enforcement actions).

[]Which of the following factors made it hard for your office to get access to 

Criminal justice data? *

Only answer this question if the following conditions are met:

Answer was 'Criminal justice data' at question '80 [notlink14]' (Using Data for Statistical, Evaluation, Research, or 

Policy Analysis Purposes Which of the following kinds of data has your office, or its representatives, considered 

using for statistical, evaluation, research, or policy analysis purposes but did not because it was too hard to get 

access to the specific dataset you needed? )

Please choose all that apply:

 Lack of staff, policies, and procedures to establish data sharing agreements 

 Statutes prohibiting data sharing 

 Regulations or policies that make it difficult to link data 

 Lack of capacity in information technology systems 

 Lack of staff or contractors with specific technical expertise needed to link data 

 Concerns about keeping the linked data secure and confidential 

Other: 

[]

Was the lack of staff, policies, and procedures to establish data sharing 

agreements: 

Only answer this question if the following conditions are met:

Answer was 'Lack of staff, policies, and procedures to establish data sharing agreements ' at question '81 

[cjac15]' (Which of the following factors made it hard for your office to get access to Criminal justice data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 46 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 69

[]

Were the statutes prohibiting data sharing: 

Only answer this question if the following conditions are met:

Answer was 'Statutes prohibiting data sharing ' at question '81 [cjac15]' (Which of the following factors made it hard 

for your office to get access to Criminal justice data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

[]

Were the regulations or policies that make it difficult to link data:

Only answer this question if the following conditions are met:

Answer was 'Regulations or policies that make it difficult to link data ' at question '81 [cjac15]' (Which of the following 

factors made it hard for your office to get access to Criminal justice data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

[]

Was the lack of capacity in information technology systems:

Only answer this question if the following conditions are met:

Answer was 'Lack of capacity in information technology systems ' at question '81 [cjac15]' (Which of the following 

factors made it hard for your office to get access to Criminal justice data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 47 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
70 The Promise of Evidence-Based Policymaking

[]

Was the lack of staff or contractors with specific technical expertise needed 

to link data:

Only answer this question if the following conditions are met:

Answer was 'Lack of staff or contractors with specific technical expertise needed to link data ' at question '81 

[cjac15]' (Which of the following factors made it hard for your office to get access to Criminal justice data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

[]

Were the concerns about keeping the linked data secure and confidential:

Only answer this question if the following conditions are met:

Answer was 'Concerns about keeping the linked data secure and confidential ' at question '81 [cjac15]' (Which of the 

following factors made it hard for your office to get access to Criminal justice data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 48 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 71

[]Which of the following factors made it hard for your office to get access to 

Decennial Census/American Community Survey (ACS) data? 

Only answer this question if the following conditions are met:

Answer was 'Decennial Census/American Community Survey (ACS)' at question '80 [notlink14]' (Using Data for 

Statistical, Evaluation, Research, or Policy Analysis Purposes Which of the following kinds of data has your office, 

or its representatives, considered using for statistical, evaluation, research, or policy analysis purposes but did not 

because it was too hard to get access to the specific dataset you needed? )

Please choose all that apply:

 Lack of staff, policies, and procedures to establish data sharing agreements 

 Statutes prohibiting data sharing 

 Regulations or policies that make it difficult to link data 

 Lack of capacity in information technology systems 

 Lack of staff or contractors with specific technical expertise needed to link data 

 Concerns about keeping the linked data secure and confidential 

Other: 

[]

Was the lack of staff, policies, and procedures to establish data sharing 

agreements: 

Only answer this question if the following conditions are met:

Answer was 'Lack of staff, policies, and procedures to establish data sharing agreements ' at question '88 

[decac15]' (Which of the following factors made it hard for your office to get access to Decennial Census/American 

Community Survey (ACS) data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 49 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
72 The Promise of Evidence-Based Policymaking

[]

Were the statutes prohibiting data sharing: 

Only answer this question if the following conditions are met:

Answer was 'Statutes prohibiting data sharing ' at question '88 [decac15]' (Which of the following factors made it 

hard for your office to get access to Decennial Census/American Community Survey (ACS) data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

[]

Were the regulations or policies that make it difficult to link data:

Only answer this question if the following conditions are met:

Answer was 'Regulations or policies that make it difficult to link data ' at question '88 [decac15]' (Which of the 

following factors made it hard for your office to get access to Decennial Census/American Community Survey (ACS) 

data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

[]

Was the lack of capacity in information technology systems:

Only answer this question if the following conditions are met:

Answer was 'Lack of capacity in information technology systems ' at question '88 [decac15]' (Which of the following 

factors made it hard for your office to get access to Decennial Census/American Community Survey (ACS) data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 50 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 73

[]

Was the lack of staff or contractors with specific technical expertise needed 

to link data:

Only answer this question if the following conditions are met:

Answer was 'Lack of staff or contractors with specific technical expertise needed to link data ' at question '88 

[decac15]' (Which of the following factors made it hard for your office to get access to Decennial Census/American 

Community Survey (ACS) data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

[]

Were the concerns about keeping the linked data secure and confidential:

Only answer this question if the following conditions are met:

Answer was 'Concerns about keeping the linked data secure and confidential ' at question '88 [decac15]' (Which of 

the following factors made it hard for your office to get access to Decennial Census/American Community Survey 

(ACS) data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 51 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
74 The Promise of Evidence-Based Policymaking

[]Which of the following factors made it hard for your office to get access to 

Education data (Federal)? 

Only answer this question if the following conditions are met:

Answer was 'Education data (Federal)' at question '80 [notlink14]' (Using Data for Statistical, Evaluation, Research, 

or Policy Analysis Purposes Which of the following kinds of data has your office, or its representatives, considered 

using for statistical, evaluation, research, or policy analysis purposes but did not because it was too hard to get 

access to the specific dataset you needed? )

Please choose all that apply:

 Lack of staff, policies, and procedures to establish data sharing agreements 

 Statutes prohibiting data sharing 

 Regulations or policies that make it difficult to link data 

 Lack of capacity in information technology systems 

 Lack of staff or contractors with specific technical expertise needed to link data 

 Concerns about keeping the linked data secure and confidential 

Other: 

[]

Was the lack of staff, policies, and procedures to establish data sharing 

agreements: 

Only answer this question if the following conditions are met:

Answer was 'Lack of staff, policies, and procedures to establish data sharing agreements ' at question '95 

[fededac15]' (Which of the following factors made it hard for your office to get access to Education data (Federal)?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 52 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 75

[]

Were the statutes prohibiting data sharing: 

Only answer this question if the following conditions are met:

Answer was 'Statutes prohibiting data sharing ' at question '95 [fededac15]' (Which of the following factors made it 

hard for your office to get access to Education data (Federal)?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

[]

Were the regulations or policies that make it difficult to link data:

Only answer this question if the following conditions are met:

Answer was 'Regulations or policies that make it difficult to link data ' at question '95 [fededac15]' (Which of the 

following factors made it hard for your office to get access to Education data (Federal)?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

[]

Was the lack of capacity in information technology systems:

Only answer this question if the following conditions are met:

Answer was 'Lack of capacity in information technology systems ' at question '95 [fededac15]' (Which of the 

following factors made it hard for your office to get access to Education data (Federal)?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 53 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
76 The Promise of Evidence-Based Policymaking

[]

Was the lack of staff or contractors with specific technical expertise needed 

to link data:

Only answer this question if the following conditions are met:

Answer was 'Lack of staff or contractors with specific technical expertise needed to link data ' at question '95 

[fededac15]' (Which of the following factors made it hard for your office to get access to Education data (Federal)?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

[]

Were the concerns about keeping the linked data secure and confidential:

Only answer this question if the following conditions are met:

Answer was 'Concerns about keeping the linked data secure and confidential ' at question '95 [fededac15]' (Which of 

the following factors made it hard for your office to get access to Education data (Federal)?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 54 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 77

[]Which of the following factors made it hard for your office to get access to 

Education data (State)? 

Only answer this question if the following conditions are met:

Answer was 'Education data (State or Private)' at question '80 [notlink14]' (Using Data for Statistical, Evaluation, 

Research, or Policy Analysis Purposes Which of the following kinds of data has your office, or its 

representatives, considered using for statistical, evaluation, research, or policy analysis purposes but did not 

because it was too hard to get access to the specific dataset you needed? )

Please choose all that apply:

 Lack of staff, policies, and procedures to establish data sharing agreements 

 Statutes prohibiting data sharing 

 Regulations or policies that make it difficult to link data 

 Lack of capacity in information technology systems 

 Lack of staff or contractors with specific technical expertise needed to link data 

 Concerns about keeping the linked data secure and confidential 

Other: 

[]

Was the lack of staff, policies, and procedures to establish data sharing 

agreements: 

Only answer this question if the following conditions are met:

Answer was 'Lack of staff, policies, and procedures to establish data sharing agreements ' at question '102 

[stedac15]' (Which of the following factors made it hard for your office to get access to Education data (State)?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

Other: 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 55 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
78 The Promise of Evidence-Based Policymaking

[]

Were the statutes prohibiting data sharing: 

Only answer this question if the following conditions are met:

Answer was 'Statutes prohibiting data sharing ' at question '102 [stedac15]' (Which of the following factors made it 

hard for your office to get access to Education data (State)?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

Other: 

[]

Were the regulations or policies that make it difficult to link data:

Only answer this question if the following conditions are met:

Answer was 'Regulations or policies that make it difficult to link data ' at question '102 [stedac15]' (Which of the 

following factors made it hard for your office to get access to Education data (State)?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

Other: 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 56 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 79

[]

Was the lack of capacity in information technology systems:

Only answer this question if the following conditions are met:

Answer was 'Lack of capacity in information technology systems ' at question '102 [stedac15]' (Which of the 

following factors made it hard for your office to get access to Education data (State)?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

Other: 

[]

Was the lack of staff or contractors with specific technical expertise needed 

to link data:

Only answer this question if the following conditions are met:

Answer was 'Lack of staff or contractors with specific technical expertise needed to link data ' at question '102 

[stedac15]' (Which of the following factors made it hard for your office to get access to Education data (State)?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

Other: 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 57 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
80 The Promise of Evidence-Based Policymaking

[]

Were the concerns about keeping the linked data secure and confidential:

Only answer this question if the following conditions are met:

Answer was 'Concerns about keeping the linked data secure and confidential ' at question '102 [stedac15]' (Which of 

the following factors made it hard for your office to get access to Education data (State)?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

Other: 

[]Which of the following factors made it hard for your office to get access to 

Federal data on business program participation? 

Only answer this question if the following conditions are met:

Answer was 'Federal data on business program participation' at question '80 [notlink14]' (Using Data for Statistical, 

Evaluation, Research, or Policy Analysis Purposes Which of the following kinds of data has your office, or its 

representatives, considered using for statistical, evaluation, research, or policy analysis purposes but did not 

because it was too hard to get access to the specific dataset you needed? )

Please choose all that apply:

 Lack of staff, policies, and procedures to establish data sharing agreements 

 Statutes prohibiting data sharing 

 Regulations or policies that make it difficult to link data 

 Lack of capacity in information technology systems 

 Lack of staff or contractors with specific technical expertise needed to link data 

 Concerns about keeping the linked data secure and confidential 

Other: 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 58 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 81

[]

Was the lack of staff, policies, and procedures to establish data sharing 

agreements: 

Only answer this question if the following conditions are met:

Answer was 'Lack of staff, policies, and procedures to establish data sharing agreements ' at question '109 

[bppac15]' (Which of the following factors made it hard for your office to get access to Federal data on business 

program participation?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

[]

Were the statutes prohibiting data sharing: 

Only answer this question if the following conditions are met:

Answer was 'Statutes prohibiting data sharing ' at question '109 [bppac15]' (Which of the following factors made it 

hard for your office to get access to Federal data on business program participation?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

[]

Were the regulations or policies that make it difficult to link data:

Only answer this question if the following conditions are met:

Answer was 'Regulations or policies that make it difficult to link data ' at question '109 [bppac15]' (Which of the 

following factors made it hard for your office to get access to Federal data on business program participation?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 59 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
82 The Promise of Evidence-Based Policymaking

[]

Was the lack of capacity in information technology systems:

Only answer this question if the following conditions are met:

Answer was 'Lack of capacity in information technology systems ' at question '109 [bppac15]' (Which of the following 

factors made it hard for your office to get access to Federal data on business program participation?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

[]

Was the lack of staff or contractors with specific technical expertise needed 

to link data:

Only answer this question if the following conditions are met:

Answer was 'Lack of staff or contractors with specific technical expertise needed to link data ' at question '109 

[bppac15]' (Which of the following factors made it hard for your office to get access to Federal data on business 

program participation?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

[]

Were the concerns about keeping the linked data secure and confidential:

Only answer this question if the following conditions are met:

Answer was 'Concerns about keeping the linked data secure and confidential ' at question '109 [bppac15]' (Which of 

the following factors made it hard for your office to get access to Federal data on business program participation?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 60 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 83

[]Which of the following factors made it hard for your office to get access to 

 Federal establishment survey data (e.g. Economic and Agricultural 

Censuses)? 

Only answer this question if the following conditions are met:

Answer was 'Federal establishment survey data (e.g., Economic and Agricultural Censuses)' at question '80 

[notlink14]' (Using Data for Statistical, Evaluation, Research, or Policy Analysis Purposes Which of the following 

kinds of data has your office, or its representatives, considered using for statistical, evaluation, research, or policy 

analysis purposes but did not because it was too hard to get access to the specific dataset you needed? )

Please choose all that apply:

 Lack of staff, policies, and procedures to establish data sharing agreements 

 Statutes prohibiting data sharing 

 Regulations or policies that make it difficult to link data 

 Lack of capacity in information technology systems 

 Lack of staff or contractors with specific technical expertise needed to link data 

 Concerns about keeping the linked data secure and confidential 

Other: 

[]

Was the lack of staff, policies, and procedures to establish data sharing 

agreements: 

Only answer this question if the following conditions are met:

Answer was 'Lack of staff, policies, and procedures to establish data sharing agreements ' at question '116 

[estbac15]' (Which of the following factors made it hard for your office to get access to Federal establishment 

survey data (e.g. Economic and Agricultural Censuses)?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 61 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
84 The Promise of Evidence-Based Policymaking

[]

Were the statutes prohibiting data sharing: 

Only answer this question if the following conditions are met:

Answer was 'Statutes prohibiting data sharing ' at question '116 [estbac15]' (Which of the following factors made it 

hard for your office to get access to Federal establishment survey data (e.g. Economic and Agricultural 

Censuses)?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

[]

Were the regulations or policies that make it difficult to link data:

Only answer this question if the following conditions are met:

Answer was 'Regulations or policies that make it difficult to link data ' at question '116 [estbac15]' (Which of the 

following factors made it hard for your office to get access to Federal establishment survey data (e.g. Economic and 

Agricultural Censuses)?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

[]

Was the lack of capacity in information technology systems:

Only answer this question if the following conditions are met:

Answer was 'Lack of capacity in information technology systems ' at question '116 [estbac15]' (Which of the 

following factors made it hard for your office to get access to Federal establishment survey data (e.g. Economic and 

Agricultural Censuses)?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 62 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 85

[]

Was the lack of staff or contractors with specific technical expertise needed 

to link data:

Only answer this question if the following conditions are met:

Answer was 'Lack of staff or contractors with specific technical expertise needed to link data ' at question '116 

[estbac15]' (Which of the following factors made it hard for your office to get access to Federal establishment 

survey data (e.g. Economic and Agricultural Censuses)?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

[]

Were the concerns about keeping the linked data secure and confidential:

Only answer this question if the following conditions are met:

Answer was 'Concerns about keeping the linked data secure and confidential ' at question '116 [estbac15]' (Which of 

the following factors made it hard for your office to get access to Federal establishment survey data (e.g. Economic 

and Agricultural Censuses)?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 63 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
86 The Promise of Evidence-Based Policymaking

[]Which of the following factors made it hard for your office to get access to 

Federal tax data? 

Only answer this question if the following conditions are met:

Answer was 'Federal tax data' at question '80 [notlink14]' (Using Data for Statistical, Evaluation, Research, or Policy 

Analysis Purposes Which of the following kinds of data has your office, or its representatives, considered using for 

statistical, evaluation, research, or policy analysis purposes but did not because it was too hard to get access to the 

specific dataset you needed? )

Please choose all that apply:

 Lack of staff, policies, and procedures to establish data sharing agreements 

 Statutes prohibiting data sharing 

 Regulations or policies that make it difficult to link data 

 Lack of capacity in information technology systems 

 Lack of staff or contractors with specific technical expertise needed to link data 

 Concerns about keeping the linked data secure and confidential 

Other: 

[]

Was the lack of staff, policies, and procedures to establish data sharing 

agreements: 

Only answer this question if the following conditions are met:

Answer was 'Lack of staff, policies, and procedures to establish data sharing agreements ' at question '123 

[taxac15]' (Which of the following factors made it hard for your office to get access to Federal tax data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 64 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 87

[]

Were the statutes prohibiting data sharing: 

Only answer this question if the following conditions are met:

Answer was 'Statutes prohibiting data sharing ' at question '123 [taxac15]' (Which of the following factors made it 

hard for your office to get access to Federal tax data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

[]

Were the regulations or policies that make it difficult to link data:

Only answer this question if the following conditions are met:

Answer was 'Regulations or policies that make it difficult to link data ' at question '123 [taxac15]' (Which of the 

following factors made it hard for your office to get access to Federal tax data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

[]

Was the lack of capacity in information technology systems:

Only answer this question if the following conditions are met:

Answer was 'Lack of capacity in information technology systems ' at question '123 [taxac15]' (Which of the following 

factors made it hard for your office to get access to Federal tax data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 65 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
88 The Promise of Evidence-Based Policymaking

[]

Was the lack of staff or contractors with specific technical expertise needed 

to link data:

Only answer this question if the following conditions are met:

Answer was 'Lack of staff or contractors with specific technical expertise needed to link data ' at question '123 

[taxac15]' (Which of the following factors made it hard for your office to get access to Federal tax data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

[]

Were the concerns about keeping the linked data secure and confidential:

Only answer this question if the following conditions are met:

Answer was 'Concerns about keeping the linked data secure and confidential ' at question '123 [taxac15]' (Which of 

the following factors made it hard for your office to get access to Federal tax data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 66 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 89

[]Which of the following factors made it hard for your office to get access 

to Federal social security data? 

Only answer this question if the following conditions are met:

Answer was 'Federal social security data' at question '80 [notlink14]' (Using Data for Statistical, Evaluation, 

Research, or Policy Analysis Purposes Which of the following kinds of data has your office, or its 

representatives, considered using for statistical, evaluation, research, or policy analysis purposes but did not 

because it was too hard to get access to the specific dataset you needed? )

Please choose all that apply:

 Lack of staff, policies, and procedures to establish data sharing agreements 

 Statutes prohibiting data sharing 

 Regulations or policies that make it difficult to link data 

 Lack of capacity in information technology systems 

 Lack of staff or contractors with specific technical expertise needed to link data 

 Concerns about keeping the linked data secure and confidential 

Other: 

[]

Was the lack of staff, policies, and procedures to establish data sharing 

agreements: 

Only answer this question if the following conditions are met:

Answer was 'Lack of staff, policies, and procedures to establish data sharing agreements ' at question '130 

[ssac15]' (Which of the following factors made it hard for your office to get access to Federal social security data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 67 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
90 The Promise of Evidence-Based Policymaking

[]

Were the statute sprohibiting data sharing: 

Only answer this question if the following conditions are met:

Answer was 'Statutes prohibiting data sharing ' at question '130 [ssac15]' (Which of the following factors made it 

hard for your office to get access to Federal social security data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

[]

Were the regulations or policies that make it difficult to link data:

Only answer this question if the following conditions are met:

Answer was 'Regulations or policies that make it difficult to link data ' at question '130 [ssac15]' (Which of the 

following factors made it hard for your office to get access to Federal social security data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

[]

Was the lack of capacity in information technology systems:

Only answer this question if the following conditions are met:

Answer was 'Lack of capacity in information technology systems ' at question '130 [ssac15]' (Which of the following 

factors made it hard for your office to get access to Federal social security data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 68 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 91

[]

Was the lack of staff or contractors with specific technical expertise needed 

to link data:

Only answer this question if the following conditions are met:

Answer was 'Lack of staff or contractors with specific technical expertise needed to link data ' at question '130 

[ssac15]' (Which of the following factors made it hard for your office to get access to Federal social security data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

[]

Were the concerns about keeping the linked data secure and confidential:

Only answer this question if the following conditions are met:

Answer was 'Concerns about keeping the linked data secure and confidential ' at question '130 [ssac15]' (Which of 

the following factors made it hard for your office to get access to Federal social security data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 69 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
92 The Promise of Evidence-Based Policymaking

[]Which of the following factors made it hard for your office to get access to 

Human services program data (e.g. SNAP and TANF)? 

Only answer this question if the following conditions are met:

Answer was 'Human services program data (e.g., SNAP and TANF)' at question '80 [notlink14]' (Using Data for 

Statistical, Evaluation, Research, or Policy Analysis Purposes Which of the following kinds of data has your office, 

or its representatives, considered using for statistical, evaluation, research, or policy analysis purposes but did not 

because it was too hard to get access to the specific dataset you needed? )

Please choose all that apply:

 Lack of staff, policies, and procedures to establish data sharing agreements 

 Statutes prohibiting data sharing 

 Regulations or policies that make it difficult to link data 

 Lack of capacity in information technology systems 

 Lack of staff or contractors with specific technical expertise needed to link data 

 Concerns about keeping the linked data secure and confidential 

Other: 

[]

Was the lack of staff, policies, and procedures to establish data sharing 

agreements: 

Only answer this question if the following conditions are met:

Answer was 'Lack of staff, policies, and procedures to establish data sharing agreements ' at question '137 

[hsac15]' (Which of the following factors made it hard for your office to get access to Human services program data 

(e.g. SNAP and TANF)?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 70 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 93

[]

Were the statutes prohibiting data sharing: 

Only answer this question if the following conditions are met:

Answer was 'Statutes prohibiting data sharing ' at question '137 [hsac15]' (Which of the following factors made it 

hard for your office to get access to Human services program data (e.g. SNAP and TANF)?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

[]

Were the regulations or policies that make it difficult to link data:

Only answer this question if the following conditions are met:

Answer was 'Regulations or policies that make it difficult to link data ' at question '137 [hsac15]' (Which of the 

following factors made it hard for your office to get access to Human services program data (e.g. SNAP and 

TANF)?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

[]

Was the lack of capacity in information technology systems:

Only answer this question if the following conditions are met:

Answer was 'Lack of capacity in information technology systems ' at question '137 [hsac15]' (Which of the following 

factors made it hard for your office to get access to Human services program data (e.g. SNAP and TANF)?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 71 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
94 The Promise of Evidence-Based Policymaking

[]

Was the lack of staff or contractors with specific technical expertise needed 

to link data:

Only answer this question if the following conditions are met:

Answer was 'Lack of staff or contractors with specific technical expertise needed to link data ' at question '137 

[hsac15]' (Which of the following factors made it hard for your office to get access to Human services program data 

(e.g. SNAP and TANF)?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

[]

Were the concerns about keeping the linked data secure and confidential:

Only answer this question if the following conditions are met:

Answer was 'Concerns about keeping the linked data secure and confidential ' at question '137 [hsac15]' (Which of 

the following factors made it hard for your office to get access to Human services program data (e.g. SNAP and 

TANF)?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 72 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 95

[]Which of the following factors made it hard for your office to get access to 

Medicaid/Medicare claims and utilization data? 

Only answer this question if the following conditions are met:

Answer was 'Medicaid/Medicare claims and utilization data' at question '80 [notlink14]' (Using Data for Statistical, 

Evaluation, Research, or Policy Analysis Purposes Which of the following kinds of data has your office, or its 

representatives, considered using for statistical, evaluation, research, or policy analysis purposes but did not 

because it was too hard to get access to the specific dataset you needed? )

Please choose all that apply:

 Lack of staff, policies, and procedures to establish data sharing agreements 

 Statutes prohibiting data sharing 

 Regulations or policies that make it difficult to link data 

 Lack of capacity in information technology systems 

 Lack of staff or contractors with specific technical expertise needed to link data 

 Concerns about keeping the linked data secure and confidential 

Other: 

[]

Was the lack of staff, policies, and procedures to establish data sharing 

agreements: 

Only answer this question if the following conditions are met:

Answer was 'Lack of staff, policies, and procedures to establish data sharing agreements ' at question '144 

[mmac15]' (Which of the following factors made it hard for your office to get access to Medicaid/Medicare claims and 

utilization data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 73 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
96 The Promise of Evidence-Based Policymaking

[]

Were the statutes prohibiting data sharing: 

Only answer this question if the following conditions are met:

Answer was 'Statutes prohibiting data sharing ' at question '144 [mmac15]' (Which of the following factors made it 

hard for your office to get access to Medicaid/Medicare claims and utilization data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

[]

Were the regulations or policies that make it difficult to link data:

Only answer this question if the following conditions are met:

Answer was 'Regulations or policies that make it difficult to link data ' at question '144 [mmac15]' (Which of the 

following factors made it hard for your office to get access to Medicaid/Medicare claims and utilization data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

[]

Was the lack of capacity in information technology systems:

Only answer this question if the following conditions are met:

Answer was 'Lack of capacity in information technology systems ' at question '144 [mmac15]' (Which of the following 

factors made it hard for your office to get access to Medicaid/Medicare claims and utilization data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 74 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 97

[]

Was the lack of staff or contractors with specific technical expertise needed 

to link data:

Only answer this question if the following conditions are met:

Answer was 'Lack of staff or contractors with specific technical expertise needed to link data ' at question '144 

[mmac15]' (Which of the following factors made it hard for your office to get access to Medicaid/Medicare claims and 

utilization data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

[]

Were the concerns about keeping the linked data secure and confidential:

Only answer this question if the following conditions are met:

Answer was 'Concerns about keeping the linked data secure and confidential ' at question '144 [mmac15]' (Which of 

the following factors made it hard for your office to get access to Medicaid/Medicare claims and utilization data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 75 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
98 The Promise of Evidence-Based Policymaking

[]Which of the following factors made it hard for your office to get access to 

Military service and veteran benefits records? 

Only answer this question if the following conditions are met:

Answer was 'Military service and veteran benefits records' at question '80 [notlink14]' (Using Data for Statistical, 

Evaluation, Research, or Policy Analysis Purposes Which of the following kinds of data has your office, or its 

representatives, considered using for statistical, evaluation, research, or policy analysis purposes but did not 

because it was too hard to get access to the specific dataset you needed? )

Please choose all that apply:

 Lack of staff, policies, and procedures to establish data sharing agreements 

 Statutes prohibiting data sharing 

 Regulations or policies that make it difficult to link data 

 Lack of capacity in information technology systems 

 Lack of staff or contractors with specific technical expertise needed to link data 

 Concerns about keeping the linked data secure and confidential 

Other: 

[]

Was the lack of staff, policies, and procedures to establish data sharing 

agreements: 

Only answer this question if the following conditions are met:

Answer was 'Lack of staff, policies, and procedures to establish data sharing agreements ' at question '151 

[vetac15]' (Which of the following factors made it hard for your office to get access to Military service and veteran 

benefits records?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 76 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 99

[]

Were the statutes prohibiting data sharing: 

Only answer this question if the following conditions are met:

Answer was 'Statutes prohibiting data sharing ' at question '151 [vetac15]' (Which of the following factors made it 

hard for your office to get access to Military service and veteran benefits records?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

[]

Were the regulations or policies that make it difficult to link data:

Only answer this question if the following conditions are met:

Answer was 'Regulations or policies that make it difficult to link data ' at question '151 [vetac15]' (Which of the 

following factors made it hard for your office to get access to Military service and veteran benefits records?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

[]

Was the lack of capacity in information technology systems:

Only answer this question if the following conditions are met:

Answer was 'Lack of capacity in information technology systems ' at question '151 [vetac15]' (Which of the following 

factors made it hard for your office to get access to Military service and veteran benefits records?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 77 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
100 The Promise of Evidence-Based Policymaking

[]

Was the lack of staff or contractors with specific technical expertise needed 

to link data:

Only answer this question if the following conditions are met:

Answer was 'Lack of staff or contractors with specific technical expertise needed to link data ' at question '151 

[vetac15]' (Which of the following factors made it hard for your office to get access to Military service and veteran 

benefits records?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

[]

Were the concerns about keeping the linked data secure and confidential:

Only answer this question if the following conditions are met:

Answer was 'Concerns about keeping the linked data secure and confidential ' at question '151 [vetac15]' (Which of 

the following factors made it hard for your office to get access to Military service and veteran benefits records?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 78 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 101

[]Which of the following factors made it hard for your office to get access to 

State unemployment insurance wage record data? 

Only answer this question if the following conditions are met:

Answer was 'State unemployment insurance wage record data' at question '80 [notlink14]' (Using Data for Statistical, 

Evaluation, Research, or Policy Analysis Purposes Which of the following kinds of data has your office, or its 

representatives, considered using for statistical, evaluation, research, or policy analysis purposes but did not 

because it was too hard to get access to the specific dataset you needed? )

Please choose all that apply:

 Lack of staff, policies, and procedures to establish data sharing agreements 

 Statutes prohibiting data sharing 

 Regulations or policies that make it difficult to link data 

 Lack of capacity in information technology systems 

 Lack of staff or contractors with specific technical expertise needed to link data 

 Concerns about keeping the linked data secure and confidential 

Other: 

[]

Was the lack of staff, policies, and procedures to establish data sharing 

agreements: 

Only answer this question if the following conditions are met:

Answer was 'Lack of staff, policies, and procedures to establish data sharing agreements ' at question '158 

[wagac15]' (Which of the following factors made it hard for your office to get access to State unemployment 

insurance wage record data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 79 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
102 The Promise of Evidence-Based Policymaking

[]

Were the statutes prohibiting data sharing: 

Only answer this question if the following conditions are met:

Answer was 'Statutes prohibiting data sharing ' at question '158 [wagac15]' (Which of the following factors made it 

hard for your office to get access to State unemployment insurance wage record data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

[]

Were the regulations or policies that make it difficult to link data:

Only answer this question if the following conditions are met:

Answer was 'Regulations or policies that make it difficult to link data ' at question '158 [wagac15]' (Which of the 

following factors made it hard for your office to get access to State unemployment insurance wage record data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

[]

Was the lack of capacity in information technology systems:

Only answer this question if the following conditions are met:

Answer was 'Lack of capacity in information technology systems ' at question '158 [wagac15]' (Which of the 

following factors made it hard for your office to get access to State unemployment insurance wage record data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 80 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 103

[]

Was the lack of staff or contractors with specific technical expertise needed 

to link data:

Only answer this question if the following conditions are met:

Answer was 'Lack of staff or contractors with specific technical expertise needed to link data ' at question '158 

[wagac15]' (Which of the following factors made it hard for your office to get access to State unemployment 

insurance wage record data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

[]

Were the concerns about keeping the linked data secure and confidential:

Only answer this question if the following conditions are met:

Answer was 'Concerns about keeping the linked data secure and confidential ' at question '158 [wagac15]' (Which of 

the following factors made it hard for your office to get access to State unemployment insurance wage record data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 81 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
104 The Promise of Evidence-Based Policymaking

[]Which of the following factors made it hard for your office to get access to 

Vital records (birth and death) data? 

Only answer this question if the following conditions are met:

Answer was 'Vital records (birth and death) data' at question '80 [notlink14]' (Using Data for Statistical, Evaluation, 

Research, or Policy Analysis Purposes Which of the following kinds of data has your office, or its 

representatives, considered using for statistical, evaluation, research, or policy analysis purposes but did not 

because it was too hard to get access to the specific dataset you needed? )

Please choose all that apply:

 Lack of staff, policies, and procedures to establish data sharing agreements 

 Statutes prohibiting data sharing 

 Regulations or policies that make it difficult to link data 

 Lack of capacity in information technology systems 

 Lack of staff or contractors with specific technical expertise needed to link data 

 Concerns about keeping the linked data secure and confidential 

Other: 

[]

Was the lack of staff, policies, and procedures to establish data sharing 

agreements: 

Only answer this question if the following conditions are met:

Answer was 'Lack of staff, policies, and procedures to establish data sharing agreements ' at question '165 

[vitac15]' (Which of the following factors made it hard for your office to get access to Vital records (birth and 

death) data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 82 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 105

[]

Were the statutes prohibiting data sharing: 

Only answer this question if the following conditions are met:

Answer was 'Statutes prohibiting data sharing ' at question '165 [vitac15]' (Which of the following factors made it 

hard for your office to get access to Vital records (birth and death) data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

[]

Were the regulations or policies that make it difficult to link data:

Only answer this question if the following conditions are met:

Answer was 'Regulations or policies that make it difficult to link data ' at question '165 [vitac15]' (Which of the 

following factors made it hard for your office to get access to Vital records (birth and death) data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

[]

Was the lack of capacity in information technology systems:

Only answer this question if the following conditions are met:

Answer was 'Lack of capacity in information technology systems ' at question '165 [vitac15]' (Which of the following 

factors made it hard for your office to get access to Vital records (birth and death) data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 83 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
106 The Promise of Evidence-Based Policymaking

[]

Was the lack of staff or contractors with specific technical expertise needed 

to link data:

Only answer this question if the following conditions are met:

Answer was 'Lack of staff or contractors with specific technical expertise needed to link data ' at question '165 

[vitac15]' (Which of the following factors made it hard for your office to get access to Vital records (birth and 

death) data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

[]

Were the concerns about keeping the linked data secure and confidential:

Only answer this question if the following conditions are met:

Answer was 'Concerns about keeping the linked data secure and confidential ' at question '165 [vitac15]' (Which of 

the following factors made it hard for your office to get access to Vital records (birth and death) data?)

Please choose all that apply:

 In your office? 

 In another office in your agency? 

 In another agency? 

 At the state or local level? 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 84 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 107

[]To what extent is each of the following a barrier to your office's ability to 

use datafor statistical, evaluation, research, or policy analysis purposes? 

Only answer this question if the following conditions are met:

-------- Scenario 1 --------

Answer was 'To evaluate a program to understand its implementation or impact on participants ' or 'To produce 

statistics about the economy, society, or environment ' or 'To support research on a population, group of entities, 

policy, or program' or 'To conduct analysis of existing policies or programs to assist decision making ' at question '26 

[purpose5]' (For which of the following purposes does your office, or its representatives, collect data from individual 

units? )

-------- or Scenario 2 --------

Answer was 'To produce statistics about the economy, society, or environment ' or 'To evaluate a program to 

understand its implementation or impact on participants ' or 'To support research on a population, group of entities, 

policy, or program' or 'To conduct analysis of existing policies or programs to assist decision making ' at question '31 

[micropurpose10]' (For which of the following purposes does your office, or its representatives, analyze micro-level 

data collected by any other Federal office or agency?)

-------- or Scenario 3 --------

Answer was '51 to 90 percent' or '11 to 50 percent' or '1 to 10 percent' or '91 to 100 percent ' at question '23 

[SERA46]' ( For the remainder of the survey, please note the following additional definitions: My/your agency: the 

Federal Agency or Department you marked in the first question Another agency: any Federal Agency or 

Department other than the one you marked in the first question My/your office: the Federal office you marked in the 

second question Another office in my/your agency: another Federal office within the Agency or Department you 

marked in the first question Representatives: individuals who receive direction from, but are not directly employed 

by, your office or agency, including contractors, grantees, and those on interagency personnel agreements (IPAs) 

Approximately what percentage of your office's total annual budget is for statistical, evaluation, research, or policy 

analysis purposes? )

Please choose the appropriate response for each item:

Not a barrier Minor barrier

Moderate 

barrier Major barrier

Lack of ability to 

hire appropriate 

staff 

Lack of ability to 

execute and 

manage contracts

Inadequate funding

Legal limitations 

Information 

Technology (IT) 

systems or 

requirements

Privacy and data 

security 

requirements

Political processes 

(e.g., transitions in 

agency/office 

leadership, 

Congressional 

oversight)

Information 

collection 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 85 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
108 The Promise of Evidence-Based Policymaking

Not a barrier Minor barrier

Moderate 

barrier Major barrier

requirements (e.g., 

Paperwork 

Reduction Act and 

Institutional Review 

Boards)

Other

Statistical, evaluation, research, or policy analysis purposes: activities intended to support the production, 

analysis, and dissemination of information, including the creation and analysis of federal statistics; production or 

use of evaluation for federally-supported or -directed programs; research on a population, group of entities, 

policy, or program; or analysis of policies or proposals to inform decision making. These purposes do NOT 

include program operations (e.g., compliance monitoring, eligibility determinations, enforcement actions). 

[]If other, please specify: 

Only answer this question if the following conditions are met:

Answer was 'Minor barrier' or 'Moderate barrier' or 'Major barrier' at question '172 [SERA49]' (To what extent is each 

of the following a barrier to your office's ability to use data for statistical, evaluation, research, or policy analysis 

purposes? (Other))

Please write your answer here:

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 86 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 109

[]Which ONE of the following is the most critical barrier to your office's 

ability to use datafor statistical, evaluation, research, or policy analysis 

purposes?  

Only answer this question if the following conditions are met:

-------- Scenario 1 --------

Answer was 'To evaluate a program to understand its implementation or impact on participants ' or 'To produce 

statistics about the economy, society, or environment ' or 'To support research on a population, group of entities, 

policy, or program' or 'To conduct analysis of existing policies or programs to assist decision making ' at question '26 

[purpose5]' (For which of the following purposes does your office, or its representatives, collect data from individual 

units? )

-------- or Scenario 2 --------

Answer was 'To produce statistics about the economy, society, or environment ' or 'To evaluate a program to 

understand its implementation or impact on participants ' or 'To support research on a population, group of entities, 

policy, or program' or 'To conduct analysis of existing policies or programs to assist decision making ' at question '31 

[micropurpose10]' (For which of the following purposes does your office, or its representatives, analyze micro-level 

data collected by any other Federal office or agency?)

-------- or Scenario 3 --------

Answer was '91 to 100 percent ' or '51 to 90 percent' or '11 to 50 percent' or '1 to 10 percent' at question '23 

[SERA46]' ( For the remainder of the survey, please note the following additional definitions: My/your agency: the 

Federal Agency or Department you marked in the first question Another agency: any Federal Agency or 

Department other than the one you marked in the first question My/your office: the Federal office you marked in the 

second question Another office in my/your agency: another Federal office within the Agency or Department you 

marked in the first question Representatives: individuals who receive direction from, but are not directly employed 

by, your office or agency, including contractors, grantees, and those on interagency personnel agreements (IPAs) 

Approximately what percentage of your office's total annual budget is for statistical, evaluation, research, or policy 

analysis purposes? )

Please choose only one of the following:

 Lack of ability to hire appropriate staff 

 Lack of ability to execute and manage contracts 

 Inadequate funding 

 Legal limitations 

 Information Technology (IT) systems or requirements 

 Privacy and data security requirements 

 Political processes (e.g., transitions in agency/office leadership, Congressional oversight) 

 Information collection requirements (e.g., Paperwork Reduction Act and Institutional Review Boards) 

 Other 

Statistical, evaluation, research, or policy analysis purposes: activities intended to support the production, 

analysis, and dissemination of information, including the creation and analysis of federal statistics; production or 

use of evaluation for federally-supported or -directed programs; research on a population, group of entities, 

policy, or program; or analysis of policies or proposals to inform decision making. These purposes do NOT 

include program operations (e.g., compliance monitoring, eligibility determinations, enforcement actions). 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 87 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
110 The Promise of Evidence-Based Policymaking

[]If applicable, please provide additional information about this most critical 

barrier. 

Only answer this question if the following conditions are met:

-------- Scenario 1 --------

Answer was 'To support research on a population, group of entities, policy, or program' or 'To conduct analysis of 

existing policies or programs to assist decision making ' or 'To produce statistics about the economy, society, or 

environment ' or 'To evaluate a program to understand its implementation or impact on participants ' at question '26 

[purpose5]' (For which of the following purposes does your office, or its representatives, collect data from individual 

units? )

-------- or Scenario 2 --------

Answer was 'To produce statistics about the economy, society, or environment ' or 'To evaluate a program to 

understand its implementation or impact on participants ' or 'To support research on a population, group of entities, 

policy, or program' or 'To conduct analysis of existing policies or programs to assist decision making ' at question '31 

[micropurpose10]' (For which of the following purposes does your office, or its representatives, analyze micro-level 

data collected by any other Federal office or agency?)

-------- or Scenario 3 --------

Answer was '91 to 100 percent ' or '51 to 90 percent' or '11 to 50 percent' or '1 to 10 percent' at question '23 

[SERA46]' ( For the remainder of the survey, please note the following additional definitions: My/your agency: the 

Federal Agency or Department you marked in the first question Another agency: any Federal Agency or 

Department other than the one you marked in the first question My/your office: the Federal office you marked in the 

second question Another office in my/your agency: another Federal office within the Agency or Department you 

marked in the first question Representatives: individuals who receive direction from, but are not directly employed 

by, your office or agency, including contractors, grantees, and those on interagency personnel agreements (IPAs) 

Approximately what percentage of your office's total annual budget is for statistical, evaluation, research, or policy 

analysis purposes? )

Please write your answer here:

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 88 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 111

[]If applicable, please list up to five statistical, evaluation, research, or policy 

analysis activities your office, or its representatives, have not been able to 

complete because it was too hard to access the right data. 

Only answer this question if the following conditions are met:

-------- Scenario 1 --------

Answer was 'To support research on a population, group of entities, policy, or program' or 'To conduct analysis of 

existing policies or programs to assist decision making ' or 'To produce statistics about the economy, society, or 

environment ' or 'To evaluate a program to understand its implementation or impact on participants ' at question '26 

[purpose5]' (For which of the following purposes does your office, or its representatives, collect data from individual 

units? )

-------- or Scenario 2 --------

Answer was 'To evaluate a program to understand its implementation or impact on participants ' or 'To produce 

statistics about the economy, society, or environment ' or 'To support research on a population, group of entities, 

policy, or program' or 'To conduct analysis of existing policies or programs to assist decision making ' at question '31 

[micropurpose10]' (For which of the following purposes does your office, or its representatives, analyze micro-level 

data collected by any other Federal office or agency?)

-------- or Scenario 3 --------

Answer was '91 to 100 percent ' or '51 to 90 percent' or '11 to 50 percent' or '1 to 10 percent' at question '23 

[SERA46]' ( For the remainder of the survey, please note the following additional definitions: My/your agency: the 

Federal Agency or Department you marked in the first question Another agency: any Federal Agency or 

Department other than the one you marked in the first question My/your office: the Federal office you marked in the 

second question Another office in my/your agency: another Federal office within the Agency or Department you 

marked in the first question Representatives: individuals who receive direction from, but are not directly employed 

by, your office or agency, including contractors, grantees, and those on interagency personnel agreements (IPAs) 

Approximately what percentage of your office's total annual budget is for statistical, evaluation, research, or policy 

analysis purposes? )

Please write your answer here:

Statistical, evaluation, research, or policy analysis purposes: activities intended to support the production, 

analysis, and dissemination of information, including the creation and analysis of federal statistics; production or 

use of evaluation for federally-supported or -directed programs; research on a population, group of entities, 

policy, or program; or analysis of policies or proposals to inform decision making. These purposes do NOT 

include program operations (e.g., compliance monitoring, eligibility determinations, enforcement actions). 

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 89 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
112 The Promise of Evidence-Based Policymaking

[]If applicable, please provide an example of amicro-level data linkageproject 

or process in your office that works well. 

Only answer this question if the following conditions are met:

Answer was 'Yes, occasionally (not as a regular part of our programs or processes) ' or 'Yes, routinely (as a regular 

part of our programs or processes) ' at question '24 [collect3]' ( Collecting Data Does your office, or its 

representatives, collect data about individual units (e.g., people, businesses, nonprofit organizations, schools) either 

directly or through an intermediary source (e.g., state or local governments, private entities)? )

Please write your answer here:

Micro-level data: a data file with an entry (record) for each individual unit.

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 90 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 113

[]If applicable, please provide an example of how your office has used data 

to build evidence for policy making. 

Please write your answer here:

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 91 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
114 The Promise of Evidence-Based Policymaking

Submit your survey.

Thank you for completing this survey.

MAX Survey - Building Evidence from Data--Capacity and Needs in the Federal Gove... Page 92 of 92

https://survey.max.gov/index.php/admin/printablesurvey/sa/index/surveyid/692148 2/7/2017
The Promise of Evidence-Based Policymaking 115

Survey response

Which Federal agency are you responding 

for?

Social Security Administration

Office of Retirement Policy

If applicable, please provide additional 

information about this most critical barrier.

It is often a time-consuming process to get 

clearances from multiple agencies to use their data 

merged with SSA data.

If applicable, please list up to five statistical, 

evaluation, research, or policy analysis 

activities your office, or its representatives, 

have not been able to complete because it

was too hard to access the right data.

If applicable, please provide an example of a 

micro-level data linkage project or process in 

your office that works well. 

If applicable, please provide an example of 

how your office has used data to build 

evidence for policy making.

Our office uses the MINT model (Modeling 

Income in the Near Term model) to analyze the 

distributional consequences of proposed Social 

Security changes. The basis of the model is SIPP 

data (Census Bureau data) merged with SSA 

earnings data.

Attachment D. Federal Office Reponses to Open-Ended Questions 

on the CEP Survey of Federal Offices, 2017
116 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you 

responding for?

Department of Veterans Affairs

National Center for Veterans Analysis and Statistics

If applicable, please provide additional 

information about this most critical 

barrier.

It is often not the actual law or policy, but a risk 

adverse application of law or policy. 

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, or 

its representatives, have not been able to 

complete because it was too hard to 

access the right data.

Linking Veteran data to IRS data to ascertain the 

veteran outcomes by looking at income as a proxy. 

If applicable, please provide an example 

of a micro-level data linkage project or 

process in your office that works well. 

If applicable, please provide an example 

of how your office has used data to 

build evidence for policy making.

We provide a predictive model of the future Veteran 

population. We collect administrative data from within 

VA, integrate force projections and separations from 

DoD to estimate the future Veteran population. This 

model is used in VA strategic planning and as input to 

VHA's healthcare enrollee projection model which is 

issued in health policy and budget formulation. 
The Promise of Evidence-Based Policymaking 117

Survey response

Which Federal agency are you 

responding for?

Department of the Interior

United States Geological Survey

If applicable, please provide additional 

information about this most critical 

barrier.

Funding is often not available for historical data 

digitization and necessary investments in information 

technology infrastructure. 

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, 

or its representatives, have not been 

able to complete because it was too 

hard to access the right data.

Field records from retiring scientists, historical data on 

tapes related to earth science, bird banding data, 

historical topographic maps 

If applicable, please provide an 

example of a micro-level data linkage 

project or process in your office that 

works well. 

If applicable, please provide an 

example of how your office has used 

data to build evidence for policy 

making.

In the last several years, USGS has been developing a 

national biodiversity observation network, the USA 

National Phenology Network (www.usanpn.org), 

including the production of data products related to 

biodiversity and environmental variation that can be used 

for research or natural resource management decisionmaking. USGS works closely with DOI National Park 

Service, Fish and Wildlife Service, the USDA 

Agricultural Research Service, the National Ecological 

Observatory Network (NEON), and the US Global 

Change Research Program. Data on national patterns of 

plant and animal phenology are collected by both 

professionals and volunteers ("citizen scientists"); data 

are immediately and freely available, and have been used 

in a number of publications, and by state/federal land 

managers. Additionally, The USGS's National Water 

Use Information Program compiles and publishes the 

Nation's water-use data. Public access to some of these 

data is provided via the USGS Water Data for the Nation 

site. Water use refers to water that is used for specific 

purposes. Water-use data is collected by area type (State, 

county, watershed or aquifer) and source such as rivers 

or groundwater, and category such as public supply or 

irrigation. Water-use data has been reported every five 

years since 1950, for years ending in "0" and "5". The 

USGS works in cooperation with local, State, and 

Federal agencies as well as academic and private 

organizations to collect and report total withdrawals.
118 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you 

responding for?

Department of Health and Human Services

Substance Abuse and Mental Health Services 

Administration - Center for Behavioral Health Statistics 

and Quality

If applicable, please provide additional 

information about this most critical 

barrier.

It is hard to fill vacancies and retain appropriate staff 

due to inefficient HR process and inadequate position 

grade levels.

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, or 

its representatives, have not been able to 

complete because it was too hard to 

access the right data.

Resource limitations (staffing and contractual) limit 

ability to support dissemination of new data set, 

including internal program administrative and 

evaluation data, as well as outside data sets and more 

analytic tools.

If applicable, please provide an example 

of a micro-level data linkage project or 

process in your office that works well. 

We support a Data Portal which enables outside 

researchers to apply for access to restricted micro-level 

data which is housed on a secure virtual data enclave 

(hosted on a physical server). The process is extremely 

efficient and safe as it includes tri-factor authentication, 

on-site visits to users, and a rigorous application 

process. The Data Portal also requires all analysis to be 

done within the data enclave, and everything must go 

through a thorough disclosure review (by the 

appropriate data groups) before being disseminated to 

the researcher for use outside of the Data Portal. It has 

been a huge success in supporting research to 

researchers all over the United States, allowing access 

from secure locations without the inconvenience of 

having to travel to an on-site data center. It also is cost 

efficient, as all monitoring of users and researchers can 

be done virtually and safely. We are able to support 

many more researchers without having to hire 

additional FTEs that would otherwise have to 

physically be present in an RDC. We have also 

increased turn-around-time for allowing access to 

research, and providing disclosure reviews for 

researchers to complete papers, presentations, doctoral 

dissertations, and allow for our data to be utilized and 

analyzed to its fullest capacity. We have made access to 

our confidential data an easy, however very safe and 

compliant, and quick process for those that need it to 

support their research. The Data Portal has also 

provided easy and safe access to sister federal agencies, 

where before they needed to travel on site for access. 

This has enable other various agencies to further their 
The Promise of Evidence-Based Policymaking 119

research agendas and expedite their own timelines. Our 

Data Portal and over all Data Archive (SAMHDA) 

have revolutionized the way that we disseminate data. 

We now have complete ownership over our data, where 

it is stored, how it is accessed and the timeline in which 

it is deployed. We are able to turnaround data and data 

publications with great speed, and having control over 

our own systems has also enabled us to respond to 

requests in a much more timely manner. In working on 

sub committees (Confidential Data Access Committee 

and others) we are hopeful that with the support of 

other agencies, OMB will continue to work on 

improving the laws surrounding data collection and 

security - CIPSEA in particular - so that one day we 

will be able to support researchers using the Data Portal 

on a cloud environment, which would only help speed 

and efficiency. 

If applicable, please provide an example 

of how your office has used data to 

build evidence for policy making.

Both CBHSQ and OPPI SAMHSA components have

been asked to participate in this survey. We defer to 

OPPI to answer this question.
120 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are 

you responding for?

Department of Health and Human Services

Indian Health Service

If applicable, please provide 

additional information about 

this most critical barrier.

IHS is funded at 40% of need. Other government agencies have 

price scales costed for agencies with deep pockets. We are 

primarily a health system for impoverished people. Any dollar 

misspent results in less medical care.

If applicable, please list up 

to five statistical, 

evaluation, research, or 

policy analysis activities 

your office, or its 

representatives, have not 

been able to complete 

because it was too hard to 

access the right data.

Misclassification by race adjustment ratios using National Death 

Index data on a routine basis down to the service unit level 

Mortality linkage studies for recent years of data due to National 

Death Index data costs on suicide, chronic disease, injury 

Identification of which of our patients is a Veteran or has a service 

connected disability Solid maternal mortality studies Studies of 

patients transferred to other facilities outside IHS and outcomes

If applicable, please provide 

an example of a micro-level 

data linkage project or

process in your office that 

works well. 

IHS medical records are accumulated annually, unduplicated, and 

put into a Indian registrant file. This is sent annually to the US 

Census to help them with their counts of American Indians and 

Alaska Natives. Census performs its linkages as needed.

If applicable, please provide 

an example of how your 

office has used data to build 

evidence for policy making.

Trends in Indian Health and Regional Differences in Indian Health 

present tables and charts that describe the IHS program and the 

health status of American Indians and Alaska Natives. Information 

pertaining to the IHS structure, American Indian and Alaska Native 

demography, patient care, and community health are included. 

Current and trend information are presented, and comparisons with 

other population groups are made, when appropriate. Current ones 

may be found at https://www.ihs.gov/dps/index.cfm/publications/ 
The Promise of Evidence-Based Policymaking 121

Survey response

Which Federal agency are you 

responding for?

Department of Defense

Army Corps of Engineers

If applicable, please provide additional 

information about this most critical 

barrier.

Information on ports is collected by the Department of 

Homeland Security, but either not at the level of detail 

that we need it, or it is and they won't provide it to us. 

Another issue is that individual ports do not want to 

share data that we could use to do economic analysis on 

ports, because they are concerned that 1) it would be 

shared with other ports, who are their competitors; and 

2) we might use that data to make budget decisions 

favoring one ports over another based on its economic 

competitiveness.

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, or 

its representatives, have not been able 

to complete because it was too hard to 

access the right data.

Harbor Maintenance Trust Fund report to Congress.

If applicable, please provide an 

example of a micro-level data linkage 

project or process in your office that 

works well. 

If applicable, please provide an 

example of how your office has used 

data to build evidence for policy 

making.

1. Development of the Corps' budget is largely based on 

project-level data that the Corps of Engineers develops 

through detailed reports/studies, involving engineering, 

economic, environmental, and related analysis. 2. We 

use data on the ability of certain projects/programs to 

execute funds as one metric we consider in making 

funding decisions. We have significantly reduced 

funding for programs that were not capable of executing 

the funds they had on hand. One example is the Dam 

Safety Program of the Corps of Engineers - it continues 

to receive a high priority for funding, but it became 

apparent that it was not executing those funds. We used 

the clear signal we saw in that data to overcome internal 

resistance to continuing to fund a "safety" program at an 

overly robust level.
122 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you 

responding for?

Department of Agriculture

Food and Nutrition Service

If applicable, please provide additional 

information about this most critical 

barrier.

The requirements of the Paperwork Reduction Act were 

primarily designed to limit the administrative burden on 

entities being regulated by the government, but are 

applied as a government-wide gatekeeper on the nature 

of statistical rigor of all data collections, including onetime sampled collections. The burdens of the 

centralized PRA clearance process, exacerbated by the 

limited capacity of OIRA to address this 

comprehensive government-wide workload, has 

dramatically increased the cost and delay in getting 

critical information to program administrators. While 

the government has an interest in ensuring that data 

collected is valid for its purpose, high-quality, and nonduplicative, all but the most potentially-burdensome 

collections could be appropriately managed at the 

agency or sub-agency level.

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, or 

its representatives, have not been able to 

complete because it was too hard to 

access the right data.

We continue to struggle with assessing the eligibility 

and participation of active-duty military households in 

our programs because of a lack of data on the 

circumstances of those families. Military authorities 

have been reluctant to share that information for 

reasons that are not entirely clear.

If applicable, please provide an example 

of a micro-level data linkage project or 

process in your office that works well. 

Several studies have linked administrative and survey 

data to assess the effect of SNAP on food insecurity 

and have found a strong association between program 

participation and reductions in food insecurity. Census 

data has also been used to demonstrate the positive

effects of SNAP in reducing poverty. Other studies 

have used both administrative and survey data to show 

how food insecurity affects learning and childhood 

development, both in the summer and during the school 

year. This evidence helped inform the Summer 

Electronic Benefits Transfer for Children (SEBTC) 

demonstration, which provided benefits on an 

electronic debit card to children eligible for free and 

reduced-price school meals to purchase food during the 

summer months when away from school. 

If applicable, please provide an example 

of how your office has used data to 

build evidence for policy making.

Effectively reaching those most in need is a critical 

concern of the Supplemental Nutrition Assistance 

Program (SNAP). The Food and Nutrition Service 

(FNS) uses administrative data – specifically, a sample 

of certification data drawn annually to measure 
The Promise of Evidence-Based Policymaking 123

payment accuracy (the Quality Control, or QC, data) to 

examine in depth the participants it serves. The latest 

annual SNAP Characteristics report developed with 

these data shows that well over half of the benefits 

(58%) go to households at or below 50% of the poverty 

line and almost all of the benefits (93%) are going to 

households at or below 100% of the poverty line. 

Rigorous evaluations of the Summer Electronic 

Benefits Transfer for Children (SEBTC) demonstration, 

demonstration found that SEBTC reduced very-low 

food security among children by one-third and 

improved participants’ diets. The study used random 

assignment into treatment and control groups, 

combined with EBT transaction data, dietary recall 

surveys, household interviews, and the Food Security 

Supplement Survey to collect data from a sample of 

over 50,000 children. 
124 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you responding 

for?

Department of Energy

Office of Environment, Health, Safety and 

Security

If applicable, please provide additional 

information about this most critical barrier.

OMB has placed FTE limits on our organization 

which prevents us from hiring necessary staff. 

If applicable, please list up to five statistical, 

evaluation, research, or policy analysis 

activities your office, or its representatives, 

have not been able to complete because it was 

too hard to access the right data.

If applicable, please provide an example of a 

micro-level data linkage project or process in 

your office that works well. 

Historical work records are transmitted to 

researchers for development of Energy Employee 

Occupational Injury Compensation claim 

development for adjudication by the Department 

of Labor. Data is provided in a secure electronic 

process.

If applicable, please provide an example of 

how your office has used data to build 

evidence for policy making.

Numbers and types of injury claims are used to 

evaluate current health and safety policy to 

determine if changes are necessary.
The Promise of Evidence-Based Policymaking 125

Survey response

Which Federal agency are you responding 

for?

Department of Justice

Office of Justice Programs - Office of Violence 

Against Women

If applicable, please provide additional 

information about this most critical barrier.

If applicable, please list up to five 

statistical, evaluation, research, or policy 

analysis activities your office, or its 

representatives, have not been able to 

complete because it was too hard to access 

the right data.

If applicable, please provide an example of 

a micro-level data linkage project or 

process in your office that works well. 

If applicable, please provide an example of 

how your office has used data to build 

evidence for policy making.

Our office uses grantee-reported data to understand 

where barriers exist to serving victims of domestic 

and sexual violence, and to identify gaps in services. 

We also fund research and evaluation to assess the 

effectiveness and impact of interventions funded 

through our grant programs.
126 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you responding for?

Department of Health and Human Services

Centers for Disease Control and Prevention -

National Center for Immunization and 

Respiratory Diseases

If applicable, please provide additional

information about this most critical barrier.

If applicable, please list up to five statistical, 

evaluation, research, or policy analysis 

activities your office, or its representatives, 

have not been able to complete because it was 

too hard to access the right data.

Any integration of data from the education 

sector due to FERPA

If applicable, please provide an example of a 

micro-level data linkage project or process in 

your office that works well. 

If applicable, please provide an example of how 

your office has used data to build evidence for 

policy making.
The Promise of Evidence-Based Policymaking 127

Survey response

Which Federal agency are you 

responding for?

Department of Veterans Affairs

Veterans Health Administration

If applicable, please provide additional 

information about this most critical 

barrier.

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, or 

its representatives, have not been able to 

complete because it was too hard to 

access the right data.

If applicable, please provide an example 

of a micro-level data linkage project or 

process in your office that works well. 

We transmit record-level clinical data to CMS to 

calculate hospital outcomes for public reporting on 

Hospital Compare

If applicable, please provide an example 

of how your office has used data to 

build evidence for policy making.

We have initiated a new patient survey, based, in part, 

on the Consumer Assessment of Health Providers and 

Systems (CAHPS) Clinician-Group Survey to assess 

the experience of Veterans who are receiving care from 

community (i.e., non-VA) providers under authorities 

established in the Veterans Choice Act of 2014. That 

data is identifying performance gaps that are being 

addressed via new processes and policies to improve 

Veteran experience with non-VA care.
128 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you responding 

for?

Department of Labor

Office of the Assistant Secretary for Policy - Chief 

Evaluation Office

If applicable, please provide additional 

information about this most critical barrier.

Recommend expanding access to NDNH data for 

research, evaluation, and statistical purposes while 

also providing strong privacy and confidentiality 

protections. Gaining access to NDNH data is very 

challenging. the PRA process also is very 

cumbersome and time consuming. 

If applicable, please list up to five 

statistical, evaluation, research, or policy 

analysis activities your office, or its 

representatives, have not been able to 

complete because it was too hard to access 

the right data.

If applicable, please provide an example of 

a micro-level data linkage project or 

process in your office that works well. 

If applicable, please provide an example of 

how your office has used data to build 

evidence for policy making.
The Promise of Evidence-Based Policymaking 129

Survey response

Which Federal agency are you 

responding for?

Department of the Treasury

Internal Revenue Service - Statistics of Income Division

If applicable, please provide 

additional information about this 

most critical barrier.

Funding affects our ability to hire adequate staff to support 

outside researchers, develop new products on our own, 

explore new data sources, etc. We have been under a hiring 

freeze for several years and have not hired from outside our 

agency in at least 4. Existing staff have been frozen at 

relatively low journeyman pay levels due to a freeze on 

promotions. All this limits our ability to bring in staff with 

new analytical skills to help the organization embrace newer 

technologies and processes, including tools associated with 

Big Data. Budgets have been flat while expenses have 

increased steadily. We have had some success leveraging 

outside academics to help innovate. Just a few new hires 

would make a significant difference in our program. 

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your 

office, or its representatives, have 

not been able to complete because 

it was too hard to access the right 

data.

If applicable, please provide an 

example of a micro-level data 

linkage project or process in your 

office that works well. 

We are extremely good at linking across tax years and tax 

forms to create longitudinal (panel) files of tax filing 

populations. We successfully link millions of information 

documents (provided by third parties such as SSA, banks, 

investment companies, businesses, etc.) to tax returns. Using 

such linkages we produce annual tabulations of county and 

state-migration, support research projects including 2 

looking at intergenerational income mobility, and produce 

income and pension fund tabulations. 

If applicable, please provide an 

example of how your office has 

used data to build evidence for 

policy making.

A few examples include: Chetty that has shown the 

importance of teacher quality in improving childhood 

outcomes Chetty, Manoli and others have provided valuable 

evidence that is being used to improve uptake of refundable 

credits by affected populations Mahon and Zwick have 

examined the effect of business stimulus programs on small 

businesses behavior during the recession; Hoxby has 

examined the uptake and estimated the effects of education 

tax credits Yagan has examined potential causes of 

individual's unwilling to relocate in the face of job loss 

during the recession
130 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you responding for? Department of Commerce

Bureau of Economic Analysis

If applicable, please list up to five statistical, evaluation, 

research, or policy analysis activities your office, or its 

representatives, have not been able to complete because it 

was too hard to access the right data.

If applicable, please provide an example of a micro-level 

data linkage project or process in your office that works 

well. 

The linkage between BEA 

multinational firm data and Census 

database for foreign trade and econ 

census

If applicable, please provide an example of how your office 

has used data to build evidence for policy making.
The Promise of Evidence-Based Policymaking 131

Survey response

Which Federal agency are you 

responding for?

Department of Homeland Security

Office of Policy - Office of Immigration Statistics

If applicable, please provide additional 

information about this most critical 

barrier.

Most of our resources go to meet basic reporting 

requirements, leaving limited time/personnel for 

research.

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, or 

its representatives, have not been able to 

complete because it was too hard to 

access the right data.

If applicable, please provide an example 

of a micro-level data linkage project or 

process in your office that works well. 

If applicable, please provide an example 

of how your office has used data to 

build evidence for policy making.

We have developed a tool that aggregates several 

different DHS and DOJ data systems to analyze how 

aliens move through the immigration enforcement 

system. The tool identifies bottlenecks and shows how 

different populations are affected by bottlenecks and 

experience different types of outcomes. the tool offers 

insight for policy development and resource allocation.
132 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you 

responding for?

Department of Labor

Bureau of Labor Statistics

If applicable, please provide additional 

information about this most critical 

barrier.

Statutes surrounding the IRS data currently prohibit 

Census from sharing business data that originates from 

IRS filings with BEA and BLS. As a result, Census, 

BEA, and BLS cannot achieve the full benefits of data 

sharing. Specifically, Title 26, Section 6103(j) needs 

edited to allow for BLS access to the data to fully 

achieve the data sharing benefits that the Confidential 

Information Protection and Statistical Efficiency Act 

sought to achieve.

If applicable, please list up to five 

statistical, evaluation, research, or policy 

analysis activities your office, or its 

representatives, have not been able to 

complete because it was too hard to 

access the right data.

Expanded estimates of wages by geography and 

industry. Expanded lay-off statistics. Firm-level 

statistical estimates. 

If applicable, please provide an example 

of a micro-level data linkage project or 

process in your office that works well. 

Linkages with BLS business register

If applicable, please provide an example 

of how your office has used data to build 

evidence for policy making.
The Promise of Evidence-Based Policymaking 133

Survey response

Which Federal agency are you responding for? Social Security Administration

Office of Research, Evaluation, and Statistics

If applicable, please provide additional 

information about this most critical barrier.

If applicable, please list up to five statistical, 

evaluation, research, or policy analysis 

activities your office, or its representatives, 

have not been able to complete because it was 

too hard to access the right data.

If applicable, please provide an example of a 

micro-level data linkage project or process in 

your office that works well. 

Our micro data linkages with the Census 

Bureau's Survey of Income and Program 

Participation and Current Population Survey 

work very well. Also our micro data linkage 

with the University of Michigan's Health and 

Retirement Study is very successful.

If applicable, please provide an example of 

how your office has used data to build 

evidence for policy making.

ORES has successfully used data to build 

microsimulation modelling capabilities for 

analyzing Social Security and SSI reform 

options and the distributional characteristics of 

Social Security beneficiaries and SSI recipients. 

ORES also uses data to study the pension 

environment, mortality patterns, the income 

distribution (poverty), and labor force 

participation, all of which contribute to the 

evidence base for making Social Security policy 

decisions. Finally, ORES supports numerous 

research projects through the Retirement 

Research Consortium and Disability Research 

Consortium, all of which are intended to 

contribute to the evidence base on retirement and 

disability policy.
134 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you responding for? Department of Health and Human Services

Office of Research, Evaluation, and Statistics

If applicable, please provide additional 

information about this most critical barrier.

We are often asked to enter into user agreements 

to obtain the data which our OGC does not 

allow. Often in working with tribal entities we 

need to get tribal IRB or tribal board approval 

which takes forever and is often denied.

If applicable, please list up to five statistical, 

evaluation, research, or policy analysis 

activities your office, or its representatives, 

have not been able to complete because it was 

too hard to access the right data.

meta-analysis using individual-level data

If applicable, please provide an example of a 

micro-level data linkage project or process in 

your office that works well. 

If applicable, please provide an example of 

how your office has used data to build 

evidence for policy making.
The Promise of Evidence-Based Policymaking 135

Survey response

Which Federal agency are you 

responding for?

National Science Foundation

National Center for Science and Engineering Statistic

If applicable, please provide additional 

information about this most critical 

barrier.

If applicable, please list up to five 

statistical, evaluation, research, or policy 

analysis activities your office, or its 

representatives, have not been able to 

complete because it was too hard to 

access the right data.

If applicable, please provide an example 

of a micro-level data linkage project or 

process in your office that works well. 

About every two years we and our contractor engage 

with SSA using SSA data and a micro data file we 

provide to identify doctoral-level scientists and 

engineers who are working in the US. We do not 

directly access SSA data, but it is SSA staff who do 

the match using full SSN and confirm those 

individuals who had wages during the period of 

interest and were therefore resident and working in the 

US. This is helpful, particularly for estimates of the 

inflow and outflow of those doctoral recipients who 

were not US citizens at the time of their degree.

If applicable, please provide an example 

of how your office has used data to build 

evidence for policy making.

The above described data on "stay rates" are included 

in a biennial report to Congress on the state of US 

science and engineering. It serves as evidence of the 

contribution of immigrants to the nation's S&E 

workforce.
136 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you 

responding for?

Department of Health and Human Services

Centers for Disease Control and Prevention - Office of 

the Associate Director for Policy

If applicable, please provide additional 

information about this most critical 

barrier.

If applicable, please list up to five 

statistical, evaluation, research, or policy 

analysis activities your office, or its 

representatives, have not been able to 

complete because it was too hard to 

access the right data.

If applicable, please provide an example 

of a micro-level data linkage project or 

process in your office that works well. 

If applicable, please provide an example 

of how your office has used data to build 

evidence for policy making.

The 6/18 and Health Impact in 5 years initiative both 

represent a process by which my office looked at the 

best available data for interventions that have evidence 

of health outcome improvement and cost effectiveness. 

Highlighting these interventions and their data makes it 

possible for policy makers to rapidly identify the best 

available evidence on these topics. We also used 

Market Scan and CMS data to generate papers on 

topics such as team based care and how team based 

care can lead to improve health outcomes and cost 

effectiveness. 
The Promise of Evidence-Based Policymaking 137

Survey response

Which Federal agency are you responding for? Department of the Treasury

United States Mint

If applicable, please provide additional information 

about this most critical barrier.

Although it may be helpful as a 

manufacturing organization to collect 

data on customers, their privacy 

concerns remain a key focus for us.

If applicable, please list up to five statistical, 

evaluation, research, or policy analysis activities your 

office, or its representatives, have not been able to 

complete because it was too hard to access the right 

data.

Customer data from the internet sales

If applicable, please provide an example of a microlevel data linkage project or process in your office that 

works well. 

If applicable, please provide an example of how your 

office has used data to build evidence for policy 

making.
138 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you 

responding for?

Department of Agriculture

Economic Research Service

If applicable, please provide additional 

information about this most critical 

barrier.

Also, lack of planning for linking by program agencies is 

a factor. Data do not contain identifiers to facilitate 

linking and and other critical data needed for high 

quality analysis; also SORNs often do not indicate as a 

routine use that data may be shared/linked/used for 

statistical research and evaluation purposes

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, 

or its representatives, have not been 

able to complete because it was too 

hard to access the right data.

1. Due to lack of access to USDA's Risk Management 

Agency Micro-level Admin data, we are unable to 

examine choices that farmers make between different 

types of insurance plans as a function of policy relevant 

variables such as the premium subsidy rate. 2. Do FSA 

programs ease credit constraints for farmers? 3. Crop 

insurance and Cash Receipts 4. Farm-labor studies 5. 

Rural manufacturing resilience 6. Impact of patents 7. 

Statistical analysis for SNAP and WIC would be 

strengthened by fuller assessment of who the programs 

reach, and who, among eligibles, are not reached. 

Identification of non-participating eligibles would be 

facilitated by improved access to Federal tax data (for 

annual income data), State UI wage records (for intrayear changes in incomes, which are volatile among the 

working poor), and vital records (for WIC’s infant 

population). Moreover, access to these income- and ageeligibility data supports development of statistical 

control groups that, in the absence of experimental 

designs, provide evidence on how program participation 

affects program outcomes. Linking NHANES to 

administrative data for SNAP and WIC (and to Census 

survey data) would provide results on how the programs 

improve health and nutrition outcomes. Understanding 

how well the program serves important subpopulations, 

such as veterans or the elderly, would be improved by 

access to data on military service and social security 

data. Understanding how SNAP interacts with other 

programs would be promoted by easier access to data on 

benefits from UI, TANF and other human services 

programs, as well as veterans benefits. 

If applicable, please provide an 

example of a micro-level data linkage 

project or process in your office that 

works well. 

1. The U.S. Census Bureau, USDA’s Food and Nutrition 

Service, which administers the Nation’s food and 

nutrition assistance programs, and ERS formed a 

strategic partnership for conducting research on USDA 

food assistance programs. ERS and FNS seek to inform 
The Promise of Evidence-Based Policymaking 139

policymakers and others on: who participates in the 

programs, how program participation affects the lives of 

those individuals, and who does not participate and why. 

The Census Bureau’s interests are primarily to improve 

their surveys, the 2020 Census, and data-linkage 

processes. The joint project is acquiring USDA 

administrative data from States and linking those data to 

Census Bureau surveys. Three ERS reports from the 

project to date have provided evidence on sub-state 

detail on program access, program targeting of benefits 

by income level, and measurement of monthly vs annual 

participation rates. 2. Linking Admin data to Survey 

data: manufacturing resilience, CRP experiments, 

conservation compliance

If applicable, please provide an 

example of how your office has used 

data to build evidence for policy 

making.

The U.S. Census Bureau, USDA’s Food and Nutrition 

Service, which administers the Nation’s food and 

nutrition assistance programs, and ERS formed a 

strategic partnership for conducting research on USDA 

food assistance programs using administrative data and 

linked data sources, obtaining results that are not 

obtainable using any one data source alone. The project 

has multiple goals. Each partner agency receives 

benefits, but different types of benefits support their 

respective missions. As USDA agencies, ERS and FNS 

seek to inform policymakers and others on: who 

participates in the programs, how program participation 

affects the lives of those individuals, and who does not 

participate and why. In contrast, the Census Bureau’s 

interests are primarily to improve their surveys, which 

collect data on food assistance program participation, the 

2020 Census, and data-linkage processes. The joint 

project is acquiring USDA administrative data from 

States. The project is linking those data to Census 

Bureau surveys, leveraging the strengths of each data 

source. Three ERS reports from the project to date have 

provided evidence on sub-state detail on program access, 

program targeting of benefits by income level, and 

measurement of monthly vs annual participation rates. 

FNS has a mission-driven interest in understanding how 

effectively SNAP is reaching the people it is designed to 

serve. The data required to produce detailed estimates 

within a State have previously not been available, and 

these can be helpful as States decide where to focus 

expenditures. The project linked SNAP data for Texas to 

the ACS and developed a new methodology for 

obtaining detailed geographic and demographic results. 
140 The Promise of Evidence-Based Policymaking

The access rate measures the share of eligible people 

who receive SNAP benefits. For Texas as a whole, 

among the people who are estimated to be eligible in the 

ACS, a check of the SNAP records shows that 63% of 

them were participants. Access rates ranges from about 

37 to 77% across Texas congressional districts. The 

Texas report also provided access rates for selected 

demographic subpopulations. Incomes vary across 

households that participate in SNAP. What percentage of 

SNAP households have the “very lowest” income? 

Linked data help address underreporting, income 

variability, and complex household structures (such as 

multi-family households and unrelated individuals). 

Using linked data for New York, the study found that 

SNAP is targeted more towards those with low annual 

incomes than is detected using ACS data alone. 

Comparing the ACS-only measure and the measure 

based on linked-data found that the share of SNAP 

households with annual income less than 50 percent of 

poverty increases from about 18 to 27 percent between 

the two measures, while the share in poverty increases 

from about 51 to 61 percent. The third study address the 

question of who participates in SNAP. People can be 

eligible for SNAP, and participate in the program, in a 

given month. ERS called that approach a “monthly” 

timeframe. Alternatively, in an “annual” timeframe, 

people can be eligible for SNAP at some time during the 

year. These are two different statistical populations. 

Federal surveys often ask: “At some time in the last year 

did you receive SNAP benefits?” If an annual timeframe 

is used for estimating eligibles, it is important to use an 

annual timeframe to measure participation. New York 

SNAP records were used to count the number of people 

who received SNAP for one or more months in the 

annual timeframe. While the monthly participation rate 

for New York was about 80%, the annual rate was about 

75%. The monthly slice-of-time tends to capture more 

people who are long-term eligibles, and they tend to 

participate at higher rates, boosting the monthly rate 

above the annual rate. Access to micro-level data and 

linkages across datasets enabled ERS to provide 

statistical analysis on three key issues of interest to 

policymakers involving participation in the country’s 

largest nutrition assistance program. 
The Promise of Evidence-Based Policymaking 141

Survey response

Which Federal agency are you 

responding for?

Department of Health and Human Services

Administration for Community Living

If applicable, please provide additional 

information about this most critical 

barrier.

The Information Collection Request process under PRA 

has posed a significant barrier to our office’s ability to 

use data. Challenges include: • Process length: The 

length of time the process takes can exceed 12 months. 

The delay alone is problematic but also has a negative 

impact on our ability to award and manage contracts to 

conduct evaluations, research or measure program 

performance. Research and evaluation are generally 

conducted through independent, research firms under 

contract. The uncertainty and length of the ICR process 

leads to contract modifications, extensions and 

increased costs. The delays also impact the ability to 

collect baseline data for new programs and initiatives 

and for those with short life-cycles. • Challenges testing 

data collection instruments. Testing of data collection 

tools (e.g. reliability and validity) if conducted with 

more the 9 individuals must go through a full ICR 

process. This includes public comment periods for 

study questions and approaches that may be altered 

based on the testing results. While public comment on 

questions prior to testing may provide some useful 

feedback a 60 day and 30 day comment period for 

questions and approach that is subject to change is not 

an efficient approach. The process does not 

accommodate a rapid cycle of question testing, revision 

and testing. An evaluation or research study will likely 

undergo two ICR approvals; one for testing and the 

second for implementation which further delays the 

research. • Use and collection of Social Security 

Numbers (SSN) to facilitate linking across data sets: 

Social security numbers are the best, most reliable 

method for linking across data sets. Data security and 

privacy are always a priority when collecting PII and 

our office uses respected research firms with extensive 

experience collecting and protecting PII. We have 

experienced extended delays during the ICR process 

when data collection includes SSN for data linking 

purposes. The extended delays related to SSN 

collection have not resulted in changes or 

improvements in our security protocols, data collection 

methods or question content. 

If applicable, please list up to five 

statistical, evaluation, research, or 

Medicaid data would greatly enhance our ability to 

evaluate program outcomes but the timeliness, 
142 The Promise of Evidence-Based Policymaking

policy analysis activities your office, or 

its representatives, have not been able 

to complete because it was too hard to 

access the right data.

completeness, quality, and cost have led to our decision 

that it is not cost effective to pursue access at this time. 

Micro-level data from our program grantees (states and 

sub-state entities) has been too difficult to access. 

Multiple challenges impact our ability to access grantee 

data including: variability in grantee capacity to collect 

and store micro-level data; lack of data standards; and 

ICR/PRA requirements. 

If applicable, please provide an example 

of a micro-level data linkage project or 

process in your office that works well. 

Our office linked outcome evaluation data with 

Medicare beneficiary data to enable measurement of 

program health outcomes. Under the same project our 

office linked USDA, FNS Store Tracker and 

Redemption Systems (STARS) data to measure food 

access. These data linkages greatly enhance the multicomponent, comprehensive process and outcome 

evaluation.

If applicable, please provide an example 

of how your office has used data to 

build evidence for policy making.

Our office has succeeded in working with CMS to 

obtain Medicare records for program evaluation 

purposes. Medicare beneficiary files have been used to 

identify a matched comparison group as well as link 

data for outcome analysis. The outcome evaluation is 

ongoing and the results will be used to inform policy 

and program decision-making. We have combined 

program process data with outcome and cost findings 

from CMS’s Evaluation of Community-based Wellness 

and Prevention Programs. Results were shared with 

Congress and are used by program and policy decisionmakers regarding evidence-based health promotion and 

disease prevention interventions. 
The Promise of Evidence-Based Policymaking 143

Survey response

Which Federal agency are you 

responding for?

Department of Health and Human Services

Agency for Healthcare Research and Quality

If applicable, please provide additional 

information about this most critical 

barrier.

Many Agencies have restrictive requirements or 

restrictive interpretations of confidentiality laws and 

regulations that make it difficult to access valuable 

supplemental data.

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, or 

its representatives, have not been able 

to complete because it was too hard to 

access the right data.

There are a number of instances in which imputation or 

editing of survey data collected by or for our Agency 

could have been improved with access to detailed 

administrative data collected by another Agency.

If applicable, please provide an 

example of a micro-level data linkage 

project or process in your office that 

works well. 

We have directly linked Medicare and Medicaid 

administrative data to our survey data to support 

methodological research that led to improvements in the 

data subsequently used in statistical and policy related 

analyses.

If applicable, please provide an 

example of how your office has used 

data to build evidence for policy 

making.

Research using AHRQ's Medical Expenditure Panel 

Survey was used to inform decisions regarding the 

reauthorization of the Children's Health Insurance 

Program (CHIP) in 2015. Applied research from our 

office based upon MEPS data led to the formulation and 

adoption of two CHIP policy recommendations by the 

Medicaid and CHIP Payment Advisory Commission 

(MACPAC): • Eliminate CHIP premiums for children in 

families under 150 percent of the Federal Poverty Line 

(FPL). • Reauthorize CHIP in 2015. These 

recommendations were accepted by Congress. 
144 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you responding 

for?

Department of Health and Human Services

Centers for Disease Control and Prevention -

National Center for HIV/AIDS, Viral Hepatitis, 

Sexually Transmitted Disease, and Tuberculosis 

Prevention

If applicable, please provide additional 

information about this most critical barrier.

Too many information collection requests are 

delayed at OMB for reasons beyond the control of 

CDC programs.

If applicable, please list up to five statistical, 

evaluation, research, or policy analysis 

activities your office, or its representatives, 

have not been able to complete because it 

was too hard to access the right data.

not applicable

If applicable, please provide an example of a 

micro-level data linkage project or process in 

your office that works well. 

not applicable

If applicable, please provide an example of 

how your office has used data to build 

evidence for policy making.

CDC used U.S. national HIV surveillance data and 

the results of systematic reviews of the literature 

on randomized, controlled trials to recommend the 

use of pre-exposure prophylaxis to prevent HIV 

infection in HIV negative persons at high risk of 

infection 
The Promise of Evidence-Based Policymaking 145

Survey response

Which Federal agency are you 

responding for?

Department of Education

National Center for Education Statistics

If applicable, please provide additional 

information about this most critical 

barrier.

My agency continues to receive funding for additional 

data collections, but does not receive support for 

additional staff to conduct those studies.

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, or 

its representatives, have not been able to 

complete because it was too hard to 

access the right data.

If applicable, please provide an example 

of a micro-level data linkage project or 

process in your office that works well. 

My agency routinely links sample survey data to 

student application and financial aid data to increase 

our nation's understanding of the use and outcomes of 

student financial aid. My agency is collaborating with 

the Veteran's Benefit Administration to identify 

Veterans among the students enrolled in sampled 

institutions for our postsecondary student aid studies; 

this will allow us to oversample Veterans in those 

studies, thus providing our agency and the Veteran's 

Administration with valuable data on the use of 

Veteran's benefits and the educational outcomes of 

Veterans.

If applicable, please provide an example 

of how your office has used data to 

build evidence for policy making.

Our agency's postsecondary student aid studies provide 

the information base that is used to inform policy 

decisions regarding student financial aid. Our agency's 

National Assessment of Educational Progress is the 

longest standing nationally representative assessment of 

student performance. It serves as a benchmark for 

numerous other state specific assessments. Our 

agency's international assessments provide the 

information base needed to evaluate the academic and 

real life performance of our students and adults relative 

to the rest of the world.
146 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you 

responding for? Nuclear Regulatory Commission

If applicable, please provide additional 

information about this most critical 

barrier.

If applicable, please list up to five 

statistical, evaluation, research, or policy 

analysis activities your office, or its 

representatives, have not been able to 

complete because it was too hard to 

access the right data.

If applicable, please provide an example 

of a micro-level data linkage project or 

process in your office that works well. 

If applicable, please provide an example 

of how your office has used data to build 

evidence for policy making.

The U.S. Nuclear Regulatory Commission (NRC) 

would like to provide the following supporting 

information to describe the context for the agency’s 

response to the question above on collecting data. The 

NRC’s mission is to license and regulate the Nation's 

civilian use of radioactive materials to protect public 

health and safety, promote the common defense and 

security, and protect the environment. In implementing 

this mission, NRC requires its regulated entities, 

including licensees, certificate holders, vendors, and 

applicants, to report information to the agency in 

accordance with specific regulatory requirements. 

Examples include qualifications for obtaining licenses, 

notifications of incidents, operating power levels, and 

biannual financial reports. NRC then evaluates the 

reported information and takes necessary regulatory 

actions in accordance with its policies and procedures. 

Should the U.S. Commission on Evidence-Based 

Policymaking be interested in such data, the NRC can 

work with the Commission to provide further detail. 
The Promise of Evidence-Based Policymaking 147

Survey response

Which Federal agency are you responding for?

Department of Justice

Office of Justice Programs -

Bureau of Justice Statistics

If applicable, please provide additional information 

about this most critical barrier.

The different statutes among the different 

agencies make it difficult to come to an 

agreement.

If applicable, please list up to five statistical, 

evaluation, research, or policy analysis activities your 

office, or its representatives, have not been able to 

complete because it was too hard to access the right 

data.

If applicable, please provide an example of a microlevel data linkage project or process in your office that 

works well. 

If applicable, please provide an example of how your 

office has used data to build evidence for policy 

making.
148 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you responding for? Small Business Administration

Office of Performance Management

If applicable, please provide additional information 

about this most critical barrier.

Less than 1% of the budget is set aside 

for evaluation work

If applicable, please list up to five statistical, evaluation, 

research, or policy analysis activities your office, or its 

representatives, have not been able to complete because 

it was too hard to access the right data.

Getting PII to link clients across 

program within the SBA Obtaining the 

counterfactual (adequate comparison 

group) 

If applicable, please provide an example of a micro-level 

data linkage project or process in your office that works 

well. 

If applicable, please provide an example of how your 

office has used data to build evidence for policy making.

Strategic Plan Development Annual 

Performance Plan Congressional 

Budget Justification Quarterly Reports 

Strategic Reviews 
The Promise of Evidence-Based Policymaking 149

Survey response

Which Federal agency are you 

responding for?

Department of Justice

Office of Justice Programs - Office of Juvenile Justice 

and Delinquency Prevention

If applicable, please provide additional 

information about this most critical 

barrier.

OJJDP has not received an appropriation for Part D of 

the JJDPA for over a decade. Consequently, all 

research and data collection must be funded through 

statutory authorities which allow us to use up to 10 

percent of programmatic funding for research. This 

funding is extremely limited. 

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, or 

its representatives, have not been able to 

complete because it was too hard to 

access the right data.

If applicable, please provide an example 

of a micro-level data linkage project or 

process in your office that works well. 

If applicable, please provide an example 

of how your office has used data to 

build evidence for policy making.

Most of the research, evaluation, and data OJJDP 

implements informs the office's policies and practice. 

For example, our Census of Juveniles in Residential 

Placement and Juvenile Residential Facility Census 

provide information about youth in juvenile residential 

facilities and the conditions of confinement including 

accesses to services. These data inform the office's 

efforts to enact juvenile justice reforms.
150 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you 

responding for? National Aeronautics and Space Administration

If applicable, please provide additional 

information about this most critical 

barrier.

The NASA Office of Education (OE) is currently 

undertaking a pilot effort to survey participants in its 

science, technology, engineering, and math (STEM) 

education programs. This effort will help NASA better 

gauge the outcomes of its education programs, and will 

enable data-driven decisions about how best to use our 

resources. Since we are still in the pilot stages of this 

effort, we cannot address all of the questions included 

in the survey above. Our intent is to eventually collect 

outcome data across all of OE's lines of business. OE 

receives funding of approximately $100 million each 

year, which is only around 0.5 percent of NASA's total 

budget. (Note that while NASA Education spends 

between 1-10 percent of its budget on evaluations, for 

the Agency as a whole, the spending is only a fraction 

of a percent.) There are resource trade-offs involved in 

spending our limited budget on studies to evaluate the 

effectiveness of our STEM education programs, rather 

than on the programs themselves. 

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, or

its representatives, have not been able 

to complete because it was too hard to 

access the right data.

If applicable, please provide an 

example of a micro-level data linkage 

project or process in your office that 

works well. 

If applicable, please provide an 

example of how your office has used 

data to build evidence for policy 

making.

The NASA Office of Education (OE) performs a critical 

role in advancing the Nation’s science, technology, 

engineering, and mathematics (STEM) education and 

workforce pipeline by working collaboratively with 

other agencies to engage students, teachers, and faculty 

in NASA’s missions and unique assets. In recent years, 

OE has worked on both expanding its existing efforts 

and developing new capabilities to objectively 

determine the impact of its programs. Following are 

some of OE’s recent and ongoing efforts to build 

strong, outcome-oriented performance measures: 

•Introducing a new annual performance indicator in FY 

2017 that tracks evaluations conducted by grantees and 
The Promise of Evidence-Based Policymaking 151

cooperative agreement awardees on educational 

activities; •Providing technical assistance to Center 

education offices and selected grantees and cooperative 

agreement awardees to ensure that educational 

investments are appropriately evaluated using common 

metrics and findings used to improve programmatic 

outcomes; and •Obtaining a generic clearance for 

methodological testing. This clearance is the first of its 

kind for NASA and allows OE to significantly enhance 

the quality of the its data collection instruments and 

overall data management through interdisciplinary 

scientific research, utilizing best practices in 

educational, psychological, and statistical measurement.
152 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you 

responding for?

Small Business Administration

Office of Advocacy

If applicable, please provide additional 

information about this most critical 

barrier.

Office would like to create new data on regulatory 

burden and other detailed data on small business but 

research budget is inadequate for the scale required for 

this type of data collection.

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, or 

its representatives, have not been able 

to complete because it was too hard to 

access the right data.

TAS, Employee Benefits and student debt as it related 

to entrepreneurism.

If applicable, please provide an 

example of a micro-level data linkage 

project or process in your office that 

works well. 

If applicable, please provide an 

example of how your office has used 

data to build evidence for policy 

making.

In accordance with our statutory mission of examining 

the role of small business in the American economy and 

recommending measures for creating an environment in 

which all businesses have the opportunity to compete 

effectively, the Office of Advocacy actively supports 

the use of evidence based policy making. As part of 

achieving this mission, the Office generates original 

data which is used for research products and by other 

regulatory agencies in the crafting of policy. For one 

specific example, the Office of Advocacy sponsors the 

Census Bureau’s annual Statistics of U.S. Businesses 

(SUSB) survey. SUSB provides comprehensive data on 

U.S. employer businesses by geography, industry, 

receipt size, employment size, and legal form of 

organization. Advocacy uses the SUSB data for a 

multitude of research projects such as our annual State 

Profiles. The State Profiles provide, among other 

information, the number of small businesses and their 

employment in each state by industry. Due to their 

comprehensive nature, the State Profiles have proven to 

be a key source of small business information for 

Congress, State policy makers and small business 

associations. The SUSB is also an important source of 

business data for regulatory agencies in the rulemaking 

process. For almost all rules affecting business, 

Agencies use the SUSB data in policy development, 

Regulatory Impact Analyses and for compliance with 

the Regulatory Flexibility Act. Data from sources such 
The Promise of Evidence-Based Policymaking 153

as the SUSB can directly alter policy decisions. For 

example, Advocacy engaged with DOJ Civil Rights 

division on a rule regarding the use of captioning and 

audio description devices in movie theatres. After 

further analysis of data from the SUSB, DOJ found their 

rule may not be economically feasible for many smaller 

movie theatres, and added an exemption for theatres 

using analog projection. Doing so allowed DOJ to 

achieve their policy objective at a smaller cost to 

society. Advocacy continues to use the best available 

data from SUSB and other sources to provide actionable 

research to policy makers and bolster regulatory 

agency’s development of evidence based policy. 
154 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you responding for? Department of the Interior

National Park Service

If applicable, please provide additional information about this most 

critical barrier.

If applicable, please list up to five statistical, evaluation, research, 

or policy analysis activities your office, or its representatives, have 

not been able to complete because it was too hard to access the 

right data.

If applicable, please provide an example of a micro-level data 

linkage project or process in your office that works well. 

If applicable, please provide an example of how your office has 

used data to build evidence for policy making.

We have field and program 

staff review draft guidance 

and comment on it.
The Promise of Evidence-Based Policymaking 155

Survey response

Which Federal agency are you responding 

for?

Department of Commerce

Bureau of the Census

If applicable, please provide additional 

information about this most critical barrier.

We have great authority to access administrative 

data, yet there are some files which would greatly 

advance our mission that we either cannot access 

or have great difficulty accessing. The NDNH and 

SNAP/WIC are examples of these two categories.

If applicable, please list up to five statistical, 

evaluation, research, or policy analysis 

activities your office, or its representatives, 

have not been able to complete because it 

was too hard to access the right data.

Full use of SNAP data to improve the accuracy of 

the 2020 Decennial Census Use of the NDNH to 

reduce the cost of the 2020 Decennial Census Full 

implementation of CIPSEA to share our business 

register with BLS and vice versa

If applicable, please provide an example of a 

micro-level data linkage project or process 

in your office that works well. 

The work conducted in the FSRDCs is a great 

success.

If applicable, please provide an example of 

how your office has used data to build 

evidence for policy making.

The American Community Provides information 

that can be used by policy makers. We do not make 

policy.
156 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you responding for?

Environmental Protection Agency

Office of Land and Emergency 

Management

If applicable, please provide additional information 

about this most critical barrier.

Often we come across questions that 

would require surveys and we don't even 

seriously contemplate requesting a ICR 

due to the time commitment. 

If applicable, please list up to five statistical, 

evaluation, research, or policy analysis activities your 

office, or its representatives, have not been able to 

complete because it was too hard to access the right 

data.

survey of voluntary partnerships; 

If applicable, please provide an example of a microlevel data linkage project or process in your office that 

works well. 

If applicable, please provide an example of how your 

office has used data to build evidence for policy 

making.

we have used evaluations to make 

changes to a program. 
The Promise of Evidence-Based Policymaking 157

Survey response

Which Federal agency are you 

responding for? Corporation for National and Community Service

If applicable, please provide additional 

information about this most critical 

barrier.

Our office and the projects we develop and implement 

are relatively new ways of doing business for this 

agency and it has tested every policy, practice and 

norm the agency has around data and its use.

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, or 

its representatives, have not been able to 

complete because it was too hard to 

access the right data.

Linking our member data to UI and New Hires data 

Conducting longitudinal research using state 

administrative data on a sample of students in an RCT 

study we conducted (didn't secure the needed consents 

initially)

If applicable, please provide an example 

of a micro-level data linkage project or 

process in your office that works well. 

If applicable, please provide an example 

of how your office has used data to 

build evidence for policy making.

We have linked our national service member data to the 

SSA disability data to determine whether or not a year 

of service can facilitate transitions to the workforce for 

individuals with disabilities. We have linked our 

national service member data to the National student 

clearinghouse to track education outcomes for members 

over time. We are trying to link member data to the 

LEHD survey data to track employment outcomes for 

members over time. We have a Social Innovation Fund 

Pay For Success Administrative Data Pilot underway 

where grantees facilitate access to various 

administrative data sets for PFS grantees to help build 

evidence in a more cost-effective way. Our office looks 

forward to receiving guidance from the CEP so that we 

can better understand the following: 1. Successful 

frameworks, policies, practices, and methods to 

overcome challenges related to evidence-building from 

state, local, and/or international governments for 

Federal evidence-based policymaking 2. Based on 

identified best practices and existing examples, the 

factors that should be considered in reasonably 

ensuring the security and privacy of administrative and 

survey data? Our office would also love to learn what 

the CEP has discovered through their fact-finding 

regarding: What are the main policy questions 

legislators and the executive branch seek to answer 

through administrative data? What are the most 

frequent, the most urgent, the most important? What 

are the challenges, opportunities, issues, barriers to 
158 The Promise of Evidence-Based Policymaking

addressing them through administrative data? What are 

the questions that CBO, CRS, GAO get asked the most, 

what information do they most need? What new 

legislation is needed and what can you do with the 

existing legislation? 
The Promise of Evidence-Based Policymaking 159

Survey response

Which Federal agency are you responding for?

Department of Education

Office of Special Education and 

Rehabilitative Services

If applicable, please provide additional 

information about this most critical barrier.

Having sufficient resources to hire, put IT 

systems in place and making statistics, 

research a priority

If applicable, please list up to five statistical, 

evaluation, research, or policy analysis activities 

your office, or its representatives, have not been 

able to complete because it was too hard to access 

the right data.

N/A

If applicable, please provide an example of a 

micro-level data linkage project or process in your 

office that works well. 

If applicable, please provide an example of how 

your office has used data to build evidence for 

policy making.

In our competitive grant programs, applicants 

for funding consideration must demonstrate 

how they intend to address evidence in their 

application and in their project should they be 

funded. 
160 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you 

responding for?

Social Security Administration

Office of Data Exchange and Policy Publications

If applicable, please provide additional 

information about this most critical 

barrier.

The most critical barrier to data exchange is legal and 

disclosure limitations. SSA often does not have legal 

authority to disclose requested data to other agencies or 

researchers (universities or research organization) for 

the requested purpose. Statutes and systems of records 

notices (routine uses) generally (with certain 

exceptions) limit our authority to disclose SSA data to 

research uses that are relevant to the administration of 

the Social Security Act.

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, or 

its representatives, have not been able to 

complete because it was too hard to 

access the right data.

If applicable, please provide an example 

of a micro-level data linkage project or 

process in your office that works well. 

If applicable, please provide an example 

of how your office has used data to 

build evidence for policy making.

My office is responsible for data exchange policy and 

operations at the Social Security Administration (SSA). 

We provide enterprise-wide, executive-level leadership, 

oversight, and support for SSA's data exchange 

programs. We work closely with SSA components 

(operational, research, legal, security, legislative) and 

external partners (federal/state agencies, private 

entities, universities and other research organizations) 

to develop and implement data exchange agreements 

for both programmatic and research purposes. On the 

research side, we seek to obtain data needed by SSA 

components to conduct research and statistical 

activities and to provide SSA data to others for research 

and statistical purposes.
The Promise of Evidence-Based Policymaking 161

Survey response

Which Federal agency are you responding for? Department of the Interior

United States Fish and Wildlife Service

If applicable, please provide additional information 

about this most critical barrier.

The approval processes for IT 

transformation in our bureau are subjected 

to a team approach, and people on the team 

often have other funding priorities.

If applicable, please list up to five statistical, 

evaluation, research, or policy analysis activities 

your office, or its representatives, have not been 

able to complete because it was too hard to access 

the right data.

If applicable, please provide an example of a 

micro-level data linkage project or process in your 

office that works well. 

Hiring systems to assist managers.

If applicable, please provide an example of how 

your office has used data to build evidence for 

policy making.

We use demographic and skills information 

for placement. We use workforce 

information to determine skills gaps. We 

use workforce information to project 

funding needs.
162 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you 

responding for?

Department of Transportation

Federal Motor Carrier Safety Administration

If applicable, please provide additional 

information about this most critical 

barrier.

The challenge is that there are numerous legacy IT 

systems that have valuable information and those 

systems were not set up to support comprehensive data 

analyses and queries. And we have difficulties getting 

information out of these legacy Federal and State 

systems. As we develop the IT systems to support new 

programs we apply the lessons learned to ensure those 

systems are structured in such a way that we can 

perform a wide range of analyses.

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, or 

its representatives, have not been able 

to complete because it was too hard to 

access the right data.

1. Determining the number of active commercial 

driver's license (CDL) holders in the U.S.; 2. 

Determining the number of new CDL holders entering 

the trucking and motorcoach industry each year; 3. 

Determining the number of Class A versus Class B 

CDL holders for #1 and #2 above; 4. Determining the 

average amount of behind the wheel training that CDL 

applicants receive when they attend the established 

training schools. This information was needed to help 

with the analyses to support my agency's entry-level 

driver training rulemaking - a statutorily mandated 

rulemaking. We completed the rulemaking through the 

negotiated rulemaking process so we were able to 

achieve a consensus among stakeholders about the 

content of the rule, but we were not able to come up 

with solid information from our own data or the data the 

State driver licensing agencies maintain. 

If applicable, please provide an 

example of a micro-level data linkage 

project or process in your office that 

works well. 

An example of a micro-level data project that has 

worked well for my office is our new National Registry 

of Certified Medical Examiners Program. Through this 

program we established minimum training and testing 

requirements for all State-licensed healthcare providers 

that perform the DOT medical exam for interstate truck 

and bus drivers. Each of the more than 52,000 

examiners must submit monthly reports to the agency 

identifying each person who is examined and the results 

of the examination (e.g., pass or fail, and the length of 

the medical certification (with a maximum length of 2 

years). The system has enabled us to determine the 

number of active truck and bus drivers and provided 

information on the percentage of drivers with health 

issues that preclude the issuance of a 2-year medical 

card. The system has also provided alarming 
The Promise of Evidence-Based Policymaking 163

information about the number of drivers who were 

medically disqualified. We can also conduct analyses to 

identify medical examiners who conduct an unusually 

large number of examinations and compare the 

performance of the various types of healthcare 

providers, as well as geographic variations by 

examiners and State of registration of the drivers.

If applicable, please provide an 

example of how your office has used 

data to build evidence for policy 

making.

One of the more recent examples involved policy 

making concerning the operation of 9-to-15 passenger 

vehicles in interstate commerce. The agency was faced 

with a statutory mandate to impose safety regulations on 

9-to-15 passenger vehicles operated for direct 

compensation in interstate commerce, but we were 

given discretion to focus on a subset that we determine 

pose the greatest risk. To assess the risks, my agency 

reviewed data from the National Highway Traffic 

Safety Administration's Fatality Analysis Reporting 

Systems (FARS). FARS data was of sufficient detail to 

enable us to identify the number of fatal crashes 

involving these small passenger-carrying vehicles, the 

number of crashes that happened while the vehicle was 

actually transporting between 9 and 15 individuals, the 

location of the crash (State and zip code), the state of 

registration for the vehicle and the state of licensure of 

the driver along with the zip code for the driver's 

residence. Using certain assumptions (e.g., Federal 

Highway Administration survey results on the average 

distance individuals commute to work), we were able to 

make the argument that operations that involved 

distances of more than 75 air-miles represent the 

greatest safety risk. We subsequently decided to move 

forward with the rulemaking focused on 9-to-15 

passenger operations traveling more than 75 air-miles to 

address the safety challenge while minimizing to the 

greatest extent the regulatory burden on short-distance 

operations of these vehicles. Although this regulatory 

approach appeared to satisfy most stakeholders, certain 

parties lobbied Congress to enact legislation several 

years later to force the agency to impose the rules on all 

the 9-to-15 passenger vehicles operated for direct 

compensation, irrespective of the data analysis and 

evidence.
164 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you 

responding for?

Department of State

Evaluation and Aid Effectiveness Branch, Office of US 

Foreign Assistance Resources

If applicable, please provide additional 

information about this most critical 

barrier.

We would use data to analyze and allocate budget funds. 

So many are already earmarked or are political 

initiatives that very little latitude for analysis is left.

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, or 

its representatives, have not been able 

to complete because it was too hard to 

access the right data.

N/A

If applicable, please provide an 

example of a micro-level data linkage 

project or process in your office that 

works well. 

If applicable, please provide an 

example of how your office has used 

data to build evidence for policy 

making.

We contracted for an evaluation of the Managing for 

Results process that F uses, which contains steps for 

managing, planning, budgeting and learning. Evaluators 

looked Department wide at the processes executed 

during the program cycle, and determined gaps and 

where streamlining was needed. F has used the 

recommendations to change processes, develop capacity 

building where needed, and restructure communication 

to the Department at large.
The Promise of Evidence-Based Policymaking 165

Survey response

Which Federal agency are you responding for?

Department of Health and 

Human Services

Office of Minority Health

If applicable, please provide additional information about this 

most critical barrier.

Office is understaffed, thus 

the time and attention 

necessary for analysis suffers,

If applicable, please list up to five statistical, evaluation, research, 

or policy analysis activities your office, or its representatives, 

have not been able to complete because it was too hard to access 

the right data.

If applicable, please provide an example of a micro-level data 

linkage project or process in your office that works well. 

If applicable, please provide an example of how your office has 

used data to build evidence for policy making.
166 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you 

responding for?

Department of Health and Human Services

Centers for Medicare and Medicaid Services - Office 

of Enterprise Data and Analytics

If applicable, please provide additional 

information about this most critical 

barrier.

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, or 

its representatives, have not been able to 

complete because it was too hard to 

access the right data.

If applicable, please provide an example 

of a micro-level data linkage project or 

process in your office that works well. 

If applicable, please provide an example 

of how your office has used data to 

build evidence for policy making.

CMS creates administrative Medicare claims data 

through the operation of the Medicare fee-for-service 

program. OEDA has used this data to develop public 

use files that present information on Medicare billing 

practices at the provider-level. For example, data 

released on skilled nursing facilities includes 

information on whether facilities are providing just 

enough therapy minutes to qualify for a higher 

payment (number of therapy minutes is 10 minutes or 

fewer above the payment threshold). This data can be 

used to identify facilities that are outliers in their 

billing practices or analyzed to determine if a policy 

change is needed to address this activity. 
The Promise of Evidence-Based Policymaking 167

Survey response

Which Federal agency are you 

responding for?

Department of Energy

Energy Information Administration

If applicable, please provide additional 

information about this most critical 

barrier.

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, or 

its representatives, have not been able to 

complete because it was too hard to 

access the right data.

If applicable, please provide an example 

of a micro-level data linkage project or 

process in your office that works well. 

If applicable, please provide an example 

of how your office has used data to 

build evidence for policy making.

Oil Export Policy: During 2014 and 2015 both 

Congress and the Administration used EIA data on 

domestic oil production, oil imports, and refinery inputs 

to assess the implications relaxing restrictions on 

exports of domestically-produced crude oil. Using both 

its data and modeling capabilities, EIA provided data 

and analytic reports that showed how different policy 

choices might affect gasoline and crude oil prices, the 

level of domestic crude oil production, domestic 

refiners, and trade in petroleum products. EIA’s 

residential Energy Consumption Survey (RECS) is 

used by several DOE offices and other government 

agencies. A few examples are: • the U.S. Department of 

Health and Human Services, Administration for 

Children and Families (HHS/ACF) to support 

evaluations of the LIHEAP program, which distributes 

energy assistance to more than 8 million low-income 

households annually to defray home heating and 

cooling costs. • The Department of Energy, Office of 

Energy Efficiency and Renewable Energy (EERE) uses 

to support the Appliances and Commercial Equipment 

Standards Program to determine the adequacy of the 

payback period for consumer of efficiency 

improvements. • U.S. Bureau of Labor Statistics (BLS) 

uses RECS data microdata file in the preparation of the 

Consumer Price Index (CPI). In particular, RECS 

microdata are the principal source for estimating utility 

costs for renters whose utility costs are included in their 

rent, which represents 20 percent of the renter sample 

used in the CPI. As a result, BLS does not have to field 
168 The Promise of Evidence-Based Policymaking

an additional survey to account for these utility costs. 

The CPI is, of course, is the principal indicator of the 

U.S. for understanding changes in cost living. 
The Promise of Evidence-Based Policymaking 169

Survey response

Which Federal agency are you 

responding for?

Department of Housing and Urban Development

Policy Development and Research

If applicable, please provide additional 

information about this most critical 

barrier.

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, or 

its representatives, have not been able 

to complete because it was too hard to 

access the right data.

We complete our activities. However, it takes way too 

long to find champions and executive support in other 

agencies, and the process to access some of the key data 

sources is way too bureaucratic. We/our representatives 

link to ACS, AHS, NDNH, NHIS/NHANES, IRS, 

FAFSA, VA, etc. 

If applicable, please provide an 

example of a micro-level data linkage 

project or process in your office that 

works well. 

We/our representatives link to ACS, AHS, NDNH, 

NHIS/NHANES, IRS, FAFSA, VA, etc. Some of our 

most successful data linkage projects have been with 

organizations with a RDCs already in place.

If applicable, please provide an 

example of how your office has used 

data to build evidence for policy 

making.

The Family Options demonstration tested 4 policy 

options for assisting homeless families, using random 

assignment of over 2000 families in a dozen sites 

ranging from Boston to Honolulu: Housing Choice 

Vouchers, Project-Based Transitional Housing, Rapid 

Rehousing, and Usual Care (usually paid from nonFederal sources). Vouchers had substantial quantifiable 

benefits relative to the other interventions, and Rapid 

Rehousing had the lowest quantifiable cost to society. 

From the perspective of ending family homelessness in 

the US, Project-Based Transitional Housing appears to 

have very little to offer, and the Office of Community 

Planning and Development at HUD has issued guidance 

tending to reduce the use of PBTH with Federal funds. 

HUD annually allocates between $1 and $2 billion a 

year for the administration of Housing Choice 

Vouchers, but the allocation system had never been 

based on evidence, and the line item could not be 

justified to appropriators in the absence of evidence. 

The HCV Administrative Cost Study produced a finely 

calibrated allocation formula based on a time and 

motion study at 60 high-performing PHAs throughout 

the US. The Office of Public and Indian Housing at 

HUD has used the study to move the HCV program to a 

new administrative fee system. HUD currently is 

running large-scale randomized experiments to 

determine: A. The efficacy of Housing Counseling for 

prospective first-time homebuyers B. The effects of the 

Family Self-Sufficiency program, especially in 
170 The Promise of Evidence-Based Policymaking

increasing employment and earnings among HCV 

families C. The effects of Rent Reform in the voucher 

program, especially on employment and termination of 

assistance HUD is also conducting paired comparison 

evaluations: 1. Rental Assistance Demonstration, in 

which 24 public housing projects undergoing 

conversion to a private-owner assisted model are 

compared with 48 similar projects not undergoing 

conversion 2. Small Area Fair Market Rents, examining 

the experience of 5 PHAs selected by lottery from a 

cluster of similar PHAs, where voucher subsidies are set 

at the zip code level rather than the metropolitan level to 

encourage deconcentration. 
The Promise of Evidence-Based Policymaking 171

Survey response

Which Federal agency are you 

responding for?

Department of Health and Human Services

National Institutes of Health – NIH Office of Science 

Policy

If applicable, please provide 

additional information about this 

most critical barrier.

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your 

office, or its representatives, have 

not been able to complete because it 

was too hard to access the right 

data.

1. Economic analyses of substance use disorder treatment; 

long term outcomes of addiction treatment. 2. accurate 

analysis of the greater research landscape, such as 

international research or biomedical industry research 3. 

Assessing the degree of NIH’s contributions to FDA new 

drug approvals in a comprehensive way. This work has 

been hampered by lack of open-source, standardized FDA 

data on the research evidence cited in FDA new drug 

applications, including the specific clinical trials considered 

in the application, as well as the limitations of search and 

export capabilities within FDA’s publicly accessible 

databases (e.g., Drugs@FDA). 4. Linking data on NIHfunded research to health interventions to healthcare 

utilization data. This is a complex undertaking that would 

require linking high-quality structured data from NIH and 

several other Operating Divisions of HHS. While not the 

most limiting factor, the cost of access to CMS utilization 

data within their research data center is an added barrier. 

If applicable, please provide an 

example of a micro-level data 

linkage project or process in your 

office that works well. 

If applicable, please provide an 

example of how your office has 

used data to build evidence for 

policy making.

1. Portfolio analysis informed expert panels on current 

funding and research gaps. Challenge competition for 

Audacious Goals identified stakeholder priorities for 

initiative planning. 2. Using NSDUH, MTF, and CDC 

Wonder data to present the scope of substance use disorder 

and overdose in the US 3. Administrative data was used to 

make a policy/program decision and create the Trailblazer 

R21 Program, a new research grant program focused on 

early career investigators that makes use of the 

Exploratory/Developmental R21 grant mechanism. This 

program emphasizes that no preliminary data is needed in 

grant applications and gives priority to early career 

investigators. Anecdotal information, which was later 

confirmed through administrative data analysis, indicated 

that high quality applications were not being funded, with 

reviewers commenting that they wanted preliminary data; 
172 The Promise of Evidence-Based Policymaking

thus senior investigators were beating out early career 

investigators as competition for funding, in general, 

increased. R21 grants were not being used for the purpose 

that they were meant to be, and instead were being used to 

fund more traditional research. As a result of this analysis, a 

portion of R21 funds was set aside for the new Trailblazer 

R21 program. 4. Additionally, the NIH Office of Science 

Policy compiled several examples of NIH program 

evaluations and other analyses performed which utilized 

administrative and other statistical data: Assessing Research 

Training and Workforce Needs/Outcomes • Race, Ethnicity, 

and NIH research awards. Ginther et al. Science 

333(6045):1015-9 2011. An analysis of NIH grant funding 

and peer review outcomes that finds several disparities in 

the process related to investigators’ race and/or ethnicity. 

Although proposals with strong priority scores were equally 

likely to be funded regardless of race, the authors find that 

Asians are 4 percentage points and black or AfricanAmerican applicants are 13 percentage points less likely to 

receive NIH investigator-initiated research funding 

compared with whites. The group conducted a follow up 

study investigating the intersection of gender and 

race/ethnicity, Gender, Race/Ethnicity, and National 

Institutes of Health R01 Research Awards: Is There 

Evidence of a Double Bind for Women of Color? Ginther et 

al. Acad. Med. 91(8):1098-107 2016. The study found that 

white women PhDs and MDs were as likely as white men to 

receive an R01 award, and that compared with white 

women, Asian and black women PhDs and black women 

MDs were significantly less likely to receive funding. • 

Biomedical Workforce Working Group Report. NIH 

Advisory Committee to the Director, 2012. A working 

group of the NIH Advisory Committee to the Director was 

tasked with developing a model for a sustainable and 

diverse U.S. biomedical research workforce that could 

inform decisions about training the optimal number of 

people for the appropriate types of positions to advance 

science and promote health. • Outcome Evaluation of the 

NCI Career Development (K) Awards Program. Discovery 

Logic, 2012. An evaluation of the outcomes of a set of 

career development awards for the National Cancer 

Institute. • Evaluation: National Institutes of Health 

Individual Mentored Career Development Awards Program. 

Discovery Logic, 2011. An evaluation of outcomes from a 

particular set of individual mentored awards sponsored by 

NIH’s Office of Extramural Research. Assessing Research 
The Promise of Evidence-Based Policymaking 173

Program Outcomes and Scientific Productivity • Predicting 

Productivity Returns on Investment: Thirty Years of Peer 

Review, Grant Funding, and Publication of Highly Cited 

Papers at the National Heart, Lung, and Blood Institute. 

Lauer et al. Circulation Research 117(3) 2015 An analysis 

of whether peer review percentile rankings predict grant 

productivity, as measured through publications and 

citations, based on 6873 cardiovascular R01 grants funded 

by the National Heart Lung and Blood Institute in 1980-

2011. The authors find a modest association, but poor 

discrimination, and a variety of mixed results. • Association 

of percentile ranking with citation impact and productivity 

in a large cohort of de novo NIMH-funded R01 grants. 

Doyle et al. Molecular Psychiatry 20(9) 2015. This study 

analyzed the citation impact and productivity for 1755 de 

novo investigator-initiated R01 grants funded for at least 2 

years by National Institute of Mental Health between 2000 

and 2009. The authors found no association between grant 

percentile ranking and subsequent productivity and citation 

impact, even after accounting for subject categories, years 

of publication, duration and amounts of funding, as well as 

a number of investigator-specific measures. • Research 

funding. Big names or big ideas: do peer-review panels 

select the best science proposals? Li et al. Science 

348(6233):434-8 2015. This paper examines the success of 

peer-review panels in predicting the future quality of 

proposed research. The authors construct new data to track 

publication, citation, and patenting outcomes associated 

with more than 130,000 research project (R01) grants 

funded by the U.S. National Institutes of Health from 1980 

to 2008. The authors find that better peer-review scores are 

consistently associated with better research outcomes and 

that this relationship persists even when we include detailed 

controls for an investigator's publication history, grant 

history, institutional affiliations, career stage, and degree 

types. • Percentile Ranking and Citation Impact of a Large 

Cohort of National Heart, Lung, and Blood Institute–

Funded Cardiovascular R01 Grants. Danthi et al. 

Circulation Research 114:600-606 2014. This study 

conducted an observational analysis of percentile rankings 

and bibliometric outcomes for a contemporary set of funded 

NHLBI cardiovascular R01 grants. The study identified 

1492 investigator-initiated de novo R01 grant applications 

that were funded between 2001 and 2008 and followed their 

progress for linked publications and citations to those 

publications. • An Outcome Evaluation of the National 
174 The Promise of Evidence-Based Policymaking

Institutes of Health (NIH) Director’s Pioneer Award 

(NDPA) Program, FY 2004–2006 Institute for Defense 

Analyses ‐Science and Technology Policy Institute, 2013. 

Sponsored by the NIH Division of Program Coordination, 

Planning, and Strategic Initiatives, this outcome evaluation 

assessed a targeted high-risk, high-reward research 

program, based on comparing NIH Pioneer Award 

recipients with recipients of more standard NIH R01 

research grants as well as awardees of the Howard Hughes 

Medical Institute (HHMI). The evaluation addresses several 

questions: To what extent does the research supported by 

the program produce unusually high impact, and to what 

extent are the research approaches used by the grantees 

highly innovative? The evaluation also addressed two 

secondary questions: To what extent is the Pioneers’ 

research interdisciplinary, and to what extent are the 

Pioneers collaborative? • Automated Research Impact 

Assessment: a new bibliometrics approach. Drew et al. 

Scientometrics 106(3) 2016. The National Institute of 

Environmental Health Sciences (NIEHS) has developed a 

new method to quantify the impact of funded research on 

the scientific and broader communities. In this article they 

describe a new bibliometric analysis method, the 

Automated Research Impact Assessment (ARIA). ARIA 

taps into a resource that has only rarely been used for 

bibliometric analyses: references cited in “important” 

research artifacts, such as policies, regulations, clinical 

guidelines, and expert panel reports. The approach includes 

new statistics that science managers can use to benchmark 

contributions to research by funding source. • Greatest 

'HITS': A new tool for tracking impacts at the National 

Institute of Environmental Health Sciences. Drew et al. 

Research Evaluation 22(5) 2013. This article presents the 

High Impacts Tracking System (HITS), a new approach to 

documenting research impacts that is in development at the 

National Institute of Environmental Health Sciences 

(NIEHS). HITS is designed to help identify scientific 

advances in the NIEHS research portfolio as they emerge, 

and provide a robust data structure to capture those 

advances. The authors have downloaded previously unsearchable data from the central NIH grants database and 

developed a robust coding schema to help track research 

products (going beyond publication counts to the content of 

publications) as well as research impacts. They describe the 

coding schema and key system features as well as several 

development challenges, including data integration, 
The Promise of Evidence-Based Policymaking 175

development of a final data structure from three separate 

ontologies, and ways to develop consensus about codes 

among program staff. • Toward the assessment of scientific 

and public health impacts of the National Institute of 

Environmental Health Sciences Extramural Asthma 

Research Program Using Available Data. Liebow et al. 

Environ. Health Perspectives. July 2009. The National 

Institute of Environmental Health Sciences (NIEHS) 

Division of Extramural Research and Training developed a 

framework to measure the scientific and health impacts of 

its extramural asthma research to improve the scientific 

basis for reducing the health effects of asthma. They then 

apply the framework to characterize the NIEHS asthma 

portfolio’s impact in terms of publications, clinical 

applications of findings, community interventions, and 

technology developments. • Scientific and Public Health 

Impacts of the NIEHS Extramural Asthma Research 

Program - Insights from Primary Data. Orians et al. Res 

Eval. 2009 Dec;18(5):375-385. NIEHS is the third largest 

source of asthma related research grant funding within the 

NIH between 1975-2005. Research results are often 

published in clinically-focused journals and these results are 

more likely to have an impact on clinical practice. While 

links to numerical public health outcomes are not discussed, 

this paper does link NIEHS funding to key breakthroughs 

related to asthma susceptibility and symptoms, showing 

NIEHS’ role in publications, clinical trials, and the adoption 

and dissemination of particular asthma interventions. 
176 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you 

responding for?

National Science Foundation

Office of Integrative Activities

If applicable, please provide additional 

information about this most critical 

barrier.

The lack of standard procedures or guidelines for 

sharing data across Federal agencies that fund research 

makes efforts to link and share data difficult or 

inefficient.

If applicable, please list up to five 

statistical, evaluation, research, or policy 

analysis activities your office, or its 

representatives, have not been able to 

complete because it was too hard to 

access the right data.

If applicable, please provide an example 

of a micro-level data linkage project or 

process in your office that works well. 

Linking NSF awards data with patent data from 

USPTO allows NSF to identify links between 

investments in basic research and patents, which are 

useful in determining the potential impact of research 

investments on technology and the economy.

If applicable, please provide an example 

of how your office has used data to build 

evidence for policy making.

Text analyses of NSF awards have revealed patterns 

about how NSF’s second merit review criterion —

Broader Impacts — is currently addressed by Principal 

Investigators (PIs) in proposals submitted to NSF and 

by reviewers in their reviews. The findings from these 

patterns will be used to design policies and provide 

guidance to PIs and reviewers on how to better address 

broader impacts of the proposed research. Analyses of 

proposals to the INCLUDES program that were 

funded and declined will allow program officers to 

revise solicitations to better align proposal with 

program objectives.
The Promise of Evidence-Based Policymaking 177

Survey response

Which Federal agency are you 

responding for?

Department of Labor

Office of Workers' Compensation Programs

If applicable, please provide additional 

information about this most critical 

barrier.

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, or 

its representatives, have not been able to 

complete because it was too hard to 

access the right data.

If applicable, please provide an example 

of a micro-level data linkage project or 

process in your office that works well. 

If applicable, please provide an example 

of how your office has used data to 

build evidence for policy making.

OWCP uses data analysis to support ongoing 

operations, evaluation, program integrity and 

accountability, and analysis for policy and decisionmaking. The majority of OWCP's data sets are 

"confidential data" (per the survey definitions) which 

contain PII about claimants as well as, in some cases, 

personal medical information related to their claims. 

The "individual unit" (per the survey definition) for 

most OWCP data sets is the claimant. For some OWCP 

programs, intermediate source data is provided by 

another agency, such as the Social Security 

Administration data to confirm employment and 

earnings, and to support offset requirements. Data 

analysis for ongoing operations include compilation 

and statistical analysis for recurring (monthly and 

quarterly) operational reviews as assessment of 

progress and results related to OWCP's five-year 

strategic plan. In FY 2016, OWCP completed 

development of the OWCP Unified Reporting System 

(OURS) to which the four programs push selected data 

on a weekly basis. OURS compiles the data and 

provides standardized reports available to managers 

and executives for quarterly reviews as well as ondemand performance analysis. OWCP has conducted a 

number of analyses using confidential data for 

decisions about proposed policies. The approach for 

these analyses varied depending on the policy issue, 

including examining claims for patterns across 

claimants or healthcare provider populations, in home 

health care usage, in wage loss compensation or 
178 The Promise of Evidence-Based Policymaking

medical benefit payments, and claims processing across 

OWCP offices. OWCP is currently conducting a 

statistical staffing analysis, using intermediate source 

(DOL Human Resource Center) data. This analysis will 

be used as input for human capital planning in the next 

OWCP strategic plan 
The Promise of Evidence-Based Policymaking 179

Survey response

Which Federal agency are you responding for? Department of Health and Human Services

National Center for Health Statistics

If applicable, please provide additional 

information about this most critical barrier.

If applicable, please list up to five statistical, 

evaluation, research, or policy analysis activities 

your office, or its representatives, have not been 

able to complete because it was too hard to 

access the right data.

IRS Data ... recently SSA data, data collected 

by the Census and economic data

If applicable, please provide an example of a 

micro-level data linkage project or process in 

your office that works well. 

Our National Death Index linkage system ... our 

linkage of our surveys with the NDI and CMS 

data and our linkage with HUD data

If applicable, please provide an example of how 

your office has used data to build evidence for 

policy making.

We do not use data to build evidence for policy 

making ... we collect and disseminate data and 

analysis that can be used by others for policy 

making ... our data can inform policy but we do 

not enter into that process directly. 
180 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you responding for?

Department of Health and Human 

Services

Centers for Disease Control and 

Prevention - National Institute for 

Occupational Safety and Health

If applicable, please provide additional information 

about this most critical barrier.

If applicable, please list up to five statistical, 

evaluation, research, or policy analysis activities your 

office, or its representatives, have not been able to 

complete because it was too hard to access the right 

data.

If applicable, please provide an example of a microlevel data linkage project or process in your office 

that works well. 

If applicable, please provide an example of how your 

office has used data to build evidence for policy 

making.

NIOSH conducts research that is used in 

policy making by state and federal 

agencies 
The Promise of Evidence-Based Policymaking 181

Survey response

Which Federal agency are you responding for? Department of Health and Human Services

Centers for Disease Control and Prevention

If applicable, please provide additional information 

about this most critical barrier.

Procuring or licensing data, whether federal 

or commercial, is expensive. The 

procurement costs are high, and the costs for 

staff time to learn about and use acquired 

sources is also high.

If applicable, please list up to five statistical, 

evaluation, research, or policy analysis activities 

your office, or its representatives, have not been 

able to complete because it was too hard to access 

the right data.

If applicable, please provide an example of a 

micro-level data linkage project or process in your 

office that works well. 

If applicable, please provide an example of how 

your office has used data to build evidence for 

policy making.
182 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you 

responding for?

Department of Commerce

Office of Performance Evaluation and Risk 

Management

If applicable, please provide additional 

information about this most critical 

barrier.

Department of Commerce has negligible funding for 

program evaluation. The best most recent evaluation 

was a very low cost collaboration between the Census 

Bureau and the International Trade Administration. A 

similar collaboration between Census and the Minority 

Business Development Administration was blocked by 

the General Council because assisted business were not 

notified that their information would be used for 

evaluation. The Grants Manual was modified and they 

are now notified in the FFO and grant terms. 

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, or 

its representatives, have not been able 

to complete because it was too hard to 

access the right data.

Evaluation of the Impact of the Minority Business 

Development Administration.

If applicable, please provide an 

example of a micro-level data linkage 

project or process in your office that 

works well. 

Open data website using Socrata application

If applicable, please provide an 

example of how your office has used 

data to build evidence for policy 

making.

We were instrumental in the International Trade 

Administration (ITA) collaboration with the Census 

Center for Economic Studies (CES). ITA gave CES 

data on businesses they assisted. Census used Census 

survey data to create a comparison group and the 

exports of the treated and untreated were compared. 

The DOC Performance staff also collaborated on the 

still to be released "Building Smarter Data for 

Evaluating Business Assistance Programs." This 

document recommends best practices in collecting data 

at the initiation of assistance to facilitate using 

statistical; data to assess impacts. The paper is a

collaboration among several agencies. It is in final form 

and is in clearance for release.
The Promise of Evidence-Based Policymaking 183

Survey response

Which Federal agency are you responding for?

Department of Health and Human 

Services

Administration for Children and 

Families - Office of Child Support 

Enforcement

If applicable, please provide additional information 

about this most critical barrier.

If applicable, please list up to five statistical, evaluation, 

research, or policy analysis activities your office, or its 

representatives, have not been able to complete because 

it was too hard to access the right data.

If applicable, please provide an example of a microlevel data linkage project or process in your office that 

works well. 

We match our wage data on 

individuals with other agencies in order 

for them to avoid erroneous costs and 

reduce overpayments.

If applicable, please provide an example of how your 

office has used data to build evidence for policy making.
184 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you 

responding for?

Department of Health and Human Services

Administration for Children and Families - Office of 

Head Start

If applicable, please provide additional 

information about this most critical 

barrier.

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, or 

its representatives, have not been able 

to complete because it was too hard to 

access the right data.

If applicable, please provide an 

example of a micro-level data linkage 

project or process in your office that 

works well. 

If applicable, please provide an 

example of how your office has used 

data to build evidence for policy 

making.

I am using this space to explain the difficulty I had in 

completing this survey. The Office of Head Start (OHS)

directly funds over 1,600 grantees to operate Head 

Start, Early Head Start and Early Head Start - Child 

Care Partnerships. The annual budget is approximately 

9 billion dollars. The Head Start Act, as amended, 

requires the Secretary to reserve no more than 20 

million dollars to fund research, annually. The Office of 

Program, Research and Evaluation, which resides in our 

Agency - the Administration for Children and Families, 

is responsible for research projects for the OHS and the 

expenditure of the 20 million dollars annually. I do not 

know, if they were asked to complete this survey; 

however, the questions may have been more applicable 

to their work than that of the OHS, which is 

overwhelmingly focused on program operations and 

oversight. Because OHS is a federal to locally funded 

program, the data we collect on individual grantees is 

generally related to day to day operations and grant 

making. Although our national funded enrollment is 

nearly a billion children, we do not collect child or 

family level data. That data resides in the grantee files 

and OHS recently promulgated news regulations to 

safeguard against unauthorized sharing or use of PII. 

Annually each grantee must complete a Program 

Information Report (PIR), which is available to the 

public, and does provide grantee aggregate data, which 

we do use in identifying trends and progress across 
The Promise of Evidence-Based Policymaking 185

states and regions and the PIR data is also used by 

independent researchers. Our major focus is to ensure 

whatever we collect is not collected by another federal 

agency and if another federal agency or state agency has 

data that we can leverage to do our job better than we 

want to be able to access that data readily. Clearly, there 

have been many challenges in these areas and 

dedicating a sufficient amount of our own resources in 

this area is a part of the challenge. 
186 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you 

responding for? Millennium Challenge Corporation

If applicable, please provide additional 

information about this most critical 

barrier.

MCC, and particularly its General Counsel, are 

seriously concerned about the limited legal assurance 

we can provide to partner countries, IRBs and survey 

respondents regarding FOIA requests for micro-level 

data that may be indirectly identifiable. For the 

purposes of scientific integrity and advancement, we 

value the ability of independent, responsible 

researchers to be able to replicate, re-analyze, or test 

new analytic hypotheses using our evaluation data. But 

uncertainty surrounding FOIA exemption rulings limits 

MCC's ability to ensure respondents and their 

governments that potentially identifiable micro-data 

shall only be used for statistical purposes.

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, or 

its representatives, have not been able to 

complete because it was too hard to 

access the right data.

MCC is unable to archive and make available restricted 

access versions of individual and establishment tax 

payer confidential interview data. On two occasions, 

IRBs have cited FOIA risks in refusing to allow MCC's 

independent evaluators to submit analytically useful 

micro-data to MCC for archival and restricted access 

dissemination. 

If applicable, please provide an example 

of a micro-level data linkage project or 

process in your office that works well. 

Public water and electric power utilities have worked 

with MCC hired evaluators to identify lists of 

households eligible for subsidized connections to 

utilities. These have worked successfully in Tanzania 

(water and power), El Salvador (power), Mozambique 

(water), and Jordan (water).

If applicable, please provide an example 

of how your office has used data to 

build evidence for policy making.

MCC's sectoral reviews of evaluations in agriculture 

and transportation have contributed to substantial 

changes in the ex-ante evidence requirements for MCC 

investments in those sectors. Additional sector-specific 

review work is currently underway in other sectors as 

well.
The Promise of Evidence-Based Policymaking 187

Survey response

Which Federal agency are you 

responding for?

Department of Education

National Center for Special Education Research

If applicable, please provide additional 

information about this most critical 

barrier.

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, or 

its representatives, have not been able to 

complete because it was too hard to 

access the right data.

It is very difficult to find out what happens with 

students with disabilities after they transition from high 

school. Are they able to find jobs that provide an 

adequate income, and do they stay in those jobs over 

time? How do students with disabilities fare in 

postsecondary schools? Do students with disabilities 

receive SSI, and if so, are there certain transition 

experiences that make it less likely for them to rely on 

this program? Using data to follow students with 

disabilities over time would help improve programs 

and policies aimed at students with disabilities in high 

school.

If applicable, please provide an example 

of a micro-level data linkage project or 

process in your office that works well. 

If applicable, please provide an example 

of how your office has used data to 

build evidence for policy making.

The increased demand for accountability in education 

focused on improved student academic performance 

has led to many questions about the most accurate 

method for capturing individual student progress, 

particularly for students with disabilities. Although 

there is a substantial amount of existing research on the 

characteristics of students with disabilities and 

assessment of their abilities and skills for purposes of 

classification and intervention, far less is known about 

the natural developmental progress in achievement for 

students with disabilities. The National Center for 

Special Education Research funded an initiative to 

study academic growth trajectories of students with 

disabilities and develop and test practical and relevant 

methods of accurately measuring academic growth for 

students with disabilities for use in accountability 

systems. The ultimate objective of the Center is to 

develop assessment methods that schools can use to (1) 

accurately assess the academic progress of students 

with disabilities and (2) improve the quality of 

education provided to students with disabilities, leading 

to improved student outcomes. Data from three States 

are used in this work. 
188 The Promise of Evidence-Based Policymaking

Survey response

Which Federal agency are you 

responding for?

Department of Labor

Employment and Training Administration

If applicable, please provide additional 

information about this most critical 

barrier.

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, or 

its representatives, have not been able to 

complete because it was too hard to 

access the right data.

If applicable, please provide an example 

of a micro-level data linkage project or 

process in your office that works well. 

We are in process of implementing common 

performance and reporting data for grantees, across 

Federally funding programs, where data submitted to 

the Federal level does not contain confidential 

information, but for which one unit (or individual) can 

be identified in multiple programs. Still significant 

work ahead. 

If applicable, please provide an example 

of how your office has used data to 

build evidence for policy making.

Conduct pilots and evaluations, build capacity of 

grantees to conduct their own research and evaluation, 

use of performance accountability measures, use of 

data analytics, etc. This information is used in policy 

development, as part of the development of budget 

requests, to identify strong performing grantees to help 

identify promising practices, to inform our operating 

plan, etc.
The Promise of Evidence-Based Policymaking 189

Survey response

Which Federal agency are you 

responding for? General Services Administration

If applicable, please provide additional 

information about this most critical 

barrier.

Finding Agency point person to coordinate on data 

sharing agreements, and put in substantial leg work to 

troubleshoot and execute and get senior level support is 

very challenging. 

If applicable, please list up to five 

statistical, evaluation, research, or 

policy analysis activities your office, or 

its representatives, have not been able to 

complete because it was too hard to 

access the right data.

We have not been able to look at program impact on 

ultimate health, wellbeing, employment, income, etc 

outcomes, but rather been restricted to intermediary 

operational outcomes that the lead Agency office owns 

and manages. 

If applicable, please provide an example 

of a micro-level data linkage project or 

process in your office that works well. 

If applicable, please provide an example 

of how your office has used data to 

build evidence for policy making.

The Federal government has incredibly rich data 

sources that, if used carefully and appropriately, can 

help answer many pressing questions regarding 

program impact, program efficiency, and help build 

needed evidence bases. Unfortunately, our ability to 

answer these questions is largely restricted to agencies, 

offices and teams who have knowledge, bandwidth, 

patience and willingness to tackle complex data sharing 

agreements and protocols. Further, many agencies may 

want to understand their programs impact on health, 

income, employment, or educational outcomes that are 

collected outside their agencies. The difficulty in 

tackling data sharing largely means that evaluation 

outcomes, specifically which the GSA Office of 

Evaluation Sciences, conducts are restricted to 

intermediary outcomes. 
This page intentionally left blank
The Promise of Evidence-Based Policymaking 191

Appendix F: CEP Public Meeting 

Materials and Presentations

Between July 2016 and March 2017, the Commission convened seven public meetings, with a total of 49 invited witnesses.

Appendix F includes each of the witnesses' presentations or written statements.
192 The Promise of Evidence-Based Policymaking

Commission on Evidence-Based Policymaking

Statement

July 22, 2016 Contact: adrm.commission.liaison.team@census.gov

Joint Statement from Chair Katharine G. Abraham and 

Co-Chair Ron Haskins on the Commencement of the 

Commission on Evidence-based Policymaking

The Commission on Evidence-Based Policymaking begins its work in the midst of an 

unprecedented movement toward evidence-based policymaking. This movement has been 

growing for more than a decade at the federal and state levels, and reflects a continued desire 

from the American public and policymakers that credible information be available to inform 

decisions about government programs and activities. Congress and the President created this 

bipartisan Commission to provide the nation with guidance on ways to further expand our 

approaches to evidence-building in government. We recognize that we have been given an 

enormous opportunity to help the country move towards better and more effective 

government in the coming decades, and look forward to developing a bipartisan strategy to 

ensure that evidence increasingly informs the important decisions that affect the lives of 

Americans.

###

July 22, 2016 Meeting
The Promise of Evidence-Based Policymaking 193
194 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 195
196 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 197
198 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 199
200 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 201
202 The Promise of Evidence-Based Policymaking

The American Dream?

 Probability that a child born to parents in the bottom fifth 

of the income distribution reaches the top fifth:

Raj Chetty

Stanford University

Improving Economic Opportunity in America

New Lessons from Administrative Data

Photo Credit: Florida Atlantic University
The Promise of Evidence-Based Policymaking 203

 Probability that a child born to parents in the bottom fifth 

of the income distribution reaches the top fifth:

Canada

Denmark

UK

USA

13.5%

11.7%

7.5%

Blanden and Machin 2008 9.0%

Boserup, Kopczuk, and Kreiner 2013

Corak and Heisz 1999

Chetty, Hendren, Kline, Saez 2014

The American Dream?

 Probability that a child born to parents in the bottom fifth 

of the income distribution reaches the top fifth:

 Chances of achieving the “American Dream” are almost 

 two times higher in Canada than in the U.S.

Canada

Denmark

UK

USA

13.5%

11.7%

7.5%

Blanden and Machin 2008 9.0%

Boserup, Kopczuk, and Kreiner 2013

Corak and Heisz 1999

Chetty, Hendren, Kline, Saez 2014

The American Dream?
204 The Promise of Evidence-Based Policymaking

The Geography of Upward Mobility in the United States

Chances of Reaching the Top Fifth Starting from the Bottom Fifth by Metro Area

San

Jose 

12.9%

Salt Lake City 

10.8%

Atlanta 4.5%

Charlotte 4.4%

Denver 8.7%

Note: Lighter Color = More Upward Mobility

Download Statistics for Your Area at www.equality-of-opportunity.org

Boston 10.4%

Chicago

6.5%

 Differences across countries have been the focus of 

policy discussion

 But upward mobility varies even more within the U.S.

 We calculate upward mobility for every metro and rural 

area in the U.S.

– Use de-identified data from IRS tax records (part of a broader 

project on effects of tax expenditures)

– 10 million children born between 1980-1982

Differences in Opportunity Within the United States

Source: Chetty, Hendren, Kline, Saez 2014: The Equality of Opportunity Project
The Promise of Evidence-Based Policymaking 205

The Geography of Upward Mobility in the Washington Metro Area

Chances of Reaching the Top Fifth Starting from the Bottom Fifth by County

Charles

Baltimore

DC

Hartford

Baltimore: 3.5%

District of Columbia: 4.7%

Prince George’s: 9.2%

Montgomery: 16.0%

Prince 

George’s

 Most of the variation in upward mobility across areas is 

caused by differences in childhood environment

 Demonstrate this by studying 5 million families that move 

between areas using tax records

Why Does Upward Mobility Differ Across Areas?

The Importance of Childhood Environments

Source: Chetty and Hendren 2015
206 The Promise of Evidence-Based Policymaking 0% 20% 40% 60% 80% 100%

10 15 20 25 30

Washington DC ($30,000)

Earnings Gain from Moving to a Better Neighborhood 

Montgomery County ($40,000)

0% 20% 40% 60% 80% 100%

10 15 20 25 30

Washington DC ($30,000)

Earnings Gain from Moving to a Better Neighborhood 
The Promise of Evidence-Based Policymaking 207

0% 20% 40% 60% 80% 100%

10 15 20 25 30

Age of Child when Parents Move

Gain from Moving to a Better Area

Washington DC ($30,000)

Earnings Gain from Moving to a Better Neighborhood 

Move at age 9  54% of gain from 

growing up in Montgomery County since 

birth

Montgomery County ($40,000)

0% 20% 40% 60% 80% 100%

10 15 20 25 30

Age of Child when Parents Move

Gain from Moving to a Better Area

Washington DC ($30,000)

Earnings Gain from Moving to a Better Neighborhood 

Montgomery County ($40,000)
208 The Promise of Evidence-Based Policymaking

1. Less residential segregation

2. Larger middle class

3. More stable family structure

4. Greater social capital

5. Better school quality

What are the Characteristics of High-Mobility Areas?

Five Strongest Correlates of Upward Mobility

0% 20% 40% 60% 80% 100%

10 15 20 25 30

Age of Child when Parents Move

Gain from Moving to a Better Area

Washington DC ($30,000)

Earnings Gain from Moving to a Better Neighborhood 

Montgomery County ($40,000)
The Promise of Evidence-Based Policymaking 209

 Results suggest that giving low-income families housing 

vouchers to move to better areas canimprove outcomes

 HUD Moving to Opportunity Experiment: gave vouchers to 

move to low-poverty areas using a randomized lottery

– 4,600 families in Boston, New York, LA, Chicago, and Baltimore in 

mid 1990’s

– Prior work found little impact of MTO on economic outcomes

– We linked MTO data to tax records to track long-term impacts on 

children who moved at younger ages

Source: Chetty, Hendren, and Katz 2015

Housing Vouchers and the Moving to Opportunity Experiment

Control

ML King Towers

Harlem

Experimental

Wakefield 

Bronx

Common MTO Residential Locations in New York
210 The Promise of Evidence-Based Policymaking

Impact of MTO Experimental Voucher on Earnings in Adulthood

by Child’s Age at Move

-6000 -4000 -2000

0 2000 4000

Treatment Effect on Household Income ($)

10 12 14 16

Child’s Age at Move

 Children who moved to low-poverty areas when young (e.g., 

below age 13) do much better as adults:

– 30% higher earnings = $100,000 gain over life in present value

– 27% more likely to attend college

– 30% less likely to become single parents

 But moving had little effect on the outcomes of children who 

were already teenagers

 Moving also had no effect on parents’ earnings

 Confirms that duration of exposure to better neighborhood 

matters, explaining why previous studies didn’t find any effects

Moving to Opportunity Experiment Re-Analysis
The Promise of Evidence-Based Policymaking 211

1. Scarce bandwidth and limited access

– Ex: complex contracting process, few physical locations for IRS; 

inadequate space in Census RDC’s

2. Outdated technology due to scarce resources

– Ex: very small investments (~$100K) would greatly relax hard disk, 

RAM, and processor constraints at IRS

3. Very difficult to link datasets

– Ex: linking Census records to tax data would permit analysis of 

upward mobility by race

 Contrast with Danish statistical agency, gold standard for research

Key Barriers to Using Administrative Data for Research

1. Create a centralized data warehouse that links datasets 

– IRS data provide an ideal spine for linking other data

– Many existing surveys redundant; focus on collecting information 

not already in administrative data

2. Provide secure, direct access to data with simplified 

access protocols

– Synthetic data do not work well for iterative analysis

– Rapid retrieval of statistical results critical; current Census RDC 

approval often too slow

Improving Evidence-Based Policy in the U.S.
212 The Promise of Evidence-Based Policymaking

0

10

20

30

40

50

60

70

80

90

100

2010 2011 2012 2013 2014 2015

Cumulative Number of Researchers

Year

Cumulative Number of Researchers who have Written Papers 

Using IRS Population Tax Data

3. Start from randomized experiments, but support a broader 

range of methods

– Experiments under-powered, especially for studying long-term effects

– Quasi-experimental methods leverage big data most directly

– Constructing descriptive statistics to monitor progress (e.g., in local 

areas) is itself very valuable

Improving Evidence-Based Policy in the U.S.
The Promise of Evidence-Based Policymaking 213

Bureau of Justice Statistics

• A principal Federal statistical agency

• Sits in the Department of Justice Office of Justice Programs

• FBI, U.S. Marshals, Bureau of Prisons, Drug Enforcement 

Administration, U.S. Attorney, Civil Rights Division,....

• Measures the U.S. justice system: A collection of independent 

systems at the local, state, and federal level.

U.S. Department of Justice

Office of Justice Programs

BUREAU OF JUSTICE STATISTICS

A Case Study: Transforming National Criminal History 

Records into a Statistical Database

for the

Commission on Evidence-Based Policymaking 

First Meeting

The National Academies of Sciences, Engineering, and Medicine

2101 Constitution Avenue NW, Lecture Room 

Washington, DC

Presented by

Jeri Mulrow

Acting Director

Bureau of Justice Statistics

July 22, 2016
214 The Promise of Evidence-Based Policymaking

The Criminal Justice System

Why Criminal History Records?

• Important policy or evaluation questions

• Existing data

• Impacts a lot of people.

• There are roughly 100 million criminal history records. 

 

 Potential for evidence-based policy making
The Promise of Evidence-Based Policymaking 215

Example of Rap Sheet Format in State A 

NAME XXXXXXXXXXXXXXXXXX STATE ID XXXXXXXXXXXX

SSN XXX-XX-XXXX FBI NO. XXXXXXXXXXXX

CITIZENSHIP UNKNOWN

BIRTH DATE XXXX/XX/XX BIRTH PLACE XXXXXXXXXXXX

DECEASED NO

RACE WHITE SEX FEMALE

ETHNICITY SKIN TONE

HEIGHT 5'05" WEIGHT 165

HAIR BROWN EYES BLUE

CRIMINAL HISTORY 

ARREST 1

DOA 1992/10/25 DOO

AGENCY POLICE DEPARTMENT

ARREST TYPE ADULT AGENCY CASE XXXXXX

CHARGE 1 OFF STATUTE 6-3-404

SEVERITY MIS

COURT DISPOSITION 

DISP DATE 1992/10/27 DOCKET # XXXXXX

OFF SEVERITY MIS OFF COUNT 1

VERDICT CONVICTED FELONY NO

BRADY/IFS CLEAR VOTING YES

SENTENCE PROBATION

FINE $100.00 RESTITUTION $85.00

SUSPENDED VICTIM COMP

MIN CONF MAX CONF

MIN PROB 180 DAYS MAX PROB ONE YEAR

Existing data: Rap Sheets

Each state maintains its own repository.

State Criminal 

History Record 

Repository

Law 

Enforcement 

Agency

Prosecutor Court

Correctional 

Agency

State Criminal 

History Record 

Repository

Law 

Enforcement 

Agency

Prosecutor Court

Correctional 

...

Agency
216 The Promise of Evidence-Based Policymaking

NAME(S): XXXXXXXXX

FBI NUMBER DPS NUMBER

XXXXXXXXX XXXXXXXXX

SOCIAL SECURITY DRIVERS LICENSE ID NUMBER

XXXXXXXXX XXXXXXXXX

SEX RACE SKIN TONE

M B MED

HEIGHT WEIGHT DATE OF BIRTH

510 180 XX-XX-XXXX

HAIR COLOR EYE COLOR FINGERPRINT PATTERN

BLK BRO

PLACE OF BIRTH CITIZEN III CODE

TX US MULTI-STATE

===================================================================

EVENT CYCLE 1

TRACKING NUMBER XXXXXXXXX

ARREST DATE 05-06-1995

TYPE ADULT

AGENCY 0180000 – STATE POLICE 

-------------------------------------------------------------------

OFFENSE DATA

OFFENSE DATE 05-06-1995

OFFENSE BURG

CITATION PC 40.02(a) 

LEVEL & DEGREE FELONY - 1ST DEGREE

DISPOSITION HELD

REFERRED DISTRICT ATTORNEY OFFICE

-------------------------------------------------------------------

COURT DATA

COURT AGENCY 5TH DISTRICT COURT 

COURT OFFENSE BURGLARY OF HABITATION

CITATION PC 30.02(a) 

LEVEL & DEGREE FELONY - 1ST DEGREE

DISPOSITION CONVICTED

DISPOSITION DATE 03-30-1996

SENTENCE DATE 03-30-1996

FINAL PLEADING GUILTY

CONFINEMENT 10Y

Example of Rap Sheet Format in State C 

NAME: XXXXXXXXXXXX DOB: XX/XX/XXXX

SID NUMBER: XXXXXXXXXXXX FBI NUMBER: XXXXXXXXXXXX

DOC NUMBER: XXXXXXXXXXXX SOC SEC XXXXXXXXXXXX 

*******************************************************************************

PERSON INFORMATION

*******************************************************************************

SEX RACE HEIGHT WEIGHT EYES HAIR PLACE OF BIRTH CITIZENSHIP

M B 600 160 BRO BLK XX US

CRIMINAL HISTORY INFORMATION

*******************************************************************************

ARREST 1 DATE OF ARREST: 06/12/1991

-------------------------------------------------------------------------------

NAME USED: XXXXXXXXXXXX

CONTRIBUTING AGENCY: POLICE DEPARTMENT

LOCAL ID: XXXXXXXXXXXX PCN: XXXXXXXXXXXX TCN: N/A

-------------------------------------------------------------------------------

ARREST OFFENSES | DISPOSITION

09930 FAIL TO COMPLY | CONTRIBUTOR OR RESPONSIBLE AGENCY:

CLASS UNKNOWN | MUNICIPAL COURT

ORIGINATING AGENCY: |

POLICE DEPARTMENT | STATUS: DISPOSITION NOT RECEIVED

OIN: XXXXXXX |

DISPO RESPONSIBILITY: COURT | CONTRIBUTOR OR RESPONSIBLE AGENCY:

COURT CASE NO: XXXXXXX | SUPERIOR COURT

COMMENT: BURGLARY |

| COURT CASE NO: XXXXXX

|

CLASS UNKNOWN | STATUS: GUILTY

ORIGINATING AGENCY: | THEFT-3

POLICE DEPARTMENT | RCW: 9-46-20

OIN | MISDEMEANOR

DISPO RESPONSIBILITY: COURT | STATUS DATE: 08/27/1991

COURT CASE NO: XXXXXX | SENTENCE: SENT. DESC.:

| CHG 01: JAIL-15 DS/SUSPENDED 

| 1 YR, SUPV-12 MOS 

-------------------------------------------------------------------------------

Example of Rap Sheet Format in State B 
The Promise of Evidence-Based Policymaking 217

Request

NY#123456

Simplified Description of III System:

A Request for a Multi-State Criminal History Record 

(or Rap Sheet)

Nlets

NY# 123456

FBI

III

Connecting together State Rap Sheets

Nlets

FBI

Interstate

Identification 

Index
218 The Promise of Evidence-Based Policymaking

III Provides Federal Portion of Rap Sheet to Nlets along 

with the Person’s State ID Numbers;

then Nlets Requests State Records 

Nlets

FBI# 635879

CA# 113458

NY# 123456

TX# 314112

VA# 006451

FBI 

III

FBI# 635879

CA# 113458

NY# 123456

TX# 314112

VA# 006451

CA#

113458 NY#

123456

TX#

314112

VA#

006451

TX#

314112

State Repositories Return State-Specific Rap Sheets and 

Nlets Sends Set to Requesting Agency 

Nlets

FBI# 635879

CA# 113458

NY# 123456

TX# 314112

VA# 006451

NY#

123456

CA#

113458

VA#

006451
The Promise of Evidence-Based Policymaking 219

Challenge 1: Gain Access (Obtain permission)

2006: Began exploring how to gain the permission to access

2008: Signed MOU with FBI

• Duties of each agency

• Terms of use

• Extensive security provisions

• Point of entry

Challenges to using Criminal History Records

1. Gaining access

2. Standardizing for statistical purposes

3. Understanding quality

4. Archiving
220 The Promise of Evidence-Based Policymaking

Challenge 2: Standardize the Criminal History 

Records

Step 1: Parse individual state rap sheet data

• Identify key data fields

• Identify location of key data fields in EACH state 

format

• Create programs to parse text strings

• Copy into a standardized relational database

Rap sheet fields in the uniform record layout

Arrest segment

• Date of offense

• Date of arrest 

• Arresting agency name and ORI

• Statute number

• NCIC code

• Charge description

• Charge severity

• Arrest disposition

• Arrest disposition date Court/Sentencing segment

• Court disposition date

• Court name and ORI

• Statute number

• NCIC code

• Offense description

• Type of court disposition 

• Sentences (prison, fine , etc.)

Subject segment

• BJS case number

• Name

• State identification number (SID)

• FBI identification number

• Date of birth

• Gender

• Race

• Social security number

Supervision segment

• Supervision date (admission, 

 release, etc.)

• Agency name and ORI

• Supervision description
The Promise of Evidence-Based Policymaking 221

Challenge 3: Evaluate quality

Data Quality

• Are there parsing and coding errors?

• Feed back identified errors to Nlets

• Check for large amounts of uncoded or unknown codes 

• Are there patterns of missing data?

• Feed back identified gaps to States

• Support states to report more complete data

Challenge 2: Standardize the Criminal History 

Records

Step 2: Recode state specific data into a common taxonomy

• Map state specific statutes into the common codes

• Initially done by hand to interpret the statutes

• Developed automated coding programs/crosswalks

• Validated the crosswalks 

• Maintain and update the library of state to state crosswalks

• Revise crosswalks as state statutes change or create new crimes
222 The Promise of Evidence-Based Policymaking

Costs*

Developmental costs – about $3.0 million.

$1.1 million for initial record acquisition, collation, parsing.

$1.9 million for initially standing up the data standardization software, 

coding crosswalks, creation of recode libraries.

Maintenance costs – about $360,000 annually

Acquisition, collation, etc. – about $60,000

Maintenance of standardization software, recode library and crosswalks –

about $300,000 per cohort study.

*excludes OJP & BJS staff salaries and overhead.

Challenge 4: Archiving

This is a remaining challenge –

Need to balance regulatory restrictions

With ability to be transparent and reproducible
The Promise of Evidence-Based Policymaking 223

Jeri M. Mulrow, Acting Director

Bureau of Justice Statistics

810 7th Street, N.W.

Washington, DC 20531

(202) 307-0765

Jeri.Mulrow@usdoj.gov

Thank you!

A Case Study...

• Enhances Nation’s criminal justice data infrastructure

• Potential is still to be fully realized for evidence-based policy decisions

• Continued data quality improvement with use

• Improves operational uses

• Improves statistical and research uses

• Improves evaluation uses
224 The Promise of Evidence-Based Policymaking

©2016 State of Illinois – Department of Innovation and Technology (DoIT) Internal Confidential

1. Public vs. Private Universities - access to UI data - 20 C.F.R.§603.

2. Data sharing agreements

▫ the Illinois Enterprise Memorandum of Understanding

Illinois Perspectives

CONFIDENTIAL – PROPRIETARY AND PRE-DECISIONAL

Any use of this material without specific permission is strictly prohibited

CEBP

9 September 2016

State Legal Perspectives

Illinois

September 9, 2016 Meeting
The Promise of Evidence-Based Policymaking 225

©2016 State of Illinois – Department of Innovation and Technology (DoIT) Internal Confidential

The Illinois Enterprise Memorandum of Understanding (eMOU) on data sharing

- Internal (but externals are in the works)

- Establishes common agreed to framework and terms

- Employs active management by an Operational Committee

- Contemplates, but does not require, a technological platform across which the 

data would be shared

- Rapid, compliant, flexible

2.  Data Sharing Agreements

©2016 State of Illinois – Department of Innovation and Technology (DoIT) Internal Confidential

Code of Federal Regulations on Employee Benefits (Title 20 of the CFR)

20 C.F.R.§603.4 requires us to keep UI data confidential

20 CFR §603.5(e) permits us to share UI data with a “public official”

20 CFR §604.2(d) says that a “public official” can include:

(5) A State educational authority, agency, or institution as those terms are used in the 

Family Educational Rights and Privacy Act, to the extent they are public entities.

1. Public vs. Private Universities: access to UI data
226 The Promise of Evidence-Based Policymaking

©2016 State of Illinois – Department of Innovation and Technology (DoIT) Internal Confidential

| 6

Thank you

©2016 State of Illinois – Department of Innovation and Technology (DoIT) Internal Confidential

The Illinois Enterprise Memorandum of Understanding (eMOU) on data sharing

 Educational institutions at all levels (public and private)

 Local units of government

 Business workforce groups

 Criminal justices groups

2.  Data Sharing Agreements
The Promise of Evidence-Based Policymaking 227

Cynthia Dwork – Evidence‐Based Policymaking

Differential Privacy and its Properties

Differential Privacy is a definition of privacy, and a collection of supporting algorithmic techniques,

tailored for privacy‐preserving statistical analysis of large datasets.

Differential privacy is a mathematical guarantee that an individual data contributor will not be affected,

adversely or otherwise, by allowing her data to be used in any study or analysis, no matter what other

studies, data sets, or information sources, are ‐‐ or will become – available.  At their best, differentially

private algorithms can make confidential data widely available for accurate data analysis, without

resorting to data clean rooms, data usage agreements, data protection plans, or restricted views.

Nonetheless, data utility will eventually be consumed: the Fundamental Law of Information Recovery

states that overly accurate estimates of too many statistics can completely destroy privacy (Dinur and

Nissim, 2003; Dwork et al, 2007; Homer et al, 2008, Dwork et al., 2015b).  The Fundamental Law can no

more be circumvented than can the laws of physics.

Every useful computation results in some loss of privacy.  Differential privacy measures and controls

privacy loss accumulating over multiple analyses.   This signal capability makes it possible to “program”

in a differentially private fashion.  In ordinary, non‐private computation, anything computable can be

computed using only addition and multiplication, but this is not how programmers work.  Algorithm

design is the creative combining of appropriate computational primitives to carry out a sophisticated

computational task, while minimizing the consumption of key resources, such as time and space.  

Similarly, differentially private algorithm design is the creative combining of simple differentially private

primitives to perform a sophisticated analytical task, while also minimizing privacy loss and inaccuracy.  

As a rule, when the dataset is large the signal dominates the noise injected for privacy; when the dataset

is small this is not the case.  This is correct; think of the case of a dataset of size one: to ensure privacy

the noise must dominate the signal.  Designed to preserve the privacy of everybody – even the needles

in the haystack – the goal is to elicit participation, without fear of repercussion, for a public good, such

as learning that smoking causes cancer, and other facts of life.  Indeed, it is often the outliers who most

need protection.   

Differential privacy also provably controls privacy loss accruing over computations on multiple, possibly

overlapping, datasets, making it especially relevant to the kinds of analyses that will be needed for

evidence‐based policy making.   

The Fundamental Law tells us that meaningful privacy guarantees come at a price.  Other disciplines,

such as ethics and economics, cannot be brought to bear without a measure of privacy loss.   Differential

privacy provides such a measure (Abowd and Schmutte, 2015).

Finally, differential privacy strengthens the scientific method in an unexpected way, even when privacy

is not a concern.  The rise of "Big Data" has been accompanied by increased risk of spurious scientific

discovery.  A great deal of effort has been devoted to reducing this risk, from the use of sophisticated

validation techniques, to deep statistical methods for controlling the false discovery rate in multiple

hypothesis testing.  However, there is a fundamental disconnect between the theoretical results and the

practice of data analysis: the theory of statistical inference assumes a fixed collection of hypotheses to

be tested, selected before the data are gathered, whereas in practice data are shared and reused, with
228 The Promise of Evidence-Based Policymaking

hypotheses and new analyses being generated on the basis of data exploration and the results of

previous studies on the same dataset.  This leads to overfitting, that is, learning about the dataset rather

than about the population from which it is drawn.  Differential privacy automatically protects against

this source of false discovery (Dwork et al., 2015a).

Key Considerations

Differential privacy holds great promise but requires great effort.  The Fundamental Law forces

economic considerations in how data should be used, increasing the imperative for high quality

differentially private algorithms, but the field is young and many of these will be the content of doctoral

dissertations not yet written.  The literature is silent on crucial preprocessing steps, such as imputation

of missing fields and other aspects of data cleaning.  Working with formal privacy guarantees requires a

new skill set, foreign to most statistical agencies, social science researchers, and data scientists.   Recent

adoption of the approach by Google and Apple will draw talent away from the public and research

sectors.   

But what is the alternative?  The distinction between Personally Identifiable Information (PII) and non‐

PII is not mathematically meaningful.  In the words of the President’s Council of Advisors on Science and

Technology, “Anonymization is increasingly easily defeated by the very techniques that are being

developed for many legitimate applications of big data.  In general, as the size and diversity of available

data grows, the likelihood of being able to re‐identify individuals (that is, re‐associate their records with

their names) grows substantially” (PCAST, 2014).  Traditional statistical disclosure limitation methods

ruling out subtraction attacks do not defend against other attacks (Fellegi, 1972, and subsequent

generalizations).   

It is a consequence of the Fundamental Law that all privacy‐preserving computations must introduce

some error.  Current statistical disclosure limitation (SDL) techniques also introduce errors. For example,

one paper states, “These algorithms [with formal privacy guarantees] are currently being implemented

on data gathered at a national statistical agency on which we empirically evaluate the utility of our

algorithms. We show that for reasonable values of the privacy loss parameter ... the error introduced by

our provably private algorithms is comparable or better than the error introduced by existing SDL

techniques” (Haney et al., 2015).  Similarly, a study on privacy in massive open online courses (MOOCs)

data from MITx and HarvardX on the edX platform reported that standard anonymization methods force

changes to datasets that “threaten replication and extension of baseline analyses”  (Daries et al., 2014).   

When the errors are introduced in a principled way, as in differential privacy, the analyst can better

interpret the results.

Even synthetic data (Rubin, 1993) can be problematic. Once constructed, synthetic data may be queried

ad libitum, with no risk of further privacy loss, using the analyst’s choice of techniques.  However,

privacy is not an automatic consequence of the data being synthetic, but depends crucially on the

process by which the synthetic data are constructed.   (The Census Bureau uses synthetic data

generated with a variant of differential privacy in OnTheMap, a website providing information on where

people work and where workers live (Machanavajjhala et al., 2008).)   
The Promise of Evidence-Based Policymaking 229

Recommendations

I close with three policy recommendations.  First, Publish Your Epsilons.  Differentially private algorithms

are equipped with a privacy parameter, usually called epsilon, capping their privacy loss.  In a non‐

private algorithm epsilon is infinite.  But what is the “meaning” of a given value of epsilon?  By

maintaining a registry of privacy loss, akin to a toxic release registry, we can observe the

accuracy/privacy tradeoffs actually made and stimulate competition to obtain better analyses at lower

privacy costs, engaging those who traffic in the data of individuals in the effort to protect their privacy.

Second, Establish a list of approved private data analysis techniques and appropriate applications, and

keep it current.   

Third, Consider Restraint.  In a data‐rich world, the challenges revolve around the trade‐off between

what can be done and acceptance of the fundamental truth that overly accurate estimates of too many

statistics can destroy privacy. If we are interested in privacy, sometimes restraint might be the right

approach.

References

J. M. Abowd and I. Schmutte, Revisiting the Economics of Privacy: Population Statistics and

Confidentiality Protection as Public Goods, http://digitalcommons.ilr.cornell.edu/ldi/22/, 2015

J.P. Daries, J. Reich, J. Waldo, E.M. Young, J. Whittinghill, A.D. Ho, A. D., D.T,. Seaton, and I. Chuang.

Privacy, anonymity, and big data in the social sciences. Communications of the ACM, 57(9), 56‐63, 2014.

I. Dinur and K. Nissim, Revealing information while preserving privacy. In Proceedings of the Twenty‐

Second ACM SIGACT‐SIGMOD‐SIGART Symposium on Principles of Database Systems, pages 202‐210,

2003

C. Dwork, V. Feldman, M. Hardt, T. Pitassi, O. Reingold, and A. Roth. The reusable holdout: Preserving

validity in adaptive data analysis. Science 349(6248), 2015a.

C. Dwork, F. McSherry, and K. Talwar, The price of privacy and the limits of lp decoding. In Proceedings

of the 39th ACM Symposium on Theory of Computing, pages pp. 85‐94, 2007

C. Dwork,  A. Smith,  T. Steinke, J. Ullman, and S. Vadhan. Robust traceability from trace amounts. In

Proc. 56th IEEE Annual Symposium Foundations of Computer Science (FOCS), 2015b

U. Erlingsson, V. Pihur, and A. Korolova, “RAPPOR: Randomized aggregatable privacy‐preserving ordinal

response. In Proc. ACM SIGSAC Conference on Computer and Communications Security (CCS), 2014.

I. P. Fellegi,  On the Question of Statistical Confidentiality. Journal of the American Statistical Association

67, no. 337, 1972

S. Haney, A. Machanavajjhala, M. Kutzbach, M. Graham, J. Abowd, L. Vilhuber. Formal privacy protection

for data products combining individual and employer frames. Presented at UNECE/Eurostat Statistical

Data Confidentiality Work Session, 2015
230 The Promise of Evidence-Based Policymaking

N. Homer, S. Szelinger, M. Redman, D. Duggan, W. Tembe, J. Muehling, J. Pearson, D. Stephan, S.

Nelson, and D. Craig. Resolving individuals contributing trace amounts of dna to highly complex mixtures

using high‐density snp genotyping microarrays. PLoS Genet, 4, 2008

A. Machanavajjhala, D.  Kifer,  J. Abowd, J. Gehrke, and L. Vilhuber. Privacy: Theory meets practice on

the map. In Proceedings of the IEEE 24th International Conference on Data Engineering, 2008

President’s Council of Advisors on Science and Technology (PCAST), Big Data and Privacy: A

Technological Perspective, May 2014.

D. B. Rubin, Statistical disclosure limitation. Journal of official Statistics, 9(2), pp.461‐468, 1993

A.D. Sarwate, S.M. Plis, J. Turner, and V.D.  Calhoun. Sharing privacy‐sensitive access to neuroimaging

and genetics data: a review and preliminary validation. Frontiers in Neuroinformatics, 2014

S. Warner. Randomized response: A survey technique for eliminating evasive answer bias. Journal of the

American Statistical Association, 60(309), 1965
The Promise of Evidence-Based Policymaking 231

A transparency initiative led by the California 

Department of Justice that publishes criminal 

justice data so we can understand how we are 

doing, hold ourselves accountable, and improve

public policy to make California safer.

What is OpenJustice?

Justin Erlich, Special Assistant Attorney General 

Office of Attorney General Kamala D. Harris

California Department of Justice

Data-driven

public safety
232 The Promise of Evidence-Based Policymaking

Theory of Change

Rigorous analytics

 Bring big data

methodologies

 Partner with

researchers

 Identify policy

improvements

Awareness

 Engage policy

makers and local

criminal justice

partners

 Inform the

community to

build trust

Transparency

 Release raw data

 Lift up clear and

actionable

insights

 Crowdsource

data analysis

 Share policies

that are working

Good data

 Improve

quality of

existing data

 Move

towards real

time

reporting

 Ensure we are

have the right

data fields

Collect Analyze Publish Engage

Vision For OpenJustice

• Ongoing national conversation on how to strengthen trust and

improve performance of the criminal justice system

• Smart on Crime approach – transparency & metrics

 Use transparency to strengthen trust

 Develop metrics to understand how we are doing and where 

to prioritize interventions 

 Identify evidence-based investigative practices

• Three key components

 Justice Dashboard to show visually how we are doing

 Open Data Portal to make raw data available to public

 Data reporting/collection improvement effort
The Promise of Evidence-Based Policymaking 233

The Dashboard visualizes the data to give 

the public a clearer statewide picture

The first six datasets
234 The Promise of Evidence-Based Policymaking

Open Data benefits & risks

 Data is a “public good”

 Transparency builds trust

 Open by default is a signaling 

function; reduces transaction costs

 Increases potential for research 

capacity by expanding access

 Unlocks innovation and 

possibilities

 Identity disclosure (e.g., 

Mugshots.com) and/or 

re-identification

 Inferential disclosure

 Bad data = bad policy

Benefits Risks

It also shows differences across counties 

and agencies throughout the state 

 Includes contextual data such as pop. demographics, education, employment, etc.
The Promise of Evidence-Based Policymaking 235

Key attributes of the data

 In addition to unique identifiers (e.g., Name), there are a generally a core set of data 

fields that can be quasi-identifiers:

 Gender

 Race

 Age

 Offense Type

 Date of Offense

 Location/jurisdiction of incident

 The problem of small numbers and geographical boundaries

 Small cities, counties, law enforcement pose the biggest challenges

 As we move to more granular geographical data collection, this will only 

get more challenging

 We are exploring “binning”

 Combining into age buckets

 Month of offense rather than date

 Aggregating jurisdictional data under a certain population

Tensions unique to criminal justice data

 Unlike most health and education data, certain criminal justice data 

tends to be available in local jurisdictions (e.g. court records, PRAs)

 But aggregated statewide “Criminal Offender Record Information” 

(CORI) is still confidential

 Criminal records can be stumbling blocks to getting jobs, housing, 

etc. so stakes are high

 There are additional access points (e.g., press releases, news 

coverage) that makes open criminal justice data particularly 

vulnerable to linkage attacks

 No guidelines like HIPAA or FERPA in the law enforcement space
236 The Promise of Evidence-Based Policymaking

“Responsible transparency”

 Tiered data access

 “Anonymized” data on open data portal + data use agreement

 Some data w/ PII available to external researchers upon application

 Some data only available to internal researchers

 Strategy for “anonymized” data

 Holding internal conversations on risk appetite

 Forming committee of experts to advise on value of granularity vs. risk of 

re-identification

 Seeking collaboration with data scientists to model risks and provide 

recommended levels of depth

 Exploring hack-a-thon approach to stress test data for disclosure risks

 Transparency on transparency

 Drafting white paper on how we approached this issue and developed our 

policies

Dataset snowflakes

 Each dataset may contemplate a different balancing test, depending on whose data we 

are dealing with 

 Different stakeholder groups, including:

 Victims of crimes (e.g., sexual assault, DV, child abuse)

 Decedents or relatives (e.g., death in custody)

 Gun owners

 Law enforcement personnel

 Special class groups (e.g., Juveniles, mental health patients)

 Individual arrested but released, acquitted etc.

 Individuals convicted
The Promise of Evidence-Based Policymaking 237

• Moving from compliance to service

• Providing value to states and districts

• Addressing new questions about privacy

Making Data Work for Students: 

The Federal Role

People won’t use data unless it is 

both valuable and trustworthy.

Building Trust is Critical to an 

Evidenced-Based Culture: Lessons 

from the Education Sector

Aimee Rogstad Guidera 

President and CEO
238 The Promise of Evidence-Based Policymaking

THE BIG IDEA

How do I know if my students are learning the material? 

How can I improve my teaching?

Data Answer Key Questions

Is my child on track to graduate ready for opportunities? 

Teachers

Are more of our students prepared for college and 

careers than last year?

Public

Are colleges, universities producing enough graduates 

with right degrees or skills to meet the employer’s needs?

School 

leaders

Policy

makers

Parents

How are the schools in my community performing?
The Promise of Evidence-Based Policymaking 239

• Moving from compliance to service

• Providing value to states and districts

• Addressing new questions about privacy

Making Data Work for Students: 

The Federal Role

What is student data and who uses it?
240 The Promise of Evidence-Based Policymaking

10 Actions to Ensure Effective Data Use: 2005  2011 

Effective Data Use: State Progress

Who Uses Student Data? 
The Promise of Evidence-Based Policymaking 241

• Moving from compliance to service

• Providing value to states and districts

• Addressing new questions about privacy

Making Data Work for Students: 

The Federal Role

As more education data becomes 

accessible and is being used, new 

questions about privacy arise.

10 Actions to Ensure Effective Data Use: 2011

Effective Data Use: State Progress

 2014 

2011 2014
242 The Promise of Evidence-Based Policymaking

2014 State Legislative Activity

110 bills related to 

student data privacy 

in 36states.

27 laws passed 

in 20states.

A Journey Through Time

1974 1998 2005 2008 2011 2012 2013 2015
The Promise of Evidence-Based Policymaking 243

112 bills related to 

student data privacy 

in 34states.

2016 State Legislative Activity

17laws passed 

in 14states.

2015 State Legislative Activity

188 bills related to 

student data privacy 

in 47states.

28laws passed 

in 15states.
244 The Promise of Evidence-Based Policymaking

Student Data Privacy: State Legislative 

Trends

2014: regulating government collection of 

student data

2015: regulating student data collection by 

education technology providers

2016: continuing to govern service providers 

and revisiting existing privacy laws

410 bills related to 

student data privacy 

in 49states.

Summary of State Legislative 

Activity

72laws passed 

in 36states.
The Promise of Evidence-Based Policymaking 245

Four Policy Priorities to Make 

Data Work for Students

• Moving from compliance to service

• Providing value to states and districts

• Addressing new questions about privacy

Making Data Work for Students: 

The Federal Role

Recommendations for Federal 

Policymakers
246 The Promise of Evidence-Based Policymaking

What Federal Policymakers Should 

Do

Provide teachers and leaders the flexibility, training, and 

support they need to answer their questions and take action.

Make Data Use Possible:

• Get your data house in order and better align regulations, 

laws, communications, and messaging across agencies. 

• Break down federal data silos. 

What Federal Policymakers Should 

Do

Be clear about what you want to achieve for students 

and have the data to ensure it gets done.

Measure What Matters:

• Conduct a review of federal data collections and sunset all 

unnecessary or duplicative collections.
The Promise of Evidence-Based Policymaking 247

What Federal Policymakers Should 

Do

Provide teachers and parents timely information on their students 

and make sure it is kept safe.

Guarantee Access and Protect Privacy:

• Build capacity at the federal level to use data effectively 

while safeguarding privacy.

 Expand the ability of the Department of Education to 

provide expert guidance and help to the field (support 

the Privacy Technical Assistance Center)

• Provide incentives for states and districts to build capacity 

for effective data use.

What Federal Policymakers Should 

Do

Ensure that every community understands how its schools and students 

are doing, why data is valuable, and how it is protected and used.

Be Transparent and Earn Trust:

• Produce timely public indicators based on stakeholders’ 

questions.

• Promote greater federal transparency to help families 

understand how the federal government collects, uses, and 

protects data.
248 The Promise of Evidence-Based Policymaking

Aimee Rogstad Guidera 

President and CEO

Data Quality Campaign

Aimee@dataqualitycampaign.org

@DQCAimee
The Promise of Evidence-Based Policymaking 249

Commission on EvidenceBased Policymaking: 

Privacy Perspectives

Marc Rotenberg 

National Academies of Science 

Washington, DC 

September 9, 2016

CEBP 2016 EPIC

Brief Background

• EPIC is an “evidence-based” policy organization. We rely on 

government statistics and reports for our work. 

• EPIC also supports the development of new laws and new 

techniques that enable the use of data while minimizing 

privacy risks 

• Served on many expert panels - AAAS, ABA, IOM, IWG, NAS, 

OECD - with goal of promoting appropriate policy responses 

to emerging challenges. Currently working with NAS on “Big 

Data and Privacy” and OECD on Risk Assessment 

• Speaking for EPIC and not for NAS or OECD

2

CEBP 2016 EPIC

Case Study I: 

Federal Wiretap Reports

• In 1968 Congress created legal authority for 

electronic surveillance in the United States 

• Multiple safeguards were established - criminal 

predicates, application requirements, internal 

accounting, judicial review and public reporting 

• The reporting requirement provides a common 

data set that allows researchers, advocates, and 

government ofcials to describe the scope of 

lawful electronic surveillance in the United States

3
250 The Promise of Evidence-Based Policymaking

CEBP 2016 EPIC

Federal Wiretap Reports: 

Key Conclusions • Stable over time. Mandated by law, not voluntary or 

dependent on private sector data sources, such as 

“transparency reports” 

• Methodology is transparent and data is provable 

• No privacy risk (no PII collected or published) 

• Ongoing relevance to policy debate (crypto 

regulations, Apple v. FBI) 

• Model for evidence-based policy

6

CEBP 2016 EPIC

Case Study II: 

NOAA

• Weather forecasting, climate data, and satellite 

imagery 

• NOAA data supports shing, shipping, 

agriculture, and many associated industries 

• NOAA data also supports mission critical 

functions, emergency services, and local and 

state government 

• No PII!

7
The Promise of Evidence-Based Policymaking 251

CEBP 2016 EPIC

Personally Identiable 

Information (PII) • PII is core concept in modern privacy law. 

• PII = “Data that identies or could identify a particular person” 

• PII creates obligations, “Fair Information Practices.” Obligations 

are asymmetric. Custodian of data has responsibilities. Data 

subjects have rights. 

• Goal is to ensure fairness, transparency, accuracy, and 

accountability 

• New techniques may expand boundaries of PII but that does not 

diminish signicance of concept. As PII becomes more readily 

identied, responsibilities necessarily follow.

9 CEBP 2016 EPIC

Privacy Enhancing 

Techniques (PETs)

• “Techniques that minimize or eliminate the 

collection of PII” (Burkhart 1998, Rotenberg 2000) 

• PETs should be robust, scaleable and provable 

• We support PETs but have also challenged poorly 

designed PETs (MD5, Ask Eraser, SnapChat) 

• CEBP could encourage the development of PETs 

10

CEBP 2016 EPIC

Risk of PII Collection

• Data breach, identity theft, nancial fraud 

• Identity theft is top consumer concern, 

2001-2014 (FTC 2015) 

• Risks are increasing (voting systems) 

• Collection of PII poses risk to institutions and to 

data subjects

11 CEBP 2016 EPIC

Data Minimization

• Video Privacy Protection Act (1988) 

18 U.S.C. 2710 (e)Destruction of Old Records.— 

“A person subject to this section shall destroy 

personally identiable information as soon as 

practicable, but no later than one year from the 

date the information is no longer necessary for the 

purpose for which it was collected and there are 

no pending requests or orders for access to such 

information under subsection (b)(2) or (c)(2) or 

pursuant to a court order.”

12
252 The Promise of Evidence-Based Policymaking

CEBP 2016 EPIC

Hard Problems Ahead

• Data is increasingly dynamic. It is more difcult to control use, 

anticipate outcomes, assess risks. 

• Data is also increasingly under attack from malicious actors. 

Even well intended data collection and analysis may end badly. 

• Increasing focus on “Big Data,” AI, and Data Analytics (“One 

Hundred Year Study on AI,” White House Report on AI) 

• Use of data for proling and prediction has direct impact on 

individuals, even when not PII. 

• “Algorithmic transparency”

13

Data is the basis of research, 

innovation, economic growth, 

and informed policy decisions,

14

Data is also the basis for 

proling, tracking, segmentation, 

and discrimination

15

Privacy protections for data are 

necessary to maximize the 

benets and minimize the risks.

16
The Promise of Evidence-Based Policymaking 253

CEBP 2016 EPIC

References

17

epic.org/bookstore

CEBP 2016 EPIC

Notes on National Data 

Center (1965) • Proposal inspired by social scientists, rise of automation, opportunity to 

gather and analyze data collected by government agencies 

• Tremendous backlash (Packard, “The Naked Society,” a NY Times bestseller) 

• Led to passage of Privacy Act of 1974 => compartmentalized records in 

federal agencies, established limitations on data matching 

• Particular concern about record linkages => additional limitations on 

collection and use of SSN 

• Renewed concerns in US about mass surveillance after 2013 disclosures of 

NSA program (led to end of domestic bulk telephone record collection) 

• EU countries have centralized record systems, but also have stronger laws 

for data protection

18

CEBP 2016 EPIC

Privacy and Innovation: “Shifting the Curve”

19

P1

P2

Protection of PII

Use of PII

A B

C

A = Inefcient outcome 

B = Efcient outcome 

C = Innovative outcome

“Privacy Enhancing 

Techniques (PETs)”
254 The Promise of Evidence-Based Policymaking

Overview

• What does CIPSEA accomplish?

• Who can use CIPSEA?

• What does CIPSEA look like in practice?

• How successful has CIPSEA been?

2

Confidential Information Protection and

Statistical Efficiency Act

(CIPSEA)

1

Katherine K. Wallman

Chief Statistician

U.S. Office of Management and Budget

September 9, 2016
The Promise of Evidence-Based Policymaking 255

Purpose of CIPSEA

• Why we need CIPSEA

– Protection

– Efficiency

• History of CIPSEA

– Challenges despite consensus and work left

undone

3

Who can use CIPSEA?

• Statistical Agencies

• Recognized Statistical Units

• Others

– Directly acquired data

– Through agreement with a statistical agency

o Applies any time an agency ‘acquires’ data for

exclusively statistical purposes, not just

surveys

4
256 The Promise of Evidence-Based Policymaking

How Effective is CIPSEA?

• Disclosure prevention

• Response rates

• Participation from agencies

6

CIPSEA in Practice

• Policies

– Minimize risk of disclosure

– Severe penalties for willful disclosure

• Procedures

– Staff training

– Physical and Information Systems Security (FISMA,

off‐site inspections, etc.)

– Disclosure review

5
The Promise of Evidence-Based Policymaking 257

Setting the Stage: Social Sector Resources

>86,000	
  foundations

have	
  $715	
  Billion	
  in	
  Assets

And	
  give	
  

~$52	
  Billion	
  

annually

From	
  the	
  Foundation	
  Center:	
  

Key	
  Facts	
  on	
  U.S.	
  Foundations	
  (2014)

Largest 1,000	
  Foundations

Give	
  $22.4	
  B	
  annually	
  to	
  nonprofits	
  in...

Health

Education

Human Services

Pub. Affairs/Society Benefit

Arts & Culture

Environment & Animals

22%

22%

16%

12%

10%

7%

11% Other

THE  EVALUATION  ROUNDTABLE

Evaluation Demand and Capacity in the

Social Sector

Tanya Beer

November 4, 2016 Meeting
258 The Promise of Evidence-Based Policymaking

Evaluation Capacity Challenges

Nonprofits:	
  Percent	
  of	
  annual	
  budget	
  spent	
  on	
  evaluation

Most	
   foundations	
   fund	
  

evaluations	
   for	
  less	
  

than	
  10%	
  of	
  their	
  

individual	
   grants	
  

From	
  State	
  of	
  Evaluation	
  (Innovation	
  Network,	
  2016),	
  and	
  Benchmarking	
  Foundation	
  Evaluation	
  

Practices (Center	
  for	
  Evaluation	
  Innovation	
  and	
  Center	
   for	
  Effective	
  Philanthropy,	
  2016)

Trendsin

Philanthropic& Nonprofit

Work

Trendsin

Evaluation& Data

Collection

Emphasis	
   on	
  outcomes	
  and	
  

effectiveness

Growing	
  demand	
   for	
  evaluation	
   to	
  

support	
  decision-­‐making

Focus	
  on	
  complex	
  problem	
  solving	
  

and	
  systems	
  change

Need	
   for	
  systems-­‐level	
   data	
  and	
  new	
  

approaches	
   to	
  evidence	
   building

Interest	
  in	
  scaling	
  in	
  partnership	
  

with	
  the	
  public	
  and	
  private	
  sector

Opportunities	
   for	
  cross-­‐sector	
  

collaboration	
  on	
  evaluation

Disproportionate	
   attention	
  to	
  

performance	
   metrics	
  

Limited	
  bandwidth	
  left	
  for	
  

understanding	
  and	
  building	
  evidence

Limited	
  capacity	
  to	
  use evaluation	
  

to	
  make	
  better	
   decisions

Experimenting	
   with	
  staffing	
  and	
  

processes	
   to	
  support	
   better	
  use
The Promise of Evidence-Based Policymaking 259

Audiences for Nonprofit Evaluation

From	
  State	
  of	
  Evaluation	
  (Innovation	
  Network,	
  2016)

How Evaluative Information is Used

From	
  State	
  of	
  Evaluation	
  (Innovation	
  Network,	
  2016),	
  and	
  Benchmarking	
  Foundation	
  Evaluation	
  

Practices (Center	
  for	
  Evaluation	
  Innovation	
  and	
  Center	
   for	
  Effective	
  Philanthropy,	
  2016)
260 The Promise of Evidence-Based Policymaking

The Edna McConnell Clark Foundation 1

Evidence Building

Overview

The Edna McConnell Clark Foundation has over a decade and a half’s 

experience helping nonprofits use data and build evidence of their 

programs’ effectiveness and improve and expand them to lift the life 

prospects of greater numbers of economically disadvantaged youth in the 

U.S. Since 2007, we have leveraged $155 million of our own resources to 

help 16 grantees secure nearly $487 million in additional private and 

public funding. 

Our grantees and the communities, families and young people they serve 

have benefited from the increased emphasis that OMB, federal agencies 

and Congress have placed on evidence-based programs. 

We are proud to have partnered with the federal government in the Social 

Innovation Fund, an initiative enlisting private intermediaries to help 

expand evidence-based programs. 

We are pleased that the Evidence Commission is advancing the “what 

works” agenda by setting clear standards and expectations for the use of 

data and evidence in policy and procurement. 

And we are honored by the opportunity to share our thoughts and 

recommendations for your consideration. 

Evidence Commission 

A Funder’s Perspective DRAFT

November 4, 2016
The Promise of Evidence-Based Policymaking 261

The Edna McConnell Clark Foundation 3

Evidence Building

Potential Solutions 

Drawing on our grantmaking experience and recent research, we offer the following ideas 

for consideration by the Evidence Commission. 

• Service providers are often on the front line of implementing federal programs, but they 

lack a close connection to government that would make it easier for them to innovate, 

contribute to, take advantage of, and execute evidence-based policy decisions

• Encourage better understanding of the role and magnitude of the private 

providers’ network in the development and delivery of evidence-based 

practices.

• Alliance for Strong Families and Communities estimates 109,515 human 

services agencies.

• $750m budget for services in Child Welfare in NYC alone for preventive and 

foster care service procured through the private sector. 

• Wisconsin has >100 agencies providing foster care

• Consider a platform for engaging nonprofits and educating the field on the 

process of evidence building as a way to extend and expand beyond its 

statutory lifetime the Commission’s efforts to reach this important sector. 

• Support efforts to promote, prioritize and incentivize the use of evidence in 

procurement. Help ensure a clear and unambiguous link of funding to the building & 

use outcomes.

• Use evidence to accurately match programs to the needs of target populations, 

avoiding one-size-fits-all thinking. 

• Support efforts to promote providers’ evidence building & outcomes, not just 

compliance.

• Set clear evidence and data standards and requirements that can be shared across 

government and the private sector. 

• Remove barriers to accessing federal, state and local data and promote the 

linkage and aggregation of data by all stakeholders. 

• Service providers need to access data from various public programs (including, 

but not limited to, the programs to which they may be directly connected) in 

order to tailor and track the outcomes of interventions for the communities they 

serve.

• Pooled data creates healthy, affordable comparison groups that facilitate 

evaluation and learning. 

The Edna McConnell Clark Foundation 2

Evidence Building

Constraints for Using & Building Evidence

EMCF has learned from long experience and recent interviews of over 80 thought 

leaders that despite increased demand for evidence-based programs, the current 

approach to “evidence building” – amassing empirical data measuring impact, 

ideally through third-party evaluation – is often too slow, expensive, incremental 

and insufficient for non profits and their funders due to current limitations. 

• Funding for evidence building and supportive capacities (technology, data, and 

policy) is limited and too often in the hands of third parties. Nonprofits are 

forced to take a "pay as you go" approach rather than follow consistent and 

embedded practice.

• The current roles and incentives of evaluation researchers are not well aligned 

with practitioners’ needs or local context. Too often non profit providers are the 

"caboose" of the evidence-building train when they should be the engine.

• Policy-makers’ demand for programs that work runs ahead of nonprofits’ ability 

to deliver such evidence-based programs. We need a greater supply of 

organizations and leaders equipped to build and use evidence-based solutions.

• While policy-makers, government and funders are demanding evidence, they 

seldom understand the operational, data and funding realities that nonprofits 

face AND they do not use such evidence consistently to set policy or award 

contracts. 

In sum, limited innovation in evaluation, constrained data access and weak links 

between evidence, public policy, implementation, and financial sustainability 

constrict the pipeline of evidence-based programs worthy and ready for large-scale 

investments by philanthropy, government, and not-for-profits.
262 The Promise of Evidence-Based Policymaking

The Edna McConnell Clark Foundation 4

Evidence Building

Partners in Building Evidence

The Evidence Commission can encourage both public and private funders to support 

the “what works” agenda in the following ways: 

• Fund data collection, evaluation, and learning as ongoing standard operating 

procedure. Evidence building is a continuous rather than intermittent, one-studyat-a-time process. Far from overhead, it is integral to a nonprofit’s success and 

requires investment in:

• Talent (leadership, data scientists, analysts, evaluators)

• Technology and tools

• Strategic evaluation planning

• Build or buy the capacity to use data to generate evidence, and ensure that 

service providers and communities have access to these assets. 

• Foster a better understanding and use of different levels of evidence and 

prioritize investment decisions based on data and evidence.

• Entry evidence reviews

• Clearinghouses 

• Inform and align policy priorities with federal data and evidence standards 

• Use data to identify and then fund gaps in evidence where it is critically needed 

to address policy priorities. 

• Incentivize innovation of less costly tools and more timely approaches to 

evaluation. 

• Understand where federal and state government is constrained and leverage 

philanthropy to expand the pipeline of policy-relevant evidence-based programs. 

• Hold researchers and technical assistance providers accountable for 

designing evaluations appropriate to the stage of organizational development and 

the capacity of nonprofits and to the communities they serve. 
The Promise of Evidence-Based Policymaking 263

Statement of

Adam Gamoran

President, William T. Grant Foundation

New York, NY 

To the

Commission on Evidence-Based Policymaking

Washington, DC

Panel on 

“Non-Governmental Demand for Evaluation: Capacity to Support Public Good Activities”

November 4, 2016
264 The Promise of Evidence-Based Policymaking

Adan Gamoran 

Statement to the Commission on Evidence-Based Policymaking

Panel: “Non-Governmental Demand for Evaluation: Capacity to Support Public Good Activities”

November 4, 2016.

Thank you for the opportunity to speak today about one of our nation’s most vexing challenges: 

serving the public good by making smart policy decisions using data that our government 

already collects.

1 The Commission on Evidence-Based Policymaking offers a rare opportunity to 

overcome significant barriers that prevent us from achieving this goal. My aim is to provide a 

non-governmental perspective on two key issues facing the Commission:

1. What data are needed to support public good activities?

2. What steps are needed to build the capacity to use data and create the evidence required for 

better policy decisions?

The William T. Grant Foundation, which I lead, supports social science research to improve the 

lives of young people ages 5 – 25 in the United States. Within this overall mission, we have two 

areas of focus. The first is to support research on reducing inequality among young people, and 

the second is to support research on improving the use of research evidence in policy and 

practice. These priorities would be greatly aided if the Commission were able to accomplish its 

goals. 

As background to my remarks, I call your attention to the legislation that established the

Commission. Your primary charge is to determine how agencies of the federal government can 

share and link administrative data sets, and accomplishing that aim would be a significant step 

forward. However, a complete reading of your mandate suggests you need not stop there. The 

Commission is uniquely positioned to consider how the federal government can use data to 

create the evidence required to achieve our policy aims, as well as how to create the 

infrastructure to support the use of evidence in policymaking. In other words, be the evidence 

commission, not just the data commission.

This broader approach is fully within your charge, and you can meet the challenge it sets forth 

by prioritizing its call to show how data “may be integrated and made available to facilitate 

program evaluation, continuous improvement, policy-relevant research, and cost-benefit 

analyses by qualified researchers and institutions,” and by emphasizing “how data and results of 

research can be used to inform program administrators and policymakers to improve program 

design.”2 It is principally these elements of your charge to which my remarks offer a response.

 1 My remarks draw on recent writing from the William T. Grant Foundation and the Forum for Youth 

Investment, including the following:

William T. Grant Foundation and Forum for Youth Investment. (2016). From data to evidence to policy. 

New York: William T. Grant Foundation. Available at: 

http://wtgrantfoundation.org/library/uploads/2016/08/From-Data-to-Evidence-to-Policy.pdf

Gamoran, A., & Ferber, T. (2016). Who would have thought: Bipartisan policymaking through evidence. 

The Hill, August 8. Available at: http://thehill.com/blogs/congress-blog/politics/290684-who-wouldhave-thought-bipartisan-policymaking-through-evidence

DuMont, K., & Smeeding, T. M. (2016). Using data to produce useful research evidence. The Digest, Issue 

1. New York: William T. Grant Foundation. Available at: http://wtgrantfoundation.org/digest/using-dataproduce-research-evidence 2 See Section 4, “Duties of the Commission,” parts (a)(1) and (b)(2)(I), in Evidence-Based Policymaking 

Commission Act of 2016, H.R. 1831. Available at: https://www.congress.gov/114/bills/hr1831/BILLS114hr1831eas.pdf
The Promise of Evidence-Based Policymaking 265

2

Data Needs: Multiple Purposes of Linked Administrative Data

Sharing and linking administrative data is necessary but will not suffice to achieve the broader 

goals of the Commission. Rather, data must be deployed in research and evaluation to create 

research evidence that informs policymaking. This would optimize the use of data we already 

collect to make smarter policy decisions. By “research evidence” I mean evidence derived from 

applying systematic methods and analyses to address a predefined question or hypothesis.

Examples include descriptive studies, intervention studies and evaluations, meta-analyses, and 

studies on cost effectiveness. Policy insights can emerge from a variety of types of research 

studies, and, consequently, linked administrative data has multiple purposes and offers diverse 

contributions to policymaking. As a private funder of research intended to improve the lives of 

young people in the U.S., the Foundation has identified a range of cases for which linked 

administrative data can address critical questions. Among these are rigorous evaluations of 

program impact, both experimental and quasi-experimental; program improvement efforts,

including performance management; and descriptive studies that contribute to policy formation.

Rigorous Evaluations of Program Impact

As is widely discussed, administrative data can be a powerful tool when attached to randomized 

evaluations of program impact. At an earlier meeting, the Commission heard from Raj Chetty of 

Stanford University, who obtained tax data that enabled him to test for long-term effects of the 

Moving to Opportunity Study. His results overturned previous findings, transforming our 

understanding of how housing voucher program effects may occur. If such federal data were 

available to a wider range of researchers, we would have more secure answers to many 

important questions about program impact. Also frequently mentioned is the value of 

administrative data for short-cycle randomized trials. Because administrative data are already 

being collected irrespective of whether a program evaluation is taking place, linking existing

data to the evaluation study substantially reduces the cost of the study, and may accelerate its 

completion and lead to more real-time information on whether programs are meeting their 

aims. For example, the Laura and John Arnold Foundation has stimulated the research 

community to pursue this course to take better advantage of data already being collected.3

Two randomized experiments that our Foundation is funding – one by Joseph Allen at the 

University of Virginia and the other by David Yeager at the University of Texas-Austin – use 

administrative data to assess the impact on academic and social-emotional outcomes of 

interventions designed to improve young persons’ social psychological functioning. These 

investigators face the laborious task of collecting administrative records from each school, one 

school at a time. Particularly in the case of the Yeager study, which includes nearly 100 schools, 

this task demands countless hours from both researchers and school personnel, and great 

expense to the funders. Yet much of the data – notably the academic outcomes – are reported up 

to the state and even the federal government, and if there were a data linkage system in place, 

they could be gathered much more efficiently. Indeed, this is one of the reasons we funded the 

Stanford Educational Data Archive, which includes test scores from all 50 states calibrated on a 

common scale.4 But since the National Center for Education Statistics that supplied these data

has no provision for releasing individual-level data from state assessments, the data are 

aggregated to the level of grades within schools, which constrains their utility for some

purposes.

 3 See: http://www.arnoldfoundation.org/laura-and-john-arnold-foundation-announces-expandedfunding-for-low-cost-randomized-controlled-trials-to-drive-effective-social-spending/

4 See: https://cepa.stanford.edu/seda/overview
266 The Promise of Evidence-Based Policymaking

3

Administrative data also have great value for quasi-experimental studies, which can often be 

conducted without additional data beyond the administrative records. Professor Chetty has 

already spoken to the Commission about how his analysis of tax records for five million 

geographically mobile American families revealed the importance of neighborhood quality for 

economic mobility. In a study funded by our Foundation, Cornell economist Michael Lovenheim 

and his colleagues were able to connect Texas state administrative data from K12 education, 

higher education, and employment and, using a difference-in-difference design that permits 

causal inference, test the effects of two university scholarship programs for high-achieving, lowincome youth on college and workforce outcomes.5 In another study we are currently funding, 

researchers at Duke University have cleverly used job loss information from North Carolina as a 

statistical instrument to test the effects of economic and family changes on children’s academic 

development.6 This research was only possible because the investigators were able to link 

administrative data from the state education and employment systems.

It is no accident that these two examples come from Texas and North Carolina. Along with 

Florida, these states have the most efficient systems for external researchers to obtain access to 

statewide education data. Consequently, much of what we know about federal and state 

education policy effects, particularly those based on rigorous quasi-experimental methods, 

comes from these three states, and we cannot be certain about how findings from these states 

generalize to other states. Ironically, thanks to federal data requirements under No Child Left 

Behind and subsequent federal incentives to states, all states now collect a large volume of 

education data, with students linked over time and to their schools and, in most cases, to their 

teachers. However, in far too many states the data languish unused except for compliance 

reports under federal law. These data are an untapped treasure the Commission could help 

uncover by advocating for federal guidelines that would facilitate the research use of state 

education data while protecting the privacy of personal information as Texas, North Carolina, 

and Florida have carefully done.

Program Improvement

Many, and perhaps most, experimental and quasi-experimental studies focus on a narrow 

question: What is the impact of a program on a particular outcome? Increasingly, however, our 

grantees recognize that the answer to this question is of limited policy value. Although it 

indicates whether a program is working or not, it does not say what works for whom and under 

what circumstances, nor does it say what steps may be undertaken to achieve the desired 

results. These questions are important because single evaluations rarely suffice to make 

decisions about program continuation or discontinuation. Instead, they are most useful if they 

point the way towards improvement. As Ron Haskins and Greg Margolis have explained, “An 

important part of a comprehensive, evidence-based strategy will be to continue funding 

programs that initially receive disappointing evaluations. Part of the federal evidence-based 

culture should be that federal agencies will work with programs and continue funding them as 

long as they are using evidence to improve their outcomes and are showing some progress.”7

 5 Andrews, R. J., Imberman, S. A., & Lovenheim, M. F. (2016). Recruiting and supporting low-income, 

high-achieving students at flagship universities. NBER Working Paper No. 17104. Cambridge, MA: 

National Bureau of Economic Research.

6 Ananat, E. O., Gassman-Pines, A., Francis, D. V., & Gibson-Davis, C. M. (2011). Children left behind: The 

effect of statewide job loss on student achievement. NBER Working Paper No. 22260. Cambridge, MA: 

National Bureau of Economic Research.

7 Haskins, R., & Margolis, G. (2015). Show me the evidence: Obama’s fight for rigor and results in social 

policy. Washington, DC: Brookings Institution Press, p. 235.
The Promise of Evidence-Based Policymaking 267

4

Program improvement cannot happen without data about when and why a program works well

or struggles. Some of our recent grants have supported methodological work to better 

understand treatment effect heterogeneity, that is, differences in how the same program may 

work differently for different people or in different places.

8 These technical advances, however, 

rest on the availability of data to identify conditions that may be related to heterogeneity. If 

randomized trials are often undertaken as “black box” studies – particularly short-cycle 

randomized trials – administrative data may allow us to peer into the box to see what is actually 

going on. 

On the government side, many are aware of the need for such administrative data, and this was 

a topic of discussion in a learning community that our Foundation organized for staff of federal 

evaluation and research agencies. As one participant commented, “research needs to provide 

more information about community context, implementation, health equity, and costs, as well 

as info about what facilitated or impeded success.” Another elaborated, “on the grant 

competition side we have been adding requests for researchers to add more on context, but that 

leads to questions about what question should be asked across studies, what data is cheaply 

available.” The Commission could encourage this work by providing guidelines for federal 

grantmakers that would strengthen researchers’ access to contextual and implementation data, 

and by calling for agencies to make administrative data available in response to these needs.

Performance management is a tool used for program improvement that relies substantially on 

administrative data.9 To assess the outcomes of a government program, however, a federal 

agency typically requires administrative data that are gathered elsewhere within the 

government. While current federal law requires agencies to set performance goals, it does not 

require them to share the data that are needed by other agencies to assess progress towards 

those goals. The Commission can bolster federal program improvement efforts by 

recommending policy changes that will support goal setting with data-sharing requirements. For 

example, laws that require evaluation could specifically require data-sharing agreements. 

Policy Formation

A third purpose of linked administrative data is to understand the nature of the challenges 

facing our nation and to identify possible programs and policies that may address them. For 

instance, President Obama has declared that income inequality is “the defining challenge of our 

time.”10 Data that our government already collects, but which are not currently well used, could 

help us formulate more effective policies to respond to this challenge. Two examples illustrate 

this point: Linkages between state education records and national surveys can identify ways to 

reduce educational inequality; and the American Opportunity Study, a national effort to link 

census, program, and survey data, can yield ways to increase upward mobility on the ladder of 

economic success.

 8 Weiss, M. J., Bloom, H. S., & Brock, T. (2015). A conceptual framework for studying the sources of 

variation in program effects. Journal of Policy Analysis and Management, 33, 778-808; Raudenbush, S.

W., & Bloom, H. S. (2015). Learning about and from variation in program impacts from multisite trials.

Working paper. New York: William T. Grant Foundation.

9 Metzenbaum, S. H., & Shea, R. (2016). Performance accountability, evidence, and improvement:

Reflections and recommendations to the next administration. Washington, DC: National Academy of 

Public Administration and The Volcker Alliance.

10 See: https://www.whitehouse.gov/the-press-office/2013/12/04/remarks-president-economic-mobility
268 The Promise of Evidence-Based Policymaking

5

Linkages between state education records and national education surveys. Educational 

inequality is a fundamental aspect of broader inequities across our nation, so reducing 

educational inequality is an important goal. Evidence about educational inequality comes from 

national surveys as well as from state administrative records. Linking the two sources of data 

would offer an especially powerful tool for crafting effective policies. While state data contain 

valuable information about the performance of students, teachers, and schools, they typically 

lack the contextual information needed to understand program and policy effects. Longitudinal 

surveys, by contrast, often have rich contextual information but lack repeated data on student 

outcomes, and often lack detailed information on teachers and course enrollment. Linking the 

two data sources would strengthen the evidence offered by each.11 In particular, linking state 

data to in-depth longitudinal surveys carried out by the National Center for Education Statistics

(NCES) would allow a much richer exploration of policy effects within and among states.

During the Obama administration, NCES officials negotiated with ten state education agencies 

to prepare to link a national longitudinal cohort study of high school students (a common type 

of NCES survey) with state education records. NCES did its part by oversampling students in 

each of the ten states so the linked samples would be representative of each state and large 

enough to test policy-relevant hypotheses. But as I have written elsewhere, “ultimately not a 

single state provided its data for linkage with the national survey. Despite agreement at the 

political level, NCES and its state counterparts were unable to resolve the bureaucratic barriers 

to linking state and federal data.”12 This unfulfilled promise is ironic because the state 

longitudinal data sets were largely built with federal grants to states, yet the federal government 

failed to use its leverage to compel or even encourage states to make their data available for 

linkage. The Commission could address this challenge by recommending legislative or 

administrative language that helps states understand the value for their own decision-making,

as well as for achieving national goals, of connecting their data systems to national data with indepth information about family background, students’ experiences within schools, school 

context, and other conditions relevant to policy formation and outcomes.

The American Opportunity Study. As David Grusky, Tim Smeeding, and Matt Snipp have 

written, given the high importance that Americans attach to equal opportunity, one might 

expect us to have a robust system for monitoring social mobility and its responsiveness to policy 

changes.13 By social mobility, we mean the chances that persons born into disadvantage can rise 

above their circumstances of origin to achieve educational and occupational success as adults.

Despite the salience of this notion, our ability to monitor changes in mobility is weak. The last 

major survey of U.S. mobility, the second study of “Occupational Changes in a Generation,” 

occurred over 40 years ago. Today, however, a standalone survey is not needed to gauge 

patterns of mobility across generations. In fact, we can do so even more effectively than in the 

past by linking data from the U.S. census to federal administrative data and to existing national 

surveys that are already being conducted. Grusky, Smeeding, Snipp, and others have proposed 

that a new American Opportunity Study (AOS) can be accomplished without the need for a new 

survey, by linking existing data. Key components of the AOS include decennial census records, 

 11 Loeb, S. (2015). Linking NCES surveys to administrative data. Paper presented at the Workshop to 

Examine Current and Potential Uses of NCES Longitudinal Surveys by the Education Research 

Community, Washington, DC. Available at: 

http://www.naeducation.org/cs/groups/naedsite/documents/webpage/naed_160699.pdf

12 Gamoran, A. (2016). Towards the next generation of U.S. longitudinal surveys: Ideas from researchers 

for the National Center for Education Statistics. AERA Open, 2 (2) 1-8.

13 Grusky, D. B., Smeeding, T. S., & Snipp, C. M. (2015). A new infrastructure for monitoring social 

mobility in the United States. Annals of the American Association of Political and Social Science, 657, 63-

82.
The Promise of Evidence-Based Policymaking 269

6

federal tax and employment records, state data from federal programs such as food stamps and 

unemployment insurance, and national surveys that are already being conducted such as the 

American Community Survey, the Survey of Income and Program Participation, and others.

The AOS would carry substantial benefits for those wishing to assess mobility trends and to 

formulate and assess policies that aim to strengthen equal opportunity processes. As the authors 

explain, 

The United States has an unassembled panel that is standing unused and that, for a 

relatively small outlay, could be transformed into a major new infrastructural resource in 

the social sciences. The AOS comes with substantial cost savings and efficiencies, allows 

the United States to formulate child development and labor market policy using highquality evidence, and would lead to a renaissance of labor market and mobility research 

that would almost surely reestablish the United States as a leader in the field.14

The AOS would permit more powerful assessments of conditions that foster and reduce 

inequality in domains such as neighborhoods, family structure, health, justice, education, and 

veterans’ affairs.15 These assessments would allow formulation of evidence-based policies. When 

linked to research study samples, the AOS could also support program evaluation in many areas. 

Despite the promise of this approach for meeting the inequality challenge that President Obama 

has called out, substantial barriers remain. Most important, much of the data about program 

participation resides at the state level, even for programs supported by federal funding. If we 

wish to know whether our policies are working, we need to know who is participating and what 

their long-term and even intergenerational trajectories are. We can accomplish this aim by 

linking program participation data to census and employment records, but only if states 

cooperate with federal agencies in data sharing. The federal government ought to have leverage 

in this regard because the programs are federally funded. The Commission could address this 

challenge by recommending legislative language that would require states to share data on 

program participation for purposes of research on policy formation and evaluation.

Linking ongoing national surveys to administrative data represents another challenge. I 

understand that for Census Bureau surveys such as the American Community Survey and the 

Survey of Income and Program Participation, such linkages are already occurring and data can 

be analyzed in the secure Federal Statistical Research Data Centers. The Commission may wish 

to urge other federal agencies and non-governmental research groups to collaborate with the 

Census Bureau to add additional surveys to this infrastructure.

More broadly, the Commission should aim to standardize and streamline procedures to 

facilitate linkages among these disparate sources of data, and encourage a bureaucratic/legal 

environment in which such linkages are viewed as assets. To accomplish this goal, the 

Commission should provide a framework that articulates the technical requirements, legal 

standing, and accepted procedures for linking and sharing data across federal agencies and with 

willing state partners. The framework should specify the different types of data that may be 

linked, including administrative, survey, and experimental data, and the different purposes of 

such linkages, including program continuation decisions, program improvement plans, and 

policy formation.

 14 Ibid, p. 79.

15 National Academies of Sciences, Engineering, and Medicine. (2016). Using linked census, survey, and 

administrative data to assess longer-term effects of policy. Proceedings of a workshop in brief. 

Washington, DC: National Academies of Sciences, Engineering, and Medicine.
270 The Promise of Evidence-Based Policymaking

7

The Commission might further recommend a pilot that would begin with specific agencies that, 

based on the investigations of the Commission, seem best prepared to implement a standardized 

procedure for linking data and making data accessible to researchers inside and outside of 

government. With support from the Laura and John Arnold Foundation and in partnership with 

the U.S. Census Bureau, Chapin Hall at the University of Chicago is sponsoring a series of 

studies to demonstrate innovative approaches to linking administrative data, and the 

Commission could look to these studies as examples.

Building Capacity for Using Linked Data to Conduct Research and Inform Policies 

Thus far I have discussed the value of administrative data for rigorous evaluations of program

impact, program improvement, and policy formation, and I outlined a series of 

recommendations for improving the linkages among, and access to, these data while protecting 

privacy. I could choose to conclude my remarks now, and indeed, the Commission could choose 

to limit its focus to the tasks of linking and sharing administrative data while protecting privacy.

But doing so would not take full advantage of the opportunity at hand. As I noted at the outset, 

the Commission is uniquely positioned to advance the capacity to support the use of evidence in 

policymaking. By capacity I am referring to the technological structures, human capital, 

organizational arrangements, and fiscal investments needed to create and use evidence. These

considerations are often an afterthought and therefore under-resourced. Because they are 

fundamental to the creation of evidence, they demand attention at the outset.

Over the course of my career, I have thought a lot about the capacity required to support the 

production and use of evidence. Prior to my current role, I directed the Wisconsin Center for 

Education Research, the oldest and largest university-based education research center in the 

country. My job was to ensure that we had the structures in place both to generate evidence and

connect with those who might use it. In my current role as president of a foundation that funds 

studies to identify and test strategies to improve the use of research evidence at the federal, 

state, and local levels, I am learning continually about the conditions needed to facilitate 

evidence use. This knowledge has been deepened by my observations of the learning community 

we facilitated for federal research and evaluation staff. These experiences make clear that

capacity is critical for supporting the production of evidence and its use. 

Yet the current capacity to accomplish this goal is limited. In a government-wide survey 

conducted by the General Accountability Office, only eleven agencies reported “committing 

resources to obtain evaluations by establishing a central office responsible for evaluation of 

agency programs, operations, or projects, although only half these offices were reported to have 

a stable source of funding. Seven agencies reported having a high-level official responsible for 

oversight of evaluation.”16 The relative scarcity of evaluation offices and funding has 

consequences: The General Accountability Office reported that “studies of organization or 

government evaluation capacity have found that it requires analytic expertise and access to 

credible data as well as organizational support both within and outside the organization to 

ensure that credible, relevant evaluations are produced and used.”17

 16 General Accountability Office. (2014). Program Evaluation: Some Agencies Reported that Networking, 

Hiring, and Involving Program Staff Help Build Capacity. GAO-15-25. Washington, DC: General 

Accountability Office.

17 Ibid, p.8.
The Promise of Evidence-Based Policymaking 271

8

Based on my experiences outside government and my interactions with those in government, I 

encourage the Commission to respond to the findings from the General Accountability Office as 

well as to similar concerns raised in a background paper prepared for the Commission.18 The 

Commission can make an essential contribution to the advancement of evidence-based 

policymaking by recommending specific steps (a) to build the capacity of researchers inside and 

outside of government to use administrative data to answer policy questions, and (b) to establish 

partnerships between researchers and policymakers that will increase the likelihood that 

research evidence permeates the policymaking process.

A Federal Infrastructure for Evidence

The Commission can meet its full mandate by providing guidance on ways that data may be used 

to create evidence that can inform and shape policy. A stronger infrastructure within the federal 

government would help build the capacity needed to meet this charge. Such an infrastructure 

could help ensure that the production and use of evidence remains a high priority for our 

nation’s decision-makers. It could also increase the capacity of individual agencies to produce 

and use evidence. Moreover, a strong infrastructure could help align efforts across agencies, 

leading to greater consistency and creating opportunities to take on challenges that require 

cross-agency collaboration.

The Commission may wish to examine infrastructure supports that are found in various 

agencies, identify best practices, and recommend their wide adoption. One of our grantees is 

currently conducting a scan of the existing federal infrastructure for statistics, data, 

performance improvement, behavioral analytics, and evaluation, and has identified a number of 

supports that could be strengthened for federal evaluation agencies. Examples of such 

infrastructure elements include:

 Leadership positions focused on evidence use

Leadership positions such as the special advisor for evidence-based policy in the Office of 

Management and Budget, and the chief technology officer in the White House Office of Science 

and Technology policy, serve to focus attention on the use of research evidence and are 

positioned to align efforts across agencies. One role of such leaders can be to foster collaboration 

between program and evaluation offices within government agencies to establish trust and make 

joint decisions about what research and evaluation is to be conducted and how the findings may 

be used.

 Interagency collaborative bodies

An interagency group that regularly convenes staff members from across government agencies 

offers opportunities for learning and collaboration. Examples include the federal Chief 

Information Officers’ Council, the Interagency Council on Evaluation Policy, and the Committee 

on National Statistics of the National Academies. In addition to supporting and formalizing such 

bodies, more could be done to align efforts across the collaboratives focused on data, statistics, 

and evaluation.

 18 Office of Management and Budget. (2016). Barriers to Using Administrative Data for Evidence 

Building. Background paper for the Commission on Evidence-Based Policymaking. Washington, DC: 

Office of Management and Budget. See: https://www.whitehouse.gov/sites/default/files/omb/mgmtgpra/barriers_to_using_administrative_data_for_evidence_building.pdf
272 The Promise of Evidence-Based Policymaking

9

 A dedicated office for research and evaluation within each agency

The capacity for building and using evidence is highly varied across federal agencies. Some 

agencies have dedicated research offices, such as the Office of the Assistant Secretary for 

Planning and Evaluation within Health and Human Services, the Chief Evaluation Office in the 

Department of Labor, the Office of Policy Development and Research within Housing and Urban 

Development, and the Institute of Education Sciences within Education. While establishing such 

offices is important, they must be accompanied by systems that foster communication between 

research and program offices. As one research leader who participated in our learning 

community of federal agency staff explained, “When I think about the program offices I 

collaborate with, and what makes them successful, I think about finding mutual goals, how to 

help those offices achieve their goals, maintaining regular communication throughout and, most 

importantly, assuming that there will be different cultures.”

 A codified set of principles and practices

As another participant in our learning community of federal agency staff noted, “There isn’t 

much of a blueprint for how federal agencies conduct evaluation.” Individual agencies have 

responded to this need; for example, both the Department of Labor and the Administration on 

Children and Families in the Department of Health and Human Services have developed policies 

for evaluation, which is one type of research evidence. A broader set of principles and practices 

for the creation and use of research evidence is needed, analogous to Principles and Practices 

for a Federal Statistical Agency, which has been influential in setting standards and approaches

that statistical agencies across policy domains have adopted. Likewise, the Common Guidelines 

for Education Research and Development have helped the Institute of Education Sciences and 

the National Science Foundation take similar approaches in understanding and supporting 

different types of research evidence. Moreover, just as there needs to be an investment in the 

skills and guidelines for conducting impact evaluations, similar supports should be established 

for program improvement efforts. The Commission could recommend the development and 

adoption of principles and practices for evidence creation and use that could become widely 

shared across federal agencies.

Partnerships for Building and Using Evidence

All too often, evidence about effective or ineffective policies or programs has little bearing on 

decisions, even when the evidence is rigorous, timely, and accessible. Sustained partnerships 

between researchers and policymakers can improve the use of research evidence by offering a 

basis for trust, incentivizing researchers to ask questions that really matter and creating a 

culture of evidence in the decision-making body. This requires an ongoing dialogue between 

researchers and decision-makers.

Throughout my remarks I have emphasized the value of administrative data for research and 

evaluation, and I gave several examples where this approach has been the case. These situations 

were fortunate; the existing administrative data happened to be what researchers and evaluators 

needed for their studies. To maximize the use of administrative data for research and evaluation, 

however, we can be more deliberate in what data are collected. When researchers act as partners 

to program and policy staff, they can collaborate on decisions about what administrative data to 

collect. These partnerships can improve the odds that programmatic and policy changes will be 

informed by the research evidence.
The Promise of Evidence-Based Policymaking 273

10

Since 2009, the William T. Grant Foundation has supported studies of the conditions that 

promote the use of research evidence in policy and practice. Partnerships between researchers 

and decision-makers emerged as a key finding from this scholarship as a mechanism for getting 

evidence produced and used. Most common in the domain of education, a research-practice

partnership is a sustained structure for facilitating relations of trust and shared goals among 

university-based researchers and government-based practitioners or policymakers, such as 

school district officials.19 The partnership carries two essential benefits. First, because the 

research agenda within the partnership is co-constructed by researchers and practitioners, the 

questions pursued in the research are ones whose answers are important to the practitioners.

Second, the partnership creates a culture of evidence within the agency, such that looking to 

evidence before making decisions becomes normative practice. Fundamental to the partnership 

is the sharing of administrative data that allows researchers to address policy questions of 

interest to school district decision-makers. Most education partnerships are between 

universities and school districts, but the state of Tennessee is leading the way by creating a new 

partnership between Vanderbilt University and the state education agency.

The Commission can improve the use of research evidence by identifying effective partnership 

models and promoting them across the federal government. Many models of partnership exist, 

such as those laid out on the website of the William T. Grant Foundation: 

http://rpp.wtgrantfoundation.org/.

Conclusion and Summary of Recommendations

The Commission on Evidence-Based Policymaking can help researchers outside government 

contribute more useful research that informs policy by recognizing the variety of purposes for 

sharing and linking administrative data, and by recommending tools and relationships that 

strengthen the capacity of policymakers to collaborate with researchers inside and outside of 

government. A number of recommendations emerge from this account:

Multiple Purposes of Linked Administrative Data

 Advocate for federal guidelines that would facilitate the research use of state data while 

protecting the privacy of personal information.

 Provide guidelines for federal grantmakers that would strengthen researchers’ access to 

contextual and implementation data, and call for federal agencies to make administrative 

data available in response to these needs.

 For federal performance management systems, recommend policy changes that support 

goal-setting with data-sharing requirements.

 Recommend legislative or administrative language that helps states understand the value for 

their own decision-making as well as for achieving national goals of connecting state data 

systems to national survey data with in-depth background information.

 Recommend legislative language that would require states to share data on program 

participation for purposes of research on policy formation and evaluation.

 19 Turley, R. N. L., & Stevens, C. (2015). Lessons from a school district-university research partnership: 

The Houston Education Research Collaborative. Educational Evaluation and Policy Analysis, 37, 6S-15S.
274 The Promise of Evidence-Based Policymaking

11

 Urge federal agencies and non-governmental research groups that conduct national surveys 

to collaborate with the Census Bureau to add additional surveys to the Federal Statistical 

Research Data Center infrastructure.

 Provide a framework that articulates the technical requirements, legal standing, and 

accepted purposes or linking and sharing data across federal agencies and with willing state 

partners.

 Recommend a pilot data-sharing effort, perhaps building on the examples of innovative 

approaches to linking administrative data.

Building Capacity of the Production and Use of Research Evidence

 Examine existing infrastructure supports found in various agencies, identify best practices, 

and recommend their wider adoption. Examples include leadership positions focused on 

evidence use; interagency collaborative bodies; dedicated offices for research and evaluation; 

and codified sets of principles and practices.

 Identify models of partnerships between researchers and policymakers, and promote their 

use across the federal government. Consider local and state examples as models for the 

federal level.

These recommendations are ambitious, but the Commission is such a unique opportunity that 

lofty goals are warranted. As my colleagues have written, “research evidence can improve public 

policies and programs, but fulfilling that potential will require honest assessments of current 

initiatives, coming to terms with outsized expectations, and learning ways to improve social 

interventions and public systems.”20 The Commission is well positioned to drive this work 

forward, especially if it focuses on the full continuum of activity from sharing and linking data, 

to using those data to create research evidence, to using that evidence to inform policymaking.

 20 See: http://wtgrantfoundation.org/tag/evidence-at-the-crossroads
The Promise of Evidence-Based Policymaking 275

To serve the public good by making smart 

policy decisions, using data our government 

already collects.

1. What data are needed to support public good 

activities?

2. What steps are needed to build the capacity to 

use data and create the evidence required?

The Challenge:

From Data to 

Evidence to 

Policy

ADAM GAMORAN

William T. Grant Foundation

Statement to the 

Commission on 

Evidence-Based 

Policymaking
276 The Promise of Evidence-Based Policymaking

Be the Evidence Commission, 

not just the Data Commission.

The Message

Supports research 

to improve the 

lives of young 

people.

Areas of focus:

 Reducing inequality

 Improving the use 

of research 

evidence

1940’s

1950’s

1930’s

1960’s

1980’s

1970’s

1990’s

2000’s

2010’s
The Promise of Evidence-Based Policymaking 277

Research evidence

Derived from applying systematic methods and 

analyses to predefined questions or hypotheses

Data Needs

Multiple purposes of linked administrative 

data

Rigorous evaluations of program impact

 Experimental

 Quasi-experimental

Program improvement

Policy formation

What works for whom and under what conditions

 Calls for data on context, implementation

 Conditions that facilitate or impede success

 Guidelines for grantmakers could strengthen access to 

data for program improvement

Performance management

 Support goal setting with data-sharing requirements

Program Improvement
278 The Promise of Evidence-Based Policymaking

Reducing inequality in education

Link national education surveys to state education 

records

 In a national survey, NCES oversampled 10 states

 Each state education agency had agreed to link 

education records to the survey

 “Not a single state provided its data for linkage with 

the national survey.”

 Bureaucratic/legal hurdles could not be overcome

 Even though the state data were largely built with federal 

grants!

Policy Formation: Reducing Inequality

Need to understand the nature of challenges 

we face, and identify policies that respond

Example: Reducing inequality, the “defining 

challenge of our time”

Policy Formation: Reducing Inequality
The Promise of Evidence-Based Policymaking 279

The American Opportunity Study

Link national surveys to administrative data

View linkages as assets

Provide a framework

 Specify types of data, including administrative, 

survey, and experimental data

 Indicate purposes of linkages, including program 

improvement and policy formation

Consider a pilot

Policy Formation: Reducing Inequality

The American Opportunity Study

A standalone survey no longer needed to gauge 

social mobility

 Link census data, federal tax & employment records, 

state data from federal programs, and national surveys

 Create capacity to understand barriers and supports 

for social mobility

 Allow formulation of evidence-based policies

 Link to research samples to support program 

evaluation

Commission could recommend legislation that 

would require states to share participation data

Policy Formation: Reducing Inequality
280 The Promise of Evidence-Based Policymaking

Elements of a federal infrastructure for 

evidence

Leadership positions focused on evidence use

Interagency collaborative bodies

Dedicated office for research and evaluation 

within each agency

Codified sets of principles and practices

Build Capacity

Capacity to use data to create evidence is 

essential to evidence-based policymaking

Capacity = technology, expertise, organizational 

arrangements, & fiscal investments

 Often under-resourced

 Demand attention at the outset

Strengthen the federal infrastructure for 

evidence

Establish partnerships between researchers and 

policymakers

Build Capacity
The Promise of Evidence-Based Policymaking 281

Ambitious but achievable goals

From sharing and linking data, to using data to 

create evidence, to using evidence to inform 

policymaking

Will result in smarter policies and greater 

opportunity for all

Be the Evidence Commission

Partnerships for building and using evidence

What is a partnership?

 Sustained structure for facilitating trust and shared 

goals

 Common in education between a university and a 

local or state partner

Benefits of partnerships

 Research questions respond to needs of decisionmakers

 Culture of evidence leads decision-makers to consider 

evidence in policymaking

See: http://rpp.wtgrantfoundation.org

Build Capacity
282 The Promise of Evidence-Based Policymaking

Statement of Naomi Goldstein

Deputy Assistant Secretary for Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

to the Evidence-Based Policy-Making Commission

4 November 2016

Dr. Haskins, Dr. Abraham, and Commission members:

Thank you for inviting me to speak with you today. You have a tremendous opportunity to further the 

creation and use of evidence in the federal government and beyond. 

At the HHS Administration for Children and Families, I serve as Deputy Assistant Secretary for Planning, 

Research, and Evaluation, overseeing a portfolio of grants and contracts to conduct research and 

evaluation related to ACF programs. I’d like to tell you a bit about how we do our work, highlighting 

implications for your deliberations, and drawing on ACF’s evaluation policy, which establishes five 

principles to govern our work. These are: rigor, relevance, transparency, independence, and ethics. 

I’d like to discuss seven main points:

1. Data are necessary but not sufficient to create evidence.

2. Administrative data and ongoing surveys are important resources, but specialized data collected 

for the purposes of specific evaluations will also continue to be important. 

3. Easier accessibility of administrative data would greatly streamline evaluation activities.

4. Implementation and descriptive studies are just as important as impact or outcome evaluations, 

and relevance is just as important as rigor.

5. Prerequisites for federal evaluation include statutory authority, funding, a skilled federal 

workforce, and a robust private sector.

6. Several bureaucratic challenges pose substantial barriers to federal evaluation efforts, in the 

areas of procurement, information technology and security, and information collection.

7. The federal evaluation enterprise lacks many elements of the infrastructure that supports and 

protects the federal statistical system.

One: Data are necessary but not sufficient for evidence that can inform policy and practice. 

It is no small thing to collect data that are valid, reliable, relevant, and representative. But data per se 

are not evidence. To turn data into evidence, we must examine them using a sound analysis plan

tailored to questions of policy and programmatic interest. In addition, for many questions – especially 

questions about the impacts of policies or programs – data must be used in the context of an evaluation 

designed and put into place before the data are collected. A randomized control trial is an obvious 

example, where the study must assign participants to treatment or control groups before providing 

services and collecting outcome data.
The Promise of Evidence-Based Policymaking 283

This point is related to the principle of rigor in ACF’s evaluation policy. Rigor means getting as close as 

we can to the truth. It is not an optional gold seal. Rather, without rigor we may be generating answers 

that are just plain wrong. ACF’s evaluation policy states that we are committed to using the most 

rigorous methods that are appropriate to the evaluation questions and feasible within budget and other 

constraints. Rigor is not restricted to impact evaluations, and it is not code for randomized control trials. 

It is necessary in all types of evaluations, including implementation or process evaluations, descriptive 

studies, outcome evaluations, and formative evaluations; and in both qualitative and quantitative 

approaches. Rigor requires ensuring that inferences about cause and effect are well founded (internal 

validity); requires clarity about the populations, settings, or circumstances to which results can be 

generalized (external validity); and requires the use of measures that accurately capture the intended 

information (measurement reliability and validity). 

In assessing the effects of programs or services, ACF’s policy is to use methods that isolate to the 

greatest extent possible the impacts of the programs or services from other influences such as trends 

over time, geographic variation, or pre-existing differences between participants and non-participants. 

For such causal questions, experimental approaches are preferred. When experimental approaches are 

not feasible, high-quality quasi-experiments offer an alternative. 

Two: Administrative data and ongoing surveys are important resources, but specialized data collected 

for the purposes of specific evaluations will also continue to be necessary. 

Administrative data and data from ongoing surveys are important resources for learning about program 

effectiveness and informing program improvement. I am confident that the Commission’s 

recommendations will strengthen the value and availability of these types of data. But specialized data 

collected for the purposes of specific evaluations will also continue to be necessary, for several reasons: 

a. Some types of information may not be included in ongoing surveys or administrative data. 

For example, consider measures of young children’s socio-emotional or cognitive 

development. 

b. Administrative data may be available to measure experiences or outcomes of program 

participants, but typically are not available for comparison groups that don’t participate in 

the programs under study. 

c. Administrative data may lack the quality, completeness, and reliability needed for 

evaluation purposes.

d. Data from ongoing surveys may not capture samples adequate for specific populations or 

for addressing specific program- or policy-related questions.

e. Administrative data and ongoing surveys are sometimes available only after a considerable 

time lag.

Three: Easier accessibility and improvements in administrative data would greatly streamline evaluation 

activities.

We don’t typically use the availability of data as a starting point for deciding which questions to pursue. 

But we do take advantage of existing data sources when we can -- for example, using the National 
284 The Promise of Evidence-Based Policymaking

Directory of New Hires or state Unemployment Insurance records to measure the outcomes of an 

employment-focused intervention. Easier accessibility of administrative data would greatly streamline 

our efforts. Particularly when we need to gather data from multiple jurisdictions, the cost, time, and 

effort of developing agreements and carrying out data exchanges can be comparable to the costs of 

collecting individual survey data. 

Although well-integrated administrative data systems suitable for supporting complex evaluations can 

take years to develop and require broad-based collaboration across many agencies and levels of 

government, such systems could reduce costs and time to carry out a range of evaluations. Some

specific improvements to make administrative data more useful include adoption of common data 

semantics and incorporating data elements to better allow data linking.

Four: Implementation and descriptive studies are just as important as impact or outcome evaluations, 

and relevance is just as important as rigor.

Implementation and impact studies together are especially powerful, allowing us not only to understand 

whether a program had desired impacts, but also to explore what aspects of the program design or

implementation may have enhanced or inhibited those impacts. It is also important to understand the

characteristics of relevant populations, and the context, so we can learn which types of services work 

best for different populations or in different settings. All of these types of work provide critical 

information for scale-up or replication of evidence-based practices, and for efforts to improve the 

effectiveness of policies and programs.

Pursuing rigor without attending to relevance can mean producing work that is elegant, but useless. To 

ensure relevance, we take into account many influences in setting evaluation priorities: statutory

requirements and Congressional interests; the interests and needs of executive branch staff and 

leadership; partners such as states, territories, tribes, and local grantees; the populations served; 

researchers; and other stakeholders. ACF’s evaluation policy calls for planning evaluations in concert 

with the planning of a program or initiative, rather than as an afterthought. To make sure that 

evaluations are relevant, we aim to build strong partnerships among evaluation staff, program staff, 

policy-makers and service providers. 

Five: Prerequisites for federal evaluation include statutory authority, funding, a skilled federal 

workforce, and a robust private sector.

In order to conduct evaluation to inform policy and program decision-making, we must have at a 

minimum both statutory authority and funding. In addition, we need a skilled federal workforce, and a 

robust private sector to compete for contracts and grants to carry out the work.

We conduct evaluations in those areas where Congress has provided resources and authority to do so. 

Several large human services programs at ACF lack both authority and resources for this type of work. 

Further, most of ACF’s funding for evaluation is linked to specific programs, with little funding available 

for cross-cutting studies, although the individuals and families we serve have complex needs that do not 
The Promise of Evidence-Based Policymaking 285

map neatly onto ACF’s programmatic structure. While some agencies (such as the Department of Labor) 

have broad authority to set aside program funds for evaluation, many agencies do not.

Our evaluation policy states that ACF will recruit and maintain an evaluation workforce with training and 

experience appropriate for planning and overseeing a rigorous evaluation portfolio. To accomplish this, 

ACF will recruit staff with advanced degrees and experience in a range of relevant disciplines. ACF will 

provide professional development opportunities so that staff can keep their skills current. The federal 

evaluation and statistical workforce is populated by experts whose skills, commitment, and integrity 

impress me every day.

Private organizations, typically research firms and universities, play an essential role in carrying out 

federal research and evaluation. We rely on them in part for practical reasons. For one thing, Congress 

often appropriates funds that can be used for grants and contracts, but not for federal staff. In addition, 

evaluations require specialized staff of different types at different times. An evaluation may need a large 

complement of survey interviewers one year, and a small staff of data analysts the next year. It is more 

practical for the government to contract for these tasks than to carry them out internally. In addition to 

these practical reasons for conducting work extramurally, this practice adds an important element of 

independence, another principle of our evaluation policy. 

ACF aims to ensure that contractors and grantees conducting evaluations have appropriate expertise 

through emphasizing the capacity for rigor in requests for proposal and funding opportunity 

announcements. This emphasis entails specifying expectations in the criteria for the competitive 

selection of grantees and contractors, and engaging reviewers with evaluation expertise.

Six: Several bureaucratic challenges pose substantial barriers to federal evaluation efforts, including 

requirements related to procurement, information technology and security, and collection of 

information. 

First, some federal contracting rules are severely constricting. For example, it is difficult to gain approval 

for awarding a contract longer than five years. But many evaluation questions require more than five 

years to answer. Even more problematic are prohibitions on incremental funding for non-severable 

contracts. 

Second, there is ambiguity about the application to evaluation of several laws related to data. While 

there may or may not be clarity in concept, in practice there is often confusion on how to apply the

Privacy Act, the Federal IT Acquisition Reform Act, and the Federal Information Security Management 

Act to evaluation projects. This confusion leads to delays and expense.

Third, information collection for evaluation purposes is subject to the same requirements as any other 

federal information collection, under the Paperwork Reduction Act. Required public notice periods and 

review by the OMB Office of Information and Regulatory Affairs take a minimum of 4 months, and often

8 months, or even more. This timeline can make it impossible to evaluate grants that are funded for 

only a few years; may prohibit the collection of baseline information on program participants; and may 
286 The Promise of Evidence-Based Policymaking

preclude the timely provision of evidence to inform policy or program decision making. While OMB has 

developed streamlined mechanisms for some limited situations, for the most part the requirements do 

not differ depending on the size or scope of the information collection. They are the same for a set of 

interviews with state officials imposing a total burden of 50 hours, and a major survey interviewing 

thousands of people and imposing thousands of burden hours on respondents.

Seven: The federal evaluation enterprise lacks many elements of the government-wide infrastructure 

that supports and protects the strength and integrity of the federal statistical system.

1. As one example, data collected by designated federal statistical agencies are covered by the 

Confidential Information Protection and Statistical Efficiency Act (CIPSEA), whereas data 

collected by evaluation offices are not. 

2. Second, the federal statistical system benefits from a formal structure for exchange of

information and ideas across agencies, as well as from a statutorily mandated leadership 

function at OMB. To support evaluation and the use of evidence, OMB has created an Evidence 

Team and an Inter-Agency Council on Evaluation Policy, which I co-chair. However, these 

activities remain limited and largely informal.

3. Third, while ACF and some other agencies have established evaluation policies, there is no 

government-wide statement of principles for federal evaluation. Just last week the National 

Academies of Sciences held a workshop to comment on existing evaluation policies of federal 

agencies as well as the desirability of a cross-agency or government-wide statement of 

principles. This might follow the model of the NAS publication, Principles and Practices for 

Federal Statistical Agencies.

ACF’s evaluation policy aims to protect the transparency, independence, and objectivity of 

evaluation, and to insulate evaluation functions from undue influence and from both the 

appearance and the reality of bias. However without a national infrastructure these protections

are somewhat fragile. While some agencies – notably the Department of Education’s Institute 

for Education Sciences – have statutory protections for independence, this is not typical. ACF 

leadership established our evaluation policy, and future leadership could choose to eliminate it.

It would be a shame to lose safeguards such as requirements in our policy to:

 Make information about planned and ongoing evaluations easily accessible, publish

study plans in advance, and release comprehensive evaluation results regardless of the 

findings. 

 Conduct evaluations through the competitive award of grants and contracts to external 

experts who are free from conflicts of interest. 

 Place authority with the career director of the Office of Planning, Research and 

Evaluation to approve the design of evaluation projects and analysis plans; and to 

approve, release and disseminate evaluation reports.
The Promise of Evidence-Based Policymaking 287

Conclusion.

The mission of my agency is to foster health and well-being by providing federal leadership, partnership, 

and resources for the compassionate and effective delivery of human services. Our vision is children, 

youth, families, individuals and communities who are resilient, safe, healthy, and economically secure.

The importance of these goals demands that we continually innovate and improve, and that we evaluate 

our activities and those of our partners. This is true for other agencies as well. 

You have the opportunity to enhance this work. I particularly hope that you will aim your 

recommendations at furthering the use of data, evidence and evaluation for learning and improvement, 

rather than primarily for answering yes/no questions that are often of surprisingly little use for these 

purposes. I look forward to your report. Thank you.
288 The Promise of Evidence-Based Policymaking

Rigor 

Relevance 

Transparency 

Independence 

Ethics 

ACF Evaluation Policy

http://www.acf.hhs.gov/programs/opre/resource/acf-evaluation-policy 

Naomi Goldstein

Deputy Assistant Secretary for Planning Research & Evaluation

US DHHS Administration for Children and Families

Commission on Evidence-Based Policymaking

4 November 2016
The Promise of Evidence-Based Policymaking 289

Seven main points

1. Data are necessary but not sufficient to create evidence.

2. Administrative data and ongoing surveys are important resources, but 

specialized data collected for the purposes of specific evaluations will 

also continue to be important. 

3. Easier accessibility of administrative data would greatly streamline 

evaluation activities.

4. Implementation & descriptive studies are just as important as impact 

or outcome evaluations; relevance is just as important as rigor.

5. Prerequisites for federal evaluation include statutory authority, 

funding, a skilled federal workforce, and a robust private sector.

6. Several bureaucratic challenges pose substantial barriers to federal 

evaluation efforts, in the areas of procurement, information 

technology and security, and information collection.

7. The federal evaluation enterprise lacks many elements of the 

infrastructure that supports & protects the federal statistical system.
290 The Promise of Evidence-Based Policymaking

 Replication of Evidence-Based Programs (Tier 1) –

approx. $75 million annually

 Research & Demonstration Projects to Develop & 

Test New and Innovative Approaches (Tier 2) –

approx. $25 million annually

 TPP Program Support - approx. $7 million annually

 Evaluation of TPP Approaches - approx. $6.8 

million annually

2

OAH Teen Pregnancy Prevention Program

Policy-Driven Demand for Government 

Evaluation: Data and Capacity Needs

HHS Teen Pregnancy Prevention 

Program and Key Issues Related to 

Evaluation

Evelyn Kappeler

Director

Office of Adolescent Health

November 4, 2016 1
The Promise of Evidence-Based Policymaking 291

4

TPP Evaluation Framework FY2010-FY2014

 Conditions for grantee-led evaluations to maintain 

high level of rigor:

 Detail criteria of rigorous evaluation

 Detailed description in funding opportunity 

announcement

 Independent evaluator

 Sufficient resources

 Grant conditions and commitment to hold grantees 

accountable to stated standards

 Programmatic and evaluation technical assistance

3

TPP Evaluation Framework FY2010-FY2014

*1 non-federally funded program was 

included for a total of 6 evaluations
292 The Promise of Evidence-Based Policymaking

 Performance Measures

 Rigorous, Independent Grantee-led Evaluations (Tiers 1 & 2)

 Federal Evaluations

 Multi-site evaluation study – (1) replications, (2) significant or 

meaningful adaptations, and (3) selected core components of 

common programs

 Qualitative study and rigorous evaluations – the feasibility & impact 

of scaling up evidence-based programs and taking a holistic approach

 Implementation & impact evaluation – a commonly implemented 

but understudied teen pregnancy prevention program 

 Secondary data analysis – interventions for middle school youth 

 Meta-analysis – program/contextual elements that affect outcomes

6

Continued Investment in Data & Evaluation

5

Using Data & Evaluation to Improve Our Work

 Tier 1

 Replicating evidence-based TPP programs to scale in 

communities with the greatest need (holistic approach)

 Capacity building to support replication of evidencebased TPP programs

 Tier 2

 Rigorous evaluation of new or innovative approaches to 

prevent teen pregnancy

 Supporting and enabling early innovation to advance 

adolescent health and prevent teen pregnancy 
The Promise of Evidence-Based Policymaking 293

 Establishing & promoting uniform evidence 

standards 

 Providing evaluation training & technical assistance 

 Collecting and using data to make continuous 

quality improvements

 Disseminating all evaluation results transparently 

and expanding the evidence

8

Strategies for Building Evaluation Capacity

 Including a planning, piloting, and readiness period 

for all grantees

 Implementing programs with fidelity and quality 

through monitoring 

 Incorporating evaluation once program 

implementation has already begun

 Ensuring strong contrast between treatment control

7

Strategies for Building Evaluation Capacity
294 The Promise of Evidence-Based Policymaking

OAH TPP Program:

http://www.hhs.gov/ash/

oah/oah-initiatives/ 

tpp_program/about/

Evaluation Results: 

http://www.hhs.gov/ash/

oah/oah-initiatives/ 

tpp_program/cohorts-fy2010-2014.html/

9

Learn More About:
The Promise of Evidence-Based Policymaking 295

2

Cities are in a unique position to pursue evidence-based 

policymaking and foster innovation.

Cities are closer to problems and to the people affected – which makes them 

better able to innovate. 

• Responsible for design, delivery and execution of services

• Receive immediate feedback from residents who viscerally experience 

their engagement with local government

Cities are both the creators and consumers of evidence. 

• Provide funding and real-world data for research 

• Use evaluation to inform effective policy making that ensures 

accountability

Commission on Evidence-Based Policymaking

Washington, DC – November 4, 2016
296 The Promise of Evidence-Based Policymaking

4

NYC is already using administrative data to improve citywide 

services and outcomes for New Yorkers.

• Citywide Data Integration is the commitment to making data 

accessible for data sharing among multiple agencies. 

• Worker Connect enhances the ability of front-line staff to access client 

information held by different agencies.

• Common Metrics Initiative seeks to create uniform metrics across 

funding streams and city agencies to make relative comparisons of 

programs possible.

3

• Poverty Measure: produces an annual NYC CEO Poverty Measure

• Innovation Fund: designs and oversees ~ $85 million portfolio of ongoing early-stage 

program models

• Evaluation: manages evaluations using 8 outside firms

o Examples of past CEO evaluations that have led to large-scale change include: 

o CUNY ASAP

o Jobs Plus

o Sector-focused workforce initiatives

The Center for Economic Opportunity (CEO) helps New York City 

government reduce poverty by advancing the use of evidence and data 

in program and policy design, service delivery, and budget decisions.

Design

Data Integration

Research

Evaluation

Program 

Management

Digital Products
The Promise of Evidence-Based Policymaking 297

Contact Information 6

For more information: www.nyc.gov/ceo

@NYCOpportunity

@mattklein_

Matthew Klein

Executive Director

Center for Economic Opportunity

Mayor's Office of Operations

Email: MKlein1@cityhall.nyc.gov

5

Recommendations: Continue to build on existing, effective strategies 

at the federal level.

1. Encourage more federal funding to be allocated to evidence-based strategies and to grantees 

who are committed to building evidence through programs. 

2. Continue developing federal Common Metrics to ensure everyone can make relative 

assessments and direct funds to the most effective programming to make funding allocations 

more impactful.

3. Amend federal laws to give local governments more freedom to share client data among its 

own divisions and employees. 

a) Amend HIPAA regulations to create a new legal exception that clearly authorizes city 

agencies to share information with each other, without requiring client consent, when 

providing or coordinating benefits, services, and care.

b) Clarify additional ways consistent with the Family Educational Rights and Privacy Act that 

cities can engage in outreach and support to vulnerable students.

c) Update privacy related laws and regulations to accommodate the application of 

technology used to support data integration. 
298 The Promise of Evidence-Based Policymaking

DOL Mission

• “...foster, promote, and develop the welfare of the wage earners, job seekers, and 

retirees of the United States; improve working conditions; advance opportunities for 

profitable employment; and assure work-related benefits and rights”

▫ Over a dozen operating agencies, each with subagencies, many programs, and field offices

 Worker protection and labor standards 

 Employment services, job training, and worker security 

 Policy and advocacy

 Many have research, analysis, and/or evaluation offices 

• CEO’s Departmental-level evaluation support-

▫ Complements not duplicates agency evaluation functions

▫ Raises the quality of evaluations and awareness and knowledge of evaluation methodology 

and standards

▫ Improves use and dissemination of evaluation results 

▫ Improves access to, quality of, and use of data, including the CEO Data Analytics Unit 

(coordinate with BLS, Performance Management Center, and agencies)

2

Evidence-based Policy: A 

Federal Evaluation 

Office’s Perspective

Demetra Smith Nightingale

Chief Evaluation Officer

U.S. Department of Labor

Presentation to the

Evidence-based Policy Commission

November 4, 2016
The Promise of Evidence-Based Policymaking 299

Main Types of Evaluations

• Formal evaluations

▫ Experimental design—random assignment treatment and control groups (especially formal 

net impact evaluations, rapid cycle behavioral tests)

▫ Quasi-experimental designs—created comparison groups (statistical matching techniques)

▫ Various analytic levels: nation, states, localities, grantees; programs, demonstrations, 

strategies, models

• Outcome evaluations

▫ Nonexperimental statistical/econometric modeling

▫ Survey analysis

▫ Statistical analysis of BLS and Census data

• Program performance analysis

▫ Quantitative output and outcome analysis

• Implementation and management evaluations

▫ Organizational and program process analysis

4

Evaluations 

& Analytic 

Projects

Agency 

Learning 

Agendas

Strategic Plan 

& Priorities

Congressional 

Requirements

OMB 

Guidance

Capacity Development

Dissemination (website, briefs, 

SnapShots)

CLEAR (standards and reviews)

Scholars Programs & Grants

Data Quality and Access

Seminars

Cross-agency Evidence Groups

Data Analytics

Consultative Analytics & Projects

DEAP-Data Exchange Platform

Public Use Files

User Groups & Seminars

Software

Statistical Modeling

Collaborative Analysis

Data Visualization

3
300 The Promise of Evidence-Based Policymaking

Priority data systems issues for 

evaluations

• Analysts’ access to the physical data infrastructure (e.g., by thirdparty researchers, federal staff analysts)

• Expert human capital (e.g., data and programming expertise)

• Timeliness of access

• Security (e.g., secure PII, informed consent usage)

• Cost efficient

6

Priority Data Issues for Evaluations

• Appropriate outcome variables

▫ (e.g., individual workers, program participants, firms, establishments, states)

• Appropriate independent variables, covariates

▫ (e.g., labor market conditions, demographic and household characteristics, work 

and earnings history, education, criminal record, occupation/industry, 

compliance history)

• Time frame aligned to evaluation goals—pre and post periods

• Micro-level data

• Longitudinal features

• Agile merging

▫ Longitudinal analysis files

▫ Using unique identifiers (e.g., individuals, firms, establishments, states, 

programs, grantees)

▫ Linking other evaluation data collected (e.g., surveys, program data)

5
The Promise of Evidence-Based Policymaking 301

For more information...

• Demetra Nightingale 

Nightingale.Demetr@dol.gov

• CEO website (including DOL Evaluation Policy Statement)

https://www.dol.gov/asp/evaluation/EvaluationPolicy.htm

• CLEAR (DOL’s evidence-based clearinghouse)

http://clear.dol.gov/

8

Priority interests (“wish list”) to improve 

data for DOL evaluations

• Earnings data. Direct, and less costly, access to earnings records produced by DOL’s 

state employment security agency partners—for evaluation, statistical products, and 

program performance measurement

• National Directory of New Hires

• Longitudinal Employer and Household Dynamics

• Firm identifiers. Common firm, employer, and establishment identification scheme

• Reformed PRA. Streamlined and less costly Paperwork Reduction Act (PRA) process 

for evaluations and evidence-building

• Streamlined IAA. Less cumbersome interagency agreement (IAA) process to 

facilitate and encourage cross-agency data sharing and matching at the Federal level

• Privacy & Security. More clarity and consistency in procedures and rules among 

Federal agencies and other data providers

7
302 The Promise of Evidence-Based Policymaking

1

Testimony to the Commission on Evidence-Based Policymaking

“Policy-Driven Demand for Government Evaluation: Data and Capacity”

Submitted by: Katherine O’Regan, Assistant Secretary for Policy Development & Research, Department 

of Housing and Urban Development

November 4, 2016

A. Importance of the Commission’s work

Good afternoon and thank you Commissioners for the invitation to join you today, to talk about data and 

capacity issues in government evaluation. Collecting, analyzing and using data and research to inform 

HUD’s programs and policies is at the core function of my office, the Office of Policy Development and 

Research. Like many other Federal agencies, HUD has been focused on learning more from the data we 

collect, and the research we conduct.

We believe that to make the most of government data and fully leverage how these data can inform 

policy, we need a leap forward:

 by making our administrative data much more widely available while protecting privacy;

 by matching them to other data – across government agencies, across levels of government, and nongovernmental data, which is particularly key to expanding cross-domain knowledge; and

 By ensuring the capacity of governmental agencies to leverage linked government data to conduct inhouse research.

We have access to our own administrative data but have both internal and external gaps.

 Internal needs include the staff time and capacity to do the research internally - with better 

infrastructure to support them, to ensure data privacy protections, and to more easily permit linking to 

data external to the agency.

 External needs include the ability to get outside researchers’ access to our data and, similarly, the 

infrastructure to support them, ensure data security, and enable linking. 

The Evidence Commission’s charge, and recommendations, could create that leap forward, greatly 

improving the use of Federal data in building evidence for policy. I will start by providing some 

background on an area we have focused considerable effort.

B. Cross-Agency Data Linking

HUD has initiated several cross-agency data linking efforts to better understand non-housing domains of 

HUD assisted households, and to support evaluation work. I will focus on two efforts in particular, and 

some lessons learned.

1. Department of Education, Free Application for Federal Student Aid (FAFSA).

This year, HUD and the Department of Education (ED) signed an MOU that allows HUD to provide ED 

with data on assisted tenants for matching to Federal Student Aid data. 

Benefits

We are using the used the linked data to learn about application for and receipt of Federal Student Aid 

among tenants, and to conduct some rapid-cycle experiments, aimed at increasing FAFSA take up and 

college enrollment.

 For the first time, HUD has national data on one measure of college attendance for those we serve.

 We are able to conduct rapid-cycle, in-house experiments.
The Promise of Evidence-Based Policymaking 303

2

 We are also able to provide aggregated data to Public Housing Agencies (PHAs) on the FAFSA take 

up for their assisted households, which some will use to evaluate their programmatic efforts – for the 

first time.

Lessons 

 Those linked-data, however, do not sit in a secured research center with access for HUD or ED staff.

 Lack of infrastructure inhibits easy in-house work, or for getting more out of the matches.

2. HUD-National Center for Health Statistics (NCHS) linked data.

The second example is work with the National Center for Health Statistics, linking HUD longitudinal 

administrative data on assisted households to 14 years of cross-sectional health survey data from two 

national surveys (National Health Interview Survey (NHIS); National Health and Nutrition Examination 

Survey (NHANES)).

Benefits

The linked data files enable researchers to examine the relationship between assisted housing and factors 

that influence health status, chronic disease, health care utilization, morbidity, and mortality.

Benchmarking data and research have already been produced. 

Lessons

 The existence of the CDC research center, with appropriate infrastructure for linking and for 

handling privacy was fundamental to the work.

 The MOU process allowed both agencies to conduct the linkage cost-free, and the interagency MOU 

underwrote free RDC access for HUD researchers, a significant cost barrier reduction.

o Sustainable financial models for data sharing need to be built in to such work.

 Legal barriers due to data privacy concerns were very challenging.

o Establishing the MOU between agencies took many years to negotiate.

o This project moved forward due solely to the prolonged commitment of a handful of staff.

o As we continue down this path, a more centralized approach is much more efficient.

Across both experiences, these one-off, time consuming interagency matches are not the most efficient 

way to exploit data and are difficult to sustain. This has led to our most recent effort.

C. HUD MOUs with the Census Bureau

HUD has entered into two agreements with Census to greatly broaden access to HUD data and matching 

with non-HUD data for approved internal and external researchers, within Census’s Controlled Access 

Infrastructure.1

1. The first is an Inter-Agency Agreement (IAA) with the Center for Administrative Records Research and 

Applications (CARRA) to link data from HUD’s tenant databases and select randomized control trials 

with the Census’ survey data collection and other administrative data. 

 HUD’s tenant databases are already maintained within CARRA, which permit outside researchers 

to gain access via a secured Federal Statistical Research Data Center.

 1 Census has the infrastructure to support IT security requirements for research with PII and other highly sensitive data. As part 

of the infrastructure support, Census provides disclosure reviews and research assistance to authorized researchers with approved 

research projects.
304 The Promise of Evidence-Based Policymaking

3

 Via this IAA, HUD has sent the Moving to Opportunity (MTO) data to CARRA (the HUD 

voucher demonstration that Raj Chetty and co-authors linked to IRS data); 

o In FY2017, we plan to send the Family Options Study (FOS) data to CARRA. 

 To jumpstart research projects, HUD will issue a solicitation for research using these data linked 

with other administrative data – all accessed through the Federal Statistical Research Centers that 

are the portal to CARRA. 

Benefits

As you all know, and will be learning more about at upcoming convenings, OMB directed the Census

Bureau to use appropriated funds for these purposes, and to administer the Evidence-Based Policymaking 

Commission.

2

 Our agreement with Census is the first Federal effort (a ‘pilot’) in a larger expansion of Census’s data 

linkage infrastructure to support evidence-building and program evaluation. 

 Our IAA and data sharing with Census will greatly increase external researchers access to HUD 

administrative and research data, within a secured infrastructure.

Lessons 

While this pilot has just started, and we believe it holds the promise for how best to proceed with cross 

agency data matching, and making the most of archived research studies, we also believe:

 A sustainable cost structure will need to be developed for a broad group of Federal agencies to 

use this model. 

 For context, the current pilot costs $50,000 per evaluation data set, per year, with potentially 

additional costs per researcher. 

o The longer-term costs need a model that does not consume agencies’ current research 

budget.

2. HUD and Census have also signed a Joint Statistical Project Agreement (JSPA) that will commit HUD 

and Census to partner on linking housing data to non-housing data sources already acquired by Census, to 

be available for internal researchers at each agency. 

Benefits

 HUD staff working on these projects are able to access the data linked via CARRA remotely from a 

secured computing environment within PD&R/HUD, for much faster policy/evaluation work.

Lessons

 Providing Federal staff with access to linked administrative data, those with program knowledge and 

the ability to move evidence much more quickly into policy is a critical piece of driving evidencebased policy.

An additional lesson on the benefits of linked data in CARRA has already occurred on the interplay 

between data availability and what we study. PD&R is conducting our research-agenda setting process, 

one that engages with stakeholders internally at HUD, and much more broadly. As we winnow down the 

more than 500 submissions, some ‘big questions’ that might not have been asked - or small ones that 

might not have been prioritized – will be included in our Strategic Plan because, through data matching 

with the Census, we considered the research feasible. The existence of a way to evaluate/study something 

does affect whether we prioritize it in our research agenda.

 2 Evidence-Based Policymaking Commission Act of 2016
The Promise of Evidence-Based Policymaking 305

4

As we move towards establishing a clearinghouse for Federal data, such as CARRA, there are ways to 

make the most of those data. 

D. Making the most of data within CARRA

1. Institutional Review Board guidance to permit indefinite data matching at CARRA

The recent work from Raj Chetty has brought attention to the benefits of data matching, and specifically 

to HUD’s Moving to Work or MTO demonstration of the 1990s. In that study, HUD received consent 

from 4,610 families to conduct data matching against administrative records.

 The consent was not time restricted and it was general about the types of administrative data for the 

matching.

 This has allowed HUD to track the impact on the heads of households and their children for nearly 

two decades. It permitted the long term impact work of Raj Chetty. And it is allowing the MTO data 

to be made available at CARRA to measure impacts over even more years.

HUD’s Family Options study, a randomized study to measure the benefits of receiving long-term versus

short-term housing assistance for families with children who are homeless, also has research questions 

that extend out 20 years or longer. However, the IRB for this study limited the consent to 5 years.

 Although HUD will be providing these data to CARRA, the limited consent period applies. 

Lesson

 It is reasonable to limit the time frame for consent for generic administrative matching.

 But if research data are eventually protected by the Census Bureau Title XIII, as part of the Federal 

clearinghouse, we are proposing that IRBs be given model language to allow indefinite consent under 

those conditions.

 When impacts take 20 or 30 years to materialize, we need to be sure that we will be able to measure 

them.

2. State data.

Demonstrations and evaluations take place in a limited number of locations, and non-Federal

administrative data is frequency crucial to many studies. There are many obstacles to accessing –and 

linking- those data too.

 In the long term evaluation of the Moving to Opportunity demonstration, researchers sought 

administrative data from multiple states on wages from unemployment insurance programs and on 

various benefits (TANF and SNAP). 

 Data access varied considerably from state to state, with some states providing only aggregate data 

which limited HUD’s ability to look at program impacts for specific subgroups. For one state, we are 

still working on securing needed data.

 Note, some of the need to link state data is itself occurring after failure to get agreement on 

linking across Federal agencies for (in this case, Medicaid data).

Lessons

Consider the role of Non-Federal administrative data in your review

 Census has initiated a pilot with Chapin Hall, bringing in a limited set of state and local data into 

CARRA, as a proof of concept. This could be expanded.

 Any data inventory the Commission creates should include non-Federal data.
306 The Promise of Evidence-Based Policymaking

5

E. Additional Issues for the Commission’s Consideration

1. Gathering data and the Paperwork Reduction Act

The Paperwork Reduction Act (PRA) requires that for any data collection involving 9 or more individuals 

or organizations, the data collection needs to go through a formal process of review that involves both a 

60-day public notice in the Federal Register seeking comment to the agency and an additional 30-day 

notice for the public to provide comment to OMB. HUD’s experience is that clearances take between 120 

to 180 days, so for most questions, it is 4 to 8 months before data collection can begin.

PRA is an effective tool for reducing burden by preventing unnecessary data collection from the 

public. But the length of time to complete the process may also be inhibiting useful data collections that 

could improve policies and programs, and those sub-optimal policies may impose also impose severe 

burden on the public.

This is particularly relevant for HUD, which does not operate its programs directly. Instead it provides 

funding to local organizations to implement its programs. These organizations include Public Housing 

Agencies (3,400), state and local governments (1,200), lenders (several hundred), multifamily-owners of 

assisted properties (20,000), and non-profit homeless providers (10,000 or so). 

Sometimes it is critical to get new information from these organizations to address an un-foreseen 

issue. PRA prevents collecting this information in a timely manner beyond nine of these 

organizations. If these programs were operated by Federal government employees directly we would not 

be prevented from collecting this information. As a result, it is not uncommon that HUD makes policy 

decisions and program changes with very little information; often reacting to anecdote rather than a more 

complete picture.

Lessons

 More robust incorporation of data and evidence in Federal policy requires improvements in PRA.

There are various ways this could be done.

As one example: allow 30-day agency reviews without OMB engagement on collections below some

thresholds, such as 1,000 responses and under 500 burden hours. This would take a 90 to 180-day

process for these collections and turn them into 30 day processes.

There are numerous other suggestions that this commission would likely hear across Federal agencies, 

along a similar vein. 

2. Address weakness of Privacy Act Protections

In a recent Freedom of Information Act (FOIA) case3, a Federal district court ordered HUD to release to 

the requester the individual records of tens of thousands of voucher-assisted tenants, omitting the most 

obvious identifying variables (name, SSN, address) but including zip code, census tract, age of all 

household members, race of head, and a wide variety of other data that in HUD’s view could be used to 

identify households receiving HUD assistance. 

 3 “The Inclusive Communities Project, Inc. v. The United States Department of Housing and Urban Development,” Civil Action 

No. 3:14-CV-3333-B
The Promise of Evidence-Based Policymaking 307

6

This was possible because the HUD administrative records, and records of many other agencies, are 

protected from disclosure only under the Privacy Act. 

 Under FOIA, when records have only Privacy Act protection, the onus is on the government to prove 

the probability of harmful disclosure. In this particular case, the court felt HUD had not met the 

burden of proof, and presumably other courts might come to the same conclusion.

 The individuals whose lives are captured in the administrative and survey data that HUD uses for 

evaluation and analysis are therefore vulnerable in a way that they would not be if these data had the 

statutory protections that Census data, income tax data, or Medicare data have. 

 HUD is concerned that the quality of administrative data may decline, if some individuals will 

eventually be less inclined to give honest answers to the questions we ask, or may decline to 

cooperate at all, if they may subsequently become subject to public exposure. 

Lesson

 The Commission should consider recommending strengthening in statutory protection for 

administrative and survey data collected for agencies like HUD.

F. Conclusion

We appreciate the chance to provide these comments to the Commission, and the forthcoming 

conversations on data and capacity issues in support of government evaluation and policy needs. 
308 The Promise of Evidence-Based Policymaking

Cross-Agency Data Linking:

Free Application for Federal Student Aid (FASFA)

• HUD provides ED assisted tenant data for matching to Federal Student Aid data. 

• Benefits

• HUD now has national data on one measure of college attendance.

• Able to conduct rapid-cycle, in-house experiments.

• Provide aggregated data to PHAs on take up for assisted households.

• Lessons

• Linked data not located in secured research center with access for HUD or 

ED staff.

• Lack of infrastructure for easy in-house work, or for getting more out of the 

matches.

2

Commission on Evidence-Based Policymaking

Katherine O’Regan

Assistant Secretary 

Office of Policy Development and Research

Department of Housing and Urban Development

November 4, 2016

Policy-Driven Demand for Government 

Evaluation: Data and Capacity 
The Promise of Evidence-Based Policymaking 309

Census MOUs:

IAA with CARRA

• Center for Administrative Records Research and Applications (CARRA) IAA to 

link data from HUD’s tenant databases and select trials with Census’ survey data 

and other administrative data. 

• Benefits

• OMB directed Census to use appropriated funds for these purposes and to 

administer the Evidence-Based Policymaking Commission.

• Pilot to support evidence-building and program evaluation.

• Will greatly increase external researchers’ access to HUD administrative and 

research data, within a secured infrastructure.

• Lessons

• A sustainable cost structure will need to be developed for broad adoption. 

• Longer-term costs need a model that does not consume agencies’ 

current research budget. 

4

Cross-Agency Data Linking:

HUD & National Center for Health Statistics 

• 14 years of HUD data (1996-2014) linked to cross-sectional health survey data.

• Benefits

• Enables researchers to examine the relationship between assisted housing 

and factors that influence health status, chronic disease, health care 

utilization, morbidity, and mortality.

• Lessons

• CDC research center (linking, securing privacy) is fundamental to supporting 

work.

• MOU included cost-free linkage, free RDC access for HUD researchers.

• Legal barriers due to data privacy concerns were very challenging; MOU 

took years to negotiate.

• Across both projects: time-consuming, one-off MOUs not the best way forward.

3
310 The Promise of Evidence-Based Policymaking

Making the Most of Data within CARRA

1. Institutional Review Board guidance to permit indefinite data matching at CARRA. 

• MTO data: consent not time restricted, general administrative matching.

• Family Options Data: 5 year consent.

• Lessons

• It is reasonable to limit the time frame for consent for generic administrative 

matching.

• But if the responses are protected by the Census Bureau Title XIII, we are 

proposing that IRBs be given model language to allow indefinite consent. 

• When impacts take 20 or 30 years to materialize, we need to be sure 

that we will be able to measure them.

6

Census MOUs:

Joint Statistical Project Agreement

• Joint Statistical Project Agreement (JSPA) commits HUD and Census to partner 

on linking housing data to non-housing data sources, available for internal 

researchers at each agency. 

• Benefits

• HUD staff can access linked data remotely from a secured computing 

environment within PD&R/HUD, for much faster policy/evaluation work.

• Lessons

• Providing federal research staff access to broad administrative data is a 

critical piece of driving evidence-based policy.

• Additional Lesson

• The interplay between data availability and what we study.

• CARRA linking is already affecting our research agenda.

5
The Promise of Evidence-Based Policymaking 311

Additional Issues to Consider

1. Gathering data and the Paperwork Reduction Act.

• More robust incorporation of data and evidence in Federal policy requires 

improvements in PRA. 

• Particularly important for HUD – which does not operate its programs 

directly (Public Housing Agencies = 3,400); state and local 

governments (1,200); multifamily-owners of assisted properties 

(20,000); etc. 

• There are various ways this could be done.

2. Weakness of Privacy Act Protections.

• Administrative data on which much of these efforts will be built has fairly 

weak privacy protections under the Privacy Act.

• The Commission could consider recommending strengthening statutory 

protection for administrative and survey data collected for agencies like 

HUD.

8

Making the Most of Data within CARRA

2. State Data

• Non-Federal administrative data is crucial to many studies. State data access 

varies widely. 

• Lessons

• Consider the role of non-Federal administrative data

• Chapin Hall pilot, whereby state and local data are being added to 

CARRA, should be expanded.

• Any data inventory the Commission creates should include nonFederal data.

7
312 The Promise of Evidence-Based Policymaking

REDUCING POVERTY AND IMPROVING LIVES 

THROUGH EVIDENCE-BASED PROGRAMS AND POLICIES 

 

 

 $200 BILLION PER YEAR SPENT BY PRIVATE SOCIAL SERVICE PROVIDERS 

 PROGRAMS TYPICALLY MONITOR OUTPUTS; RARELY MEASURE IMPACT 

 EVIDENCE OF PROGRAM IMPACT IN THIS SECTOR IS NECESSARY FOR 

PROMOTING EVIDENCE-BASED POLICIES AND PROGRAMS NATIONALLY 

 FUNDING IS NOT ALIGNED WITH MEASURING IMPACT 

Most private and government funding for private providers does not require impact 

measurement, and consequently limited resources are available to build a body of evidence 

of what works. 

 ACCESS TO DATA AND ANALYSIS IS LIMITED 

Social service providers collect the necessary data for eligibility and for reporting, but lack 

access to administrative data to measure outcomes and the bandwidth to analyze it. 

 NOT CONNECTED TO EVIDENCE OF WHAT WORKS

Providers looking to start a new, evidence-based program often have a difficult time 

finding information on validated programs and how to implement them. 

 INCENTIVIZE IMPACT EVALUATION 

Funding for private service provider programs should require rigorous evaluation AND 

include appropriate funding to pay for it so providers can measure impact, but not cut 

into scarce program budget resources to do so. 

 PROVIDE ACCESS TO ADMINISTRATIVE DATA 

Providers need access to the administrative data sets that include key outcomes such as 

earnings records, government program participation (e.g. TANF, SNAP, SSDI), 

hospitalizations and health care utilization, arrest records and education records. 

 CREATE A WHAT WORKS CLEARINGHOUSE 

Make the best evidence available and easily accessible so that impactful programs are 

widely known and can be replicated throughout the private service provider sector. 

CHALLENGES (FOR PROVIDERS) 

RECOMMENDATIONS (FOR COMMISSION) 

NON-PROFIT SERVICE PROVIDERS 

AND EVIDENCE-BASED POLICY 

NON-PROFIT SERVICE PROVIDERS 
The Promise of Evidence-Based Policymaking 313

EXAMPLE 1: HOMELESSNESS PREVENTION

PROGRAM: The Homelessness Prevention Call Center (HPCC) in Chicago is one of the largest 

call centers in the nation, taking approximately 70,000 calls each year. The HPCC connects 

those at risk of homelessness with emergency financial assistance, but the availability of 

funding varies unpredictably on a day-to-day basis. 

STUDY AND DATA: LEO’S study examines the impact of financial assistance for 4,500 

individuals and families who called the HPCC between 2010 and 2012. In order to observe 

shelter entry for these callers, we linked the call center information to administrative data on 

entries into and exits from homeless shelters in Chicago. 

RESULTS: Emergency financial assistance prevents homelessness. 

 Assistance reduced shelter entry 6 months after the call by 76 percent; effect persists for 

more than a year. 

 Impact of financial assistance is largest for those with especially low income. 

 

EXAMPLE 2: JUVENILE DIVERSION 

PROGRAM: Reading for Life (RFL) is a diversion program in St. Joseph County, Indiana 

designed for non-violent juvenile offenders. A unique and innovative alternative to prosecution 

in the court system, RFL allows participants to study works of literature in small groups led by 

trained volunteer mentors. 

STUDY AND DATA: LEO’s study examines the impact of RFL for 400 first time juvenile 

offenders randomly placed into RFL or into community service diversion between 2010 and 

2014. In order to measure recidivism, we linked RFL enrollment data with administrative data 

collected by the Juvenile Justice Center and adult arrest records collected by the State. 

RESULTS: The Reading for Life diversion program reduces two-year recidivism rates for 

prosecuted felonies by 50%. 

 Participants were 36% less likely to be arrested 

 Participants were 68% less likely to be prosecuted for a felony 

The Wilson Sheehan Lab for Economic Opportunities (LEO) is a research lab housed in 

the Department of Economics at the University of Notre Dame. LEO matches top researchers 

with social service providers to conduct impact evaluations that identify the innovative, effective, 

and scalable programs and policies that support self-sufficiency. LEO's research is conducted by 

Notre Dame faculty as well as an interdisciplinary network of scholars from across 

the country with expertise in designing and evaluating the impact of domestic programs aimed 

at reducing poverty and improving lives. LEO disseminates its key findings to policymakers and 

front-line providers in order to support evidence-based policy and programming decisions that 

effectively and jointly reduce poverty in the United States. 

Learn more at www.leo.nd.edu

EXAMPLES OF LEO’S WORK WITH NON-PROFIT SERVICE 

PROVIDERS USING ADMINISTRATIVE DATA 
314 The Promise of Evidence-Based Policymaking

INFORMATION NOT RELEASABLE TO THE PUBLIC UNLESS AUTHORIZED BY LAW:

This information has not been publicly disclosed and may be privileged and confidential. It is for internal government use only and must not be disseminated, 

distributed, or copied to persons not authorized to receive the information. Unauthorized disclosure may result in prosecution to the full extent of the law.

Introduction to CMS

2

Over 140 million Americans receive healthcare 

coverage through programs administered by CMS

Medicare Medicaid/CHIP Health Insurance 

Marketplaces

Health insurance for 

individuals age 65 

and older, as well as 

those with 

disabilities

Health insurance 

managed by the 

states for individuals 

with lower incomes

A resource that 

allows individuals to 

sign-up for private 

health insurance 

with tax credits to 

offset premiums

INFORMATION NOT RELEASABLE TO THE PUBLIC UNLESS AUTHORIZED BY LAW:

This information has not been publicly disclosed and may be privileged and confidential. It is for internal government use only and must not be disseminated, 

distributed, or copied to persons not authorized to receive the information. Unauthorized disclosure may result in prosecution to the full extent of the law.

Commission on Evidence-Based 

Policymaking

Niall Brennan

CMS Chief Data Officer 

Director, Office of Enterprise Data & Analytics

@N_Brennan

December 12, 2016 Meeting
The Promise of Evidence-Based Policymaking 315

INFORMATION NOT RELEASABLE TO THE PUBLIC UNLESS AUTHORIZED BY LAW:

This information has not been publicly disclosed and may be privileged and confidential. It is for internal government use only and must not be disseminated, 

distributed, or copied to persons not authorized to receive the information. Unauthorized disclosure may result in prosecution to the full extent of the law.

Releasing Machine Readable Data – Provider Payment 

and Utilization Files

4

May 2013

● Public data sets with payment and utilization information for 

services and procedures provided to Medicare beneficiaries

● Data released to date covers > 85% of Medicare program payments

*Updated to include demographic and health information associated with the 

provider’s beneficiary panel in October 2015

 3,000 

Hospitals

 150,000 

records

 $62B in 

payments 

May 2013

Hospital 

Inpatient & 

Outpatient

 880,000 

NPIs

 9M 

records

 $90B in 

payments

 Over 1M 

NPIs

 23M 

records

 $103B in 

drug costs

 385,000 

NPIs

 1.9M 

records

 $11B in 

payments

 11,000 

HHAs

 100,000 

records

 $18B in 

payments

Physician 

& Other 

Supplier*

Part D 

Prescriber

DME and 

POS

Home 

Health 

Agency

Skilled 

Nursing 

Facility

 15,000 

SNFs

 80,000 

records

 $28B in 

payments

April 2014 April 2015 Oct 2015 Dec 2015 Mar 2016 Oct 2016

Hospice

 4,000 

Hospices

 4,000 

records

 $15B in 

payments

INFORMATION NOT RELEASABLE TO THE PUBLIC UNLESS AUTHORIZED BY LAW:

This information has not been publicly disclosed and may be privileged and confidential. It is for internal government use only and must not be disseminated, 

distributed, or copied to persons not authorized to receive the information. Unauthorized disclosure may result in prosecution to the full extent of the law.

CMS makes lots of data 

available to download for 

free

Finding CMS Publicly Available Data

3

Dnav.CMS.gov
316 The Promise of Evidence-Based Policymaking

INFORMATION NOT RELEASABLE TO THE PUBLIC UNLESS AUTHORIZED BY LAW:

This information has not been publicly disclosed and may be privileged and confidential. It is for internal government use only and must not be disseminated, 

distributed, or copied to persons not authorized to receive the information. Unauthorized disclosure may result in prosecution to the full extent of the law.

Part D Prescribers in Albuquerque (ZIP-code 87109)

6

These 8 providers all have at least 1,000 total drug claims 

and a generic drug dispensing rate (GDR) < 50%.

Albuquerque, NM: ZIP 87109

545 Providers, Average GDR = 71.5%

Provider Name Provider Specialty

Drug 

Claims

Generic 

Drug Claims

Generic Drug 

Dispensing Rate

A. Weinstein Ophthalmology 4,834 594 12.3%

J. Dorf Pulmonary Disease 2,963 515 17.4%

T. Watterberg Ophthalmology 2,123 433 20.4%

V. Valentine Clinical Nurse 

Specialist 1,078 348 32.3%

R. Allen Ophthalmology 1,713 619 36.1%

J. Krawchuck Nurse Practitioner 1,371 604 44.1%

B. Monson Ophthalmology 1,096 521 47.5%

R. Patel-Trujillo Internal Medicine 1,757 877 49.9%

The most prescribed brand 

name drug by this provider 

is Vigamox, which 

accounted for 34.2% of his 

total claims ($188,354)

INFORMATION NOT RELEASABLE TO THE PUBLIC UNLESS AUTHORIZED BY LAW:

This information has not been publicly disclosed and may be privileged and confidential. It is for internal government use only and must not be disseminated, 

distributed, or copied to persons not authorized to receive the information. Unauthorized disclosure may result in prosecution to the full extent of the law.

Medicare Part D Generic Dispensing Rate

5

HRR Level

ZIP-code 87109 

(Albuquerque) 

= 71.5%

The national average for dispensing 

generic drugs is 76.0%. NM’s state 

average (77.2%) is slightly higher.

Across NM ZIP-codes, generic dispensing 

rates range from 0% to 100%. One-third of 

NM ZIP-codes have rates below the state 

average.
The Promise of Evidence-Based Policymaking 317

INFORMATION NOT RELEASABLE TO THE PUBLIC UNLESS AUTHORIZED BY LAW:

This information has not been publicly disclosed and may be privileged and confidential. It is for internal government use only and must not be disseminated, 

distributed, or copied to persons not authorized to receive the information. Unauthorized disclosure may result in prosecution to the full extent of the law.

● Research Data Assistance Center (ResDAC) provides 

assistance to researchers interested in using Medicare and/or 

Medicaid data

● The Chronic Condition Warehouse (CCW) is CMS’ research 

data warehouse designed to support external researchers and 

internal CMS research and analytic functions 

 Contains over 315B records with 1B records added monthly

 Unique beneficiary ID allows data linkages across all CCW data

● New innovator research data access expands current data 

release policy to allow innovators to access CMS data to create 

products they intend to sell

Enabling Cutting Edge Health Care Research

8

INFORMATION NOT RELEASABLE TO THE PUBLIC UNLESS AUTHORIZED BY LAW:

This information has not been publicly disclosed and may be privileged and confidential. It is for internal government use only and must not be disseminated, 

distributed, or copied to persons not authorized to receive the information. Unauthorized disclosure may result in prosecution to the full extent of the law.

Building Consumer Friendly Interfaces

7
318 The Promise of Evidence-Based Policymaking

INFORMATION NOT RELEASABLE TO THE PUBLIC UNLESS AUTHORIZED BY LAW:

This information has not been publicly disclosed and may be privileged and confidential. It is for internal government use only and must not be disseminated, 

distributed, or copied to persons not authorized to receive the information. Unauthorized disclosure may result in prosecution to the full extent of the law.

● CMS makes two types of files available to researchers

 Limited Data Set (LDS) files which excludes specific direct identifiers, including 

name, address, HIC, SSN, DOB, ZIP Code and medical record number

 Research Identifiable Files (RIFs) which are custom CMS data extracts 

that may contain direct beneficiary identifiers

● LDS files are easier to request (less documentation and CMS review) 

but users face additional limitations on use of the data

Types of Research Data Files

10

Research 

Identifiable

Limited 

Data Sets

Requires CMS Privacy Board Review? Yes No

Data file can be customized to only include a 

specific cohort (e.g., diabetics residing in MN) Yes No

Data can be linked to non-CMS data using a 

beneficiary identifier (SSN or Medicare id) Yes No

INFORMATION NOT RELEASABLE TO THE PUBLIC UNLESS AUTHORIZED BY LAW:

This information has not been publicly disclosed and may be privileged and confidential. It is for internal government use only and must not be disseminated, 

distributed, or copied to persons not authorized to receive the information. Unauthorized disclosure may result in prosecution to the full extent of the law.

● Medicare enrollment and claims (1999-current)

● Medicare Part D event data (2006-current)

● Medicaid eligibility and claims (1999-2013)

● Assessment data (instrument inception-current): 

 Long Term Care Minimum Data Set (MDS)

 Home Health Outcome and Assessment Information Set (OASIS) 

 Inpatient Rehab Facility – Patient Assessment Instrument (IRF-PAI)

● Shared Savings Program and Pioneer ACO files (2013-current)

● Medicare Data on Provider Practice and Specialty (2008-current)

● Administrative data linked to surveys (e.g., Health and Retirement 

Survey, National Health and Aging Trends Study)

Available Research Data Files

9
The Promise of Evidence-Based Policymaking 319

INFORMATION NOT RELEASABLE TO THE PUBLIC UNLESS AUTHORIZED BY LAW:

This information has not been publicly disclosed and may be privileged and confidential. It is for internal government use only and must not be disseminated, 

distributed, or copied to persons not authorized to receive the information. Unauthorized disclosure may result in prosecution to the full extent of the law.

● Researchers have been receiving CMS data for decades

 Requests have grown exponentially over the past couple years

 Researchers are requesting more timely and less expensive data

● CMS developed the VRDC to meet researchers evolving needs

● VRDC is a secure and efficient means for researchers to virtually 

access and analyze the vast store of CMS data in the CCW

Virtual Research Data Center (VRDC)

11

● VRDC benefits

 Lower cost, more timely data

 Researchers use own laptop to 

securely access and analyze data 

remotely 

 Increased security for sensitive data

 Additional privacy protections – no 

beneficiary identifiable data may leave 

the secure environment
320 The Promise of Evidence-Based Policymaking

2 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Agenda

 QCEW general background

 Data acquisition

 Quality and curation

 Linking

 Uses and analysis

 Sustainable, scalable, matchable

 Archiving

1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

BLS Federal-state cooperative 

program:

Quarterly Census of 

Employment and Wages

Erica L. Groshen

Commissioner

Bureau of Labor Statistics

Commission on Evidence-Based Policymaking

December 12, 2016
The Promise of Evidence-Based Policymaking 321

4 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Data acquisition

 Fed-state cooperative program

BLS provides funding, training, IT system, quality 

standards, processing, analysis, enhancements

– $58 million in FY2016; over half goes to states

– 100 BLS and 300+ state employees

State LMI agencies prepare and provide quarterly

UI tax micro data

– Establishment-level

– Within 15 weeks

Shared governance

council (BLOC)

3 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

QCEW general background 

 Essential economic information 

Number of establishments 

Monthly employment

Total and average wages

 Issued quarterly (frequent)

Available 6 months after quarter ends

 Published detail

Nation, states, MSAs and counties 

By 6-digit NAICS industry

 Covers 96.5% of civilian employees 

U.S., Puerto Rico and U.S. Virgin Islands
322 The Promise of Evidence-Based Policymaking

6 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Linking

 Establishments linked across quarters 

to create longitudinal records

Sampling frame for BLS establishment 

surveys

Input to Local Area Unemployment 

Statistics 

Source of Business Employment 

Dynamics (BED) and new variables

– Births, deaths, expansions, contractions

– Age, survival rates, firm size

5 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Quality and curation

 Working together to ensure gold-standard 

data

Acquiring

Transforming

Reviewing and editing

Processing

 Data enhancements

Annual Refiling Survey

Multiple Worksite Report

Linking...
The Promise of Evidence-Based Policymaking 323

8 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Uses and analysis – cont’d.

 QCEW’s strong, wide user base

Bureau of Economic Analysis 

Census Bureau

Outside researchers—top ranked

Programmatic

– Employment and Training Administration

– State and local government

Private sector

– Modeling and weighting big data

– Sales plans and location decisions

– Forecasts

7 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Uses and analysis

 An example of high-quality data BLS 

provides to inform smart, evidence-based 

decisions
324 The Promise of Evidence-Based Policymaking

10 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Uses and analysis – cont’d.

 Users of BED job dynamics 

data include

Federal Reserve System

Small Business Administration

Academics

9 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

UI Tax Rate & Actuarial 

Analysis

UI-Covered Employment

Local Area Unemployment

Personal Income (BEA) Largest single input

Gross Domestic Product (BEA)

Economic Forecasting

Current Employment 

Statistics

Occupational Employment 

Statistics BED: Job Creation/Destruction

• Quarterly and Annual

• Size Class Dynamics

• Firm and establishment births/ 

deaths

• Business Survival Rates

• Monthly Labor Review

Occupational Employment Statistics

Occupational Safety and Health Statistics

Current Employment Statistics

National Compensation Survey

Industrial Price Program

Occupational Safety 

and Health Statistics

Programmatic Uses

Benchmarking

(Employment Base)

General Economic Uses

Quarterly 

Census of 

Employment and 

Wages (QCEW)

Analytical 

Uses Sampling

Social Security Administration

State Revenue Projections

Jobs Openings & Labor 

Turnover Survey

Job Openings & Labor Turnover Survey

BLS Publications

• QCEW: Quarterly Releases: County 

by detailed NAICS - 5 month lag

• Annual Employment and Wages

• Special publications/maps

Federal Funds Allocation

$321 Billion

(HUD, USDA, HCFA/CHIP)

State and Local Government

• Services Planning

• Transportation planning

• Emergency planning

Economic Development Indicators

• Cluster Analysis

• Shift Share

• Industry Diversity Indexes

• Location Quotients

• Local Impact analysis

• Site planning decisions

Census Bureau Uses of QCEW

• LEHD: QCEW is essential matching 

key and source of geography and 

industry and link to other datasets, 

and employment

• Business Register: Sharing to 

reduce costs, burden and improve 

consistency

Industry Code Sharing (3 

Million/year)

• COS: Mullti-unit sharing- reduces cost 

and improves consistency

• CPS Redesign: QCEW used after 

decennial Census to improve sample 

design

Map of QCEW uses

Other Agencies and States:

• NAWS/DOL

• NASS: Rural Innovation Survey

• States: Job Vacancies, Green Jobs 

Wage and Benefits Surveys
The Promise of Evidence-Based Policymaking 325

12 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Sustainable, scalable, matchable

– cont’d.

 More initiatives to leverage QCEW

Intra-BLS projects

– Foreign Direct Investment

– Quarterly Refiling Survey pilot

Enterprise data

Wage records

11 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Sustainable, scalable, matchable

 QCEW can be leveraged for new uses with 

little to no additional:

Respondent burden

Cost

 Examples

Non-profit economy

Hurricane flood zone maps
326 The Promise of Evidence-Based Policymaking

14 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

QCEW bottom line

 Product of robust Fed-state partnership

 Comprehensive, detailed, accurate, 

relevant and timely

Supports decentralized decision-making

Cannot be replicated by private sector

Underlies key official and nonofficial statistics

Serves all parts of country and economy

 Flexible—supporting modernization, 

add-ons, matching, modeling and upgrades 

to other programs

13 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Archiving

 Linked micro data

1990-present

Confidential

 Unlinked micro data

1975-present

Confidential

 Detailed tabulations

1975-present

Public website
The Promise of Evidence-Based Policymaking 327

Contact Information

16 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Erica L. Groshen

Commissioner

Bureau of Labor Statistics

www.bls.gov

202-691-7800

groshen.erica@bls.gov

15 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

QCEW: High-ROI component of our 

national informational 

infrastructure 
328 The Promise of Evidence-Based Policymaking

STANFORD CENTER ON

POVERTY AND INEQUALITY

CENTER ON POVERTY AND INEQUALITY
The Promise of Evidence-Based Policymaking 329

STANFORD CENTER ON

POVERTY AND INEQUALITY

•

•

•

•

STANFORD CENTER ON

POVERTY AND INEQUALITY
330 The Promise of Evidence-Based Policymaking

STANFORD CENTER ON

POVERTY AND INEQUALITY

•

•

STANFORD CENTER ON

POVERTY AND INEQUALITY
The Promise of Evidence-Based Policymaking 331

STANFORD CENTER ON

POVERTY AND INEQUALITY

•

•

•

•

STANFORD CENTER ON

POVERTY AND INEQUALITY
332 The Promise of Evidence-Based Policymaking

STANFORD CENTER ON

POVERTY AND INEQUALITY

STANFORD CENTER ON

POVERTY AND INEQUALITY
The Promise of Evidence-Based Policymaking 333

The Approach

STANFORD CENTER ON

POVERTY AND INEQUALITY

STANFORD CENTER ON

POVERTY AND INEQUALITY
334 The Promise of Evidence-Based Policymaking

100 xthe 

ition Matrix (or Copula)

0.21% 0.21% 0.18% 0.19% 0.20% ... 3.70% 4.39% 4.97% 6.19% 9.65%

STANFORD CENTER ON

POVERTY AND INEQUALITY

Measuring Absolute Mobility

STANFORD CENTER ON

POVERTY AND INEQUALITY
The Promise of Evidence-Based Policymaking 335

STANFORD CENTER ON

POVERTY AND INEQUALITY

STANFORD CENTER ON

POVERTY AND INEQUALITY
336 The Promise of Evidence-Based Policymaking

Measuring Absolute Mobility

STANFORD CENTER ON

POVERTY AND INEQUALITY

STANFORD CENTER ON

POVERTY AND INEQUALITY
The Promise of Evidence-Based Policymaking 337

STANFORD CENTER ON

POVERTY AND INEQUALITY

Measuring Absolute Mobility

STANFORD CENTER ON

POVERTY AND INEQUALITY
338 The Promise of Evidence-Based Policymaking

STANFORD CENTER ON

POVERTY AND INEQUALITY

•

•

STANFORD CENTER ON

POVERTY AND INEQUALITY
The Promise of Evidence-Based Policymaking 339

STANFORD CENTER ON

POVERTY AND INEQUALITY

•

•

STANFORD CENTER ON

POVERTY AND INEQUALITY
340 The Promise of Evidence-Based Policymaking

•

•

STANFORD CENTER ON

POVERTY AND INEQUALITY

STANFORD CENTER ON

POVERTY AND INEQUALITY
The Promise of Evidence-Based Policymaking 341

CEP questions

1. Does the model acquire (or ingest) data?

2. Does the model curate data?

3. Does the model link data?

4. Does the model analyze data?

5. Does the model provide for sustainability and 

scalability?

6. Does the model archive data?

2

U.S. Census Bureau 

Infrastructure Overview

Ron Jarmin

Associate Director, Economic Programs

December 12, 2016
342 The Promise of Evidence-Based Policymaking

Overview 

1. Census Bureau mission and authorities

2. Infrastructure and data assets

3. Enhancements and opportunities

4

In a word...

Yes...the Census Bureau acquires, ingests, curates, 

links, analyzes, and archives data. And, yes, we are 

striving to promote a sustainable and scaleable

model for accessing a range of high-value, sensitive, 

confidential information.

3

Link and 

Access

Acquire

..................................

....................................

....................................

....................................

....................................

....................................

....................................

....................................

....................................

....................................

AGREEMENT

....................................

....................................

....................................

....................................

...................................

Data Protection Clauses

....................................

....................................

....................................

....................................

Ingest and 

Curate

Analysis Maintain 

and Archive
The Promise of Evidence-Based Policymaking 343

Legal framework and authority

Title 13 provides authorities for 

for protecting and accessing 

high value information about 

the nation’s population and 

economy.

6

Protecting Accessing

 6: acquire and utilize records to the greatest extent possible

 8: reimbursable studies and joint statistical projects

 9: protect confidential individual and establishment data, limit 

access, and statistical uses

 23(c): swear in researchers to assist the Census Bureau

Mission

The Census Bureau's mission is to measure the nation's people 

and economy. We honor privacy, protect confidentiality, share 

our expertise globally, and conduct our work openly.

More specifically, how do we realize our mission in the context of 

evidence-building?

 Increase the awareness and acceptance of administrative 

data in federal statistics

 Identify, acquire, ingest, process, link, and analyze 

administrative data

 Create products that demonstrate the value of data linkage 

and linked data

5
344 The Promise of Evidence-Based Policymaking

CARRA: individuals and households

8

Federal data

•Internal Revenue Service

•Housing and Urban 

Development

•Childcare Development 

Fund

•Medicaid and Medicare

•Social Security 

Administration

•Veteran’s Administration

•U.S. Postal Service

•Selective Service

State data

•Women, Infants, and 

Children

•Temporary Assistance for 

Needy Families

•Supplemental Nutrition 

Assistance Program

•Low Income Energy 

Assistance Program

•Child Care Subsidy

3rd-party data

•Contact frame

•Public schools

•Property and tax 

foreclosure

Infrastructure and data assets

7

3 Examples

 CARRA data linkage infrastructure

 Longitudinal-Employer Household Dynamics (“LEHD”)

 Federal Statistical Research Data Centers (FSRDC)

censuses, surveys, and frames

external data sources: administrative and 3rd-party data

linked data: products, research, and program evaluation
The Promise of Evidence-Based Policymaking 345

Current evidence-building projects

 Joint work with BJS, BOP, CMS, ERS, IRS, SSA, and VA 

programs, as well as a forthcoming project with 

HUD

 Longitudinal linkage projects with ten institutions in 

seven FSRDCs, the Census Longitudinal 

Infrastructure Project (CLIP)

 Twelve evidence building pilots, including Chapin 

Hall pilot projects

10

Coverage? Utility? Frequency?

Childcare Development Fund

Deeds

FHA Borrowers

Foreclosures

HUD Assisted Renters

Indian Health Service 

IRS 1040s 

IRS Information Returns 

LEAP

MBR

Medicaid

Medicare

National Change of Address 

National Corrections Reporting System

Numident

Property tax

Selective Service Registrations

SNAP

SSI

TANF

Telephone numbers

WIC

Childcare Development Fund Deeds

FHA Borrowers

Foreclosures

HUD Assisted Renters Indian Health Service 

IRS 1040

IRS 1099 LEAP

MBR

Medicaid

Medicare

National Change of Address

National Corrections Reporting System 

Numident

Property tax

Selective Service Registrations

SNAP

SSI

TANF

Telephone numbers

WIC

9
346 The Promise of Evidence-Based Policymaking

Physical Locations

12

Research Data Centers: FSRDCs

 Program created: Census Bureau creates program at 

HQ in mid-1980’s for access to business data

 Institutional Partners

 First remote location added in 1994

 24 locations as of 2016, 6 in development

 259 active projects and 100 projects “on deck” 

 Federal Partners 

 AHRQ and NCHS join RDC program as partners in 2008.

 BLS joined in 2016 as a partner

 Interagency Council on Statistical Policy (ICSP) approves rebranding to Federal Statistical RDCs (FSRDC) in 2015

11
The Promise of Evidence-Based Policymaking 347

LEHD Infrastructure

QCEW*

Economic 

Survey Data Business 

Register

UI* Wage 

Records

Federal 

Records Demographic 

Census/Survey 

Data

OPM*

Public-Use

Data 

Products...

QCEW = Quarterly Census of Employment and Wages

UI = Unemployment Insurance

OPM = Office of Personnel Management

Linked 

National 

Jobs 

Data

Firm 

Data

Jobs 

Data

Person 

Data

• Job data cover over 95% of private employment and most state, local, 

and federal jobs

• Data availability: 1990-2015, start year varies by state, rolling end date

14

Interconnectivity

13
348 The Promise of Evidence-Based Policymaking

How to access LEHD data

 Internal Projects

 Partner with Census Bureau researchers

 Must abide by specific rules stated in Memorandum of 

Understanding (MOU)

 Limitations: capacity constraints, state laws, release of 

state-specific results

 Federal Statistical Research Data Center 

(FSRDC) Projects

 Well-established process

 Additional hurdles for access to full LEHD data

16

Current and planned LEHD projects

 Internal Projects

 Job-to-Job Flows Data Set

 Labor Market Outcomes Associated with Military Service

 Education Pilot

 Other Federal Projects (DOL)

 Homeless Veterans Reintegration

 Trade Adjustment Act Evaluation

 External Projects 

 Firm Financial Constraints and Employment

 Labor Market Implications of External Shocks

15
The Promise of Evidence-Based Policymaking 349

Discussion

Ron Jarmin

John Abowd

Lucia Foster

Amy O’Hara

18

Enhancements to infrastructure

 Cloud-based data facility in partnership with NYU as Software 

as a Service (SaaS) platform

 Short-term and long-term solutions for creating and displaying 

metadata

 Data infrastructure website 

[http://www.census.gov/about/adrm/linkage.html]

 Research partnerships with DARPA XDATA performers and 

academic collaborators

 Hardware and software upgrades

17
350 The Promise of Evidence-Based Policymaking

The content of this presentation is the opinion of the writer(s) and does not necessarily represent the opinion of the Internal Revenue Service

 The Revenue Act of 1916 requires the 

Secretary of the Treasury to publish 

statistics on the operations of the tax 

system, at least annually (Internal 

Revenue Code Section 6108(a))

Statistics of Income Mission

Formulate and execute the statistical policies, 

practices, and programs of the Internal Revenue 

Service. Collect, analyze, safeguard, and 

disseminate information on Federal taxation in 

support of tax administration, economic policy 

development, and financial analysis. Serve a 

broad range of users in the IRS, the Federal 

government, the public, and the nonprofit 

sectors. Provide statistical support within the 

Service for a broad range of program evaluation 

and measurement analytics. Lead efforts to 

modernize federal statistical programs and 

practices through engagement with the federal 

statistical community.

SOI History and Mission

2 Joint Statistical Research Program

The content of this presentation is the opinion of the writer(s) and does not necessarily represent the opinion of the Internal Revenue Service

IRS Statistics of Income 

Division’s Joint Statistical 

Research Program

Barry Johnson,

Director, Statistics of Income
The Promise of Evidence-Based Policymaking 351

The content of this presentation is the opinion of the writer(s) and does not necessarily represent the opinion of the Internal Revenue Service

Joint Statistical Research Program

Goals: 

o Provide new understandings of taxpayer behavior that 

impact the administration of the U.S. tax system

o Provide new insights into ways that existing tax policies 

affect people, businesses, and the economy

o Suggest tax policy solutions to advance the common 

good

4 Joint Statistical Research Program

The content of this presentation is the opinion of the writer(s) and does not necessarily represent the opinion of the Internal Revenue Service

SOI Mission (cont.)

o Unlike agencies that collect data through censuses and 

surveys, SOI collects its data from the administrative records 

created from IRS processing of tax and information returns

o Data collected from almost 250 forms and schedules

o SOI uses data from administrative processing to form the 

core of its products

o SOI adds value by:

• Collecting additional information from forms, schedules, and 

attachments

• Reorganizing information to add consistency

• Coding data items to make them statistically useful

• Performing rigorous quality checks to improve accuracy and 

reliability

3 Joint Statistical Research Program
352 The Promise of Evidence-Based Policymaking

The content of this presentation is the opinion of the writer(s) and does not necessarily represent the opinion of the Internal Revenue Service

Project Management - Data

o Access to all available IRS data and SOI samples, as 

required to support specific projects including 

• Longitudinal administrative data sets 

• Curated cross-section and panel sample files

o Able to bring in non-IRS data for linkage to IRS 

files:

• Researcher must obtained license/ permission 

• License must explicitly permit linkage

o Access data primarily at IRS facilities using IRS 

equipment

• Exploring placing IRS equipment at Federal Statistical 

Data Centers for researcher convenience

Joint Statistical Research Program

The content of this presentation is the opinion of the writer(s) and does not necessarily represent the opinion of the Internal Revenue Service

Research Projects Must Support Tax Administration

Internal Revenue Code Section 6103(b)4:

i) the administration, management, conduct, 

direction, and supervision of the execution and application of 

the internal revenue laws or related statutes (or equivalent 

laws and statutes of a State) and tax conventions to which 

the United States is a party, and 

(ii) the development and formulation of Federal tax 

policy relating to existing or proposed internal revenue laws, 

related statutes, and tax conventions

5 Joint Statistical Research Program
The Promise of Evidence-Based Policymaking 353

The content of this presentation is the opinion of the writer(s) and does not necessarily represent the opinion of the Internal Revenue Service

Project Management – SOI Staff Role

SOI assigns all projects to staff with subject-matter

expertise:

o Serve as technical resources and ensure compliance 

with project scope and performance period 

o Review all interim and final products for compliance with 

data-use limitations, contract guidelines, and disclosure 

limitation standards

o Manage administrative processes

• SOI must rely on shared HCO and contract management 

resources 

• Technology support and equipment availability vary 

depending on geographic location of researcher

8 Joint Statistical Research Program

The content of this presentation is the opinion of the writer(s) and does not necessarily represent the opinion of the Internal Revenue Service

Project Management - Arrangements

o Some projects do not require research access to data

o Data access vehicles include: 

• Intergovernmental Personal Mobility Arrangement (IPA) 

• Student Volunteer Agreement

• Contract

o When accessing data, researchers:

• Must undergo full background investigation and data security 

and protection training 

• Are subject to disclosure penalties and prosecution

o Projects ideally last 2 years, but extensions are possible to:

• Facilitate peer review comments

• Expand work particularly beneficial to tax administration

7 Joint Statistical Research Program
354 The Promise of Evidence-Based Policymaking

The content of this presentation is the opinion of the writer(s) and does not necessarily represent the opinion of the Internal Revenue Service

Research Projects 

Call for 

proposals

Number of 

applicants

Number of 

projects

selected

Number of 

institutions

represented*

2008 10 4 5

2012 51 17 13

2014 80 12 17

2016 Call for proposals announced 11/9/16 

and closes 12/31/16

10 Joint Statistical Research Program

* Some projects bring together researchers from multiple institutions 

The content of this presentation is the opinion of the writer(s) and does not necessarily represent the opinion of the Internal Revenue Service

Project Solicitation and Selection

oSolicitation of proposals:

•Email to SOI Listserv subscribers and through various 

professional organization mailing lists

•Presentations at NTA, NBER

oEvaluation criteria: 

• Tax administration relevance, contribution to IRS goals and 

mission

• Data availability

• Availability of SOI staff to participate in and/or oversee work

• Researcher skills, quality of research plan, proposed timeframe

oProjects selected by committee of IRS and U.S. Treasury 

Department staff

9 Joint Statistical Research Program
The Promise of Evidence-Based Policymaking 355

The content of this presentation is the opinion of the writer(s) and does not necessarily represent the opinion of the Internal Revenue Service

Additional Benefits of a Research Program

Administrative Data Enhancements

o IRS Data Bank

• Basic individual level data

• Variables needed to link to additional information

• Improved metadata

• Adding data from information documents (W-2, 1099, etc.)

o Improved analytical tools - Stata and R

o Improved management of computing space and processing 

techniques

Enhancements to publicly released products

o New income data tabulations

o Redesigned public-use individual income tax file

o New data on low-income earners (forthcoming)

o Better understanding of flow-through businesses

12 Joint Statistical Research Program

The content of this presentation is the opinion of the writer(s) and does not necessarily represent the opinion of the Internal Revenue Service

Selected Research Contributions: 

o The impact of tax expenditures and methods for 

increasing program participation

o Measures of intergenerational income mobility

o The impact of education tax credits

o Understanding how workers use taxable withdraws of 

pension funds

o The effects of tax rates on labor-force participation and 

migration

o Compliance studies – auditor effects, impact of offshore 

voluntary disclosure

o New understandings of business behavior

11 Joint Statistical Research Program
356 The Promise of Evidence-Based Policymaking

The content of this presentation is the opinion of the writer(s) and does not necessarily represent the opinion of the Internal Revenue Service

Questions?

Contact:

Barry.W.Johnson@irs.gov

(202) 803-9794

13 Joint Statistical Research Program
The Promise of Evidence-Based Policymaking 357

Legal Context: Privacy Law

The Privacy Act of 1974 (5 U.S.C. 552a)

– Governs the collection, maintenance, use, and 

dissemination of personally identifiable information (PII) 

about individuals maintained by Federal agencies

– No agency shall disclose any record to any person 

without the prior written consent of the individual (there 

are certain exceptions)

– A willful disclosure could result in a misdemeanor and 

fine of not more than $5,000

2

IES Data Licensing Program

Marilyn M. Seastrom, Ph.D.

Presentation for the

Commission on Evidence-Based Policymaking 

December 12, 2016

Washington, DC
358 The Promise of Evidence-Based Policymaking

Legal Context: Confidentiality Law

Confidential Information Protection and Statistical

Efficiency Act (CIPSEA) of 2002 (P.L. 107-347, 

Title V)

– Protects personally identifying information from 

disclosure when information is collected under a 

pledge of confidentiality and for statistical purposes

– A willful disclosure could result in a class E felony 

with fines up to $250,000 and/or up to five years in 

prison

– Allows an agency’s agents to access restricted data.

4

Legal Context: Confidentiality Law

The Family Educational Rights and Privacy Act (20 

U.S.C. § 1232g; 34 CFR Part 99) protects the privacy 

of student education records 

• FERPA applies to all schools that receive funds under 

an applicable program of the U.S. Department of 

Education

• FERPA allows schools to disclose those records to 

specified officials for audit or evaluation purposes

• FERPA applies to administrative record data that NCES 

obtains from the school or institution without the explicit 

written consent of the parent or student

3
The Promise of Evidence-Based Policymaking 359

Legal Context: IES Confidentiality Law

• Under the IES confidentiality law, no person may:

– Use any individually identifiable information for a 

non-statistical purpose (except in the case of 

terrorism)

– Release data that could identify a person, or

– Permit unauthorized persons to examine the 

individual data 

6

Legal Context: IES Confidentiality Law

Education Sciences Reform Act (ESRA) of 2002 -

Confidentiality Section (20 U.S.C. 9573)

– All individually identifiable information about students, 

their families, and their schools shall remain 

confidential

– Individually identifiable information is immune from the 

legal process and cannot be used in any judicial 

proceeding (except in the case of terrorism

5
360 The Promise of Evidence-Based Policymaking

Data Curation: IES Confidentiality Procedures

• Disclosure Review Board—IES technical staff

– Approve data perturbations

– Clear files for release 

• Direct identifiers are removed—de-identified 

restricted use file

• Sensitive information is removed or recoded—

anonymized public use file

–Top- and bottom-coding

–Categories

8

Legal Context: IES Confidentiality Law

• Under the IES confidentiality law, Federal 

employees, contractors, and licensees who are 

obligated to obey this law and who knowingly 

disclose any individually identifiable information, 

will be subject to a class E felony with fines up 

to $250,000 and/or up to five years in prison

7
The Promise of Evidence-Based Policymaking 361

Data Access: IES Confidentiality Procedures

• Data Analysis Systems—user specified 

tabulations are provided online using restricted 

use data

– National Data Explorer (NDE)—NAEP

– International Data Explorer (IDE)—TIMSS, PISA, PIRLS, 

PIAAC

– PowerStats—NPSAS, BPS, B&B, ELS, HSLS, SSOCS

• Restricted Use Data Licensing—Qualified external 

researchers are provided access through a licensing 

agreement

10

Data Linkage: IES Confidentiality Procedures

• Restricted use data may be linked to

– Department of Education administrative data 

at the student and institution levels

– Other Federal data at the student level

– External third party data at the student and 

institution levels

– Aggregate data from Census/ACS using 

geocodes

9
362 The Promise of Evidence-Based Policymaking

Background: IES Data Licensing Program

• Non-IES data users are loaned restricted use 

data through a license (contract)

• The license is between IES, the user, and the 

user’s organization (e.g., a university, a research 

institution, or company)

• Licensed users agree to the terms of the IES 

confidentiality law

• IES loans restricted-use data only to qualified 

organizations in the 50 States and DC 

12

Background: IES Data Licensing Program

• Legislation provides the basis for the IES Data 

Licensing Program

• The IES Director may utilize temporary staff, 

including employees of Federal, State, or local 

agencies and employees of private organizations to 

assist in performing the agency’s responsibilities, 

but only if such temporary staff are sworn to observe 

the IES confidentiality law (see 20 U.S.C. 9573 

(d)(3)).

11
The Promise of Evidence-Based Policymaking 363

Application: IES Data Licensing Program:

• Submit a formal online request, including 

- Name, title, institutional affiliation, full address, 

phone number, and e-mail address (researcher 

only) for the

- Researcher

- Senior Official 

- System Security Officer 

- A list of all authorized users on the data license, 

and

- Information describing the project (see next slide)

14

Background: IES Data Licensing Program

• History of program:

– 1989: Initiated talks with OMB to start a trial data 

licensing system; Developed protocol and legal 

documents

– 1991: First license issued

– 2000: 502 restricted-use licenses

– 2002: NCES program expanded to IES (ESRA)

– 2007: Implemented electronic application system

– 2016: 1,261 restricted-use licenses

13
364 The Promise of Evidence-Based Policymaking

Application: IES Data Licensing Program

• Information describing the project includes 

(continued)

- A description of any data that will be linked to the 

requested data

- An indication of which education sectors will be 

served by the project

- An agreement that the requested data will not be 

used for any administrative or regulatory purpose

- The length of the requested loan 

16

Application: IES Data Licensing Program

• Information describing the project includes

- The name, year and subject matter of the data 

file(s) requested 

- Non IES FERPA protected data, whether the 

proposed work is to be conducted as a special 

study, or for or on behalf of a school or institution

- The project title and a brief description of the 

research objective and how the requested data 

will be used

- An explanation of why the public-use files cannot 

meet the researcher’s need 

15
The Promise of Evidence-Based Policymaking 365

Data Access: Data Licensing Program

• The licensee must agree to

- Keep the data safe from unauthorized 

disclosures at all times

- Participate in unannounced, unscheduled 

inspections to ensure compliance with license 

terms and required security procedures

- Read the Restricted-Use Data Procedures 

Manual. http://nces.ed.gov/pubs96/96860rev.pdf

- Ensure that all users listed on the license read, 

understand, and follow the license and security 

plan requirements.

18

Application: IES Data Licensing Program

• The licensee must submit hard copy of

- Signed license (data use agreement)

- Signed and notarized affidavits of nondisclosure 

for each user on license

- Signed security plan form

17
366 The Promise of Evidence-Based Policymaking

Analysis: IES Data Licensing Program

• To maintain confidentiality in reports, tabular 

results must be published 

– With unweighted counts rounded to the nearest 

10 or 50

– As is, if perturbations are present

– Collapsing tables (columns and/or rows) until 

there are at least 3 respondents in every cell, if 

no perturbations

20

Analysis: IES Data Licensing Program

• The licensee must agree to

- Use the ED approved procedure for reporting 

out tabular results

- Submit all draft information products to IES for 

disclosure review

19
The Promise of Evidence-Based Policymaking 367

Maintenance: IES Data Licensing Program

• The licensee must (continued)

- Only keep and access the restricted-use data on a 

standalone desktop computer. 

- Never place the data on a laptop computer, 

server, external hard drive or UBS memory stick 

- Purge and overwrite the computer’s hard drive 

before attaching the computer to a modem or 

network (LAN connection/internet).

22

Maintenance: IES Data Licensing Program

• The licensee must

• Notify IES immediately if licensee receives any 

legal or other demand for the data

- Use the online license system to notify IES of the 

following types of changes

- Project operations or security procedures

- Departures or additions of the project staff 

- The need for additional data

- Maintain a file of all license documents, 

amendments and affidavits

21
368 The Promise of Evidence-Based Policymaking

Closeout: IES Data Licensing Program

• The licensee must:

- Close the license when the research is 

completed or the license terminates 

- The restricted-use data and all other individually 

identifiable information (e.g., the one backup copy, 

working notes) must be returned to IES or destroyed 

under IES supervision and procedures

- The Close-out Certification Form must be sent to 

IES to complete the license close-out process.

24

Maintenance: IES Data Licensing Program

• The licensee must (continued)

- Ensure that only those persons listed on the 

license have access to the secure project 

office

- Ensure that secure project office keys are returned 

and computer login disabled within 24 hours after 

any staff leave the project 

- Immediately notify IES of staff changes via 

amendment

23
The Promise of Evidence-Based Policymaking 369

Operations: IES Management of Licenses

• Maintain complete, detailed license records 

(amendments and users) in searchable format

• Maintain up-to-date contact information on all 

licensees

• Send security reminders to licensees (via email) 

• Track license expiration dates and notify licensee 

about pending license closure

• Employ agency licensing staff that are detail 

oriented, efficient, and vigilant

26

Operations: IES Management of Licenses

• Review license applications in a transparent matter 

and track all review outcomes

• Conduct disclosure reviews of work products within 

5 business days

• Use physical site inspections and personal 

interviews to ensure security compliance

• Immediately follow-up with licensee on minor and 

major license violations

• Review the site inspector’s work

25
370 The Promise of Evidence-Based Policymaking

IES Confidentiality Program Information

Confidentiality Program overview: 

http://nces.ed.gov/statprog/confproc.asp

Confidentiality laws:

http://nces.ed.gov/statprog/conflaws.asp

Confidentiality standard:

http://nces.ed.gov/statprog/2002/std4_2.asp

28

Questions? Contact:

Marilyn M. Seastrom Shelley Burns

marilyn.seastrom@ed.gov shelley.burns@ed.gov

(202) 245-7766 (202) 245-7279 

Useful Web Resources for Licensing Info

Restricted-use Data License Program:

http://nces.ed.gov/statprog/instruct.asp

Restricted-Use Data Procedures Manual:

http://nces.ed.gov/pubs96/96860rev.pdf

Access Your License in the IES Online System:

https://nces.ed.gov/statprog/licenseapp/requestemail.asp

How to Submit a License Amendment:

https://nces.ed.gov/statprog/instruct_mod.asp

Frequently Asked Questions:

https://nces.ed.gov/statprog/instruct_licensing_faq.asp

https://nces.ed.gov/statprog/instruct_access_faq.asp

27
The Promise of Evidence-Based Policymaking 371

Definitions

• De-identified information is used to describe records 

that have enough personally identifiable information 

removed or obscured, such that the remaining 

information does not identify a specific individual. 

Each record includes a re-identification code that can 

be used by the data manager with a linking key to 

identify the individual. 

• Anonymization—data are de-identified, AND do not 

include a re-identification code; that is, the data 

cannot be linked back to a file with identifiable data. 

Anonymized data may also have been subjected to 

additional statistical disclosure limitation techniques.
372 The Promise of Evidence-Based Policymaking

Data Access in Germany: Historical Development

Where do we come from?

1990s 1998 2001 2004 2016

Page 2

13 January 2017

Bender: United States Commission on Evidence-Based Policymaking 

Access to Administrative Data in the Bundesbank

Stefan Bender, Head of Research Data and Service Center (RDSC), Deutsche Bundesbank

US Commission International Panel

Washington, D.C. 13 January 2017

(The views expressed here do not necessarily reflect the opinion of the Deutsche 

Bundesbank or the Eurosystem.)

January 13, 2017 Meeting
The Promise of Evidence-Based Policymaking 373

Scope of the Bundesbank‘s

Research Data and Service Center (RDSC)

• The RDSC is part of the Bundesbank internal project Integrated 

MicroData-based Information and Analysis System (IMIDIAS)

• Goals of IMIDIAS:

• Support policymaking process

• Encourage cooperation with (external) researchers

• Promote evidence-based policy-making 

• Key principles:

• Data as a public good

• Transparent data access

• Data protection 

Page 4

13 January 2017

Bender: United States Commission on Evidence-Based Policymaking 

Policy evaluation can make better use of existing datasets

• The Bundesbank – like other central banks – produces datasets which 

are highly valuable for policy analysis and research.

• So far, most of these datasets have been used to provide aggregate 

statistics and ad hoc analysis of specific policy issues.

• There is significant knowledge of data and institutional background. 

• Systematic use of these data for policy analysis is often constrained 

by 

• Time

• IT-resources

• Legal restrictions

• The Bundesbank has launched a large-scale initiative aimed at making 

better use of existing data both, for policy analysis as well as internal 

and external researchers.

Page 3

13 January 2017

Bender: United States Commission on Evidence-Based Policymaking 
374 The Promise of Evidence-Based Policymaking

Factsheet on the RDSC of the Bundesbank

• The RDSC has started in 2014 as part of the Statistics Department 

of the Bundesbank. 

• The RDSC offers access for non-commercial research to 

(highly sensitive) micro data of the Bundesbank for free! 

• Over 100 new projects in 2016.

• 14 employees (in 2017 at least 18).

• 12 working places for guest researchers.

Page 6

13 January 2017

Bender: United States Commission on Evidence-Based Policymaking 

Additional Aspects and Arguments for a RDSC

• Trust in researchers needed

• Data quality will increase

• More results on needed content and topics

• Better knowledge on data and content (recruitment)

• „Branding“, „Marketing“

Page 5

13 January 2017

Bender: United States Commission on Evidence-Based Policymaking 
The Promise of Evidence-Based Policymaking 375

Microdata Structure at Deutsche Bundesbank

What treasures does the Bundesbank hide?

Page 8

13 January 2017

Bender: United States Commission on Evidence-Based Policymaking 

Microdata Structure at Deutsche Bundesbank

What treasures does the Bundesbank hide?

Page 7

13 January 2017

Bender: United States Commission on Evidence-Based Policymaking 

Monetary

Financial 

Institutions

(Banks)

Households

Securities

Companies
376 The Promise of Evidence-Based Policymaking

“German Lessons”

• Development was/is fast, but incremental:

trust building, growing data complexity, learning process ...

• (New) skills for researchers / data producers.

• Engagement of researchers (value of data work?).

• Efficiency: researcher passport, metadata system (with elements 

of tripadvisor, amazon), project management in a RDC, ...

• Harmonization/Internationalization: G20 initiative on data sharing 

and data access of central banks. INEXDA network of 5 central 

banks has started: France, Italy, Germany, Portugal, UK.

Page 10

13 January 2017

Bender: United States Commission on Evidence-Based Policymaking 

The 5 Safes in the RDSC (Portfolio Approach)

• Safe people: non-disclosure agreement, contract (with penalty up to 

60,000 Euro, publishing the name, exclusion from access up to 2 

years). 

• Safe projects: non-commercial research, project description.

• Safe environment: working places without internet connection, (cell) 

phone, photo, printer and drive.

• Safe data: (weakly) anonymized data.

• Safe results: output control, papers/presentations are checked.

• Access to real data, anonymization is only one dimension, others 

have more effects on data protection. 

Page 9

13 January 2017

Bender: United States Commission on Evidence-Based Policymaking 
The Promise of Evidence-Based Policymaking 377

Tasks of the RDSC (in more Detail)

The RDSC offers access for non-commercial research to (highly 

sensitive) micro data of the Bundesbank: 

• Generate (linked) micro data

• Offer advisory service on data selection and data access (data 

handling, research potential, scope and validity of data)

• Provide data access and data protection

• Document data and methodological aspects of the data

• Work on own research projects (in close cooperation with the 

Bank’s business areas and the Research Centre)

• Organize conferences and workshops

Page 12

13 January 2017

Bender: United States Commission on Evidence-Based Policymaking 

Information and Contact

Page 11

13 January 2017

Bender: United States Commission on Evidence-Based Policymaking 

German Data Forum: http://www.ratswd.de/en/start

Deutsche Bundesbank: http://www.bundesbank.de/rdsc

Stefan Bender: stefan.bender@bundesbank.de

Website: www.bundesbank.de/rdsc

Contact: fdsz-data@bundesbank.de
378 The Promise of Evidence-Based Policymaking

Modes of Data Access

Page 14

13 January 2017

Bender: United States Commission on Evidence-Based Policymaking 

Off-Site Access On-Site Access

Email, encrypted

(Scientific Use File)

Remote Execution Guest Stay

Factually anonymous Weakly anonymous (= confidential)

Output control

Data Access at the RDSC

Page 13

13 January 2017

Bender: United States Commission on Evidence-Based Policymaking 

Data Producers

 Survey studies

 Official statistics

 (External/Big Data)

Data users

RDSC

• RDSC mediates between data producers and external users. 

• RDSC controls for compliance with data protection

regulations.
The Promise of Evidence-Based Policymaking 379

German Data Forum: Key Facts

• Advisory council to the federal government

• 16 members: 8 data producers / 8 data users 

from research

• Development of a research data infrastructure 

for the social, behavioral, and economic 

sciences

• Accreditation of 30 research data centres

• Facilitating access to high-quality data

• Result of independent initiatives from within the 

scientific community

Page 16

13 January 2017

Bender: United States Commission on Evidence-Based Policymaking 

RDC of the Federal Employment Agency

• The RDC of the Federal Employment Agency in the 

Institute for Employment Research offers access for 

non-commercial research to (highly sensitive) admin 

data (social security records), surveys and their linkages 

for free.

• Access thru 2 sites in the UK and 6 sites in the US.

• Remote Access via JoSuA (Job Submission Application) 

with

• Two modes of output: internal use or publication

mode,

• Internal use mode is treated like a microdata use.

Page 15

13 January 2017

Bender: United States Commission on Evidence-Based Policymaking 
380 The Promise of Evidence-Based Policymaking

Public attitudes to 

the use and sharing of their data

Research for the Royal Statistical Society

Exploring the public’s views on using 

administrative data for research 

purposes. Research for ESRC and ONS

Commission on Evidence Based Policy | ADRN | 13/01/2017 2

Presentation to US 

Commission on 

Evidence Based Poicy

Tanvi Desai
The Promise of Evidence-Based Policymaking 381

Commission on Evidence Based Policy | ADRN | 13/01/2017 4

Commission on Evidence Based Policy | ADRN | 13/01/2017 3
382 The Promise of Evidence-Based Policymaking

Commission on Evidence Based Policy | ADRN | 13.01.2017 6
The Promise of Evidence-Based Policymaking 383

Commission on Evidence Based Policy | ADRN | 13/01/2017 8

Commission on Evidence Based Policy | ADRN | 13/01/2017 7
384 The Promise of Evidence-Based Policymaking

Commission on Evidence Based Policy | ADRN | 13/01/2017 10

1. Smart 

glass

2. 

Fingerprint 

reader 

authenticat

ion

3. Panic 

alarm

Commission on Evidence Based Policy | ADRN | 13/01/2017 9
The Promise of Evidence-Based Policymaking 385

t.desai@essex.ac.uk

help@adrn.ac.uk

https://adrn.ac.uk/

THANK YOU
386 The Promise of Evidence-Based Policymaking

Increased Demand for High-Quality 

Research in Education Policy

• Prior to 2000, studies in education policy had been 

plagued by poor-quality data with small, nonrepresentative samples and no prospective follow-up.

• Era of evidence increased demand for better analysis.

• No Child Left Behind Act increased collection of highquality data, for accountability purposes. In NC, data 

collection had started in 1990s with its ABCs program.

The North Carolina Education 

Research Data Center

Kenneth A. Dodge

Founding Director
The Promise of Evidence-Based Policymaking 387

Basic Terms

• NCERDC acts as agent of NC DPI.

• DPI provides electronic data files to NCERDC.

– 4,000 files per year; identifiers intact

• Legal authorization: “to improve education of children in NC”

• NCERDC cleans, stores, merges files.

• Researchers apply to receive customized, de-identified 

data files for a specific study.

• Four-person board establishes policy, ensures data 

security, and reviews requests.

Establishment of the

NC Education Research Data Center

• A partnership between Duke University and the NC 

Department of Public Instruction

• Established in January 2001, renewed through 2021

• Initially funded by Spencer Foundation, now fees

• 375 approved research projects, 50 dissertations

– Half initiated outside of NC

– All for research, not for political use or journalism
388 The Promise of Evidence-Based Policymaking

Data Files in NCERDC

 Annual student files

 1,459,852 children in 2,592 charter & public schools

 Test scores, special ed, graduation, discipline

 Teacher and principal credentials; $$$ school information 

 Merged longitudinally back to 1993

 Can be merged with other files

 Birth records, child abuse registry, arrest, Medicaid, higher education

 Can create peer context information 

 Add local context data, such as job losses and natural disasters

NCERDC Structure

DPI Files

NCERDC

Identified 

files; 

Merged

Qualified 

Scholars NCERDC

Deidentified 

files

Public; 

Journalists

Birth 

Records

Arrest 

Records

Other 

Records

XX
The Promise of Evidence-Based Policymaking 389

National Opportunities

 NSF Network co-led by D. Figlio and K. Dodge

 60 scholars and state-level ed policy leaders

 Statewide Longitudinal Data Systems (SLDS)

 > $600 million to states to develop databases

 Little used for research, to date

 Race to the Top and Early Learning Challenge 

 Barriers can be overcome

 Legal, through statute

 Data security, through standards

 Relationships of trust, through incentives and careful use

 Public will, through leadership

Examples

• Evaluate impact of funding for pre-kindergarten program

– State funding varied (randomly) across 100 counties and 20 years

– Find that: Children living in well-funded counties at age 4 have higher 

test scores in elementary school at least through grade 5

• Evaluate impact of lead exposure on student outcomes

– Find that: Higher lead in housing walls associated w/ lower test scores

• Evaluate impact of middle school on student trajectories

– 85% of districts have k-5, 6-8, 9-12 system (others have k-6 or k-8)

– Sixth graders in k-5 system have more substance use infractions

• Same students have worse outcomes in high school
390 The Promise of Evidence-Based Policymaking

Goals for presentation

• Chapin Hall background

• Partnerships with data providers (federal, state 

and local agencies) to facilitate access to data

• Importance and uniqueness of state and local 

data for federal evidence and policy purposes

• Discuss a pilot project to understand the 

demand for linkage to federal data sources, 

methods and use cases

Increasing the use of state and local 

administrative data for evidence-building

Robert M. Goerge

Presentation to the 

Commission on Evidence-Based Policymaking 

January 13, 2016
The Promise of Evidence-Based Policymaking 391

Data that supports large scale evidence-building

• Focus on:

– Linked administrative datasets or administrative 

data linked to survey data

– Microdata on individual, families or providers 

(organizations or individuals) with personally 

identifiable information (PII)

– “Universe” data or data on the entire population so 

that sub-state or sub-group analyses can be done

– Historical data to do longitudinal analysis

– Going to scale!

Chapin Hall at the University of Chicago

• Provide public and private decision-makers 

with rigorous data analysis and achievable 

solutions for improving the lives of children, 

youth and families

• Our audiences are policymakers and funders, 

government and private agency leaders, and 

researchers

• Our ongoing partnerships with public systems, 

institutions, organizations, and programs are a 

core strategy to achieve our mission
392 The Promise of Evidence-Based Policymaking

The partnership

• What is different in order to build a 

relationship

– Need to include input from agency staff

– Need participation from agency staff in the 

substance and design, but not the doing of it

– Need them to review results before external 

audiences see them

– Need them to have a chance to respond through 

actions or words to the research

• Only with such a partnership will they perhaps 

see a benefit and provide their data

Need a partnership with the data providers

• Most federal, state and local agency leaders and 

policymakers don’t want to be ”researched”

• It’s their data!! The vast majority (if not all) do 

not have to provide their data to researchers

• Therefore, they need to see the benefit in providing 

access to their data to researchers OR to other 

government agencies

• They need to feel confident that they will be included 

and not be treated at arm’s length in any specific 

research endeavor

• This is a different way of how research has been done 

in the past
The Promise of Evidence-Based Policymaking 393

Examples of federal datasets that are close, but richer 

when accessed from the state

• UI (Unemployment Insurance) quarterly wage 

data at the Census 

– All states, no sampling, up to date

– Common format

– However, not all states allow it to be used for nonLEHD purposes

• Adoption and Foster Care Reporting System

– All states, no sampling, up to date

– Common format

– However, no identifiers, 6 month summaries

Why administrative data from states

• Many federal programs implemented by states report 

microdata to federal agencies.

• These data make up many of the federal datasets

• This federal data, however, is very different in 

format and content from the data that is maintained 

and analyzed by the state agencies themselves or by 

external parties that are provided the state’s data

• The state data is what should be put into an 

administrative data clearinghouse

• Data may be transformed, de-identified, sampled or 

be restricted in its use when sent to the federal agency
394 The Promise of Evidence-Based Policymaking

De-identified data – Child care subsidies

• Child Care Development Fund - CCDF

• Often the largest work support program in the 

state

• Recently, the reauthorization of the CCDF 

program removed the requirement of providing 

PII (Social Security Numbers) to HHS for 

parents and children participating in this 

program

Transformed data example - TANF

• Temporary Assistance to Needy Families

• HHS Office of Family Assistance receives an 

annual summary record of characteristics, 

benefits, services and outcomes of individual 

and families on TANF from each state in a 

specific format so that the data is comparable 

across states and can be used for national 

reporting purposes

• Universe data from 30 states
The Promise of Evidence-Based Policymaking 395

Restricted data - NDNH

• National Directory of New Hires contains

– New Hires 

– Quarterly Wage (QW)

– Unemployment Insurance (UI) 

• Researchers can only use if it is de-identified, or

• “for research purposes found by the Secretary of Health and 

Human Services (HHS) to be likely to contribute to achieving 

the purposes of Part A or Part D of the Social Security Act.”

• 2 years of data is maintained by HHS

Sampled data – SNAP Quality Control

• Data for “conducting quality control (QC) 

reviews of Supplemental Nutrition Assistance 

Program (SNAP) cases”

• Statistical sample

• While other data could be linked to this 

sample, the size of the sample prohibits substate analysis

• Cannot look at SNAP receipt as an outcome
396 The Promise of Evidence-Based Policymaking

Multi-state studies with state data

• If we are not going to use federally help data 

which is comparable, what had to be done?

• Data has to be made comparable

– Often little or no metadata

– Requires researchers have significant subject 

matter and local service system expertise

• However, the richness of a particular state’s 

data can enhance the analyses

– A state may have more historical data

“Raw” state administrative data is richer

• Data pulled from state information systems for 

either their own analysis or analysis by 

external parties (Chapin Hall, CARRA ...)

• Contains state-specific variables (fields) and 

identifiers necessary for the state to 

implement the program

• Richness of sub-year variation, nonsummarized data and the ability to calculate 

and transform to fit research question
The Promise of Evidence-Based Policymaking 397

Response to RFP

• 17 responses to the call for full project 

proposals

• 25 responses to the call for letters of interests

• Individuals and organizations ranged from 

researchers in state and local government 

agencies, local and national advocacy 

organizations, research organizations, and 

universities

• Less than a handful were below par

Using Linked Data to Advance EvidenceBased Policymaking

• Demonstrate an efficient way to link state and local data to 

Census-held data to answer important questions while 

protecting privacy

• Create compelling use cases for strengthening the Census 

linkage infrastructure to serve multiple levels of government

• Inform Federal, state, and local strategies for facilitating data 

linkage across programs

• Supported by the Laura and John Arnold Foundation

• CARRA is a collaborator with CH on this project

• Distributed an RFP for research projects linking ‘PI-held’ data 

to data held by the Census Bureau
398 The Promise of Evidence-Based Policymaking

Data being brought by investigators

• County-level integrated human services data

• County court records 

• State birth certificates

• State post-secondary records

• State juvenile justice youth population

• State/county public benefit receipt 

• K-12 student data

• Applications for state and federal financial aid 

for college 

Primary interests for Census held data

• Employment

• Post-secondary education

• Public benefits: Medicaid, Medicare, SSI, SNAP, 

TANF, HUD

• Decennial Census, American Community Survey

• Topics included: 

– long-term follow-up of welfare reform experiments; 

– study of evicted households; 

– long-term follow-up of students K-12

– public aid for post-secondary education

– health outcomes over the life course
The Promise of Evidence-Based Policymaking 399

Appendix B: Proposal Methodologies 

• Descriptive studies

• Needs assessment

• Eligibility/program take-up

• Long-term follow-up of 

RCTs

• Quasi-experimental studies

• Regression discontinuity 

• Propensity score matching

• Difference-in-difference 

• Longitudinal analysis 

• Life course/trajectory 

models

• Policy analysis 

• Predictive analytics 

• Data linkage/warehouses 

• Cluster analysis 

• GIS/mapping 

Appendix A: Proposal Topics 

• Education: preschool, K-12, 

post-secondary

• Employment

• Minimum wage

• Housing: homelessness, 

eviction, mobility

• Criminal justice

• Health 

• Lead exposure

• Suicide

• Disaster preparedness 

• Intergenerational poverty

• Multi-system families

• Child support

• Eligibility determination

• Food insecurity/SNAP

• Refugees

• Immigration

• Predictive analytics

• Taxation 

• Federal regulation 
400 The Promise of Evidence-Based Policymaking

WASHINGTON STATE DEPARTMENT OF SOCIAL AND HEALTH SERVICES

DSHS | Research and Data Analysis

2

Analytics in the State Social and Health Service Environment

 Medicaid expenditures are disproportionately concentrated in 

populations with multiple comorbid physical and/or behavioral health 

conditions

Overall social and health service program costs are driven by a 

relatively small number of persons with overlapping risk factors and 

service needs, often exacerbated by extreme poverty, trauma, mental 

illness, substance use disorders, cognitive limitations or functional 

impairments

High-cost clients often have significant social support needs such as the 

need for economic, housing or employment support, or interventions to 

reduce the risk of criminal justice involvement

 Increased demand to use state agency data to directly inform care

 Increased emphasis on quality/outcome measurement and valuebased payment structures

WASHINGTON STATE DEPARTMENT OF SOCIAL AND HEALTH SERVICES

DSHS | Research and Data Analysis

1

David Mancuso, PhD  January 13, 2017

Getty Images

Supporting Delivery System Transformation 

Through Data Integration and Analytics
The Promise of Evidence-Based Policymaking 401

WASHINGTON STATE DEPARTMENT OF SOCIAL AND HEALTH SERVICES

DSHS | Research and Data Analysis

4

How do we use integrated administrative data?

Policy analysis

 Example: describing the link between ED utilization and 

prescription narcotic drug-seeking behavior

Program evaluation

 Example: evaluating the impact of SUD treatment on health 

care costs and criminal justice involvement

Predictive modeling and clinical decision support

 Example: dynamic patient-level risk scoring to identify high-risk 

dual Medicare/Medicaid enrollees for engagement in Health 

Homes and to support direct patient care

Performance measurement 

 Example: monitoring health care quality, utilization and “social 

determinant” outcome measures

WASHINGTON STATE DEPARTMENT OF SOCIAL AND HEALTH SERVICES

DSHS | Research and Data Analysis

3

Data Sources in the DSHS Integrated Client Databases

School Outcomes 

Preschool – College

Internal

Arrests Charges

Convictions

Incarcerations

Community 

Supervision

Dental Services

Medical Eligibility

Hospital Inpatient/ Outpatient

Managed Care 

Physician Services

Prescription Drugs

Hours

Wages

Housing Assistance

Emergency Shelter

Transitional Housing

Homeless Prevention 

and Rapid Re-housing 

Permanent 

Supportive Housing

Public Housing

Housing Choice 

Vouchers

Multi-Family 

Project-Based 

Vouchers

External

Administrative 

Office of the Courts

Employment 

Security 

Department

Department 

of Corrections Washington State Patrol Department of Commerce Health Care Authority

Housing 

and Urban 

Development 

Public Housing 

Authority

WASHINGTON STATE

Department of Social and Health Services 

Integrated Client Databases

Nursing Facilities

In-home Services

Community 

Residential 

Services

Functional 

Assessments

Case 

Management

Community 

Residential 

Services 

Personal Care 

Support

Residential 

Habilitation 

Centers and 

Nursing Facilities

Medical and 

Psychological 

Services 

Training, 

Education, 

Supplies

Case 

Management

Vocational 

Assessments 

Child Protective 

Services

Child Welfare 

Services 

Adoption

Adoption Support

Child Care

Out of Home 

Placement

Voluntary Services

Family 

Reconciliation 

Services

Institutions 

Dispositional 

Alternative

Community 

Placement

Parole

Food Stamps

TANF and State 

Family Assistance

General 

Assistance

Child Support 

Services

Working 

Connections Child 

Care

DSHS 

Juvenile Rehabilitation

DSHS 

Economic 

Services

DSHS 

Aging and Long- Term Support

DSHS 

Developmental 

Disabilities

DSHS 

Vocational 

Rehabilitation

DSHS 

Children’s 

Services

Child Study 

Treatment Center

Children’s Longterm Inpatient 

Program

Community 

Inpatient 

Evaluation/ 

Treatment

Community 

Services

State Hospitals

Assessments

Detoxification

Opiate 

Substitution 

Treatment

Outpatient 

Treatment

Residential 

Treatment

DSHS Behavioral Health

Mental Health and Substance Abuse Services

Education 

Research Data Center

De-identified

Births

Deaths

Department 

of Health
402 The Promise of Evidence-Based Policymaking

WASHINGTON STATE DEPARTMENT OF SOCIAL AND HEALTH SERVICES

DSHS | Research and Data Analysis

6

 Example: “Care Coordination Program for Washington State Medicaid Enrollees 

Reduced Inpatient Hospital Costs” published in April 2015 Health Affairs

– Statistically significant reduction in hospital costs

– Promising reduction in overall Medicaid medical costs

http://content.healthaffairs.org/search?submit=yes&fulltext=care+coordination+program+for+

washington+state+medicaid+enrollees+reduced+inpatient+hospital+costs&x=0&y=0

Inpatient 

Hospital 

Admission

All Long-Term 

Care Costs

Nursing Home

OVERALL

Savings

TOTAL

MEDICAL

Cost Detail

Estimated per member per 

month impact

 $248

 $318

 $18

+ $23

Program Evaluation

Peer-Reviewed Journal Quality Is Possible on a Rapid-Cycle Timeline

WASHINGTON STATE DEPARTMENT OF SOCIAL AND HEALTH SERVICES

DSHS | Research and Data Analysis

5

Examples of Policy Analyses (and Other Content) Published in 2016

https://www.dshs.wa.gov/sesa/rda/research-reports
The Promise of Evidence-Based Policymaking 403

WASHINGTON STATE DEPARTMENT OF SOCIAL AND HEALTH SERVICES

DSHS | Research and Data Analysis

8

Lessons Learned: Keys to Washington State’s Success

 Senior agency leadership recognizing potential for integrated data 

analytics to support improved service delivery

 Maintaining connection between analytic staff and program operations

 Supporting service delivery systems rather than “academic” interests

 Maintaining a commitment to analytical integrity to build trust with 

other agencies, the Legislature, and external stakeholders

 Commitment to engage data owners in timely review of sensitive 

results before public release

 Initial development occurred within a single large umbrella agency

 Integration of new sources dependent on external partner agency 

interest

WASHINGTON STATE DEPARTMENT OF SOCIAL AND HEALTH SERVICES

DSHS | Research and Data Analysis

7

continued

Washington State’s PRISM

predictive modeling and 

clinical decision support 

application supports a 

Medicare-Medicaid Dual 

Eligible Demonstration that 

produced $21.6 million in 

Medicare savings in its first 

year

Predictive Modeling and Clinical Decision Support

Achieving Profound Savings Supporting Direct Client Care
404 The Promise of Evidence-Based Policymaking

WASHINGTON STATE DEPARTMENT OF SOCIAL AND HEALTH SERVICES

DSHS | Research and Data Analysis

10

Questions?

https://www.dshs.wa.gov/sesa/rda/research-reports

Microsoft

WASHINGTON STATE DEPARTMENT OF SOCIAL AND HEALTH SERVICES

DSHS | Research and Data Analysis

9

Lessons Learned: Data Integration Challenges

Obtaining the necessary financial resources

 Establishing effective cross-agency governance structures

 Building and maintaining trust among data owners, including 

addressing privacy concerns

 Conscripting time from state agency subject matter experts

 Maintaining support of constantly evolving state agency leadership

 Maintaining an analytical data infrastructure in a constantly evolving 

policy, program and IT system environment

 Recruiting and retaining internal staff with analytical expertise, or 

finding external contractors with relevant subject matter expertise

Data are plentiful – analytical skills informed by policy and program 

expertise are scarce
The Promise of Evidence-Based Policymaking 405

What is a Statewide Longitudinal 

Data System (SLDS)?

An SLDS allows for the alignment of 

administrative data from multiple 

sources to examine and improve 

education, workforce, and economic 

outcomes along the entire education 

and workforce lifecycle of individuals 

and programs.

A Public Engagement 

Approach to Establishing 

Statewide Longitudinal 

Data Systems

Dr. Domenico “Mimmo” Parisi

Professor of Demography and Applied Statistics

Executive Director, National Strategic Planning and Analysis Research Center
406 The Promise of Evidence-Based Policymaking

Data-centric Approach

●“Begins with data...ends with data”

●Focused on technical issues - acquisition, 

curation, and alignment of data.

Data Lifecycle

Providers Management

• Acquisition

• Curation

• Alignment

Users
The Promise of Evidence-Based Policymaking 407

Public Engagement: Data Providers

Who are the data providers?

●Education: Pre-K, K12, Career 

Technical, and Higher Education

●Workforce: ES/UI, TANF/SNAP, 

Ex-Offender programs, 

Internships, Wage Records

Providers

The Public Engagement Approach

●Begins with a common understanding of the value of data

●Focus on creating a common understanding of the value 

of data between data providers and data users.

Providers Users
408 The Promise of Evidence-Based Policymaking

Public Engagement: Data Management

●Data Governance—ensuring 

privacy and confidentiality and 

establishing common rules 

guided by a common 

understanding of the value of 

data.

●Data Security and 

Expertise—establishing a 

center of excellence for 

storage, processing, and 

analysis.

Management

• Acquisition

• Curation

• Alignment

Public Engagement: Data Access

Who will use the data?

●Researchers and Scientists

●Policymakers

●Administrators

●The Public

●Economic Developers

Users
The Promise of Evidence-Based Policymaking 409

Sustainability

●Show Value - demonstrate that people are using 

the system to:

• Improve education and workforce outcomes 

• Help avoid costs in the implementation of education and 

workforce programs

●Leverage - A data tool becomes an economic 

development tool (e.g., Workforce on Demand 

powered by Mississippi Works)

●Legislative and Executive Support

Capacity

U.S. Department of 

Education created multiyear SLDS grants that 

were designed to 

support design, 

development, 

implementation, and 

expansion of longitudinal 

data systems at the state 

level. lifetracks.ms.gov
410 The Promise of Evidence-Based Policymaking

Questions

Dr. Domenico “Mimmo” Parisi

Professor of Demography and Applied 

Statistics

Executive Director, National Strategic Planning 

and Analysis Research Center

mparisi@nsparc.msstate.edu

https://nsparc.msstate.edu
The Promise of Evidence-Based Policymaking 411

Purpose and Use for Vital Records

• Civil Registration: permanent legal record of vital events 

registered and processed by vital records offices

• National Security: protect against fraudulent use; used for 

proof of citizenship; birth certificates used by SSA to 

generate Social Security numbers, by State for passports

• Public Health: surveillance, monitoring trends, identifying 

emerging issues

The National Vital Statistics System

National Center for Health Statistics

Charles J. Rothwell

Director, National Center for Health Statistics

Presentation for the 

Commission on Evidence Based Policy Making

January 13, 2017
412 The Promise of Evidence-Based Policymaking

Registration of Vital Events

• Registration of births and deaths is not a Federal activity –

the legal authority resides with the states and jurisdictions

• The US has 57 independent registration areas that provide 

the permanent legal record for births and deaths:

• 50 States

• 5 U.S. Territories

• New York City

• Washington, DC

Background

• 1893 International List for Causes of Death

• 1907 Model Legislation for states on vital registration

• 1933 NAPHSIS established - all states and DC 

adopted laws consistent with Model legislation

• 1967 and beyond: NCHS developed automated 

coding system and software

• Today: Electronic registration of vital events
The Promise of Evidence-Based Policymaking 413

Vital Statistics Cooperative Program

• Oldest Federal – State data sharing partnership

• NCHS contracts with 57 jurisdictions

• Collaboration results in the use of standard certificates, 

classifications (ICD), coding rules, and reporting elements

• Records are sent to NCHS:

• NCHS provides quality review and mortality coding

• NCHS returns electronically coded records in 1 day, 

returns manually coded records in 10 days

Federal Role in Vital Statistics

• Public Health Service Act, Section 306(h)(1):

• NCHS mandated to collect data annually from the 

records of births and deaths in registration areas

• Secretary of HHS shall encourage the collection of 

detailed data on ethnic and racial populations

• States and registration areas shall be paid by the 

Secretary the Federal share of its reasonable costs
414 The Promise of Evidence-Based Policymaking

Timeliness

• Improved timeliness = increased value

• Substantial improvements in timeliness for:

• Receipt of records by NCHS from jurisdictions

• Return of records to jurisdictions from NCHS

• Release of data by NCHS

• Timeliness of mortality data differs by cause of 

death – important for mortality surveillance

Robust Collaboration

• NCHS provides funding, training, technical assistance

• Records sent to NCHS must meet certain requirements 

for coding rules and coding structures:

• NCHS provides quality review and mortality coding

• NCHS returns electronically coded records in 1 day, returns 

manually coded records in 10 days

• Result: NCHS produces national statistics based on 6 – 7 

million records of births and deaths each year and 

quarterly estimates on major outcomes
The Promise of Evidence-Based Policymaking 415

Access to Information for the 

Public and Researchers

Mortality Records Received by NCHS within 10 

Days of the Date of Death

0

10

20

30

40

50

2010 2011 2012 2013 2014 2015 2016 (JanSept)

Percent

Year
416 The Promise of Evidence-Based Policymaking

Products and Dissemination

• Data Briefs

• National Vital Statistics Reports

• Vital Statistics Rapid Release Program- Quarterly 

• Natality estimates for birth rates, delivery method, 

preterm, other Gestational age

• Mortality estimates for 15 leading causes of death plus drug 

overdoses, falls (age 65+), HIV, homicide, and firearms
The Promise of Evidence-Based Policymaking 417

Release and Access Policy

Access to Data

• Interactive, web-based services

• WONDER

• WISQARS

• Full micro-data files

• Public use

• Restricted
418 The Promise of Evidence-Based Policymaking

Restricted Data continued

NCHS Review:

Federal Privacy and Confidentiality Requirements:

• Public Health Service Act § 308(d) Information must be used 

for the purpose for which it was supplied, and no identifiable 

information may be published or released without consent

• Following approval from NAPHSIS and NCHS, researchers 

must sign a Data Use Agreement on terms and use

• Data for approved projects will be provided at no cost

Restricted Data

Requests must be reviewed by NAPHSIS and NCHS

NAPHSIS Review:

• Researchers may request customized micro-data files 

containing geographic data on States and counties

• Some states have their own laws, regulations or policies 

restricting access and prohibiting release of certain items

• NAPHSIS must review and approval all requests for 

custom micro-data prior to review by NCHS
The Promise of Evidence-Based Policymaking 419

Contact Information:

Charles J. Rothwell

Director, National Center for Health Statistics

CRothwell@cdc.gov

National Death Index

• Centralized database of death record information

• Assists investigators in the identification of study 

participants who have died

• Records are available from 1979 – 2015

• Available to investigators solely for statistical 

purposes in medical and health research
420 The Promise of Evidence-Based Policymaking

4 POINTS

 Key points in the legal frameworks context 

 The overall process for access

 Governance 

 CASD role and technological solution 

2

Secure Access to Administrative Data in France

Context, Governance and Technology 

Roxane Silberman (CASD-GENES, and CNRS Paris)

Commission on Evidence-Based Policymaking meeting, US

January 13, 2017

1
The Promise of Evidence-Based Policymaking 421

KEY POINTS IN THE LEGAL FRAMEWORKS FOR

ADMINISTRATIVE DATA

• Under various legal frameworks 

• Started by changing specific legal frameworks one by 

one 

– 2014 Tax data 

– 2015 Medico-administrative data

• Extension to all administrative data - 2016

– By clearing of the responsibility of producers regarding 

professional secrecy

• NB Central Bank data depend on the European 

framework: Banque de France opened an on-site access 

• Last step: Facilitate linkage on the basis of the NIR 

(National Identity Number) for statistical and research 

purpose - 2016

4

INTRODUCTION

• Opening access to confidential government microdata

for research in France : a 2 steps process

– Access to official confidential microdata (census and surveys) 

2008 

– Access to administrative data in France since 2014-2016

• Administrative data under a rather dispatched 

organisation and legal frameworks

• However organization of access to administrative data 

has benefited from the centralized process and 

infrastructure set up for access to official microdata

– The approval process: Comité du secret statistique

– The remote access infrastructure: CASD

3
422 The Promise of Evidence-Based Policymaking

GOVERNANCE (1): THE APPROVAL PROCESS

• Comité du secret statistique

• An independent authority chaired by a lawyer 

• Representatives of the Parliament

• Representatives of the Unions

• Representatives of the producers 

• Representatives of CNIL (National Authority in charge of privacy protection) 

and of Archives de France (National Archives)

• 2 Representatives of researchers (proposed by the ministry of research in 

consultation with the Data Archives infrastructure) 

• Provides an advice based on the research project and 

public interest

+ Agreement of the producer

+ Formal approval from the Authority in charge of privacy 

protections if personal data (CNIL)

+ Formal approval from the National Archives 

• Does not require accreditation of institutions 

• Open to researchers from European Union 6

THE OVERALL PROCESS

• A researcher can ask any data under the legal perimeter

• The researcher contacts the producer to check data fit 

the research project 

• Approval process via independent authorities 

+ Agreement of the producer (see above)

• Remote access process: CASD acts as third party 

between the producers and the users 

5
The Promise of Evidence-Based Policymaking 423

THE ACCESS PROCESS: CASD ROLE

• Acts as third party for secure access between data

producers and researchers

• Equal access for all researchers (no specific access for

the founding institutions)

o Preparation and transfer of data with the producers

o Metadata harmonised under DDI

o Establish contracts with researchers (responsibility of

the researcher)

o Enrolment and training of researchers

o Secure remote access to data

o Outputs checked before export or automatic exports

depending on agreements with data holders

o Can act as third party for data linkage

8

GOVERNANCE (2): THE SECURE ACCESS CASD

• A department of GENES, an independent

institution of high education

o Representatives of Ministry of research and

Ministry of finances on board

• CASD governing board includes 5 Founding

Research institutions + Data archive

infrastructure + INSEE

• A Scientific Advisory Board

• 15 members from France and other countries

• Researchers from various disciplines, IT and

methodologists + a representative of INSEE, Eurostat

invited

7
424 The Promise of Evidence-Based Policymaking

10

:

o A central IT infrastructure highly secure

o The SD-Box : the only way to access the central infrastructure. They are sent to

users.

CASD : an integrated system 

10

CASD TECHNOLOGICAL SOLUTION

A distributed architecture designed by GENES IT department

– High security 

o An integrated system

o Physical and logical security including a biometric strong 

authentication

o No possibility to download data

– Usability for the researchers 

o Remote access: researchers can work from their university with their 

SD-Box (+ access card +biometric control)

o Researchers can see the data 

o A research environment for working and writing 

– Flexibility

o Easy to install and deploy in the pre-existing researchers’ institutions 

environment: no compatibility issues, no test to run

9
The Promise of Evidence-Based Policymaking 425

12

Insertions Extractions

Inputs and outputs are controlled

No internet access

The bubble: an hermetic infrastructure 

The hermetic bubble

An hermetic set of secure server

A Hadoop cluster for 

SD-Box are the unique way BigData

to access the bubble

Access via internet is 

encrypted

All data treatments are done 

inside the bubble Confidential data cannot be 

extracted from the bubble

Servers & 

Application

s

Confide

ntial 

data Teralab

12

11

The SD-Box®

Biometric reader Card reader 

Screen and on site 

parameters button

Access box

RJ45 connector

VGA connector

USB connector

11
426 The Promise of Evidence-Based Policymaking

14

THANKS ! 

roxane.silberman@casd.eu

https://casd.eu

CONCLUSION

The technological solution: 

• From the researcher’s perspective: everything is at hand, data, a 

well-known environment with all software, storage, computing power, 

also designed for big data 

• From the security perspective: everything happens at CASD in the 

bubble with no dependence of other systems that may compromise 

security, no possibility to download data 

• From a service perspective (installation, supervision, assistance) 

SD-Box are identical, monitored, configured, updated remotely by 

CASD, easy to replace

• From a cost perspective: the solution integrates software and 

services. SD-Box can be shared by several teams 

The overall unified system :

• A central point for access with harmonised criteria for approval, 

same procedures for access and possibility to easily work with 

datasets from different administrations 13
The Promise of Evidence-Based Policymaking 427

Person

id: 

Person Number

Enterprise

id:

CBR-No

Dwelling

id:

Address

Health

EmploymentEducation

Social

Etc.

CPR

BDR CBR

Questionnaire

Interview

Cadastre

VAT

2

The Danish Statistical Information 

System

Research Services at Statistics 

Denmark. 

Ivan Thaulow

Head of Research Services, Statistics

Denmark 
428 The Promise of Evidence-Based Policymaking

Health data

 National Register of Patients

 Hospitalizations and diagnoses etc. from 1977 

onwards 

 Register of Medicinal Product Statistics

 Total population data from 1995 – prescripted

medicines 

 Health insurance

• Contacts covered by the public health insurance e.g. 

visits to a doctor 

• 1992-

4

Demographic data

 Longitudinal demographic information from 1980 

onwards

 E.g. immigration and emigration

 Marriages and divorces 

 Births and deaths (causes of death)

 Family formation

 References between parents and children

3
The Promise of Evidence-Based Policymaking 429

Labourmarked, business register etc

 IDA – Integrated Database for Labour Market Research

 IDA linkages (persons - workplaces – employees)

 1981-

 Central Business Register (enterprises and accounts)

 Central Register of Buildings and Dwellings (BBR) 

 and real estate registers (owners of real estate)

6

Income, education etc.

 Income statistics, income of the year 1980-

 Now E-income, monthly employment figures

 Cohesive social statistics / Public beneficiaries

 Recipients of transfer benefits

 1984 -

 Highest completed level of education (status 1 

January) 1981-

 Integrated Student Register (longitudinal data)

• 1973-

• detailed educational training programs 

incl. interrupted training programs

5
430 The Promise of Evidence-Based Policymaking

Remote Desktop 

Protocol

8

 High data quality

 Comprise entire 

population

 Longitudinal studies –

cover several years 

(many registers back to 

1980)

Researchers have access to 

a gold mine of data!

7
The Promise of Evidence-Based Policymaking 431

Limitations to Data access

 Only access to data according to a ”need to know” 

principle

 Statistics Denmark may reduce the data applied for 

(samples and grouping)

10

Autorization:

Who can access microdata?

 Only researchers or analysts from specially 

authorized institutions (stable institutions with a 

responsible manager and with a number of 

researchers/analysts)

 Private companies can be authorized and have 

access if they have a stable research environment

9
432 The Promise of Evidence-Based Policymaking

Data Security

Output

 Output with statistical results is forwarded only by e- mail

 Researchers are not allowed to print individual 

records and cannot download data or results

 Mail is logged

 Output is randomly checked by SD

12

Data Security

Users

 The head of the research institute signs Authorization: 

Responsibility & supervision 

 The researcher signs declaration to follow rules for 

access to micro data

 Possible to close an authorization if rules for access to 

micro data is not followed

Data

 All data will be de-identified

 Micro data stays at Statistics Denmark on special 

dedicated servers

 The research server are separated from the statistical 

production at Statistics Denmark

11
The Promise of Evidence-Based Policymaking 433

25. oktober 2011

The End!
434 The Promise of Evidence-Based Policymaking

BIRTH DEATH

+ Hospital reports to vital records office (VRO)

+ VRO reviews data for quality

- If needed, VRO follows up with hospital

+ VRO where birth occurred shares birth record with mother’s 

state of residence, if different

+ VRO submits birth data to NCHS

+ NCHS reviews data for quality

- If needed, NCHS sends back to VRO

- If needed, VRO follows up with hospital

+ VRO conducts final year end edits and submits complete, final 

data to NCHS

+ NCHS conducts final data quality review

- If needed, NCHS sends back to VRO

+ NCHS and vital records jurisdictions each release aggregate 

non-identifiable vital statistics data

There are many permutations of registering and 

reporting vitals; the process depends on where and how 

an event occurs, which determines who initiates the 

process. This flow chart is meant to depict the most 

typical scenarios for births and deaths.

The process for registering vital events, reporting data to NCHS, and releasing vital statistics is complex, with many steps 

performed by many actors from start to finish. When an event occurs, data providers—typically hospitals for birth information and 

funeral homes, physicians, and coroners for death information—submit birth and death data to the vital records jurisdictions so that 

the vital event can be reviewed, edited, processed and officially registered. The jurisdictions are then responsible for maintaining 

registries of such vital events and for issuing certified copies of birth and death records. 

DATA FLOW: PROCESS FOR REGISTERING

AND REPORTING VITALS

Depending upon the circumstances of the death - natural 

causes or questionable circumstances - the death certificate is 

initiated by:

+ Funeral home to supply personal and demographic information

+ Medical official (e.g., certifying physician, coroner, or medical 

examiner) to supply the cause and manner of death and other medical 

information

+ Funeral home completes death certificate with demographic 

information about the decedent and provides to VRO

+ VRO reviews data for quality

- If needed, VRO follows up with funeral home and/or medical official

+ VRO shares death record with state of residence if person died out 

of state and with the state of birth so that state may mark the birth 

record "deceased"

+ VRO submits data to NCHS

+ NCHS reviews data for quality

- If needed, NCHS sends back to the VRO

- If needed, VRO follows up with funeral home and/or medical official

+ VRO conducts final year end edits and submits complete, final data 

to NCHS

+ NCHS conducts final data quality review

- If needed, NCHS sends back to VRO

+ NCHS and vital records jurisdictions each release aggregate 

non-identifiable vital statistics data

+

+

+

+

+

+

+

+

+

+

+

+

+

+

+

+

+

+

OR

RIP

Excerpt from NAPHSIS. (2013). More Better Faster: Strategies for Improving the Timeliness of Vistal Statistics. 

Washington, D.C., p. 9.
The Promise of Evidence-Based Policymaking 435

What Problem are We Trying to Solve?

1. Public Goods Problem for Open Data 

Due to free riding and Þnancial challenges. 

2. Externality/Spillover Problem for Evidence 

Due to inevitable validity and privacy leakage. 

3. Transaction Costs Problem for Researchers 

Due to lack of trusted intermediary platforms.

Economics of Evidence: 

Public Sector Problems and Solutions

Daniel L. Goroff 

Vice President and Program Director, Alfred P. Sloan Foundation

Informed by grantees such as: Cynthia Dwork, Aaron Roth, Adam 

Smith, John Abowd, Jerry Reiter, Sahlil Vadhan, Micah Altman, Gary 

King, Alessandro Acquisti, and at ICPSR, IQSS, BITSS, COS, ADRN...

Opinions are not necessarily theirs or those of the Sloan Foundation. 

February 24, 2017 Meeting
436 The Promise of Evidence-Based Policymaking

Evidence & the “Externality” Problem

• Data is not a “public good” (excludable). 

• Evidence for policy isn’t either (actually rival). 

Need models, hypotheses, and causal inference. 

• “Externality” or “spillover” is when you affect others 

without their choice, e.g., air or water pollution. 

• Every query answered leaks privacy and validity! 

• Solution: regulate bad behavior, facilitate good. 

Data & the “Public Goods” Problem

• Open data is a “public good,” technically speaking. 

• I.e., a commodity that’s non-rival & non-excludable. 

• E.g., lighthouses, parks, discoveries, defense, etc. 

• Problem is to Þnance and sustain public goods. 

• Solutions to free riding are taxes or philanthropy. 

• Works for look-up data: SDSS, Wikipedia, GPS, etc.
The Promise of Evidence-Based Policymaking 437

Facts about Privacy [from DR14]

• Database Reconstruction Theorem: Too many 

statistics answered too accurately from a conÞdential 

database will expose the entire database for sure. 

• Data cannot be fully anonymized & remain useful. 

• Re-identifying anonymized data is not the only risk. 

• Queries over large sets are not protective. 

• Query auditing is problematic & provably impractical. 

• Neither summary statistics nor ordinary facts are safe. 

Accuracy & the Externality Problem 

• Validity of testing a hypothesis against a null H? 

Reject null if p= prob of data D given H is < .05. 

• Say another project tests D against another null H’. 

But should publish only if prob of D given H or H’<.05. 

• Or try 100 tests. Noise should make 5 look signiÞcant. 

If put other 95 away, literature will differ from evidence. 

Called p-hacking, hypothesis Þshing, or data mining. 

• Solutions: Limit access. Or pre-register hypotheses. 

Or use some data to explore, set-asides for testing. 

Or control validity-leakage rate using DP methods...
438 The Promise of Evidence-Based Policymaking

Differential Privacy Properties 

• Note: Because exp(εε)~1+εε for small εε, this means 

M(U) tells you almost nothing new about U=D vs D’. 

• DP Theorem: There exist useful M that satisfy εε-DP. 

E.g., given a standard statistical question about U, 

compute the answer then add noise of “size” εε. 

• Participation: Anything learned from M(U) or after is 

essentially the same whether or not your info is in U. 

• Composition: Doing M1 then M2 is (εε1+εε2)-DP.

Privacy Solutions [DMNS 06]

• Idea: allow researchers to ask certain questions about 

a dataset D to a mechanism M that adds noise to the 

true answer, then gives an approximate answer M(D). 

• DeÞnitions: Let εε>0 and let U be a database I cannot 

see. It has a row for each individual’s information. 

Call a pair of datasets D and D’ neighbors if they differ 

in at most one row. Before learning M(U), I have prior 

beliefs about the odds that U=D vs. U=D’. We say M 

satisÞes εε-differential privacy if learning M(U) cannot 

change those odds by more than a factor of exp(εε).
The Promise of Evidence-Based Policymaking 439

Produce Evidence but Limit Externalities? 

• Let data scientists explore away at synthetic data. 

• Given a hypothesis so generated, access data to test 

it using DP to control privacy and validity leaks. 

• Yes, Differentially Private methods also control 

overÞtting and false positive rates by ignoring D vs DÕ. 

• Thus distinguish between exploratory work on data vs. 

conÞrmatory research that can produce evidence. 

• Who will help facilitate all this for researchers? 

Privacy & the Externality Problem 

• Only shows how to regulate the leakage of privacy. 

Still can’t answer too many questions, or researchers 

could average out the noise. Need a privacy budget. 

• Small εε means more privacy. But requires more noise. 

So can ask more questions, but get less accuracy. 

• Synthetic Dataset Theorem: Given D, you can run an 

M that approximately answers certain statistical 

questions in such a way that researchers can hardly 

ever tell M(D) from M(D’), even after many queries. 
440 The Promise of Evidence-Based Policymaking

Basic References

• Dinur and Nissim (2003) [link] 

• Dwork, McSherry, Nissim, and Smith (2006) [link]

• Machanavajjhala, Kifer, Abowd, Gehrke, and Vilhuber 

(2008) [link] 

• Dwork and Roth (2014) [link]

• Dwork, Feldman, Hardt, Petassi, Reingold, and Roth 

(2015) [link]

• Goroff (2015) [link] 

High Transaction Costs for Researchers 

• Gov’t can try to reduce such costs: currency, FOIA. 

• Administrative data use is now ad hoc: Hard to 

obtain, prepare, protect, supply, sustain, study, link. 

• Need trusted intermediaries with sector expertise. 

Call these Administrative Data Research Facilities. 

• For gov’t or proprietary data, e.g., IRIS, Kilts, AISP, 

CDRC. ADRF’s may also help with federal statistics. 

• Make a network, call it the ADRN, to share standards 

and best practices for producing reliable evidence. 
The Promise of Evidence-Based Policymaking 441

What are We Talking About?

• Getting answers out of data...

... without revealing individual records

• Not a new concept (remember the ‘70s?)! 

• But now the computational machinery may 

be powerful enough for prime time

2

6th Meeting of the Commission on Evidence-Based Policymaking

Learning Statistics Secure from Administrative 

to DNA Records:

Are We There Yet?

Bradley Malin, Ph.D.

Professor of Biomedical Informatics, Biostatistics, & Computer Science

Director, Health Data Science Center

Vanderbilt University

February 24, 2017
442 The Promise of Evidence-Based Policymaking

First Last Age Education

1 7 14 20

2 8 15 21

3 9 16 24

4 10 17 22

5 11 18 25

6 12 19 23

First Last Age Earnings

26 28 31 34

2 8 15 35

3 8 16 36

27 29 32 37

5 11 18 38

6 30 33 39

Department of Education Internal Revenue Service

Traditional Cryptography

This leaks alot of information

4

First Last Age Education

John Smith 32 High School

Jim Jones 45 College

Mary Little 39 Medical School

Mike Glasgow 22 College

Abby Hightower 51 Medical School

Raj Ramesh 62 College

First Last Age Earnings

Tyler Tooney 27 $45,000

Jim Jones 45 $60,000

Mary Little 39 $150,000

Bill Blast 75 $275,000

Abby Hightower 51 $75,000

Sandy Tunep 62 $66,000

Department of Education Internal Revenue Service

3
The Promise of Evidence-Based Policymaking 443

Crypto Advances: Secret Sharing

• Split... then combine

• In practice, this is done through higher order functions

• And once you can count – you can create complex statistical models

-6, -12, 10

2, 3, -10

14, 14, 7

Site C: 7 Site B: 5 

Site A: 10

22!

6

Crypto Advances: Secret Sharing

• Split... then combine

-6 2

14

Site C: 7 Site B: 5 

Site A: 10

5
444 The Promise of Evidence-Based Policymaking

Real World Example: Cybernetica (Caveat)

D. Bogdonov, et al. Students and taxes: a privacy-preserving social study using secure computation. 

Proceedings of the Privacy Enhancing Technologies Symposium. 2016. 

Process Time (testing facility) Time (in the wild)

Aggregation of education data 30 minutes 2 hours

Aggregation of tax data (monthly income) 18 hours 221 hours

Aggregation of tax data (yearly income) 2 hours 15 hours

Data join 30 minutes 4 hours

Analysis of data 29 hours 141 hours

Total time 60 hours 384 hours

8

Real World Example: Cybernetica

• Statisticians from the Estonian 

Center of Applied Research

• Sharemind System

• Linked:

• Individual tax payments from 

Estonian Tax and Customs Board 

(10 million)

• Higher education events from 

Ministry of Education and 

Research (500 thousand)

D. Bogdonov, et al. Students and taxes: a privacy-preserving social study using secure computation. 

Proceedings of the Privacy Enhancing Technologies Symposium. 2016. 7
The Promise of Evidence-Based Policymaking 445

Traditional Cryptography

Hello, my 

name is...

Encryption

Function

Secret Key

Asd;;io1329

Plaintext Encrypted

10

What Happens 

When You Can’t Find 

Trusted Servers?

9
446 The Promise of Evidence-Based Policymaking

10

57

23

Yes!

Me

You

“Garbled” Circuit Evaluation

Encrypted input

Process encrypted data

Extend circuit and carefully randomize order

12

Crypto Advance - Circuit Evaluation:

Am I Older than You?

57

23

Yes!

Me

You

11
The Promise of Evidence-Based Policymaking 447

Crypto Advances: Homomorphisms

• Long word, simple idea

Encrypt(X + Y) = Encrypt(X) + Encrypt(Y)

Decrypt(Encrypt(X + Y) = X + Y

• Can perform arbitrary mathematical computations! 

• Major recent advances in academia and industry.

14

10

57

23

Yes!

Me

You

“Garbled” Circuit Evaluation

Encrypted input

Process encrypted data

Extend circuit and carefully randomize order

The circuits can get big...

The circuits can get slow. 13
448 The Promise of Evidence-Based Policymaking

Application in Health Data Mining

• EPFL & Sophia Genetics

• Integration of 

homomorphic crypto into 

popular health database 

exploration software (i2b2)

• Issue queries like “how 

many patients have 

congestive heart failure 

and genetic variant x?”

J.L. Raisaro, et al. Privacy-preserving explocation of genetic cohorts with i2b2 at Lausanne University 

Hospital. 3rd International Workshop on Genome Privacy & Security. 2016. 16

• EPFL & Sophia Genetics

• Integration of 

homomorphic crypto into 

popular health database 

exploration software (i2b2)

• Issue queries like “how 

many patients have 

congestive heart failure 

and genetic variant x?”

J.L. Raisaro, et al. Privacy-preserving explocation of genetic cohorts with i2b2 at Lausanne University 

Hospital. 3rd International Workshop on Genome Privacy & Security. 2016.

Application in Health Data Mining

15
The Promise of Evidence-Based Policymaking 449

Real Working Software

• Time to compute is quick – seconds (or less)!

• I encourage you to watch 

Kristen Lauter’s talk from the 

2016 Genomics and Patient 

Privacy Conference

https://www.youtube.com/watc

h?v=vUtyuw7YLVM

18

https://www.theregister.co.uk/2016/02/09/researchers_break_

homomorphic_encryption/

Server (Cloud)

Encrypt Encrypt

Encrypt

 Encrypted 

Question

 Encrypted 

Answer

17
450 The Promise of Evidence-Based Policymaking

What are the Challenges?

• Secure computation is NOT a panacea

• Computation can be secured, but the answers can still leak information

• Example: Queries that reveal answers with very small counts

• There will always be a need for good key management, authentication, 

and (to a certain extent) trust – both in the system and the data

20

Where are We?

• The basic math behind secure computation is there.

• The software is there ... for special circumstances.

• The software is not there... for arbitrary on the fly 

computations*

*It is for secret sharing, but not for homomorphic computation

19
The Promise of Evidence-Based Policymaking 451

One More Thing – Secure Hardware

• Multiparty crypto may not be necessary in the future.

• Tamper resistant hardware may provide an opportunity for 

performing statistical analysis on plaintext.

• Hardware obscures individuals from viewing what’s taking place.

• Example: our work on the IBM Secure co-processor in 2012*

• More recently: UCSD’s work on the Intel SGX system (secured RAM)**

*Canim M, Kantarcioglu K, Malin B. Secure management of biomedical data with cryptographic 

hardware. IEEE Transactions on Information technology in biomedicine. 2012; 16(1): 166-175.

**F. Chen, et al. PRINCESS: Privacy-protecting rare disease international collaboration via encryption 

through software guard extensions. Bioinformatics. 2017: in press 21
452 The Promise of Evidence-Based Policymaking

Acknowledgements

 Research supported by 

 National Science Foundation

 ACI 14-43014, SES-11-31897, CNS-10-12141

 National Institutes of Health: R21-AG032458

 Alfred P. Sloan Foundation: G-2015-20166003

 US Bureau of the Census

 Any views expressed are those of the author and not 

necessarily of NSF, NIH, the Sloan Foundation, or the 

Census Bureau

2

Jerry Reiter

Department of Statistical Science

Information Initiative at Duke

Duke University

jerry@stat.duke.edu

1
The Promise of Evidence-Based Policymaking 453

Synergies of integrated system

 Use synthetic data to develop code, explore data, 

determine right questions to ask

 User saves time and resources when synthetic data 

good enough for her purpose

 If not, user can apply for special access to data

 This user has not wasted time

 Exploration with synthetic data results in more efficient 

use of the real data 

 Explorations done offline free resources (cycles and 

staff) for final analyses

4

The vision we are working towards

 Integrated system for access to confidential data 

including 

 unrestricted access to fully synthetic data, coupled 

with

 means for approved researchers to access confidential 

data via remote access solutions, glued together by

 verification servers that allow users to assess quality of 

inferences from the synthetic data.

3
454 The Promise of Evidence-Based Policymaking

Verification servers:

Where are we now?

 Allowable verifications depend on user characteristics

 We have developed verification measures that satisfy 

differential privacy

 Plots of residuals versus predicted values for regression

 ROC curves in logistic regression

 Statistical significance of regression coefficients

 Tests that coefficients exceed user-defined thresholds

 R software package in development

 Open question: how to scale up while respecting 

privacy budgets

6

Synthetic data: 

Where are we now? 

 Available data products (released by Census Bureau)

 Synthetic Longitudinal Business Database, 

 Synthetic Survey of Income and Program Participation

 OnTheMap

 Off-the-shelf software to generate synthetic data? Not yet.

 General plug-and-play routines? 

 Model based synthesis – yes, but hard to characterize 

disclosure risks beyond re-identification

 Formally private synthesis – much theoretical development, 

but not much practical experience for complex datasets

5
The Promise of Evidence-Based Policymaking 455

Illustrative application:

Verification of regression

 Regress log salary on demographics, including gender 

and race 

 Hypothetical results from the synthetic data 

(dummy numbers as we are vetting final analyses):

 Median salaries for Asian men are about 1.5% lower 

than median salaries for white men, holding all else 

constant

 Huge sample sizes, so statistically significant

 Is the result from the synthetic data believable?

8

Illustrative application: 

The OPM Synthetic Data Project

 Created fully synthetic version of the OPM CPDFEHRI status file

 Longitudinal work histories of civil servants from 1988 

to 2011

 Simulate careers, demographics, grades and steps, 

salaries, ....

 Only available to OPM and Duke IRB approved 

researchers at the moment

7
456 The Promise of Evidence-Based Policymaking

Illustrative application:

Verification of regression

 User defines a threshold that represents a result of 

practical significance

 Test if true coefficient for Asian male B < -.01

 Verification software returns differentially private 

answer that reflects uncertainty due to noise 

 Goal: estimate the probability, p = Pr( B < -.01 ) 

 Output: 95% credible interval for p

 Examples: 

 interval for p is (.92, 1.0), conclude synthetic data result valid

 interval for p is (.52, .64), don’t trust synthetic data result

9
The Promise of Evidence-Based Policymaking 457

Tabulations

publication trade-offs

← Loss of detail

Ease of access

Raw microdata

Privacy loss

Tabulations

Public-use 

microdata

Confidentiality protection 

and physical safeguards

Lars Vilhuber

Cornell University

Funding acknowledged under NSF-#1131848 (NCRN) and a grant from the Alfred P. Sloan Foundation
458 The Promise of Evidence-Based Policymaking

public use data

Data provider/ 

custodian

Data user/ 

researcher

How to provide easy and convenient 

access to data with more detail than 

public-use microdata, less privacy loss 

than direct publication of raw data?

The goal of data access
The Promise of Evidence-Based Policymaking 459

basic paradigm

What type of room?

Where is the data?

“Data Enclave” or “Secure Room”

confidential data

What type of room?

What type of access device?

Where is the data?

What type of person?

?

How do results leave the room?
460 The Promise of Evidence-Based Policymaking

virtual data enclaves

Synonyms:

VDI

(virtual desktop infrastructure)

Thin clients

Remote desktop

making things virtual

What type of room?

What type of access device?

Where is the data?

“Virtual Data Enclave”
The Promise of Evidence-Based Policymaking 461

Examples in 2017

• BLS HQ ?

• BJS data access

• Department of Education data

• Census Bureau RDCs 

• Canadian RDCs

• HRS restricted-access data

• and many more

Physical data enclaves Virtual data enclaves

(data remains in secure data center) 

• Census Bureau/Federal 

Statistical RDCs (since early 2000s)

• German IAB RDCs (since mid 2000s)

• French CASD (since late 2000s)

• Cornell’s CRADC, NORC (early 2000s)

• HRS restricted access data (2015)

• and many many more

Examples in 1990s

• BLS HQ 

• BJS data access

• Department of Education data

• Census Bureau RDCs 

• Canadian RDCs

• HRS restricted-access data

• and many more

Physical data enclaves Virtual data enclaves

(data remains in secure data center) 
462 The Promise of Evidence-Based Policymaking

basic levers

Where?

How?

basic levers

What type of room?

What type of access device?
The Promise of Evidence-Based Policymaking 463

Public-use 

microdata

Tabulations

access methods: enclaves

← Loss of detail

Ease of use

Tabulations

Thin client

RDC

Remote 

desktop

Remote 

execution Submitting analysis programs by 

email or through website (possibly 

combined with synthetic 

microdata)

Software on your own PC giving a 

view onto secure data 

environment

Secondary secure PC giving a view 

onto secure data environment

Public-use 

microdata

Tabulations

access methods: enclaves

← Loss of detail

Ease of use

Tabulations

RDC Physically secure room housing 

access devices and/or data

Privacy loss
464 The Promise of Evidence-Based Policymaking

Access matrix for confidential data

# access

points

Access 

computers

Access rooms Avail. analysis 

methods

Type disclosure 

avoidance

FSRDC 

researcher

24 sites 

(~700 users)

Full Full (badge 

access)

Some (choice of 

software)

Manual/ 

variety of rules

Census staff 

researcher

n.d. None (VDI) None (VDI) Some (choice of 

software)

Manual/ self/ 

variety of rules

IAB: JoSuA

researcher

414 users None (Web 

application)

None (Web 

application)

Smaller (software, 

whitelist 

commands)

Manual/ 

variety of rules

CASD 

researcher

371 sites 

(1471 users)

Extra Full 

(custom-built 

hardware)

Some (university 

office, EU)

Some (choice of 

software)

Manual/ 

variety of rules

€300/ pack of 10

Stat.Denmark

(typical EU)

? None (VDI)

- Some

(host institution)

None (VDI) -

Some

(host institution)

Some 

(choice of software)

Manual/ self/ 

variety of rules

Control by data provider of:

What type of room?
The Promise of Evidence-Based Policymaking 465

How do results leave the room?

Typically, the researcher asks an authorized 

agent of the data provider to review the 

results for risks of disclosure, and he will 

then send them to the researcher

How do results leave the room?
466 The Promise of Evidence-Based Policymaking

self-controlled release of results

• Researcher controls release of results

• Prepares results herself

• According to certain prescribed rules

• Sends them through a system

• Automatically receives results typically per email

• Used

• Most often by contractually-controlled non-enclave data

• Data in some university- or faculty-controlled enclaves (HRS, Dept. of Ed)

• Danish researcher access system

What if the “authorized agent” 

were the researcher?
The Promise of Evidence-Based Policymaking 467

Tabulations

access methods: enclaves with researcher-controlled release

← Loss of detail

Ease of use

Raw microdata

Tabulations

Public-use microdata

Thin client

RDC

Remote 

desktop

Remote 

execution

Self-controlled 

release

Tabulations

access methods: enclaves with researcher-controlled release

← Loss of detail

Ease of use

Raw microdata

Tabulations

Public-use microdata

Thin client

RDC

Remote 

desktop

Remote 

execution

Self-controlled 

release
468 The Promise of Evidence-Based Policymaking

penalties

Access matrix for confidential data

# access

points

Access 

computers

Access rooms Avail. analysis 

methods

Type disclosure 

avoidance

FSRDC 

researcher

24 sites 

(~700 users)

Full Full (badge 

access)

Some (choice of 

software)

Manual/ 

variety of rules

Census staff

researcher

n.d. None (VDI) None (VDI) Some (choice of 

software)

Manual/ self/ 

variety of rules

IAB: JoSuA

researcher

414 users None (Web 

application)

None (Web 

application)

Smaller (software, 

whitelist 

commands)

Manual/ 

variety of rules

CASD 

researcher

371 sites 

(1471 users)

Extra Full 

(custom-built 

hardware)

Some (university 

office, EU)

Some (choice of 

software)

Manual/ 

variety of rules

€300/ pack of 10

Stat.Denmark

(typical EU)

? None (VDI)

- Some

(host institution)

None (VDI) -

Some

(host institution)

Some 

(choice of software)

Manual/ self/ 

variety of rules

Control by data provider of:
The Promise of Evidence-Based Policymaking 469

penalties

• IAB:

• Loss of data access for up to two (2) years for researcher and institution

• Contractual penalty up to €60,000 paid by the institution

• Denmark:

• Researcher: Loss of data access for life, or up to three (3) years for 

“minor breaches”

• Institution: Loss of access for a positive but limited (undefined) period

• No financial or penal penalties Of Note: the FSRDC contract explicitly excludes a 

responsibility of the university for the actions of 

its employees, though university remains bound by 

FWA/IRB.

penalties

• FSRDC and federal employee:

• federal prison sentence of up to five (5) years, a fine of up to $250,000, 

or both.

• France:

• prison sentence of up to one (1) year, a fine of up to €15,000, or both.
470 The Promise of Evidence-Based Policymaking

trust and access

penalties

•Does ease of application matter (penal vs. 

contractual rules)?

•Is it conducive to more strongly engage the 

researcher’s employer (typically but not 

exclusively a university)?
The Promise of Evidence-Based Policymaking 471

hypothesis: culture matters

• Researchers and agencies create the communities in which rules are 

applied and enforced

• Training and “indoctrination”:

• Training of FSRDC researchers (short, decentralized) 

vs. FedStat employees (≥1 day on-site)

• 1 full day on-site (in Paris) training for French researchers 

• Common forums: 

• Conferences: Canadian, US (FSRDC, NCHS) yearly RDC conferences

• Discussion, local groups: users of FSRDC share a common physical space

• More or less tight binding of researchers into a community is 

important

What type of person?
472 The Promise of Evidence-Based Policymaking

Concerns about centralized compute 

infrastructure

• Scope

• FSRDC infrastructure dwarfed by 

other federal research 

investments (e.g. XSEDE) that 

cannot be utilized

Cluster Cores Tflops

As a 

multiple of 

FSRDC

FSRDC 240 4.36 1x 

Wrangler (TACC) 2304 62 14x 

Stampede (TACC) 102400 9600 2202x 

virtual enclave = centralization
The Promise of Evidence-Based Policymaking 473

some concluding thoughts

• How to enable a scalable and secure system

• Does it require changes in the legal framework?

• How to build a culture of responsible and secure data access among 

researchers?

• What kind of devices or access mechanisms do we want to enable?

• Who gets to hold the data that researchers actually access?

summary
474 The Promise of Evidence-Based Policymaking

Thanks

• Stefan Bender (formerly IAB and now Bundesbank, Germany)

• Jörg Heining (IAB, Germany)

• Roxanne Silberman (CASD, France)

• Kamel Gadouche (CASD, France)

• Jean Poirier (CIQSS, Canada)

thank you

lars.vilhuber@cornell.edu
The Promise of Evidence-Based Policymaking 475

Some References

• Walter Wilcox (1914) cited in Anderson, Margo J., and Seltzer, William. “Federal Statistical Confidentiality 

and Business Data: Twentieth Century Challenges and Continuing Issues’.” Journal of Privacy and 

Confidentiality 1.1 (2009): 7-52, 55-58.

• Kohlmann, Annette (2005): “The Research Data Centre of the Federal Employment Service in the Institute for Employment Research.” In: Schmollers Jahrbuch 125, 437-447

• Allmendinger, Jutta and Kohlmann, Annette (2005) “Datenverfügbarkeit und Datenzugang am 

Forschungsdatenzentrum der Bundesagentur für Arbeit im Institut für Arbeitsmarkt- und Berufsforschung”. 

In: Allgemeines Statistisches Archiv 89, S. 159-182

• Heining, Jörg (2010): “The Research Data Centre of the German Federal Employment Agency: data supply and demand between 2004 and 2009.” In: Zeitschrift für ArbeitsmarktForschung, Jg. 42, H. 4, S. 337-350. 

http://www.iab.de/389/section.aspx/Publikation/k100128n09

• Kargus, Andrea; Müller, Anne (2014): “Auch in Nürnberg möglich: Von der zweiten Liga in die Champions 

League - ein Gespräch mit Stefan Bender.” In: IAB-Forum, Nr. 2, S. 38-45. 

http://www.iab.de/188/section.aspx/Publikation/k141201301

• Kraus, Rebecca S. (2011): “Statistical Déjà Vu: The National Data Center Proposal of 1965 and Its 

Descendants.” Presentation at JSM 2011. https://www.census.gov/history/pdf/kraus-natdatacenter.pdf
476 The Promise of Evidence-Based Policymaking

These opinions are my own. They are not the opinions of the 

Berkman Klein Center, any of our funders, nor (with the exception of 

co-authorship on previously published work) my collaborators.

Legal Standards for De-identification

Alexandra Wood

Berkman Klein Center for Internet & Society at Harvard University

Presentation before the Commission on Evidence-Based Policymaking

February 24, 2017
The Promise of Evidence-Based Policymaking 477

Overview of US legal framework for de-identification

 De-identification standards are highly sector- and context-specific and vary 

widely depending on the setting. For example, some standards provide an 

objective for de-identification, while others prescribe a method for deidentification.

 Applicability is typically a binary determination that turns on the 

interpretation of terminology such as personal information, personally 

identifiable information, or individually identifiable information.

 Practices also vary, but generally are heuristic and focus on withholding, 

removing, or coarsening pieces of information considered to be identifying.

Evolving landscape for government data releases

 Government agencies are making efforts to release more information to the 

public for a wide range of purposes from transparency and accountability to 

scientific research and innovation.

 Releasing data about individuals inherently carries privacy risks.

 De-identification has long been used to enable the release of data while 

addressing privacy concerns.

 However, scientific understanding of privacy is evolving and traditional 

approaches to de-identification are increasingly shown to be inadequate.
478 The Promise of Evidence-Based Policymaking

HIPAA Privacy Rule

Method #1 for de-identifying data: Expert determination

A person with appropriate knowledge of and experience with generally accepted 

statistical and scientific principles and methods for rendering information not 

individually identifiable:

(i) Applying such principles and methods, determines that the risk is very small 

that the information could be used, alone or in combination with other reasonably 

available information, by an anticipated recipient to identify an individual who is a 

subject of the information; and

(ii) Documents the methods and results of the analysis that justify such 

determination

Variations in standards: Selected laws

 HIPAA Privacy Rule

 Family Educational Rights and Privacy Act

 Confidential Information Protection and Statistical Efficiency Act

 Massachusetts data security regulation
The Promise of Evidence-Based Policymaking 479

Family Educational Rights and Privacy Act

Permits the release of de-identified information, without consent, “after the removal of all 

personally identifiable information provided that the educational agency or institution or 

other party has made a reasonable determination that a student’s identity is not personally 

identifiable, whether through single or multiple releases, and taking into account other 

reasonably available information.” (20 C.F.R. § 99.31(b)(1))

Personally identifiable information includes, but is not limited to, names, addresses, 

personal identifiers (e.g., SSNs, student numbers, biometric records), indirect identifiers 

(e.g., date of birth, place of birth, mother’s maiden name), other information that, alone or in 

combination, is linked or linkable to a specific student that would allow a reasonable person 

in the school community, who does not have personal knowledge of the relevant 

circumstances, to identify the student with reasonable certainty, or information requested by 

a person who the educational agency or institution reasonably believes knows the identity of 

the student [in the requested record]. (20 C.F.R. § 99.3)

HIPAA Privacy Rule

Method #2 for de-identifying data: Safe harbor

(i) Categories of information from a list of 18 identifiers (e.g., names, geographic 

units containing 20,000 or fewer people, dates (except year), telephone numbers, 

Social Security numbers, etc.) are removed, and

(ii) The covered entity does not have actual knowledge that the information could 

be used alone or in combination with other information to identify an individual 

who is a subject of the information.

(45 C.F.R. § 164.514)
480 The Promise of Evidence-Based Policymaking

Massachusetts data security regulation

Personal information is defined as the combination of

(1) a Massachusetts resident’s first name (or first initial) and last name, and

(2) any one or more of the following:

(a) Social Security number,

(b) Driver’s license number or state-issued identification card number, or

(c) Financial account number, or credit or debit card number.

This definition explicitly excludes publicly available information.

(201 Mass. Code Regs.§ 17.00)

Confidential Information Protection and Statistical 

Efficiency Act

CIPSEA protects data in identifiable form, meaning “any representation of 

information that permits the identity of the respondent to whom the information 

applies to be reasonably inferred by either direct or indirect means.”

Confidential Information Protection and Statistical Efficiency Act of 2002, Pub. L. No. 107-347, 

tit. V, § 502 (4) (2002).
The Promise of Evidence-Based Policymaking 481

Limitations of de-identification more generally

 Advances in the scientific understanding of privacy have demonstrated that 

privacy approaches relying exclusively on de-identification fail to provide 

reasonable protection.

 While they may reduce some risks, traditional de-identification approaches

• Do not prevent all disclosures or protect information in the manner that 

most individual subjects would expect,

• Address only a subset of privacy attacks and attackers,

• Are not readily scalable for use by non-experts, and

• Often result in the redaction or withholding of useful information.

Gaps in the current framework

 De-identification standards in the US often rely on concepts such as 

personally identifiable information that are not precisely defined.

 Guidance on selecting among and applying privacy measures is limited.

 Standards focus on releases of data in microdata (individual-level) formats.

 Standards and guidance encourage use of a narrow subset of the privacy 

measures available and hinder adoption of stronger techniques.

 Lack of clear guidance leads to inconsistent practices and uncertainty. As a 

result, similar privacy risks (or even identical data) are sometimes treated 

differently by different actors.
482 The Promise of Evidence-Based Policymaking

A modern approach

Selecting combinations 

of appropriate privacy 

and security controls 

based on informational 

risks

A modern approach

 In light of the limitations of de-identification, practitioners may consider:

• Conducting a systematic analysis of informational risks and intended uses, 

and

• Implementing a combination of privacy and security controls rather than 

relying solely on de-identification.

 For example, a tiered access model can

• Closely match combinations of privacy controls to different risks and 

intended uses at each stage of the information lifecycle, and

• Bring gains in both privacy and utility for a broad range of uses across 

different types of data.
The Promise of Evidence-Based Policymaking 483

Micah Altman, Alexandra Wood, David R. O’Brien, Salil Vadhan, and Urs Gasser, 

Towards a Modern Approach to Privacy-Aware Government Data Releases, 30 

Berkeley Technology Law Journal 1967 (2015).

Alexandra Wood, Edo Airoldi, Micah Altman, Yves-Alexandre de Montjoye, Urs 

Gasser, David O’Brien, and Salil Vadhan, Comments on the Proposed Rules to 

Revise the Federal Policy for the Protection of Human Subjects (2016).

Available from http://privacytools.seas.harvard.edu

References

Open query access via 

differentially private tools

Repository/

Curator

Data

Holder

Data 

User

Download w/ sanitization 

and terms of service

Download w/ approval and 

data use agreement

Protected analysis via 

secure data enclave

A modern approach: Example tiered access model
484 The Promise of Evidence-Based Policymaking

Thank you

Commission on Evidence-Based Policymaking

National Science Foundation

Alfred P. Sloan Foundation

Berkeley Center for Law & Technology

Microsoft Corporation

and our collaborators through the Privacy Tools for Sharing Research Data 

project
The Promise of Evidence-Based Policymaking 485

Context

• Uncoordinated programs and services are wasteful and 

often less effective

• Sharing information is necessary for coordination and 

responsible management of taxpayer resources

• We need a “Yes, Unless” expectation of data sharing:

– Technology and systems with capacity to safely and efficiently share 

data 

– Rules and policies that support responsible data sharing

– Empowered governance structures to manage data sharing

– Agency culture that recognizes the risks of failing to share

2

Federal-State Partnerships in Support of 

Data-Informed, Evidence-Based Programs

Maria Cancian

Commission on Evidence-Based Policymaking 

March 13, 2017

1

March 13, 2017 Meeting
486 The Promise of Evidence-Based Policymaking

Recent Examples from ACF (1)

• Information Memorandum on “Data sharing between 

TANF and child welfare agencies”

– Clarifying existing permissions

– Articulating the logic and benefits of data sharing

• Comprehensive Child Welfare Information Systems 

Regulation (CCWIS) makes data sharing a fundamental 

component, requiring:

– New data exchanges with education, courts, and Medicaid 

– A data exchange standard for interfaces with service providers and 

other agency systems

4

Opportunities for Federal Leadership

3

• Clarify permissible practice and expectations

– Articulate a “Yes, Unless” culture

– Remove regulatory barriers, actively support and promote 

appropriate data sharing as a fundamental responsibility

– Hold federal agencies and states responsible when failure to share 

data imposes costs on participants and systems 

• Provide Resources

– Guidance and TA on sharing data across program silos

– Model systems, infrastructure, and tools

– Support for federal agency efforts (technology, governance, data 

management), and state integrated data systems
The Promise of Evidence-Based Policymaking 487

Recent Examples from ACF (3)

• Proposed State Human Services Longitudinal Data System

– Aligns federal and state interests; States as willing partners, with federal 

government offering TA, peer learning, funding

– Supports program management and delivers directly to program 

administrators – improving buy-in and data quality

– Addresses challenge of harmonization across programs in different 

organizations, with capacity to roll up to national level

– Could include incentives and capacity to share data with federal system 

for administrative use, research/statistical use, or both

• Proposed Systems Innovation Technical Assistance Center

– Supports shared software and standards/architecture development

– Current program funding “vertical” with limited support for integration

– Current funding state-specific, not leveraging reuse

6

Recent Examples from ACF (2)

• New Division of Data and Improvement within the ACF 

Office of Planning, Research, and Evaluation (OPRE)

– Integrate existing previously disconnected efforts:

• State system assessment

• PARIS data matching

• Support for interoperability and data exchange (e.g. NIEM)

– Support data sharing across ACF programs (e.g. TANF, child welfare, 

child support), with other agencies and programs in HHS (e.g. 

Medicaid) and beyond (e.g. SNAP) with technical and legal resources

– Support data-informed decision making through improved data 

systems combined with analytic capacity, planning, and evaluation 

resources within OPRE

5
488 The Promise of Evidence-Based Policymaking

Questions?

Maria Cancian

mcancian@wisc.edu

8

Key Objectives of Data Sharing 

• Improving Services, e.g.:

– Streamline eligibility determination by sharing verified information across 

programs to reduce administrative costs and burden on participants

– Connect information from schools or health providers to child welfare records 

to improve continuity of educational and health services, avoid waste

• Measuring Performance

– Individual program performance and variation over time or across 

jurisdictions 

– Indicators of performance drawn from other systems serving the same 

participant (e.g. the earnings of job training program participants, the 

educational outcomes of children in foster care)

• Supporting Formal Evaluations and Research

– Reduce the costs and respondent burden associated with surveys

– Capture detailed and accurate information on government services

7
The Promise of Evidence-Based Policymaking 489

2

1

Using Data and Evidence to Improve 

Labor Department Performance

Seth D. Harris

Commission on Evidence-Based Policymaking

March 13, 2016
490 The Promise of Evidence-Based Policymaking

. . . an ongoing dialogue between 

leaders and others in their 

organization about “how we do 

things around here.”

4

Accountability: evidence connecting 

strategies and outputs to outcomes. 

Leadership: vision, what business you 

are in, and promises about outcomes.

Accountability: strategies, 

output targets, evidence/ 

hypotheses about outcomes.

Reporting on data and 

deadlines.

Sustained effort to put 

performance first 

3
The Promise of Evidence-Based Policymaking 491

 Leadership

 Opportunity

 Definition of Success

 Metrics/Visibility Systems

 Accountability Systems

 Predictable and Repeatable Processes

 Mission-Focus Among Staff/Incentive 

Structures

5
492 The Promise of Evidence-Based Policymaking

The Role of ICSP

• Advises and assists OMB in the coordination 

of Federal statistical activities

• Fosters interagency coordination and 

collaboration

• Membership: 13 statistical agency heads + 1 

rotating member, led by the Chief Statistician 

of the United States

• Authorized by 44 U.S.C. § 3504

2

Expanding the Statistical System’s Capacity to 

Support Evidence-Building

Recommendations to the Commission on Evidence-Based Policymaking

from the Interagency Council on Statistical Policy (ICSP)

Brian C. Moyer

March 13, 2017
The Promise of Evidence-Based Policymaking 493

View Around the President’s Cabinet Table

Agriculture Commerce Defense Education Energy

Health, 

Human

Services

Homeland

Security

Justice

Interior

Labor

State

Transportation

Veteran’s

Affairs

Treasury

Independent

Agencies

Housing,

Urban Dev

Office of 

Management 

and Budget

4

The U.S. Federal Statistical System

• The U.S. decentralized system includes about 

120 agencies

• A substantial portion of official statistics are 

produced by 13 principal statistical agencies

3
494 The Promise of Evidence-Based Policymaking

6

BEA Uses a Variety of Data

Federal Statistical System Decentralized

Agriculture Commerce Defense Education Energy

Health, 

Human

Services

Homeland

Security

Justice

Interior Transportation

Veteran’s

Affairs

Treasury

Independent

Agencies

Housing,

Urban Dev

Office of 

Management 

and Budget

NASS,

ERS

Census,

BEA

NCES EIA

NCHS SSP

NCSES/NSF, 

ORES/SSA

SOI

BTS

BLS

BJS

Labor

State

5
The Promise of Evidence-Based Policymaking 495

Establish a new research environment

• Build a research and computing environment that more fully 

supports the use of statistical information for a range of 

activities, including evidence-based policy research

• Available to governmental and non-governmental 

researchers

• Provides convenient data access for users who are not 

located geographically close to statistical agency 

operations

• Equipped with systems designed to enhance privacy and 

protect data confidentiality

8

Recommendations

• Establish a research environment that brings together data 

sets from across the statistical agencies

• Streamline the research approval process to expedite 

evidence-building work

• Modify legislation to expand the allowed use of administrative 

data for statistical purposes

• Implement strategies that increase access to and improve the 

quality of State and Local government data

7
496 The Promise of Evidence-Based Policymaking

Modify legislation to expand allowed use of data

• Regulations that are in place to protect data confidentiality 

and privacy sometimes prevent statistical use of data

• Example 1: Use of federal tax data by non-Treasury 

employees is limited to “tax administration” purposes

• Example 2: Census Bureau has broad access to federal tax 

data, BEA has limited access, and BLS has no access

• Recommend broadening/amending language in the Internal 

Revenue Code to address above examples

10

Streamline the research approval process

• Current approval process is cumbersome

• Recommend modifications to existing process, not elimination

• Establish review boards composed of technical 

representatives from the agencies that own the data to be 

used in the proposed project

• Harmonized, transparent processes for requesting and 

accepting research project proposals

• ICSP can oversee and coordinate these modifications

9
The Promise of Evidence-Based Policymaking 497

Increase access to and quality of State and Local data

• Many valuable types of administrative data are collected by 

State and Local governments but made available only in a 

limited manner to Federal agencies

• Process for Federal agencies to acquire the data can be very 

time-consuming and costly (e.g. BLS QCEW agreements)

• ICSP can oversee a review of existing State and Local data 

collections to assess current needs and avoid duplication of 

requests made by Federal agencies

• Use results to formulate consolidated requests by Federal 

agencies, identify pilot projects to improve data quality

11
498 The Promise of Evidence-Based Policymaking

In Today’s Discussion:

• Describe Colorado WIC, the services it provides and the 

data it collects

• Present Colorado WIC’s strategic planning efforts including 

a focus on measuring progress towards goals

• Describe the current data sharing efforts in WIC and across 

the state (specifically with the Census Bureau)

• Share barriers to expanding data sharing across efforts for 

Colorado and other Mountain Plains Region WIC States

From Strategy to Action: 

Improving Program Outcomes through 

Targeted Data-Sharing Across Programs

Commission on Evidence-Based Policymaking

March 13, 2017

Presented by:

Erin Ulric, MPH, Nutrition Services Branch Chief (WIC, CACFP, Early 

Childhood Obesity Prevention)

Colorado Department of Public Health and Environment
The Promise of Evidence-Based Policymaking 499

What data do we collect?

Basic demographic information 

Quantitative and Qualitative Nutrition 

Measures 

Information about all services provided 

during the appointment

What is WIC?

WIC is The Special Supplemental Nutrition Program for Women, Infants and 

Children. 

WIC’s goal is to help keep low-income pregnant and breastfeeding women, 

infants and children younger than age 5 healthy. To do this, WIC provides:

• Personalized nutrition consultation

• Breastfeeding information, support and referrals

• Free, healthy food

• Referrals for medical and dental care, health insurance, child care, housing, 

lactation support, and other services that can benefit the whole family
500 The Promise of Evidence-Based Policymaking

Colorado WIC and the Census 

Bureau

• Agreement between US Census and USDA was signed in 

August of 2014.

• The agreement between Colorado WIC was formalized in 

March of 2015 and the final data compilations were 

provided to Colorado WIC in May of 2016. 

• Data provided showed potentially eligible populations from 

calendar years 2013 and 2014
The Promise of Evidence-Based Policymaking 501

Use of Census Bureau Data

• Track program 

performance and 

identify 

populations for 

targeted outreach

• Colorado Gap Map 

www.gapmap.org
502 The Promise of Evidence-Based Policymaking

Future support for datasharing

Increased technical assistance and legal guidance around 

acceptable data-sharing from the Federal level to the States

Targeted funding for research, data collection and analysis

Involve third party entities (i.e. National WIC Association, 

American Public Health Association) in disseminating best 

practices to their respective members to increase buy-in

Barriers to Data Sharing

• Complexity of data and a lack of resources to understand 

and extract the data

• State leadership buy-in and encouragement of data-based 

decision making

• Different legal interpretations of data-sharing by 

state/agency/program

• Collaboration between programs to understand what data is 

available and how it can be used
The Promise of Evidence-Based Policymaking 503

Questions?

Erin Ulric, MPH

erin.ulric@state.co.us 
This page intentionally left blank
The Promise of Evidence-Based Policymaking 505

Appendix G: CEP Public Input–Hearing 

Testimony and Other Public Comments

The Commission provided numerous opportunities for any member of the public to provide input to the Commission’s fact-finding. The 

Commission convened three open public hearings—in Washington, DC, Chicago, and San Francisco—during which any member of the public 

who requested to testify before the Commission 

was allowed to present. A total of 39 members of 

the public presented information during the three 

hearings.

The Commission also issued a Request for 

Comments in the Federal Register and accepted 

comments by email, which generated over 350 responses from the public. 

Appendix G includes written testimonies, a 

copy of comments submitted to the Request for 

Comments, and other public comments provided 

to Commissioners and staff.
506 The Promise of Evidence-Based Policymaking

DC  CA  IL  NY  TX

younginvincibles.org

Facebook: /together.invincible | Twitter: @younginvincible 

Abstract 

Policymakers are making decisions about higher education without crucial performance 

measurements. The Commission on Evidence-Based Policymaking represents a unique 

opportunity to address this problem, and collect and use the information students prioritize the 

most: how different colleges serve today’s diverse student bodies, which majors and programs 

lead to specific occupations and industries, and whether students are repaying their student 

loans.

Statement 

Thank you for the opportunity to speak before you today. My name is Tom Allison and I am the 

Deputy Director of Policy and Research for the Young Invincibles, a national research and 

advocacy organization working to expand the economic opportunities for young adults. 

We know a lot about college and universities: how much schools charge for tuition and fees, 

how many students they enroll, and what types of programs or majors they offer.

We also know a lot about jobs and workforce trends: how many people are unemployed, how 

much money different types of workers make, which industries are growing and shrinking, and 

what skills employers are looking for in their workers.

The problem arises however, when attempting to draw connections between what we know 

about colleges and universities, and what we know about jobs and the workforce. Preventing 

us from connecting that link, is the Student Unit Record Ban, a single paragraph in the 2008 

reauthorization of the Higher Education Opportunity Act, prohibiting the Department of 

Education from collecting and using student-level data. This is frustrating for today’s students, 

who carry challenges and aspirations unique from previous generations, and the majority of 

which pursue higher education to improve their economic opportunities. Choosing where to go 

to school, what to study, and how to pay for it comes with the highest stakes of any decision in 

their life. Students and families need and deserve better insight to inform these decisions. 

Moreover, without outcomes information on which schools and programs lead to jobs and 

ultimately financial security for their graduates, policymakers are left in the dark, unable to 

intelligently align funding with policy priorities. Colleges are also blind to students’ trajectories 

after they leave campus and cannot adjust academic programs or systems to ensure students 

can land good jobs or pursue further education.

Over the course of two years, Young Invincibles conducted workshops, listening sessions, and 

roundtable discussions with current and aspiring college students across the country to better 

understand their priorities and values in attending and paying for college. We synthesized their 

voices in the Student Agenda for Data Reform and organizations representing over one million 

students currently support it. We will submit it in our written comments to the Commission, but 

in brief the agenda calls for overturning the student unit record ban, collecting more information 

about innovative platforms and alternatives to traditional higher education, and to protect the 

privacy and security of sensitive student information.

H1_2016_06

October 21, 216 Hearing

Young Invincibles

Presenter: Tom Allison

Washington, DC Public Input–Hearing Testimony
The Promise of Evidence-Based Policymaking 507

DC  CA  IL  NY  TX

younginvincibles.org

Facebook: /together.invincible | Twitter: @younginvincible

I’d like to dedicate the remainder of my time to read comments from a former student leader 

and recent graduate from the University of Nebraska, named Thien, whose story illustrates the 

need to improve our postsecondary infrastructure:

As a 17-year-old, I did not have nearly enough knowledge of federal loan programs, 

extra college fees, trends in increasing tuition costs, or credit transferability to make the 

best possible decision when considering the investment I was making in paying for 

school. Some online tools can be helpful in estimating front-end costs, but they do little 

to educate on what life after graduation, or dropping out, would bring. It only takes a few 

clicks for a student to receive thousands of dollars in loans, but some can end up 

repaying them for decades afterward. Colleges need to be more transparent when 

advertising their costs by also informing prospective students on the costs that go along 

with repayment.

We need more information on which schools best serve first generation and minority 

students like myself to feel comfortable and assured we’ll find a college committed to 

our success. A college campus can be a very unfamiliar environment when you don’t 

have family members to help navigate the strange new setting. Our institutions of higher 

education need to paint a more accurate picture of their minority communities, and the 

rate of success of those communities experience after graduation, including how 

prepared they are for the workforce. It’s a great resource for some of us, who are not 

used to asking for help and may let ourselves fail out of college before mentioning 

anything to anyone, but it’s frankly not enough.

I can’t speak for every low-income, first generation, minority college student in America, 

but I know these words resonate with a lot of my peers. While we know we need to take 

the reins of our own success, we need to be empowered to do so, and it is clear that 

there is a lot of information that needs to be made available before students can make a 

decision that will impact the rest of their lives and those close to them. What we need 

right now is better data, more of it, and to have it in a transparent and easily digestible 

form.

You can read the rest of Thien’s story and others on our website. Thank you for your time.
508 The Promise of Evidence-Based Policymaking

October 14, 2016

Dear Members of the Commission on Evidence-Based Policymaking: 

The Pew Charitable Trusts promotes transparency and accountability in government through the use of 

rational, reliable decision-making based on facts and evidence. We bring forth research that shows 

which policies, practices, and programs are effective. We have used this evidence-based approach to 

support successful home visiting programs for new mothers, evaluations of state-based tax incentives, 

and public safety programs to reduce recidivism. Our experience shows that helping policymakers enact 

evidence-based policies—those that improve states’ fiscal health and enjoy broad bipartisan appeal—

shifts policymakers’ thinking about how to invest taxpayer dollars. As lawmakers see the benefits of 

evidence-based policymaking in one key policy area, they are more inclined to explore reforms in 

others. 

One of our most successful evidence-based initiatives is the Pew-MacArthur Results First project, a joint 

effort of The Pew Charitable Trusts and the John D. and Catherine T. MacArthur Foundation. A 

growing number of states and counties are partnering with the project to make evidence-informed 

decisions in eight policy areas. At present, we work with 22 states and seven counties to incorporate 

rigorous research into their policy and budget processes and use evidence to identify and invest in 

programs that achieve successful outcomes and positive returns on investment. 

We applaud the federal Evidence-Based Policymaking Commission’s work and share your commitment 

to bring data and evidence to the forefront of federal decision making. We stand ready to be a resource 

as you consider how federal policies and practices could support state efforts to use data and research in 

the policymaking process—and offer our experiences at the state and county level that can inform 

federal level policies and practices.

The Results First approach includes:

 Creating an inventory of currently funded programs;

 Assessing which programs are most likely to work, based on the best available research;

 Utilizing the customized Results First cost-benefit model to compare programs based on their

expected return on investment; and

 Using the results to inform budget and policy choices.

New Mexico has used the Results First approach to compare the expected outcomes of adult criminal 

justice, child welfare, early education, and behavioral health programs. Using their Results First 

analysis, state leaders directed more than $100 million to evidence-based programs. In addition, the state 

is building a culture of evidence by incorporating evidence into their policymaking processes. For 

example, the Corrections Department adopted a policy that mandates that 70 percent of funds are 

directed to evidence-based programs. The department also adopted contracting standards that require 

vendors to document their use of evidence-based practices and monitor outcomes for programs that are 

developed in New Mexico to ensure that they meet the state's goals. 

H1_2016_10

October 21, 2016 Hearing

Pew-Macarthur Results First Initiative

Presenter: Sara Dube
The Promise of Evidence-Based Policymaking 509

Mississippi passed legislation in 2014 establishing evidence standards for evaluating the state's 

corrections, health, education, and transportation programs. Using the Results First model, the state 

determined that a shock incarceration program—a paramilitary, boot-camp intervention—currently 

required by statute has been proven ineffective by national research. The legislature subsequently moved 

to eliminate the program in 2017, and is now developing an evidence-based alternative. In addition, the 

state, through its budget instructions, now requires executive agencies to justify funding for any new 

program by identifying evidence supporting the program's effectiveness. Mississippi policymakers 

expect to use this information to bolster the state's reinvigorated performance-based budget system.

New York State has used the Results First framework to target more than $50 million in state general 

funds over three years toward effective evidence-based alternatives to incarceration programs. 

Recipients of these funds are required to show that the programs are being implemented according to 

their original design—demonstrating fidelity—and that they are achieving expected outcomes. The state 

also leveraged its Results First analysis to compete for and win a $12 million “Pay for Success” grant 

from the U.S. Department of Labor.

Iowa’s Results First analysis confirmed that the state’s existing community-based domestic violence 

treatment program was not effective in reducing recidivism among domestic abusers. In fact, the model 

showed that the state was losing $3 for every $1 invested in the program. To improve outcomes for both 

victims and taxpayers, the department partnered with the University of Iowa to pilot an alternative 

program known as Achieving Change Through Value-Based Behavior, or ACTV. Preliminary results of 

the pilot demonstrated positive effects in reducing recidivism and the department subsequently began 

shifting funds away from the ineffective program and toward ACTV.

Colorado has completed program inventories and cost-benefit analyses in the adult criminal justice, 

juvenile justice, and child welfare policy areas, and is using their results to re-allocate funds in the FY 

16-17 state budget. For example, the state will repurpose $1.9 million in FY 2016- 17 and $2.4 million 

in subsequent years for a new community corrections pilot project for at-risk offenders, centering the 

offender’s treatment on cognitive behavioral therapy (an evidence-based program). The state has also 

dedicated $7.2 million (in FY 2016-17, with investments of $9.5 million each year after) to 

Communities That Care, a prevention system designed to reduce levels of adolescent delinquency and 

substance use through the selection and use of effective evidence-based preventive interventions tailored 

to a community's specific profile of risk and protection.

These are just a few of many examples of states using evidence to inform their budget and policy 

choices. We will submit additional examples and information in response to your request for comments 

via the Federal Register. 

As you develop your recommendations, please consider the effect of federal policies on these state and 

local efforts, and feel free to contact us and our partners with any questions about the lessons learned at 

the state level and how they could be applied at the federal level.

Thank you,

Sara Dube

Director, Pew-MacArthur Results First Initiative

The Pew Charitable Trusts
510 The Promise of Evidence-Based Policymaking

An Automated Evidence-Based-Policy Clearinghouse for Researchers, Practitioners, Federal 

Agencies, and Policymakers: A Proposal to the New Evidence-Based Policymaking Commission 

We are aware of your new Evidence-based Policymaking Commission, recently created by Congress 

and signed into law by President Obama. The bipartisan members who conceived of the need for this 

Commission are to be congratulated—a recognition of the need to infuse scientific evidence into the 

decisions of policy-makers is the first step to effectively designing policies that improve our lives while 

not wasting tax-payer money on unproven strategies. 

Our understanding is that Commissioners have been charged with three general tasks: (1) to improve 

the federal data infrastructure while respecting privacy and security concerns; (2) to incorporate 

outcomes measurement, cost-benefit data, evaluation, randomized controlled trials (RCTs), and 

rigorous impact analysis into federal program design; and (3) to consider the value and nature of a 

clearinghouse that would facilitate access to data by various constituencies and enable the research 

community to judge what works and what does not. 

The Commission will focus on ways to incentivize the rigorous evaluation of programs and policies that 

aim to reduce the problems associated with detrimental prevailing conditions and promote more 

healthful and productive outcomes. Until now, many programs we invest in do not possess stringent 

indicators of their effectiveness and, thus, there is no justification for their continuation. 

There is a wealth of data already collected by the federal government and other agencies and 

organizations reflective of a broad range of phenomena, from physical health to juvenile and criminal 

justice to climate change. Existing data reserves are currently not well organized and thus an 

infrastructure is needed to increase the utilization of these data. 

To facilitate the process of organizing and fully utilizing the data, we recommend a means to directly 

and expeditiously improve policy decisions. Our proposal is highly compatible with the law by 

incorporating federal agency and other data, as well as methodological components that will be readily 

accessible and understandable to those who stand to benefit. And we believe there will be widespread 

support from Congress, the White House and a number of organizations which have an interest in 

evidence-based policy-making. 

We propose that the federal government (and expert contractors) develop an automated 

clearinghouse—perhaps called the “National Evidence-Based Toolkit for Intuitive Navigation” (NETIN)—

that will provide comprehensive information regarding evidence-based programs and policies (EBPs) to 

users; e.g., researchers (who can populate the database), policy-makers (who need to know what to 

legislate and fund), and community organizations, practitioners and government agencies (that need to 

identify best practices). The data populating this toolkit will provide parameters needed to readily map 

available EBPs to existing needs, whether that be to identify best violence prevention practices for any 

given community or to determine which policies to fund to reduce poverty. Also needed is flexibility to 

include innovative and/or promising programs that have yet to be subjected to rigorous evaluation but 

are in the database denoted by their stage of development and need for further study (as per the #2 

mandate above). 

Parameters will be intuitively searchable and fields will be delineated by relevant characteristics; e.g., 

outcome of interest (e.g., diabetes, violence, contaminated water); setting (e.g., school, family, 

community, national); target population (e.g., special needs children, parents, community 

stakeholders, minorities); implementation protocols and frameworks (costs, timeline expectations to 

achieve impact, strategies to shift resources from existing to promising or evidence-supported 

approaches); pertinent literature and resources on assessing and utilizing research; cost-benefit 

analyses; and other information deemed helpful. The goal is to provide a comprehensive, one-stop 

resource that is more user-friendly and searchable on dimensions that are not currently available, 

providing an efficient and valid method to guide evidence-based policy-makers and others who might 

benefit from the resource. 

H1_2016_02

October 21, 2016 Hearing

National Prevention Science Coalition to Improve Lives:

Diane Fishbein
The Promise of Evidence-Based Policymaking 511

The system would be both iterative and interactive; e.g., a search for a category of programs may 

elicit a notation about the need for extra diligence or a particular protocol for implementation. Or 

reference materials may be recommended if using certain interventions. At all stages of navigation, 

weblinks would lead to relevant information. 

Finally, the Clearinghouse would provide a searchable methodology section for researchers who want 

to fill in gaps in the Clearinghouse database. There would be guidance on design, methods, statistical 

techniques, evaluation protocols, and strategies for translation. 

We realize this will be a very large and complex undertaking that will take years to complete and will 

require continual updating. There will also be a need to establish criteria and thresholds for 

designating programs and policies as evidence-based, not only relative to the statistical findings from 

RCTs, but the population significance of those results (e.g., how broadly are effects achieved?). 

Fortunately, there are a number of existing registries that evaluate programs; they can be utilized and 

integrated as best seen fit. The Commission and their advisors will also want to make decisions about 

what policy areas to cover (from human behavior and health to security, the economy, and the 

environment). These objectives for a clearinghouse can be accomplished with sufficient funding and 

commitment, as well as by calling upon the expertise of evidence-based policy-making organizations, 

academics, researchers, current registry experts, federal government database keepers, 

implementation scientists, methodologists, computer scientists, and statisticians. And critical to this 

effort, to ensure its usability and utility, input must be sought from all potential users (e.g., 

community groups, policy-makers, agencies, foundations) working in concert with experts. 

This proposal is reflective of what policy-makers, practitioners, stakeholders and others need to make 

informed, adequately justified, and effective decisions when identifying programs and policies that will 

serve communities and the nation. We have outlined a general roadmap for the creation of a 

clearinghouse—the Commission’s 3rd consideration—with details to be fleshed out after thorough 

discussion and consultation. Our hope is that the Commission will include such a plan that will bring to 

fruition their charge to design a data infrastructure and incorporate results from existing and newly 

conducted studies. There is potential to greatly improve the operations of government, the services 

provided to citizens, and their financial impact. 

Diana H. Fishbein PhD is Co‐Director of the National Prevention Science Coalition to Improve Lives. She is

the C. Eugene Bennett Chair in Prevention Research at the Edna Bennett Pierce Prevention Research

Center at The Pennsylvania State University in State College, Pa.

Neil Wollman PhD is Co‐Director of the National Prevention Science Coalition to Improve Lives and Senior

Fellow at Bentley Service‐Learning Center, Bentley University in Waltham, Mass. Nwollman@bentley.edu

260‐568‐0116
512 The Promise of Evidence-Based Policymaking

Rachel Fishman, Senior Policy Analyst at New America 

Submission for the First Public Hearing of the Commission on Evidence-Based Policymaking (CEP)

Abstract: The US Department of Education puts out $130 Billion a year on federal financial aid to help

students go to college, and billions more are spent by other federal agencies on higher education through

tax credits, the GI Bill, and more. Despite having a tremendous amount of administrative data, policymakers,

students, and families know shockingly little about how particular schools and programs are serving

students due to a law banning the connecting of these data sets. In an era when college has never been more

important nor more necessary, we believe this issue is one the Commission should address directly. 

Oral and Written statement: Thank you for the opportunity to speak today about the better use of existing 

higher education data to support improved decision making by families and policymakers. My name is Rachel 

Fishman and I am a Senior Policy Analyst at New America in the Education Policy Program which uses original 

research and policy analysis to help solve the nation’s critical education problems.

It’s hard to open a newspaper or turn on the television these days without finding another report of the

questionable value of college degrees. As anxiety over student debt and college costs reaches new heights, the

public is growing increasingly uncertain about the value of a college education. The answer to the question “Is 

college worth it” is an unequivocal “yes.” On average. But the real question is: In which program, at which 

college, at which price and for which students is it worth it? 

Students, families, and taxpayers are spending unprecedented amounts on higher education, but remain largely

in the dark about how to spend these precious dollars. And while colleges and universities spend hundreds of

thousands of hours collecting and reporting data, they don’t know how their students are faring compared with

similar students at similar schools. Institutions of all types are subsidized with hundreds of billions of dollars a

year in federal financial aid (not to mention billions more in tax credits, GI Benefits, Department of Labor funds,

and more), but taxpayers don’t know if these dollars are being wasted at diploma mills or poor-performing

institutions. Policymakers have no sense of whether their reforms and investments are helping or hurting the

families that most need the boost higher education can provide. At a time when higher education has never

been as important or as expensive, it’s unimaginable that we can’t answer these critical questions.

Why can’t we answer them? Because the federal government either doesn’t have—or can’t use—the right

data. That’s true, not because it is technically impossible, but because it is illegal. In 2008, Congress passed a

law that banned the creation of a federal student unit record system to enable existing data systems to speak 

with one another and answer critical questions. 

H1_2016_04

October 21, 2016 Hearing

New America

Presenter: Rachel Fishman
The Promise of Evidence-Based Policymaking 513

2

The current hodgepodge of data systems cannot answer basic question like:

● How do part-time and older learners fair in the current system?

● What happens to students who transfer from particular colleges?

● How many—and which—students complete at particular colleges?

● Do students who get some of the more than $30 Billion spent annually on Pell Grants

graduate?

● Are graduates able to find jobs that allow them to pay down their debts?

A system that uses already-collected administrative data would allow us to answer these questions.

Creating a Student Unit Record would not require the collection of additional student data, but would allow the

connecting of existing data already held by a variety of federal and state agencies. Protecting these data at all

points of the lifecycle is crucial, and it is worth considering housing such a system in the Department of

Education’s National Center for Education Statistics, which is classified as a statistical agency and therefore

subject to stringent privacy and security requirements under the Privacy Act of 1974, the Education Sciences

Reform Act of 2002 (ESRA), and the E-Government Act of 2002. We can also look to state level systems for best

practices that could be implemented at the federal level.

We believe using existing administrative data to better understand the outcomes of students at our nation’s

colleges is exactly the type of critical policy issue the Commission was designed to address. We know we have

just scratched the surface here today and we will provide much more detail about the existing administrative

data sources as well as privacy and policy considerations in separate written comments. Thank you for your time

and attention and I look forward to answering any questions.

1899 L Street, NW, Suite 400 • Washington, DC 20036 • Phone: 202-986-2700 • Fax: 202-986-3696 • www.newamerica.org
514 The Promise of Evidence-Based Policymaking

Page 1 of 7

Statement of

George F. Grob 

Chair, Evaluation Policy Task Force 

American Evaluation Association 

For the Commission on Evidence-Based Policymaking 

The Value and Methods of Evaluating

Government Programs Throughout Their Life Cycle 

Rayburn House Office Building 

Washington, D.C. 

October 21, 2016 

H1_2016_03

October 21, 2016 Hearing

American Evaluation Association

Presenter: George Grob
The Promise of Evidence-Based Policymaking 515

Page 2 of 7

American Evaluation Statement

for the

Commission on Evidence-Based Policymaking

Thank you for the opportunity to discuss the important topic of evaluation of federal programs. I 

am a professional independent evaluator, formerly Director of Evaluation at the Office of 

Inspector General, Department of Health and Human Services and the Federal Housing Finance 

Agency. I am writing as Chair of the Evaluation Policy Task Force of the American Evaluation 

Association (AEA), the professional organization devoted to the application and exploration of 

evaluation in all its forms since 1986. 

AEA has approximately 7,000 members across all 50 states, as well as 80 other countries. 

Members have gathered together from many interdisciplinary fields (such as public policy and 

administration, political science, economics, statistics, psychology, sociology, education, public 

health, demography, ethnology, etc.) to create a community of learning and practice over the past 

three decades. Members in academe have worked to develop, refine, and teach evaluation 

methods, while members in practice have served the evaluation needs of many organizations 

including agencies across the federal government. Members serve in many federal evaluation, 

policy, and inspector general offices, and at the Office of Management and Budget.

AEA has developed professional standards for the quality of studies and ethics for the 

multidisciplinary members of the field. Of particular interest today is AEA’s paper: An 

Evaluation Roadmap for a More Effective Government. This document describes many types 

of evaluation that can address management requirements, as well as principles and practices for 

ensuring evaluation quality and usefulness, including methods, human resources, budgets, 

independence, transparency, and professional ethics in a government setting. For your 

convenience, I have attached a copy for your reference. 

I focus here on three main topics: 1) the importance of evidence and the availability of data for 

government decision makers; 2) evaluation methods; and 3) evaluation in government settings.

Evidence and Data for Decision Makers

Government decision makers, including both the Congress and Executive Branch agencies, need 

appropriate evidence to make informed decisions to assess and improve the relevance, efficiency, 

and effectiveness of government programs, policies, and activities (hereafter “programs”). AEA 

applauds the work of the Commission to help Congress embed evaluation into program design 

and to ensure that quality data are available for evaluation.

Federal program design should include an appropriate evaluation framework to guide data 

collection and use over the life of a program. This includes data needed for rigorous impact 

evaluations as appropriate. Measures of a program's key processes and outcomes should be 

established while the program is being conceived and developed. In fact, taking time during the 

process of conceptualizing a new program to specifically define expected outcomes is most 

useful in establishing relevant metrics. Preliminary metrics should be put into place by the time 

program implementation begins, thus allowing key data to be collected to monitor program 
516 The Promise of Evidence-Based Policymaking

Page 3 of 7

implementation, determine progress, and set the stage for methodologically rigorous studies. It 

is vital for some study methods that data be collected prior to the program intervention.

We support the Commission in its efforts to consider whether and how a clearinghouse for 

program and survey data should be established. We encourage efforts to ensure that verifiable, 

reliable, and timely data are available to permit the objective evaluation of programs, including 

an assessment of assumptions and limitations in such evaluations. Agencies should use 

evolving best practices for data security, and ensure that publicly available data are aggregated or 

otherwise stripped of all information that could be used to identify particular individuals or 

businesses.

The proposed clearinghouse could also serve as a repository for the evidence contained in 

evaluation reports, providing an archive capacity for the collection, dissemination, and 

preservation of knowledge and lessons learned from evaluation studies. This would provide an 

enhanced base for guiding future program design and management, which often requires a 

critical mass of knowledge to properly comprehend and address the complexity of program 

processes and influences. It would also be a great benefit for future meta-analyses of evaluation 

findings. 

While recognizing the high value and strategic importance of large-scale archives and datasets, 

the availability of these existing data should not reduce the capacity to gather targeted data as 

needed to address important program evaluation questions. 

Evaluation Methods 

The Commission is charged with making “recommendations on how best to incorporate 

outcomes measurement, institutionalize randomized controlled trials, and rigorous impact 

analysis into program design.” We believe that the key to such evaluation activities is for 

federal entities, (including Congress, as well as Executive Branch agencies and the White House 

Office of Management and Budget (OMB)), to identify the important evaluation questions that 

they need answered to effectively direct the future of Federal programs. Such questions could be 

effectively embedded in authorizing legislation or in congressional committee reports associated 

with legislative authorizations or renewals. Executive Branch agencies can do so in their budget 

documents and implementation plans. Evaluators, in consultation with other experts, can then 

identify which scientific methods are best suited to answer those questions. Specifically defining 

program activities and expected outcomes has proven very useful in choosing relevant evaluation 

questions about program operations and impact.

Such questions, and associated evaluations, are needed throughout the life cycle of programs, 

from their initial authorization through all phases of their implementation. For example, during 

early stages in the life of programs, key questions might center on the fidelity of their 

implementation with statutory requirements and on early implementation problems and 

successes. As the program matures, decision makers might want to establish metrics to track 

such features as enrollment of intended beneficiaries or establishment of required administrative 

systems and other infrastructures. Gradually, interest may shift to outputs, in terms of benefits 

provided and beneficiaries served. Ultimately, decision makers and citizens will want 

information about the impact of programs on people’s lives, the economy, public health, safety, 
The Promise of Evidence-Based Policymaking 517

Page 4 of 7

or other factors or goals associated with the purpose of the programs. Throughout the life of the 

programs, government officials and taxpayers will want to know if funds are being misspent or 

wasted. 

While interest will ultimately focus on program impact and value received for investments made, 

citizens and decision makers do not want to wait until a program has run its course and then 

determine whether it has been working. Along the way, they will want to know if 

implementation problems can be corrected and whether the programs can be improved.

All of these questions are important. But the methods for answering them can be complex. 

Evaluation professionals have a broad range of methods—based on research— from which to 

draw on to answer both impact and operational process questions. Rather than legislating, 

requiring, or overemphasizing any single specific method for impact analysis in federal 

guidance, AEA recommends that federal policy require that careful consideration be given to a 

range of evaluation methods that may be appropriate or feasible in any given circumstance.

 

Over the years, the evaluation field has developed an extensive array of analytic approaches and 

methods that can be applied and adapted to a wide variety of programs and circumstances, 

depending on the program’s characteristics and implementation stage, the way the results will be 

used, and the kinds of decisions that need to be made. In designing evaluation studies it is 

important to recognize that every method has pros and cons and strengths and weaknesses that 

must be addressed in matching them to answer the specific questions, circumstances, and

intended uses of results. There are real-life factors which can render designs infeasible, 

impractical, or inappropriate. To ensure adequate deployment, every study design must examine 

and address feasibility constraints, including resources (funding and time limits), conditions in 

the field, ethical considerations, stakeholder concerns, etc. All evaluation methods should be 

context-sensitive and have cultural relevance.

Agencies should not only focus on tools for evaluation inquiry, but foster evaluation thinking as 

well. High-stakes program decisions should be based on a preponderance of evidence developed 

using sound methods. Some programs may need a high level of credibility and precision in the 

portfolio of evidence upon which leaders base a decision. This may require multiple studies and 

methods as well as a combination of process and impact evaluations to assess and understand the 

effectiveness of an approach within the portfolio of evidence. A range of analytic methods may 

be needed, and often several methods—including quantitative and qualitative approaches—

should be used simultaneously. Multiple methods can offset the shortcomings of any one 

method with the strengths of another.

In fact, some decisions about how to improve the reach and impact of a given program may not 

require a high level of precision or a large portfolio of evidence. Some evaluation approaches 

are particularly helpful in a program’s early developmental stages, whereas others are more 

suited to ongoing and regularly implemented programs or to ex-post analysis of temporary 

programs upon their completion. The broader policy and decision-making context also can 

influence which approach is most appropriate. Sometimes information is needed quickly, 

requiring studies that can use existing data or rapid data-collection methods; at other times, more 

sophisticated long-term studies are required to understand fully the dynamics of program 
518 The Promise of Evidence-Based Policymaking

Page 5 of 7

administration and beneficiary behaviors. Moreover, different approaches can complement one 

another.

The opportunity to capitalize on early successes, identify implementation impediments, or make 

mid-course corrections is critical. So it is essential to conduct ongoing formative evaluation 

throughout the program’s life cycle. For example, evaluation can address questions that arise 

during implementation of the program, such as the validity of assumptions that underlie program 

design, or challenges to implementation in the field. Early in the program’s history, relatively 

simple information may be needed quickly (e.g., regarding obstacles to participation in the 

program). Evaluators should match the methodology to the questions at each stage of program 

development and to information needs, which may call for a range of methods over time, 

including targeted data collection that may not always include outcomes measurement.

Today we see considerable interest in impact analyses, including randomized controlled trials. 

No doubt these are valuable tools and have their place. But we wish to emphasize that they 

represent only some of the methods that can and should be applied, depending on the questions 

that need to be answered. They are not intrinsically better than other methods, except in those 

circumstances where they are most appropriate and feasible. An overarching focus on these 

methods to the exclusion of others will deprive decision makers of valuable insights about ways 

to improve program effectiveness and efficiency, and, when appropriate, whether to increase or 

diminish program funding.

Most federal evaluations need to go beyond estimating aggregate impacts to also addressing 

"what works for whom, and under what circumstances." If the data from evaluation studies are 

to be of most use in guiding evidence-based decision making, they need to be able to support 

conclusions about how program impacts vary across subgroups of those affected by the programs 

and also conclusions about the contexts in which the specific program activities are most 

effective.

Numerous examples are available of evaluations that have enhanced the effectiveness and impact 

of programs but that were conducted early in the program’s life. One that comes to mind is the 

evaluation conducted by the Institute of Medicine during the first five year of the PEPFAR 

program. This was done at the request of Congress, embedded in the original authorizing statute. 

It provided feedback on implementation issues that was available to decision makers at the time 

of the program’s first reauthorization. It is fair to speculate that this early feedback contributed to 

the impact of that program from that time forward, and in many ways was as or more impactful 

than studies performed in later years.

Other studies can affect programs and their impacts when performed several years into their 

implementation. One example is a series of evaluations and audits that identified serious 

problems of service quality, cost, and fraud within Medicare’s home health program. Based on 

those studies, the Congress reformed the structure of the program, leading to savings of some 

forty plus billions of dollars and the abandonment of participation in the program by many high 

risk providers. These impacts were verified by independent reviews conducted by the 

Government Accountability Office. It is especially noteworthy that it was not a single impact 

study that led to these reforms, but rather by a body of work spanning several years. 
The Promise of Evidence-Based Policymaking 519

Page 6 of 7

Many other evaluation studies that lead to improved program impact and efficiency are 

documented on the websites of various Federal Offices of Inspectors General. They illustrate the 

value of using a body of work and mixed methods to assess both implementation and 

effectiveness of federal programs. 

Evaluation Capacity in Government Settings

AEA believes that all federal entities should have the ability and should be encouraged (and in 

some cases, required) to evaluate programs. However, each agency or department should 

develop structures and plans for their evaluation functions that are best suited to their missions, 

organizational structures, stakeholders, environments, timing of and need for evidence in 

decision making, and available resources. 

Because evaluation should serve as an essential core function in good governance, agencies 

should be required to apply the findings and conclusions of evaluations to program design, 

management, reform, expansion, or termination—ensuring that policy formulation will be more 

open, consultative, and evidence-informed. Agencies should, to the extent practical, conduct 

impact evaluations on pilot programs before attempting to expand or replicate them. 

A framework for the planning and conduct of evaluations should also include: 

a. A public evaluation policy statement

b. A sound procedure for establishing annual and multi-year evaluation agendas and 

timetables

c. Consultation with appropriate congressional committees, OMB, and other external 

program stakeholders on their information needs

d. A dissemination plan, preferably with public access

e. Resources needed to support evaluation, and 

f. Plans regarding how the findings and conclusions of evaluations shall be considered 

in subsequent program design, program management, and decisions regarding 

program reform, expansion, or termination.

The organizational structure of evaluation efforts is also important. Thus, it is vital to ensure an 

appropriate mixture of independence and collaboration between the evaluators and program 

offices with regard to evaluation design, conduct, and reporting. Consultation is needed to 

ensure relevance, but independence is needed to ensure impartiality. Depending on the unique 

organizational structure of each agency, an independent central evaluation office could be 

responsible for: developing and promoting program evaluation expertise throughout an agency; 

planning, conducting or procuring evaluation studies; and ensuring appropriate follow-up of 

evaluation findings and recommendations.

Adequate staffing of evaluation units and support for professional development is also necessary 

if the Commission’s work is to achieve the kind of benefits foreseen by Congress. To ensure that 

decision makers use the evidence produced in evaluation studies, agencies should invest in 

training those staff responsible for program design, implementation, and management regarding 

the proper conduct of evaluation and the use of findings in program decision making.
520 The Promise of Evidence-Based Policymaking

Page 7 of 7

Summary

1. Government decision makers must have appropriate credible evidence to make informed 

decisions regarding the structure and operations of federal agencies and policies, and to 

maximize their effectiveness and efficiency.

2. For key decision points within programs, federal entities (including Congress, executive 

branch agencies, and OMB), should identify important evaluation questions. Evaluators, 

in consultation with program officials, should select methods best suited to answer those 

questions. 

3. All federal entities should have the authority and resources to conduct evaluations, and 

should be encouraged (and in some cases, required) to evaluate various programs. 

However, each agency or department should develop structures and plans for those 

evaluation functions best suited to their mission, organizational structure, stakeholders, 

environment, and timing of and need for evidence in decision making.

4. Recognizing the importance of assessing program effectiveness, the opportunity to 

capitalize on early successes or to make mid-course corrections is also critical. So it is 

essential also to conduct ongoing formative evaluation throughout the program’s life 

cycle.

5. The proposed Evidence Clearinghouse should serve as a repository for the evidence 

contained in evaluation reports and as an archive for the collection, dissemination, and 

preservation of knowledge and lessons learned from evaluation studies.
The Promise of Evidence-Based Policymaking 521

Closing the gaps in opportunity and

achievement, pre-k through college.

1250 H STREET N.W., SUITE 700 WASHINGTON, D.C. 20005 T 202/293-1217 F 202/293 -2605 

WWW.EDTRUST.ORG

Commission on Evidence-Based Policymaking – Public Hearing 

Friday, October 21, 2016 

Comments submitted by: 

Tiffany Jones, Director of Higher Education Policy 

The Education Trust 

Abstract: The availability of high-quality, robust data systems is essential to helping the 

public understand how students are faring at particular institutions, identifying equity 

gaps, and better incentivizing improvement and success. Having better information on the 

college participation and outcomes of all students also helps ensure that students can make 

the best postsecondary decisions for themselves, with the billions of dollars that the federal 

government annually invests in student aid. We believe that the most efficient and effective 

way to gather complete and more honest data is through a (modified) unit record system, 

and we support both an effort to eliminate the current student unit record ban and an 

effort to expand and improve the current data collections, including, for example, data on 

part-time students and transfers, and making Pell status transparent in the collections. 

Oral and Written Statement: 

Thank you for the opportunity to speak. My name is Tiffany Jones and I am the Director of 

Higher Education Policy at The Education Trust. Ed Trust is committed to advancing 

educational opportunities for all students, but especially low-income students and students 

of color. We aim to advance equity in higher education by encouraging policymakers and 

the public to hold campuses accountable for student outcomes and supporting 

improvement at campuses committed to serving low-income students and students of 

color. 

Since the original Higher Education Act (HEA) was passed in 1965, the U.S. has made 

substantial progress in college access. College-going rates have climbed for students from 

all economic and racial groups. Yet despite this progress, low-income students today enroll 

in postsecondary education at rates lower than high-income students did in the mid-1970s. 

In every category of postsecondary education, low-income students and students of color 

are less likely than others to earn the degrees that they want and need, and far more likely 

to end up with debt and no degree. 

Before disaggregation of data was required in K-12, we knew anecdotally that schools were 

not educating all groups of students well. But we did not know just how significant the 

inequities were, and we didn’t know which schools were making progress and which 

weren’t.

H1_2016_08

October 21, 2016 Hearing

The Education Trust

Presenter: Tiffany Jones
522 The Promise of Evidence-Based Policymaking

THE EDUCATION TRUST

1250 H STREET N.W., SUITE 700 WASHINGTON, D.C. 20005 T 202/293-1217 F 202/293-2605 

WW.EDTRUST.ORG

That, unfortunately, is where we still are in higher education — especially in regard to lowincome students. We have some limited research on, for example, overall Pell graduation 

rates, but we don’t currently know which institutions are serving these students well and 

which aren’t. Pell graduation rate data will be incorporated into IPEDS in the coming years, 

but to date these data have not been included in annual IPEDS data collections. IPEDS also 

doesn’t include data on part-time students or students who don’t start in the fall or 

students who transfer in from another college.

If we have learned anything from past experience, it is this: that students who aren’t 

measured don’t count. If you want these students to count, and I know you do, you need to 

make the same shift to demanding better data that you have made in K-12. 

We believe that the most efficient and effective way to have complete and more honest data 

is through a (modified) unit record system. The current ban on a federal student unit 

record system makes it impossible for federal policymakers to get a comprehensive picture 

of how students are moving through postsecondary education and attaining degrees and 

certificates. The commission should recommend the overturn of the unit record ban and 

the ban on a federal database of WIOA data, so that we can have a nationwide, inclusive 

data set to show how people are moving through a variety of education pathways. 

Creating a student unit record system can begin by leveraging existing resources from the 

U.S. Department of Education (which houses the National Student Loan Data System and 

Integrated Postsecondary Education Data System), Social Security Administration, the 

Department of Defense and Department of Veterans Affairs, among others, to create a more 

complete picture of the higher education landscape. These sources provide valuable data 

on important subgroups of students who are often overlooked, including Pell Grant 

recipients, student loan borrowers, and student veterans. If linked, these data would 

produce valuable information about enrollment and completion rates, and post-college 

employment and earnings.

We also support an effort to expand and improve the current data collections, including, for 

example, data on part-time students and transfers, and making Pell status transparent in 

the collections.

Thank you for having this hearing and taking an important and critical step toward 

advancing the quality and availability of higher education data — specifically, a step that 

ensures better data that can be used to empower students, families, the public, advocates, 

and campuses as we aim to increase higher education equity and student success.

I look forward to answering any questions.
The Promise of Evidence-Based Policymaking 523

October 14, 2016

Dear Chair Katharine G. Abraham, Co-chair Ron Haskins, and members of the Commission:

On behalf of The National Campaign to Prevent Teen and Unplanned Pregnancy, thank you for 

the opportunity to submit a statement for the record for the Commission on Evidence-Based 

Policymaking (CEP) meeting to be held on October 21, 2016. The National Campaign, a 

research-based, nonpartisan, nonprofit organization was founded in 1996. We work to improve 

the lives and future prospects of children and families by ensuring that all children are born into 

families committed to and ready for the demanding task of raising the next generation by 

reducing unplanned pregnancy among teens and young adults. The National Campaign works

towards three ultimate outcomes: 

● Reduce the rate of teen pregnancy by 50% by 2026.

● Reduce the rate of unplanned pregnancy among women age 18-29 by 25% by 2026.

● Reduce the disparities in teen pregnancy and unplanned pregnancy rates among

racial/ethnic and socioeconomic groups by 50% by 2026.

Ensuring that young people have access to high quality, evidence-based teen pregnancy 

prevention education is one critical element in helping more young people delay pregnancy and 

parenting. 

Given our long-standing commitment to research, evidence and evaluation, we applaud the 

establishment of the Commission and appreciate the important issues it is tackling. In this 

statement, we offer feedback about several of the duties the Commission is tasked with, along 

with information about two tiered evidence-based programs—the Personal Responsibility 

Education Program (PREP) and the Teen Pregnancy Prevention (TPP) Program—which we hope 

will be helpful as the Commission goes about its important work.

The Commission poses several important questions with respect to data infrastructure and 

access, including a request for examples of best practices related to linking local, state and 

federal data. Not surprisingly, this type of endeavor raises many technological, ethical, and legal 

challenges, particularly as they relate to the balance between data access and privacy. One 

example that may be helpful to consider is the Longitudinal-Employer Household Dynamics 

(LEHD) program. We highlight this program for its ability to successfully navigate challenges 

associated with partnership formation, privacy protection, and data access while producing data 

that have greatly impacted policy.

Similarly, the Commission poses several questions related to the potential benefits and 

challenges of developing a clearinghouse for administrative and survey data. While The 

Campaign strongly supports greater access to administrative and survey data, and a 

H1_2016_10

October 21, 2016 Hearing

National Campaign to Prevent Teen and 

Unplanned Pregnancy

Presenter: Kelleen Kaye
524 The Promise of Evidence-Based Policymaking

clearinghouse would be beneficial in theory, we believe such an effort would likely fall short of 

its goals and would be difficult to maintain. It is particularly difficult to imagine a single 

clearinghouse that gathered data and evidence across all policy domains in a way that adequately 

captured the complexities of these data and the programs they reflect. Rather, we believe those 

resources would be better committed to helping agencies maintain and enhance the data access 

they already have in place. In our experience, as these agencies try to meet growing data 

collection costs with fixed or even diminishing budgets, the availability of policy relevant data 

has been shrinking in critical ways. Key questions have been cut from surveys and online access 

to data has been curtailed. This is particularly true as it pertains to tabulating results for states or 

localities. For example, one can no longer use the online vital statistics data to look at key policy 

questions like variation in Medicaid or WIC participation at the state level. It is also the case 

that some particularly rich data, such as the Medicaid Max files, are not available as de-identified 

files, thus making them difficult to obtain and underutilized. There are likely similar limitations 

in other policy domains as well. We believe that with relatively modest investments and vocal 

champions, data access could be greatly expanded.

We also would like to comment on the Commission’s interest in how data and findings from 

evaluations can best be used to improve policies and programs. We offer two examples of tiered 

evidence grant making from the US Department of Health and Human Services (HHS) that use 

evaluation results to continually improve those programs.

The TPP Program and PREP, like the Maternal, Infant and Early Childhood Home Visiting 

program, have been recognized as pioneering examples of tiered evidence-based policymaking,

i

and represent an important contribution to building a body of evidence of what works. They 

include high quality implementation, evaluation, innovation, and learning from results. The 

majority of funding from the TPP Program and PREP goes toward replicating program models 

that have been demonstrated to change behavior using well recognized high standards of 

evidence. A smaller portion of funding is reserved for research and demonstration projects to 

develop, replicate, refine, and test additional models and innovative strategies. This ensures that 

the menu of effective approaches to reducing teen pregnancy will continue to grow and be 

refined. 

TPP Program and PREP grantees can choose from a list of effective models that have been 

identified through HHS’ ongoing systematic review of the teen pregnancy prevention research 

literature. Since 2009, HHS has sponsored this review of the literature to help identify models 

with evidence of effectiveness in reducing teen pregnancy, sexually transmitted infections 

(STIs), and associated sexual risk behaviors. The review, conducted by Mathematica Policy

Research, looked at hundreds of evaluations and initially identified 28 models that met Tier 1 

criteria. That is, they must have been evaluated using a randomized controlled trial or quasiexperimental design, demonstrate changes in behavior (not just knowledge or behavioral intent), 

and results must be published in a peer-reviewed journal. The evidence review is updated 

periodically to capture the latest evaluation studies, and now includes 44 models.

ii The wide 

range of models on the HHS list of evidence-based programs gives grantees the flexibility to 

choose an effective approach that reflects their needs, population, and values, recognizing that 

what people in New York City may choose for high school age teens might be different from 

what people in Mississippi choose for middle school youth.
The Promise of Evidence-Based Policymaking 525

The TPP Program is a discretionary program administered by the Office of Adolescent Health 

(OAH) that was originally funded in FY 2010 at $110 million. It supported an initial cohort of 

102 grants for a five-year period. Funded at $101 million for FY 2016, the TPP Program 

currently supports 84 competitive grants to a broad range of organizations and agencies serving 

youth in 39 states and the Marshall Islands. The grantees focus intensely on communities with 

the highest teen birth rates and the most at-risk youth. These five-year grants were awarded in 

FY 2015 and are contingent on continued appropriations. As noted above, approximately 75% 

of the grant funds are used to replicate program models that have already been shown through 

careful evaluation to change teen behavior (Tier 1), and approximately 25% of the funds support 

research and demonstration projects to develop, replicate, refine, and test additional models and 

innovative strategies to prevent teen pregnancy (Tier 2). 

PREP, established in FY 2010, continues to be funded at $75 million in mandatory funding 

annually through FY 2017. Administered by the Administration on Children and Families 

(ACF), PREP supports states, communities, and tribes to educate adolescents on both abstinence 

and contraception to prevent pregnancy and STIs, and on other adulthood preparation topics such 

as healthy relationships, communication with parents, and financial literacy. PREP focuses on 

youth at greatest risk of teen pregnancy and geographic areas with high teen birth rates. For 

example, 34% of grantees targeted youth in foster care and 74% target youth in high need areas.iii 

Most of the PREP funding ($58 million) supports grants to states, territories, and tribes and 

emphasizes the use of evidence-based programs. Indeed, more than 95% of youth served by the 

state grants received one of the evidence-based programs from the HHS list referenced above.iv 

An additional $10 million supports competitive grants to public and private entities to develop, 

replicate, refine, and evaluate innovative strategies to reduce teen pregnancy and repeat 

pregnancies among youth up to age 21. These grants are subject to rigorous evaluation and 

reflect a “Tier 2” approach that supports innovation, fills gaps in existing programs for 

underserved populations, and expands knowledge about what works. 

Both programs have invested heavily in the highest standards of evaluation and learning, as well 

as in innovation. OAH funded 41 rigorous evaluations during the first round of TPP Program 

grants that ran from 2010-2014.v

 The recently released findings—90% of which were from 

randomized control trials—indicate that four of the Tier 1 programs were found effective in 

changing behavior in additional settings and new populations. Among the Tier 2 grantees, 8 

new, innovative models were found to be effective. Overall, these evaluations help build a body 

of evidence about where, when, and with whom specific models are most effective, and have 

expanded the menu of effective program models from which communities can choose. The 

results, along with implementation lessons, also help guide the second round of TPP Program 

grantees, and the many communities that look to the HHS list of evidence-based programs for 

guidance on what approaches will work best for them. Many of these findings and valuable 

implementation lessons were recently published in a special supplement of the American Journal 

of Public Health.

vi PREP grantees have also been subjected to rigorous evaluations through 

several different federally sponsored studies, and several studies have already been added to the 

HHS evidence review.

The commitment to evidence-based investments and innovation in the area of teen pregnancy 

prevention has been pivotal in changing the landscape. Before these two programs began, there 
526 The Promise of Evidence-Based Policymaking

were no federal investments dedicated to evidence-based teen pregnancy prevention programs; 

research in this area had primarily come from private investments, with few resources available 

to replicate or further evaluate the existing models.

The National Campaign also offers PREP and the TPP program as two examples where 

evaluation—specifically randomized control trials and quasi-experimental designs—

have been successfully incorporated into the program designs. These are two of the few 

government programs that use evidence and evaluation criteria throughout the grant life cycle.vii

In fact, only about $1 out of every $100 spent on federal programs is backed by any evidence 

that the money is being spent wisely. 

We believe rigorous evaluations have been successfully implemented for a few reasons. 

Importantly, the legislation for both programs specifies that some portion of funds should be 

used for evaluation. Program requirements also signify that evaluations are a priority. For 

instance, PREP grantees must participate in a federally-led evaluation, if chosen, and the “Tier 

2” innovation grantees are required to conduct their own rigorous evaluations, unless selected to 

be part of a federally-led evaluation. All TPP Program grantees are required to conduct some 

program evaluation, with a subset selected for rigorous impact evaluation.

viii In addition, there 

are several federally-led evaluation studies that include large, multi-state, rigorous evaluations 

conducted under contract to OAH.ix Besides rigorous evaluations, mandatory reporting of 

performance measures is another way that OAH and ACF ensure grant projects are making 

sufficient progress toward their stated missions and that there is continuous quality improvement.

Of course, providing support for grantees is another vital component to ensuring evaluations are 

successful. From review of initial evaluation designs to preparation of the final evaluation 

reports, TPP Program and PREP grantees received ongoing evaluation training and technical

assistance support to ensure rigorous methods and reporting.

x, xi In addition, it is essential to 

have a commitment to evaluation and learning from program leadership and adequate federal 

staff capacity to carry out that commitment. Leadership at OAH and at ACF demonstrated such

commitment, built staff capacity, and worked closely with evaluation experts at the ACF Office 

of Planning, Research and Evaluation (OPRE) and the Office of the Assistant Secretary for 

Planning and Evaluation (ASPE).

In closing, thank you for considering our input for the Commission for Evidence-Based 

Policymaking. If you have any questions or need additional information, please contact me at 

202-478-8512 or kkaye@thenc.org.

Sincerely,

Kelleen Kaye

Vice President, Research and 

Evaluation 

i Results for America. (2015). Invest in What Works Fact Sheet: Federal Evidence-Based Innovation Programs.

Washington, DC: Author. Retrieved October 12, 2016 from http://results4america.org/policy-hub/invest-works-

 
The Promise of Evidence-Based Policymaking 527

 

fact-sheet-federal-evidence-based-innovation-programs/. Also, Haskins, R. (2014). Show Me The Evidence. 

Washington, DC: Brookings Institution Press. 

ii Lugo-Gil, J., Lee, A., Vohra, D., Adamek, K., Lacoe, J., & Goesling, B. (2016). Updated findings from the HHS Teen 

Pregnancy Prevention Evidence Review: July 2014 through August 2015. Washington, DC: Mathematica Policy 

Research. Retrieved October 12, 2016 from 

http://tppevidencereview.aspe.hhs.gov/pdfs/Summary_of_findings_2015.pdf. iii Administration for Children and Families/Family and Youth Services Bureau. (2015). Adolescent Pregnancy 

Prevention Program Fact Sheet. Washington, DC. Author. Retrieved October 12, 2016 from 

www.acf.hhs.gov/programs/fysb/resource/app-fact-sheet. iv Administration for Children and Families/Family and Youth Services Bureau. (2015). Personal Responsibility 

Education Program: How States Planned and Implemented Adolescent Pregnancy Prevention Programs: State PREP 

Performance Measures of Structure, Cost, and Support for Implementation. Washington, DC: Author. Retrieved 

October 10, 2016 from www.acf.hhs.gov/sites/default/files/fysb/prep_pm_brief_20151216.pdf. v U.S. Department of Health and Human Services, Office of Adolescent Health. (2016). Results from the first round 

of TPP grantees. Retrieved October 12, 2016 from www.hhs.gov/ash/oah/oah-initiatives/tpp_program/cohorts-fy2010-2014.html. vi Morabia, A. (Ed.). (2016). Building the Evidence to Prevent Adolescent Pregnancy: Office of Adolescent Health 

Impact Studies (2010-2015) [Special issue]. American Journal of Public Health, 106(S1). Retrieved October 12, 2016 

from http://ajph.aphapublications.org/toc/ajph/106/S1. vii A recent GAO report includes TPP in its review of five tiered evidence grant programs, noting evidence is used 

throughout, including for assessing the evidence base and identifying evidence-based approaches, implementing 

evidence-based approaches with fidelity, conducting rigorous independent evaluations, and disseminating 

evaluation results.

viii U.S. Department of Health and Human Services/Office of Adolescent Health. (2016). TPP Evaluation and 

Performance Measurement. Retrieved October 10, 2016 from http://www.hhs.gov/ash/oah/oahinitiatives/evaluation/grantee-led-evaluation/grantees-2010-2014.html. ix http://www.hhs.gov/ash/oah/oah-initiatives/evaluation/federal-led-evaluation/index.html x http://www.hhs.gov/ash/oah/oah-initiatives/evaluation/ta.html. xi Goerlich Zief, S., Knab, J., & Cole, RP. (2016) A Framework for Evaluation Technical Assistance (2016). American 

Journal of Public Health, 106(S1): S22–S24. Retrieved October 10, 2016 from

http://ajph.aphapublications.org/doi/full/10.2105/AJPH.2016.303365.
528 The Promise of Evidence-Based Policymaking

- 1 -

Commission on Evidence-Based Policymaking (CEP)

First Public Hearing: Friday, October 21, 2016

Comments Submitted by 

Christine M Keller, PhD

Association of Public & Land-grant Universities (APLU)

Vice President, Research and Policy Analysis

ckeller@aplu.org

Abstract: The most significant barrier for evidence-based policymaking and decisions within higher education is 

the ban within the Higher Education Opportunity Act of 2008 that prohibits the Department of Education from 

collecting student-level data for postsecondary students. The lack of national student-level data prevents the 

Department from calculating comprehensive progress and completion outcomes for students as they move 

across different higher education institutions, especially as they cross state boundaries. It also prevents the 

linking of postsecondary data with federal data from other agencies that would allow better evidence of 

outcomes after college (e.g., earnings, employment) as well as outcomes for students in key federally-funded 

programs (e.g., Pell grants, veterans benefits). The result is that students and families are left in the dark as they 

make the critical decision of which college or university is the right fit; policymakers struggle to appropriately 

hold accountable institutions receiving taxpayer dollars; and institutions lack the information they need to assess 

their performance and improve.

Oral and Written Statement

Co-chairs Abraham and Haskins and distinguished members of the Commission, thank you for the opportunity to 

submit comments for your consideration during this hearing. My name is Christine Keller and I am the Vice 

President of Research and Policy Analysis at the Association of Public and Land-grant Universities (APLU). APLU 

is a research, policy, and advocacy organization dedicated to strengthening and advancing the work of public 

universities in North America.

A top priority for APLU is to foster the widespread use of data and evidence to support decision-making - on 

university campuses and as the basis for sound and effective federal policy. APLU believes the most significant

barrier for evidence-based policymaking and decisions within higher education is the ban that prohibits the 

Department of Education from collecting individual-level data for postsecondary students (Higher Education 

Opportunity Act of 2008). Lifting the ban in order to create a limited federal postsecondary student-level data

system would produce more accurate details of student enrollment patterns, progression, completion, and postcollegiate outcomes. Such a system would assist with national priorities such as providing students and their 

families with more complete and accurate information when selecting a college. And better ensure that 

policymakers can appropriately allocate public resources and evaluate program effectiveness.

A student-level postsecondary data system would address one of the most significant shortfalls of the current 

institution-level data collections within the Department of Education – the inability to accurately report the 

H1_2016_16

October 21, 2016 Hearing

Association of Public and Land Grant Universities

Presenter: Christine Keller
The Promise of Evidence-Based Policymaking 529

- 2 -

progress and completion of all students across multiple institutions and state boundaries. The current federal 

graduation rate only reports the completion of full-time students who start and finish at their first institution. 

Students who start their studies part-time or transfer institutions are not reported in the federal rate. Data from 

the National Student Clearinghouse demonstrate that these students comprise an increasingly large proportion 

of today’s students. Sixty-four (64) percent of bachelor’s degree recipients from public universities attend more 

than one institution before graduation and more than 60 percent of community college students attend parttime. Yet these students are missing from the federal graduation rate. 

The Department of Education, to its credit, has made multiple attempts to address these shortcoming within the 

constraints of an institutional level collection. However, the information provided remains inadequate for 

consumers and policymakers, adds reporting burden for institutions, and, the latest attempt was judged too 

unreliable by the Department of Education to release the data publicly after the first year of collection. A 

student-level data collection would simplify the creation of progress and outcomes measures as well as increase 

reliability and consistency of the metrics across institutions.

A second significant shortfall of the current postsecondary data is the inability to create linkages between 

postsecondary education data and other federal data systems. Linking with other federal data systems would 

harness the data already collected through other agencies to provide key information such as employment and 

earnings after college for all students. Progress and completion rates could be reliably and accurately 

calculated for student participating in federal programs such as Pell grants or GI Benefits. Combining information 

across federal agencies would streamline data collection, minimize duplicate reporting by institutions, and 

reduce the chance of errors in the resulting metrics.

Any student-level data system must include a robust set of protections and protocols to safeguard student data 

from unauthorized use or disclosures and to secure its collection and storage. Policies and procedures to 

protect data must be transparent and utilize evolving best practices for data security to address real and 

legitimate concerns about privacy and security, but privacy and security should not be used as an excuse for 

blocking transparency and access to more complete data.

Following are two specific examples from APLU members that illustrate why lifting the ban to create a system 

with more comprehensive and accurate data is a top priority for our association.

Example 1: Student Achievement Measure 

As I mentioned earlier, the current federal graduation rate only includes first-time, full-time students who start 

and finish at their first institution. To help fill the data gaps in the federal system, the higher education 

community created the Student Achievement Measure (www.studentachievementmeasure.org) or SAM. SAM 

is a voluntary initiative that allows institutions to report the progress and completion of full-time, part-time, and 

transfer students. Over 600 colleges and universities from all 50 states and the District of Columbia are 

participating in SAM and reporting the outcomes of 600,000 more students than the federal government’s 

measure.
530 The Promise of Evidence-Based Policymaking

- 3 -

One of the SAM participants is the University of North Texas. The federal graduation rate for UNT shows that 

just under half of first-time, full-time student who started in Fall 2009 graduated within 6 years. With only the 

information provided by the federal graduation rate, it appears that the other half of the students who started 

at the University of North Texas failed. However, by using the SAM methodology, UNT is able to show that 

another 13% of students graduated from another institution and another 14% are still enrolled pursuing their 

degree, for a total of 76% students who have graduated or are still enrolled.

SAM also reports the outcomes for the nearly 3,000 students who started at the University of North Texas as 

transfer students in Fall 2009 and are missing from the federal rate – 80% have graduated or are still enrolled

after 6 years. All totaled, SAM includes another 3,000 of the University of North Texas’s students who are not 

included in federal graduation rate.

SAM provides a powerful model of the type of information that would be available if the ban on collection of 

student-level data were lifted. However, SAM is not a substitute for a federal student-level data system. SAM is 

voluntary and does not include all postsecondary institutions. Nor is SAM the official data included in the U.S. 

Department of Education’s College Scorecard, the U.S. Department of Veterans Affairs’ college comparison tool, 

or other consumer information tools. A more complete federal solution is still needed.

Example #2: University of Texas System

A second example comes from the University of Texas System. The UT System has demonstrated the usefulness

of student-level data as part of their consumer information tool – seekUT (http://utsystem.edu/seekut/). By 

combining student-level data and the state workforce data, the UT System is able to present information such as 

the average cumulative student loan debt and median earnings at one, five, and ten years post-graduation for 

students graduating from specific programs at the UT institutions. 

However, as useful and powerful as having earnings data at the state level has been, there were key limitations 

that hindered the UT System’s ability to answer critical questions. Without a federal postsecondary data 

solution they were limited to data for those graduates that remained in Texas after graduation and unable to 

account for the earnings of graduates that move out of state. UT System administrators recognized that they 

needed national data across all states to evaluate and improve academic programs. And provide students with

more comprehensive employment and earnings information to show a realistic picture of earnings after 

graduation. This information would help students make more informed decisions about their choice of majors 

and appropriate amounts of debt. 

In fact, the UT System felt that national employment and earnings data would be of such high value that they 

recently finalized an agreement with the US Census Bureau to provide national post-college outcomes for UT 

graduates through a pilot research project. The Census-UT System collaboration is an important demonstration 

of how higher education and federal agencies can break down silos and work together. Imagine how valuable 

would it be if all colleges and universities and state systems could have access to similar information to support 

institutional evaluation and improvement and student decision-making – without each entity negotiating a 

separate agreement? A national student-level data system could help all institutions more readily reach that 

goal.
The Promise of Evidence-Based Policymaking 531

- 4 -

In sum, lifting the ban on the collection of student-level postsecondary data would provide access to more 

comprehensive, meaningful data and allow for the better alignment and integration with other federal data 

systems. The results would strengthen the federal government’s ability to provide essential information on 

higher education – for student and families to make more informed decisions about where to attend college; for 

policymakers to determine allocations of public resources and evaluate program effectiveness; and for college 

leaders to develop institutional policies and practices that support successful outcomes for all students. 

As the Commission continues to develop a strategy for increasing the availability, alignment, and use of high 

quality data to inform policy and decision-making, we encourage your consideration of the acute need for more 

accurate and complete postsecondary data for all users.

APLU is a research, policy, and advocacy organization dedicated to strengthening and advancing the work of public 

universities in the U.S., Canada, and Mexico. With a membership of 236 public research universities, land-grant institutions, 

state university systems, and affiliated organizations, APLU's agenda is built on the three pillars of increasing degree 

completion and academic success, advancing scientific research, and expanding engagement. Annually, its 194 U.S. 

member campuses enroll 3.9 million undergraduates and 1.2 million graduate students, award 1 million degrees, employ 1 

million faculty and staff, and conduct $40.2 billion in university-based research.
532 The Promise of Evidence-Based Policymaking

American Principles Project

1130 Connecticut AVE, NW, Suite 425, Washington, DC 20036

www.AmericanPrinciplesProject.org

Abstract:		We	urge	the	Commission	to	resist	calls	to	repeal	the	statutory	prohibition	on	the	

development,	implementation,	or	maintenance	of	a	federal	student	unit-record	system.		Such	a	

system	would	curtail	liberty	interests	of	the	individual,	would	invite the	collection	and	use	of	

ever-more	data,	and	would	fundamentally	alter	the	relationship	between	the	individual	and	

government	in	a	way	that	is	incompatible	with	our	constitutional republic.

Statement	by	Emmett	J.	McGroarty,	JD

Before	the	Commission	on	Evidence-Based	Policymaking

The	Commission	on	Evidence-Based	Policymaking	was	created	to	pursue	a	laudable	goal:	To	

improve	analysis	of	the	effectiveness	of	federal	programs.1 But	when	such	a	pursuit	is used	to	

justify	collecting,	conglomerating,	and	tracking massive	amounts	of	Americans’	personal	data,	

as	is	certainly	true	in	the	realm	of	education,	it’s	necessary	to	examine	the	dangers	and	the	

tradeoffs.	American	Principles	Project	(APP)	believes	that	such	activities	suppress	the	liberties	

of	the	people	and	pervert	the	relationship	between	the	people	and	government.		We	urge	the	

Commission	to	reject calls	to	establish a	federal	student	unit-record	system and to	engage	in	

such	Orwellian	activity.

Section	134	of	the	Higher	Education	Act	wisely	prohibits	the development,	implementation,	or	

maintenance	of	a	federal	student	unit-record	system	(one	that	would	allow	the	government	to	

collect	personally	identifiable	information	(PII)	on	individual	higher-education	students	and	link	

education	data	to	workforce	data).2 Recently,	though,	an	orchestrated	demand	for	repeal	of	

this	prohibition	has	been swelling.3

According	to	well-funded	organizations4 with	a	vested	interest	in	accessing	that	data	for	their	

own	purposes,	the	federal	government	suffers	from	data-deprivation.	Think	how	much	more	

efficiently	our	nation	could	operate,	and	how	much	more	the	government	could	help	people	

run	their	own	lives,	if	it	maintained	a	centralized	repository tracking	almost	every	conceivable	

data	point	about	every	citizen	– where	he	attended	school,	what	courses	he	took,	what	grades	

he	earned,	what	extracurricular	record	(good	or	bad)	he	compiled,	what	jobs	he	applied	for,	

what	jobs	he	got,	what	salary	he	made,	whether	he	was	promoted,	what	salary	he	earned	in	his	

new	position,	whether	he	lost	his	job and	why,	whether	he	joined	the	military,	what	sort	of	

1 David	B.	Muhlhausen,	“A	Commission	for	Evidence-Based	Policymaking:	A	Step	in	the	Right	Direction,”	

Heritage.org	(March	9,	2015),	available	at	http://www.heritage.org/research/reports/2015/03/a-commission-onevidence-based-policymaking-a-step-in-the-right-direction. 2 20	U.S.C.	sec.	1001	et	seq.,	available	at	 http://naicu.edu/docLib/20081030_HEA101-studentunit.pdf. 3 See	Libby	Nelson,	“Idea Whose	Time	Has	Come?”	Inside	Higher	Education	(May	13,	2013),	available	at

https://www.insidehighered.com/news/2013/05/13/political-winds-shift-federal-unit-records-database-howmuch. 4 Kelly	Field,	“Rescind	Ban	on	Federal	Unit-Record	System	to	Track	Students,	Report	Says,”	The	Chronicle	of	Higher	

Education	(March	11,	2014),	available	at http://www.chronicle.com/article/Rescind-Ban-on-Federal/145279/.

H1_2016_12

October 21, 2016 Hearing

American Principles Project

Presenter: Emmett McGroarty
The Promise of Evidence-Based Policymaking 533

2

military	record	he	established,	whether	he	was	arrested	and	for	what,	whether	he	went	to	jail,	

and	on	and	on	ad	infinitum.	

This	is	not	a	description	of	a	free	and	open	United	States	of	America.	This	is	a	description	of	a	

totalitarian	society	that	keeps	tabs	on	its	own	citizens	– for	their	own	good,	of	course. It’s	also	a	

description	of	what	would	inevitably	happen	with	the	establishment	of	a	student	unit-record	

system,	all	in	the	name	of	“better	consumer	information,”	“accountability,”	and	

“transparency.”

What’s wrong	with	a	federal	unit-tracking	system?	

First,	it	would	compile	students’	personally	identifiable	information	(PII)	without	their	consent	–

or	even	their	knowledge	that	their	data	is	being	collected	and	disclosed.	It’s	one	thing	to	collect	

data	from	a	student	who	voluntarily (which	of	course	presumes	actual	notice	of	the	program)

participates	in	a government	program	and	understands	that	participation will	expose	his	PII	to	

program	administrators; it’s	quite	another	to	forcibly	suck	every	individual into	a	datacollection	system	simply	because	he	enrolled	in	an	institution	of	higher	education. Telling	that	

student	that	he	must hand	over	his	personal	data to	promote	a	greater	good	as	defined	by	

bureaucrats	and	lobbyists – or	even	worse,	just	dragooning	him	without	telling	him	anything	–

is	simply	un-American.

Second,	the	purposes	of	the	proposed	system	would	be	so	open-ended	that	the	repository	is	

certain	to	be	expanded	over	time	to centralize	data	far	beyond	collegiate	and	employment	

data.	In	the	creative	bureaucratic	mind,	literally	everything	can	be	linked	to	education.	So	why	

stop	with	employment	data?	Why	not	see	how	one’s	education	affects	his	participation	in	the	

military?	Or	his	health?	Or	his	criminality?	Or	his	housing	patterns?	Or	the	number of	children	

he	has?	Or	whether	he	purchases	a	gun?	Or	his	political	activity?	Inquiring	bureaucrats	want	to	

know,	and	every	question	can	be	justified	by	citing	“better	consumer	information.”

And	will	this	dossier	created	on	every	citizen	become	permanent?	Presumably	so.	If	the	goal	of	

providing	maximum	consumer	information	is	to	be	achieved,	both	historical	and	current	data	–

constantly	updated	and	expanded	– must	be	compiled	and	preserved.	

Perhaps	this	expansion	won’t	happen.	Perhaps	the	federal	government,	in	stark	contrast	to	its

behavior	over	the	last	100	years,	will	stay	within	its	boundaries.	But	reality-based	Americans	

know	the	government	will	push	the	envelope	as	far	as	it	possibly	can,	as	it	always	does. And	

they	know	that	giving	that	government	access	to	such	a	treasure	trove	of	data	is	dangerous	to	

privacy	and	to	individual	liberty.

Third,	the	idea	that	this	massive	repository	of	PII	will	be	protected	against	unauthorized	access	

and	data	breaches	is quite	simply	delusional.	Less	than a	year	ago,	a	hearing	of	the	House	

Committee	on	Government	Oversight	and	Reform5 revealed	the	shocking	lack	of	student-data	

																																																												 5 U.S.	Department	of	Education:	Information	Security	Review”	(Nov.	17,	2015),	available	at

https://oversight.house.gov/hearing/u-s-department-of-education-information-security-review/.
534 The Promise of Evidence-Based Policymaking

3

security	throughout	the	U.S.	Department	of	Education	(USED).	The	problems	encompass	both	

lax	controls	over	the	people	allowed	access	to	sensitive	data,	as	well	as	outdated	technology	

and	inadequate	security	to	prevent	unauthorized	access.

USED’s	system	contains	over	139	million	Social	Security	numbers	(largely	through	its	office of	

Financial	Student	Aid),	along	with	sensitive	borrower	information	about	students	and	families	

contained	in	the	National	Student	Loan	Database.	The	findings	of	the	Office	of	the	Inspector	

General	(OIG)	and	the	General	Accounting	Office	were	disturbing:

n Of	the	97,000	account/users	with	access	to	this	information	(government	

employees	and	contractors),	fewer	than	20	percent	have	undergone	a	

background	check	to	receive	a	security	clearance.

n The	security	mechanisms	protecting	that	data	are	grossly	inadequate.	As	one	

OIG	witness	testified,	“During	our	testing	.	.	.	OIG	testers	were	able	to	gain	

full	access	to	the	Department’s	network	and	our	access	went	undetected	by	

Dell	[the	vendor]	and	the	Department’s	Office	of	the	Chief	Information	

Officer.”	

n USED	ignored repeated	warnings	from	OIG	that	its	information	systems	are	

vulnerable	to	security	threats.

That	the	federal	government	should	now	consider	ballooning	the	sensitive	data	contained	in	

these	insecure	systems	is	at	best	misguided	and	reckless.

Even	if	the	data	systems	were	secure,	the	Obama	administration’s	gutting	of	the	Family	

Educational	Rights	and	Privacy	Act	(FERPA)	means	that	government	education	officials	(federal,	

state,	and	local)	now	have	enormous	leeway	to	disclose	PII	on	individual	students	without	their	

consent.	Pursuant	to	the	recent	FERPA	regulations,	these	officials	may	share	private	PII	with	

other	government	agencies,	nonprofit	entities,	corporations,	researchers,	and	literally	anyone	

on	the	planet	as	long	as	the	disclosure	can	be	characterized	as	an	audit	or	evaluation	of	a	

(broadly	defined)	“education	program.”6

Will	the	new	conglomeration	of	student	data	be	fair	game	for	disclosure	under	these	

regulations?	The	danger	is	too	real	to	dismiss.

The	philosophical	problem	with	a	federal	student	unit-record	system	is	that	it	treats	free-born	

American	citizens	as	objects	of	research	and	study.	It	assumes	that	the	goal	of	benefitting	

others	in	society,	in	vague	and	theoretical	ways,	authorizes	the	powerful	federal	government	to	

collect	and	disseminate millions	of	data	points	on	individuals	– without	their	consent.	This	

fundamentally	changes	the	relationship	between	the	individual	and	government.		Collecting	

and	holding	massive	amounts	of	data	about	an	individual	has	an	intimidating	effect	on	the	

individual—even	if	the	data	is	never	used.		This	is	even	more	so	the	case	when	the	collector	has	

																																																												 6 Letter	from	American	Association	of	Collegiate	Registrars	and	Admissions	Officers	to	Regina	Miles	of	U.S.	Dept.	of	

Education	(May	23,	2011),	available	at https://aacrao-web.s3.amazonaws.com/migrated/FERPA-AACRAOComments.sflb.ashx_520501ad842930.77008351.ashx.
The Promise of Evidence-Based Policymaking 535

4

the	force	of	the	law	behind	it.		Our republic	rests	on	the	idea	that	the	citizen	will	direct	

government.		That	cannot	happen	where	government	sits	in	a	position	of	intimidation	over	the	

individual.		

Submitted	by:

Emmett	J.	McGroarty,	Esq.

Senior	Fellow

American	Principles	Project
536 The Promise of Evidence-Based Policymaking

1

September 9, 2016

Dr. Katharine G. Abraham, Chair

Mr. Ron Haskins, Co-Chair

Evidence-Based Policymaking Commission

U.S. Census Bureau

4600 Silver Hill Road

Suitland, MD 20746

Dear Chairwoman Abraham and Co-Chairman Haskins,

We are writing to encourage you to consider including the attached policy recommendations in 

your final report to Congress and the Administration. 

We believe that the Commission can help invest taxpayer dollars in what works by assisting

policymakers at all levels of government in:

 Building evidence about the practices, policies and programs that will achieve the most

effective and efficient results so that policymakers can make better decisions;

 Investing limited taxpayer dollars in practices, policies and programs that use data, evidence

and evaluation to demonstrate they work; and

 Directing funds away from practices, policies, and programs that consistently fail to achieve

measurable outcomes.

Although the Evidence-Based Policymaking Commission Act of 2016 directs the Commission to 

study and report on several important topics including data privacy and data sharing, our attached 

policy proposals focus on the provision that directs the Commission to “make recommendations on 

how best to incorporate outcomes measurement, institutionalize randomized controlled trials, and 

rigorous impact analysis into program design.” 

We thank you in advance for your consideration of our recommendations.

Sincerely,

Actionable Intelligence for Social Policy (AISP)

America Forward

Center for Employment Opportunities

Center for Research and Reform in Education, Johns Hopkins University

KIPP

REDF

Results for America

Sorenson Center for Impact

Success for All Foundation

Sunlight Foundation

cc: Members of the Evidence-Based Policymaking Commission

H1_2016_01

October 21, 2016 Hearing

Results for America

Presenter: David Medina
The Promise of Evidence-Based Policymaking 537

2

INVEST IN WHAT WORKS COALITION RECOMMENDATIONS

Data Collection

 Federal Data Infrastructure: The Commission should consider recommending that Congress

and the Administration provide sufficient funding to help the U.S. Census Bureau accelerate

the process of acquiring key administrative data-sets from local, state, and federal agencies,

and strengthen its infrastructure for processing, standardizing, linking, and making data

available to other government agencies and independent researchers via data use agreements

with strong privacy protections. As part of this effort, the Census Bureau should develop an

inventory of data-sets at the local, state, and federal levels and make this inventory accessible

to government agencies and independent researchers.

 Federal Data Inventories: The Commission should consider recommending that Congress

and the Administration codify into law what is already required by the May 2013 Executive

Order by passing the OPEN Government Data Act. This legislation would mandate that every

federal agency create an enterprise data inventory of all data sets held by the agency and

make these lists public in machine-readable formats with strong privacy protections.

 Federal Data Information Technology: The Commission should consider recommending that

Congress and the Administration provide sufficient funding to allow every federal agency to

update and modernize its IT infrastructure that supports data collection, analysis, sharing, and

usage so that data can be appropriately structured, protected, analyzed and disclosed in line

with the updated information policy of the United States. A 2016 report by the U.S. General

Accountability Office highlighted the urgent need for the U.S. government to modernize its

aging legacy systems.

 Workforce Data: The Commission should recommend that Congress and the Administration

allow the linking of workforce datasets (including but not limited to state and federal

unemployment insurance and new hires data sets) to improve the effectiveness and efficiency

of publicly-supported workforce development programs, as long as the linking is consistent

with strong privacy protections. For example, many states cannot determine the impact of their

job training programs without the ability to link their participant information with information

about wage earnings across multiple states where participants obtain employment.

 State Education and Workforce Data Systems: The Commission should recommend that

Congress and the Administration support the enhancement of the existing State Longitudinal

Data Systems (SLDS) program administered by the U.S. Department of Education, which

helps states integrate education and workforce data, and the proposed expansion of the

Workforce Data Quality Initiative that would help build state and local capacity to track

employment and educational outcomes of Workforce Innovation and Opportunity Act program

participants, including those with disabilities, and provide information about job success rates

and training programs.

 Federal Education Data Identifiers: The Commission should consider recommending that

Congress and the Administration direct federal agencies to standardize the way they collect
538 The Promise of Evidence-Based Policymaking

3

and share student-level identifiers (e.g., de-identified but encrypted) so that researchers can 

more effectively evaluate publicly-supported education and workforce development programs. 

This information should be housed in one federal agency in order to promote appropriate 

sharing and usage of this standardized data.

 Federal Programmatic Data: The Commission should consider recommending that Congress 

and the Administration authorize every federal agency to set aside 1% of their program funds 

for program evaluations that generate programmatic outcomes data that can help make federal 

programs more effective and efficient. 

Data Analysis

 Data Leadership and Infrastructure: The Commission should consider recommending that

Congress and the Administration direct every federal agency to have a senior staff member

(i.e., Chief Evaluation Officer or equivalent position) with the authority, staff, and budget to

develop important programmatic data through the evaluation of its major programs and to use 

this programmatic data and available administrative data to inform the agency’s policies and 

improve its programs.

Data Sharing

 Local and State Data Systems: The Commission should consider recommending that 

Congress and the Administration clarify that local and state agencies can invest federal 

program funds in strengthening their data infrastructures for processing, standardizing, linking, 

and making data available to other government agencies and independent researchers via 

data use agreements with strong privacy protections.

 Federal Education Data Infrastructure: The Commission should consider recommending

that Congress and the Administration strengthen the U.S. Department of Education’s (ED’s)

data infrastructure, including the hiring and training of key analytic staff, to manage the 

collection, quality, release, and analysis of education data with strong privacy protections and 

the support the proposed InformED initiative that would pull together ED’s diverse array of data 

and studies on a particular topic, and allow open data access to help unlock answers to 

pressing education questions and needs.

Data Usage

 “What Works” Clearinghouses: The Commission should consider recommending that 

Congress and the Administration direct every federal agency to develop a “What Works” 

clearinghouse or evidence exchange with the purpose of making evaluation reports available 

to the public.

 Performance Management/Continuous Improvement: The Commission should consider 

recommending that Congress and the Administration direct every federal agency to develop 

and operate a performance management system with clear and prioritized outcome-focused 
The Promise of Evidence-Based Policymaking 539

4

goals and aligned program objectives and that frequently collects, analyzes, and uses

administrative and programmatic outcomes data to improve outcomes, return on investment, 

and other dimensions of performance.

 Federal Grant Programs: The Commission should consider recommending that Congress 

and the Administration direct every federal agency to use evidence of effectiveness, including 

impact analysis and other outcomes measurements based on high-quality administrative and 

programmatic outcomes data, when allocating funds from its 5 largest competitive and noncompetitive grant programs.

 Evaluation and Research: The Commission should consider recommending that Congress 

and the Administration direct every federal agency to have an evaluation policy, evaluation 

plan, and research/learning agenda which ensures that the agency has an intentional 

approach to the collection, analysis, sharing, and usage of administrative and programmatic 

data and publicly release the findings of all completed evaluations to improve the effectiveness 

and efficiency of federal programs.

 Repurpose for Results: The Commission should consider recommending that Congress and 

the Administration direct every federal agency to use its administrative and programmatic data 

to determine when to shift funds away from practices, policies, and programs which 

consistently fail to achieve desired outcomes and toward evidence-based, results-driven 

solutions.
540 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 541
542 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 543
544 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 545
546 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 547
548 The Promise of Evidence-Based Policymaking

1 

Commission on Evidence-Based Policymaking—Public Hearing 

Friday, October 21, 2016 

Comments submitted by: 

 Amanda Janice Roberson, Research Analyst 

Institute for Higher Education Policy 

Abstract: The current postsecondary data infrastructure is fragmented and incapable of answering a 

number of important questions about how our students fare in the higher education system. Key 

stakeholders, including policymakers, institutions, researchers, and the students themselves, need better 

information about college access, progression, completion, and post-college outcomes. Given the federal 

government’s substantial investment in postsecondary education, it is imperative that existing data – at 

the institutional, state, and federal levels – are leveraged to answer these critical questions. By fostering 

these data linkages and removing existing legal barriers, the Commission can create a system where data 

drive efforts to increase postsecondary success and close equity gaps. 

Oral and Written Statement: Chairman Abraham, Co-Chair Haskins, and commissioners: thank you for the 

opportunity to address the Commission on Evidence-Based Policymaking on the importance of a cohesive 

postsecondary data infrastructure and its impact on evidence-based policymaking. 

My name is Amanda Janice Roberson and I am a research analyst with the Institute for Higher Education 

Policy. IHEP is a nonpartisan, nonprofit organization committed to promoting access to and success in 

higher education for all students, with a focus on students who have been underserved by our 

postsecondary system. Based here in Washington, D.C., we believe that all people, regardless of 

background or circumstance, have the opportunity to reach their full potential by participating and 

succeeding in higher education. 

In support of this goal, IHEP leads the Postsecondary Data Collaborative (PostsecData), a partnership 

between more than 35 organizations committed to the responsible use of high-quality postsecondary 

data to improve student outcomes. PostsecData partners represent a broad range of constituents, 

including groups that represent students, postsecondary institutions, the workforce community, and state 

and federal policy influencers and researchers. 

Since 2014, IHEP has spearheaded research on which data should be collected, how metrics should be 

defined, and through which mechanisms our currently disconnected, duplicative, and incomplete data 

systems can work together to create a cohesive postsecondary data ecosystem. IHEP supports the mission 

of this Commission to analyze and make recommendations for streamlining federal data and data 

systems. We suggest the following actions to improve the landscape of postsecondary data for use by 

policymakers, students and families, institutions, and researchers. 

 Promote best practices in privacy and security for interconnected data systems.

Recommendations by the Commission for data linkages should address the importance of privacy,

security, and confidentiality. As institutional practices and changing laws at the state level have

led to confusion around when it is permissible to share or link data, policies and procedures from

the Commission should be transparent, consult with data security experts to implement fieldrecognized best practices, and ensure that all publicly reported, aggregate data are stripped of

personally identifiable information.

H1_2016_05

October 21, 2016 Hearing

Institute for Higher Education Policy

Presenter: Amanda Janice Roberson
The Promise of Evidence-Based Policymaking 549

2

 Leverage existing data to decrease burden, streamline reporting, and answer critical questions.

Data from sources like the U.S. Department of Education (which houses the National Student Loan 

Data System [NSLDS] and Integrated Postsecondary Education Data System [IPEDS]), Social 

Security Administration (SSA), the Department of Defense (DoD) and Department of Veterans 

Affairs (VA), among others, should be linked and leveraged to create a more complete picture of 

the higher education landscape. These sources provide valuable data on important subgroups of 

students who are often overlooked, including Pell grant recipients, student loan borrowers, and 

student veterans. If linked, these data would produce valuable information about enrollment and 

completion rates, and post-college employment and earnings. The Commission should consider 

ways to increase capacity and funding available to streamline processes and link data, as these 

are the primary challenges for state and local level data linkages. 

 Expand access to wage and labor market information for postsecondary outcomes. In an era of 

scarce resources, the value of a postsecondary degree has never been greater, and post-college 

outcomes are increasingly important to policymakers and students. Now, data and metrics on 

employment and earnings are limited to voluntary initiatives, like College Measures, state 

dashboards, and the College Scorecard, revamped in September 2015. The Commission should 

explore datasets, like the Census Bureau’s Longitudinal Employer Household Dynamics (LEHD) 

program or the National Directory of New Hires (NDNH), which both utilize state Unemployment 

Insurance (UI) wage records, or the Social Security Administration and Internal Revenue Service 

tax records, to understand the return on personal investment of students and families and federal 

investment in higher education.

 Align definitions and metrics across federal laws. Establishing common definitions for data 

metrics across federal laws like the Higher Education Act, the Workforce Innovation and 

Opportunity Act, and the Perkins Career and Technical Education Act could reduce administrative 

burden and create comparable outcomes across federal programs. Common and consistent 

metric definitions in the postsecondary ecosystem would make it much easier to link data

between local, state, and federal sources and allow for accurate comparisons. 

 Recommend that Congress overturn the ban on a federal student-level data system. The 

statutory ban on a federal student unit record system stifles the ability of policymakers to answer 

questions about our postsecondary system, limits the information available to consumers, and 

imposes unnecessary burden onto institutions. The Commission should recommend to overturn 

the ban and direct the U.S. Department of Education to engage with the higher education 

community to design and implement a student-level data system. This system would create a 

nationwide, inclusive data set that shows how students move through higher education and their 

post-college outcomes. This system would allow for disaggregation by key student characteristics, 

like Pell Grant receipt, race/ethnicity, and others, and illuminate evidence for future policymaking 

around closing equity gaps and the federal investment in higher education and postsecondary 

programming. Given the sensitive nature of record level data, the Commission should also 

recommend rigorous data privacy and security policies to govern this system.

Thank you for your time and the opportunity to provide a statement to the Commission. I look forward to 

answering any questions.
550 The Promise of Evidence-Based Policymaking

Commission for Evidence-Based Policymaking Public Hearing: Remarks 

from the American Statistical Association’s Scientific and Public Affairs 

Advisory Committee 

October 21, 2016 

My name is Clyde Tucker and I am a member and former chair of the American Statistical 

Association’s Scientific and Public Affairs Advisory Committee. The American Statistical 

Association was founded in 1839 and is the oldest continuously operating scientific society 

in the United States. With more than 19,000 members working in academia, government, 

and industry, the ASA works to promote the practice and profession of statistics, the science 

of learning from data, and measuring, understanding, and communicating its uncertainty. 

We believe that statistics and the ability to interpret statistical evidence are integral to the 

success of evidence-based policy making. 

The ASA lauds Speaker Ryan and Senator Murray for their actions and efforts to bring 

evidence-based policymaking into the limelight. The ASA heartily supports the Commission 

for Evidence-Based Policymaking and looks forward to the Commission’s efforts to improve 

the science surrounding evidence-based policymaking. We appreciate that three 

Commissioners are members of the ASA, two of which are former heads of federal statistical 

agencies. In my brief comments today, I will focus on five issues: (i) the stature and 

autonomy of the federal statistical system; (ii) data sharing that leads to data 

synchronization; (iii) concerns related to privacy and confidentiality that may present 

barriers to the release of data needed for evidence-based policy making; (iv) nurturing 

evidence-based policymaking capacity across the federal government; and (v) statistical 

evidence. 

 To ensure that the statistical analysis used to support evidence-based policy making is both 

impartial and accurate, the integrity of the research process must be maintained. In 

particular, the ASA believes that the federal statistical agencies will play a vital role in 

evidence-based policymaking, and ASA is committed to supporting the historical autonomy 

of these agencies in order to ensure the integrity of their work. In a 2015 letter to Congress, 

twenty former statistical agency heads wrote, 

As the foundation for policy making and policy administration, objective and 

credible statistical data are vital to our democracy, economy, governance, and wellbeing. All sides of a policy debate should be able to look to the statistical data as 

objective and high quality. Any perception that the data have been influenced by a 

partisan perspective undermines the policy making and its administration. The 

independence of a federal statistical agency is a critical element in an agency 

producing objective and credible statistical data... Statistical agencies should have 

complete control over data collection, analysis, and publication. Such autonomy 

H1_2016_16 American Statistical Association Letter

October 21, 2016 Hearing

Presenter: Clyde Tucker
The Promise of Evidence-Based Policymaking 551

should include control over an agency’s planning, budget, press releases, and 

information technology. 

In the past, this independence for some agencies has been protected from direct political 

interference by requiring that the heads of statistical agencies be appointed by the 

President and confirmed by the Senate. Currently, however, Senate confirmation is no 

longer required to appoint the director of the Bureau of Justice Statistics or the 

commissioner of the National Center for Education Statistics (NCES). There is also a Housepassed bill pending in the Senate that would remove presidential appointment of the NCES 

Commissioner. 

The ASA also supports data linkage and collaboration between the federal statistical 

agencies. Indeed, greater information sharing across agencies would enhance the research 

supporting evidence-based policymaking. One example of this type of data sharing is data 

synchronization. For example, ASA advocates, as a follow up to the Confidential Information 

Protection and Statistical Efficiency Act (CIPSEA), providing the Bureau of Labor Statistics 

and the Bureau of Economic Analysis the same access to the Internal Revenue Service’s 

business information that the Census Bureau currently has. Supported not just by ASA but 

also by a wide variety of other stakeholders, this carefully crafted proposal costs no money, 

but would result in substantial improvements to the quality of our nation’s official economic 

statistical data, ultimately benefitting policymakers, US businesses, and many other 

Americans. For more information on this, I provide links to three resources at the end of my 

written comments that I understand you will have access to. 

Although privacy and confidentiality are important concerns with respect to the release of 

data for research purposes, barriers created to ensure privacy and confidentiality could 

limit analysis critical for evidence-based policymaking. In particular, researcher’s access to 

data often is delayed as a result of the need to undergo reviews by multiple Institutional 

Review Boards (IRBs). This process can impede the ability to respond quickly to the needs 

of policymakers. I have provided a link to a new National Academies report that notes the 

delays caused by the multiple IRB approvals sometimes needed for a single study and the 

serious problems this creates for timely policymaking. 

While perhaps beyond the charge of this committee, ASA also encourages more resources 

for federal agencies to develop their internal capacities for evidence-based policymaking. 

Although the statistical agencies have statistical expertise needed for evidence-based 

policymaking, a number of agencies do not. More could be done to increase this analytical 

capacity across the government. Given the constrained budgets of the federal government, 

we understand adding personnel with appropriate expertise isn’t widely feasible. However, 

we strongly encourage alternative solutions, such as guidance documents and professional 

development on this topic. In fact, staff at the statistical agencies might be involved in these 

endeavors. We would also support greater agency collaboration through reduced barriers 

and greater access to data for trusted and vetted users in ways that ensure confidentiality 

protection. 

Let me close with comments on statistical evidence for the wider evidence-based 

policymaking community. We encourage the use of modern statistical and data science 

methods in program evaluations—methods such as Bayesian modeling, decision analysis, 

and big data techniques. To put our comments in context, we recently saw statistical 

language in pending legislation that seemed artificially restrictive. Specifically, we were 

concerned that the language might limit analysis to significance testing or p-values alone. 
552 The Promise of Evidence-Based Policymaking

The ASA Scientific and Public Affairs Advisory Committee recently released a statement 

with guidance on statistical language in legislation, 

http://ww2.amstat.org/misc/SPAAC_GuidanceStatisticalEvidence.pdf. We look forward to 

continued conversation on this issue. 

To reiterate, we fully support the Commission on Evidence-Based Policymaking and look 

forward to engaging the statistical community in your efforts. Thank you for your time. 

Links for 2015 letter to Congress from twenty former statistical agency heads: 

• http://ww2.amstat.org/misc/FormerAgencyHeadLetter.pdf 

• http://community.amstat.org/blogs/steve-pierson/2015/01/29/former-federalstatistical-agency-heads-urge-stronger-nces-as-senate-panel-advances-bill-toweaken-it

References on Data Synchronization 

• http://www.aei.org/publication/data-asymmetry-public-policy/ 

• http://www.copafs.org/UserFiles/file/FederalBusinessRegistryLetterSenatewithAt

tach.pdf 

• http://magazine.amstat.org/blog/2011/11/01/data-synchronizationscipolicy/

 Reference to the National Academies report on multiple IRB approvals 

• https://www.nap.edu/catalog/23670/strengthening-data-science-methods-fordepartment-of-defense-personnel-and-readiness-missions 
The Promise of Evidence-Based Policymaking 553

PUBLIC SUBMISSION 

As of: 10/14/16 4:20 PM 

Received: October 14, 2016 

Status: Pending_Post

Tracking No. 1k0-8sgj-iuzd

Comments Due: November 14, 2016

Submission Type: Web 

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking 

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking 

Document: USBC-2016-0003-DRAFT-0010

Comment on FR Doc # 2016-22002 

Submitter Information 

Name: Christel Swasey 

Address:

Pleasant Grove - Utah, 84062 

Email: 212christel@gmail.com 

General Comment 

Dear Commission on Evidence-Based Policymaking, 

I love the American concept of voter-based, Constitution-based, elected representative-based, 

policymaking. It's why I live in America. In contrast to voter-based policymaking there is 

evidence-based policymaking, which I don't love because it implies that one entity's "evidence" 

trumps individuals' consent to new policy changes. 

Former Secretary of Agriculture Ezra Taft Benson said something about education that also 

applies to educational data and policymaking: 

"The best way to prevent a political faction or any small group of people from capturing control 

of the nation's educational system is to keep it decentralized into small local units... This may not 

be as efficient as one giant super educational system (although bigness is not necessarily 

efficient, either) but it is far more safe. There are other factors, too, in favor of local and 

independent school systems. First, they are more responsive to the needs and wishes of the 

parents and the community. The door to the school superintendent's office is usually open to any 

parent [or teacher]... But the average citizen would be hard pressed to obtain more than a form 
554 The Promise of Evidence-Based Policymaking

letter reply from the national Commissioner of Education in Washington, D.C." 

Local control, and consent of the governed, are two foundational principles in our great nation. 

Because the CEP is not an elected body, it does not hold authority to collect, or to recommend 

collection, of student-level evidence, or of any evidence, without written consent; and, for the 

same reasons, neither does the Department of Education. 

Because the fifty, federally-designed, evidence-collecting, State Longitudinal Database Systems 

never received any consent from the governed in any state to collect data on individuals (as the 

systems were put into place not by authority, but by grant money) it follows that the idea of 

having CEP study the possible removal of barriers to federal access of those databases, is an 

egregious overstep that even exceeds the overstep of the State Longitudinal Database Systems. 

Because federal FERPA regulations altered the original protective intent of FERPA, and 

removed the mandate that governments must get parental (or adult student) consent for any use 

of student level data, it seems that the idea of having CEP study and possible influence removal 

of additional "barriers" to federal use of data, is another egregious overstep. 

As a licensed teacher in the State of Utah; as co-founder of Utahns Against Common Core 

(UACC); as a mother of children who currently attend public, private and home schools; as 

acting president of the Utah Chapter of United States Parents Involved in Education (USPIE); as 

a patriot who believes in "consent of the governed" and in the principles of the U.S. Constitution; 

and, as a current tenth grade English teacher, I feel that my letter represents the will of many who 

stand opposed to the study of the removal of protective barriers on student-level data, which the 

CEP's website has outlined it will do. 

I urge this commission to use its power to strengthen local control of data, meaning parental and 

teacher stewardship over student data, instead of aiming to broaden the numbers of people with 

access to personally identifiable student information to include government agencies and/or 

educational sales/research corporations such as Pearson, Microsoft, or the American Institutes 

for Research. 

To remove barriers to federal access of student-level data only makes sense to a socialist who 

agrees with the Marc Tucker/Hillary Clinton 1998 vision of a cradle-to-grave nanny state with 

"large scale data management systems" that dismiss privacy as a relic in subservience to modern 

government. It does not make sense to those who cherish local control. 

It is clear that there is a strong debate about local control and about consent of the governed, 

concerning data and concerning education in general. NCEE Chair Mark Tucker articulated one 

side of the debate when he said: "the United States will have to largely abandon the beloved 

emblem of American education: local control. If the goal is to greatly increase the capacity and 

authority of the state education agencies, much of the new authority will have to come at the 

expense of local control." 
The Promise of Evidence-Based Policymaking 555

Does that statement match the philosophical stand of this commission? I hope not. Local control 

means individual control of one's own life. How would an individual control his or her own 

destiny if "large scale data management systems" in a cradle-to-grave system, like the one that 

Tucker and Clinton envisioned, override the right to personal privacy and local control? It is not 

possible.

I urge this commission to use any influence that it has to promote safekeeping of unit-record data 

at the parental and teacher level, where that authority rightly belongs. 

Sincerely,

Christel Swasey 
556 The Promise of Evidence-Based Policymaking

United States Parents Involved in Education (USPIE) is a nonprofit, nonpartisan, nationwide 

coalition of state leaders focused on restoring local control of education by eradicating federal 

intrusion. State leaders from around the country fed up with being ignored on education policy 

have joined forces to abolish the US Education Department and put an end to all federal 

education mandates.

USPIE endeavors to inform Americans of the trillions of dollars wasted on federal education in 

the last 35 years with nothing to show for it but stagnant, and declining test scores. It is the goal 

of USPIE to return American's education to its proper local roots and restore parental authority 

over their children's education.

USPIE’s STOP FED ED campaign is led by parents, taxpayers and educators committed to 

ending the U.S. Department of Education. The fight against Common Core has exposed the 

failures of those trying to force a federally-based one size fits all curriculum on states and local 

school districts. 

So we ask, “why even have a federal department?” Because it’s not about children. It’s about 

control. Control through federal dollars. And it’s big business. It’s about pushing an agenda. And 

it’s about ending something that had worked for years and replacing it with something no one 

even understands. 

For half a century now this experiment with federal control of local public schools has gone on 

and it’s failed. Let’s stop treating our children like rats in some social engineering laboratory and 

start treating children like children again. The first step is ending the Department of Education 

and that’s what STOP FED ED is all about. 

H1_2016_14

October 21, 2016 Hearing

United States Parents Involved in Education

Presenter: Erin Knowles

Note: Erin Knowles was unable to be present, but submitted written 

remarks.
The Promise of Evidence-Based Policymaking 557

Breaking Good

The effective use of good data is vital to achieving the results about 

which people care most.

The CEP environment includes a global consensus, bi-partisan support, 

high ambitions, new resources and recent successes in the use of data, 

analysis and evidence to improve public program performance.

The Commission agenda on data access and protection can help 

broaden and deepen the use of data and analysis for that purpose.

The value of this agenda can be increased and sustained to 

simultaneously take two big steps: 

◦ Providing information and other resources to help speed the 

improvement process; and 

◦ Linking programs performance to the high-level results about which 

people care most.

From Data to Action: 

Achieving Results People Care About Most

PRESENTATION TO THE

COMMISSION ON EVIDENCE-BASED POLICYMAKING

QUENTIN WILSON

WASHINGTON, DC

OCTOBER 21, 2016

H1_2016_11

October 21, 2016 Hearing

Presenter: Quentin Wilson (independent researcher)
558 The Promise of Evidence-Based Policymaking

You’ve Got a Friend:

Emerging Research, Advocacy and 

Support Resources

Governing Institute Living Cities

Results for America USC Civic Data

Pew-McArthur Results First Initiative and Clearinghouse

Bloomberg Philanthropies What Works Cities

Harvard Data-Smart City Solutions

Hewlett Foundation Effective Philanthropy Group 

Evidence-Based Policymaking Collaborative

Postsecondary Education Data Collaborative

Bill & Melinda Gates Foundation

Institute for Higher Education Policy

Global Consensus and 

Advancement: Using Data to 

Achieve Better Results

◦ Tim Berners-Lee, Founder of the World Wide Web: The 

Next Web

◦UN Guidelines on Open Government Data

◦ White House Executive Order and 9/28/16 Open Data 

Innovation Summit

◦ Commission on Evidence-Based Policymaking

◦ Results Washington

◦ Maryland StateStat

◦ Baltimore OutcomeStat
The Promise of Evidence-Based Policymaking 559

College Completion: 

Career & College Clubs

Caring about Data:

Making a Difference on Results that Matters Most

Key Result Area Data-Based Initiatives

Priority Outcomes

Education

College Completion Career & College Clubs

Child Development First Five/Parents as Teachers 

Public Safety

Violent Crime Gang intervention

Worker Safety Highway construction process

Health

Substance Abuse SAMHSA E-B Program Registry 

Prevention
560 The Promise of Evidence-Based Policymaking

Economic Prosperity: Bridgespan’s Billion 

Dollar Bet on Pathways to Careers

Data-based research-driven proposal

Potential investments in six result areas “emphasized the need to 

better track and manage data.”

Results measured by Return on Investment (ROI) model. (10X)

Data Making a Difference - II

Key Result Area Data-Based Initiatives

Priority Outcomes

Economic Prosperity

Strong Neighborhoods Blight Reduction

Housing 

Employment Opportunity Pathways to Careers

Effective Government Santa Monica FD

Missouri Dept. of Revenue

Energy and the Environment
The Promise of Evidence-Based Policymaking 561

Setting the Stage for Faster, Better Results: 

What’s Needed Now

Create a repository of consensus outcomes and measures, 

and resources for improvement.

Encourage leadership and collaboration among agencies, levels 

of governments and sectors.

Build a culture of support for the use of data and outcomes 

that matter, not punishment and misguided “accountability.”

Develop more accessible, commonly accepted and usable data.

◦ “80% of data lives in forms and places our teams and 

systems can’t easily process.”- IBM Watson Team

◦ World Wide Web founder Tim Berners-Lee: Linked Data

State Initiatives:

California Performance Excellence Resources

◦California Data Collaborative

◦California Evidence-Based Clearinghouse

◦California Open Justice initiative

◦Open Data tech firms 

◦Local government leaders
562 The Promise of Evidence-Based Policymaking

For More Information:

Contact:

Quentin Wilson

1230 Rosecrans Avenue

Suite 300

Manhattan Beach, CA 90266

310-800-4715

Next Steps: Using Data and Analysis for 

Faster, Better Results People Care About

Prioritize organizational goals, outcomes and measures from the 

inventory of options.

◦ Utilize data and information about the current, projected and 

comparative performance on these outcomes.

◦ Involve the public, elected officials, researchers, advocates and 

practitioners in the prioritization process.

Speed effective implementation with access and use of resources.

◦ Policy and program research and advocacy

◦ Promising practices

◦ Technical assistance and training

Integrate performance assessment and review with strategic 

planning.
The Promise of Evidence-Based Policymaking 563

1

Commission on Evidenced-Based Policymaking

Submission by 

Carrie Wofford, President, Veterans Education Success &

Mark Schneider, Vice President, American Institutes for Research

The Need to Share and Link Federal Data on the Post-9/11 GI Bill

We urge the Evidence Based Policymaking Commission to take steps to end the siloization of 

federal data and ensure that federal agencies share data. 

The Post-9/11 GI Bill provides an important example of the detrimental impact of siloed federal 

data.

Why it Matters:

Historians and economists frequently credit the original GI Bill with helping to build America’s 

Middle Class following WWII. After the 9/11 terror attacks on American soil, a new GI Bill was 

enacted to provide the current generation of veterans with their ticket to the American dream, 

helping 1.5 million veterans at a cost to taxpayers of $61 billion since August of 2009. The goal 

is to assist veterans in the transition to a successful civilian career.

In order to best serve veterans and the federal taxpayer investment, government officials, 

higher education leaders, and policymakers need to know how the GI Bill is succeeding and 

“what works” under the GI Bill. 

Such an assessment is impossible because federal data regarding student outcomes and 

occupational outcomes for veterans remain siloed across several federal agencies.

At this time, nobody in America knows the student veteran graduation rate, debt rate, default 

rate, or whether the Post-9/11 GI Bill is succeeding in supporting veterans’ transition to civilian 

occupational and income success. Little is known about veterans’ educational attainment, debt 

or default, because the U.S. Education Department (ED) does not know which students are 

veterans and the U.S. Department of Veterans Affairs (VA) does not track student outcomes. 

Although ED formerly tracked veterans in its database, it stopped doing so in 2009, when it 

introduced a skip-pattern in its FAFSA form, such that most veterans never seeing the question 

about military service. Unfortunately, this change at ED was launched the exact same year 

(2009) that the Post-9/11 GI Bill went into effect. A 2014 U.S. Senate Committee reported that 

for-profit colleges dominate the Post-9/11 GI Bill and generally provide poor outcomes for 

students overall, while costing taxpayers twice as much per veteran as public colleges and 

universities, but student outcomes specific to veterans was unknown. Indeed, reporters asked 

the Senate Committee what the student veteran graduation rate was, and this question was 

H1_2016_13

October 21, 2016 Hearing

Veterans Education Success

American Institutes for Resarch

Presenters: Carrie Wofford and Mark Schneider
564 The Promise of Evidence-Based Policymaking

2

impossible to answer because the data needed to answer this basic question is held in separate

agencies.1

In addition, occupation and income data from IRS and Census is not linked to either VA or ED 

data. 

What Data-Linking or Sharing Could Achieve:

If data from VA and ED were shared or cross-walked and if occupational and income data from 

IRS or Census were added, the resulting combined data set would provide solid answers on the 

student veteran graduation rate, debt and default rates, job placement rate, and income. The 

shared data would:

Help veterans make an informed college choice as they decide where to use their GI Bill by 

arming them with data about veterans’ probability of graduation and likely earnings trajectory 

from each college (and each program). VA’s GI Bill College Comparison Tool is currently the 

best source of information for veterans choosing a college, but it does not provide veteranspecific data. If federal data were shared, VA’s GI Bill College Comparison Tool could be as 

robust as ED’s College Scorecard. The College Scorecard was possible only because federal 

agencies shared data. The College Scorecard provides students with important data-points 

about student graduation (specifically, the graduation rate within 150% of expected time to 

completion for first-time, full-time students) and salary after attending (specifically, the median 

earnings of former students who received federal financial aid 10 years after entering the 

school). It should be noted that much of ED’s data is limited to first-time, full-time students, 

which is an outdated limitation and one the Commission should urge ED to change. 

Help Congress and policymakers improve regulation by providing data on Post-9/11 GI Bill 

students and their debt and occupational rates. Currently, Congress and policymakers have no 

data on the student outcomes, nor on the occupational and income outcomes, of educational 

paths under the Post-9/11 GI Bill. Understanding a return on investment would assist 

 1 Private efforts have tried to determine the student veteran graduation rate, but it is impossible 

without VA and ED actually sharing data. The largest private effort (by Student Veterans of America, 

known as the 2014 “Million Records Project” and its 2016 update “NVEST”) undertook to match VA data 

on Post-9/11 GI Bill use against data from the National Student Clearinghouse to try to determine the 

student veteran graduation rate for a subset of GI Bill users. But the Clearinghouse data is limited to 

degree-granting schools (and covers most, but not all veterans at degree-granting schools), so it 

provides only a limited answer. Most notably, Clearinghouse does not track students at certificate, nondegree programs, nor vocational/technical programs (both of which are covered under the GI Bill). VA 

estimates that nearly half of the GI Bill is spent at non-degree schools, meaning that the Clearinghouse 

data and the “Million Records Project” are missing half of GI Bill students. Specifically, VA reports that 

among Post-9/11 GI Bill students starting their education in 2015: some 54,000 Post-9/11 GI Bill 

students were starting non-degree college programs, and 30,000 started vocational and technical 

programs, while 87,275 started undergraduate degree programs and 19,222 started graduate degree 

programs. 
The Promise of Evidence-Based Policymaking 565

3

policymakers greatly, but requires federal data matches. Data-linkage would enable 

exploration of the effectiveness of the Post-9/11 GI Bill in ensuring a successful civilian career 

for the current generation of veterans. Adding in data from the U.S. Defense Department’s test 

scores on service members’ abilities and skills could serve as “controls” in determining the 

impact of the GI Bill. 

Executive Order 13607 (April 27, 2012) required VA and ED to share data to determine veteran 

student outcome measures, but, four and a half years later, the agencies still have not 

completed an MOU to do so.

Detailed Questions That Could Be Answered if Federal Data Were Combined:

1. Participation in the Post-9/11 GI Bill: 

a. Nearly half of the Post-9/11 GI Bill goes unclaimed. What are the demographics

of veterans who skip the GI Bill? How are they faring? What are their 

occupations and incomes, and how do those compare to their occupations and 

incomes prior to military service? Are they reliant on public assistance? Did they 

skip the GI Bill because they already had a college degree, or because they had a 

strong career before military service? By historic contrast, only 20% of eligible 

veterans skipped the original GI Bill following WWII, and such eligible nonparticipants were often older (over the age of 35).

2 In terms of occupation prior 

to military service, most veterans who utilized the original GI Bill after WWII had

“little or no pre-war experience in jobs requiring extensive skill or training,”

while those who skipped it (eligible non-participants) had been “working in jobs 

of a fairly high level” before the war.

3 Are these trends true today under the 

Post-9/11 GI Bill? Veterans organizations report anecdotal evidence that many 

veterans skipping today’s GI Bill do need higher education but feel intimidated

by the college search process and fear their academic skills are not up to par.

 2 The President’s Commission on Veterans’ Pensions, Veterans Benefits in the United States(April 1954), 

Omar Bradley, Chairman (known popularly as the “Bradley Commission”), available at

http://www.cnas.org/sites/default/files/Bradley_Commission_Report1956.pdf (page 251, 259). The 

Bradley Commission found that 83-86% of veterans in the two youngest groups (under 20 or between 

20-24 years of age) used the GI Bill. 

3 Bradley Commission, available at

http://www.cnas.org/sites/default/files/Bradley_Commission_Report1956.pdf (page 261). The Bradley 

Commission also reported that that many of the younger veterans “had held no regular job before 

entering service. In general, those who had held jobs were in relatively unskilled occupations.” (page 

258). In contrast, only 30% of pre-war managers and proprietors used the original GI Bill, as did only 

39% of pre-war full-time employees. (page 261)
566 The Promise of Evidence-Based Policymaking

4

b. Participation under the Post-9/11 GI Bill is increasing each year. VA data shows 

that nearly 1 million eligible individuals participated in 2011, a 15% increase over 

FY 2010 and a 71% increase over FY 2009. Do the data show better student 

outcomes (such as persistence and completion) and civilian employment success 

among more recent GI Bill students as compared to 2009 and 2010? Are student 

loan debt and default levels rising? Has the return on taxpayer investment 

changed over time?

c. Non-Veteran vs. Veteran Participants. Veterans can choose to give some or all 

of their Post-9/11 GI Bill to their spouse or dependents, and 18% of GI Bill 

students are spouses or dependents. Do non-veteran GI Bill students enjoy 

better outcomes than veterans? Do they have better persistence and 

completion rates in college, perhaps indicating that veteran students need more 

support on campus? How do non-veteran Post-9/11 GI Bill students’ loan debt 

and default rates compare to their veteran counterparts? Do non-veteran 

participants have higher incomes and better correlation between their 

occupation and field of degree? 

d. Outcomes by Demographics. Are there differences by age, race, gender, 

ethnicity, or residential region in outcomes for Post-9/11 GI Bill users? For 

example, some VA analyses suggest that women veterans are more likely to use 

the Post-9/11 GI Bill than men. Is their persistence better? Are their outcomes 

better?

2. GI Bill Effectiveness. How effective is the Post-9/11 GI Bill in ensuring a successful 

civilian career for the current generation of veterans?

a. Income & Public Assistance. Do veterans have higher incomes and less 

dependence on public assistance programs after using the Post-9/11 GI Bill than 

before they used it? By point of comparison, the original GI Bill, following WWII, 

reportedly reduced reliance on unemployment assistance from 20% of veterans.

4 

What is known about veterans who rely on public assistance programs? How 

many have a college degree, whether through the Post-9/11 GI Bill or otherwise? 

What was their field of study in college? What is their occupation? 

b. Degree and Occupation. Do veterans have different occupations after using the 

Post-9/11 GI Bill than before they used it? Among Post-9/11 GI Bill users, which 

occupations, degrees, and fields of study result in the highest income and least 

reliance on public assistance? Do the degrees and occupations correlate, or is 

occupational success (at least in some occupations) independent of degree 

 4 Bradley Commission, available at

http://www.cnas.org/sites/default/files/Bradley_Commission_Report1956.pdf (pate 251).
The Promise of Evidence-Based Policymaking 567

5

obtained? What about licensed occupations? Does the GI Bill help veterans 

obtain work in licensed occupations? (Approximately 20% of a sample of 300 

degree programs approved by VA for Post-9/11 GI Bill in licensed occupations 

are improperly accredited and fail to leave the graduates eligible to work.5 What 

percent of veterans working in licensed occupations used the Post-9/11 GI Bill in 

that field of study? Are public colleges more likely to have the right 

accreditation for graduates to be eligible to work in licensed fields?) Which 

degrees and institution types produce graduates employed in which fields, with 

which licenses?

c. Debt and Default: What are the student loan debt and default rates for both 

veteran and non-veteran beneficiaries using the Post-9/11 GI Bill? Are debt and 

default rates higher for veterans than non-veteran students using the Post-9/11 

GI Bill? Are there correlations in student loan debt and default by degree and 

field of study obtained? By college type (online vs. brick and mortar) and 

institutional sector? By occupation and income?

d. Montgomery GI Bill vs. Post-9/11 GI Bill: Does the Post-9/11 GI Bill deliver 

better return on investment, including student and occupational outcomes, than 

its immediate predecessor, the Montgomery GI Bill? 

3. Which College Experience Yields the Best Return on Investment under the Post-9/11 

GI Bill?

a. Brick and Mortar vs. Online. Among Post-9/11 GI Bill users, what type of college 

is most effective? Many experts assume brick and mortar colleges are more 

effective than online education, but online education is very popular with 

military students. What are the educational outcomes and civilian employment 

success for online student veterans? 

b. Institutional Sector. Among Post-9/11 GI Bill users, which sector of college is 

associated with the best student outcomes and civilian employment success: 

public, non-profit, or for-profit? What is the return on investment from each 

sector? 

 5 See Veterans Education Success, “The GI Bill Pays for Degrees That Do Not Lead to a Job,” (Sept. 2015) 

available at

http://static1.squarespace.com/static/556718b2e4b02e470eb1b186/t/56ba65f8356fb040f04fb56a/145

5056377419/GI+Bill+Dollars+do+not+pay+for+accredited+programs.pdf
568 The Promise of Evidence-Based Policymaking

6

i. Several government6 and private reports7 have concluded that for-profit 

colleges do not serve students well. Do for-profit colleges have lower 

persistence and completion rates among Post-9/11 GI Bill students than 

other sectors? Do for-profit colleges produce higher student loan debt 

levels and default rates? What are the civilian employment results for 

Post-9/11 GI Bill graduates? Also, what is the return on investment? 

ii. Experts often claim public community colleges produce better results for 

veterans, and significant public and private funds are invested in 

community college programs for veterans. What are the student 

outcomes and civilian employment success of veterans at community 

colleges compared to other sectors? What is the return on investment 

for veterans at community colleges?

c. Specific Colleges: Because much of the Post-9/11 GI Bill expenditures are 

concentrated in a handful of colleges, it is possible to derive robust samples at 

those colleges, enabling an assessment of how well the colleges are serving 

veterans and the taxpayer investment.

d. Type of Degree: Are there differences in the return on investment among GI Bill 

students attending college degree programs (e.g., Bachelor of Science in 

Engineering) vs. non-degree, certificate programs (e.g., Certificate in Radiology 

Technology) vs. vocational/technical programs (e.g., truck driving) – taking into 

account the cost of the program and the resulting civilian success? (Defense 

Department scores of service members’ skills and abilities could serve as a 

control when measuring post-education occupational success.) Do demographic 

or household income correlate with type of degree chosen?

e. Field of Study: Is it possible to determine program-specific outcomes, such as 

the return on investment of a B.S. in Engineering vs. a B.S. in Nursing, factoring in 

demographic and pre-education differences?

We hope the Evidence Based Policymaking Commission can take strong steps to end the 

siloization of federal data and ensure that federal agencies share data to benefit the public as 

well as policymakers.

 6 See, e.g., U.S. Senate Committee 2012 report, “For-Profit Education: The Failure to Safeguard the 

Federal Investment and Ensure Student Success,” available at

http://www.help.senate.gov/imo/media/for_profit_report/PartI.pdf

7 See, e.g., National Bureau of Economic Research, “Evaluating Student Outcomes at For-Profit Colleges,” 

(June 2012), available at http://www.nber.org/papers/w18201 
The Promise of Evidence-Based Policymaking 569

Critical Issues for the Commission on Evidence-Based Policymaking

Statement from Workforce Data Quality Campaign

The Commission on Evidence-Based Policymaking’s examination of federal administrative and survey 

data provides an exceptional opportunity to address the management and use of data for measuring 

postsecondary education and workforce outcomes. The Commission’s findings and impact could lead 

to more inclusive, aligned and market-relevant data systems to help educators, students, employers, 

workers and policymakers all make more informed decisions.

Workforce Data Quality Campaign (WDQC) is a non-profit initiative that promotes inclusive, aligned, 

and market-relevant education and workforce data. We engage hundreds of national experts, state 

officials, and workforce development advocates, encouraging the use of data to ensure that all of our 

nation’s education and training programs are preparing students and workers to succeed in a 

changing economy. Given our mission, we are excited about the promise of the Commission’s work, 

and are pleased to have the opportunity to share our recommendations. 

Data collected and held by the government could help to answer a range of important postsecondary 

education and workforce questions, such as: 

• Which skilled positions are employers having a difficult time filling, and what institutions

might they look to for recruitment?

• Are recent college graduates finding jobs and earning good wages?

• How much do students borrow, and can they repay these loans?

• What types of education and training pathways are helping people succeed in careers?

• Which program models are most effective at helping target populations (e.g. ex-offenders,

veterans, low-income individuals) gain skills and find stable employment?

• How can workers know which short-term credentials would likely raise their earning

potential?

• What job search strategies are most effective, and for whom?

In some instances, surveys have been able to answer those questions over a limited time frame, but 

with great effort and expense. A growing number of state longitudinal data systems are linking 

administrative records to answer questions. However, geographical coverage is limited, so they 

cannot answer questions about students who leave the state, or compare outcomes across states. 

The federal government already collects data through numerous administrative sources, in addition 

to conducting regular surveys. With improved coordination, these data could be systematically 

shared and linked to answer those and other critical questions for generations to come. 

H1_2016_07 

October 21, 2016 Hearing

Workforce Data Quality Campaign

Presenter: Rachel Zinn
570 The Promise of Evidence-Based Policymaking

Issues for Action

Strides have been made in recent years, but much information remains separated between agencies 

because of technological, cultural, and legal barriers. WDQC encourages the Commission to 

recommend the following actions in its final report: 

1.) Expand access to wage information

The Commission should examine how the federal government can build on existing data collections 

and facilitate the linking of employment and earnings data across higher education and training 

programs. 

Students and workers want to know which education and training programs will help improve 

their chances of having successful careers. Researchers need access to more detailed and 

comparable data on programs to analyze which pathways are working for students and workers. 

Agencies at all levels of government want to know the short- and long-term employment 

outcomes of those they have served. 

Potential relevant data sources include the National Directory of New Hires and the Census 

Bureau, which contain Unemployment Insurance wage records submitted by states. The Internal 

Revenue Service and the Social Security Administration have individual tax records. In limited 

instances, agencies have found ways to use these data to show employment outcomes for 

programs, but the federal government needs to create efficient, strategic processes for managing 

employment data. The Commission should consider how a federal clearinghouse could

streamline employment data collections and rationalize processes for access, while protecting 

privacy and enhancing security. 

2.) Improve information on postsecondary progress and outcomes

The Commission should examine ways ─ such as establishing a federal student record system ─ to 

measure postsecondary student progress and more effectively and efficiently answer important 

consumer and policy research questions. 

Stakeholders do not have access to comparable information at the program level, and in many 

cases, only students receiving financial aid; attending first-time, full-time; or those pursuing two 

or four year degrees are counted. These limitations exclude non-degree credentials that are 

growing in number and importance, as well as the transfer, part-time, and adult students who 

now outnumber ‘‘traditional’’ postsecondary students. The Department of Education’s College 

Scorecard and planned changes in the Integrated Postsecondary Education Data System (IPEDS) 

reflect progress toward providing and linking data for analysis and consumer-friendly interfaces, 

but the information remains scattered and incomplete. Overturning legal prohibitions on federal

collection of data on individuals involved in postsecondary education and training programs, and 

implementing a national student record system, would allow for building a more complete 

picture with lower administrative burden.
The Promise of Evidence-Based Policymaking 571

3.) Provide more accessible labor market information 

The Commission should include in its examination how labor market information (LMI) might be 

better integrated to provide more comprehensive and clear information. 

Having access to LMI (e.g. occupational projections) may strengthen a worker’s ability to make 

decisions about employment and training, and help to improve the alignment of education and 

training programs with labor market demand. The Bureau of Labor Statistics, Census, and other 

statistical agencies could more effectively collaborate and incorporate additional information 

from federal programs to enhance data about employment, worker characteristics, and the job 

market. If the Commission examines LMI, it should coordinate with the newly established 

Workforce Information Advisory Council (WIAC), which reports to the Secretary of Labor.

4.) Harmonize definitions and metrics across federal laws

The Commission should explore how the federal government could implement similar definitions and 

metrics to streamline reporting and improve opportunities for data linkages between programs. 

State agencies and service providers often face the burden of having to report on program results 

using different definitions and measures, which increases staff time and cost. Using common 

definitions and metrics from the Workforce Innovation and Opportunity Act (WIOA) for other 

programs as appropriate, such as those operating under the Perkins Career and Technical 

Education Act, would reduce administrative burden, make outcomes more comparable, and 

facilitate coordination across human capital programs. 

5.) Clarify privacy and security protections

The Commission should account for best practices in privacy and security as it conducts its review.

Institutional practices and changing laws at various levels of government have often created 

confusion around what is possible and led to blockages in sharing and linking data, even when 

doing so is legal. Policies and procedures recommended by the Commission should be 

transparent, utilize evolving best practices for data security, and ensure that publicly available 

data are aggregated or otherwise stripped of all information that could be used to identify 

particular individuals or employers. As noted by presenters in an earlier Commission meeting on 

privacy and security issues, the Commission should respect varying viewpoints on privacy rights. 

In order to strike an appropriate balance between privacy concerns and optimal use of data to 

improve publically-funded programs, the Commission should ensure that federal policy accounts 

for emerging technologies that can help protect sensitive information. 

As the Commission conducts its examination, we encourage the elevation and promotion of highquality data sources that can be used to inform human capital development policy. We hope the 

Commission will focus on maximizing the use of data to enhance decision-making and continually 

improve education and training services that allow all Americans to contribute to a 21st Century 

economy. 
572 The Promise of Evidence-Based Policymaking

Rhode Island Innovative Policy Lab at Brown University

Science Library - Level 10, 201 Thayer Street, Providence, RI 02912

Page 1 of 2

Commission on Evidence-Based Policymaking Written-Statement:

On behalf of the Rhode Island Innovative Policy Lab (RIIPL), we are pleased to provide 

stakeholder input on the Commission on Evidenced-Based Policymaking.

The Rhode Island Innovative Policy Lab (RIIPL) at Brown University is a new collaboration 

between researchers at Brown University and the Office of the Governor in Rhode Island. It is a 

prototype innovative policy lab, centered around synergetic partnerships with State and Federal 

government partners to develop new policies and measure their successes. Together, RIIPL and

the Rhode Island Government are using data and rigorous research to support the development of 

more effective public policy. The goals are to increase equity of opportunity for all Rhode Islanders 

through more effective policy, and provide rigorous evidence on what works for policy makers 

throughout the country.

In this statement, we wish to voice support for the work of the Commission on Evidence-Based 

Policymaking (CEP) which is part of an important movement to create a new approach to policymaking. The commission is charged with making recommendations for how data and results of 

research can be used to evaluate and better design government programs. At present, there are 

thousands of policies at the federal, state and local levels, yet little is known about their 

effectiveness. Efforts like RIIPL's and the CEP's are important steps in measuring what works, and 

implementing it to improve lives through increased policy effectiveness.

Researchers at RIIPL combine data science, behavioral science, economics and public policy in 

our approach to creating evidence-based reforms. We foster collaboration between scientists and 

practitioners to come up with real, workable and testable solutions. RIIPL also integrates education 

at Brown University into the research and dissemination process to change the way policy is 

approached in the future by training the next generation of leaders as part of the collaborative 

research and policy innovation process. At RIIPL, communication and collaboration between 

policy makers and researchers facilitate research, discovery and learning which feed directly into 

actionable policy innovation - the goal of evidence-based policy.

Through research-government collaboration, we are discovering what works in five key areas:

• Closing the Achievement Gap by increasing equity of opportunity so that all children, 

regardless of background, have the ability to reach their potential.

Rhode Island Innovative Policy Lab (RIIPL)

Brown University

December 18, 2016

Chicago Public Input–Hearing Testimony
The Promise of Evidence-Based Policymaking 573

Rhode Island Innovative Policy Lab at Brown University

Science Library - Level 10, 201 Thayer Street, Providence, RI 02912

Page 2 of 2

• Developing Smart Social Programs with higher impact per dollar so that we can fight 

poverty effectively.

• Restoring Community through improved criminal justice programs which recognize that 

society gains most when all people are productively engaged.

• Designing Regulation that Works for business and the environment by creating transparent 

regulations which protect the environment and grow the economy.

• Learning from the Past by using prior policy changes to inform future decisions because 

we can learn from our failures and successes alike.

Together, RIIPL and the Rhode Island Government are putting data and science to work to increase 

equity and opportunity in society, making Rhode Island an incubator for Smart Policy and a leader 

in data-driven policy innovation. We support national efforts such as the CEP's work to foster and 

support government-research relationships and research-driven policy.

Justine Hastings

Director of RIIPL

Professor of Economics and International and Public Affairs

Brown University

Lint Barrage

Faculty Research Associate at RIIPL

Assistant Professor of Economics and Environmental Studies

Brown University

Eric Chyn

Faculty Research Associate at RIIPL

Assistant Professor of Economics

University of Virginia

John Friedman 

Faculty Research Associate at RIIPL

Associate Professor of Economics and International and Public Affairs

Brown University

Emily Oster

Faculty Research Associate at RIIPL

Professor of Economics and International and Public Affairs

Brown University
574 The Promise of Evidence-Based Policymaking

TesƟmony as prepared for

Commission on Evidence‐Based Policymaking

Midwest Public Hearing

5 January 2017

I am tesƟfying today on behalf of Raise Your Hand and the Parent CoaliƟon for Student Privacy.

Raise Your Hand is a Chicago‐based grassroots parent group that advocates for high‐quality public

educaƟon for all students in Chicago and Illinois. We are a primarily volunteer‐run organizaƟon. We

formed in 2010 to work on the issue of inadequate and inequitable funding of Chicago Public Schools

and have expanded to work on several other educaƟon policy areas, including student privacy.

The Parent CoaliƟon for Student Privacy formed in 2014 and is a naƟonal coaliƟon of parents and

advocates defending the protecƟon of student data privacy.

The Parent CoaliƟon for Student Privacy wrote a leƩer in November to the Commission opposing the

creaƟon of a centralized, federal clearinghouse of the personally‐idenƟfiable informaƟon of all students,

commonly referred to as a student unit‐record system or naƟonal database. This leƩer was signed by five

other groups as well, including the American Civil LiberƟes Union and the Network for Public EducaƟon.

The risk that a federal database of student unit records would pose to student privacy is immense;

including the very real possibility of breach, malicious aƩack, or the use of this informaƟon for purposes

not iniƟally intended. In the years since a federal student unit‐record system was banned by the Higher

EducaƟon Act in 2008, the reasons against creaƟng it have only become more compelling.

In the past few years, much highly personal data held by federal agencies has been hacked, including the

release of the records of the Office of Personnel Management involving more than 22 million individuals,

not only federal employees and contractors but also their families and friends.

The US Department of EducaƟon in parƟcular has been found to have especially weak security standards

in its collecƟon and storage of student data, and received a grade of D on the government scorecard

created to assess how well federal agencies were implemenƟng data security measures this past May.

In addiƟon, preK‐12 student data currently collected by state departments of educaƟon that would

potenƟally be shared with the federal government include upwards of 700 highly sensiƟve personal data

elements, including students’ immigraƟon status, disabiliƟes, disciplinary records, and homelessness

data.

As privacy advocates in the UK recently discovered, the personal informaƟon in a similar naƟonal student

database that the government promised would be used only for research purposes has been secretly
The Promise of Evidence-Based Policymaking 575

requested by the police and by the Home Office, in part to idenƟfy and locate undocumented children

and their families.

We are also very concerned about recent revelaƟons of the widespread surveillance on ordinary ciƟzens

by the federal government, and the way in which a naƟonal student data system would be used to

expand the tracking of students from preschool into high school, college, the workforce and beyond. A

federal data clearinghouse of student informaƟon could effecƟvely create life‐long dossiers on nearly

every individual in the naƟon.

The rapid adopƟon of the use of digital technology in preschool through high school has been

accompanied by a similarly rapid increase in the generaƟon of data Ɵed to individual students and

collected and stored by third‐party organizaƟons. Dozens of so�ware and hardware vendors have

products in use in the Chicago Public Schools alone. The regulaƟon and protecƟon of the data generated

by such programs remains an open quesƟon. As this data is almost always Ɵed to a student’s

personally‐idenƟfiable informaƟon, it too could be connected to and at risk from a naƟonal

student‐record system.

There have been at least two major, known data breaches in Chicago Public Schools in the last two years.

In May of 2015, 4000 students had their names, addresses, phone numbers, disability status and other

personal informaƟon inadvertently shared with vendors responding to a district RFP. This past fall, a CPS 1

employee was fired for unauthorized sharing of personal informaƟon of more than 28,000 students with

a charter management organizaƟon who then used the data for markeƟng. Student data is already 2

highly vulnerable even without a federal data clearinghouse.

In light of all these concerns, we urge you to strongly oppose the creaƟon of any centralized federal data

system holding students’ personally idenƟfiable informaƟon and to support the conƟnuaƟon of the ban

in the report you provide to Congress.

Although I am now a full‐Ɵme advocate for public educaƟon, my professional training was as a research

scienƟst in a quanƟtaƟve field, computaƟonal linguisƟcs. As a scienƟst, I certainly agree that high‐quality

data collecƟon is a crucial ingredient in the research process. I also know that the ethical consideraƟons

in research using data from human subjects are paramount and that well‐supported conclusions can be

drawn from staƟsƟcal samples derived from carefully designed experiments.

We do not need to track every student from preschool to the workforce in order to create an efficient

and successful public educaƟon system, and given the risks and costs of doing so, we should not do it.

1 “Data breach triggers sharing of personal info for 4,000 students“ Catalyst Chicago. May 19, 2015.

hƩp://catalyst‐chicago.org/2015/05/data‐breach‐triggers‐sharing‐of‐personal‐info‐for‐4000‐students/ 2 “3 Noble charter staffers OK’d using CPS student data to recruit” Chicago Sun‐Times. Dec. 23, 2016.

hƩp://chicago.sunƟmes.com/news/3‐noble‐charter‐staffers‐oked‐using‐cps‐student‐data‐to‐recruit/
576 The Promise of Evidence-Based Policymaking

If we want evidence‐based policy for educaƟon, we need to put the burden on experimental design, not

on our children’s private data. Researchers must devise ways to test hypotheses that require the least

amount possible of individuals’ private data—just as we minimize the risk for physical or mental harm in

clinical trials—because universal, lifelong data collecƟon is an unacceptably unethical course of acƟon.

I urge this Commission to consider the principles in the Belmont Report, wriƩen more than 40 years ago

under the charge of an earlier federal commission, the NaƟonal Commission for the ProtecƟon of Human

Subjects of Biomedical and Behavioral Research: the principles of respect of persons, beneficence and

jusƟce. The creaƟon of a naƟonal database of student records violates all three of those principles.

Once privacy is lost it is nearly impossible to restore. And so, we hold a moral and ethical obligaƟon to

our children – and our ciƟzens ‐‐ to minimize such a risk in any way possible.

Cassandre Creswell, PhD

Co‐execuƟve director

Raise Your Hand AcƟon

2507 N Sacramento Ave

Chicago IL 60647

cassie.creswell@gmail.com
The Promise of Evidence-Based Policymaking 577

1

College of Liberal Arts

& Sciences

TESTIMONY

Donna K. Ginther

Professor, Department of Economics

Director, Center for Science, Technology & Economic Policy

University of Kansas

and

Research Associate

National Bureau of Economic Research

Before the 

Commission on Evidence-Based Policy Making—Public Hearing

January 5, 2017

Introduction

Chair Abraham, Chair Haskins, and distinguished members of the Commission, thank you for 

the opportunity to testify before the Commission on Evidence-Based Policy. My testimony will 

address the goals of the commission through the lens of my personal experience as an evidencebased policy researcher. 

I have the following observations for the committee: First, evidence-based policy is 

critically important for the efficient allocation of government resources. The Commission’s 

report, Using Administrative and Survey Data to Build Evidence, provides several examples of 

how administrative and survey data have informed policy. I highlight some of my contributions 

to policy in this testimony. 

Second, evidence-based policy is easier said than done. In my experience, evidence-based 

policy only takes place when policy-makers initiate the process. When researchers approach 
578 The Promise of Evidence-Based Policymaking

2

government agencies with a request to use administrative or survey data in ways that stray from

the norm, the barriers to using these data become insurmountable. 

Third, to the extent that states gain more control over administering federal programs (for 

example Temporary Assistance to Needy Families), obtaining permission to use state data and 

the complexity of state policies will raise additional barriers to evidence-based policy. State 

governments do not typically have the resources or expertise to maintain high-quality data. 

Some states, such as Wisconsin, have worked closely with researchers to understand policy. The 

Institute for Research on Poverty at the University of Wisconsin-Madison has had access to state 

child support data for years, and assisted the state with improving child support payments. The 

Texas Schools project, based at the University of Texas-Dallas has longitudinal student data and 

is a model for providing access to researchers. North Carolina, Florida and Michigan have also 

shared student data with researchers. However, these states are the exception in granting access 

to researchers. 

Finally, I have two recommendations for the Commission: First, the Commission must 

think carefully about the incentives for policy makers to support evidence-based policy. Second, 

I recommend that a series of experiments be conducted with federal agencies to incorporate 

evaluation of current programs and research design in new programs in order to evaluate their 

effectiveness. I am not convinced that every program requires a randomized controlled trial 

before implementation. However, quasi-experimental designs, field experiments, and good “old 

fashioned” data analysis with administrative data can go a long way towards informing policy. 

Once the “proof of concept” has been established and refined, then evidence-based policy could 

be more widely implemented across the government.
The Promise of Evidence-Based Policymaking 579

3

In the testimony that follows, I will provide a brief overview of my work related to 

evidence-based policy, describe my experience with the barriers to using administrative and 

survey data, discuss successful examples of evidence-based policy, and provide 

recommendations to the Commission on policy-maker incentives and research design in program 

evaluation.

Research Related to Evidence-based Policy

I am a proponent of the concept of “Big Data to Knowledge,”1 and my research and the 

work of the Center for Science, Technology & Economic Policy is united by our approach of 

linking several data sources together to create big/complex data that yield insights that are 

greater than the sum of its parts. I have received several federal and foundation grants for work 

in the interdisciplinary field of the Science of Science and Innovation Policy (SciSIP). At the 

core of Science Policy research are two related economics questions: 1) How should the federal 

government allocate scarce research and development resources to maximize economic growth 

and development? 2) What is the return on the federal government’s research and development 

investments? My research contributes to the study of science policy by examining the allocation 

of grant funding, gender and race/ethnicity differences in academic careers, and scientific 

entrepreneurship and innovation. I was also the principal investigator on the NSF grant to fund 

 1 Big Data to Knowledge (BD2K)—an initiative started by the National Institutes of Health in 2012 that is related to 

some of the goals of the Commission on Evidence-Based Policy. According to the NIH: “The BD2K initiative 

addresses four major aims that, in combination, are meant to enhance the utility of biomedical Big Data: 1) To 

facilitate broad use of biomedical digital assets by making them Findable, Accessible, Interoperable, and Reusable 

(FAIR); 2) To conduct research and develop the methods, software, and tools needed to analyze biomedical Big 

Data. 3) To enhance training in the development and use of methods and tools necessary for biomedical Big Data 

science. 3) To support a data ecosystem that accelerates discovery as part of a digital enterprise.” 

https://datascience.nih.gov/bd2k/about.
580 The Promise of Evidence-Based Policymaking

4

the Kansas City Research Data Center,2 one of the twenty-four Federal Statistical Research Data 

Centers.

Under my direction, the Center for Science, Technology & Economic Policy (CSTEP) at 

the Institute for Policy & Social Research at the University of Kansas has undertaken several 

“big data” projects. Between 2010 and 2015, CSTEP was the administrative home of the 

Kansas City Area Education Research Consortium (KC-AERC), an education research 

consortium modeled on the University of Chicago Consortium of School Research. Unlike the 

Chicago Consortium that has researchers from one university working with one school district, 

KC-AERC consists of a multidisciplinary team of researchers from the University of Kansas, the 

University of Missouri-Columbia, the University of Missouri-Kansas City, and Kansas State 

University who work with the 32 Kansas City-area school districts in both Kansas and Missouri 

to conduct data-driven education policy research and evaluation. CSTEP has also received 

contracts from the Kansas Board of Regents (KBOR) to match KBOR enrollment and graduation 

records for all public community colleges, technical colleges and four-year universities in the 

state to employment records from the Kansas Department of Labor in order to analyze the 

efficacy of KBOR programs. 

In what follows, I share the lessons I have learned from working on policy-relevant 

research using administrative and survey data from multiple governmental entities. The barriers 

to obtaining data for evidence-based policy are significant and in some cases, insurmountable. 

However, when researchers become “trusted partners” with policy-makers, the evidence 

generated by administrative data is persuasive, and policies do change. Two projects illustrate 

the barriers and successes I have encountered.

 2 “Kansas City Research Data Center.” National Science Foundation SES-1359527.
The Promise of Evidence-Based Policymaking 581

5

A Tale of Two Projects

Barriers. In 2004, the National Science Foundation (NSF) funded my proposal: “Gender 

Differences in Employment Outcomes for Academics in Science and Social Science” SES0353703. This grant proposed to merge publication data from Thomson-ISI’s Web of Science

onto the NSF’s Survey of Doctorate Recipients (SDR) in order to examine whether productivity 

differences explain the gender gap in salary and promotion in academia. The SDR is a biennial, 

longitudinal survey that collects information on demographics, education, employment and 

earnings of U.S. trained doctorates in science, technology, engineering and mathematics (STEM) 

fields. Although the grant was funded in 2004, the creation of the SDR Productivity Database 

has been a work in progress ever since. 

Essentially, a proposal to merge SDR data with other data sources puts legitimate 

research of importance to Congress3 at odds with the Confidential Information Protection and 

Statistical Efficiency Act (CIPSEA) of 2002.

4 It took until 2006 for NSF to establish a policy to 

permit matching SDR data with other sources. In 2008, the NSF funded a research conference, 

“Collaborative Research: Workshop on linking NSF SED/SDR Data to Scientific Productivity” 

SRS-0725475 which brought together researchers interested in using the SDR Productivity data, 

statistical experts on linking data sets, and staff from the NSF Division of Science Resource 

Statistics to discuss the issues involved in creating the data with the least amount of matching 

error, ensuring its confidentiality, and providing access to the research community.5

 Between 

2006 and 2009, I drafted several versions of the data-matching proposal, and ultimately in 2010 

 3 This section of the testimony is an update of my testimony before the Subcommittee on Research and Science 

Education of the U.S. House of Representatives on the Fulfilling the Potential of Women in Academic Science and 

Engineering Act of 2008. May 8, 2008. 4 One NSF staff member initially responded to my request to link publication data to the SDR by saying, “No, I 

could go to jail if we were to do so.” This is presumably a reference to provisions in CIPSEA. 

5 Information from this conference is available at http://www.albany.edu/~marschke/Workshop/. The Division of 

Science Resource Statistics was renamed the National Center for Science and Engineering Statistics.
582 The Promise of Evidence-Based Policymaking

6

the National Center for Science and Engineering Statistics (NCSES) at NSF took over the 

funding and oversight of matching publications and patents to the SDR. 

The creation of the SDR Productivity Database is still a work in progress, twelve and 

half years later. I could not complete the proposed work in my original NSF grant, and I have 

three projects that are still waiting on the data. The data have been matched and NCSES is now 

reviewing the data quality before releasing it to researchers; something I hope will happen in 

2017. However, NCSES dropped two-thirds of the SDR longitudinal panel between 2013 and 

2015 waves of the survey, reducing the quality and viability of the SDR for future research, and 

creating another barrier. It remains to be seen whether the SDR data are useful for my research 

purposes beyond the 2013 wave.

Successes. In 2008, I was contacted by Thomson Reuters/Discovery Logic (a contractor 

for NIH) about assisting with a project to examine the diversity of NIH-funded researchers. The 

NIH provided our research team with every variable I requested from NIH’s IMPAC II 

administrative database on over 88,000 NIH R01 grant proposals from 2000-2006 to examine 

race/ethnicity differences in NIH funding. In the process of creating these data, our research 

team linked publications and bibliometric measures for the principal investigators to the grant 

proposals. Ironically, the work I proposed to do with the SDR, I was allowed to complete for the 

NIH. 

The resulting paper was published in Science: “Race, Ethnicity, and NIH Research 

Awards” was co-authored with Raynard Kington, former Deputy Director of the National 

Institutes of Health, Walter Schaffer in the Office of the Director at NIH, and colleagues at 

Thomson Reuters/Discovery Logic. Our study found that Asians were 4 percentage points and 

black or African-American applicants were almost half as likely (13 percentage points) to 
The Promise of Evidence-Based Policymaking 583

7

receive NIH investigator-initiated research funding compared to whites. After controlling for the 

applicant’s educational background, country of origin, training, prior research awards, 

publication record, and employer characteristics, we found that black applicants remained onethird less likely (10 percentage points) than whites to be awarded NIH research funding. 

The findings in Ginther et al. garnered significant media attention and changed NIH 

policy. As a result of the study, NIH Director Francis Collins appointed the Advisory 

Committee to the Director Working Group on the Diversity in the Biomedical Research 

Workforce. Based on the findings of the Working Group, NIH made a $500 million, ten-year 

commitment to improve diversity through increasing the number of underrepresented minorities 

who study biomedicine (Building Infrastructure Leading to Diversity) and creating a national 

mentoring network (National Research Mentoring Network).6

 NIH also hired a chief officer for 

Scientific Workforce Diversity.

Why did the NIH project change policy while the NSF project still languishes? The NIH 

study was initiated by a policy-maker, Raynard Kington, who at the time was Deputy Director of 

the NIH. Institutional support yielded immediate data access. In contrast, in 2004 I was a 

relatively unknown researcher asking permission to link data to the SDR shortly after a federal 

law made this type of request open to interpretation. No bureaucrat has ever lost his or her job 

by saying “no” to a researcher request. Thus, in my experience, when the study is proposed by 

the policy-maker, access to the administrative data will be forthcoming. 

Evidence-based policy at the state and local levels. This iron-law of evidence-based 

policy research has played out again and again in my experience. In 2009 we started the Kansas 

City Area Education Research Consortium with the goal of creating a data repository for all 

 6 NIH BUILD: https://www.nigms.nih.gov/training/dpc/pages/build.aspx. NIH NRMN: 

https://www.nigms.nih.gov/training/dpc/pages/nrmn.aspx.
584 The Promise of Evidence-Based Policymaking

8

Kansas City metropolitan school districts using the state longitudinal data systems (SLDS). But 

our vision for a data repository for Kansas City metropolitan area schools has never been 

realized, because the state departments of education in Kansas and Missouri did not trust KCAERC as partners. In the case of Kansas, KC-AERC researchers obtained student-level data on 

two occasions for specific projects. In the case of Missouri, KC-AERC researchers developed a 

measure of teacher-value added using student-level data, only to have access to those data 

rescinded after the project was completed. 

Despite our failure to develop lasting partnerships with the state departments of 

education, over time school districts learned about KC-AERC and started hiring us to conduct 

evaluation projects. As we became trusted partners, school districts provided administrative data 

to KC-AERC to evaluate a number of different programs.

7 The same has been true in our work 

with the Kansas Board of Regents. We have received funding for several contracts to link 

KBOR data to employment and earnings records in Kansas, evaluate the efficacy of adult basic 

education, high school technical education, and the graduation rates of community college 

transfer students. From the perspective of state and local policy-makers, “qualified researchers 

and institutions” need to be known, trusted, and local. Research studies based on other school 

districts or states do not have external validity for state and local policy-makers.

States are even wary about sharing data with the federal government: Kansas was the last 

state to sign an agreement with the US Census Bureau to allow its data to be incorporated in the 

LEHD. Currently states review all proposals related to using their LEHD data and retain the 

right of refusal for these projects. In addition, when 50 state governments make myriad changes 

to state-administered federal programs (e.g. TANF), the policy environment becomes very 

complex, very quickly. I have a new project that proposed to examine the impact of TANF 

 7 See current projects at www.kcaerc.org.
The Promise of Evidence-Based Policymaking 585

9

policy changes. Welfare reform happened twenty years ago, but there are very few studies that 

have traced the impact of state TANF policy changes since then, in part because it is a very 

difficult and complicated policy environment to study. To the extent that other federal programs 

are block-granted to the states, our ability to understand how these policies affect outcomes may 

be severely compromised because so many state policies will be changing all at once. 

Recommendations for the Commission

I offer two recommendations to the Commission. First, policy-makers are the key to 

implementing evidence-based policy. Thus, the Commission needs to think carefully about how 

to align the incentives for policy-makers to share their data and be receptive to findings. After 

completing the NIH study of race/ethnicity differences, I spoke with administrators in another 

science funding agency about conducting a similar study. These administrators declined my 

offer. Assuming that the Commission is successful in breaking down legal and administrative 

barriers, risk aversion on the part of policy-makers will remain a stubborn obstacle. 

Second, I recommend that a series of experiments be conducted with federal agencies to 

incorporate evaluation of current programs and research design in new programs in order to 

evaluate their effectiveness. In particular, I recommend that the Commission work with the 

science funding agencies (NIH and NSF) to develop prototypes for evaluating existing programs 

and incorporating evaluation into the design of new programs. I suggest the science agencies, 

because they will be predisposed to using evidence-based approaches to understanding their 

programs. I suggest these experiments and prototypes since it is useful to have evidence to 

support building evaluation into program design. I will provide some examples of how this 

might work using current programs that I am familiar with at the NIH. 
586 The Promise of Evidence-Based Policymaking

10

After we published our paper in Science, NIH announced the Early Career Reviewer 

(ECR) program,8 motivated by our finding that individuals who served on NIH study sections 

were more likely to receive NIH funding. However, our analysis did not demonstrate that 

serving as a reviewer had a causal effect on funding. NIH would know the causal effect of 

serving as a reviewer now if ECR applicants were randomly assigned to an ECR treatment or 

control group when the program was started. These types of policy experiments could take two 

or three years to evaluate, but it would be useful to know whether the program works as expected 

instead of implementing the policy and hoping for the best.

NIH has also been concerned that the age of obtaining the first R01 grant has increased, 

and it created policies to maintain award rates for new investigators that were similar to 

experienced investigators.9 This policy has been in place since 2008, and it would be possible to 

use quasi-experimental methods to evaluate whether NIH new and early stage investigator 

programs had the desired effect. 

Conclusion

Chair Abraham and Chair Haskins, once again I thank you for this opportunity to testify 

today. I believe that evidence-based policy is critically important to the efficient allocation of 

governmental resources. However, evidence-based policy is much easier said than done 

because policy-makers are the gatekeepers to their data. Evidence-based policy analysis cannot 

take place without the support of policy-makers. Thus, I encourage the Commission to consider 

ways to align policy-maker incentives to facilitate the goals of data access and robust evaluation. 

Finally, I recommend a series experiments be conducted with federal agencies to incorporate 

 8 https://public.csr.nih.gov/reviewerresources/becomeareviewer/ecr/pages/default.aspx. 9 https://grants.nih.gov/policy/new_investigators/index.htm.
The Promise of Evidence-Based Policymaking 587

11

evaluation of current programs and research design in new programs to demonstrate the 

effectiveness and build support for evidence-based policy. 
588 The Promise of Evidence-Based Policymaking

1

December 20, 2016

Statement and Recommendations Concerning 

Commission on Evidence-Based Policymaking

by

V. Joseph Hotz

Arts & Sciences Professor of Economics

Duke University

Durham, NC 27708

hotz@econ.duke.edu

The following comments are related to comments submitted on behalf of the Population 

Association of America (PAA) and the Association of Population Centers (APC) on November 

14, 2016. Nonetheless, all of the views and opinions expressed below are my own and ones I 

support.

Let me begin by applauding passage of the Evidence-Based Policymaking Commission Act of 

2016, which created the bipartisan Commission on Evidence-Based Policymaking and charged 

this Commission with: 

• determining how to integrate administrative and survey data and to make those data available 

to facilitate research, evaluation, analysis, and continuous improvement while protecting 

privacy and confidentiality; 

• recommending how data infrastructure, database security, and statistical protocols should be 

modified to best fulfill the integration and increased availability of data as described above;

• recommending how best to incorporate rigorous evaluation into program design; and 

• considering whether a federal clearinghouse should be created for government survey and 

administrative data. 

In what follows, I focus my comments on two related issues: the data infrastructure to support 

evidence-based policy analysis and the importance of insuring access to these data for qualified 

researchers, both within and outside of government.

A. There are important benefits to the use of administrative data, especially when linked, for 

conducting policy-relevant research. Administrative records have been used in a variety of 

research areas and provide an essential source of data for conducting important policy-relevant 

research. For example, such records have been used to study participation in and impacts of 

social programs (e.g., welfare programs, manpower training, food stamps, the earned income tax 

credit, etc.) on various outcomes. Often these outcomes are measured with linked administrative 

data, such as wage earnings (from linked unemployment insurance wage records), health 

conditions (from linked Medicaid records) or fertility (from linked birth certificate records). The 

availability of administrative records from federal, state or local sources provide a cost-effective 

way of supporting evaluations of these programs, regardless of whether the evaluations made use 
The Promise of Evidence-Based Policymaking 589

2

of randomized designs for allocation of program participants to different “treatments,” or other 

studies that have made use of non-experimental designs. 

But social program evaluation is not the only place where administrative records can and will be 

the primary source of data to monitor particular programs and/or evaluate particular policies or 

“treatments.” Furthermore, they do not only use government records. Here I reference two 

examples. First, biomedical research, including research that is relevant to policies affecting 

health-related behaviors, such as smoking bans or regulation of the nutritional content of foods, 

uses increasingly electronic health records (EHRs) from public and private health care systems to 

measure the health effects of variation in such policies. Second, administrative records from 

private firms that construct credit scores for use by financial institutions have been used by 

researchers, including the research division of the New York Federal Reserve Bank, to monitor 

and conduct policy-relevant research on student loan debt in the U.S. In both of these areas, 

administrative records support important policy-relevant research in a way that is both costeffective and potentially more accurate than data collected via other means, e.g. surveys.

B. At the same time, there are important legal and other constraints that limit the use of 

administrative records and the ability of researchers to link records across different sources of 

these records. In particular, different sources of administrative data are subject to varying and 

divergent laws and regulations that can inhibit their use. For example, administrative records 

from social programs administered at the state or local level (e.g., TANF programs) are often 

subject to laws and regulations that make it hard for one agency to share their records with 

another agency. And, as noted in the NRC report on the Reengineering of the Survey of Income 

and Program Participation (SIPP), existing state laws that cover the privacy and access of 

administrative records from TANF, Medicaid, unemployment insurance, and the workers’ 

compensation programs make it very difficult, if not impossible, for these programs to share their 

data with the Census Bureau (or other) surveys like the SIPP.1

Historically, this issue has complicated the conduct of biomedical research that makes use of 

electronic (or non-electronic) health care records of individuals as institutional review boards 

(IRBs) have required studies to obtain informed consent from subjects in these studies for any 

follow-up use of subjects’ EHRs and/or updating of these records. Recently proposed revisions 

to the Common Rule2 will reduce and/or eliminate this re-consenting requirement for certain 

types of studies and types of administrative records so long as subjects are provided with a clear 

statement regarding potential future use of administrative records as part of their initial consent 

process. Many population scientists welcome this change and suggest it may represent a model 

for the Commission to examine as it considers how to facilitate access to records like EHRs 

while still providing participants with the opportunity to make informed decisions about research 

access to their records. 

More generally, I strongly urge the Commission to investigate the various laws and regulations 

 

1 Constance Citro and John Karl Scholz, eds., Reengineering the Survey of Income and Program 

Participation, National Research Council, 2009. [NOTE: I served as a member of the NRC expert panel 

that developed this report.]

2 HHS–OPHS–2015–008, Federal Policy for the Protection of Human Subjects, Federal Register,

80(173), Sept 8, 2015, 53933-54058.
590 The Promise of Evidence-Based Policymaking

3

governing access to administrative records for research purposes. In particular, I encourage the 

Commission to look closely at the laws affecting access to state and local government data and 

policies restricting record linkage across various federal agencies. 

C. To facilitate the conduct of evidence-based, policy-relevant research, I encourage the 

Commission to examine and improve access to administrative records to qualified researchers 

outside and inside the government. I understand and appreciate that there are important 

confidentiality and security concerns that necessarily limit access of researchers to various types 

of government-based administrative records and/or restricted-use data sources. Furthermore, I

appreciate why restrictions on the access of non-governmental researchers may need to be 

different, and possibly more restrictive, than that applied to researchers employed by authorized 

government agencies. But, at times, these restrictions have made access to such data very 

difficult for academic and non-governmental researchers. 

Over the last 20 years, U.S. statistical agencies, initially led by the U.S. Census Bureau, have 

made great strides in improving access to restricted-use versions of federal data sources through 

the Federal Statistical Research Data Centers (RDCs) program. This program now allows access 

to data products from 12 different federal statistical agencies for qualified governmental and 

non-governmental researchers in 20 different centers around the country. While some the 

research covered by the data agreements approved for use of these centers is often not directly 

related to policymaking, much of it is. 

A similar effort for providing access to data from the Internal Revenue Service (IRS) under the 

Joint Statistical Program of the Statistics of Income (SOI) Division of the IRS has enabled 

qualified researchers to submit proposals for access to IRS data and to link it to various data for 

research purposes. This program has facilitated a number of highly visible and widely cited lines 

of research by Professors Raj Chetty (Stanford) and Emmanuel Saez (UC Berkeley). For 

example, Chetty and co-authors analyzed the association between income and the life expectancy 

of individuals in the U.S. since 2000 by linking IRS tax records on income with Social Security 

Administration death records.3 The findings of this research, especially the finding of differences 

by area in the associations of longevity by income, raises important questions about the sources 

of these disparities and how to alleviate these differences. Such research could not have been 

conducted without this program. 

A large body of research shows that geography (e.g. neighborhoods) affects the social and 

economic well-being and health of individuals and families. But, state and local policymakers, 

researchers, and program officials often lack the data needed to measure differences in 

community environments, to isolate how neighborhood characteristics shape micro-level 

outcomes, or to test the efficacy of neighborhood-level interventions. Most survey data files lack 

such key contextual information, while most administrative data lack key demographic, 

socioeconomic, behavioral, and outcome information. While individual-level record linkage of 

survey and administrative data could provide such critical data for state and local-level evidencebased policymaking, most state and local researchers/program evaluators lack the resources to 

submit proposals and conduct these types of linkages and research within a RDC. The 

 

3 Chetty, R. et al. (2016). “The Association between Income and Life Expectancy in the United States, 

2001-2014, Journal of the American Medical Association, 315(15): 1750-1766.
The Promise of Evidence-Based Policymaking 591

4

Commission should also encourage statistical agencies and other researchers to create spatiallylinked administrative and survey data that could be provided to state and local 

researchers/program evaluators outside of RDCs to increase evidence-based policymaking at the 

state and local level. 

I call on the Commission to encourage expanding access of data and records from federal, state 

and local sources to qualified non-governmental and governmental researchers. This expansion 

should include state and local government researchers, whose access to data can provide support 

for accurately assessing needs, creating programs to address those needs, and delivering services 

in more cost-effective ways. While such efforts may include expanding the RDC and/or IRS’s 

Joint Statistical Programs or similar programs, they should also include expanding access to 

spatially-linked administrative and survey data that could be provided outside of RDCs. Efforts 

also may include providing more funding for merit-based grants to undertake these projects, 

especially in light of the limited resources available to researchers in local governments and 

those working in non-governmental settings. 

I also encourage the Commission to consider recommending any necessary legal revisions that 

would allow federal statistical agencies to share data with researchers conducting evidence based 

research. For example, the Census Bureau’s authorizing regulation, Title 13, does not explicitly 

recognize the use of sensitive data for conducting scientific research, be it policy-relevant or not, 

as a “benefit to the Bureau.” Rather, Title 13 only supports data access to improve the quality of 

Census (and other) data products. A more explicit acknowledgment that qualified research 

projects can be conducted for scientific purposes would allow Census to approve studies using 

confidential data that are primarily designed to replicate existing studies and/or determine the 

robustness of findings from previous research. Such changes would help ensure the legitimacy of 

research uses of these data and give greater credibility to the findings based upon these data. 

D. I encourage greater attention be given to the population representativeness of the policyrelevant research produced using administrative records and population-based surveys to 

better assess and characterize the population-representativeness of findings from 

administrative data. Many studies use administrative records to “evaluate” the impact of some 

particular policy or program. As I have argued above, the administrative records provide a 

potentially cost-effective way of conducting such evaluations—particularly when compared to 

the alternative of collecting survey data that is collected from a sample representative of the 

population relevant for the study. But such benefits of using administrative records in evaluative 

research does not mitigate the importance of assessing the sampling properties of this data 

source. 

Consider the following example regarding the design of the Precision Medicine Initiative (PMI).

One of the key components of the PMI’s initial plan is to assemble a million-person sample of 

individuals who would provide access to their Electronic Health Records (EHRs) as a condition 

of the study. Access to EHRs on this large sample would provide data to study a wide range of 

health conditions, including conditions that are relatively rare and only affect population 

subgroups. One of the study’s recruitment strategies was to use social media and other methods 

to attract participants who would grant access to their EHRs and undergo one or more physical 

examinations. 

While the goals of the PMI are important and have the potential to provide evidence-based 

assessments of health conditions relevant for U.S. health policy, population scientists and other 
592 The Promise of Evidence-Based Policymaking

5

social scientists are concerned about lack of attention to the properties of what amounts to a 

“volunteer” sample of people with EHRs, even if the sample includes data on one million 

participants. In public comments, PAA and APC raised these concerns and strongly suggested 

that the NIH leadership consider using existing population-based health studies to form at least 

part of the PMI cohort to assess the population-representativeness of the recruitment strategy 

based on volunteers. In developing both policies and best practices for policy-relevant research, I

encourage the Commission to advocate for the designs of data collection that explicitly account 

for the sampling properties and population-representativeness of its studies. 

Lastly, I encourage the Commission to ensure that population-representative data sources 

collected by the Federal government continue to be viewed as an important source of data for 

policy-relevant research, both as a way to monitor behaviors and phenomena relevant to public 

policy. For example, data sources like the Current Population Survey (CPS), the American 

Community Survey (ACS) and the Survey of Income and Program Participation (SIPP) all play 

roles in the monitoring and implementing public policy in the U.S. The CPS is the populationrepresentative data that enables the BLS to construct estimates of unemployment and labor force 

participation rates of the U.S. population on a monthly basis. The ACS provides data on poverty 

rates at the lowest levels of geography, such as school districts and communities, which are used 

to allocate funding for programs such as the USDA’s National School Lunch Program and State 

Children’s Health Insurance Program (SCHIP). The SIPP has facilitated a broad range of 

research on the distribution of income and participation in a range of social programs using a 

survey that is designed to be population representative for most states in the U.S. These surveys, 

and others, are important components of the U.S. data infrastructure and are needed to support 

evidence-based policymaking.
The Promise of Evidence-Based Policymaking 593

Comments to the Commission on Evidence-Based Policymaking

from Virginia Knox, MDRC

January 5, 2017

My name is Virginia Knox, and I am the Director of the Family Well-Being and 

Children’s Development Policy area at MDRC. We’re a nonprofit, nonpartisan education 

and social policy research organization dedicated to learning what works to improve 

policies and programs that affect low-income individuals and families. 

The ideas I’ll outline today are laid out at greater length in written comments that MDRC 

submitted to the commission in November. MDRC hopes to see the Commission use its 

mandate to recommend that the federal government put rigorous evidence at the center of 

policymaking. 

My comments will briefly outline six of the elements we see as necessary to achieving 

that goal: 

• Validate the role of third-party evaluations 

• Create a culture of continuous improvement

• Build on tiered evidence strategies

• Embed evidence within existing funding streams

• Improve access to administrative data (while acknowledging its limitations) 

• And protect data confidentiality. 

First is to validate the role of independent evaluation of programs and policies in the 

federal government: Evaluation findings that are credible, relevant, accurate, and timely 

are critical for policymakers and practitioners to make informed decisions about how to 

spend the resources of government. This is an issue of particular urgency in a time of 

budget constraints and fiscal austerity.

Second -- Create a culture of continuous improvement: Rather than being focused on 

up-or-down judgments about programs or policies, government should continue to 

develop incentives for using research evidence to make programs more effective over 

time. A good example is the Maternal, Infant, and Early Childhood Home Visiting 

(MIECHV) program, a large evidence-based federal initiative that has several elements 
594 The Promise of Evidence-Based Policymaking

2

worth emphasizing: 

• Rigorous prior evidence is being used to influence how federal funds are spent, 

making it more likely that the funds will make a difference for families.

• The legislation recognizes that innovation is important for any field to keep 

advancing, and offers states a subset of funds to test new approaches. 

• Funds were also set aside for research to make sure that learning continues under 

MIECHV and can influence future realizations of home visiting.

Third -- Build on the tiered evidence strategies embodied in the Investing in 

Innovation Fund at the Department of Education, the Workforce Innovation Fund at the 

Department of Labor, the Teen Pregnancy Prevention Program at the Department of 

Health and Human Services, and the Social Innovation Fund at the Corporation for 

National and Community Service. These funds set clear guidelines about standards of 

evidence and provides incentives for both innovative new programs and for testing the 

scaling of models with evidence of effectiveness — which we see as truly the next 

frontier in the evidence-based policymaking agenda.

Fourth -- Embed evidence within existing funding streams: As the MIECHV example 

illustrates, when we have evidence of what works, we should build incentives into 

current funding streams to make sure that dollars follow the evidence. And while the 

innovation funds have been a source of effective new ideas for a given field, 

incorporating resources for research within major program funding streams would allow 

federal agencies to develop evaluation agendas that focus on continuous improvement of 

existing programs. 

Fifth - Improve Access to Administrative Data 

In evaluating the effectiveness of social programs, agencies and the researchers who 

evaluate their programs need ready access to administrative data. At the same time, 

access to administrative data is not a panacea, so we shouldn’t dismiss the importance of 

surveys and other types of data. 

There are at least four factors that affect the fit of administrative records for a given 

research study.

• Access – The process of gaining permission and approval to use the admin data 

may be cumbersome and expensive

• Standardization – if there are multiple sources of admin data and they are all in 

different formats, then the researcher will have to spend considerable time 

standardizing it before she can conduct the needed analysis

• Completeness - administrative records often do not include all the data required to 

answer important questions

• Data Lags - Federal data sets like the National Directory of New Hires or tax 

records can have very long lags before the data are assembled, cleaned, and 

available for use in a study. This has significant implications for timeliness and 
The Promise of Evidence-Based Policymaking 595

3

limits the use of administrative data in quick-turnaround studies with multiple 

follow-up periods.

My sixth and final point is about protecting confidentiality. 

There are a number of important tensions that underlie this high stakes issue. As an 

example, Congress is considering amending The Family Education Rights and Privacy 

Act (FERPA) because of concerns over threats to the privacy of student data, originally 

prompted by public outcry over educational technology vendors and their use of 

children’s information for advertising and commercial gain. Unfortunately, education 

researchers from academia and other nonprofit institutions have gotten swept up in the 

furor.

Under current federal law, education agencies can share data with researchers only for 

research projects designed to benefit students and improve instruction — and only under 

extremely strict privacy conditions. But some are suggesting that Congress should 

significantly scale back even that authority. Indeed, many states are interpreting FERPA 

to preclude the sharing of any individually identifiable data with researchers, even though 

that data would only be reported in aggregate form for policy purposes. Without access to 

student data, little education research could be conducted at all. The bottom line is that 

it’s essential to continue to protect the security and privacy of student data, but we must 

be careful to not unintentionally end the analysis of student data for its original purpose: 

assessing and improving education.

This is just one example of how we need to protect the privacy of data for people who 

participate in studies, while still making it possible to use available data to improve social

policy.

Thank you for allowing me this time to provide input from MDRC into the important 

work of the commission. 

Sincerely,

Virginia Knox

Director, Family Well-Being and Children’s Development Policy

MDRC

16 E. 34th Street, 19th Floor

New York, NY 10016

virginia.knox@mdrc.org
596 The Promise of Evidence-Based Policymaking

Commission on Evidence-Based Policymaking

Hearing, Chicago, January 5, 2017

Submission from: Institute for Social Research, University of Michigan, P.O. Box 1248, Ann

Arbor, MI 48106-1248. Contact: Margaret Levenstein, Director, Inter-university Consortium for

Political and Social Research and Research Professor, Institute for Social Research,

MaggieL@umich.edu.

The University of Michigan’s Institute for Social Research (ISR) is pleased to present the

comments below for consideration as the Commission deliberates to expand access to and the

use of government data for the purposes of building evidence-based evaluation of program and

policy outcomes while concurrently protecting the privacy and confidentiality of the citizens and

organizations studied. There are several general principles that we believe should guide the

Commission as it examines opportunities to build and evaluate evidence-based programs and

policies with administrative and survey data. These principles are woven throughout our

comments to the Commission’s questions below and are summarized as follows:

● Confidentiality of individual data and the independence of the federal statistical system

must remain paramount. Participation in our federal data programs, whether they collect

survey or administrative data, is premised on the promise that individual data will remain

confidential and will be used for statistical purposes only. It is never to be used for

enforcement purposes or for the benefit of particular commercial or political interests.

Confidence in the estimates produced by our federal statistical system requires

adherence to these principles at all times. As articulated in the Office of Management

and Budget’s Statistical Policy Directive No. 1 (2014), it is critical to “Protect the trust of

information providers by ensuring the confidentiality and exclusive statistical use of their

responses. Maintaining and enhancing the public’s trust in a Federal statistical agency’s

or recognized statistical unit’s ability to protect the integrity of the information provided

under a pledge of confidentiality is essential for the completeness and accuracy of

statistical information as well as the efficiency and burden of its production.” This is just

as true when administrative data is re-purposed for statistics. Undermining this trust

undermines statistical measurement as well as the effectiveness of the programs upon

which the statistics are based.

● Two important steps to uphold these principles and assure the independence and

reliability of the estimates produced by our federal statistical system include the

following:

a. Data originally generated outside the federal system, either from state and local,

commercial, non-profit, social media, web-based, or other programs, should be

aggregated outside the federal system. These data can be cleaned and

documented, and secure, confidentiality-protecting crosswalks to PII can be

created. Data can then be transferred to the federal statistical system for

matching to federal data resources. This will preserve respondent and data

provider confidentiality and increase confidence in the security of the system. It
The Promise of Evidence-Based Policymaking 597

also provides a mechanism for state and local civil servants and third-party (e.g.,

academic researchers) to access the non-federal data in a secure environment

without burdening the federal research data system and its supporting security

clearance mechanisms. This is the model that has been adopted by the Institute

for Research on Innovation in Science (IRIS) in its collaboration with the Census

Bureau’s Innovation Measurement Initiative. This model leverages the expertise

of those outside the federal statistical system to improve, harmonize, and

document the non-federal data. This kind of expertise is often lost or reduced

when data are moved exclusively inside the federal statistical system.

b. Data generated by federal agencies and programs will almost surely be legally

required to stay within the federal firewall. These data should be made available

systematically (promptly, with transparent access procedures, and where there is

no or limited documentation, with a mechanism for researchers to contribute to

the data and documentation). This will harness the energies and expertise of

researchers to improve the data resources of the federal statistical system as

well as state-of-the-art analyses of policies in order to assure that the inference

drawn from evidence is scientifically sound. This also assures that there is

competition in program analysis, so that multiple approaches can contribute to

the analysis and program evaluation. The Federal Statistical Research Data

Centers are an important mechanism for providing researcher access to these

data, but, given the significant ongoing hurdles to their use, they should not be

the exclusive mode for researcher access.

Collaborations between federal agencies and academic organizations can help to address

several challenges in using administrative data for evidence-based policy making. First and

foremost, collaboration with multiple, external academic organizations can help to assure data

availability without excessive centralization that might compromise Americans’ right to privacy

and security in the data that is generated by and about them in the course of their interactions

with federal and state agencies. Second, collaboration between the federal government and

academic organizations can bring to bear the efforts of the large number of faculty and students

who would be willing to work to improve administrative data. This is particularly important as

administrative data, like other non-design data, require significant investments in cleaning and

documentation to assure that the target measurement concept is what is actually being

measured. Third, those same academic collaborators bring with them expertise in both

measurement and analysis that contributes to the scientific rigor of the policy analysis. Fourth,

collaboration with academic and other research organizations increases the likelihood that

multiple analytical approaches are considered, avoiding any tendency for monolithic or even

self-serving analysis. Finally, this kind of collaboration develops skills, both on the part of the

civil servants on whom we rely for producing the critical statistical resources of our country and

and on the part of students who are the next generation of scholars and civil servants. This kind

of collaboration will produce a generation of students who better understand the challenges of

measurement and policy-relevant analysis as a result of their participation in such a
598 The Promise of Evidence-Based Policymaking

collaboration.

For example, the Institute for Research on Innovation and Science (IRIS), a collaboration of

approximately 50 universities based at the Institute for Social Research (ISR) at the University

of Michigan, provides an excellent model in which universities (mostly state, but also private)

have voluntarily chosen to share confidential, proprietary data, including individual identifiers,

with a federal statistical agency for the production of new estimates of the impact of national

investments in research and development. IRIS has developed the capacity to ingest,

harmonize, and de-identify transaction-level data from its member institutions. It uses these data

to produce reports back to its members, restricted datasets available to the research

community, and datasets that it transmits to the U.S. Census Bureau for linkage to Census data

assets. The Census Bureau produces additional estimates and reports from these linked

datasets, and makes the linked data available to qualified researchers with approved projects in

the network of Federal Statistical Research Data Centers (FSRDCs). This model has been able

to achieve the participation of a large number of institutions, systematic access for the research

community, and a more valuable research dataset than was the case for an earlier initiative

strictly within the federal government. These data provide the basis for an evidence-based

evaluation of a wide range of federal and private programs investing in science and academic

R&D. IRIS’s model leverages the interest and abilities of the research community to analyze

these data as well as leveraging the existing data resources of Census Bureau and the

computing resources of the FSRDCs.

In another initiative, in this case between the Institute for Employment Research (IAB) of the

German Federal Employment Agency (BA) and the University of Michigan’s ISR, now expanded

to five other U.S. universities and locations in the UK and on the Continent, hundreds of

researchers have contributed to the analysis of German labor market reforms through their

access to restricted, linked survey and administrative data. These initiatives demonstrate both

the feasibility and the value of academic-government collaborations in overcoming the

challenges to creating appropriate data infrastructure and harnessing scientific expertise to

analyze those data for evidence-based policy evaluation.

State and local governments produce large amounts of administrative data on programs that

they implement, whether funded locally or by the federal government. State and local civil

servants have important expertise and knowledge about the operations of these programs and,

therefore, the meaning of the administrative data generated by them. They often lack the data or

statistical scientific expertise or computing environments in which to analyze or link these

resources. Partnerships between state and local governments, federal governments, and

academic institutions can provide the relevant training while developing data resources of value

to all parties for evidence-based program evaluation.

We recommend two complementary approaches to providing security and assuring privacy. One

is to take steps to assure that the data are safe. This can be achieved through traditional

anonymization methods (aggregation, suppression, swapping, and noise infusion), data
The Promise of Evidence-Based Policymaking 599

encryption, and the creation of fully or partially synthetic data. The other is to take steps to to

assure that the researcher who analyzes the data is safe and is working in a safe computing

environment. This is achieved through researcher training and credentialing, scientific peer

review and pre-registration of research proposals, and the use of secure computing

environments, such as virtual or physical research data enclaves. These two approaches are

complementary. Many tasks associated with the work of turning administrative datasets into

useful analytical datasets, including data cleaning, the production of metadata, and dataset

linkages, can only be accomplished with access to identifiable data. This then requires secure

computing and a system for training and vetting researchers. The UK’s Administrative Data

Research Service has made steps in the direction of researcher training and credentialing. The

European Union’s Data Without Boundaries project envisioned a researcher “passport” to

facilitate credentialed access across the European statistical agencies. The Sloan Foundation

has recently supported the Inter-university Consortium for Social and Political Research

(ICPSR) to build on these earlier projects to establish durable researcher credentials for access

to confidential data.

Exploiting the potential power of administrative and survey data for evidence-based policy and

program evaluation requires that both government and non-government analysts are able to

discover appropriate data resources and gain access necessary to analyze these data

effectively. To address these needs, an infrastructure characterized by rich metadata about

administrative and survey data sources, a secure platform for researchers to analyze datasets

held in other locations, and a standardized and broadly accepted system of researcher

credentialing must be developed. The existing network of Federal Statistical Research Data

Centers provides an important mechanism for non-governmental researchers to contribute their

expertise to the challenges faced by the federal statistical system and to the evaluation of

programs.

● The adoption of standardized researcher credentialing, accepted by multiple federal

statistical agencies, similar agencies in other countries (such as the German IAB and UK

Data), and non-governmental providers of confidential data to the research community

(ISR, NORC, RTI) can reduce barriers to accessing data by enabling qualified

researchers to analyze data through a modality that is appropriate to their level of

data-security training and experience. It also assures that access is obtained for

legitimate research purposes on an equitable basis.

● In order to make data useful, and the research arising from it replicable, investments

should be made in data documentation via well-defined metadata fields, and

infrastructure should be built that enable researchers to locate and analyze datasets

held in multiple, distinct, secure locations. Community curation, provided by researchers

who are invested in understanding the data and enabled with appropriate software, can

assist in building this documentation.

The characteristics of a data-sharing infrastructure designed to increase the availability and use

of government data for evidenced-based program evaluation include:
600 The Promise of Evidence-Based Policymaking

● Robust search and browse capabilities that leverage standardized metadata, permitting

researchers to discover data and learn about data in depth

● Capacity to facilitate crowdsourcing (active curation) and improvement of metadata to

capture and leverage newly acquired knowledge about the data

● Capacity to recognize varying levels of credentials assigned to a researcher ID

● Functionality that enables researchers to analyze datasets held in multiple, distinct,

secure locations, that is, a computing backbone that can support secure, multi-party

computing.

Two white papers prepared for the Commission on Evidence-Based Policymaking by the Office

of Management and Budget, “Using Administrative and Survey Data to Build Evidence” and

“Barriers to Using Administrative Data for Evidence Building,” explicitly point to the key

challenges. These include statutory prohibitions that hinder access to the data; policy and legal

interpretations, which can vary across agencies and federal, state, and local governments; and

resource and capacity constraints, specifically the lack of appropriate and reliable infrastructure

to address data sharing and access, management and curation of data, and security and

privacy concerns.

The Longitudinal Employer-Household Dynamics/Local Employment Dynamics data program is

our best example of such a collaboration. LEHD highlights both the enormous potential and the

enormous challenges to creating and making use of linked state and federal data. This

collaboration has made possible very important new data assets which have revolutionized our

understanding of local and internal labor markets, job creation and destruction, and job mobility

for workers in different industries, cohorts, and demographic groups. These valuable data

remain underutilized because of limitations to access. Providing resources to strengthen state

and local government statistical capacity would allow those agencies and their civil servants to

participate more effectively in research using these data. Increased capacity within state and

local governments would allow these agencies and civil servants to benefit from collaboration

with external researchers and reduce their incentives to impede research.

There are benefits and limitations to both the single- and multiple-clearinghouse approaches.

Overall we endorse a principle of union catalogs, so that data can be discovered and compared.

A single clearinghouse would facilitate the process of finding and gaining access to the data and

potentially linking multiple datasets. A clearinghouse would also act as a single point of entry for

an analyst searching for appropriate data with which to address his/her question, and one might

expect that a single catalog would have the benefit of consistent metadata to assist the

researcher in evaluating the options and identifying the most useful source of data. Having a

single clearinghouse to more efficient linking of datasets, for example if the clearinghouse

functioned as a trusted third party and provided de-identified, linked data to researchers. A

single clearinghouse, using appropriate software to track dataset versions would also increase

reproducibility of analyses by make it easier for researchers and policymakers to identify a

specific instance of a dataset. Given that administrative datasets are updated regularly as new
The Promise of Evidence-Based Policymaking 601

data are generated versioning of data is particularly important for rigorous and reproducible

analysis. The most important benefit to a single clearinghouse is that it would reduce the

bureaucratic hurdles to analyses that required to access multiple datasets; on the other hand,

these hurdles creates checks and balances and privacy protection that can be undermined by

centralization.

Multiple clearinghouses, however, would allow for specialization and expertise around particular

data sources and/or types (which lends itself to strong user support as well) and the flexibility to

respond more efficiently to changes in formats or uses of data in a particular domain. One of the

challenges to using administrative data for research and analysis is the lack of accompanying

documentation about the fields in the dataset. A series of specialized clearinghouses could

begin to address this because domain-specific staff expertise could, over time, be used to

create such documentation -- for example, noting when the underlying meaning of a particular

field has changed or even simply pointing out that distributions on key variables changed at a

specific point in time so that the researcher could do the detective work necessary to figure out

why. Having multiple clearinghouses also spreads and develops the capacity necessary in both

person-power (tagging, data checking, user support) and hardware/software for storing and

disseminating the data across multiple organizations. This decentralization provides robustness

to the infrastructure while increasing privacy protections.

An efficient and privacy-protecting solution would be to have integrated data catalogs and

multiple clearinghouses, but secure, multi-party computing across clearinghouses and common

standards to gain access to data, including:

● common researcher credentials

● peer review and pre-registration of research project proposals

● data use agreements

● required metadata fields

High quality data requires investment in curation. High quality analyses require investment in

training researchers and civil servants and providing them with up-to-date computing facilities.

Democracy requires that the data be well-protected. All of these require resources that have to

be provided by someone.

One model for self-sustaining data access is the consortium model. Most relevant to the types

of data discussed here is the Institute for Research on Innovation and Science (IRIS), a project

based at the University of Michigan’s Institute for Social Research. Started with funding from

external sources, IRIS’s model was to become self-funded by charging institutions (colleges and

universities) an annual membership fee. This fee provides the member institution with

campus-level reports based on their data, a seat at the table to help prioritize and design future

IRIS products and initiatives, and access to de-identified and aggregate IRIS data for

researchers on their campus. One benefit to universities is that, although they are required to

deposit data about their campuses with IRIS annually, the IRIS staff has automated the ability to

produce charts and reports based upon those data (possible because the same information
602 The Promise of Evidence-Based Policymaking

fields are collected from each institution). An organization or government agency that is required

to share data and/or provide reports based on those data can find that it is in their interest to

pay those with the skills and resources to properly support the data sharing efforts to carry out

those tasks rather than reinventing the wheel and building that capacity within each agency. The

data center providing a service such as creating reports or demonstrating use of the data within

the research community is seen by the data producer as an added benefit. Similarly, the

Inter-university Consortium for Political and Social Research (ICPSR) began as a consortium of

22 institutions in 1962 and continues the model with over 760 member institutions today. These

institutions pay an annual membership fee in exchange for access to data curated (and tools

created) using member funding as well as reduced tuition for students enrolling in the ICPSR

Summer Program in Quantitative Methods of Social Research.

It is important to remember, however, that there is no free lunch; if we want better data and

better analysis than currently exists, resources will have to be obtained to support this. The

value of this research may well provide the basis for self-financing, but it is more likely that such

research creates positive externalities without the ability to generate much revenue to support it.

It is critical that administrative and survey data held by government agencies be made available

to qualified researchers and institutions for scientific research and evidence-based analysis.

Such access provides the policy community with much greater resources for informing policy

decisions than if we rely exclusively on government analysis. It also increases the likelihood

that there is diversity in the analytical approaches brought to bear on important policy questions.

“Qualified researchers and institutions” should be established through external scientific peer

review of proposals and a system of researcher credentialing that creates an incentive for

researchers to be good data stewards. Because of the current lack of consistency across

agencies in defining these terms, ICPSR is undertaking a project to research, propose, and test

recommendations for researcher credentialing, the result of which will be a tiered set of

characteristics that describe “qualified researchers and institutions.” These characteristics will

stem from those currently employed/accepted by providers of restricted data, in so far as those

requirements are related to protecting against disclosure risk (i.e., not requirements put into

place to add bureaucracy or additional “hoops” that must be jumped for data access). We

anticipate using factors such as whether one has completed requisite training in ethical data

use, is employed at an accredited academic institution, has secured federal funding, and

proposes a project that is scientifically sound and that requires access to the data in question.

The ability to disseminate data using a variety of modes (providing metadata only, synthetic

data, use restricted to a physical or virtual enclave, or encrypted download) also allows for

flexibility in determining access. That is, if a researcher does not have accepted credentials, or

is not affiliated with an institution with appropriate technical and legal protections for data, a

researcher might still be allowed access to de-identified data. More sensitive data can be

restricted to access in a virtual or physical data enclave. In other words, the same data may be
The Promise of Evidence-Based Policymaking 603

made available to different researchers under different access modalities based on the

characteristics of the researcher and the sponsoring institution.

The integration (linking) of administrative and/or survey data in a clearinghouse without question

increases the risk of disclosure of entities within the data; however, the federal statistical

community and the research data community have a long history and reputation for protecting

confidentiality. This reputation must be maintained and protected by adhering to the Office of

Management and Budget’s Directive 1 (2014). Policies to maintain these protections and the

reputation of and confidence in the statistical agencies of the United States include:

● Providing access only to credentialed analysts with well-articulated research plans and

objectives

● Provision by the clearinghouse of disclosure review of output, notes and other materials

that are to be taken out of the clearinghouse (secure environment) to prevent unintended

disclosure of subjects within the dataset(s)

● Developing and implementing privacy preserving analytical techniques as well as

disclosure avoidance techniques such as creating synthetic populations that preserves

statistical information

Clearinghouses can and should require researchers or analysts to submit a detailed proposal of

the project for which the data are to be used, specifically addressing why the dataset in question

is necessary for addressing the research question. Once these are vetted, by scientific peer

review, clearinghouse staff and perhaps an external review board representing the data

producer and the study population, a conclusion can be drawn on whether the benefits of the

research project outweigh potential risks. Other restrictions should be consistent with the factors

listed above -- explicitly agreeing to use the data in an ethical manner (and potentially

demonstrating completion of training in doing so), restrictions on the computing environment in

which the data can be analyzed, agreeing to terms of use, and the like.

There are a number of private and governmental organizations that offer technological options

for data sharing and management. Colectica, a Minneapolis-based firm, is an example of a

research and development firm specializing in data management, integration services, and

Internet technologies for government, academic, and commercial computing; it offers a range of

highly specific products and services useful for supporting data sharing and management. They

offer tools for working with metadata using a variety of documenting standards (e.g., the Data

Documentation Initiative, DDI,). Colectica also has a portal that offers search, browse,

visualization, and data management capabilities.

Other projects exist that could offer either the technology or the functionality considerations that

would be helpful. One such project is the Sustainable Environment/Actionable Data (SEAD)

project, funded by the National Science Foundation and based at the University of Michigan.

SEAD provides a collaborative platform for researchers to curate their data as they undertake

analyses, so that the documentation is created and captured and can be harvested when the
604 The Promise of Evidence-Based Policymaking

data are shared (i.e., in a clearinghouse). A number of organizations, such as ICPSR, NORC,

and the Michigan Center for the Demography of Aging, use technology to create virtual spaces

in which researchers can analyze data that have significant disclosure issues. Generally, these

spaces require the researcher to log in to a server housed at the data provider, conduct their

analyses, and have output vetted before it is released to them. These virtual data enclaves often

disable connections to the internet, print functionality, email, and other programs/features that

might compromise data security. Lastly, software (e.g., Fedora) exists for creating and

managing digital repositories and could be employed by a clearinghouse.

Resources to train civil servants in state, local, and federal agencies to evaluate their own data

will also increase their capacity to learn from and absorb the analyses done by others. The

implementation of multiple randomized control trials could also reduce the inclination to limit

data sharing, as analyses can examine the question of which policies or interventions should be

supported at scale (not simply whether an individual policy or program should be continued).

Building continuous evaluation and improvement, based on progress toward measurable

objectives for the relevant population, into policy design provides programs with incentives to

collect and analyze data in order to identify potential improvements.

We should also work to develop a culture that highlights the intrinsic benefit that most civil

servants, researchers, and the general population receive from having better answer questions

about program effectiveness and other social issues. Researchers and civil servants will then be

more likely to suggest improvements to data collection (methods and/or content) that would

provide more effective analytic data to use in program evaluation. Researchers who are

analyzing data are also likely to catch anomalies or potential inaccuracies that might be missed

without researcher engagement. Having multiple researchers with multiple perspectives

working with the same data will support models that might be more robust than if a single party

were solely responsible for producing the evaluations. The research community’s embrace of

data transparency and replicability may provide reinforcement to governmental agencies to

adhere to similar principles. Sharing data among agencies and with researchers increases the

return on investments in data creation. It is more efficient use of government resources. It is

rarely the case that a single researcher or organization can study everything that can be

examined using a given data source. Differences in disciplinary perspectives mean that data

collected for one purpose might be seen by another investigator as having value for his/her

project that is completely different. Our statistical agencies employ dedicated civil servants who

value improvements in the quality of measurement that they produce for our country.

Recognition and respect for these values and these individuals will enable them to be more

effective and take the steps necessary to continuously improve our data infrastructure.

There are currently significant barriers in accessing and using such data, including challenges in

discovering the existence and location of appropriate data, uncertainty about legal infrastructure

and processes for providing access to data, lack of documentation of file contents or data

provenance. There are also limited resources for analyzing data (e.g., appropriate training for
The Promise of Evidence-Based Policymaking 605

government employees and non-governmental researchers, appropriate computing

infrastructure).

Simply using such information is the first step. Organizations may collect data from program

participants but not use it in evaluating the effectiveness of the program, may create summary

statistics based on the data but not move further into more sophisticated statistical models,

and/or may not be aware of existing research that could inform program/policy decisions. The

ability to link data sources provides an opportunity to put data about program participation into

context in ways that have not been possible before. For example, having information about

students’ performance for a given school by itself is helpful, but having the ability to link the

information to such things as parental earning records and teacher characteristics allows an

educational policy analyst to determine which shifts in student performance are likely a result of

new policies implemented at the school, characteristics of the school or teachers themselves, or

other issues related to outside influences such as food insecurity. Comparing data across

similar contexts or programs is helpful in that the similarities and differences between the

contexts create quasi-experimental designs, allowing researchers to identify the parts of the

program that are most effective and those where improvement might be needed. Making data

available to researchers also provides an avenue for dialogue between academics and

policymakers that otherwise might not exist.

Program and policy evaluation should be included in program design so that evaluation is based

on evidence that is available and analyzable by multiple, even competing, research teams, held

to standards of reproducibility so that all parties can learn from evidence as it accrues in the

process of program implementation.
606 The Promise of Evidence-Based Policymaking

Commission chair Abraham, Commission co-chair Haskins, other commissioners and Dr. 

Martinez, thank you for holding this hearing today and including me in it. 

The development of a statistical infrastructure that integrates administrative and program data, as 

well as commercial and other non-designed data, has enormous potential to provide the basis for 

improvements in our knowledge and understanding of the impact and effectiveness of alternative 

policies. This is an extremely important effort, and I commend you for working to build the

statistical infrastructure that we need to create an empirical, evidence-based foundation to 

undergird policy discussions. 

I’d like to emphasize today the important ways that collaborations between federal agencies and 

academic organizations can help to address several challenges in using administrative data for 

evidence-based policy making. 

First and foremost, collaboration with multiple, external academic organizations can help to 

assure data availability without excessive centralization that might compromise Americans’ right 

to privacy and security in the data that is generated by and about them in the course of their 

interactions with federal and state agencies. Confidentiality of individual data and the 

independence of the federal statistical system must remain paramount. Participation in our 

federal data programs, whether they collect survey or administrative data, is premised on the 

promise that individual data will remain confidential and will be used for statistical purposes 

only, and is never used for enforcement purposes or for the benefit of particular commercial or 

political interests. As articulated in the Office of Management and Budget’s Statistical Policy 

Directive No. 1 (2014), it is critical to “Protect the trust of information providers by ensuring the 

confidentiality and exclusive statistical use of their responses.” This is just as true when 

considering administrative data rather than the responses of survey participants. Undermining 

this trust undermines statistical measurement as well as the effectiveness of the programs which 

the statistics are intended to measure.

Collaboration with academic organizations can help to address this privacy challenge by 

providing the basis for a network of data resources that can be analyzed jointly, without 

concentrating data within a single federal agency. As an example, IRIS, the Institute for 

Research on Innovation and Science, a collaboration of dozens of universities based at the 

University of Michigan’s Institute for Social Research, is aggregating administrative data from 

those universities into a single data infrastructure. Those data can be shared with and linked to 

federal data assets, but they are produced and reside outside the federal government. Other 

academic collaborations are doing the same for administrative data from state and local 

governments on K-12 education, criminal justice, transportation, and land use. 

This model of a networked data infrastructure, based on collaborations between academic 

organizations and federal statistical agencies, allows us to reap the benefits from using 

administrative and program data, especially given the emerging capabilities of secure multi-party 

computing, without the potential threats to privacy that might be associated with a more 

centralized system. 
The Promise of Evidence-Based Policymaking 607

Second, collaboration between the federal government and academic organizations can bring to 

bear the efforts of the large number of faculty and students who would be more than willing to 

work to improve administrative data. This is particularly important as administrative data, like 

other non-design data, require significant investments in cleaning to assure that the target 

measurement concept is what is actually being measured. Investments in preservation and 

documentation are necessary to meet basic scientific standards of reproducibility; crowdsourcing 

improvements to data and metadata from the academic research community can do at least some 

of this. The creation of good documentation and metadata is critical to the effectiveness of a 

networked system, as it allows for the creation of a “union catalog” of data, and coherent 

analysis of data, even when those data resources are located in different places. Given the very 

real resource constraints of the federal statistical agencies, effectively leveraging these external 

resources is critical to the construction and design of a rigorous statistical infrastructure. 

Third, those same academic collaborators bring with them expertise in both measurement and 

analysis that contribute directly to the scientific rigor of the policy analysis. 

Fourth, collaboration with academic and other research organizations increases the likelihood 

that multiple analytical approaches are considered when evaluating a policy or program, 

avoiding any tendency for monolithic or even self-serving analysis. Open access to alternative, 

competing approaches to analytical questions provides the basis for legitimacy of the analysis 

that is done with administrative data and increases the public’s trust and willingness to have data 

about them and their activities used in this way. 

Finally, this kind of collaboration develops skills, both on the part of the civil servants on whom 

we rely for producing the critical statistical resources of our country and on the part of students 

who are the next generation of scholars and civil servants. The civil servants who make up the 

federal statistical system deserve our respect and appreciation. We need to elevate their status, 

and we do that, not by isolating them within the federal bureaucracy, but by allowing them to 

engage with their academic peers. This kind of collaboration will also produce a generation of 

students who better understand the challenges of measurement and policy-relevant analysis as a 

result of their participation in such a collaboration.

Thank you for your attention and consideration, and thank you for the work that you are doing to 

modernize our statistical infrastructure, the very basis of our knowledge of ourselves and our 

society.
608 The Promise of Evidence-Based Policymaking

1

Submitted December 6th 2016

Written Statement for the Commission on Evidence-Based Policymaking 

Public Hearing in Chicago, January 5th, 2016

Introduction

The Abdul Latif Jameel Poverty Action Lab North America (J-PAL NA), based in the 

Department of Economics at the Massachusetts Institute of Technology, leverages scholarship 

from 143 affiliated professors to generate and disseminate rigorous evidence about anti-poverty 

policies. J-PAL NA provides pro-bono technical support, capacity building, and matchmaking 

with researchers to government agencies and nonprofits seeking to design and implement 

randomized evaluations, many of which rely extensively on administrative data. Affiliates in our

network have conducted 154 ongoing or completed randomized evaluations in North America 

across sectors such as health care, housing, criminal justice, education, and labor markets. J-PAL 

NA also creates training materials to build research capacity, including a comprehensive, 

practical guide to obtaining and using administrative data for randomized evaluations.1 We 

appreciate the opportunity to submit a statement to the Commission on Evidence-Based 

Policymaking. 

J-PAL affiliated researchers have relied heavily on administrative data to conduct policy-relevant 

research. Data from IRS tax records enabled an almost 20-year follow-up of families involved in 

the Moving to Opportunity Experiment. The follow-up study demonstrated that young children 

who moved to low-poverty neighborhoods increased their college attendance and expected 

lifetime earnings.2 Data from the U.S. Department of Education, the Ohio Board of Regents, and 

the National Student Clearinghouse collectively enabled a randomized controlled trial showing 

that simplifying the financial aid application process increased college attendance and 

persistence.3 Data from hospitals in the Portland area revealed that Medicaid insurance, for 

which opportunities to apply were allocated through a lottery in Oregon, increased emergency 

room usage by 40 percent.4 Access to administrative data was critical to generating these 

insights.

Executive Summary 

This statement reflects J-PAL’s expertise concerning randomized evaluations, administrative 

data access, and collaboration between government agencies and external researchers. It 

 1 Feeney, Laura, Jason Bauman, Julia Chabrier, Geethi Mehra, and Michelle Woodford. “Using Administrative Data for 

Randomized Evaluations.” J-PAL North America. December 2015. 

https://www.povertyactionlab.org/sites/default/files/documents/AdminDataGuide.pdf

2 Chetty, Raj, Nathaniel Hendren, and Lawrence F. Katz. "The Effects of Exposure to Better Neighborhoods on Children: New 

Evidence from the Moving to Opportunity Experiment." NBER Working Paper #21156, May 2015.

3 Bettinger, Eric P., Bridget Terry Long, Philip Oreopoulos, and Lisa Sanbonmatsu. 2012. "The Role of Application Assistance 

and Information in College Decisions: Results from the H&R Block Fafsa Experiment." The Quarterly Journal of Economics 

127(3):1205-42. 4 Taubman, Sarah, Heidi Allen, Bill Wright, Katherine Baicker, Amy Finkelstein, and the Oregon Health Study Group. 2014. 

"Medicaid Increases Emergency Department Use: Evidence from Oregon's Health Insurance Experiment." Science 

343(6188):263-8.
The Promise of Evidence-Based Policymaking 609

2

incorporates recommendations from an open letter penned by several leading economists, 

including multiple J-PAL affiliates5

, and a short paper published by a subset of the same 

authors.6 This comment elaborates on these key recommendations:

 Establish clear data documentation and standard data request forms, building on the 

example set by the Centers for Medicare and Medicaid Services. 

 Expand secure access to real microdata to qualified researchers, prioritizing secure 

remote connections while also increasing capacity at Census Research Data Centers.

 Develop a data clearinghouse within the Census Bureau for currently hard-to-access data, 

particularly microdata on earnings and income, and link the data across agencies.

 Avoid flat per-user fees for data access to encourage validation and double-checking of 

data analysis.

 Clearly articulate program objectives and build ongoing process evaluation into every 

program to lay the foundation for impact evaluation. 

 Institutionalize a process for identifying questions for program evaluation and appropriate 

conditions for randomized evaluations, focusing on three cases:

o Demand for a program exceeds capacity to supply the program.

o Gradual roll out of a program to different individuals or locations over time.

o Refinement or reconsideration of eligibility criteria for a program.

Responses to Specific Questions Posed in the CEP Request for Comments

3. Based on identified best practices and existing examples, how should existing 

government data infrastructure be modified to best facilitate use of and access to 

administrative and survey data?

Build on the example of the Centers for Medicare and Medicaid Services to establish clear 

data documentation and standard data requests. 

Existing government data infrastructure should incorporate standard data request forms with 

clear data dictionaries, using the Centers for Medicare and Medicaid Services (CMS) Research 

Data Assistance Center (ResDAC) data documentation as a model of best practices. The 

ResDAC system allows researchers to understand specifically what variables are available and to 

submit requests with data protection plans. Because the ResDAC system allows CMS to review 

those requests systematically as opposed to on an ad hoc basis, ResDAC facilitates routine, 

secure access to administrative data that culminates in several hundreds of medical studies each

year.7

Applying the ResDAC model to an administrative data clearinghouse or other data repositories

would allow researchers to see exactly what variables they are permitted to request, along with a 

 5 Card, David, Raj Chetty, David Cutler, Steven Davis, Martin Feldstein, William Gale, Jonathan Gruber, Michael Greenstone, 

Caroline Hoxby, Lawrence Katz, and Emmanuel Saez. 2010. “An Open Letter on Expanding Access to Administrative Data for 

Research in the United States. https://eml.berkeley.edu/~saez/dataaccess_final.pdf 6 Card, David, Raj Chetty, Martin S. Feldstein, and Emmanuel Saez. "Expanding access to administrative data for research in the 

United States." American Economic Association, Ten Years and Beyond: Economists Answer NSF's Call for Long-Term 

Research Agendas (2010). https://eml.berkeley.edu/~saez/card-chetty-feldstein-saezNSF10dataaccess.pdf 7 Ibid.
610 The Promise of Evidence-Based Policymaking

3

brief description of each variable, before submitting a request. This explicit listing of available

variables would enable data discovery and save program administrators and researchers hours of 

valuable time by avoiding long correspondences about whether the desired data exist.

Publicizing exactly which variables exist and what agency houses the data increases transparency 

with no risk of revealing personally identifiable information. Better data documentation can thus 

facilitate use of and access to administrative and survey data without raising concerns for data 

security and privacy protection. 

Moreover, sensitive variables that would trigger additional levels of review or security could be 

clearly labeled as sensitive as part of this clearer data documentation. Currently, researchers may 

request a variable that is not central to their analysis, without realizing that it captures sensitive 

information. This could delay or jeopardize the entire request or allow access to sensitive data 

that, with clearer data documentation, would not have been requested from the agency.8

4. What data-sharing infrastructure should be used to facilitate data merging, linking, and 

access for research, evaluation, and analysis purposes?

Expand secure access to microdata to qualified researchers through remote and on-site 

connections rather than creating synthetic data.

An optimal infrastructure for integrating administrative, survey, and statistical data to facilitate 

research and evaluation while ensuring data security and privacy will provide secure 

environments where qualified researchers can directly access microdata. Microdata enable 

researchers to perform more informative analyses by controlling for individual characteristics 

(such as educational attainment or race) to better determine the impact of a program. Microdata 

also allow researchers to evaluate how a program affects specific subpopulations, such as lowincome individuals. Researchers can use microdata to validate and adjust their analysis as they 

learn from the data in real time—a crucial step in the research process. Moreover, for analysis in 

rigorous randomized evaluations, researchers require microdata to link individuals to their 

treatment status. 

There are currently twenty-four Federal Statistical Research Data Centers (RDCs), which are 

physical, secure environments established through partnerships between the Census Bureau and 

research institutions where researchers who have undergone special sworn status can access 

restricted microdata. However, capacity in these RDCs is limited, both physically and according 

to Census bandwidth, and access is artificially restricted to researchers based on geographic 

proximity rather than on the merit of their research proposal.

9 Similar constraints apply to 

researchers working with statutorily restricted tax data through contracts with the IRS Statistics 

of Income Division (SOI)—the type of arrangement that enabled the Moving to Opportunity 

follow-up study. The SOI is small, has a limited budget, and can accommodate few research 

projects at a time. 10 More secure, direct access to microdata should be provided in two ways: (1) 

 8 Feeney et al., “Using Administrative Data for Randomized Evaluations,” 2015. 

9 Card et al., "Expanding access to administrative data,” 2010b.

10 Office of Management and Budget, Barriers to Using Administrative Data for Evidence-Building (white paper submitted for 

the Commission on Evidence-Based Policymaking, July 15, 2016). https://www.whitehouse.gov/sites/default/files/omb/mgmtgpra/barriers_to_using_administrative_data_for_evidence_building.pdf
The Promise of Evidence-Based Policymaking 611

4

preferably through remote, secure connections such as the “flexiplace” systems used by federal 

employees who work with restricted data from home, but also (2) through expansion of on-site 

secure environments in the form of additional space and funding for Research Data Centers and 

similar centers at other statistical agencies.

11

Synthetic data, one alternative to expanding remote and on-site secure connections to restricted 

data, are a far inferior option for enabling policy-relevant research and program evaluation. 

Synthetic data are constructed to mimic certain features and aggregate characteristics of real data 

without containing real individual-level information. Although this appears—on its surface—to 

enable research while protecting privacy, synthetic data suffer severe disadvantages relative to 

real microdata. Synthetic data may be incompatible with randomized evaluations and other

rigorous program evaluations because researchers must be able to link individuals to their 

treatment status—i.e., whether a particular person received a program or not. Furthermore, 

synthetic data make it difficult or impossible to study subpopulations, such as low-income 

individuals, which may be of particular policy interest. Researchers would have to specify each 

subpopulation they intend to study and all necessary contents of the data in advance. This may be 

impossible, in part because researchers often revise their analyses to address observations they 

learn from the raw data.12 Meanwhile, data administrators would have to create new synthetic

datasets for each request to study a specified subpopulation, which would require significant 

infrastructure and personnel.

7. What data should be included in a potential U.S. government data clearinghouse? What 

are the current legal or administrative barriers to including such data in a clearinghouse or 

linking the data?

Establish a data clearinghouse within the Census Bureau for currently hard-to-access data, 

particularly microdata on earnings and income. 

For reasons discussed in response to question 4, the data should be real microdata rather than 

aggregated, de-identified, synthetic, or perturbed data. The clearinghouse should prioritize data 

that do not already benefit from strong infrastructure for access. Specifically, a clearinghouse 

should be developed for federal income and earnings microdata and focus on enabling 

researchers to link these data to the extent legally possible.

Income and earnings data have less well-developed access infrastructure and face several legal 

barriers to use for program evaluation, meaning that the clearinghouse would not be redundant.13

For example, individual states maintain their own data system for Unemployment Insurance (UI) 

records, with individual discretion and statutory protections on providing access to this data. The 

Department of Labor does not store the data in a central location. Although the Census Bureau 

has made a significant contribution to accessing state UI data through the Longitudinal Employer 

 11 Card et al., "Expanding access to administrative data,” 2010b. 12 Ibid. 13 Ibid.
612 The Promise of Evidence-Based Policymaking

5

Household Dynamics Program (LEHD), the LEHD program requires that researchers be on-site 

at a designated Research Data Center.14

With infrastructure secure enough for the highly restricted data from UI and tax records 

established, the clearinghouse should then focus on facilitating linkage of these data with other, 

less restricted data. Federal data on income, namely tax records or Unemployment Insurance 

records, are in high demand because income can serve as a key outcome variable for many 

government programs or policies in education, job training, criminal justice, and place-based 

interventions.15 For example, earnings—as measured by tax records—was a key outcome 

variable in the follow-up study of the Moving to Opportunity Experiment.

8. What factors or strategies should the Commission consider for how a clearinghouse(s) 

could be self-funded? What successful examples exist for self-financing related to similar 

purposes?

Avoid flat per-user fees for data access to encourage validation and double-checking of 

data analysis. 

As is customary, a clearinghouse may charge fees for accessing data, such as a fee per project, a 

fee for sets of users, or an initial fee for the first user followed by much smaller fees for 

additional users. It should not charge the same flat fee per person for accessing the data because 

this severely discourages the double checking crucial to correcting human errors. Despite its 

clear data request process, CMS charges a fixed fee of $25,000 per person who accesses

identifiable data through their Virtual Research Data Center.

16 This may create problems because 

researchers often need multiple people to work with the data to ensure accuracy—including 

people who effectively proofread to correct for human coding errors. With fixed per-person 

costs, researchers either pay a large inflexible sum of money for someone to double check the 

analysis or—facing tradeoffs given limited research funds—forego a set of “fresh eyes” to 

double check the analysis at risk of making mistakes. Therefore, by creating a high marginal cost 

to adding additional users, charging a fixed fee per data user effectively institutionalizes 

mistakes. 

18. How can or should program evaluation be incorporated into program designs? What 

specific examples demonstrate where evaluation has been successfully incorporated in 

program designs?

Clearly articulate program objectives and build process evaluation into every program to 

lay the foundation for impact evaluation. 

Process evaluation is always needed and constitutes a critical prerequisite for impact evaluation. 

It is not sensible to ask whether the program is succeeding or failing to deliver outcomes without

first knowing whether the program itself is being delivered with fidelity. In some cases, 

 14 Ibid. 15 Office of Management and Budget, Barriers to Using Administrative Data, 2016. 16 Research Data Assistance Center. “Fee List for RIFs: Physical Research Data Requests.” Centers for Medicare and Medicaid 

Services. https://www.resdac.org/sites/resdac.umn.edu/files/CMS%20Price%20List%20for%20Research%20Files_23.pdf

Accessed October 31, 2016.
The Promise of Evidence-Based Policymaking 613

6

important questions about how a program can or should function may be sufficiently answered 

by process evaluations, needs assessments, or literature reviews. Rigorous impact evaluation, 

particularly randomized evaluation, should be pursued when the benefits in terms of knowledge 

generated would likely outweigh the costs of the evaluation, and when planning during program 

design can facilitate impact evaluation. 17

Many components that aid process and impact evaluation should be developed during program 

design:

 Precisely articulated program objectives. 

 A needs assessment clearly articulating the problem that the program will address.

 Standard outcome measures used in research literature about similar programs that allow 

potential impact evaluation results to be compared to those in other studies and used in 

cost-effectiveness analyses.

 A plan for data collection and flow from program practitioners to administrators. This 

includes planning in advance to collect identifying information, such as Medicaid ID 

numbers, to enable later matching of program-level records to administrative records for 

impact evaluation. 

As an example of incorporating evaluation into program design, Benefits Data Trust (BDT) is 

working with J-PAL North America in an ongoing randomized evaluation of different outreach 

strategies to increase enrollment in the Supplemental Nutrition Assistance Program (SNAP) 

among eligible but unenrolled individuals in Pennsylvania. BDT had a clear grasp of the need its 

program addressed: despite awareness among eligible households that SNAP exists, many people 

could not imagine navigating the enrollment process alone. BDT had the clear program objective 

of increasing benefits enrollment, and change in program enrollment is a standard outcome that 

could be compared across different studies. BDT and researchers agreed that based on a review 

of the existing research, there was little rigorous evidence about what interventions can increase 

SNAP enrollment. 

Although BDT was already providing enrollment assistance and sending outreach, BDT worked 

with researchers to design and test two distinct outreach activities—one high-touch intervention 

including a letter plus enrollment assistance and one low-touch intervention including a letter 

only. The researchers also worked with BDT to design a new letter for the evaluation based on 

marketing and psychology literature. Seeing quickly that this newly designed letter was more 

effective, BDT plans to incorporate this letter design in other states outside of Pennsylvania. 

Ultimately, the impact of the different outreach strategies will be measured using administrative 

data, which can be accessed according to a data use agreement with the Pennsylvania 

Department of Human Services. 

19. To what extent should evaluations specifically with either experimental (sometimes 

referred to as “randomized control trials”) or quasi-experimental designs be 

 17 Glennerster, Rachel, and Kudzai Takavarasha. Running randomized evaluations: A practical guide. Princeton University Press, 

2013.
614 The Promise of Evidence-Based Policymaking

7

institutionalized in programs? What specific examples demonstrate where such 

institutionalization has been successful and what best practices exist for doing so?

Institutionalize a process for developing specific research questions and determining the 

appropriate conditions for randomized control trials or other evaluation methods. 

Federal agencies should institutionalize a process of developing high-priority research questions 

and determining the most appropriate evaluation methodology, following the precedent set by

the Social and Behavioral Sciences Team (SBST). SBST launches demonstration projects—

usually in the form of randomized evaluations—to rapidly evaluate low-cost applications of 

behavioral science to achieve desired outcomes, such as increasing workplace savings plan 

enrollment among military service members or increasing the rate at which indebted graduates 

apply for income-based loan repayment plans.

18

When properly designed and implemented, randomized evaluations rigorously demonstrate the

causal impact of a program by establishing the counterfactual—what outcomes would exist for

program participants if they had not received the program. Random assignment ensures that, 

with a large enough sample, the group that receives the program and the group that does not are 

similar on average before the start of the program. Therefore the impact estimate from a 

randomized evaluation offers confidence that any differences in outcomes between the two 

groups are a result of the program. The ability to isolate program impact from self-selection or 

other confounding factors is why randomized evaluations are widely recognized as a highly 

credible method for estimating program impact. Where there is little internal experience 

implementing randomized evaluations, agencies should seek partnerships with external or 

academic researchers who are vested in similar questions.

Randomized evaluations can only occur when randomization is built into the program design.

However, randomization should not be incorporated indiscriminately; rather, randomization

should be incorporated into programs to facilitate randomized evaluations where appropriate on 

three grounds:

 The current evidence for answering the well-defined research question is non-existent, 

insufficient, or inconclusive.

 There is a clear unit of randomization—individual program participants, schools, clinics, 

etc.—for which there is a large enough sample size and a clear means of tracking 

outcomes for both the treatment group and the control group.

 Randomization is feasible and ethical. Although not an exhaustive list, the following 

conditions offer opportunities where randomization may be feasible and ethical:

o Demand for a program exceeds capacity to provide the program. A lottery may be 

a fairer alternative than allocating slots on a first come, first served basis—

particularly when a goal of the program is equity of access—and offers an 

alternative to imposing increasingly narrow eligibility criteria under funding 

constraints. 

 18 Social and Behavioral Sciences Team. 2016 Annual Report. Executive Office of the President, National Science and 

Technology Council. September 15, 2016. https://sbst.gov/assets/files/2016%20SBST%20Annual%20Report.pdf
The Promise of Evidence-Based Policymaking 615

8

o A program is being expanded by gradually offering it to individuals, schools, or 

districts until full coverage is reached. A lottery can be used to randomly assign 

the order in which individuals or units receive the program. The individuals or 

units that have not yet received the program serve as the control group until all 

units receive the program. 

o A new intervention—such as a financial incentive or care coordination services—

will be added to an existing program. Program participants can be randomized to 

receive different versions of the program, e.g., with or without the added 

intervention, to isolate the impact of the new intervention.

o Program eligibility criteria are being refined or reconsidered. People just

above/below the eligibility cutoff can be randomly assigned to receive or not 

receive the program to determine whether it is effective for this marginal group. 

Meanwhile, those well within the program eligibility cutoff, automatically receive 

the program, and those well outside the cutoff do not qualify for the program.

As a specific example of institutionalizing a randomized evaluation, the South Carolina 

Department of Health and Human Services is partnering with J-PAL North America to 

incorporate a randomized evaluation in its expansion of a nurse home-visiting program for lowincome mothers delivered by the nonprofit organization Nurse-Family Partnership (NFP). The 

specific, high-priority research question is whether a new, less expensive version of the NFP 

program that South Carolina is expanding will be as effective as the pre-existing version that has 

been rigorously evaluated before. 

A randomized evaluation was found to be an appropriate method for answering this question 

given excess demand for the program. Although South Carolina is expanding this less expensive 

version of NFP to thousands of mothers through an innovative pay-for-success initiative, the 

program does not have sufficient resources to serve all of the women who are eligible. 

Applicants will be randomly assigned, on a rolling basis from 2016 to 2020, to either a treatment 

group that is offered access to the program, or to a control group that is not.19 We will assess the 

effect of NFP on a range of short- and long-run maternal and child outcomes using 

administrative data that will be available for all members of both treatment and control groups. 

This will yield useful evidence for South Carolina and for policy makers nationally, who are 

interested in the broader health and financial consequences of expanding Medicaid to include 

similar services.

 19 The Abdul Latif Jameel Poverty Action Lab. “The Impact of a Nurse Home Visiting Program on Maternal and Child Health 

Outcomes in the United States.” Accessed October 24, 2016. https://www.povertyactionlab.org/evaluation/impact-nurse-homevisiting-program-maternal-and-child-health-outcomes-united-states
616 The Promise of Evidence-Based Policymaking

Date: December 22, 2016

To: Commission on Evidence-Based Policymaking

From: Tom Schenk Jr, Chief Data Officer, City of Chicago

RE: Input on CEBP mission as defined in Public Law 114-140

Data privacy is crucial so Americans can trust the systems they rely upon for their well-being. 

People must have full faith in the ability to talk to their doctors, knowing that their details will 

not be reassociated with them again; students and parents must have faith that their grades, 

disciplinary records are only made to select few; and every member of the household must have 

faith that their responses to Census records will not be shared widely until many decades later.

Those protections should not inhibit their government from providing better, more

comprehensive services. The whole sum of social sciences research has demonstrated that 

outcomes in health, education, well-being and other areas are highly dependent on outside 

factors, such as nutrition, education levels of parents and guardians, family wealth, and many 

other factors. Often, researchers and policymakers need to account for external factors, such as 

these, to help understand program effectiveness.

Enabling evidence-based, data-driven policy is crucial for governments to be more efficient and 

effective for Americans. This is a progression of many steps where the bipartisan coalitions in 

U.S. Congress and the President have made several large steps to enabling greater data sharing 

with the explicit goal of improving education, workforce outcomes, health, and human 

services. The America COMPETES Act of 2007 and subsequent American Recovery and 

Reinvestment Act provided the targets and funding, respectively, to allow state governments to 

build longitudinal data systems which facilitate the ability to track students from the school 

system into the workforce. The recent authorization of Workforce Innovation and Opportunity 

Act (WIOA) has extended the call for linked data between education and the workforce to help 

job seekers.

The impact of these bills have been useful at every level of government. While heading 

institutional effectiveness and accountability for the Iowa Department of Education, we were 

able to use these longitudinal systems to conduct sophisticated analysis of the effectiveness of 

state-funded programs. Our team was able to calculate comprehensive rates of return to 

education for community colleges by each individual program for each individual college. By 

combining those records with state prison records, we could also determine the amount of 

diversion of costs by reducing the likelihood of crime. We could also determine the additional 
The Promise of Evidence-Based Policymaking 617

tax revenues the state earned by those student’s higher wages and lower expenses on other 

support programs.

Other research included sophisticated pseudo-experimental evaluations of science, technology, 

engineering, and mathematics (STEM) programs. These studies used sophisticated analytical 

techniques to follow students from middle school, through high school, and into college to judge

the ability of programs to increase math and science test scores, improve high school graduation, 

articulation to college, and the entry into STEM college majors.

As a senior researcher with the Department of Medical Social Sciences at Northwestern Feinberg 

School of Medicine, we used linked patient records to understand the long-term impact of cancer 

interventions on not only the progression of cancer, but also the quality of life for cancer 

patients. And today, as Chief Data Officer for the City of Chicago, we link multiple data sources 

together to predict where to send food inspectors, the next outbreak of rodents, and to allocate 

our workforce to pro-actively

The task set for the CEBP is also enormous. If privacy is compromised or if we are unable to

effectively link data to drive better decisions, it could setback progress on evidence-based policy. 

Based on my conversations with hundreds of residents on how governments use data, I have 

found that Americans are happy when this data is being used to improve lives and to make their 

government more efficient. However, we also know this optimism can be undermined by 

unethical uses of data or disclosing personal details of an individual. It is a careful balancing act 

that can be achieved, but takes considerable thought and attention.

II. Clearinghouse

One task laid out for CEBP is to “consider whether a clearinghouse for program and survey data 

should be established and how to create such a clearinghouse”. Single clearinghouses of data are 

often too big and too complicated of a task to be done well. We all have ideals of a single large 

repositories of all data that can be used by every researcher. However, we must take rational first 

steps and focus on “quick wins”. Attempts at creating websites for everyone often leads to 

creating something too confusing and ultimately not useful for anyone. 

Instead, a two-pronged approach should be taken for the Commission to meet its target by 

creating smaller, linked hubs and to create a service to allow for ad hoc data matching.

A. Setup smaller “hubs” as initial steps to clearinghouse

In order to provide a tractable mission for CEBP, the organization must focus on data matching 

problems to help answer specific domains of policy questions. For instance, congress has passed 

several pieces of legislation focused on linking education and workforce outcomes. There are 
618 The Promise of Evidence-Based Policymaking

clear outcomes that can be measured through these linked records. The initial hubs should focus 

on the highest priority questions, incorporating the most critical sources of data.

Additional hubs should be created to explore topics such as recidivism, health care, impacts of 

poverty, and other areas deemed of high importance. As new legislation is passed, Congress and 

the respective departments can continue to integrate data into these hubs so outcomes under the 

legislation can be effectively tracked and new research questions can be explored.

These hubs should continue to grow, but with a goal of eventually becoming interoperable 

themselves. This approach can help ensure that researchers get access to valuable data earlier 

while continuing to progress 

B. A service to link data and remove unique identifiers should be provided for researchers

Researchers need access to a variety of linked data but while balancing privacy. One way to 

achieve this is to create a centralized service which empower teams whose responsibility is to 

match personally identifiable data for the purpose of research and then discard personally 

identifiable data before it is provided to researchers. 

Of course, other data besides unique identifiers can compromise privacy. A unique combination 

of data—such as gender combined with age, race, and ethnicity—could allow someone to tie 

seemly anonymous data back to individuals. For instance, there may be a lone 32 year-old 

female immigrant in a ZIP code, so the release of that data could undermine privacy. Further 

steps on imputation, sometimes called “hot deck imputation”, are needed to mask personally 

identifiable data.

The United Kingdom's Data Services—operated by the UK's Economic and Social Research 

Council (ESRC), an organization akin to the United States' National Science Foundation—

provides similar services. The State of Florida’s Integrated Network for Data Exchange and 

Retrieval (FINDER) has also set a benchmark for such services.

There should also be efforts to capitalize on the existing open data movement. Federal 

government agencies and hundreds of state and local governments have launched open data 

portals which make data easily accessible without any barriers. Linked data—at its most 

summary level—would be a great addition to open data portals.

III. Barriers to data exchange

A. Data needs to be exchanged between levels of government
The Promise of Evidence-Based Policymaking 619

Policymakers are sometimes surprised to find their respective government does not have all of 

the data relevant to their operations. In contrast to wide belief, the federal government does not 

have access to all of the data available for local and state agencies. There are significant and 

legal barriers that prohibit governments and agencies to sharing data between levels of 

government.

Congress and government agencies can take steps to facilitate data sharing between localities and 

states, between states, and between Federal governments and other government agencies. For 

instance, WIOA requires the tracking of educational outcomes even though states are unable to 

track those outcomes—such as wages and state certifications—in nearby states. Legislation 

should encourage, if not require, these data sharing exchanges to facilitate research studies.

B. Guidance on existing privacy laws

Some of the limited ability to share data is simply caused by misunderstanding privacy laws, 

perceiving or misinterpretation an inability to share data when it is possible. There will always be 

value in providing further guidance for governments to clarify permissible data sharing between 

departments and governments.

IV. Limitations on data usage

These new ideas must also evolve our notions of ethical uses of data. Currently, legislation exists 

on whether some can or cannot have data. A future clearinghouse, hubs, and robust linked data 

needs to consider how data is allowed to be used.

Further guidance will need to clarify ethical and unethical uses of data, which should extend to 

whether researchers would be permitted to access and use linked data. Violating these terms 

should remove the researcher’s ability to be able to access linked data in the future.

V. Making research relevant and useful for policymakers

Research must be relevant to policymakers that answers their immediate questions and also be 

able to foresee other valuable research questions. A crucial aspect of relevant research is 

geographical relevance. National studies can indicate important trends, but often are not 

representative of each state or city or every participant. Research should be prioritized if it 

provides geographic breakdowns, breakdowns by subgroups, and other subsets deemed 

important for programs.

These summary statistics are incredibly useful for policymakers and should be frequently 

published. When data systems to talk with each other, agencies should still issue the reports that 
620 The Promise of Evidence-Based Policymaking

take technical details and summarize high-level statistics. These statistics should also be 

downloadable.

Researchers should be encouraged to improve their communication channels for policymakers. 

They should rely on recent developments in data visualization research and issue short-form 

abstracts to quickly communicate findings and become less reliant on long-form journal articles 

and reports.

VI. Rigorous impact analysis

Randomized control studies have been considered the “gold standard” of evidence-based policy. 

However, these studies are not always possible because of logistical constraints, ethical 

concerns, or would be too narrow to represent overall performance of a program. Meanwhile, 

other sophisticated techniques, such as pseudo-experimental methods like propensity score 

matching, difference-in-difference, and other methods can be used to measure causal impact of 

programs. The framework on evaluation should consider the “gold standard”, but also a “silver” 

and “bronze” standard of rigor.
The Promise of Evidence-Based Policymaking 621
622 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 623

TESTIMONY MATHEMATICA POLICY RESEARCH

1

Commission on Evidence-Based Policymaking

Using Evidence to Improve Programs and Inform Policy Decisions

Matthew Stagner, Senior Fellow

Mathematica Policy Research

January 5, 2017

For nearly 50 years, Mathematica has been dedicated to delivering high quality evidence and 

objective analysis to help policymakers and public program leaders uphold their missions to 

improve public well-being. As senior fellow and director of human services research in 

Mathematica’s Chicago office, I want to thank you for the opportunity to contribute to the 

important work of the Commission with my testimony today.

Earlier this month, Mathematica President Paul Decker emphasized the importance of 

evidence-based policymaking in a letter submitted during the Commission’s open comment 

period. Today, I will underscore several of Paul’s key points while also highlighting my own 

experience in evaluating policies and programs for vulnerable youth and at-risk families. I will 

focus my comments today on: 

 Effectively translating research evidence into actionable policy changes and program 

improvements

 Expanding access to administrative data in support of that research 

 Ensuring that evidence used in policymaking is rigorous

I begin by highlighting how research can, and should, be translated into actionable 

policy changes and program improvements. I have seen this happen firsthand. For example, 

Mathematica is working with the U.S. Department of Health and Human Services, Office of 

Adolescent Health to expand the use and understanding of evidence-based programs to prevent 

teen pregnancy. Over the years, we have identified dozens of these programs that have

demonstrated evidence of success in reducing sexual risk behaviors among adolescents. This is 

important because the federal government has invested millions of dollars to disseminate 
624 The Promise of Evidence-Based Policymaking

TESTIMONY MATHEMATICA POLICY RESEARCH

2

knowledge about these programs and to implement and evaluate them in communities around the 

country. Currently, we are designing new evaluations of teen pregnancy and other risk 

prevention programs that build on this growing portfolio of research.

In work we are doing for the Administration for Children and Families (ACF), we are 

studying interventions designed to help former foster children who are at risk of homelessness

and face significant challenges in the transition to adulthood. We provided evaluation technical 

assistance to state and local grantees as they developed comprehensive and innovative programs 

to support at-risk youth in making a successful transition to independent adult life. This project is 

helping strengthen these grantees’ interventions and evaluation plans, build their local evaluation 

capacity, assess the evaluability of their programs, and document the progress they make during 

the planning period. It also provides insight into potential designs and implementation of broader 

programs intended to reduce homelessness among former foster youth. A key lesson from this is 

that building evidence can take significant upfront work with programs to make sure we evaluate 

approaches that are innovative, well implemented, and significantly different from existing 

services.

Also for ACF, Mathematica is evaluating the effectiveness of maternal, infant, and early 

childhood home visiting programs, which are designed to achieve better outcomes for mothers, 

fathers, and their babies. Adverse birth outcomes, such as low birth weight and preterm birth, are 

more common in the United States than in other developed countries, and they extract emotional 

and economic costs from families and communities. Our studies are providing evidence on 

which programs are effective in improving children’s health and development, as well as other 

outcomes.

Conducting this work effectively, however, requires expanded access to sophisticated 

administrative data. For child welfare agencies in particular, the ability to link administrative 

data across systems and apply them to decision making can have a real impact on vulnerable 

children and families. Although administrative data give agencies the ability to inform program 

development and improve outcomes, expanded access comes with challenges, such as 

maintaining confidentiality and understanding the meaning and usefulness of data in the various 

human services systems. Learning to use these data effectively and efficiently requires expert 

guidance from state and local leaders who specialize in data systems, policy, and practice. 

In one project for Casey Family Programs, we are examining administrative data in two 

states to identify children and youth who are heavy users of child welfare, Medicaid, and other 

services. Identifying subpopulations of children and youth who are continuously using intensive 
The Promise of Evidence-Based Policymaking 625

TESTIMONY MATHEMATICA POLICY RESEARCH

3

services may shed light on those who lack support at critical junctures, are placed in overly 

restrictive environments, receive too many ineffective services, or are in need of better and 

different ways to meet their extensive needs. We expect this project to help child welfare and 

allied agencies achieve better outcomes for youth and families by tailoring more effective 

services to them earlier. We also expect that our findings will demonstrate the need for service 

coordination across systems to identify better strategies to serve these children.

Our project team has encountered some barriers in accessing the data, however. First, we 

live in a world of decentralized data, where each state or county collects and stores data in a 

different way. This leads to the need for many months of work to understand each data set, what 

it can provide for analysis, and what may be missing or inaccurate. Second, the privacy 

protection laws—important for preventing the misuse of information—can create long delays in 

access for legitimate policy research. We should seek ways to standardize data across 

jurisdictions, while allowing for some state flexibility, and also find ways to fast-track policy 

research requests while continuing to protect privacy.

Evidence used in policymaking must be rigorous, using a strong research methodology 

and starting with a carefully designed research question. I also support Paul’s view that too 

often, policy is made using subjective judgment or poorly designed research. While randomized 

controlled trials (RCTs) are the “gold standard” of research methodology, high quality research

can also encompass a wide range of research methods. Program managers seeking to refine a 

program may use descriptive or non-experimental analyses that help generate hypotheses about 

what is working and what is not. They may also use machine-learning methods and big data to 

predict which services might be most effective for each program participant based on his or her 

characteristics and needs. These managers could then propose changes to their program and 

evaluate these changes using an RCT or a quasi-experimental design (QED) before deciding to 

roll out the change widely. 

For another ACF project, I am working with a team of researchers to explore the potential of 

advanced research methods and data to address unanswered questions about the incidence of 

child maltreatment. Although research on child maltreatment has advanced substantially over the 

past 20 years, new data and innovative research tools make it possible to better understand the 

incidence of maltreatment and related risks and, in turn, to improve practices and policies. Our 

findings will inform the direction of future research. We will draw on existing administrative 

data, innovative methods, and advanced statistical techniques by identifying and prioritizing key 

research questions and exploring innovative methodological approaches. In each design option, 

we will discuss primary research questions; describe how to access, use, and link relevant data 
626 The Promise of Evidence-Based Policymaking

TESTIMONY MATHEMATICA POLICY RESEARCH

4

sources; identify relevant survey items and statistical analysis plans; pre-test instruments; 

estimate statistical power, respondent burden, and needed resources; and discuss the barriers, 

challenges, and feasibility.

The common thread running through all of the points and examples cited above is the 

importance of strengthening partnerships between the research community and the public policy 

and program communities. Working together more effectively and efficiently will help us 

achieve our shared mission to improve public well-being. Researchers can provide critical 

insights at all stages of policy and program design and improvement. In the same vein, 

government agency staff can be important partners throughout the research process by 

contributing vital perspectives on identifying and implementing evaluations and providing access 

to key data. 

Fostering a culture of evidence throughout our federal, state, and local governments is 

essential to ensuring that policies and programs are developed and refined to meet the needs of 

our fellow citizens. I am honored to be able to present this testimony to the Commission today, 

and we at Mathematica look forward to following the Commission’s work in the months ahead. I 

am happy to respond to any questions about my remarks today. Thank you.
The Promise of Evidence-Based Policymaking 627

1

College of Liberal Arts

& Sciences

RESPONSE TO COMMISSION QUESTIONS

Donna K. Ginther

Professor, Department of Economics

Director, Center for Science, Technology & Economic Policy

University of Kansas

and

Research Associate

National Bureau of Economic Research

Commission on Evidence-Based Policy Making

February 10, 2017

Introduction

Chair Abraham, Chair Haskins, and distinguished members of the Commission, you asked me to 

address questions based on my previous testimony. My response is detailed below. I conclude 

with a discussion of logistical hurdles associated with access to these confidential data and a 

suggestion for how to provide access while maintaining confidentiality.

1. What are specific, actionable recommendations for how policymakers can be 

incentivized to encourage evidence-based policymaking? 

This is a very hard problem. As I mentioned in my testimony, no bureaucrat has ever lost 

his or her job by saying “no” to a researcher request. Furthermore, policy-makers at the highest 

levels of federal agencies need to advocate for evidence-based policy. Essentially, evidencebased policy clashes with typical management decisions because a policy-maker needs to be 

willing to risk learning about bad outcomes. To fully take advantage of evidence-based policy, 
628 The Promise of Evidence-Based Policymaking

2

the federal government needs to promote a culture of learning from evidence. Policies should be 

implemented that do not punish policy-makers and administrators for revealing “bad news” 

about a given program. Models of these types of changes can be found in evidence-based 

medicine and evidence-based management. Jeffrey Pfeffer and Robert Sutton tackled the 

challenges confronting managers and policy-makers in their book, Hard Facts, Dangerous HalfTruths And Total Nonsense: Profiting From Evidence-Based Management. My 

recommendations are based in part on their research. 

First, evidence-based policy requires evidence. The federal government should provide 

funding for evidence-based research via the NIH and NSF. The Science of Science and 

Innovation Policy (SciSIP) at NSF provides a good model for how science agencies could fund 

evidence-based policy research. SciSIP was created in response to John Marburger’s call for 

using evidence-based policy to understand the economic impact of the federal government’s 

scientific investments. The science agencies have existing infrastructure that can be used to 

facilitate evidence-based policy. For example, federal research funding could be used to support 

the use of the data clearinghouse. In addition, the peer review process at the funding agencies 

can be used to identify qualified researchers and projects. The non-science federal agencies 

should be allocated funds for the evaluation of their programs and have periodic reviews of their 

programs in order to facilitate improved outcomes of federal investments. These programmatic 

reviews should be conducted by individuals outside of the agency (for example, researchers who 

have had their projects peer-reviewed) in order to provide an unbiased assessment of the 

program. 
The Promise of Evidence-Based Policymaking 629

3

Second, policy-makers in the federal government’s Senior Executive Services (SES) need 

training and professional development in evidence-based policy and management. According to 

Pfeffer and Sutton, “The Centre for Evidence-Based Medicine says that identifying and applying 

effective strategies for lifelong learning are the keys to making this happen for physicians. The 

same things are surely critical to evidence-based management.”

1

 The Office of Personnel 

Management offers several executive development courses for members of the SES including 

formal training, mentoring and executive coaching. The curriculum could be expanded to 

provide courses about using evidence in decision-making, an overview of research methods 

associated with evidence-based policy, and how to assess the quality of research studies that 

inform evidence-based policy. 

Third, as evidence-based policy evaluations are put in place, a “negative outcomes” 

reporting system should be developed to facilitate learning from the unintended consequences of 

various policy initiatives. To incentivize admissions of “negative outcomes,” policy-makers that 

report these unintended consequences would be held harmless as long as they take proactive 

measures to address policy shortcomings that they have identified. This system could be 

modeled on medical models of patient safety reports or the Aviation Safety Reporting System. 

The goal of a reporting system would be encourage the admission of negative outcomes and limit 

assigning blame. Using “negative outcomes” as additional evidence, policy-makers could then 

recalibrate policies with the goal of improving outcomes.

Finally, the federal government should encourage and fund experiments and prototypes 

before making wholesale policy changes. Although controversial, the Affordable Care Act 

(ACA) provides a good example of how this might work. The ACA was based on 

 1 Harvard Business Review, January 2006 https://hbr.org/2006/01/evidence-based-management.
630 The Promise of Evidence-Based Policymaking

4

Massachusetts’ universal health insurance law that was enacted in 2006 and the Oregon 

Medicaid expansion lottery. The evidence from these state policies informed the implementation 

of the ACA. Although no policy is perfect, the ACA has provided health insurance coverage for 

over 18 million, previously uninsured people. As Congress debates changes to the ACA, an 

evidence-based approach would slow the process of changing the program in order to experiment 

with changes for the purpose of developing evidence on what works and what does not. For 

example, states could be granted waivers to experiment with changes to the ACA. After three to 

five years, the evidence associated with these state waivers will accumulate to show how the 

ACA can be improved prior to making wholesale changes to national policy. 

2. What other datasets, if any, would facilitate your work that are not currently accessible 

to you?

The federal and state governments collect terabytes of data that would be useful for my 

research purposes. In general, I am in complete agreement with Commissioner Troske’s 

argument that if the federal government funds the collection of administrative data, then these 

data sets should be part of the data clearinghouse. Below I list data that would facilitate my 

work. While I have had limited access to some of these data sets, broader access would further 

future research and evaluation for me and may others.

First, I recommend that the commission prioritize the creation of unique Protected 

Identification Keys (PIKs) for all survey and administrative data that are included in the data 

clearinghouse. Since almost all of my research and evaluation depends on linking individuals 

that appear in several data sets, this would facilitate improved research and evaluation. Specific 

data by research subject include:
The Promise of Evidence-Based Policymaking 631

5

K-12 Education and Social Safety Net Data

• The U.S. Department of Education funded the creation of the Kansas Individual Data on 

Students (KIDS), a student longitudinal data system (SLDS). These SLDS data for all 

states should be available to the federal government and researchers. 

• Administrative records on state and federal social safety net programs could be linked to 

student records. Linking these data to student records would allow researchers and policy 

makers to understand the impact of social safety net policies on child’s educational 

outcomes. These include administrative records for: 

o Temporary Assistance to Needy Families (TANF) 

o Supplemental Nutrition Assistance Program (SNAP)

o Earned Income Tax Credit (EITC)

o Women, Infants and Children nutrition (WIC)

o Supplemental Security Income (SSI)

o Head Start and other Pre-K program participation records

o Medicaid Records

o Childcare assistance records

o Federal housing applications, support and vouchers. 

• The ability to link employment records of parents from the LEHD and child support 

payments to student records in SLDS would allow researchers to understand the effect of 

income on child outcomes. 
632 The Promise of Evidence-Based Policymaking

6

Higher Education Data

• Student administrative records from systems of higher education (e.g. the Kansas Board 

of Regents). These data could be linked to the LEHD to understand the economic return 

to investments in higher education. The University of Michigan Institute for Research 

on Innovation and Science (IRIS) has these records for about 20 universities. Having this 

information for federally funded colleges and universities would facilitate a deeper 

understanding of the private returns on the education investment as well as the impact of 

federal funding of graduate training in science and engineering.

Science Policy Data

The scientific agencies have a treasure-trove of administrative data that would facilitate 

the science of science policy studies. These data include:

• NIH IMPAC II extramural funding applications and awards data. It would be useful to 

create the ability to link principal investigator IDs to their publications in PUBMED.

• NSF Enterprise Information system data on applications and awards.

• Similar grant administration data from the Department of Defense, Department of 

Agriculture and Environmental Protection Agency

• Data from SciENcv: Science Experts Network Curriculum Vitae and My Bibliography 

from the National Center for Biotechnology Information.

• Information from iEdison that links patents to federal funding.

• A crosswalk that links scientists to their federal research funding IDs to publication 

record IDs such as Clarivate Analytics Researcher ID or ORCID.

• A crosswalk that links institutional IDs in federal funding research databases (e.g. the ID 
The Promise of Evidence-Based Policymaking 633

7

for the University of Kansas in NIH IMPAC II) to the Department of Education’s 

Integrated Postsecondary Education Data System.

• Information on state of residence and employer industry in the Science and Engineering 

Statistics Data System.

Internal Revenue and Social Security Earnings Records

• The ability to link Internal Revenue W-2 records and Social Security Administration

disability and retirement records to federal survey data including the American 

Community Survey, Current Population Survey and Survey of Income and Program 

Participation data would facilitate improved estimates of labor market outcomes.

3. Logistical issues Associated with Access to these data.

Although the Commission did not ask me to comment on access to administrative data, 

this is a pressing issue, and I would like to share my experiences. The Commission would like to 

preserve the privacy and confidentiality of these administrative records. However, we have 

encountered significant problems with existing models of confidential data access. The Research 

Data Center (RDC) network has regularly experienced severe connectivity issues with accessing 

data housed at the Census Bureau. These include losing access to Census data for hours or an 

entire day as well as problems with password resets and security questions. In the programming 

environment, large programming jobs can shut down an entire RDC. There is no plan to expand 

the number of computing nodes at Census headquarters despite the addition of six RDCs this 

year. Thus, the RDC model as it currently exists would be over-burdened by the additional 

demands of supporting a data clearinghouse.
634 The Promise of Evidence-Based Policymaking

8

One alternative to the RDC model is the NORC Data Enclave—a kind of private version 

of the RDC that allows researchers to connect to data secured at NORC via VPNs and secure 

laptops. Although this method provides more flexibility than the RDC, it is plagued by slow 

response to user requests and an inability for researchers to easily upload programs or data to the 

secure system. 

One promising alternative would be to create a series of synthetic databases that could be 

used for research development purposes. Once researchers are satisfied with their analysis on 

the synthetic data, they could send their analytical files to Census or the Data Clearinghouse for 

validation on the internal data. A version of synthetic Survey of Income and Program 

Participation (SIPP) data matched to IRS and SSA records is now available 

(http://www.census.gov/programs-surveys/sipp/guidance/sipp-synthetic-beta-data-product.html).

Conclusion

Once again, I would like to thank the Commission for allowing me to testify about 

evidence-based policy. Researchers’ access to administrative data has the potential to 

revolutionize policymaking and improve the functioning of all levels of government.
The Promise of Evidence-Based Policymaking 635

• In your written testimony, you urged the Commission to investigate the various laws and

regulations governing access to administrative records for research purposes. What are some

specific examples of cases where Federal or state law or ambiguity around Federal or state

law can be a barrier to accessing data?

The basis for my comments, as noted in a footnote in my testimony, is an National Research

Council report, Reengineering the Survey of Income and Program Participation, edited by

Constance Citro and John Karl Scholz. I attach a copy of this Report. I was a member of the NRC

Expert Panel that produced this Report. The section of the report relevant for the Commission’s

question to me is Chapter 3, “Expanded Use of Administrative Records,” pp. 58-64 in particular.

This section addresses the following question:

If you wanted to link state records for social programs, e.g., TANF, Medicaid, UI, 

Workers’ Compensation (WC), and other case benefits, to a Census Bureau data set like 

the SIPP, are there legal barriers to Census acquiring these data? 

To answer this question, the Panel commissioned a study by Professor Douglas J. Sylvester, who 

is the dean of the Sandra Day O’Connor College of Law at Arizona State University. I attach a 

copy of his final report on his findings. The pages noted above in the attached NRC report 

summarize his findings. Sylvester focused his analysis on 3 categories of state laws, constitutions 

and judicial rulings concerning confidentiality and access provisions that applied to 4 “sets” of 

programs (TANF, Medicaid, UI & WC, & other Cash Benefits) for each of the 50 states, giving a 

total of 200 = 50 x 4 state-programs. The 3 categories were: 1. Ready Access; 2. Restricted 

Access; 3. No Access. (See p. 59 of the NRC report for definitions and Table 3-1 on pp. 60-62 for 

the summary results.) The Sylvester study concluded that 113 of the state-programs fell in 

Category 3 of No Access. This finding is the basis for the comment I made in my testimony that 

the Commission has noted. 

There are a host of qualifications to the above conclusion about access to state administrative 

records. Many are made in both the NRC report (pp. 63-64) and in the Sylvester final report. One 

of them is that the analysis undertaken by Professor Sylvester was not asked to examine 

provisions in federal law (see p. 3/22 in the Sylvester report) and I have nothing more I can say 

with any authority about the federal laws and titles that apply to this issue, beyond what the 

Commission already knows about such laws as CIPSEA, HIPAA, FERPA and the titles that govern 

statistical agencies like Census and the IRS. But I would draw the Commission’s attention to pp. 

18/22 of the Sylvester Report concerning the SSA Access to State Records Online (SASRO) 

operation that provides SSA to access to state-held records for various state-level programs like 

TANF, Medicaid and others. As I understand it, SASRO is a data-sharing program is for the 

purposes of the administration of SSA programs; I don’t know that it covers uses “for research 

purposes.” 

Also, Sylvester notes on p. 18/22, several federal regulations (20 C.F.R. 401.110) apply to the 

sharing of data between SSA and the Census Bureau for “purposes of planning or carrying out of 

a census or survey or related activity...” Again, I don’t have sufficient expertise to say anything 

Hotz -- QFR's 
636 The Promise of Evidence-Based Policymaking

more about these provisions in federal statutes about data sharing, but I note them as potential 

federal provisions that seem relevant to the issue at hand.

• What datasets do you think are most vital for research and as such should be included in a

federal data clearinghouse if the Commission were to recommend one be developed?

My preferences would be for the following data and administrative records to be included:

(1) SSA Master Earnings File and Benefits Files for all of the programs it administers (OASDI, SSI,

etc.)

(2) IRS Records on Earnings, claiming for EITC and college attendance

(3) TANF and General Assistance from states, including non-cash benefits (e.g., child care and

training)

(4) SNAP, WIC, CSFP and other food/nutrition programs

(5) Section 8 and other housing programs, including Low Income Home Energy Assistance

(6) Medicare and Medicaid (presumably from CMS) and SCHIP

(7) Child care (Head Start) and other programs

(8) UI and Worker’s Compensation Records (from state)

(9) Death records

I also think that attention needs to be paid to getting other sources of data, including nonfederal data that can be linked either to governmental admin records or surveys. These include:

(10)Credit Score data

(11)Student Debt (federal guaranteed and non-guaranteed)

(12)Housing data on housing values, foreclosures, etc.

(13)Health data, such as from Electronic Health Records from health care provider systems

(14)Place-based data like neighborhood crime data, access to food outlets by quality (e.g., food

deserts and swamps) and access to other vital services (heath care facilities), “neighborhood 

walkability”. The key for these data is being able to link it with to admin records of 

individuals and/or survey respondents by location. 

These data are needed to assess the impacts and effectiveness of various federal (and state) 

policies.

I will stop here. Let me know if there is more you want on this issue.
The Promise of Evidence-Based Policymaking 637

1 

TO: Commission on Evidence-Based Policymaking

FROM: Virginia Knox, PhD 

MDRC, Director of Policy Area on Family Well-being and Children’s 

Development 

RE: Response to follow-up questions

DATE: February 10, 2017

Thank you for the opportunity to provide additional input for the Commission’s consideration. 

My responses are aimed at acknowledging the solid foundation that Congress has built for 

evidence-based policymaking and to indicate the most critical elements to include in future 

legislation. The discussion below draws on MDRC’s forty-plus years of working with a variety of 

federal agencies to learn what works to improve the well-being of low-income individuals and 

families. 

• What specific, actionable recommendations do you have for the Commission

related to building a culture of continuous improvement and building on tiered

evidence strategies employed at ED, HHS, and Labor? What resources would be

most helpful to agencies?

Congress has already begun to build a culture of continuous improvement and to expect 

agencies to rely on evidence to make funding decisions. Therefore, some of the most 

actionable recommendations are a continuation of current trends in legislation that has used 

tiered evidence approaches, such as the Education Innovation and Research program that was 

created through the Every Student Succeeds Act. Congress can support evidence-based 

policymaking by clearly stating that: 

a) Independent evaluations of programs and policies in the federal government are critical

to making informed decisions about how federal resources are spent. This includes

building a body of evidence over time that is aimed at informing high priority questions

about the most efficient and effective allocation of public resources and design of

policies and programs.

b) In fields where there is a rigorous evidence base upon which to make funding decisions,

the majority of federal funds should be used for policies and programs that have

evidence of effectiveness. Where there is not an adequate evidence base, or where

there are important gaps in the evidence base, federal funds should be accompanied by

evaluation requirements so that a base of evidence is built to inform future decisionmaking.
638 The Promise of Evidence-Based Policymaking

2

c) Even in fields that have a substantial base of evidence, federal funds can provide 

incentives for continual improvement and innovation to address newly emerging issues,

by allowing a proportion of federal funds to be used for promising programs that need 

more research to determine their effectiveness.

d) Rigorous evaluation should be used to assess the impacts, benefits, and costs of policies 

and programs.

e) Evaluations should aim to not just measure current program effects but also to inform 

future improvement and decision-making. This can be done through research designs 

aimed at learning from variation in impacts where possible and by including

implementation studies that examine issues such as the fidelity of programs to the 

intended policies or program models; local innovations and adaptations; the 

infrastructure needed to scale up the policy or program; and the mechanisms by which 

the program has its effects. 

f) The tiered evidence strategy described above can be applied to existing funding streams 

by using the above principals and setting aside funding for research and evaluation. 

ESSA’s provisions that reformed Title I are an example of movement in this direction. 

Other funding streams have also used research set-asides to help build a body of 

evidence that accumulates and provides newly needed information over time. These are 

typically 1% or more, depending on the size of the funding stream and the state of 

evidence in the particular field. Set-asides should preferably be structured so that the 

programs and the research that can improve their effectiveness over time are not put in 

the position of competing with each other for scarce public resources.

Evaluations using these set-asides can play several critical functions, including 

understanding the effects of new policies or programs; providing data-driven guidance 

about how to improve existing policies and programs over time; and to refresh the 

evidence base for programs in which the social and economic context has evolved since 

the original research was conducted. 

• Can you provide examples of where the “special rights clause” was used to limit 

the public release of the results of a program evaluation?

In our experience, use of the special rights clause hampers efficient and effective evaluation. 

Federal contracting provisions governing data ownership establish the agency/contractor 

relationship in evaluation research, specifically how both quality control and independence are
The Promise of Evidence-Based Policymaking 639

3

maintained. Federal agencies have several options to strike the appropriate balance between 

the need for accountability on the part of the government and the need for independence on 

the part of the social policy evaluator. Use of the special rights clause (52.227-17) is an exercise 

of governmental control that undermines any claim to independence that is vital for social 

program evaluation credibility. 

In our experience, it is usual for agencies to claim that its rights over the content and 

dissemination of reports derive from the special data rights provision. This provision prohibits 

the contractor from using, releasing and publishing any data produced during contract 

performance without prior written permission of the contracting officer. The provision, 

according to which government is given unlimited rights to all data, including copyright and 

release, is associated with restrictive governmental control over the contractor’s work 

products: line editing and writing of reports prepared by the contractor; delays in the release 

of findings to accommodate politically-motivated review; the ability to forbid release; and, the 

assertion of total control by the government over individually identifiable survey and 

administrative data that the contractor is collecting as part of its impact analysis. However, this 

exercise of governmental control undermines the independence and transparency that is vital 

for social program evaluation credibility. And it is contrary to the purpose of the special rights 

clause, which is to govern the production of data for the internal use of government. 

Further, the clause’s severe restriction on publication raises major issues for work with 

universities, which hold academic freedom principles that do allow work on federal contracts 

including the special rights clause. We have witnessed considerable inefficiencies in the use of 

federal resources for evaluation because universities who would have been the highest quality 

or lowest cost partners in a federal contract were unable to sign a subcontract that includes this 

provision. This has led to delays or increased management costs in two different ways: We 

have held protracted discussions about mitigating the effects of this clause so that the 

university is able to be part of the research team, or the university partner has simply had to 

leave the planned team and we have had to spend time and resources identifying other 

partners.

There is a solution to this problem. The general rights in data clause contains “alternate 

provisions” that are mandated in contracts for basic and applied research with universities or 

colleges and are permitted in other contracts upon agency determination that the alternate 

provision is appropriate. Federal agencies can ensure research independence by using 

Alternate IV of the general rights in data clause of the FAR (52.227-14 Alternate IV), rather than 

the special rights clause, in policy evaluation contracts.
640 The Promise of Evidence-Based Policymaking

4

Alternate IV mandates agencies to allow contractors to copyright any data first produced “in 

contracts for basic or applied research...to be performed solely by universities and colleges.” 

Alternate IV also permits agencies to use the same provision in other contracts when the 

agency determines that the contractor should have permission, the same as given academic 

institutions, to claim copyright in any data produced during contract performance. This is 

blanket permission, extending beyond the limited permission for scientific and technical articles 

already set forth in 52.227-14. 

Alternate IV directs agencies to loosen the restrictions that apply to contractors under the 

general rights in data clause when those contractors are colleges or universities performing 

applied or basic research OR when the agency determines that similar treatment is warranted 

for other contractors. And, of course, the section recognizes governmental interests by 

awarding an unlimited license to government for all data other than computer software.

Government contractors performing evaluations are conducting applied research. The interests

of research organizations like MDRC in producing high quality independent evaluations and 

disseminating results are no different from academic institutions that are engaged in the 

conduct of applied research. Governmental interests in ensuring that the evaluation is 

conducted independently, with quality controls imposed by the sponsoring agency, can be 

rigorously advanced regardless whether the evaluation is conducted by a government 

contractor or a university.

With Alternate IV as the baseline, agencies can use Section H of the contract, which contains

particular clauses related to contract performance, to set review requirements consistent with 

appropriate oversight, quality control and the goals of the general rights clause. For example, 

the agency can stipulate in the contract a reasonable period of time for agency review of the 

evaluator’s findings, a process for making comments and suggestions regarding the contractor’s 

reports, and an opportunity for the agency to attach to the final report any dissenting opinion 

regarding the contractor’s findings. Such conditions regarding the use or release of data can 

also be inserted into the contract’s scope of work or in an appendix. These provisions, tailored 

to the specific context of a research project, can accommodate the agency’s need to ensure 

that its views are incorporated into a final report with the evaluator’s need for independence in 

producing the report.

In sum, contracts drafted using Alternate IV advance the objectives of research independence 

and reliable evidence unfettered by agency censorship or interference and allow for efficient 

partnerships across the research community. 
The Promise of Evidence-Based Policymaking 641

5

• You described several administrative and procurement obstacles that evaluators 

face. Which one or two do you see as the biggest barriers to expanding evaluation 

capacity? 

Two obstacles are fundamental.

a) It is of critical importance that Congress affirm that data that are collected for research 

purposes may not be used by federal agencies for any other purpose, including 

enforcement activities. As a general rule, federal agencies should not require the transfer of 

individually identifiable data from the research contractor to the federal agency during or 

after contract performance. Sometimes, however, such a transfer is necessary or beneficial 

for the work, and when that is the case, the government must be able to provide assurances 

to study participants that data collected for research purposes will not be used for any nonresearch purpose, including enforcement.

Identifiable data should not be transferred to the government because such disclosure has

the unintended consequences of reducing the completeness, validity, and value of the 

research to the federal government. As required by human subjects regulations and ethical 

principles for research, research contractors must inform potential research subjects of how 

their data is to be used. Researchers want to be able to assure potential subjects that the 

data the subject provides or authorizes for collection will be held confidentially, to the 

extent permitted by law, by the researcher and the researcher alone. Requiring that the 

research contractor inform prospective research subjects that their private information may 

be provided to a federal agency is highly likely to have a chilling effect on the ability of the 

research contractor to enroll study participants or, once enrolled, to collect reliable and 

complete answers to survey questions and access to other private information. Once in the 

possession of the governmental agency, subjects’ individually identifiable information might 

be subject to a myriad of unrelated government purposes, and even public release under 

the Freedom of Information Act (FOIA). 

On occasion, there is a research need for a federal agency to handle the identifiers of 

individuals who have agreed to participate in an independent evaluation. This has come up 

either because an agency can help the researchers gain access to administrative data that 

the agency possesses or because for procurement reasons the agency wants the flexibility 

to continue the study with a different contractor, and to accomplish this they need access 

to the identifiers for members of the research sample. In these circumstances, there is an 

inherent tension between the needs of the research and the necessity of a confidentiality 

pledge to the research participants. The governmental agency often cannot make the same 

commitment to confidentiality that a research organization can make, because of statutory 

requirements regarding access to data held by the government. This means that a clear 
642 The Promise of Evidence-Based Policymaking

6

statutory commitment that agencies may not use research data or the identity of 

individuals participating in federal research for any purpose other than conducting the 

research is needed. That assurance would make it more likely that people will continue to 

participate in research despite the increased sharing of research data or identifiers that may 

be needed occur as data systems become more integrated and accessible for a variety of 

research purposes. 

Such an affirmation could be similar to the Confidential Information Protection and 

Statistical Efficiency Act (CIPSEA), which protects data collected by federal statistical 

agencies.

b) Reforming the Paperwork Reduction Act to facilitate research and evaluation would 

facilitate a wide range of data collection efforts. Without this change, agencies will be 

stopped in their tracks as they try to take advantage of the kinds of quick turnaround

evaluations that could -- with the help of emerging technologies -- transform the quality and 

frequency of continuous program improvement activities in the coming decade. 

An example of a productive reform would be to develop standards for providing agency 

research and evaluation offices with delegated authority to review and approve Paperwork 

Reduction Act clearances for planned evaluations.1 Authority could be delegated to 

agencies, for example, if they (1) have robust evaluation capacity that routinely utilizes the 

most rigorous and appropriate evaluation design for the questions being asked; and (2) use 

existing administrative and survey data when possible.

• MDRC has conducted numerous evaluations of Federal human service programs 

under the authority of Section 1115 of the Social Security Act. To what extent do 

you think this provision or authority has encouraged evaluation that may not have 

otherwise been conducted? Is it worth considering such an authority for other 

types of programs? Is there a model statute that allows for independent 

evaluation and ensures that structures and resources are available to conduct 

such evaluations?

Yes, a substantial amount of experimentation that occurred in the past four decades would not 

have been possible without the waiver authority provided under Section 1115. The welfare-towork experiments that were conducted by MDRC and others in the 1980s and 1990s2 are an 

 1 For additional details of potential PRA reforms, see Feldman, A. (2016). Strengthening Results-Focused 

Government: Strategies to Build on Bipartisan Progress in Evidence-Based Policy. Washington, DC: The Brookings 

Institution. 2 Gueron, J. M. and Rolston, H. (2013). Fighting for Reliable Evidence. New York, NY: Russell Sage Foundation.
The Promise of Evidence-Based Policymaking 643

7

example of an agency and researchers using this authority to create a coherent body of 

evidence that informed future policy.

However, the waiver process is just the starting place for building rigorous evidence. Giving an 

agency the authority to waive current rules does not guarantee that evidence will be built. In 

fact, there are cases (such as No Child Left Behind) in which a waiver process was established 

without a requirement of rigorous evaluation, and the federal government lost the opportunity 

to inform future decisions with strong research. Therefore, if waiver authority is used as one of 

the foundations for evidence-based policy, it is critical for it to be accompanied by the following

provisions:

Any waiver statute should:

1. Require rigorous evaluation to measure the impacts, implementation, and costs of the 

new approach compared to current policy, with adequate sample sizes and research 

designs that can reliably answer the questions being asked. Where cost neutrality is 

required in the waiver authority, rigorous evaluation should be used to reliably ascertain 

whether cost neutrality was maintained. 

2. Authorize the agency’s Office of Research to review the research designs of proposed 

evaluations of waiver activities. To maximize the reliability of research results, their 

ratings of evaluations should give preference in impact designs to those that rely on 

random assignment. 

3. Include a set-aside of at least 1% for research and evaluation, and in some cases a 

higher percentage.

4. Include periodic audits by GAO to monitor the number of waivers granted, the use of 

evaluation for program improvement purposes, and any other requirements.

5. Authorize the funding agency to request evaluators to communicate what they are 

learning to the state or local programs that are participating in the study while the study 

is still going on, to the extent that such communication will not undermine the study’s 

ability to answer the particular research questions that are agreed upon. Such 

communication can improve the efficiency and effectiveness with which federal 

resources are being used even during the period of the evaluation, and can increase the 

benefits that state and local evaluation sites receive from the time and effort they 

contribute to the study. 
644 The Promise of Evidence-Based Policymaking

8

6. For some kinds of waiver-based experiments, it can be useful to provide an incentive for 

the private sector to contribute to the initiative by requiring programs that apply for 

grants to provide matching funds.
The Promise of Evidence-Based Policymaking 645

Date: February 13, 2017

To: Commission on Evidence-Based Policymaking

From: Tom Schenk Jr, 

Chief Data Officer, City of Chicago

Chair, Civic Analytics Network, Harvard University

RE: Follow-up Input on CEBP mission as defined in Public Law 114-140

It was my pleasure to have an opportunity to provide verbal testimony to the Commission on 

ways to increase evidence-based policy in our government. The conversation was very helpful 

and I hope I was able to provide some useful information to members of the Commission. I am 

following-up on my testimony to provide clarity and more information based on our 

conversation.

Briefly, my verbal testimony discussed the following ideas:

 Local governments, like Chicago, need to be able to share data between our own 

departments and between levels of government (e.g., county, state, and Federal 

governments) to provide better services. We face a number of challenges in this respect, 

including privacy laws that are interpreted in various ways that increases the time to 

create data-sharing agreements.

 Commission members should avoid the immense task of building a central Clearinghouse 

of data; instead, focus on smaller hubs that are topically relevant. This approach is not for 

security reasons, but simply for the pragmatic way to provide some central data services.

 Instead, the Commission should recommend a service where data is merged and 

deidentified. This service should be a paid service to help fund the activity itself.

Members of the Commission had a number of questions and asked for more details in each of 

these categories.

Data Sharing Within Government

Since providing testimony, I have reached out to my peers in the Civic Analytics Network—a 

network of 24 municipal Chief Data Officers (or similar roles) based out of Harvard’s Kennedy 

School of Government and which I am currently serving as a chair. Below, I have incorporated 

their feedback to provide more detail on the challenges facing cities in using data for evidencebased policy.

Privacy laws are broadly written to provide guidance on sharing data between governments, 

researchers, private companies, community groups, or anyone else who may want to access 

education, health, workforce, and other records. However, the experience of anyone seeking data 
646 The Promise of Evidence-Based Policymaking

sharing agreements—including attempts to share relevant data between government 

departments—is frustrating because of the high degree of local interpretation of privacy laws. 

Two reasonable attorneys can have different interpretations of the same provisions of privacy 

laws.

For instance, interpreting the “hard-code” and “minimum necessary” rules from HIPAA has 

caused difficulty in sharing data. Due to the ubiquity of data that could potentially contain health 

information, HIPAA guidelines are often used for non-health data decisions, as a precaution 

against the possibility of the inclusion of some health data.

Subtle but significant differences in legal terminology such as "permit", "prohibit", "allow", 

"may", and "shall", to name a few. Often, the laws are vague and exhaustive at the same time, 

such as the HIPAA requirement of the "minimum necessary" use and disclosure of data, which 

has its own set of complex and subtle rules and exceptions which contain language such as 

"reasonable steps" and "minimum necessary to accomplish the intended purpose." The reality of 

this framework is that a very limited number of experts fully understand the privacy laws that 

govern an entity's activities.

It would be much easier if FERPA / HIPAA / CORI / etc. gave clear guidance about appropriate 

uses of protected data by affiliated partner organizations and details how about what those 

partners need to do on their end to ensure privacy protections are maintained (standards for data 

security, removal of PII, etc.), and (critically) absolves the source agencies of liability if they 

share in good faith and the recipients don't hold up their end of the bargain. The big worry that 

impedes sharing even when there is a legal justification is that the data owners remain liable for 

what their partners do, so that provides a strong disincentive.

Avoiding a Clearinghouse – not because of information security

I proposed that the Commission avoid the task of creating a central clearinghouse of data. The 

reason to avoid this is because such clearinghouses are extraordinarily difficult to build and 

attempt to serve all audiences, thereby, tend to serve no audience very well. While the concept of 

a clearinghouse is attractive, it is not practical.

Others who testified with similar recommendations had mentioned it was partly due to security 

reasons. Generally, it was argued that decentralized data sources would make it harder for 

hackers or anyone with dubious intentions to quickly download personal data and subsequently 

selling the data, providing it to others, or using it in other inappropriate ways.

However, this is dubious. If a Clearinghouse was formed, that does not imply the data would be 

in the same physical location or anyone who accessed the system would have automatic access to 
The Promise of Evidence-Based Policymaking 647

all of the data. That is, data sitting on the same physical computer can be separated so data is 

separated. In a simple example, many basic computers have the ability to setup distinct users. 

While all of the data is stored on the same physical drive, no one user needs to have access to all 

of the data stored on the computer. This notion is logical separation of data.

For example, hackers have certainly accessed the systems of social media platforms. Often, you 

will see these companies clarify which data was precisely accessed. That is because “hacking” a 

system does not imply all of the data is compromised. Architects of data warehouses—such as a 

Clearinghouse—use logical separation throughout the system.

This is an important distinction because the most significant risk is the people who can access the 

data regardless if it is in a central Clearinghouse or stored in various physical locations. If 

academic researchers are able to combine data, that will naturally increase the risk that the 

professor, postdoc, research assistant, or anyone else with access to the data could make a 

mistake, such as storing data on a flash drive1 or laptops2 and losing it, or simply sharing the data 

through email or cloud-storage (e.g., Dropbox) accounts because of elaborate scams3 or 

mistakes4

.

The Commission needs to be aware of the true source of risk so it can make recommendations to 

mitigate those risks. Researchers who access any linked data should have very strict rules on the 

use and storage of data, such as never storing it on unencrypted flash drives or other mobile 

devices that could be lost or stolen. The Commission should even consider that the storage of 

such data should only be on computers that are disconnected from the internet and data must 

only be stored on encrypted hard drives that meets NIST standards5

.

 1 Greenberg, Adam. “Lost flash drive compromises data for thousands of students”. SC Magazine, August 12, 2013. 

https://www.scmagazine.com/lost-flash-drive-compromises-data-for-thousands-of-students/article/541926/ 2 Keane, Jonathan. “Why stolen laptops still cause data breaches, and what’s being done to stop them”. PCWorld, 

January 13, 2016 http://www.pcworld.com/article/3021316/security/why-stolen-laptops-still-cause-databreaches-and-whats-being-done-to-stop-them.html 3 McKinnon, Ryan. “Email Scam Threatens School Employee Data”. GovTech, February 7, 2017 

http://www.govtech.com/education/Email-Scam-Threatens-School-Employee-Data.html 4 Owens, Adam. “Wake schools sent postcards containing Social Security numbers”. WRAL, December 4, 2009 

http://www.wral.com/news/education/story/6547340/

5 National Institutes of Standards and Technology. Security and Privacy Controls for Federal Information Systems 

and Organizations. NIST Special Publication 800-53. April 2013 (revised January 22, 2015) 

http://dx.doi.org/10.6028/NIST.SP.800-53r4
648 The Promise of Evidence-Based Policymaking

An Affirmative Action/Equal Opportunity Employer

MEMORANDUM www.mathematica-mpr.com

TO: Commission on Evidence-based Policymaking

FROM: Matthew Stagner DATE: 2/8/2017

SUBJECT: Responses to follow up questions

Thank you again for the opportunity to testify to the Commission and for the follow up 

questions. Below are the questions you asked and my brief responses. Please let me know if 

there are further ways that staff of Mathematica Policy Research or I, in particular, can be of 

assistance. We look forward to following the work and publications of the Commission and 

wish you well in your work.

You noted the importance of strengthening partnerships between the research community 

and the public policy and program communities. Do you have specific suggestions the 

Commission could consider to do this? Are there examples of such partnerships that have 

worked well? 

Thank you for the opportunity to elaborate on this issue. We believe there are multiple 

approaches to continue testing and strengthening research-policy partnerships. A first step in 

improving this relationship is improving the layering of evaluation requirements and technical 

assistance. When federal grant funds require rigorous evaluation, agencies should ensure their 
The Promise of Evidence-Based Policymaking 649

MEMO TO: Commission on Evidence Based Policy

FROM: Matthew Stagner

DATE: 2/8/2017

PAGE: 2

grantees get adequate funds to support local evaluation and technical assistance. Rigorous 

evaluation can feel like it is “taking money away” from grantee activities, but there is a strong 

“public good” case for building an on-going evidence base. On top of this, we have found that 

significant evaluation technical assistance efforts can help local evaluators improve the quality 

and usefulness of their evaluations. If grantees, their evaluators, national technical assistance 

providers, and federal agencies work together, the evidence base will grow quickly. We are 

engaged in many such technical assistance efforts for the Department of Health and Human 

Services and the Department of Education, including the Regional Education Laboratory 

program and the Youth At-Risk of Homelessness (YARH) program.

Second, the commission may want to enhance role of associations and institutions that 

already bring together researchers and policy makers. Federal agencies could encourage 

policymaker membership organizations, such as the American Public Human Services 

Association, to include researchers in their activities. Through this process, researchers will hear 

what is most on the minds of state and local policy makers and will learn more about the contexts 

and constraints facing policy makers as they attempt to use evidence. It may also help to have 

research organizations such as the Association for Public Policy Research and Analysis connect 

more frequently with policy makers. These associations may help to bridge the “cultural divide” 

between researchers and policy makers.

Third, federal efforts to improve the training of researchers should focus on the important 

skills of conducting practical evaluation research. The next generation of researchers will need 

to listen carefully to policy makers’ needs and assess the best methods for working quickly to 
650 The Promise of Evidence-Based Policymaking

MEMO TO: Commission on Evidence Based Policy

FROM: Matthew Stagner

DATE: 2/8/2017

PAGE: 3

address those needs. Training programs for social scientists, such as the Department of 

Education training programs, could be encouraged to do more to build these skills. 

Finally, the commission may want to consider supporting expanded federal efforts that 

specifically build and enhance such partnerships. One such effort is the Researcher-Practitioner 

Partnerships in Education Research, funded by the Department of Education, Institute of 

Education Sciences. These grants focus on building long-term relationships between policy 

makers and local researchers. The partnerships carry out initial research on education issues of 

interest to policy makers and develop plans for future research. Through this joint research, an 

education agency's capacity for taking part in research and using research results can increase. 

Funding goes to launch new partnerships and to support the expansion of existing partnerships. 

This model may transfer to other policy areas beyond education, particularly human services 

efforts.

What datasets do you think are most vital for research and as such should be included in a 

federal data clearinghouse if the Commission were to recommend one be developed?

From the Chicago hearing, it appears that much of the Commission’s focus has been on data 

from the education and labor areas, such as earnings tracked in the unemployment insurance 

system or by the Internal Revenue System. Therefore, we focus on other others that may not 

have received as much attention, particular issues of health and disability. Without these data, 

findings on the connections between education and labor market outcomes may be misleading.
The Promise of Evidence-Based Policymaking 651

MEMO TO: Commission on Evidence Based Policy

FROM: Matthew Stagner

DATE: 2/8/2017

PAGE: 4

To support research in disability, health and retirement policy, the federal data clearinghouse 

should include the Social Security Administration’s (SSA) administrative and survey data, as 

well as the Rehabilitation Services Administration’s administrative records on state vocational 

rehabilitation services (annual RSA 911 file). SSA has developed and maintains several analytic 

files built from multiple administrative file to support internal researchers. 

The largest of these is the Disability Analysis File, which contains the most recent 10 years 

of records on SSI and SSDI applications and disability awards (about 2.5 million records). This 

database includes a longitudinal record for every Supplemental Security Income or Social 

Security Disability Beneficiary between the ages of 10 and 65 from 1996 forward (on the order 

of 25 million records). SSA also maintains a historical Continuous Work History File including a 

1% sample of all workers and beneficiaries that is extremely valuable for long-term research and 

not available to researchers outside of the agency.

Researchers have used SSA administrative data, including the Disability Analysis File, to 

link to other administrative records that include Vocational Rehabilitation and earnings 

information. For example, recent SSA research studies have used SSA administrative data 

linked to the RSA-911 and the Master Earnings File (the SSA file that contains IRS earnings 

histories) on a regular basis. 

Other important sets of data are Medicare and Medicaid records maintained by the Centers 

for Medicare and Medicaid Services (CMS) and records from the Department of Veterans 

Affairs, though record linkages between these sources and SSA administrative data are less 

common. A useful clearinghouse would support linkages between SSA records and state 
652 The Promise of Evidence-Based Policymaking

MEMO TO: Commission on Evidence Based Policy

FROM: Matthew Stagner

DATE: 2/8/2017

PAGE: 5

program records, including Medicaid claims, unemployment wage and benefit records, 

vocational rehabilitation records, workforce system records, and public education records.

SSA has also conducted large survey efforts of beneficiaries to gather information not 

included in administrative records. For example, SSA’s National Beneficiary Survey, which is 

routinely linked to administrative records, includes several rounds of data collection for a 

nationally representative survey of SSI and DI beneficiaries. SSA also conducts surveys as part 

of demonstration efforts that can serve as additional sources of information about demonstration 

participants. It would also be useful to include the one-time National Survey of Child Recipients 

and Families, with linked longitudinal administrative records. 

National surveys are another potential source of record linkage, particularly for SSA 

administrative data. Specifically, it would be very valuable to include the surveys that SSA, 

HHS and Census routinely link to SSA and other administrative records now: six National 

Center for Health Statistics Surveys (many for multiple years), the Health and Retirement Study, 

the Survey of Income and Program Participation, and the Current Population Survey. Although 

these surveys are more accessible to policy researchers than most SSA administrative data, they 

remain difficult to access. Maintaining them in a clearinghouse and making them more 

accessible would improve synergies between survey and administrative data.

A final consideration is the potential to use administrative records from a clearinghouse to 

facilitate web-based surveys. This approach is low cost and an added advantage of broadening 

access to administrative data. Administrative data provides a shallow look at a full program 

population, and web-based surveys of a sample drawn from that population can provide muchneeded context and a deeper view of particular issues. A recent article by Jaszczak, Sevak, and 
The Promise of Evidence-Based Policymaking 653

MEMO TO: Commission on Evidence Based Policy

FROM: Matthew Stagner

DATE: 2/8/2017

PAGE: 6

Wright entitled “Personal Characteristics of VR Applicants: Findings from the Survey of 

Disability and Employment” (now under review) provides an example of such a study, focused 

on a vocational rehabilitation population.
654 The Promise of Evidence-Based Policymaking

www.studentprivacymatters.org	
  

info@studentprivacymatters.org

@parents4privacy

124	
  Waverly	
  Place

New	
  York,	
  NY	
  10011

303.204.1272

10	
  February	
  2017

Commission	
  on	
  Evidence-­‐Based	
  Policymaking

Washington, DC

Dear	
  Commission	
  Members:

Thank	
  you	
  for	
  requesting	
  additional	
  information	
  about	
  the	
  Parent	
  Coalition	
  on	
  Student	
  Privacy’s	
  position	
  

on	
  unit	
  record	
  systems.	
  Below	
  we	
  address	
  all	
  three	
  of	
  the	
  Commission’s	
  follow-­‐up	
  questions:

• Can	
  you	
  please	
  clarify	
  whether	
  your	
  objection	
  is	
  to	
  unit	
  record	
  systems	
  for	
  elementary	
  and	
  

secondary	
  school	
  students	
  or	
  if	
  you	
  also	
  object	
  to	
  unit	
  record	
  systems	
  for	
  postsecondary	
  

students?

See	
  the	
  discussion	
  of	
  this	
  issue	
  below.	
  

• You	
  argue	
  for	
  local	
  control	
  of	
  student	
  data	
  and	
  specifically	
  for	
  parental	
  and	
  teacher	
  data	
  

stewardship.	
  At	
  what	
  point	
  should	
  adult	
  students	
  in	
  postsecondary	
  education	
  become	
  the	
  

stewards	
  of	
  their	
  own	
  data?

The	
  control	
  of	
  their	
  data	
  should	
  pass	
  to	
  post-­‐secondary	
  students when	
  they	
  become	
  adults,	
  

as	
  occurs	
  currently	
  in	
  Federal	
  law.

• Would	
  your	
  concerns	
  about	
  the	
  intrusiveness	
  of	
  a	
  student-­‐unit	
  record	
  system	
  be	
  mitigated	
  if	
  the	
  

data	
  were	
  only	
  maintained	
  without	
  personal	
  identifiers	
  that	
  could be	
  used	
  to	
  track	
  an	
  individual	
  

student,	
  and	
  if	
  there	
  were	
  statutory	
  protections	
  that	
  guaranteed	
  that	
  the	
  data	
  could	
  only	
  be	
  

used	
  for	
  aggregate,	
  statistical	
  analysis?

No,	
  because	
  data	
  can	
  easily	
  be	
  re-­‐identified.	
  If	
  only	
  aggregate	
  data	
  is	
  used,	
  only	
  aggregate	
  

data	
  should	
  be	
  collected.

We	
  have	
  significant	
  reservations	
  about	
  the	
  creation	
  of	
  any	
  universal	
  unit	
  record	
  system	
  for	
  students,	
  

whether	
  for	
  elementary,	
  secondary	
  or	
  post-­‐secondary	
  students.	
  

Elementary	
  and	
  secondary	
  student	
  unit	
  record	
  systems	
  present	
  a	
  particular	
  set	
  of	
  risks	
  because	
  the	
  

majority	
  of	
  information	
  in	
  a	
  child’s	
  K12	
  educational	
  record	
  should	
  not	
  be	
  made	
  “permanent.”	
  Childhood	
  

is	
  a	
  time	
  of	
  growth,	
  experimentation	
  and	
  development;	
  and	
  mistakes	
  and	
  challenges	
  should	
  not	
  be	
  part	
  

of	
  a	
  record	
  that	
  could	
  follow	
  one	
  into	
  adulthood	
  and	
  hamper	
  a	
  child’s	
  chance	
  of	
  future	
  success.	
  

A	
  unit	
  record	
  system	
  for	
  post-­‐secondary	
  education	
  does	
  not	
  present	
  an	
  identical	
  set	
  of	
  concerns.	
  

Students	
  in	
  post-­‐secondary	
  institutions	
  do	
  expect	
  that	
  some	
  aspects	
  of	
  their	
  transcript,	
  including	
  grades	
  

and	
  credits,	
  will	
  persist	
  into	
  adulthood	
  with	
  the	
  expectation	
  of	
  being	
   shared	
  with	
  employers	
  and	
  other	
  

educational	
  institutions—with	
  their	
  consent.	
  Yet	
  other	
  contents	
  of	
  their	
  education	
  records	
  should	
  never	
  

be	
  made	
  public.	
  

Records	
  from	
  these	
  years	
  may	
  also	
  contain	
  sensitive	
  information	
  about	
  immigration	
  status,	
  counseling	
  

records,	
  mental	
  and	
  physical	
  health	
  and	
  disabilities,	
  etc.	
  At	
  the	
  age	
  of	
  18,	
  control	
  of	
  the	
  record	
  is	
  
The Promise of Evidence-Based Policymaking 655

transferred	
  from	
  a	
  parent	
  or	
  guardian	
  to	
  the	
  students	
  themselves,	
  but,	
  crucially,	
  privacy	
  controls	
  are	
  still	
  

maintained.

We	
  have	
  several	
  concerns	
  about	
  the	
  need	
  for	
  and	
  use	
  of	
  any	
  universal	
  post-­‐secondary	
  unit	
  record	
  

system:

1. The	
  efficacy	
  of	
  methods	
  to	
  de-­‐identify	
  or	
  anonymize	
  personally-­‐identifiable	
  data	
  is	
  questionable.	
  

De-­‐identified	
  data	
  can	
  often	
  be	
  re-­‐identified	
  and	
  exposed.1

2. The	
  government	
  should	
  not	
  have	
  access	
  to	
  a	
  comprehensive	
  database	
  for	
  all	
  post-­‐secondary	
  

students	
  as	
  this	
  information	
  could	
  be	
  easily	
  abused	
  for	
  political	
  or	
  immigration	
  reasons.	
   This	
  is	
  

especially	
  of	
  concern	
  given	
  the	
  current	
  political	
  climate.	
  The	
  Home	
  Office	
  in	
  Great	
  Britain	
  has	
  

now	
  requested	
  access	
  to	
  a	
  similar	
  government	
  student	
  database	
  for	
  the	
  purposes	
  of	
  

“immigration	
  control”	
  that	
  was	
  promised	
  to	
  only	
  be	
  used	
  for	
  research.2

3. Once	
  the	
  federal	
  government	
  starts	
  collecting	
  post-­‐secondary	
  data,	
  this	
  could	
  easily	
  lead	
  to	
  a	
  

creeping	
  expansion	
  of	
  data	
  collection	
  from	
  K12	
  institutions	
  and	
  districts.	
  

4. The	
  quality	
  of	
  research	
  based	
  on	
  large-­‐scale	
  correlational	
  studies	
  is	
  of	
  greatly	
  varying	
  quality3

and	
  does	
  not	
  justify	
  the	
  risks	
  of	
  universal	
  tracking.

5. Large	
  amounts	
  of	
  data	
  used	
  for	
  the	
  purposes	
  of	
  evaluating	
  post-­‐secondary	
  institutions’	
  

effectiveness	
  are	
  already	
  available,	
  including	
  the	
  Department	
  of	
  Education’s	
  College	
  Scorecard,	
  

the	
  Mobility	
  Report	
  Card	
  Project—a	
  collaboration	
  of	
  the	
  US	
  Treasury	
  and	
  the	
  Department	
  of	
  

Education,	
  and	
  the	
  privately-­‐run	
  National	
  Student	
  Clearinghouse.	
  

6. Extensive	
  regulations	
  have	
  already	
  been	
  implemented	
  to	
  ensure	
  that	
  post-­‐secondary

institutions	
  are	
  protecting	
  student’s	
  long-­‐term	
  financial	
  interests,	
  e.g.	
  Negotiated	
  Rulemaking	
  on	
  

Gainful	
  Employment implemented	
  in	
  Fall	
  2014.

We	
  acknowledge	
  that	
  given	
  the	
  investment	
  of	
  taxpayer	
  funds	
  that	
  support	
  institutions	
  of	
  higher	
  

education,	
  the	
  federal	
  government	
  has	
  a	
  strong	
  practical	
  interest	
  to	
  make	
  certain	
  that	
  those	
  funds are	
  

being	
  used	
  efficiently	
  and	
  effectively.

We	
  do	
  not,	
  however,	
  think	
  that	
  a	
  universal	
  student	
  record	
  system	
  created	
  and	
  administered	
  by	
  the	
  

federal	
  government	
  is	
  a	
  necessary	
  component	
  of	
  fulfilling	
  that	
  interest	
  and	
  duty.

The	
  federal	
  government	
  spends	
  billions	
  of	
  dollars	
  in	
  medical	
  research	
  and	
  health	
  care	
  as	
  well,	
  and	
  yet	
  

there	
  has	
  been	
  no	
  proposal	
  that	
  we	
  know	
  of	
  for	
  the	
  federal	
  government	
  to	
  collect	
  the	
  personal	
  health	
  

data	
  for	
  every	
  person	
  in	
  the	
  United	
  States.	
  

	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
   1 See,	
  for	
  example,	
  the	
  research	
  of	
  Latanya	
  Sweeney	
  on	
  identifiability. 2 “The	
  Home	
  Office	
  are	
  turning	
  teachers	
  into	
  immigration	
  officers”	
  G.	
  Bhattacharyya.	
  Politics.co.uk. 3 See	
  discussion	
  from	
  a	
  variety	
  of	
  fields	
  (medicine,	
  psychology	
  and	
  linguistics)	
  in	
  “Data	
  dredging,	
  bias,	
  or	
  

confounding.”	
  Smith,	
  G.D.	
  and	
  S.	
  Ebrahim.	
  BMJ.	
  2002;	
  “Why	
  Most	
  Published	
  Research	
  Findings	
  Are	
  False.”	
  

Ioannidis,	
  J.	
  P.	
  A.	
  PLOS	
  Medicine.	
  2005;	
  “False-­‐Positive	
  Psychology:	
  Undisclosed	
  Flexibility	
  in	
  Data	
  Collection	
  and	
  

Analysis	
  Allows	
  Presenting	
  Anything	
  as	
  Significant.”	
  Simmons,	
  J.P.,	
  L.D.	
  Nelson	
  and	
  U.	
  Simonsohn.	
  Psychological	
  

Science.	
  2011;	
  and	
  “Linguistic	
  Diversity	
  and	
  Traffic	
  Accidents:	
  Lessons	
  from	
  Statistical	
  Studies	
  of	
  Cultural	
  Traits.”	
  

PLoS	
  ONE.	
  2013.	
  Roberts	
  S.	
  and	
  J.	
  Winters.
656 The Promise of Evidence-Based Policymaking

We	
  support	
  only	
  the	
  use	
  of	
  aggregated student	
  data	
  for	
  the	
  evaluation	
  of	
  postsecondary	
  institutions.	
  

The	
  collection	
  of	
  such	
  data	
  must	
  include	
  asking	
  for	
  consent	
  for	
  participation	
  from	
  either	
  a	
  

parent/guardian	
  or	
  the	
  students	
  themselves	
  if	
  over	
  the	
  age	
  of	
  18.	
  Clear,	
  transparent	
  information	
  about	
  

how	
  any	
  data	
  is	
  to	
  be	
  used	
  and	
  who	
  it	
  may	
  be	
  shared	
  with	
  must	
  be	
  presented	
  before	
  asking	
  for	
  consent.	
  

And,	
  there	
  should	
  be	
  no	
  financial	
  benefit	
  or	
  loss	
  contingent	
  on	
  granting	
  the	
  consent.

We	
  continue	
  to	
  urge	
  the	
  Commission	
  to	
  recommend	
  against	
  the	
  creation	
  of	
  any	
  universal federal	
  

student	
  unit	
  record	
  system.

On	
  behalf	
  of	
  the	
  Parent	
  Coalition	
  for	
  Student	
  Privacy,

Sincerely,

Cassandre	
  Creswell,	
  PhD

Co-­‐executive	
  director

Raise	
  Your	
  Hand	
  Action

Chicago	
  IL
The Promise of Evidence-Based Policymaking 657

1 

Submitted January 26th, 2017 

Written Statement for the Commission on Evidence-Based Policymaking 

Public Hearing in San Francisco, February 9th, 2017 

Introduction 

The Abdul Latif Jameel Poverty Action Lab North America (J-PAL NA), based in the 

Department of Economics at the Massachusetts Institute of Technology, leverages scholarship 

from 143 affiliated professors to generate and disseminate rigorous evidence about anti-poverty 

policies. J-PAL NA provides pro-bono technical support, capacity building, and matchmaking 

with researchers to government agencies and nonprofits seeking to design and implement 

randomized evaluations, many of which rely extensively on administrative data. Affiliates in our 

network have conducted 154 ongoing or completed randomized evaluations in North America 

across sectors such as health care, housing, criminal justice, education, and labor markets. J-PAL 

NA also creates training materials to build research capacity, including a comprehensive, 

practical guide to obtaining and using administrative data for randomized evaluations.1 We 

appreciate the opportunity to submit a statement to the Commission on Evidence-Based 

Policymaking. 

J-PAL affiliated researchers have relied heavily on administrative data to conduct policy-relevant 

research. Data from IRS tax records enabled an almost 20-year follow-up of families involved in 

the Moving to Opportunity Experiment. The follow-up study demonstrated that young children 

who moved to low-poverty neighborhoods increased their college attendance and expected 

lifetime earnings.2 Data from the U.S. Department of Education, the Ohio Board of Regents, and 

the National Student Clearinghouse collectively enabled a randomized controlled trial showing 

that simplifying the financial aid application process increased college attendance and 

persistence.3 Data from hospitals in the Portland area revealed that Medicaid insurance, for 

which opportunities to apply were allocated through a lottery in Oregon, increased emergency 

room usage by 40 percent.4 Access to administrative data was critical to generating these 

insights. 

Executive Summary 

This statement reflects J-PAL’s expertise concerning randomized evaluations, administrative 

data access, and collaboration between government agencies and external researchers. It 

1 Feeney, Laura, Jason Bauman, Julia Chabrier, Geethi Mehra, and Michelle Woodford. “Using Administrative Data for 

Randomized Evaluations.” J-PAL North America. December 2015. 

https://www.povertyactionlab.org/sites/default/files/documents/AdminDataGuide.pdf

2

 Chetty, Raj, Nathaniel Hendren, and Lawrence F. Katz. "The Effects of Exposure to Better Neighborhoods on Children: New 

Evidence from the Moving to Opportunity Experiment." NBER Working Paper #21156, May 2015. 

3

 Bettinger, Eric P., Bridget Terry Long, Philip Oreopoulos, and Lisa Sanbonmatsu. 2012. "The Role of Application Assistance 

and Information in College Decisions: Results from the H&R Block Fafsa Experiment." The Quarterly Journal of Economics 

127(3):1205-42. 4

 Taubman, Sarah, Heidi Allen, Bill Wright, Katherine Baicker, Amy Finkelstein, and the Oregon Health Study Group. 2014. 

"Medicaid Increases Emergency Department Use: Evidence from Oregon's Health Insurance Experiment." Science 

343(6188):263-8.

Mary Ann Bates, J-PAL North America, 

Massachusetts Institute of Technology 

San Francisco Public Input–Hearing Testimony
658 The Promise of Evidence-Based Policymaking

2

incorporates recommendations from an open letter penned by several leading economists, 

including multiple J-PAL affiliates5

, and a short paper published by a subset of the same 

authors.6 This comment elaborates on these key recommendations:

 Establish clear data documentation and standard data request forms, building on the 

example set by the Centers for Medicare and Medicaid Services. 

 Expand secure access to real microdata to qualified researchers, prioritizing secure 

remote connections while also increasing capacity at Census Research Data Centers.

 Develop a data clearinghouse within the Census Bureau for currently hard-to-access data, 

particularly microdata on earnings and income, and link the data across agencies.

 Avoid flat per-user fees for data access to encourage validation and double-checking of 

data analysis.

 Clearly articulate program objectives and build ongoing process evaluation into every 

program to lay the foundation for impact evaluation. 

 Institutionalize a process for identifying questions for program evaluation and appropriate 

conditions for randomized evaluations, focusing on three cases:

o Demand for a program exceeds capacity to supply the program.

o Gradual roll out of a program to different individuals or locations over time.

o Refinement or reconsideration of eligibility criteria for a program.

Responses to Specific Questions Posed in the CEP Request for Comments

3. Based on identified best practices and existing examples, how should existing 

government data infrastructure be modified to best facilitate use of and access to 

administrative and survey data?

Build on the example of the Centers for Medicare and Medicaid Services to establish clear 

data documentation and standard data requests. 

Existing government data infrastructure should incorporate standard data request forms with 

clear data dictionaries, using the Centers for Medicare and Medicaid Services (CMS) Research 

Data Assistance Center (ResDAC) data documentation as a model of best practices. The 

ResDAC system allows researchers to understand specifically what variables are available and to 

submit requests with data protection plans. Because the ResDAC system allows CMS to review 

those requests systematically as opposed to on an ad hoc basis, ResDAC facilitates routine, 

secure access to administrative data that culminates in several hundreds of medical studies each

year.7

Applying the ResDAC model to an administrative data clearinghouse or other data repositories

would allow researchers to see exactly what variables they are permitted to request, along with a 

 5 Card, David, Raj Chetty, David Cutler, Steven Davis, Martin Feldstein, William Gale, Jonathan Gruber, Michael Greenstone,

Caroline Hoxby, Lawrence Katz, and Emmanuel Saez. 2010. “An Open Letter on Expanding Access to Administrative Data for 

Research in the United States. https://eml.berkeley.edu/~saez/dataaccess_final.pdf 6 Card, David, Raj Chetty, Martin S. Feldstein, and Emmanuel Saez. "Expanding access to administrative data for research in the 

United States." American Economic Association, Ten Years and Beyond: Economists Answer NSF's Call for Long-Term

Research Agendas (2010). https://eml.berkeley.edu/~saez/card-chetty-feldstein-saezNSF10dataaccess.pdf 7 Ibid.
The Promise of Evidence-Based Policymaking 659

3

brief description of each variable, before submitting a request. This explicit listing of available

variables would enable data discovery and save program administrators and researchers hours of 

valuable time by avoiding long correspondences about whether the desired data exist.

Publicizing exactly which variables exist and what agency houses the data increases transparency 

with no risk of revealing personally identifiable information. Better data documentation can thus 

facilitate use of and access to administrative and survey data without raising concerns for data 

security and privacy protection. 

Moreover, sensitive variables that would trigger additional levels of review or security could be 

clearly labeled as sensitive as part of this clearer data documentation. Currently, researchers may 

request a variable that is not central to their analysis, without realizing that it captures sensitive 

information. This could delay or jeopardize the entire request or allow access to sensitive data 

that, with clearer data documentation, would not have been requested from the agency.8

4. What data-sharing infrastructure should be used to facilitate data merging, linking, and 

access for research, evaluation, and analysis purposes?

Expand secure access to microdata to qualified researchers through remote and on-site 

connections rather than creating synthetic data.

An optimal infrastructure for integrating administrative, survey, and statistical data to facilitate 

research and evaluation while ensuring data security and privacy will provide secure 

environments where qualified researchers can directly access microdata. Microdata enable 

researchers to perform more informative analyses by controlling for individual characteristics 

(such as educational attainment or race) to better determine the impact of a program. Microdata 

also allow researchers to evaluate how a program affects specific subpopulations, such as lowincome individuals. Researchers can use microdata to validate and adjust their analysis as they 

learn from the data in real time—a crucial step in the research process. Moreover, for analysis in 

rigorous randomized evaluations, researchers require microdata to link individuals to their 

treatment status. 

There are currently twenty-four Federal Statistical Research Data Centers (RDCs), which are 

physical, secure environments established through partnerships between the Census Bureau and 

research institutions where researchers who have undergone special sworn status can access 

restricted microdata. However, capacity in these RDCs is limited, both physically and according 

to Census bandwidth, and access is artificially restricted to researchers based on geographic 

proximity rather than on the merit of their research proposal.

9 Similar constraints apply to 

researchers working with statutorily restricted tax data through contracts with the IRS Statistics 

of Income Division (SOI)—the type of arrangement that enabled the Moving to Opportunity 

follow-up study. The SOI is small, has a limited budget, and can accommodate few research 

projects at a time. 10 More secure, direct access to microdata should be provided in two ways: (1) 

 8 Feeney et al., “Using Administrative Data for Randomized Evaluations,” 2015. 

9 Card et al., "Expanding access to administrative data,” 2010b.

10 Office of Management and Budget, Barriers to Using Administrative Data for Evidence-Building (white paper submitted for 

the Commission on Evidence-Based Policymaking, July 15, 2016). https://www.whitehouse.gov/sites/default/files/omb/mgmtgpra/barriers_to_using_administrative_data_for_evidence_building.pdf
660 The Promise of Evidence-Based Policymaking

4

preferably through remote, secure connections such as the “flexiplace” systems used by federal 

employees who work with restricted data from home, but also (2) through expansion of on-site 

secure environments in the form of additional space and funding for Research Data Centers and 

similar centers at other statistical agencies.

11

Synthetic data, one alternative to expanding remote and on-site secure connections to restricted 

data, are a far inferior option for enabling policy-relevant research and program evaluation. 

Synthetic data are constructed to mimic certain features and aggregate characteristics of real data 

without containing real individual-level information. Although this appears—on its surface—to 

enable research while protecting privacy, synthetic data suffer severe disadvantages relative to 

real microdata. Synthetic data may be incompatible with randomized evaluations and other

rigorous program evaluations because researchers must be able to link individuals to their 

treatment status—i.e., whether a particular person received a program or not. Furthermore, 

synthetic data make it difficult or impossible to study subpopulations, such as low-income 

individuals, which may be of particular policy interest. Researchers would have to specify each 

subpopulation they intend to study and all necessary contents of the data in advance. This may be 

impossible, in part because researchers often revise their analyses to address observations they 

learn from the raw data.12 Meanwhile, data administrators would have to create new synthetic

datasets for each request to study a specified subpopulation, which would require significant 

infrastructure and personnel.

7. What data should be included in a potential U.S. government data clearinghouse? What 

are the current legal or administrative barriers to including such data in a clearinghouse or 

linking the data?

Establish a data clearinghouse within the Census Bureau for currently hard-to-access data, 

particularly microdata on earnings and income. 

For reasons discussed in response to question 4, the data should be real microdata rather than 

aggregated, de-identified, synthetic, or perturbed data. The clearinghouse should prioritize data 

that do not already benefit from strong infrastructure for access. Specifically, a clearinghouse 

should be developed for federal income and earnings microdata and focus on enabling 

researchers to link these data to the extent legally possible.

Income and earnings data have less well-developed access infrastructure and face several legal 

barriers to use for program evaluation, meaning that the clearinghouse would not be redundant.13

For example, individual states maintain their own data system for Unemployment Insurance (UI) 

records, with individual discretion and statutory protections on providing access to this data. The 

Department of Labor does not store the data in a central location. Although the Census Bureau 

has made a significant contribution to accessing state UI data through the Longitudinal Employer 

 11 Card et al., "Expanding access to administrative data,” 2010b. 12 Ibid. 13 Ibid.
The Promise of Evidence-Based Policymaking 661

5

Household Dynamics Program (LEHD), the LEHD program requires that researchers be on-site 

at a designated Research Data Center.14

With infrastructure secure enough for the highly restricted data from UI and tax records 

established, the clearinghouse should then focus on facilitating linkage of these data with other, 

less restricted data. Federal data on income, namely tax records or Unemployment Insurance 

records, are in high demand because income can serve as a key outcome variable for many 

government programs or policies in education, job training, criminal justice, and place-based 

interventions.15 For example, earnings—as measured by tax records—was a key outcome 

variable in the follow-up study of the Moving to Opportunity Experiment.

8. What factors or strategies should the Commission consider for how a clearinghouse(s) 

could be self-funded? What successful examples exist for self-financing related to similar 

purposes?

Avoid flat per-user fees for data access to encourage validation and double-checking of 

data analysis. 

As is customary, a clearinghouse may charge fees for accessing data, such as a fee per project, a 

fee for sets of users, or an initial fee for the first user followed by much smaller fees for 

additional users. It should not charge the same flat fee per person for accessing the data because 

this severely discourages the double checking crucial to correcting human errors. Despite its 

clear data request process, CMS charges a fixed fee of $25,000 per person who accesses

identifiable data through their Virtual Research Data Center.

16 This may create problems because 

researchers often need multiple people to work with the data to ensure accuracy—including 

people who effectively proofread to correct for human coding errors. With fixed per-person 

costs, researchers either pay a large inflexible sum of money for someone to double check the 

analysis or—facing tradeoffs given limited research funds—forego a set of “fresh eyes” to 

double check the analysis at risk of making mistakes. Therefore, by creating a high marginal cost 

to adding additional users, charging a fixed fee per data user effectively institutionalizes 

mistakes. 

18. How can or should program evaluation be incorporated into program designs? What 

specific examples demonstrate where evaluation has been successfully incorporated in 

program designs?

Clearly articulate program objectives and build process evaluation into every program to 

lay the foundation for impact evaluation. 

Process evaluation is always needed and constitutes a critical prerequisite for impact evaluation. 

It is not sensible to ask whether the program is succeeding or failing to deliver outcomes without

first knowing whether the program itself is being delivered with fidelity. In some cases, 

 14 Ibid. 15 Office of Management and Budget, Barriers to Using Administrative Data, 2016. 16 Research Data Assistance Center. “Fee List for RIFs: Physical Research Data Requests.” Centers for Medicare and Medicaid 

Services. https://www.resdac.org/sites/resdac.umn.edu/files/CMS%20Price%20List%20for%20Research%20Files_23.pdf

Accessed October 31, 2016.
662 The Promise of Evidence-Based Policymaking

6

important questions about how a program can or should function may be sufficiently answered 

by process evaluations, needs assessments, or literature reviews. Rigorous impact evaluation, 

particularly randomized evaluation, should be pursued when the benefits in terms of knowledge 

generated would likely outweigh the costs of the evaluation, and when planning during program 

design can facilitate impact evaluation. 17

Many components that aid process and impact evaluation should be developed during program 

design:

 Precisely articulated program objectives. 

 A needs assessment clearly articulating the problem that the program will address.

 Standard outcome measures used in research literature about similar programs that allow 

potential impact evaluation results to be compared to those in other studies and used in 

cost-effectiveness analyses.

 A plan for data collection and flow from program practitioners to administrators. This 

includes planning in advance to collect identifying information, such as Medicaid ID 

numbers, to enable later matching of program-level records to administrative records for 

impact evaluation. 

As an example of incorporating evaluation into program design, Benefits Data Trust (BDT) is 

working with J-PAL North America in an ongoing randomized evaluation of different outreach 

strategies to increase enrollment in the Supplemental Nutrition Assistance Program (SNAP) 

among eligible but unenrolled individuals in Pennsylvania. BDT had a clear grasp of the need its 

program addressed: despite awareness among eligible households that SNAP exists, many people 

could not imagine navigating the enrollment process alone. BDT had the clear program objective 

of increasing benefits enrollment, and change in program enrollment is a standard outcome that 

could be compared across different studies. BDT and researchers agreed that based on a review 

of the existing research, there was little rigorous evidence about what interventions can increase 

SNAP enrollment. 

Although BDT was already providing enrollment assistance and sending outreach, BDT worked 

with researchers to design and test two distinct outreach activities—one high-touch intervention 

including a letter plus enrollment assistance and one low-touch intervention including a letter 

only. The researchers also worked with BDT to design a new letter for the evaluation based on 

marketing and psychology literature. Seeing quickly that this newly designed letter was more 

effective, BDT plans to incorporate this letter design in other states outside of Pennsylvania. 

Ultimately, the impact of the different outreach strategies will be measured using administrative 

data, which can be accessed according to a data use agreement with the Pennsylvania 

Department of Human Services. 

19. To what extent should evaluations specifically with either experimental (sometimes 

referred to as “randomized control trials”) or quasi-experimental designs be 

 17 Glennerster, Rachel, and Kudzai Takavarasha. Running randomized evaluations: A practical guide. Princeton University Press, 

2013.
The Promise of Evidence-Based Policymaking 663

7

institutionalized in programs? What specific examples demonstrate where such 

institutionalization has been successful and what best practices exist for doing so?

Institutionalize a process for developing specific research questions and determining the 

appropriate conditions for randomized control trials or other evaluation methods. 

Federal agencies should institutionalize a process of developing high-priority research questions 

and determining the most appropriate evaluation methodology, following the precedent set by

the Social and Behavioral Sciences Team (SBST). SBST launches demonstration projects—

usually in the form of randomized evaluations—to rapidly evaluate low-cost applications of 

behavioral science to achieve desired outcomes, such as increasing workplace savings plan 

enrollment among military service members or increasing the rate at which indebted graduates 

apply for income-based loan repayment plans.

18

When properly designed and implemented, randomized evaluations rigorously demonstrate the

causal impact of a program by establishing the counterfactual—what outcomes would exist for

program participants if they had not received the program. Random assignment ensures that, 

with a large enough sample, the group that receives the program and the group that does not are 

similar on average before the start of the program. Therefore the impact estimate from a 

randomized evaluation offers confidence that any differences in outcomes between the two 

groups are a result of the program. The ability to isolate program impact from self-selection or 

other confounding factors is why randomized evaluations are widely recognized as a highly 

credible method for estimating program impact. Where there is little internal experience 

implementing randomized evaluations, agencies should seek partnerships with external or 

academic researchers who are vested in similar questions.

Randomized evaluations can only occur when randomization is built into the program design.

However, randomization should not be incorporated indiscriminately; rather, randomization

should be incorporated into programs to facilitate randomized evaluations where appropriate on 

three grounds:

 The current evidence for answering the well-defined research question is non-existent, 

insufficient, or inconclusive.

 There is a clear unit of randomization—individual program participants, schools, clinics, 

etc.—for which there is a large enough sample size and a clear means of tracking 

outcomes for both the treatment group and the control group.

 Randomization is feasible and ethical. Although not an exhaustive list, the following 

conditions offer opportunities where randomization may be feasible and ethical:

o Demand for a program exceeds capacity to provide the program. A lottery may be 

a fairer alternative than allocating slots on a first come, first served basis—

particularly when a goal of the program is equity of access—and offers an 

alternative to imposing increasingly narrow eligibility criteria under funding 

constraints. 

 18 Social and Behavioral Sciences Team. 2016 Annual Report. Executive Office of the President, National Science and 

Technology Council. September 15, 2016. https://sbst.gov/assets/files/2016%20SBST%20Annual%20Report.pdf
664 The Promise of Evidence-Based Policymaking

8

o A program is being expanded by gradually offering it to individuals, schools, or 

districts until full coverage is reached. A lottery can be used to randomly assign 

the order in which individuals or units receive the program. The individuals or 

units that have not yet received the program serve as the control group until all 

units receive the program. 

o A new intervention—such as a financial incentive or care coordination services—

will be added to an existing program. Program participants can be randomized to 

receive different versions of the program, e.g., with or without the added 

intervention, to isolate the impact of the new intervention.

o Program eligibility criteria are being refined or reconsidered. People just

above/below the eligibility cutoff can be randomly assigned to receive or not 

receive the program to determine whether it is effective for this marginal group. 

Meanwhile, those well within the program eligibility cutoff, automatically receive 

the program, and those well outside the cutoff do not qualify for the program.

As a specific example of institutionalizing a randomized evaluation, the South Carolina 

Department of Health and Human Services is partnering with J-PAL North America to 

incorporate a randomized evaluation in its expansion of a nurse home-visiting program for lowincome mothers delivered by the nonprofit organization Nurse-Family Partnership (NFP). The 

specific, high-priority research question is whether a new, less expensive version of the NFP 

program that South Carolina is expanding will be as effective as the pre-existing version that has 

been rigorously evaluated before. 

A randomized evaluation was found to be an appropriate method for answering this question 

given excess demand for the program. Although South Carolina is expanding this less expensive 

version of NFP to thousands of mothers through an innovative pay-for-success initiative, the 

program does not have sufficient resources to serve all of the women who are eligible. 

Applicants will be randomly assigned, on a rolling basis from 2016 to 2020, to either a treatment 

group that is offered access to the program, or to a control group that is not.19 We will assess the 

effect of NFP on a range of short- and long-run maternal and child outcomes using 

administrative data that will be available for all members of both treatment and control groups. 

This will yield useful evidence for South Carolina and for policy makers nationally, who are 

interested in the broader health and financial consequences of expanding Medicaid to include 

similar services.

 19 The Abdul Latif Jameel Poverty Action Lab. “The Impact of a Nurse Home Visiting Program on Maternal and Child Health 

Outcomes in the United States.” Accessed October 24, 2016. https://www.povertyactionlab.org/evaluation/impact-nurse-homevisiting-program-maternal-and-child-health-outcomes-united-states
The Promise of Evidence-Based Policymaking 665

9

Appendix to the Original Written Statement

During J-PAL North America Executive Director Quentin Palfrey’s testimony at the public CEP 

hearing in Chicago on January 5th, Commissioner Abraham asked a question about J-PAL’s 

views on user fees. To clarify our initial statement, we do not suggest that data repositories avoid 

charging fees in general. We suggest only that data repositories avoid flat per-user fees for data 

access, since these can discourage researchers from having multiple team members access the 

data and reduce opportunities to prevent data analysis errors. As is customary, we recommend 

that a clearinghouse charge reasonable fees for accessing data, such as a fee per project, a fee for 

sets of users, or an initial fee for the first user followed by much smaller fees for additional users.

As one example, J-PAL has ongoing research partnerships with the state of South Carolina 

which have been benefited from South Carolina’s integrated data system and clear data access 

procedures. The integrated data system South Carolina has built over time, which is housed at

the Revenue and Fiscal Affairs Office makes it easier for government and independent 

evaluators to assess the impact of government programs. This has made South Carolina a 

promising location for researchers interesting in answering crucial policy-relevant questions 

related to maternal and child health, the opioid crisis, health insurance rating systems, and much 

more. As just one example, South Carolina’s Department of Health and Human Services 

(SCDHHS) has partnered with J-PAL affiliates to use this infrastructure to evaluate how 

expanding the Nurse-Family Partnership through a pay-for-success contract affects the health 

and long-term economic outcomes of mothers and children enrolled in the program.

Commissioner Haskins also asked us to provide examples of specific datasets for which the 

Commission should prioritize improving access. Based on conversations with our professional 

staff and affiliates, we suggest prioritizing access to administrative data on earnings and income 

that are currently hard to access, such as unemployment insurance wage records and Social 

Security Administration data. There are a number of administrative datasets for federal programs 

that are maintained at the state level, such as records for UI, SNAP, and TANF, and researchers 

must gain access to these data on a state-by-state basis. There is a great deal of arbitrary variation 

between states in the application process, the time it takes for applications to be considered, and 

the likelihood that an application is approved. These state-level access burdens can be very 

discouraging for researchers interested in important policy questions related to these programs. 

We suggest that the Commission look for ways to ease access to these state-level datasets by 

harmonizing data collection and application processes. For example, the Commission could 

recommend a carrot/stick strategy to encourage states to share their data with CARRA for 

research purposes, allowing researchers with CARRA-approved projects to access these data 

more systematically. The Commission could use a similar strategy to encourage the use of

common metrics across states so data can be more readily linked across states.

In addition, the Commissioners expressed interest in examples of institutions or structures that 

have successfully created opportunities for policy-relevant research and partnerships between 

academics and government. J-PAL has worked with a number of governments internationally 

and in the United States to create mutually beneficial partnerships between these public entities 

and academic researchers. These partnerships have allowed public officials and academics to 
666 The Promise of Evidence-Based Policymaking

10

work together to identify important research questions, conduct high-quality evaluations, and 

implement the learnings from these evaluations into policy action.

We have found that specific structures can be put in place that facilitate governments sharing the 

priority questions they would like research to address, and then developing specific prospective 

research projects in close collaboration with external researchers. The intent is to have 

prospective evaluations emerge not in isolation either from government or from external 

researchers alone, but rather as part of an on-going collaboration. Two examples include sectorspecific commissions and innovation initiatives. 

Commissions that have a mandate to test policy innovations in a particular policy area are 

one example of such a structure that has been implemented internationally. For example, 

MineduLAB is an innovation lab for education policy housed within the government of Peru. 

The MineduLAB team of four monitoring and evaluation professionals is nested inside the 

Ministry of Education (Minedu) Secretariat of Strategic Planning (SPE). The lab pilots and 

evaluates the effectiveness of innovations with the ultimate goal of allowing the Ministry to use 

evidence to improve children’s learning throughout the country.

Critical to this success is the lab’s close ties with academia. The collaborative MineduLAB 

process combines the researchers’ expertise with the political and operational experience of

policymakers, ensuring that the innovations tested before scaling-up are informed by existing 

rigorous evidence and cutting-edge theories on education and behavioral economics.

Innovation Initiatives can create a mechanism for developing collaborative research 

partnerships that focus on policy maker’s priority policy issues. International examples 

include the Government of Tamil Nadu in India, which has set up the Tamil Nadu Innovation 

Initiative, which works with researchers to call for proposals on an ongoing basis, select the most 

promising ideas based on local needs and existing evidence, design pilot programs, and 

rigorously evaluate them to understand their impact. In France in 2008, the government created

the Experimental Fund for Youth, which set aside 320 million Euros to learn which programs are 

effective in helping youth improve educational outcomes and social and professional integration. 

The fund sets aside funding for rigorous evaluations, with a focus on randomized evaluations. In 

the US, competitions such as the Government Performance Lab’s technical assistance 

competition and the J-PAL State and Local Innovation Initiative provide platforms for state and 

local governments to bring their priority research questions to external research teams.
The Promise of Evidence-Based Policymaking 667

To: Commission on Evidence-Based Policymaking (CEP)

From: Joy Bonaguro, Chief Data Officer, City and County of San Francisco

Date: January 23, 2017

Re: Issues relevant to CEP’s charge established in Public Law 114-140

Evidence-based policies help government agencies provide more efficient and effective services for

Americans. Individuals tend to support data use that improves their lives, whether it be as

consumers or citizens. At the same time, individuals expect government use of data to protect their

individual privacy and confidentiality. These two concepts need not stand in opposition to one

another - constructive and responsible data use can both improve lives and uphold privacy.

Against this context, social service delivery is in the midst of a migration from program to people

centric care. Our most vulnerable individuals touch multiple systems - education, human services,

and housing - which have historically operated in silos. The transition to coordinated,

person-centered care will better meet the needs of our clients by tailoring care to meet the needs of

each individual, rather than using a one-size-fits-all approach.

A coordinated care approach is most effective when jurisdictions and programs can share data

about the individuals they jointly serve to avoid duplication and offer the right mix of services. As a

result, evidence based policy is not simply a function of retrospective studies, but continuous

improvement, which the commission here today was charged to study. Continuous improvement is

predicated on real time data linkage to facilitate care coordination. While the bulk of the

Commission's charge is focused on data linkage for study, we also encourage the commission to

recognize the importance of care coordination to ensure quality care in real time.

The remainder of my comments focus on the Commission's duties per Public Law 114-140.

Section I. How best to incorporate outcomes measurement, randomized

controlled trials, and rigorous impact analysis

More rigorous evaluation and a focus on outcomes is essential to continuously improve government

policy and programs. The recommendations below help promote this locally.

1. Establish and harmonize outcomes-based measures for major areas of program

funding. Too many agencies require a variety of reports often focused on activities (e.g. how

many served) and not outcomes (e.g. how much better off). Those reports vary across

funding streams and over time, often resulting in costly changes to administrative data

systems. The result is time and effort spent on reporting compliance versus assessing what

works. A word of caution for any outcomes framework: not achieving an outcome is not

necessarily grounds for marking a program as a failure. Some programs may be preventing

even worse outcomes. Instead, outcomes metrics should be used to continuously improve

and modify programs; and

2. Selectively fund RCTs and lottery based services. Some portion of federal program dollars

could be predicated on lotteries leading to randomized controlled trials. While not

appropriate in all cases, a lottery requirement as a function of funding would help local

programs overcome resistance to experimenting more with lotteries. Lotteries are not

appropriate for services where a clear and consistent set of criteria is known for prioritizing

program access and program resources are sufficient.

1
668 The Promise of Evidence-Based Policymaking

Section II. Legal and administrative barriers.

Perhaps the most significant barrier to data linking is navigating a thicket of sector-specific,

jurisdiction-specific and sometimes conflicting statutes and standards regarding data privacy and

security. The recommendations below provide several avenues to remove these barriers and spur

evidence-based policy through integrated data, while maintaining privacy.

A. Statutory and regulatory changes could broaden opportunities for data sharing. The

following recommendations identify opportunities to modify privacy statutes and regulations.

1. Streamline and harmonize complicated and often contradictory federal data and

privacy regulations. Changes should allow for more flexibility in data sharing across

health, human services, housing, and education agencies, while continuing to ensure strong,

consistent privacy protections. Regulations and standards can be streamlined and

harmonized around key principles (e.g. role-based access, limited use) and consistent

definitions (e.g. of personally identifiable information);

2. Implement exemptions to existing data and privacy regulations to allow for data

sharing between local government services and agencies and with designated partners

for limited purposes. Purposes covered by exemptions should include program evaluation,

continuous improvement, policy-relevant research, cost-benefit analysis by qualified

researchers and institutions, and care and service coordination; and

3. Consider the creation of an omnibus federal information privacy law. This avenue is

supported by advocates across industry, academia and government as a means of promoting

efficiency, accuracy and integrity of information as well as encouraging a clearer

understanding of privacy requirements by individuals, industry and government.

B. Administrative actions could ease the burden of legal interpretation. Often, the most

significant barrier to confidential data sharing is not the interpretation of any single statute, but

navigating a series of statutes. Each jurisdiction navigates this thicket afresh, which concentrates

risk on individuals and localities interpreting the law and serves as a disincentive to integrated data.

More regulatory guidance per the following recommendations would help local governments

consistently and responsibly interpret confidentiality laws.

1. Provide cross-agency guidance for data sharing across multiple bodies of law (e.g.

FERPA, HIPAA, HMIS etc). While policy specific guidance is helpful, integrated guidance

across policies is more useful for the local practitioner. For example, the Administration for

Children & Families released a Toolkit for sharing data across six programs; 1

2. Provide model data use agreements (DUAs) that take into account the cross-agency

guidance. Model DUA terms, which build on the prior recommendation, could accelerate

adoption among agencies. A 2013 GAO report notes that model or example DUAs are 2

especially needed in health and human services;

3. Partner with states to provide guidance as to the interaction between state and federal

privacy requirements. State specific statutes can further complicate the legal path towards

integrated data. Developing model frameworks in partnership with states can reduce the

local burden of interpretation;

1 Available at

https://www.acf.hhs.gov/sites/default/files/assets/acf_confidentiality_toolkit_final_08_12_2014.pdf. 2

 “Human Services: Sustained and Coordinated Efforts Could Facilitate Data Sharing While Protecting

Privacy” available at http://www.gao.gov/assets/660/652058.pdf.

2
The Promise of Evidence-Based Policymaking 669

4. Convene major states and select localities to identify opportunities to harmonize

requirements and guidance. States and localities provide a “boots on the ground”

perspective on existing and emerging data linking challenges; and

5. Create and maintain a single website for cross-agency guidance. Finding timely guidance

requires hunting across multiple sites. A cross agency webiste could serve as the umbrella to

shepherd the recommendations above.

Section III. Clearinghouse and data-sharing infrastructure.

To promote evidence-based government, further investments in data and digital infrastructure,

could include the following:

1. Pilot policy “labs” or “hubs” that focus on using data integration to answer a specific

problem or issue. Pilots could focus on a key health care outcome or topic such as

recidivism. Pilots would allow experimentation at a small level and achieve “quick wins”

ahead of introducing any larger clearinghouse. Pilots could evolve in an agile manner

through continued integration of more data and eventually interoperable hubs;

2. A central service for matching personally identifiable data and then de-identifying for

the purpose of research. Matching and linking data across systems can be a challenge and

burden for local entities due to the expertise required and may disincent research

partnerships. A central service could lend consistent technical expertise to the task of data

matching; and

3. Ensure government data is broadly available as open data whenever possible. The

burden of data access is reduced when government data is proactively made available. A

combination of open and closed data is needed for evidence-based policy.

Section IV. Limitations on data access and use

Limitations on data access and use should be developed from a risk management perspective, with

the limits commensurate to the privacy or security risk posed by the data. Rather than focusing on

whether others can have data, a risk management approach allows agencies to designate who

should have access and for what purposes based on a data classification scheme. Specific methods

for managing risk in a data sharing context might include:

1. Establishing review committees to assess the risk/benefit of the proposed research;

2. Ensuring the data-sharing infrastructure supports role-based access, and developing model

protocols and procedures to support role-based access; and

3. Developing model data use agreements and/or Terms of Use for data-sharing that restrict

researchers and other individuals with access from using confidential data to re-identify

individuals or for purposes not specified in the scope of work.

3
670 The Promise of Evidence-Based Policymaking

Section V. Incentives to facilitate interagency sharing of information

Eliminating legal and administrative barriers and providing information resources and guidance, as

discussed in Section II, are major steps toward achieving greater interagency sharing of

information. Additional incentives to promote interagency sharing of information could include:

1. Allocation requirements for analytics and evaluation in federally funded programs;

2. Allocation requirements for data and infrastructure in federally funded programs to ensure

the consistent collection and availability of quality data;

3. Grant and funding criteria that reward interagency proposals; and

4. Waivers that would allow states and localities to pursue a coordinated care approach and

build the case for needed regulatory and statutory changes.

Section VI. How data and research can inform policy-makers and programs

A gap often exists between research and practice. This gap exists in at least two forms: 1) a gap

between what researchers find and its application to practice and 2) a gap between what

practitioners need and what researchers find compelling to study given a historical focus on original

research and a bias towards positive results.

To close the first gap:

1. Researchers should provide practical information and concrete steps or

recommendations to guide decision-making, not just journal articles. This could include

research with short-form extracts, sections on practical policy recommendations, data

visualizations, geographic breakdowns, or breakdowns by subgroups relevant to programs.

Frameworks for measuring impacts and evaluating programs should take into account the

resources and ability of agencies, including the recognition that academic peer review and

practical implementation may have different standards (that is don’t let perfect be the

enemy of good); and

2. Establish centers of practice responsible for translating and consolidating research

results. While journals provide the service of publishing results, few entities combine the

results into practitioner handbooks. It is the rare local agency that can stay on top of the

research across disparate fields to understand what programs or actions have the strongest

evidence base in their field.

To close the second gap of aligning incentives between researchers and practitioners, consider the

following recommendations:

1. Incent researchers to partner with programs and practitioners. Funders could require

or favor research and grant submissions that included a program or policy partner and an

applied research question; and

2. Seed research centers at local universities with joint academic and practitioner

governance. For example, the John W. Gardner Center at Stanford develops its research

agenda in partnership with the San Francisco Unified School District and a pool of funds

helps incent applied research in collaboration with the school district.

4
The Promise of Evidence-Based Policymaking 671

Commission	on	Evidence-Based	Policymaking

Doing	What	Works	in	Community-Based	Organizations	

Susan	N.	Dreyfus,	president	and	CEO

Alliance	for	Strong	Families	and	Communities

January 22,	2017

Thank	you	for	the	opportunity	to	contribute	to	the	important	work	this	Commission	on	

Evidence-Based	Policymaking	has	undertaken.	It	is	an	honor	to	speak	with	you	today	on	

behalf	of	the	Alliance	for	Strong	Families	and	Communities,	where	I	am	the president	and	

CEO.	Prior	to	joining	the	Alliance,	I	was	secretary	for	the	Washington	State	Department	of	

Social	and	Health	Services,	where	I	had	responsibility	for	Medicaid,	child	welfare,	

behavioral	health,	juvenile	justice,	economic	assistance,	and	other	human	services.	I	also	

served	as	Wisconsin’s	first	administrator	of	the	Division	of	Children	and	Family	Services.	

Most	recently,	from	2014-2016,	I	served	as	a	commissioner	for	the	federal	Commission	

to	Eliminate	Child	Abuse	and	Neglect	Fatalities.	

The	Alliance’s	network of	human	service	organizations is	grateful	that	you	are	facilitating	

this	conversation,	as	we	believe	this	is	the	right	time	for	significant	steps	forward	in	

developing	a	modern	human	services	sector	that	leads	to	stronger,	healthier families	and	

communities.	The	Alliance	strategic	action	network	is	uniquely	positioned	to	share	its	

perspective	on	evidence-based	policymaking	because	community-based	organizations	

provide	the	majority	of	services	in	the	social	sector	and	generate	the	majority	of	

outcomes	data	used	to	inform	best	practice	development.	

I	will	begin	with	some	background	information	about	the	Alliance	for	Strong	Families	and	

Communities.	The	Alliance	is	a	national	strategic	action	network	of	nearly	450	nonprofit,

community-based	organizations	dedicated	to	achieving	a	vision	of	a	healthy	and	

equitable	society for	all	children,	adults,	and	families.	Many	organizations	in	this	network	

have	more	than	100	years	of	human	service	leadership	experience.	

It	was	formed	in	1998	by	the	merger	between	Family	Service	America	and	the	National	

Association	of	Homes	and	Services	for	Children, and	expanded	during	a	2014	merger	with	

the	United	Neighborhood	Centers	of	America.	The	result is	the	nation’s	largest	network	

of	human-serving	organizations	working	with	their	neighbors	on	the	front	lines	to	

enhance	the	lives	of	Americans.	The	Alliance	works	for	transformational	change	by	

representing	and	supporting	its	network	as	members	translate	knowledge	into	best	

practices	that	improve	their	communities.	We	help	members	successfully	meet	the	

challenges	of	today	and	prepare	for	those	to	come.	

The	sector	stands ready	to	fully	engage	in	policy	conversations	that	could	shape	human	

services	and	impact	equity	and	opportunity	in	our	nation	for	years	to	come.	We	bring	
672 The Promise of Evidence-Based Policymaking

Testimony																		Alliance	for	Strong	Families	and	Communities															Page	2 of	5

extensive	knowledge	and	have	a	responsibility	to	inform	changes	being	considered	at	the	

local,	state, and	federal	levels	with	the	lived	experiences	of	the	people	we	serve.	For	

example,	when	elected	leaders	talk	about	jobs	as	the	single	solution	to	a	vibrant	

economy,	our	sector’s	voice	is	critical	to	broadening	the	lens	to	encompass	the	social	

determinants of	health	and	the	latest	advances	in	neurosciences	to	that	conversation.	As	

important	is	our	effort	to	advance	equity	across	our	country	to	ensure	all	of	our	citizens	

have	equal	access	and	opportunity	to realize	their	full	potential	and	share	in	the	

prosperity	of	our	great	nation.		

There	are	three	main	ideas	I	would	like	to	discuss	with	you	today:

1. Context	matters.	Organizations	must	align	and	integrate	with	others	in	their	

ecosystems	to	promote	efficiencies	and	better	outcomes.	

2. Developing	a	deeper	appreciation	for	research	evidence	among	human	

service	staff	can	produce	better	outcomes	for	children	and	families.

3. Evidence-based	and	informed	practices	need	to	be	implemented	within	a	

research	and	development	infrastructure	that	is	equipped	to	support fidelity,	

quality	service	delivery,	positive	outcomes,	and	innovation.	

Context	Matters

Organizations	have	the	capacity	to	be	feeders	of	knowledge,	but	they	face	contextual	

factors	that	are	outside	of	their	control.	The	context	in	which	people,	programs,	and	

systems	operate	must be	taken	into	consideration	when	discussing	evidence-based	

programs	and	policymaking.	If	organizations	embrace	evidence-based	outcomes	without	

aligning	with	other	organizations and	sectors in	their	ecosystem,	the	intended	outcomes	

are	less	likely	to be	achieved.	Take	the	case	of	a	community-based	organization	that	

enrolls	a	family	in	an	evidence-based	program	to	meet	the	family’s	needs.	The	same	

family	has	children	who	attend	a	school	with policies	and	procedures	that run	counter	to

the	latest	knowledge	in	brain	development.	As	human-serving	programs	apply	evidence,	

they	often	come	across	external	barriers. The	context	in	which	social	sector	organizations	

operate	needs	to	support	their	use	of	evidence.	

It	is	important	to	think	about	the	four	levers	of	systems	change.	We	always	focus	on	the	

first	one, practice, but	we	rarely get	to	the	other	three:	policy,	regulatory,	and	fiscal	

change.	The	Alliance, through	our	strategic	action	network,	is	aiming	to	be	more	

intentional	about	the	use	of	research	and	its	implications	for	our	work	beyond	merely	

changing	practice	and	interventions. Because	if	we	change	practice,	it	changes	for	one	

person,	one	worker,	and	one	agency,	at	one	given time.	But	until	the	evidence	gets	

hardwired	into	policy	and regulatory	change,	along	with	fiscal	alignment,	we	will	have	

programs	operating	in	unsupportive environments.	True	systems	change	requires	us	to

pull	all	four	levers,	not	just	the	quick	and	easy	lever	of	doing	our	practice	differently. If

the	other	three	levers	are not	also	addressed, evidence-based programs	are here	today,	

gone	tomorrow,	and	we	will	not attain	sustainable	and	meaningful	systems	change	that	

can	truly	improve	the	lives	of	children,	families,	and	communities.
The Promise of Evidence-Based Policymaking 673

Testimony																		Alliance	for	Strong	Families	and	Communities															Page	3 of	5

When	I	was	secretary	in	the	state	of	Washington,	I	asked	researcher	Dr.	Mark	Courtney	

to	look	at	all	of	the	child	neglect	cases in	our	child	protection	agencies. He	found	that	two	

of	the	biggest	challenges	these	families	faced	were	related	to	housing	and	economics.	

The	challenges	were	unrelated	to what	the	child	welfare	agency	was	doing	or	not	doing

to	support	those	families.	It	was	a	much	broader	picture	than	whether	the	programs	that	

we	had	control	over	were	working. For	this	reason,	it is critical	that	systems	are

integrated	so	they	are	familiar	with the	programs	and	services	offered	by	one	another	

and	can	help	put	needed	services	in	place	in	a	timely	manner	for	families.		

During	my	experience	on	the	Commission	to	Eliminate	Child	Abuse	and	Neglect	Fatalities,	

I	was	reminded	of	a	principle	in	social	psychology	called	the	fundamental	attribution	

error,	thanks	to	testimony	from	Dr.	Gary	Melton	at	the	Kempe	Center	in	Colorado.	This	

principle	is	the	tendency	to	attribute	behaviors	to	a	person’s	internal	qualities	while	

underestimating	situational	influence.	Both	the	general	public	and	human	service	

professionals	tend	to	overemphasize	individual	factors	and	underemphasize	situational	

ones.	Organizations	focus	on	what	they	are	doing	to	help	one	person,	but	they	have	a	

limited	ability	to	influence	the	context	in	which	that	person	lives	every	day.	We	know	

from	the	research	there	are	social	determinants	of	health	that	influence the	well-being	of	

children,	families,	and	communities.	For	example,	if	a	program	seeks	to	connect	a	family	

to	medical	care	or	substance	abuse	treatment,	it	may	need	to	examine	the	underlying	

issues	of	education	and	employment	in	order	to	support	that	family’s	success.	The	

recommendation	is	that	context	matters,	and	health	and	human	services	leaders	must	be	

encouraged	to	align	and	integrate	with	others	who	operate	in	their	ecosystems	in	order	to	

achieve	efficiency	and	better	outcomes.

Developing	a	deeper	appreciation	for	research	evidence	among	staff	can	produce	better	

outcomes	for	families.

Evidence	from	child	welfare	research	can	and	should	be	used	when	making	decisions,

whether	those	decisions	are	being	made	by	policymakers,	agency	directors,	or social	

workers	who	directly	interact	with	families.	However,	there	is	a	lack	of	research	about	

how	research	evidence	is generated	and	used	and	whether	its	use	produces	better	

outcomes	for	children	and	families.	To	help	answer	these	questions,	26 private	child	

welfare	agencies,	some	of	which	are	members	of	the	Alliance	network,	participated	in	a	

project	on	research	evidence	use	funded	by	the	W.T.	Grant	Foundation	in	2013-2014.	

The	project	was	led	by	Fred	Wulczyn,	Ph.D.,	senior	research	fellow	at	Chapin	Hall	Center	

for	Children	at	the	University	of	Chicago.	This	project	sought	to	accomplish	three	main	

objectives:

1.			Establish	the	extent	to	which	individuals	who	work	for	child	welfare	

agencies	use	research	evidence	when	making	decisions	about	how	to	

work	with	children	and	families.

2.			Determine	whether	staff	use	of	research	evidence	is	associated	with	

characteristics	of	agencies	and	improvements	in	the	permanency	rate	at	

the	agency	level.

3.			Examine	the	connection	between	evidence	use	and	outcomes.
674 The Promise of Evidence-Based Policymaking

Testimony																		Alliance	for	Strong	Families	and	Communities															Page	4 of	5

The	findings	of	this	research	indicate	that	developing	a	deeper	appreciation	for	research	

and	evidence	use	among	staff	can	produce	better	outcomes	for	families.	Findings	

include:

• Child	welfare	staff	with	a	positive	attitude	toward	evidence,	more	child	welfare	

experience,	and	the	know-how	to	find	research	evidence, were	more	likely	to	

acquire	research	evidence	as	part	of	decision	making.

• Agencies	with	more	staff	who	use	research	evidence	move	children	to	

permanency	significantly	more	quickly	than	agencies	with	less	staff	who	use	

research	evidence.

• Leadership	and	organizational	investment	are	linked	to	more	evidence	use	by	

staff.

• Agencies	that	generate	their	own	research	evidence	have	the	greatest	effect	on	

improved	outcomes	for	children	and	families.	

The	policy	implications	for	these	findings	are	related	to	talent	and	workforce	

development,	especially	staff	attitude,	skills,	and	use	of	evidence.	The	results	indicate	

that	an	investment	in	staff	development—particularly	around	fostering	positive	attitudes	

about	evidence	and	strengthening	the	ability	to	find	and	use	evidence—would	bolster	

the	chances	of	success	for	families	engaged	in	those	programs. Supervisors	can	play	a	key	

role	in	supporting	the	use	of	evidence	by	front-line	workers. The recommendation	I

submit	on behalf	of	the	Alliance	network	is	to invest	in	workforce	development to	lift	skills,	

foster	positive	attitudes,	and	improve	organizational	culture	to	elevate	the	use	of	research	

evidence	and	promote	positive	outcomes	for	families.

Evidence-based	and	informed	practices	need	to	be	implemented	within	a	research	and	

development	infrastructure	that	is	equipped	to	support	fidelity,	quality	service	delivery,	

positive	outcomes,	and	innovation.	

Programs	that	implement	evidence-based	practices	are	less	likely	to	achieve	intended	

outcomes	if	the	practice	is	not	implemented	with	fidelity	or if	the	context	in	which	the	

program	operates	is	not	supportive.	In	other	words,	having	an	evidence-based	practice	is	

not	a	recipe	for	success.	

Programs	should	be	involved	in	continuous	quality	improvement	evaluations,	along	with	

determining	which	practices	can	be	proven	to	be	effective	at	any	given	time.	The	

American	Public	Human	Services	Association	encourages	public	human	service	leaders	to	

“invest	in	infrastructure	capacities	to	scale	up	promising	approaches	and	evidenceinformed/evidence-based	practices.” 1 We	should	promote	evidence-informed	practices	

that	improve	outcomes	while	also	ensuring	that	we	don’t	get	in	the	way	of	practical,	lowcost	supports	that	can make	a	big	difference	to	a	family’s	health	and	well-being.

 1 APHSA’s	2016	Annual	Report.
The Promise of Evidence-Based Policymaking 675

Testimony																		Alliance	for	Strong	Families	and	Communities															Page	5 of	5

There	is	wisdom	from	the	medical	community	that	can	be	beneficial	for	the	human	

services	sector,	especially	child	welfare,	to	consider.	For	example,	medical	professionals	

attend	grand	rounds,	formal	conferences	in	which	experts	present	and	discuss	clinical

issues	intended	to	be	educational	for	all	participants.	A	child	welfare	version	of	grand	

rounds,	where	a	specific	case	is	presented	and	barriers	identified,	could	be	useful	to	

social	workers	and leaders	in	the	field.	The	medical	community	also	has	standards	of	care	

for	medical	conditions.	Child	welfare	might	consider	developing	standards	of	care	for	

different	types	of	abuse	and	neglect	so	that	professionals	know	what	steps	to	take	and	

how	to	connect	children	and	families	to	resources	and	other	systems.	

Organizations	need	to	be	innovation-driven.	The	Center	on	the	Developing	Child at	

Harvard	University	promotes	an	approach	to	innovation	that	takes	risks,	shares	results	

early,	and	learns	quickly	from	ideas	that	do	not	work2

.	When	an	organization	rewards	

innovation,	people	can	work	together	to	test	ideas	and	engage	in	active	learning.	These	

ideas	should	be	grounded	in	the	latest	science	available	and	informed	by	on-the-ground	

experience.	The	Center	recommends	working	across	sectors	to	test	promising	ideas,	

learn	from	failure,	and	promote	fast-cycle	sharing.	It	is	critical	that	policy,	funding,	and	

professional	environments	support	this	type	of	experimentation	and	entrepreneurial	

investment	in	new	ideas.	The	recommendation	is	to	invest	in	a	research	and	development	

infrastructure	that	is	equipped	to	support fidelity,	quality	service	delivery,	positive	

outcomes,	and	innovation.	

Conclusion

It	is	important	to	our	sector	that	any	changes	being	proposed	to	promote	evidence-based	

policymaking	will	result	in	gains,	not	losses,	for	children,	families,	and	communities.	As	

we	consider	directing	resources	toward	programs	that	produce	outcomes,	elevating	our	

network’s	on-the-ground	experience	is	essential.	Legislative	solutions must	align the	

practice,	policy,	regulatory,	and	financial	aspects	of	how	we	do	our	work	in	order	to yield

the	best	outcomes	that	all	families	deserve.	Making	sure	that	programs	are	operating	in	

contexts	that	support	the	use	of	evidence,	building	the	infrastructure	and	human	capital

for	research	evidence,	and	focusing	on	research	and	development	investments are	

critical	considerations	for	any	policy	about evidence-based	practices.	

It	is	a	pleasure	to	share	this	testimony	with	you,	and	I	look	forward	to	hearing	more	

about	the	Commission’s	findings	in	the	coming	months.	I	welcome any	questions	about	

my	remarks.

 2 http://developingchild.harvard.edu/innovation-application/key-concepts/science-based-innovation/
676 The Promise of Evidence-Based Policymaking

 

Education Liberty Watch: 9601 Annapolis Boulevard N Maple Grove, MN 55391*office@edlibertywatch.org*952-361-4931*www.edlibertywatch.org

Florida Stop Common Core Coalition: 116 Cousley Drive SE Port Charlotte, FL 33952 * office@flstopcccoalition.org *888-376-5550 *www.flstopcccoalition.org

Comments to the Commission on Evidence-Based Policymaking Opposing Lifting the Prohibition on a Student Unit Record 

System

Karen R. Effrem MD

President of Education Liberty Watch & Executive Director of the Florida Stop Common Core Coalition

The thousands of parent, grandparents, teachers, and citizen supporters across the nation of Education Liberty Watch and 

the Florida Stop Common Core Coalition, as well as many allied groups1 representing hundreds of thousands more that have 

repeatedly raised student data privacy concerns, are firmly against the lifting of the prohibition on a federal student unitrecord for many reasons. Here are a just a few:

 The government has no constitutional or moral right to collect data and offer it, without consent, to governmental 

and non-governmental organizations for research without consent. It is a blatant violation of the Fourth Amendment 

to the US Constitution and of the numerous Supreme Court precedents affirming parents’ inherent right to direct 

the education and upbringing of their children as outlined in this letter from Liberty Counsel.2

 The Obama administration has already gutted the outdated and fairly weak data protections of Family Educational 

Rights and Privacy Act (FERPA) for K-12, allowing multiple federal agencies and outside researchers to have access to 

sensitive student and family data without parental consent.3 Lifting the student unit-record prohibition will create a 

womb-to-tomb database of information that citizens will not be able to monitor themselves for any purpose, 

including to correct errors. Access to this data by so many entities will likely have life-altering consequences for the 

individual regarding college, jobs, military service, insurance, and other issues. Privacy expert Barmak Nassirian4 said 

of this idea:

“Tracking autonomous free individuals through most of their lives in the name of better information for the 

benefit of others may be justifiable, but its extremism should at the very least be acknowledged and 

addressed.”

 There is abundant evidence from multiple federal5 and state bills, from the National Assessment of Educational 

Progress (NAEP) effort to track mindsets and school climate data, from the National Commission on Social Emotional 

Learning,6 from the new Head Start Performance Standards,

7 from policy efforts of foundations8 and business 

groups,9 etc. that there is a concerted effort to psychologically profile American students and workers starting at a 

very young age. This effort makes the plan to lift the prohibition on the student unit-record even more dangerous, 

very likely leading to the egregious violation of the most fundamental of American rights, the private right of 

conscience.

 1 http://edlibertywatch.org/wp-content/uploads/2014/11/Congressional-ESEA-letter-10-24.pdf 2 Liberty Counsel Letter Regarding Assessing Mindsets in the NAEP - http://edlibertywatch.org/wp-content/uploads/2014/11/Final-LtrNAEP-legal-and-privacy-concerns-06272016.pdf 3 http://edlibertywatch.org/2014/01/formal-response-to-the-chief-state-school-officers-letter-on-student-data-privacy/ 4 http://www.studentprivacymatters.org/barmak-nassirian-is-the-student-right-to-know-bill-worth-the-risk-to-privacy/ 5 http://www.flstopcccoalition.org/files/7C7D2880-68E4-4C61-AF31-FF29E8FA9789--B4BCFE9D-8881-4C5A-B267-D9863CA280DB/setrahandout-final.pdf 6 https://www.conservativereview.com/commentary/2016/11/think-common-core-is-bad-new-standards-crank-the-creep-factor-up-toeleven

7 http://truthinamericaneducation.com/federalized-education/enter-baby-common-core-new-head-start-performance-standards/ 8 http://www.casel.org/from-casel/ 9 http://www.usnews.com/news/articles/2016-05-13/op-ed-educational-data-mining-can-enhance-science-education
The Promise of Evidence-Based Policymaking 677

 We believe that student privacy and parental consent should always be considered pre-eminent compared to the 

research desires of the government or private sector, especially in the realm of psychological profiling.

 Congress and the rest of the federal government already ignore research that does not fit the paradigm of 

expanding the federal role in education and further usurping the parental role. Disastrous federal welfare policies

that subsidize unwed parenthood have resulted in terrible academic performance, more out-of-wedlock births, and 

increased rates of crime, drug abuse, suicide, etc. Multiple studies have shown that very few, if any, of the 

government programs designed to address these government-created problems – for example, schoolimprovement10 early childhood,11 or home visiting programs

12 -- have had significant sustainable benefit or reached 

their stated national goals.13 According to data presented to the US House Education and Workforce Committee by 

the Cato Institute several years ago, federal involvement in education has yielded either stagnant or declining 

academic performance:

And often federal education programs have made things worse. But of course, the federal government simply 

disregards the research it disagrees with – and insists, through advocacy of a unit-record system, that it should be 

given even more data for even more research, which will be ignored if it reaches the “wrong” conclusions. The 

research of the esteemed co-chairman, Dr. Haskins, is an important example of this situation as he has authored 

several early-childhood studies, including analysis of Head Start14 and Abecedarian,

15 and has seen first-hand the 

failure and or harm of these programs on poor children. Regardless of his views on lifting the prohibition on the 

creation of a student unit-record system, it is somewhat ironic that he is serving on this commission that is hearing 

so many pleas to expand access to more privacy-destroying data for more government programs.)

 The US Department of Education (USED) has terrible data-security:

The US House Oversight and Government Accountability Committee chaired by Rep. Jason Chaffetz held two 

hearings dealing with the deplorable state of student data-security in USED. Here are some of its findings 

(emphasis added):

At the November 17, 2015 hearing,

16 the Committee learned that the US Department of Education:

 10 https://edexcellence.net/commentary/education-gadfly-daily/flypaper/2012/the-disappointing-but-completely-predictable-resultsfrom-SIG.html 11 http://edlibertywatch.org/wp-content/uploads/2014/11/Updated-Preschool-Compilation-4-16.pdf 12 http://thefederalist.com/2016/07/13/sending-government-agents-into-peoples-homes-wont-fix-preschools-failures/ 13 http://thefederalist.com/2016/12/07/7-things-betsy-devos-needs-immediately-becomes-education-secretary/ 14 Dr. Haskins was on the Head Start Research Committee involved in Head Start Impact Study that showed very early fade out and harm 

to math skills of three year olds - https://www.acf.hhs.gov/sites/default/files/opre/hs_impact_study_final.pdf 15 Dr. Haskins’s study found that Abecedarian children were “more aggressive than children in the control group” in elementary school. 

http://www.jstor.org/stable/1129759?seq=1#page_scan_tab_contents 16 https://oversight.house.gov/hearing/u-s-department-of-education-information-security-review/
678 The Promise of Evidence-Based Policymaking

o Holds 139 million unique Social Security numbers;

o Continues to be "vulnerable to security threats" according to the IG, and has repeat 

findings in annually required FISMA audits;

o Failed to detect a penetration test of its systems conducted by the IG during its 

FY2015 [Federal Information Modernization Security Act];

o Received an "F" on the Committee's FITARA scorecard. [Federal Information Technology 

Acquisition Reform Act]

At the February 2, 2016 hearing,

17 the Committee learned:

o The Department of Education's (ED) Chief Information Officer (CIO) Danny Harris 

received substantial bonuses despite poor performance in securing IT systems at the 

Agency and significant ethical lapses in judgment.

o Despite the IG's evidence to the contrary, Acting Secretary King asserted that Mr. Harris 

did not violate any law, regulation, policy, or standard of ethical conduct.

o Mr. Harris testified his home theatre installation and car detailing activities were 

"hobbies" and not businesses. The IG testified that these activities qualified as businesses.

o It was in excess of two years before ED responded to the IG's initial report of findings 

and referral for administrative action.

o The Department of Justice (DOJ) declined to prosecute the IG's criminal referral and 

deferred to ED leadership for action. Acting Sec. King deemed verbal counseling and a three 

page ethics guidance letter as appropriate consequences.

Does anyone really want psychosocial research and assessment results housed in the US Department of Education 

under these circumstances?

Perhaps most disturbing is the appearance of arrogant dismissal of citizens’ privacy concerns by in this discussion during this 

Commission’s meeting that occurred on December 13, 2016.

18

A question asked by Commission member Bruce Meyer19 to witness Ron Jarmin of the Census Bureau starting at 

1:26:47:

“Let me try and ask what I think is a very difficult question, and I don’t expect you to be able to answer it, 

but maybe we can start a conversation that could be useful to us. So, I see Census as having made a lot of 

steps to move in the kinds of directions that are suggested or anticipated by the Commission bill, in that you 

are working to bring data from other agencies or you have, into the RDCs — you’ve broadened their mission 

and you are bringing together data from many agencies and allowing researchers in and outside of 

government to access the data that you’ve brought together. What are the ways that you could expand 

those efforts? Um, and I’m not suggesting that we talk about a single statistical agency across 

government, but how could there be more of a coordination or maybe a virtual one statistical 

agency where Census is playing a coordinating role, or what kinds of movements in that direction should we 

think about? What kinds of things have you thought about? What are the barriers to moving towards more 

coordination between the statistical agencies?”

The response at 1:28:54 from the Mr. Jarmin:

“...One of the biggest constraints that everybody involved in this sort of endeavor faces is the different rules 

that are attached to data that are sourced from different agencies or different levels of, you know, whether 

 17 https://oversight.house.gov/hearing/u-s-department-of-education-investigation-of-the-cio/ 18 https://www.youtube.com/watch?v=MXasJLAWgtc as transcribed at https://whatiscommoncore.wordpress.com/2016/12/13/fedscomment-on-childrens-privacy-ripping-the-band-aid-would-probably-not-fly/ 19 https://www.cep.gov/commissioners/meyer.html
The Promise of Evidence-Based Policymaking 679

it’s federal or state... that if there was broad agreement in, that, you know, if there was one law that prosc–

[sic – word not fully pronounced] had the confidentiality protections for broad classes of data, as opposed 

to, you know, here’s data with PII [personally identifiable information] on it that’s collected from SSA, 

here’s data with PII on it that’s collected from the IRS; here’s data with PII on it that’s collected from a 

state; versus from a statistical agency– if data with PII on it was treated the same, you know I think that 

would permit, you know, organizations that were collecting PII-laden data for different purposes to make 

those data available more easily. Now, that’s probably a pretty heavy lift... to show people through 

examples like that, that there is not an explosion and the world keeps working, maybe even works better, 

um, to do this in sort of baby steps showing by example, as opposed to trying to rip the BandAid off. I think ripping the band-aid would probably not fly.”

This attitude that all PII, no matter from where it comes, belongs to the federal agencies and corporate researchers, is 

extremely distasteful to American citizens,20 especially for parents when it comes to womb-to-tomb education and health 

data about their children. This is especially true when one considers that this is advocated by the Census Bureau, whose only 

constitutional charge is to:

Representatives and direct Taxes shall be apportioned among the several States which may be included within this 

Union, according to their respective Numbers...The actual Enumeration shall be made within three Years after the 

first Meeting of the Congress of the United States, and within every subsequent Term of ten Years, in such Manner 

as they shall by Law direct.21

We therefore strongly recommend the following for CEP and for Congress: 

1. Retain the prohibition of a student unit-record system.

2. Strongly consider a moratorium on further federal research until programs already shown to be ineffective and 

harmful are transformed or eliminated and until effective measures are actually implemented.

3. Prohibit social emotional data-gathering and the use of data for predictive testing in the FERPA and Strengthening 

Education Through Research Act (SETRA) reauthorizations.

4. Put in strict data-transparency language and update the data-security language per the recommendations of 

technical experts like Dr. Joel Reidenberg22 or Barmak Nassirian23 in any FERPA reauthorization.

5. Require third-party software and testing vendors to notify parents of what data is collected on students and how 

it is used.

6. Find a way for students whose identity and privacy is compromised to be compensated, in addition to penalizing

researchers or private vendors that breach data-security. 

7. Close the curriculum and assessment loophole for invasive surveys in the Protection of Pupil Rights Amendment.

24

8. Demand that the US Department of Education immediately repair the federal data-security failures found in the 

Inspector General’s recent report and uncovered by the House Oversight Committee.

 20 http://thepulse2016.com/karen-r-effrem/2016/06/22/memo-to-big-data-parents-are-not-fearful-they-are-furious-about-data-mining/ 21 U.S. Constitution, Article 1 Section 3 Clause 1 22 http://edworkforce.house.gov/uploadedfiles/reidenberg_testimony_final.pdf 23 http://www.studentprivacymatters.org/barmak-nassirian-is-the-student-right-to-know-bill-worth-the-risk-to-privacy/ 24 http://edlibertywatch.org/wp-content/uploads/2015/02/Common-Core-FERPA-and-ESRA-Update-for-Senate.pdf
680 The Promise of Evidence-Based Policymaking

TESTIMONY

Before the

Commission on Evidence-Based Policy Making—Public Hearing

San Francisco, CA

February 9, 2017

Hello, my name is Maryann Feldman. I am a professor in the department of 

Public Policy at the University of North Carolina, Chapel Hill.

I am speaking as a member of the community of scholars that seeks to 

advance the scientific basis of science and innovation policy. My testimony 

outlines some of the needs of that community related to 1) open science, (2) 

ready access to administrative data, (3) rationalization of data access and use 

polices, (4) the important for researchers to understand context or for them to 

collaborate with practitioners, which also enhances trust, and finally (5) 

offering suggestions for a governance structure for data access and use while 

maintaining privacy.

My own research uses a variety of administrative data from public, private 

and non-profit organizations at the local, state and national level. I am a 

recipient of funding under the National Science Foundation (NSF) Science of 

Science and Innovation Policy (SciSIP) program, which is part of the Division 

of Social, Behavioral and Economic Sciences. The program funds research to 

develop models, analytical tools, data and metrics that can be applied in the 
The Promise of Evidence-Based Policymaking 681

2

science policy decision making process and affect the use and allocation of 

scarce scientific resources. 

I am currently on detail to the NSF. The views expressed here do not 

necessarily represent the views of NSF or the U.S. Government. In my 

comments, I represent my own views as a researcher and member of the 

Science of Science Policy (SoSP) community of interdisciplinary researchers 

and policymakers, who seek to provide a scientifically rigorous basis for 

informing science policy. 

The fundamental question that motivates the Science of Science and 

Innovation Policy community is, how can the United States use scarce 

scientific resources more efficiently and effectively to promote scientific and 

technological progress, economic development and economic growth. This 

research is empirical and relies on access to data. 

It is only by studying the scientific research enterprise, the range of actors 

engaged in innovation and specific public programs and incentives that 

scholars can provide evidence to policy makers. 

The United States conducts $456 billion of research and development (R&D) 

activity annually (NSF 2015). The Federal government, the second largest 

R&D funder behind business, invests $122 billion each year through the work 

of over 15 federal mission agencies (e.g., Department of Defense and 

Department of Health and Human Services) and 70 sub-agencies (e.g., the 

Defense Advanced Research Projects Agency and National Institutes of 

Health), as well as 15 independent agencies such as the National 

Aeronautics and Space Administration (NASA) and the National Science 

Foundation (NSF). 
682 The Promise of Evidence-Based Policymaking

3

Understanding this diverse and dynamic system is a daunting task. Yet our 

ability to study and understand this field is greater now that it has ever been 

because of our ability to work with new digital data sources, and benefit from 

advances in research design and empirical methods. Progress for my

community of scholars is dependent on access to administrative data. I 

applaud the mission of the Commission and want to be sure that the federal 

science agencies are included in your efforts to make administrative data 

available.1

I would like to offer three examples of projects that used administrative data 

to provide policy-relevant analysis. Federal agencies routinely publish 

information on funded projects yet access to all applications is rare. Access 

to administrative records enables analysis of differences between funded and 

non-funded projects and allows the construction of quasi-experimental 

research designs to better discern causal effects. 

Each of these three projects I mention was conducted through special 

negotiated agreements that involved contracts or exceptions to gain access to 

proposal submissions. All have been published in top journals.

First, Ginther et al (2011) conducted the first systematic investigation of racial 

and ethnic differences in National Institutes of Health (NIH) funding, using 

88,000 Research Project Grants (R01), which is the most common 

mechanism NIH uses to fund science. The NIH IMPAC II grant database

provides applicant’s self-identified race or ethnicity. Ginther et al (2011)

 1 There	is	utility	to	differentiating the	difference	between	administrative	data	and	

scientific	data.	Administrative	data	capture agency	processes,	policies,	and	

outcomes in	pursuit	of	their	mandated	mission.		In	contrast,	scientific	data	are the	

product	of	the	agency’s	efforts.
The Promise of Evidence-Based Policymaking 683

4

found, after controlling for the applicant’s educational background, country of 

origin, training, previous research awards, publication record, and employer 

characteristics, that black applicants were 10 percentage points less likely 

than whites to be awarded NIH research funding. Published in Science, this 

work was co-authored with Raynard Kington, former Deputy Director of the 

National Institutes of Health, and Walter Schaffer in the Office of the Director 

at NIH.

With this evidence, NIH Director Francis Collins, Ph.D., M.D., established 

the Working Group on Diversity in the Biomedical Research Workforce to 

develop effective strategies to recruit and promote the professional growth of 

underrepresented groups in biomedical research, from graduate study to 

granting of tenure. Based on the findings of the Working Group, NIH made a 

$500 million, ten-year commitment to improve diversity through 

• Increasing the number of underrepresented minorities who study 

biomedicine (Building Infrastructure Leading to Diversity); 

• Creating a national mentoring network (National Research Mentoring 

Network); and, 

• Hiring a Chief Officer for Scientific Workforce Diversity.

Second, Lui and Agha (2015) examined the success of peer-review panels in 

predicting the future quality of proposed research using administrative data to 

track publication, citation, and patenting outcomes associated with more than 

130,000 (R01) grants funded by the NIH from 1980 to 2008. Their analysis 

found that better peer-review scores are consistently associated with better 

research outcomes and that this relationship was robust even after 

accounting for an investigator’s publication history, grant history, institutional 

affiliations, career stage, and degree types. A one–standard deviation worse 

peer-review score among awarded grants is associated with 15% fewer 
684 The Promise of Evidence-Based Policymaking

5

citations, 7% fewer publications, 19% fewer high-impact publications, and 

14% fewer follow-on patents. The results suggest that peer-review is doing a 

good job of picking projects. Further research can help program managers 

further refine the process. 

Third, Sabrina Howell, in her dissertation and a forthcoming article in the 

American Economic Review, conducted the first large-sample, quasiexperimental evaluation of R&D subsidies using administrative data on 

ranked applicants to the U.S. Department of Energy’s Small Business 

Innovation Research (SBIR) grant program. The results found that an early 

stage award approximately doubles the probability that a firm receives 

subsequent venture capital and has large, positive impacts on patenting and 

revenue, with stronger effects for more financially constrained firms. The 

results suggest that the SBIR program is accomplishing its objective: the 

results are not due to certification, where the award contains information 

about firm quality, but instead the SBIR program funds are used for

technology prototyping, helping commercialize new ideas. 

These readily accessible academic studies are in contrast to the often-used 

contract mechanism, which provides access to administrative data while 

limiting publication rights. Projects with limited publication rights are not 

useful for academic careers, thereby deterring academics from investing in 

such research. As a result, much of the analysis of these data is done by

consultants, of varying quality and motives, conducting rather routine 

analyses, usually not employing the most advanced methods nor publishing 

the results in the peer-reviewed scientific journals. 

These few examples demonstrate the need for ready access to administrative 

records to inform science policy and address the practices of federal 
The Promise of Evidence-Based Policymaking 685

6

agencies. In these few studies researchers have negotiated access on an 

individual basis, overcoming each agency’s own unique procedures for 

accessing non-funded proposals, reviewer and panel rankings, and 

administrative records, which include additional principal investigator 

characteristics, project monitoring data and outcomes information. 

Imagine what might be possible if procedures for access to administrative 

data were streamlined making their use relatively painless. This would 

require amending the Federal Information Security Management Act of 2002, 

and subsequent updates, which provide the framework of information security 

standards for the federal government’s information systems. One 

straightforward work around is to ask program applicants to provide informed 

consent to use their data for research purposes. 

There are currently no standard procedures for researchers to gain access to 

federal science agency administrative data. The situation can be even more 

onerous for obtaining data from state and other publicly funded programs. 

Existing procedures for gaining access, where they exist, are time-consuming 

to navigate, difficult to understand and present barriers to the use of 

administrative data. Certainly, removing legal barriers that limit access for 

the research community is a needed first step. 

In my work with federal agencies, one of the things that surprised me was the 

great degree of heterogeneity. Each agency has its own policies and 

procedures for gaining access to administrative data. For example, NIH 

established internal policies to determine when data from their records 

management systems can be disclosed for research purposes.2 NIH provides 

different levels of access depending on a researcher’s role. Federal 

 2 https://oma.od.nih.gov/forms/Privacy%20Documents/PAfiles/0036.htm
686 The Promise of Evidence-Based Policymaking

7

researchers can access data if they take yearly training in information security 

and privacy. NIH hosts a secure, encrypted environment that can be remotely 

accessed through a virtual private network. NIH staff who wish to access 

research grants data that are considered confidential and sensitive, such as 

unfunded grant proposals and reviewer rankings, require additional security 

background screenings. Personally identifiable information is highly restricted, 

often limited to a single person record at a time. If the need for expanded 

access to confidential and sensitive information arises, NIH staff can obtain a 

special data access agreement, which is limited to one-year duration. NIH 

staff can also sponsor a ‘guest researcher’ to obtain a data access agreement 

to access disaggregated awards data.3 Sponsorship is at the discretion of 

NIH Institutes, Centers, and Offices. In most cases, these researchers have 

NIH grant support. External researchers are required to undergo the same 

clearances required for NIH staff. Researchers must request NIH clearance 

for any manuscript or presentations showcasing their work using NIH 

administrative data. 

In contrast, many other federal agencies do not have specific policies or 

procedures in place to grant research access to administrative data. EPA and 

USDA participate in Census’ FSRDC program, providing pollution abatement 

and food security population survey data, respectively.4

The Commission could strengthen efforts by developing overarching 

directives that coordinate and encourage data sharing with researchers 

across Federal agencies. One avenue for coordination and harmonization of 

agency policies and procedures is the Interagency Council for Statistical 

Policy. In addition, in October 2016, OMB released a notice of solicitation for 

 3 NIH policy	for	the	guest	researcher	appointment	is	available	in	NIH	manual	chapter	

2300-308-1	at	https://oma1.od.nih.gov/manualchapters/person/2300-308-1

4 http://www.census.gov/about/adrm/fsrdc/federal_partners.html
The Promise of Evidence-Based Policymaking 687

8

comments on a new policy directive “Release and Dissemination of Statistical 

Products Produced by Federal Statistical Agencies.”5 Interagency dialogue 

and assessments could identify how statistical agency policy development 

could be adapted and adopted for Federal research award and other 

administrative data. 

Another avenue for coordination and information sharing has been the 

Interagency Working Group on Science Policy, coordinated by the White 

House Office of Science and Technology Policy (OSTP) and NSF’s SciSIP 

program. The primary function of this Group has been to identify 

opportunities to develop tools, theories, and methodologies that will advance 

their common interest, recommend joint research, data, and evaluation 

projects and enable Federal agencies to collaborate, coordinate, and 

leverage resources and efforts.

One question that the commission has asked about is alternative ways to 

share administrative data, providing access to researchers while 

guaranteeing confidentiality, privacy and data security. For the science 

agencies I would like to advocate for a decentralized system of agency 

specific data enclaves. There are three reasons. 

First, there are many nuances to agency operations: the way programs are 

operated, data are collected, and even how variables are defined. Pece 

(2016) provides an overview of the challenges of obtaining reliable R&D data 

from federal agencies. Standardization is required in order to be able to 

make interagency comparisons. The National Center for Science and 

Engineering Statistics (NCSES) and the Office of Management and Budget 

 5

https://www.whitehouse.gov/sites/default/files/omb/inforeg/directive4/frn_comm

ent_stat_policy_dir_4_addendum.pdf
688 The Promise of Evidence-Based Policymaking

9

(OMB) are working to establish a Federal R&D Community-of-Practice (R&D 

COP) to address agency questions on the interpretation of how to apply 

OECD Frascati manual. Adoption of standards would certainly reduce 

barriers to research using administrative data. 

Second, my experience is that to best inform policy requires an understanding 

of context and the specific operation of the agency. The best efforts involve 

collaboration between academics and practitioners. Rather than simply taking 

data and conducting analysis there are many nuances and definitions to 

understand and appreciate and to make informed interpretations of the 

results. 

Third, agencies have specific questions to address their mandate and to help 

them achieve their objectives. Policy relevant research projects are bestformulated and designed through interaction and collaboration. These 

interactions will benefit the scholarly community by providing more nuanced 

research questions and a deeper understanding the mechanisms that govern 

science funding and agency operations. 

Some Science policy questions are universal across agencies but there are 

many hypothesis-driven studies that can be conducted using single agency 

data, perhaps linked to publications and patents. I think that there is plenty 

that we might say based on administrative records at the single science

agencies. These would be questions about the effectiveness of different 

program interventions, selection processes and outcomes. Studies 

conducted at one agency can be replicated at other agencies. 

Another opportunity is an expanded use of fellowships, and other temporary 

hiring authorities and mechanisms to bring in academic and other researchers 
The Promise of Evidence-Based Policymaking 689

10

into agencies to access administrative data and conduct evaluations. 

Mechanisms to do this include the Schedule A excepted service authority and 

Intergovernmental Personnel Act.6 The contract terms outline the access to 

program administrative data and analyses that the researcher would conduct. 

There seem to be various mechanisms in place across Federal agencies. 

Sharing these experiences across the Federal Government could provide a 

set of lessons learned to improve adoption and use. 

I believe that a decentralized system allows for experimentation. Over time 

better practices will emerge and there will be more confidence in the 

relationships. This model would also create a dialogue between researchers 

and the policy community that is likely to increase trust on the part of the 

agencies and to provide researchers with more interesting and informed 

questions. 

One question that I ask myself is if these administrative data need to be 

linked to other sources and what would those other sources be. Ultimately 

there is a hope that individual researchers can be tracked over their careers, 

however building such longitudinal data takes time. U-Metrics at the 

University of Michigan is one effort to accomplish this task. With greater 

familiarity with agency data, new questions emerge. A viable second phase 

could link agency administrative data with other records, such as those at the 

Census Federal Statistical Research Data Center (FSRDC). Indeed, Sabrina 

 6 More	on	Schedule	A	available	at	https://www.opm.gov/policy-dataoversight/disability-employment/hiring/#url=Schedule-A-Hiring-Authority;	the	

Intergovernmental	Personnel	Act	Mobility	Program	provides	for	the	temporary	

assignment	of	personnel	between	the	Federal	Government	and state	and	local	

governments,	colleges	and	universities,	Indian	tribal	governments,	federally	funded	

research	and	development	centers,	and	other	eligible	organizations,	more	

information	available	at	https://www.opm.gov/policy-data-oversight/hiringinformation/intergovernment-personnel-act/
690 The Promise of Evidence-Based Policymaking

11

Howell has such a project underway to extend her EPA work by matching to 

applicant and other firms over time and working with the FSRDC. 

Great work has been done at the FSRDC but if administrative data will be 

housed there then the Commission should address some of the current 

barriers and limitations. The application and vetting process is very lengthy 

and time consuming – unsuited to academic clocks which are short and tick 

loudly. Currently, proposals must demonstrate value to the Census Bureau 

under Title 13 of the US Code. That requirement might be too stringent 

considering there are many worthy projects that may not have direct value to 

the Census or another agency but might have great social value.

Currently, the federal agencies are working together to provide the data on 

funded projects to researchers and the public in a standard format for funded 

projects with the Federal RePORTER platform.7 I invite you to take a look at 

this website and it's capabilities to query on topics across the participating 

agencies. There are negotiations to add other agencies. Indeed these efforts 

would be aided if the Commission could encourage or otherwise incentivize 

all the federal science agencies to provide their data.

Finally I would like to address something that I have noted that really 

concerns me and has potential to derail this undertaking. There is fear at 

many agencies that allowing researchers to work with their administrative 

records will reveal negative findings about the agencies. At a time when the 

federal science agencies feel under attack this is certainly understandable. 

But only by looking at data we can identify problems, bottlenecks and improve 

processes and efficiency. If we accept the premise that government programs 

are necessary to address market failures, facilitate network externalities, 

 7 https://federalreporter.nih.gov/
The Promise of Evidence-Based Policymaking 691

12

reduce barriers to collaboration and take on fundamental projects that are 

beyond the scope of most American businesses, then innovation and 

American competitiveness can be improved. It is only by studying the 

scientific research enterprise, the range of actors engaged in innovation and 

specific public programs and incentives that can scholars provide evidence to 

policy makers. 

Administrative data related to science and innovation policy cover a large 

number of administrative units beyond the federal agencies. At least one 

entity operates in each of the 50 states, with a mission of promoting 

innovation, entrepreneurship or science-based economic development. On 

top of this, consider all the activity in firms, research universities, institutes 

and labs and other non-profit and quasi-government organizations that have 

some stake in the outcomes related to science, scientific progress and 

commercial outcomes. Each has administrative data that could be studied to 

improve our understanding of the research and innovation enterprise. The 

federal effort in which the Commission is engaged can lead the way and set a 

model that states and other entities can follow. 

References: 

Ginther, D. K., Schaffer, W. T., Schnell, J., Masimore, B., Liu, F., Haak, L. L., 

& Kington, R. (2011). Race, ethnicity, and NIH research awards.

Science, 333, 1015–1019.

Howell, Sabrina T. (forthcoming). Financing Innovation: Evidence from R&D 

Grants. America Economic Review. 

Li, Danielle, and Leila Agha (2015). "Big Names or Big Ideas: Do PeerReview Panels Select the Best Science Proposals?" Science 348: 

434–438.
692 The Promise of Evidence-Based Policymaking

13

National Science Foundation, National Center for Science and Engineering 

Statistics (2015). U.S. R&D Increased in 2013, Well Ahead of the 

Pace of Gross Domestic Product. Info Brief NSF 15-330. 

Pece, Christopher V. (2016). Putting the Cart Before a Lame Horse: A Case 

Study for Future Initiatives to Automate the Use of Administrative 

Records for Reporting Government R&D. OECD Bly Sky Meetings. 
The Promise of Evidence-Based Policymaking 693

1

Testimony to the Commission on Evidence-Based Policymaking

Building Evidence for Communities of Color and Integrating Data for Public Good

Submitted by Cynthia Guy, Ph.D., Vice President of Research, Evaluation, Evidence and Data

The Annie E. Casey Foundation

February 9, 2017

I. Overview

Good morning/afternoon, Commissioners. Thank you for the opportunity to testify before you. I 

am delighted to be here because your mission is closely aligned with the Annie E. Casey 

Foundation’s commitment to build a brighter future for millions of children at risk of poor 

educational, economic, social and health outcomes. 

Today, I would like to share with you the Foundation’s insights into using data and evidence to 

develop the best possible policies for those children and families. To get better results, all levels 

of government must allocate scarce resources to strategies that work. But too often, 

policymakers fund well-intentioned approaches that lack evidence of effectiveness and end up 

failing the children they seek to help. 

Most social scientists agree that randomized controlled trials (RCTs) are the gold standard for 

classifying evidence-based programs. Yet researchers and policymakers also understand that 

RCTs can be conducted only in a very limited set of circumstances. And even when an RCT is 

feasible, the results may not be applicable to other people, places and contexts. Because of 

these limitations, the Foundation urges the use of best evidence available rather than solely 

relying on RCTs. 

The Foundation supports evidence-based work carried out by professionals with backgrounds in 

education, health and child and adolescent development. 

In my testimony, I would like to cover three areas. First, the data tell us clearly that children of 

color face the greatest obstacles to opportunity. Programs and strategies should therefore 

demonstrate success in communities of color, and we should develop a more diverse group of

researchers to evaluate those efforts. Second, government should adopt integrated data 

systems to identify broad community trends and patterns and develop solutions to respond 

accordingly. And finally, I will discuss why policymakers cannot afford to lose the critical 

information gained through the U.S. Census, American Community Survey and Supplemental 

Poverty Measure.

II. Using Evidence to Make Better Decisions for Communities of Color

Our nation is more of a quilt than a melting pot, so the traditions, social practices and cultural 

dynamics in communities of color have formed very differently over generations compared to

those in predominantly white communities. They can also vary greatly from one community of 

color to the next. A program that works well in urban Chicago, for example, will likely have little 

relevance in a rural Native American community. 

Yet the traditional approach to developing evidence-based programs and practices often does 

not incorporate the perspectives, experiences and input of communities of color. These 

programs are largely tested with primarily white trial groups and are created by an 

overwhelmingly white scientific community. By failing to appropriately consider children and 
694 The Promise of Evidence-Based Policymaking

2

families of color, developers and researchers miss the opportunity to gain critical knowledge 

about what does and does not work in diverse settings. 

A close look at the development of evidence-based programs for communities of color reveals 

three major themes: 1) an absence of adequate funding for rigorous impact evaluations of 

programs geared toward communities of color; 2) a lack of culturally specific and sensitive data

collection approaches in evaluations; and 3) a dearth of evaluation professionals with adequate 

knowledge and training on the cultural issues facing these communities. Indeed, many 

evaluators and researchers are not well versed in community engagement and participatory 

research techniques that build trust and partnership within communities already wary of being 

research subjects. Investments to address these three themes will help minimize the gaps in 

evidence-based programs and practices for communities of color.

Increasing and engaging more researchers of color will enable human service providers to learn 

more about what works best in communities of color from people who are more intimately 

familiar with the intricacies of those communities. The Casey Foundation’s Expanding the 

Bench initiative aims to increase the small but growing number of researchers and evaluators of 

color and to strengthen their influence in the field. Expanding the Bench provides networking 

opportunities that connect researchers of color with funders and supports professional 

development activities that place those researchers in leading evaluation centers and research 

firms. We encourage more efforts in this vein.

We have seen progress. Some evidence-based programs have emerged to support 

communities of color. Strong African American Families (SAAF), for example, helps rural 

African-American families strengthen their relationships, improve parenting and help young 

people develop. SAAF has included specific ways for parents to help young people cope with 

discrimination in response to research that exposure to racial discrimination is a strong predictor 

of preadolescent and adolescent depressive symptoms and substance abuse,. 

SAAF was developed by child psychologist Gene Brody with a foundation of rigorous research. 

The program follows the standards of the National Institute of Mental Health prevention 

research cycle, which requires longitudinal, epidemiological research on target populations 

before developing prevention programs designed for them. For more than a decade, Brody and 

his colleagues at the University of Georgia’s Center for Family Research have worked to identify 

protective factors that allow some rural African-American children in the state to thrive despite 

living in more challenging circumstances than their white peers.

Evaluations of SAAF showed several areas of statistically valid success. When compared with a 

control group, youth participants experienced fewer problem behaviors, such as theft, truancy or 

suspension. Those who didn’t drink alcohol were less likely to begin drinking, and those who did 

increased their use at a significantly slower rate. The program also showed improvements in 

parenting, with increases in positive communication and in youth protective factors, such as

negative attitudes about alcohol and sex. SAAF also delivered some unanticipated health 

benefits: Youth participants showed lower levels of stress hormones, inflammation and cellular 

aging.

Another program, Con Mi MADRE (Mothers And Daughters Raising Expectations), works with 

Latina mothers and daughters in Austin, Texas, to instill a shared vision of higher education and 

make that vision a reality. Con Mi MADRE provides tailored services to mothers and their 

daughters, from the child’s sixth-grade year until she graduates from college. Services include 

personal visioning techniques, development of intergenerational communication skills, college 

visits and financial literacy education.
The Promise of Evidence-Based Policymaking 695

3

Con Mi MADRE is designed to take advantage of close family ties in Latino culture to build 

confidence in two generations simultaneously. Once moms develop these skills, every child and 

grandchild in their homes benefit. In a program that serves 750 children per year, 100 percent of 

the participants graduate from high school, and 77 percent go on to college. Of those who 

attend college, 54 percent persist and/or graduate, compared with a 15 percent college 

graduation rate for Latinas throughout central Texas. 

Con Mi MADRE demonstrates the importance of taking culture into consideration. While 

cognitive behavioral therapy is probably one of the best-known clinical evidence-based 

practices, the organization found it to be less effective with its Latino families. 

To develop additional evidence about the effectiveness of programs serving communities of 

color, it is essential to listen to the communities themselves. We can learn from organizations 

applying evidence-based practices in a culturally relevant way and use their experiences to 

enhance scientific knowledge about what works for children and families of color. We can also 

learn from many successful local programs that are not deemed evidence-based but are 

building proof of their effectiveness by engaging in research, by placing joint emphasis on 

qualitative and quantitative outcomes and by developing community-centered approaches. Such 

programs combine academic and community knowledge to deliver effective, culturally relevant 

solutions. 

III. Promoting State and Local Integrated Data Systems

Through decades of work with child welfare, juvenile justice and other public systems, we have 

encountered numerous challenges and frustrations caused by the inability of government 

agencies to share information that can help children. We advocate the adoption of an integrated 

data system (IDS) to link individual-level administrative data from multiple government agencies. 

By linking records across agencies and over time, an IDS creates a rich picture of individual 

service needs, participation and outcomes over many years. In some cases, individual records 

are linked together to form comprehensive, long-term household and family histories. We 

believe it is important to develop these systems for state and local governments, both of which 

play key roles in the management and delivery of education and human services essential to 

child and family well-being. Facilitating and accelerating data-driven decisions across agencies 

will also enable them to respond more swiftly to address needs and issues as they arise.

The federal government has been an important partner in promoting IDS development at the 

state and local levels. The field has benefited from continued federal efforts to clarify and 

emphasize that privacy regulations such as the Health Insurance Portability and Accountability 

Act and Family Educational Rights and Privacy Act actually allow and encourage data 

integration among state and local public agencies. Also important has been the federal role in 

supporting the costs of state and local IDS development and encouraging comprehensive 

approaches to public systems planning and management that require cross-system data 

sharing. For example, the Office of Management and Budget has increasingly exercised 

leadership here, directing agencies to use a portion of their budget for low-cost evaluations 

using administrative data and issuing State System Interoperability and Integration Project 

grants to seven states in 2014.1 More broadly, the federal government has mandated and 

 1 See OMB-M-12-14. Grants were awarded to California, Colorado, Illinois, Indiana, Maryland, New York 

and Oklahoma, and interviewees confirmed their importance to IDS development in these states.
696 The Promise of Evidence-Based Policymaking

4

funded efforts to build longitudinal databases in every state with information on student 

achievement from early learning through workforce entry.

A significant number of state and local governments, as well as nonprofit and university partners 

under the governance of public agencies, already operate integrated data systems. By offering 

large sample sizes, longitudinal data and the ability to identify multiagency clients, an IDS is a 

valuable tool for policy analysis, program planning and monitoring and evaluation. Because of 

the sensitive nature of the data, organizations that house an IDS carefully follow privacy laws to 

protect the confidentiality of children and families, securely store data and maintain rigorous 

standards for use and access.

By using an IDS, states can understand whether their approaches are working or having 

unintended consequences for children. In Wisconsin, for example, child welfare workers and 

child support enforcement workers have routinely pursued child support orders to offset the cost 

of a child’s out-of-home placement. But when researchers analyzed the administrative records 

in the state’s IDS, they discovered this policy not only failed to recover costs but actually 

lengthened the time children spent in foster care. Acting on these findings, officials in the 

Department of Children and Families crafted a new policy. When it goes into effect, 

caseworkers will not collect child support from a parent during the first six months a child is in 

foster care and will apply new criteria in deciding to do so after that period.

An IDS can provide proof points to show a program is working and worth continuing. In 

Washington, for example, analysts used the state’s IDS to evaluate the effectiveness of an 

innovative policy that reversed the typical practice of reducing or terminating Temporary 

Assistance for Needy Families (TANF) benefits when a child is removed from home and placed 

in care. Designed to speed family reunification by easing economic hardship, the new policy 

allows a parent to receive full TANF benefits for up to 180 days after a child enters care. The 

evaluation’s positive findings on improving reunification rates without additional costs enabled 

state officials to keep the benefits in place. 

Integrated data systems also advance two-generation approaches, which simultaneously 

address the needs of children and their parents so both can succeed. For example, the South 

Carolina IDS links inmate and household records (including use of mental health services, 

involvement with child welfare and juvenile justice systems, reliance on income support and 

student academic performance), which enables researchers to study the impact of incarceration 

on families. The state can use this information to improve family services and ease reentry.2 In 

Oregon, the state’s Center for Evidence-Based Policy used a combination of birth and parent 

records (from vital statistics, child welfare and programs related to self-sufficiency and 

substance abuse) to develop a model that predicts the likelihood of a child being maltreated by 

age 2 and placed in foster care by age 6. The data are geocoded down to the census-block 

level to create hot-spot maps that illustrate the prevalence of child maltreatment and foster care 

placement rates by location and are used to target preventive services to families.3

IV. Using Federal Data to Inform Better Decisions

Two of the most valuable data resources for the Casey Foundation and all practitioners of datadriven, evidence-based decision making are the decennial census and the American 

 2 DeHart, D, & Shapiro, C. (2016). Integrated administrative data and criminal justice research. American 

Journal of Criminal Justice, 1–20. doi:10.1007/s12103-016-9355-5 3 Actionable Intelligence for Social Policy interview with C. Kelleher in 2016, retrieved from 

www.aisp.upenn.edu/qa-with-christopher-kelleher
The Promise of Evidence-Based Policymaking 697

5

Community Survey. Both are currently at risk. Another key resource for providing accurate 

assessments of government programs designed to help families, especially those with children, 

is the Supplemental Poverty Measure. 

No data resource is more fundamental to the operation of our federal system than the decennial 

census, which plays a key role in allocating political representation and federal funds. Despite 

the importance of ensuring a complete and accurate count in 2020, Congress has required the 

U.S. Census Bureau to hold the cost of the next census at the same level as the 2010 census, 

approximately $13 billion. This significant budget constraint will make it all but impossible to 

meet, let alone exceed, the precedent for accuracy set in 2010. Even within the constrained 

spending levels, the census has been bound up in the ongoing continuing resolutions for 

funding government. In most cases, this means level funding with the previous year. For the 

census, which has a planned schedule of increased appropriations as we near 2020 

implementation, level funding translates into a significant cut. We hope the Commission would 

consider two recommendations: 1) an increase in spending for the 2020 census, and 2) either 

an approach outside the traditional appropriations process for funding the census or a 

commitment to avoid level funding through continuing resolutions. 

While the census provides basic, once-a-decade numbers essential to government operations, 

the U.S. Census Bureau’s American Community Survey (ACS) provides a rich body of 

continually updated social and economic indicators that the public, private and nonprofit sectors 

rely on to target and improve policies, programs and investment. For example, staff at the 

Casey Foundation analyze ACS data to provide accurate information on child well-being 

disaggregated by geography, income, race and many other factors. Our KIDS COUNT Data 

Center has 4 million data points, many of which are derived from analyses of the ACS. These 

data are important to public and private organizations that are developing evidence-based 

programs. 

For example, Congress uses the ACS data to allocate grants for homeland security, highway 

planning and construction, Medicaid, substance abuse treatment, community development, rural 

electrification, public transit and dozens of other programs. Community leaders use ACS data to 

analyze the emerging needs of their neighborhoods; plan for the future; and locate new schools, 

recreational areas, hospitals and police and fire departments. Businesses use the data to make 

key marketing, site selection and workforce decisions. In recent years, there have been 

repeated efforts in Congress to undercut the ACS by making participation voluntary (a change 

that will reduce response rates and raise costs of data collection, according to Census Bureau 

tests and international experience) and by reducing or eliminating its funding. To maintain this 

fundamental data resource for evidence-based policymaking, we must defend the ACS against 

attempts to undermine the accuracy and reliability of the survey. We hope the Commission 

would oppose making the ACS voluntary by acknowledging that the ACS is the long form of the 

census and part of the constitutional obligation for enumeration, and we urge you to address the 

importance of funding the ACS.

In 2011, the U.S. Census Bureau created the Supplemental Poverty Measure (SPM) to provide 

a more accurate measurement of poverty than the official poverty measure reported. The SPM 

measures the impact of a number of programs such as the Supplemental Nutrition Assistance 

Program and Earned Income Tax Credit, and it accounts for rising costs and other changes that 

affect a family’s budget. The measure has advanced the nation’s understanding of child poverty 

and the effects of safety net programs and tax policies on families. By using the SPM, 

researchers have determined that the rate of children in poverty has declined since 1990, while 

the official measure shows almost no change. Data-based benchmarks and measures like the 
698 The Promise of Evidence-Based Policymaking

6

SPM are essential tools for evidence-based policymaking, and federal policymakers should use 

and sustain them. 

V. Lessons and Recommendations

Discovering what works to serve children and families, particularly in communities of color, and 

proving the efficacy of culturally relevant approaches are still very much works in progress. 

Questions of balance between scientific knowledge and community experiences are ongoing. 

However, those involved in the program examples described earlier share the following 

observations: 

 Evidence is necessary for broader adoption and scaling. Whether or not a program is 

recognized as “evidence-based,” the presence of compelling proof provides greater 

assurance that an investment in a program will indeed deliver the desired outcomes. 

However, creating that assurance requires significant investments of time and money on the 

part of program developers.

 Partnerships are key in developing evidence-based programs. Research institutions, 

universities and government research agencies such as the Agency for Healthcare 

Research and Quality all have experience with creating evidence-based protocols and 

securing funding for promising programs that seek to prove their efficacy. Partnering with 

those institutions can give small organizations a leg up in achieving evidence-based status. 

 Culture is key. Programs that deliver great results for one group may not do so for another. 

Understanding culture and incorporating it into the creation of an evidence-based program 

will help ensure success in communities of color. In addition, understanding community 

culture will help create local support and buy-in during the research and pilot phases of an 

evidence-based program.

 Evidence comes in multiple forms. When examining cultural relevance, qualitative 

evidence is just as important as quantitative in terms of determining a program’s efficacy 

and effectiveness. 

 Evidence-based research must respect community values. Data collection processes 

must be sensitive to communities to avoid making people uncomfortable, especially in 

cultures that consider family matters private or have a deep distrust for outside 

organizations. By respecting community values, data may be collected with more fidelity and 

more comprehensively and can be used to inform stronger research design and more 

culturally competent methodology. 

 The federal government must continue to play an important role in promoting IDS 

development at the state and local levels. It can do so by defining federal privacy 

regulations in ways that encourage cross-agency data integration, by helping support the 

costs of state and local IDS development and by encouraging comprehensive approaches to 

public systems planning and management that require cross-system data sharing.

 A complete count of the 2020 census, a mandatory American Community Survey and valid 

benchmarks of economic well-being such as the Supplemental Poverty Measure are 

essential to support data-driven decision making at every level of government. Federal 

policymakers should protect and sustain these data resources and tools.
The Promise of Evidence-Based Policymaking 699

7

VI. Resources to Help Employ Evidence-Based Programs and Practices and Promote 

Data-Driven Decision Making 

“Implementation Research: A Synthesis of the Literature,” by Dean L. Fixen, Sandra F. Naoom, 

Karen A. Blase, Robert M. Friedman, and Frances Wallace (University of South Florida, 2005)

http://ctndisseminationlibrary.org/PDF/nirnmonograph.pdfhttp://ctndisseminationlibrary.org/PDF/

nirnmonograph.pdf

“Evidence-Based Programs and Practices: What Does It All Mean?” by Lisa Williams-Taylor 

(Research Review, Children’s Services Council of Palm Beach County, 2007)

California Reducing Disparities Project website. 

National Evidence-Based Practice Databases

While these databases may not hold a significant number of evidence-based programs focused 

on communities of color, they may be helpful in providing useful general information such as 

definitions of different types of evidence-based practices.

The Social Work Policy Institute (www.socialworkpolicy.org/research/evidence-based-practice2.html) lists the following databases and registries, among others:

Blueprints for Healthy Youth Development (Center for the Study and Prevention of Violence)

California Evidence-Based Clearinghouse for Child Welfare

National Registry of Evidence-Based Programs and Practices (Substance Abuse and Mental 

Health Services Administration) 

Social Programs That Work

Additional Resources

“Counting for Dollars: The Role of the Decennial Census in the Distribution of Federal Funds,” 

www.brookings.edu/research/counting-for-dollars-the-role-of-the-decennial-census-in-thedistribution-of-federal-funds
700 The Promise of Evidence-Based Policymaking

8

“Surveying for Dollars: The Role of the American Community Survey in the Geographic 

Distribution of Federal Funds,” www.brookings.edu/wpcontent/uploads/2016/06/0726_acs_reamer.pdf

Actionable Intelligence for Social Policy, www.aisp.upenn.edu
The Promise of Evidence-Based Policymaking 701

TESTIMONY

James J. Hill, Jr., EdD

CEO, Proofpoint.net

Los Altos, CA

Before the Commission on Evidence-Based Policy Making—Public Hearing

February 9, 2017
702 The Promise of Evidence-Based Policymaking

Abstract

Policy and program success requires four evidence-related factors: (1) effective goal setting, (2) 

accurate, yet speedy, issue analysis, (3) an impartial solution selection process, and (4) follow-through 

and reinforcement. However, having a robust and interconnected infrastructure and “Big Data” isn’t the 

same as having “Big Evidence”.

With just two weeks of observation of the new Administration, one theme seems to be emerging: A 

desire – maybe an insistence – for rapid actions that make a big impact.

So, while there are many long term considerations that the Commission can address, delivering two or 

three quick, ground level, wins will serve as a notable indicator of the bipartisan promise of the 

evidence-based law on which this Commission was founded.

Dr. Hill Biography

Dr. Jim Hill is the founder and CEO of Proofpoint Systems, a provider 

of cloud applications that advance evidence-based organizational 

performance. Since 2003, he has led Proofpoint in digitizing the 

processes associated with gap and root cause analysis; planning, 

execution, and evaluation; and fair talent management to help clients 

and consultants produce better and more cost effective results.

Prior to founding Proofpoint, Jim was a career officer in the United 

States Marine Corps and an executive with Sun Microsystems. He is 

also a past president of the International Society for Performance 

Improvement.

Jim’s extensive writings consist of numerous articles and book 

chapters including the Handbook of Human Performance Technology 

(3d Ed., 2006) and Performance Interventions (1999). His book, Giving 

Away Power, was published in 2013.

Jim has been featured in Training, Sales and Marketing Management, Japanese Management Journal, 

and BPTrends for his visionary approach to organizational performance. Along with his clients and the 

organizations he has led, he is a 3-time finalist for the American Business Award (Best New Software 

Product), the recipient of the CyberFEDS Awesome New Technology Award, and a 6-time recipient of the 

ISPI Award of Excellence. He is the 2016 recipient of ISPI’s Distinguished Service Award.

Jim is a graduate of The Ohio State University and received his doctorate (human and organizational 

improvement) from the University of Southern California.

He and his wife, Aiko, have four children and reside in Los Altos, California where they are active in their 

community.
The Promise of Evidence-Based Policymaking 703

Written Statement for the

Commission on Evidence-based Policy-making

Introduction and Overview

Chair Abraham, Chair Haskins, and Commissioners joining us virtually, I’m Jim Hill and I’m delighted to 

be here with you this morning.

Having reviewed early drafts of the legislation, I appreciate the opportunity to contribute.

As the CEO of Proofpoint.net, I represent a cloud technology that helps organizations solve their biggest 

issues faster and more cost effectively than ever before. That takes evidence.

What Leaders Want

Our experience is that leaders just want clear answers to five key questions:

1. Do our program priorities align with achievable objectives? 

2. How do we responsibly audit, govern, and allocate resources?

3. Who are we contracted with and are they performing?

4. How well are we integrated?

5. How do we assess ongoing programs and new requirements?

I founded Proofpoint.net to help answer these questions. Timely, accurate, and unbiased information 

leads to better and faster decisions, improved performance, and accelerated organizational evolution.

Government Needs Practical Evidence Now

The question facing the Commission is simple, “How do we make government smarter with all the data 

floating around?”

However, multiple factors call for a major change in the way we think about work-related evidence. 

Beyond policy and technology considerations, there is a social aspect that must be considered. As 

examples:

 It’s estimated that world data will grow 50x over the next 10 years1

 There is virtually no data related to day-to-day transactional work2

 Organizational improvement programs have about a 70% failure rate3 and probably higher4

 With the growth of virtual work and expansion of BYOD, people want easier access to their work 

tools and data

 And, Big Data is not Big Evidence.

This issue is so wide-ranging, that it may be prudent to tackle it in short term, low cost chunks that align 

to desired decision cycles and what will likely be a high demand, resource constrained fiscal 

environment.

I’d like to highlight some examples and offer recommendations for quick wins that advance your goals.

 1 J.P. Morgan Chase & Co. (2016). Unpublished venture capital briefing. San Francisco. January.

2 Ibid.

3 Sources include Hammer & Champy, 1993; Beer & Nohria, 2000; Kotter, 2008. See additional reference 

information at the end of this paper.

4 Druckman, Singer & Van Cott, 1997; Smith, 2002; Hill, 2004.
704 The Promise of Evidence-Based Policymaking

Written Statement for the

Commission on Evidence-based Policy-making

Reference Case #1: Practical Program Monitoring

In 2013 a federal executive approached us with a problem: His staff was manually producing 80 daily 

and weekly reports.5

Little of the information they needed was in enterprise systems. Instead, it was in static documents and 

the heads of people.

So, rather than focus on planning and management, they were exchanging scores of emails and phone 

calls each day.

You can appreciate the inefficiency. You’ll also sense that the millions of data points they were using and 

creating would never be available to others.

Nearly every federal program works this way. And, popular management applications don’t help. They 

are designed for individual projects or, maybe, small groups. As a result, there is no effective way to 

bring thousands of projects or programs together.

This leads to my first recommendation to pilot a common enterprise project management tool in a 

willing department, agency, or program.

Doing so will demonstrate the power of capturing day-to-day transactional data that is now lost.

This recommendation is relevant to the Commission’s charter in that it provides a use case for creating 

practices for monitoring programs and assessing their outcomes.

Reference Case #2: Simplifying Data Collection & Access

My second example begins with a recent report that more than 240 hospitals have been penalized three 

years in a row for infection-related issues. With an average penalty of $500K, 6 that is more than onethird of a billion dollars wasted.

In a related event, I recently visited a notable acute care facility where I was shown two walls covered in 

8 1⁄2 x 11 sheets of paper.7 Each sheet contained a graph that staff members were assigned to manually 

update each day – using a marker.

At the end of each month, the graphs were filed away.

While hospitals spend many millions on electronic health records, patient informatics, and bill collection

technologies, healthcare administration is nearly all done via paper, spreadsheets, and white boards –

and it’s a massive vulnerability to the Nation’s health.

So, my second recommendation to pilot a common quality management tool with a willing healthcare 

system.

This will lead to better patient outcomes and reduced readmissions. It will also cut administrative time, 

increase staff morale, and reduce turnover.

This recommendation aligns to your charter by eliminating barriers to accessing data that organizations 

already collect. In this case, the collection effort is so manual that the data will never get used by the 

right people or at the right scale.

 5 Proofpoint.net use case. Client not disclosed.

6 Punke, 2016.

7 Proofpoint.net use case. Client not disclosed.
The Promise of Evidence-Based Policymaking 705

Written Statement for the

Commission on Evidence-based Policy-making

A Final Thought & Recommendation

My final comment is that data access and exchange are not technically hard. They are simply electronic 

handshakes.

But the barriers are enormous. Beyond longstanding policies, on the vendor side the core philosophy of 

your major providers is to intentionally not integrate, or to do so at great expense.

However, the “social” side of evidence exchange presents an opportunity. Make it personal.

So, I recommend the establishment of a federal “Goal Repository,” starting with a pilot program that

rewards participation.

The Goal Repository would use interpersonal sharing to drive interagency sharing. It would cultivate 

thinking along the lines of, “This is what I’m trying to do (goals). I wonder if anyone else is working on a 

similar issue.” Leaders would connect and share information of mutual value based on mutual interests.

Conclusion

Finally, in your report to the President and Congress, I urge the Commission to recommend an approach 

characterized by “low cost immediacy.”

Addressing the issue via multiple, successful, pilots will produce frameworks that others can easily 

adopt. That will result in speed, cost efficiency, and improved productivity at every level of government.

References

Beer, M., & Nohria, N. (2000). Cracking the code of change. Harvard Business Review, 78(3), 133–141.

Druckman, D., Singer, J. E., & Van Cott, H. (1997). Enhancing organizational performance. Washington, DC: National Academy 

Press.

Hammer, M., & Champy, J.A. (1993). Reengineering the corporation: A manifesto for business revolution. New York, NY: Harper.

Hill, J. (2000). Are executives overconfident? Unpublished doctoral dissertation. University of Southern California. Los Angeles.

Hill, J. (2004). Stewardship. Performance Improvement, 43(2), 18–23. doi: 10.1002/pfi. 4140430206

Hill, J. (2013). Using the cloud to accelerate transformation and influence change. Performance Improvement, 52(5), 19–27. doi: 

10.1002/pfi. 21347

J.P. Morgan Chase & Co. (2016). Unpublished venture capital briefing. San Francisco. January.

Kotter, J.P. (2008). A sense of urgency. Boston, MA: Harvard Business School Press.

Punke, H. (2016). The 241 hospitals punished 3 years in a row for high infection rates. Becker’s Hospital Review. December 27, 

2016.

Smith, M.E. (2002). Success rates for different types of organizational change. Performance Improvement, 41(1), 26–33. 

doi:10.1002/pfi.4140410107
706 The Promise of Evidence-Based Policymaking

1

Statement submitted to the Commission on Evidence-based Policymaking

Public Meeting, February 9, 2017

San Francisco, CA

Submitted by: David S. Johnson, Research Professor and Director of the Panel Study of Income 

Dynamics, University of Michigan1

Co-signers: Rebecca Blank (University of Wisconsin – Madison), Sheldon Danziger (Russell 

Sage Foundation), Kathryn Edin (Johns Hopkins University), Irv Garfinkel (Columbia 

University), David Grusky (Stanford University), Michael Laracy (Annie E. Casey Foundation),

Luke Shaefer (University of Michigan), Timothy Smeeding (University of Wisconsin –

Madison), Laura Speer (Annie E. Casey Foundation), Jane Waldfogel (Columbia University),

Renee Wilson-Simmons (National Center for Children in Poverty, Director), Christopher Wimer

(Columbia University), James P. Ziliak (University of Kentucky)

The timing for a bipartisan commission on developing evidence-based policymaking is almost 

prescient. Many of us have been involved with the process of creating this commission, and we 

thank Speaker Paul Ryan and Senator Patty Murray, Chair Katharine Abraham, Co-Chair Ron 

Haskins and all of the Commissioners on their work and dedication to this important goal –

“...developing a bipartisan strategy to ensure that evidence increasingly informs the important 

decisions that affect the lives of Americans.”

Our purpose in this statement is to focus on three points:

• Support and independence must be provided to the federal statistical agencies to

“improve statistical protocols” for producing timely, accurate and relevant statistics,

specifically, the Supplemental Poverty Measure (SPM).

• Support must be provided to statistical agencies for the continued production of the SPM,

as it is the only official statistic that measures the effectiveness of government tax, 

transfer, and social insurance programs.

• Support must be provided to construct a data infrastructure that facilitates linkages 

between survey and administrative data, which can be used to improve the SPM.

Our request is slightly different than many of the other statements to the Commission. We focus 

on the Commission goals related to improving statistical data series and statistical protocols as 

described in the Commission’s charge.2 These common protocols and commonly accepted 

statistical indicators are critical. As Katherine Wallman, the recent Chief Statistician of the U.S.,

stated in last week’s Guardian article (Chalabi (2017)), “We should all be starting from the same 

 1 Contact info: Survey Research Center, 426 Thompson St. Room 3234, Ann Arbor, MI 48106;

johnsods@umich.edu

2

“(1) determine the optimal arrangement for which administrative data on Federal programs and tax expenditures, 

survey data, and related statistical data series may be integrated and made available to facilitate program 

evaluation, continuous improvement, policy-relevant research, and cost-benefit analyses by qualified researchers 

and institutions while weighing how integration might lead to the intentional or unintentional access, breach, or 

release of personally-identifiable information or records; (2) make recommendations on how data infrastructure, 

database security, and statistical protocols should be modified to best fulfill the objectives identified in paragraph 

(1).”
The Promise of Evidence-Based Policymaking 707

2

numbers. I think that’s a fear that many of us have at this point - it’s that picking and choosing 

your numbers to suit your politics is not the way that we ought to be doing it.”

Importance of Federal Statistics

The white paper presented to the Commission as background, “Using Administrative and Survey 

Data to Build Evidence,”3 stressed the importance of the foundational evidence represented in 

aggregate indicators as one key component of evidence-based information; the SPM and the 

Principal Federal Economic Indicators (PFEI) are key foundational evidence. The other white 

paper, “Overview of Federal Evidence-Building Efforts,” describes the responsibilities of 

statistical agencies, which include producing reliable data, accurate and objective statistics, and 

protecting. Without a statistical system that can work in this framework, the task of building a 

data infrastructure to improve statistics will be difficult, if not, impossible.

The country’s need for reliable and trustworthy statistics on the economic well-being of the 

nation is critical. This can only be accomplished with official statistics, government survey 

collection,4 and data integration with government administrative data. In fact, the recent 

National Academies of Science’s Committee on National Statistics report (Groves and HarrisKojetin, 2017) states it best – “Federal statistics provide critical information to the country and 

serve a key role in a democracy.”

The need for reliable and trustworthy statistics are summarized in a variety of Statistical Policy 

Directives issued by OMB, and in a sense, provide guidelines for the statistical protocols used in 

the statistical system. Statistical Policy Directive 1 ensures the quality and independence of the 

statistical agencies. Statistical Policy Directives 3 and 4 provide guidance in the dissemination 

of the principle federal economic indicators, and other important foundational evidence. 

The Importance of Poverty and the SPM

The poverty measure has an elevated distinction as recognized by OMB Statistical Policy 

Directive 14. Even before these important directives (1, 3 and 4) were issued by OMB, 

Statistical Policy Directive 14 (released in 1978) instructed the Census Bureau to produce an 

official poverty measure, and stated: “Other measures of poverty may be developed for 

particular research purposes.” It is essential that a poverty measure reveals the effect of major 

income support programs designed to aid the poor. Many such programs, however, are excluded 

from the official poverty measure. Such programs include housing supports, major food and 

nutrition programs, and refundable tax credits. Only the SPM can provide evidence on these 

major social programs.

Because the official poverty measure does not account for any of the government tax and in-kind 

transfer programs, it does not provide an accurate view of the financial status of low-income 

 3 See links for white papers, https://obamawhitehouse.archives.gov/sites/default/files/omb/mgmtgpra/using_administrative_and_survey_data_to_build_evidence_0.pdf and 

https://obamawhitehouse.archives.gov/sites/default/files/omb/mgmtgpra/overview_of_federal_evidence_building_efforts.pdf

4 See Joe Hotz’s statement to the CEP on Jan 5, 2017, where he stresses the importance of government survey data.
708 The Promise of Evidence-Based Policymaking

3

people. It only includes direct cash social insurance (e.g. Social Security and Unemployment 

Insurance) and welfare (e.g. Temporary Assistance for Needy Families and Supplemental 

Security Income) programs. As such, any changes in benefits from the Supplemental Nutrition 

Assistance Programs (food stamps) or tax credits like the Earned Income Tax Credit and 

Additional Child Tax Credit will have no direct impact on the poverty rate, even though outlays 

on these programs far surpass those of the means-tested cash transfer programs. In addition, the 

official poverty threshold, the dollar amount such that families with income below it are poor, 

was based on food spending in the 1960s, updated with inflation until today. But housing today, 

not food, is the single largest element of households’ budgets. The official thresholds are also 

the same for San Francisco and Merced, and New York City and rural Mississippi. Finally, the 

official poverty measure does not account for the income sharing between a single parent and a

live-in partner – they could both be poor under the official measure even though they live 

together and share resources.

The SPM has been developed over the past 25 years with bipartisan participation by researchers

and policymakers, and remedies the flaws in the official poverty measure. The main impetus for 

the new measure arose out of the National Academy of Sciences report, Measuring Poverty: A 

New Approach, released in 1995. In this report, a bipartisan and distinguished group of 

academics and researchers (Rebecca Blank and Sheldon Danziger, both co-signers today, were 

members of the panel) agreed on recommendations for a new method to measure poverty in 

America. This new measure has been examined by many bipartisan groups, including meetings 

at The Brookings Institution, American Enterprise Institution, and Wisconsin’s Institute for 

Research on Poverty. All along the process, these discussions were guided by the direct 

involvement and continued research at the federal agencies, especially BLS, Census and OMB.

Three events solidified the support for an SPM measure – Rebecca Blank’s Presidential Address 

at APPAM that suggested that the new measure be a supplement for, and not a replacement of,

the official poverty measure; Mayor Bloomberg’s call for the Center for Economic Opportunity 

to produce an alternative poverty measure to provide more accurate evidence for NYC; and the 

proposed Congressional legislation, the Measuring American Poverty Act. These activities 

centered the discussion on using a common supplemental poverty measure in conjunction with 

the official measure. 

Finally, in 2010, the President proposed funding for the SPM in the 2011 Budget, and OMB 

issued guidance via the Interagency Technical Working Group. The detailed Department of 

Commerce congressional budget stressed the importance of the new measure for evidence-based 

policymaking stating: “A more accurate measure of poverty will provide policymakers with 

more accurate information for modeling changes to their programs and improving the 

evaluation of the effectiveness of programs for the disadvantaged. Poverty is a critical indicator 

of how widely prosperity is shared in our economy and is a benchmark for targeting resources 

toward the disadvantaged.”5

The SPM is much more accurate than the current official poverty measure, which has been 

basically unchanged since its inception in 1968. Even Adam Smith realized the importance of 

defining poverty in terms of the conditions of the day “...a creditable day-laborer would be 

 5 See http://www.osec.doc.gov/bmi/budget/11CJ/Census%2011%20portrait%20final.pdf
The Promise of Evidence-Based Policymaking 709

4

ashamed to appear in public without a linen shirt, the want of which would be supposed to 

denote that disgraceful degree of poverty...” Yet the official poverty thresholds are based on 

food costs from the 1960s.

Unlike the official measure, the SPM thresholds are updated to reflect a current basket of 

necessities (based on food, clothing, shelter and utilities), different thresholds are created for 

different living arrangements, for renters vs owners, and for differences in the cost of living 

across geographic areas.6 Poverty is determined by combining all of the income from spouses 

and cohabiting partners and includes all children, as opposed to the official measure. As 

mentioned earlier, income includes not just cash income but also the value of government inkind transfers as well as tax credits (and tax payments). Finally, the market basket continues to 

relevant for the current period.

In addition to a much more inclusive definition of resources, the SPM also takes into account the 

increasing costs of health care and child care and work-related expenses. As such, it is definitely 

a “more accurate measure of poverty” as stated in the Commerce Department budget (mentioned 

above). Because the SPM is annually published by the federal statistical agencies,7 it represents 

a critical component of the foundational evidence required to determining the effectiveness of 

policy.

The SPM also changes our perception of the poor – who they are. Due to the inclusion of inkind transfers and refundable tax credits, children and single parent families are a smaller 

fraction of the poor. With child care and work-related expenses, workers are a larger share of the 

poor under the SPM, as well as families spending on private health insurance. The SPM 

demonstrates that people living in poverty are not just those without any income. Just as Adam 

Smith recognized, it takes more to live in today’s society. 

As Speaker Ryan (when he was chair of the House Budget Committee) stated in his budget 

document, “Economic Opportunity in America,” July 2014,

8 (where he introduced the concept of 

the Commission), “The first task is to determine how we should define progress,” and he 

continued by suggesting that the official poverty measure was flawed, and that the SPM would 

better measure the progress of poverty, and even stating, “Using more comprehensive poverty 

measures strengthens this case. Using the SPM, Wimer et al show that child poverty plummeted 

beginning in 1994.”9

Since its inception in 2010,10 there has been extensive research using the SPM to evaluate 

government programs. The 2014 Economic Report of the President highlighted the SPM in their 

chapter on the War on Poverty to demonstrate that with government programs we have not lost 

the War on Poverty as the flat trend of the official poverty measure suggests. In addition, the 

bipartisan report, “Opportunity, Responsibility and Security: A Consensus Plan for Reducing 

Poverty and Restoring the American Dream,” (on which Ron Haskins was a coauthor) 

 6 Garner (2010) discusses the details of the SPM thresholds.

7 See Renwick and Fox (2016) for the most recent publication. 8 See http://budget.house.gov/uploadedfiles/expanding_opportunity_in_america.pdf 9 See also Winship (2014) for support for using the SPM. 10 Short (2011) is the first official report that produces an SPM.
710 The Promise of Evidence-Based Policymaking

5

highlighted the SPM as the preferred poverty measure, by a wide spectrum of policy researchers,

in evaluating the trends in poverty and the impact of government tax and transfer programs. The 

attached reference list shows many papers that examine the impact of the SPM on the poverty 

rates of a variety of demographic groups, including the official publications at the Census Bureau 

and BLS.

Recent research shows that only in using the SPM can we measure the impact of public policy. 

For example, Bitler, Hoynes and Kuka (2016) demonstrate the impact on child poverty reduction 

from the social safety net using an SPM-like measure, and Wight, Kashal and Waldfogel (2014)

show that the SPM finds impacts on very low food security for children. 

Finally, some of the most important work that uses the SPM, in conjunction with the American 

Community Survey, to measure the effectiveness of government programs has been conducted at 

the state and local levels. The Center for Economic Opportunity in NYC has led the way on 

developing an SPM for NYC. In fact, Mayor DeBlasio initiated the OneNYC project that 

includes poverty reduction goals using their SPM.11 For example, NYC has also used the SPM 

to evaluate the effects on poverty for workers negotiating wage contracts and to inform the 

Mayor’s housing policy on annual rent changes in regulated units.

Wisconsin and California also have state-specific SPMs, which are used to determine 

effectiveness of state-specific polices like state-level EITC and child care. For example, in 

California the state level SPM has been used to evaluate the expansion of the California EITC, 

showing that this expansion will “...increase the resources of a substantial number of 

Californians living in poverty and deep poverty.”12 Oregon and Minnesota have worked to 

develop measures, the Urban Institute has initiated a number of projects with states (Minnesota, 

Connecticut, Georgia, Illinois, and Massachusetts), and the Center on Poverty and Social Policy 

at Columbia University provides the SPM for all states.13

Importance of Administrative and Survey Data Linkages

The SPM is well-positioned to be an example of an economic indicator that integrates survey and 

administrative data, while highlighting the importance of the new data infrastructure. All of the 

state-level SPMs already adjust their estimates using the state-level administrative data on 

program participation to ensure that the aggregate benefits in the survey match the state totals 

(e.g., SNAP and housing). The Urban Institute’s TRIM model has been using integrated data to 

model improved estimates of SNAP and TANF in the SPM (see CBPP).

It is widely known that the government surveys do not completely account for all government 

transfer programs; many respondents mis-report or under-report their benefits. Recent research 

by Meyer, Mok and Sullivan (2015) documents this for a variety of surveys, especially the 

amounts of income from government transfer programs.

 11 See also Matthew Klein’s statement to the CEP on Nov 4, 2016 12 See http://inequality.stanford.edu/sites/default/files/EITC-california.pdf 13 See https://www.povertycenter.columbia.edu/data-release/2016/8/23/historical-spm-data
The Promise of Evidence-Based Policymaking 711

6

Meyer and Mittag (2015) demonstrate that the under-reporting for SNAP creates a downward 

bias in the estimate of the poverty reduction impact of SNAP. Integrating the administrative data 

on SNAP with the survey data on other income shows that SNAP is much more effective at 

reducing poverty than shown in uncorrected survey data. Hence, this data infrastructure with all 

State-level SNAP benefits (and other programs such as WIC and TANF) will provide much 

better foundational evidence on the effectiveness of these programs.

Alternatively, Hokayem, Bollinger and Ziliak (2015) show that earnings are over-reported for 

low income families in the surveys as compared to their earnings records at IRS and SSA. By 

integrating survey and administrative data, they demonstrate that poverty is higher than shown in 

the official measure, suggesting such a correction would similarly show higher poverty in the 

SPM. Hence data infrastructure that included administrative earnings and income data would 

improve the estimates provided by the SPM.

Because of this under-reporting and measurement error in income surveys, some researchers 

suggest that using a consumption-based measure of poverty may be more accurate (see Meyer 

and Sullivan, 2012). While multiple poverty measures would help to assess the poverty trends 

and composition of the poor, the SPM is currently the only official measure that can account for 

the impact of government programs on a timely basis. Hence, we believe that the focus should 

be on improving the measure of income in the SPM. This is also supported in the recent 

legislation proposed by Senator Mike Lee, the Poverty Measurement Improvement Act, which 

recommends the data linkages necessary to obtain an improved measure of resources. As a 

result, building on the current data infrastructure that exists in the federal statistical agencies is 

the best method to further statistical protocols and highlight the SPM as critical foundational 

evidence.

Recommendations

We recommend that the Commission’s report include a statement that OMB should continue 

support and funding to Census and BLS to produce the SPM alongside the official poverty 

measure.

We recommend that the Commission’s report encourage statistical agencies to use the integration 

of survey and administrative data to improve the measure of resources and thresholds in the 

SPM, and in national statistics more generally.

We recommend that the Commission’s report include a statement supporting statistical analysis 

and research at statistical agencies.
712 The Promise of Evidence-Based Policymaking

7

References

Annie E. Casey Foundation, “Measuring Access to Opportunity,” Annie E. Casey Foundation 

kids count data snapshop, February 2016

Ben-Shalom Y, Moffitt R, Scholz JK. An assessment of the effectiveness of anti-poverty 

programs in the United States. 2011

Betson, D., Giannarelli, L. and Zedlewski. S, 2011. A summary of the Workshop on State 

Poverty Measurement Using the American Community Survey, April 1, 2011

Bitler, M., Hoynes, H. and Kuka, E., “Child Poverty, the Great Recession, and the Social Safety 

Net in the United States,” NBER Working Paper No. 22682, September 2016

Blank, R. (2008), “Presidential Address: How to Improve Poverty Measurement in the United 

States, Journal of Policy Analysis and Management Vol 27:2, 233-254.

Bohn, S., Danielson, C., Levin, M., Mattingly, M. and Wimer, C., “The California Poverty 

Measure: A New Look at the Social Safety Net,” PPIC publication, October 2013.

Brookings/AEI, “Opportunity, Responsibility and Security: A Consensus Plan for Reducing 

Poverty and Restoring the American Dream”

Center for Economic Opportunity, "The CEO Poverty Measure, 2005 – 2014," Mayor’s office 

publication, 2016

Chalabi, Mona, “Statisticians fear Trump White House will manipulate figures to fit narrative,”

The Guardian, Jan 30, 2017.

Citro, C. F., and R. T. Michael (eds.). 1995. Measuring Poverty: A New Approach. Washington, 

D.C.: National Academy Press.

Council of Economic Advisors, Economic Report of the President, GPO, 2014 

Fox, Liana, Christopher Wimer, Irv Garfinkel, Neeraj Kaushal, and Jane Waldfogel. (2015). 

“Waging War on Poverty: Historical Trends in Poverty Using the Supplemental Poverty 

Measure.” Journal of Policy Analysis and Management 34(3): 567-592.

Fox, Liana, Christopher Wimer, Irv Garfinkel, Neeraj Kaushal, JaeHyun Nam, and Jane 

Waldfogel (2015). “Trends in Deep Poverty from 1968 to 2011: The Influence of Family 

Structure, Employment Patterns, and the Safety Net.” The Russell Sage Foundation Journal of 

the Social Sciences 1: 14-34.

Garfinkel, Irv, David Harris, Jane Waldfogel, and Christopher Wimer (2016). “Doing More for 

Our Children: Modeling a Universal Child Allowance or More Generous Child Tax Credit.” 

New York, NY: The Century Foundation.
The Promise of Evidence-Based Policymaking 713

8

Garner, Thesia (2010). “Supplemental Poverty Measure Thresholds: Laying the Foundation.” 

Washington, DC: Bureau of Labor Statistics.

Groves, R. and Harris-Kojetin, B. (eds), “Innovations in Federal Statistics: Combining Data

Sources While Protecting Privacy,” National Academy Press, prepublication, January 2017.

Haveman, R., Blank, R., Moffitt, R., Smeeding, T., Wallace, G., “The War on Poverty: 

Measurement, Trends, and Policy,” Journal of Policy Analysis and Management 34:3, Summer 

2015.

Hokayem, Charles, Christopher Bollinger, and James P. Ziliak. 2015. “The Role of CPS

Nonresponse in the Measurement of Poverty,” Journal of the American Statistical 

Association 110(511): 935-945.

Interagency Technical Working Group. 2010. “Observations from the Interagency Technical 

Working Group on Developing a Supplemental Poverty Measure,” March 2010. 

https://www.census.gov/hhes/povmeas/methodology/supplemental/research/SPM_TWGObservat

ions.pdf

Meyer, B. and Mittag, N., "Using Linked Survey and Administrative Data to Better Measure 

Income: Implications for Poverty, Program Effectivenss and Holes in the Safety Net," NBER 

Working Paper No. 21676, October 2015

Meyer, B. and Sullivan, J. "Identifying the Disadvantaged: Official Poverty, Consumption 

Poverty, and the New Supplemental Poverty Measure,” Journal of Economic Perspectives,

Summer 2012, 111-136

Meyer, B., Mok, W., and Sullivan, J. "Household Surveys in Crisis," Journal of Economic 

Perspectives, Fall 2015, 29(4), pp. 199-226.

Nolan, L., J. Waldfogel, and C. Wimer. “Long-term Trends in Rural and Urban Poverty: New 

Insights Using a Historical Supplemental Poverty Measure.” The ANNALS

Nolan, Laura, Irwin Garfinkel, Neeraj Kaushal, JaeHyun Nam, Jane Waldfogel, and Christopher 

Wimer (in press). “Trends in Child Poverty by Race/Ethnicity: New Evidence Using a Historical 

Supplemental Poverty Measure.” Journal of Applied Research on Children.

Nolan, Laura, Irwin Garfinkel, Neeraj Kaushal, JaeHyun Nam, JaneWaldfogel, and 

ChristopherWimer (2016). “A New Method for Measuring Historical Poverty Trends: 

Incorporating Geographic Differences in the Cost of Living Using the Supplemental Poverty 

Measure.” Journal of Economic and Social Measurement 1: 1-28.

Nolan, Laura, Jessica Pac, Christopher Wimer, Irwin Garfinkel, Neeraj Kaushal, JaeHyun Nam, 

and Jane Waldfogel (2016). Poverty in the 50 States: Long Term Trends and the Role of Social 

Policies.
714 The Promise of Evidence-Based Policymaking

9

Pac, Jessica, Jane Waldfogel, and Christopher Wimer, “Poverty among Foster Children: 

Estimates using the Supplemental Poverty Measure.” (Social Service Review).

Renwick, T. and Fox, L. The Supplemental Poverty Measure: 2015, U.S. Census Bureau, Report

Number: P60-258, 2016

Sherman, A. and Trisi, D. “Safety Net More Effective Against Poverty Than Previously Thought

Correcting for Underreporting of Benefits Reveals Stronger Reductions in Poverty and Deep 

Poverty in All States, CBPP brief, May 6, 2015

Short, Kathleen (2011). “The Research Supplemental Poverty Measure: 2010.” Current 

Population Reports P60-241.

Smeeding, Timothy M., and K. Thornton. 2016. “Wisconsin Poverty Report: Poverty Levels 

Flat on Average More Diverse within State in 2014 ” Eight Annual Report of the Wisconsin 

Poverty Project. Institute for Research on Poverty, University of Wisconsin–Madison: (June)

Wight, V., Kaushal, N., J. Waldfogel. (2014). “Understanding the Link between Poverty and 

Food Insecurity among Children: Does the Definition of Poverty Matter?” Journal of Children 

and Poverty.

Wimer, Christopher, Liana Fox, Irwin Garfinkel, Neeraj Kaushal, and Jane Waldfogel (2016). 

“Progress on Poverty? New Estimates of Historical Trends Using an Anchored Supplemental 

Poverty Measure.” Demography 53(4):1207-18.

Winship, S. “Testimony before the Committee on the Budget,” U.S. House of Representatives, 

Washington, D.C. January 28, 2014.
The Promise of Evidence-Based Policymaking 715

1

Strengthening	
  Data	
  and	
  Evidence	
  to	
  Improve	
  Education Programs

Karen	
  Levesque,	
  Dorothyjean	
  Cratty,	
  and	
  Nicole	
  Ifill

RTI	
  International

RTI	
  International	
  is	
  an	
  independent,	
  nonprofit	
  institute	
  whose	
  mission	
  is	
  to	
  improve	
  

the	
  human	
  condition	
  by	
  turning	
  knowledge	
  into	
  practice.	
  RTI	
  staff	
  are	
  experts	
  in	
  

education	
  data	
  from	
  system	
  development	
  to	
  data	
  collection,	
  linking,	
  analysis,	
  

reporting,	
  communication,	
  and	
  use,	
  and	
  including	
  administrative	
  and	
  survey	
  data	
  at	
  

both	
  the	
  K12	
  and	
  postsecondary	
  education	
  levels.

Submitted	
  to	
  the	
  Commission	
  on	
  Evidence-­‐Based	
  Policymaking (CEP)

January	
  24,	
  2017

Introduction

US	
  education	
  is	
  an	
  enormous	
  enterprise	
  facing	
  daunting educational	
  challenges.	
  

Practitioners,	
  policymakers,	
  researchers,	
  parents, and	
  other	
  stakeholders seek	
  

evidence-­‐based	
  strategies	
  for	
  improving	
  student	
  outcomes. In	
  this	
  brief	
  we	
  describe	
  

several	
  important	
  opportunities	
  for strengthening state and	
  federal education	
  data	
  

to	
  help	
  improve K12	
  and	
  postsecondary	
  education	
  programs.

State Education Data	
  and	
  Evidence	
  

Important	
  Data	
  Differences

Some federally	
  funded	
  programs that	
  are	
  administered	
  at	
  the	
  state	
  and	
  local	
  

levels—such	
  as	
  unemployment	
  insurance	
  and the	
  Supplemental	
  Nutrition	
  

Assistance	
  Program	
  (SNAP)—collect standardized	
  individual-­‐level	
  data, which	
  are

reported to	
  the	
  relevant	
  federal	
  agencies.	
  While	
  K12	
  education	
  is	
  also	
  administered	
  

at	
  the	
  state	
  and	
  local	
  levels (with	
  some	
  federal	
  funding),	
  state	
  and	
  local	
  education

data	
  systems	
  vary	
  widely and,	
  in	
  addition	
  to	
  being	
  used	
  for	
  state	
  and	
  local	
  purposes,

report	
  aggregate	
  data	
  only	
  to	
  the	
  US Department	
  of	
  Education	
  (USED).	
  

Federal	
  Investments in State	
  Education Data

Historically,	
  state	
  education	
  agencies	
  collected	
  aggregate administrative	
  and

performance	
  data	
  from	
  school	
  districts	
  for	
  state	
  and	
  federal	
  reporting

purposes.	
  After	
  No	
  Child	
  Left	
  Behind was	
  enacted	
  in 2001,	
  most	
  states began

building	
  student-­‐level	
  longitudinal	
  data	
  systems	
  to	
  track	
  individual	
  student

performance over	
  time.	
  Since	
  2006,	
  USED has	
  awarded six	
  rounds	
  of	
  competitive	
  

grants	
  to	
  states	
  to	
  support the	
  development,	
  building, securing,	
  and	
  use	
  of	
  these	
  

statewide	
  longitudinal	
  data	
  systems	
  (SLDSs).	
  Early	
  grants	
  focused	
  on	
  building	
  K	
  

through	
  12	
  data	
  systems	
  and	
  states	
  have	
  increasingly	
  been	
  able	
  to	
  collect	
  and	
  link	
  

the	
  following at	
  the	
  student	
  level:
716 The Promise of Evidence-Based Policymaking

2

• demographics,	
  disabilities,	
  and	
  behavior

• courses,	
  tests,	
  and	
  grades

• assigned	
  teachers’	
  qualifications	
  and	
  experience

• participation	
  in	
  career	
  and	
  technical	
  education,	
  gifted	
  programs,	
  etc.

• K12	
  outcomes	
  such	
  as high	
  school	
  graduation	
  and	
  college	
  readiness

• school	
  climate	
  survey	
  data and	
  social	
  emotional	
  learning	
  data

Subsequent SLDS	
  grants	
  focused	
  on	
  integrating data	
  from additional	
  sectors,	
  

including	
  early	
  childhood	
  education,	
  postsecondary	
  education,	
  and	
  workforce.	
  

Additional federal	
  grants	
  (USDOL	
  Workforce	
  Data	
  Quality	
  Initiative	
  and	
  USDHHS	
  

early	
  childhood	
  learning)	
  have	
  also	
  helped	
  to	
  expand SLDSs	
  beyond	
  K12 to	
  P20W+	
  

data	
  systems, covering	
  the	
  complete	
  pipeline from	
  early	
  learning	
  through	
  

postsecondary	
  and	
  workforce	
  preparation	
  and	
  outcomes,	
  and	
  including	
  

nontraditional	
  paths	
  through	
  adult	
  education.

Turning	
  Data	
  into	
  Evidence

Student-­‐level	
  longitudinal	
  data	
  make	
  it	
  possible	
  for	
  educators,	
  policymakers,	
  

researchers,	
  and	
  even	
  parents	
  and	
  students	
  to	
  understand	
  the	
  factors	
  that	
  are	
  

associated	
  with	
  better	
  outcomes as students	
  transition	
  to	
  the	
  next	
  education	
  level	
  or	
  

into	
  the	
  workforce,	
  including	
  improved	
  learning;	
  persistence,	
  promotion,	
  and	
  

completion;	
  and	
  labor market	
  success.	
  These	
  data	
  can	
  help	
  to	
  determine	
  predictors	
  

of	
  positive	
  outcomes,	
  develop	
  early	
  warning	
  tools,	
  and	
  identify	
  effective	
  programs	
  

and	
  practices.

In	
  addition	
  to	
  addressing state	
  and	
  local	
  information	
  needs,	
  SLDSs	
  can	
  support	
  

federal	
  evidence	
  requirements.	
  The	
  new	
  Every	
  Student	
  Succeeds	
  Act,	
  Workforce	
  

Innovation	
  and	
  Opportunity	
  Act,	
  and	
  Education	
  Department	
  General	
  Administrative	
  

Regulations	
  now	
  require	
  state	
  and	
  local	
  agencies	
  and	
  awardees	
  to	
  link	
  cross-­‐sector	
  

data	
  and	
  to	
  propose,	
  develop,	
  and use	
  an	
  evidence	
  base	
  for	
  federally	
  funded	
  

programs.	
  The	
  new	
  grant	
  evidence	
  categories	
  encompass	
  traditional	
  causal	
  studies,	
  

such	
  as	
  RCTs	
  and	
  quasi-­‐experiments,	
  as	
  well	
  as	
  well-­‐designed	
  correlational	
  studies,	
  

all	
  of	
  which can	
  be	
  informed	
  by	
  SLDS	
  data.

States	
  and	
  districts	
  have	
  developed sophisticated	
  and	
  secure	
  data	
  systems	
  that	
  link

cross-­‐sector	
  data	
  and	
  offer access	
  under	
  the	
  relevant	
  federal,	
  state,	
  and	
  district	
  

privacy	
  laws. However,	
  more	
  support	
  is	
  needed	
  to	
  develop	
  new administrative	
  data	
  

science	
  methods	
  that	
  produce	
  credible	
  evidence	
  and	
  to	
  encourage	
  appropriate	
  use	
  

of	
  the	
  data and	
  evidence to	
  inform	
  policy	
  and	
  practice.

Recommendations

We	
  recommend	
  that	
  CEP:

1. Encourage	
  continued	
  federal	
  investment in	
  the development	
  and	
  use	
  of	
  SLDSs;	
  

and
The Promise of Evidence-Based Policymaking 717

3

2. Promote development	
  of	
  administrative	
  data	
  science	
  methods	
  that	
  produce

sound	
  and	
  accessible	
  correlational,	
  quasi-­‐experimental,	
  and causal	
  evidence.	
  

Developing	
  these systems and	
  methods	
  will simultaneously	
  help	
  to	
  meet federal,

state, and	
  local	
  demands for evidence	
  of	
  effective	
  education	
  practices,	
  programs,	
  

and	
  policies.

Federal	
  Education Data	
  and	
  Guidance

RTI	
  is a	
  member	
  of	
  the	
  Postsecondary	
  Data	
  (PostsecData)	
  Collaborative,	
  which	
  

made	
  several	
  recommendations	
  in	
  comments	
  provided to	
  CEP in	
  November.	
  Below,	
  

we develop several	
  of	
  these recommendations further.

1. Make	
  recommendations	
  that	
  address	
  the	
  administrative	
  and	
  legal	
  barriers	
  to	
  

data	
  linking	
  and	
  access.	
  

Administrative	
  data,	
  when	
  de-­‐identified	
  and	
  linked	
  appropriately,	
  can	
  provide	
  

important context	
  for	
  understanding the	
  education	
  landscape.	
  Understandably,	
  

access	
  to	
  Internal	
  Revenue	
  Service	
  (IRS) and	
  Social	
  Security	
  Administration	
  (SSA)

data	
  is	
  highly	
  controlled	
  for	
  privacy	
  and	
  security	
  reasons.	
  However,	
  as	
  the	
  recent	
  

release	
  of	
  Raj	
  Chetty’s	
  paper, Mobility	
  Report	
  Cards:	
  The	
  Role	
  of	
  Colleges	
  in	
  

Intergenerational	
  Mobility, shows, linked	
  data	
  can	
  be	
  handled	
  properly	
  and	
  enable

new	
  analyses	
  that	
  cannot	
  be	
  undertaken	
  using	
  currently	
  available	
  data such	
  as	
  the	
  

institution-­‐level	
  Integrated	
  Postsecondary	
  Education	
  Data	
  System	
  (IPEDS)	
  or the	
  

National	
  Center	
  for	
  Education	
  Statistics’	
  sample	
  surveys.	
  

While	
  IPEDS	
  and	
  the	
  sample	
  surveys	
  are	
  important	
  sources	
  of	
  data,	
  each	
  has	
  its	
  

limitations.	
  IPEDS	
  can	
  only	
  be	
  used	
  to	
  explore	
  differences	
  at	
  the	
  institution	
  level	
  and	
  

does	
  not,	
  for	
  example, currently	
  collect	
  detailed	
  outcomes	
  on	
  all	
  student	
  

populations.	
  Because	
  these	
  data	
  are	
  not	
  student	
  level,	
  they	
  cannot	
  be	
  merged with

other	
  data	
  that	
  could	
  provide the	
  missing	
  student	
  characteristics. Further,	
  while	
  a	
  

useful source	
  of	
  student-­‐level	
  education	
  data,	
  the	
  National	
  Postsecondary	
  Student	
  

Aid	
  Study,	
  Beginning	
  Postsecondary	
  Students	
  Longitudinal	
  Study,	
  and	
  Baccalaureate	
  

and	
  Beyond	
  Longitudinal	
  Study are	
  constrained	
  by	
  their	
  sampling designs.	
  Because	
  

they	
  are	
  nationally	
  representative,	
  but not	
  state	
  representative in	
  most	
  cases,	
  these	
  

surveys	
  cannot	
  currently	
  be	
  used	
  to make	
  comparisons across	
  or within	
  states,	
  

despite a growing	
  interest in	
  making	
  such comparisons.	
  

The Western	
  Interstate	
  Commission	
  for	
  Higher	
  Education Multistate	
  Longitudinal	
  

Data	
  Exchange,	
  the	
  Southern	
  Regional	
  Education	
  Board	
  State	
  Data	
  Exchange,	
  and	
  

the	
  University	
  Innovation	
  Alliance	
  are examples	
  of	
  individual	
  states	
  and	
  institutions	
  

agreeing	
  to	
  share	
  data	
  with	
  a	
  common	
  goal	
  of	
  understanding	
  the	
  enrollment	
  and	
  

completion	
  patterns	
  of	
  students and	
  improving	
  student	
  outcomes.	
  By	
  

recommending	
  actions	
  to	
  simplify	
  the	
  sharing	
  and	
  comparing of	
  data	
  across	
  states,	
  

the	
  Commission	
  can	
  signal	
  its support	
  of	
  these	
  existing	
  agreements	
  and	
  encourage	
  

other	
  states	
  to	
  explore	
  similar	
  agreements.	
  
718 The Promise of Evidence-Based Policymaking

4

2. Align	
  definitions	
  and	
  metrics	
  across	
  federal	
  laws.	
  

Establishing	
  common	
  definitions	
  for	
  data	
  metrics	
  across	
  federal	
  laws	
  would	
  reduce	
  

administrative	
  burden	
  and	
  create	
  comparable	
  outcome measures	
  across	
  federal	
  

programs.	
  Federal	
  laws	
  such	
  as the	
  Higher	
  Education	
  Act,	
  the	
  Workforce	
  Innovation	
  

and	
  Opportunity	
  Act (WIOA),	
  the	
  Perkins	
  Career	
  and	
  Technical	
  Education	
  Act,	
  and	
  

the	
  Elementary	
  and	
  Secondary	
  Education	
  Act	
  contain	
  similar	
  metrics	
  that	
  could be	
  

streamlined.	
  While	
  these	
  recommendations	
  require	
  Congressional	
  action,	
  consistent	
  

definitions	
  can cut	
  costs	
  and	
  reduce	
  security	
  risks	
  by	
  minimizing	
  the	
  number	
  of

redundant	
  data	
  collections	
  across	
  the	
  federal	
  government.	
  The	
  Commission	
  can

recommend a	
  legal	
  framework	
  that	
  uses	
  the	
  proposed	
  national	
  clearinghouse	
  to	
  

align metrics	
  around:	
  

• student	
  enrollment	
  rates in	
  colleges	
  and	
  programs	
  

• college	
  readiness	
  

• college	
  and	
  program	
  completion	
  

• employment	
  rate	
  or	
  job	
  placement	
  rate	
  

• earnings	
  

• credential	
  attainment	
  

Aligning metrics	
  across	
  federal	
  agencies	
  would streamline existing	
  reporting	
  and	
  

allow	
  consumers	
  to	
  make	
  informed	
  decisions	
  about	
  programs	
  that	
  span	
  agencies.	
  

Standard	
  definitions	
  would	
  also	
  create	
  opportunities	
  to	
  combine	
  and	
  expand	
  data	
  

dashboards	
  and	
  reporting	
  for	
  additional	
  purposes such	
  as consumer	
  information	
  

and	
  regulatory	
  compliance.	
  For	
  example,	
  the	
  College	
  Scorecard	
  could	
  include	
  data	
  

from	
  DOL’s WIOA	
  reporting	
  in	
  order	
  to	
  show	
  training	
  program	
  outcomes	
  and	
  serve	
  

as	
  a	
  resource	
  for	
  students	
  to	
  understand	
  outcomes	
  for	
  different	
  career	
  pathways.	
  

Source:	
  Adapted	
  from	
  PostsecData	
  Collaborative	
  November	
  14,	
  2016	
  comments	
  to	
  the	
  

Commission.

3. Leverage	
  existing	
  data	
  to	
  decrease	
  burden,	
  streamline	
  reporting,	
  and	
  answer	
  

critical	
  stakeholder	
  questions.	
  

Existing	
  administrative	
  data	
  can help policymakers,	
  institutions,	
  and	
  students	
  

answer	
  critical	
  questions	
  about	
  postsecondary	
  access,	
  success,	
  post-­‐college	
  

outcomes,	
  and	
  affordability,	
  such	
  as:	
  

o How	
  many	
  low-­‐income,	
  Pell	
  Grant-­‐recipient,	
  first-­‐generation,	
  veteran,	
  adult,	
  

transfer,	
  and	
  part-­‐time	
  students—who	
  make	
  up	
  the	
  new	
  majority	
  on	
  today’s	
  

campuses—attend	
  each	
  college?	
  Do	
  these	
  students	
  graduate?	
  

o How	
  long	
  does	
  it	
  take	
  students,	
  particularly	
  those who	
  enter	
  college	
  with	
  less	
  

academic	
  preparation	
  or	
  fewer	
  financial	
  resources,	
  to	
  complete	
  college?	
  

o Do	
  students	
  who	
  don’t	
  graduate	
  transfer—or	
  do	
  they	
  drop	
  out?	
  

o How	
  much	
  money	
  do	
  different	
  types	
  of	
  students	
  borrow for	
  college	
  and	
  how	
  

do	
  their	
  repayment	
  outcomes	
  vary?	
  
The Promise of Evidence-Based Policymaking 719

5

o Can	
  students	
  find	
  jobs	
  in	
  their	
  chosen	
  field and	
  how	
  much	
  do	
  they	
  earn?	
  

Data	
  from	
  the USED,	
  SSA, Department	
  of	
  Defense	
  (DoD),	
  and	
  Department	
  of	
  

Veterans	
  Affairs	
  (VA) provide	
  valuable	
  information on	
  important	
  subgroups	
  of	
  

students	
  who	
  are	
  often	
  overlooked,	
  including	
  Pell	
  Grant	
  recipients,	
  student	
  loan	
  

borrowers,	
  and	
  student	
  veterans. For	
  example,	
  SSA	
  and	
  IRS tax	
  records	
  have,	
  in	
  

limited	
  instances,	
  been	
  used to	
  report	
  employment	
  and	
  earnings	
  outcomes	
  at	
  the	
  

program	
  level	
  to	
  better	
  understand	
  students’ workforce	
  outcomes.	
  USED’s IPEDS	
  

and	
  National	
  Student	
  Loan	
  Data	
  System	
  include	
  information	
  on	
  financial	
  aid	
  and	
  

student	
  access	
  and	
  success	
  that	
  can	
  be	
  disaggregated	
  to	
  examine progress	
  for	
  

student	
  subpopulations. DoD	
  and	
  VA,	
  which	
  house	
  data	
  on	
  student	
  veterans,	
  

financial	
  aid,	
  and	
  recruiting, can	
  be	
  used	
  to	
  answer	
  these	
  questions	
  specifically	
  for	
  

the	
  veteran	
  population. Linking	
  these data	
  sources	
  can reveal opportunities	
  to	
  

eliminate	
  duplicative	
  data	
  collections	
  and	
  decrease	
  reporting	
  burden	
  on data	
  

providers.	
  

RTI	
  recognizes	
  the	
  importance	
  of	
  limiting	
  data	
  collections	
  to	
  those	
  metrics	
  that	
  have	
  

a	
  specific and	
  valuable	
  purpose	
  in meeting administrative,	
  policymaking,	
  and	
  

research	
  needs.	
  These	
  data	
  collections	
  should	
  also	
  include	
  a	
  robust	
  reporting	
  

function,	
  returning	
  data	
  back	
  to	
  colleges	
  and	
  programs	
  for	
  benchmarking	
  and	
  

improvement	
  purposes	
  and	
  to	
  add value	
  for	
  the	
  data	
  providers.	
  Any	
  clearinghouse	
  

or	
  data	
  system	
  solution	
  that	
  is	
  considered	
  by	
  the	
  Commission	
  should	
  focus	
  on	
  

answering	
  critical	
  questions about	
  student	
  outcomes; balancing	
  reporting	
  burden	
  

with	
  analytic	
  value; and	
  including	
  outcomes	
  data	
  on employment	
  and	
  earnings	
  for	
  

all	
  students,	
  at all	
  postsecondary	
  levels	
  and	
  in	
  workforce	
  programs.

Source:	
  Adapted	
  from	
  PostsecData	
  Collaborative	
  November	
  14,	
  2016	
  comments	
  to	
  the	
  

Commission.
720 The Promise of Evidence-Based Policymaking

At the October 21st CEP hearing, Emmet McGroarty of the American Principles 

Project was the only testimony at the hearing opposing the continuation or 

increase of federal collection of individual student data. Mr. McGroarty is not the 

only man or woman in America that opposes the federal unit record system of 

student data. Grassroots organizations, made up of highly skilled, organized and 

well-researched individuals are also fighting to protect our children and 

grandchildren. The pushback on data collection is vigorous and we won’t stop 

until we have a victory.

Numerous experts are seeking to repeal the prohibition in the current Higher Ed 

Act Section 134 against a federal student-unit record system that would collect 

personally identifiable information on higher education students and link 

education data to workforce data. I stand with Mr. McGroarty in asking that the 

protective barrier to government intrusion into citizen’s private lives remain intact.

This United Commission report to the President and Congress required by the 

law, is considering how the integration of data “might lead to the intentional or 

unintentional access/breach, or release of personally identifiable information or 

records.” This is a very serious question for the Commission to answer. The 

Commission is also faced with the question of whether the government even has 

the right to collect such personal data.

States were incentivized to create massive databases in 2002, allowing more 

than just the collection of academic performance and biographical information on 

individual students in K-12, as well as higher education. Prior to this, the law

prohibited the collection of PII such as disciplinary history, social emotional 

development and extra curricular activities of individual students at the federal 

level. However, Obama’s gutting of FERPA in 2012 now means that government 

education officials now have the freedom to disclose a student’s PII without 

parental consent.

Mr. McGroarty also presented this profound statement to the Commission on 

Evidence Based Policymaking: “Such databases make freeborn American 

citizens objects of research and study. It assumes that the goal of benefiting 

others in society justifies the powerful federal government collecting and 

disseminating millions of data points on individuals often without their expressed 

consent. This fundamentally changes the relationship between the individual and 

government. Our republic rests on the idea that the citizens control the 

government. That cannot truly happen when government sits in the position of 

intimidation over the individual.”

Having the government know the state of your social and emotional 

development, along with data from other data sets, like medical records that the 

Commission is considering, is very disturbing. The Commission is focused on the 
The Promise of Evidence-Based Policymaking 721

collection of data from higher education for consumer benefit and workforce 

planning. It would be one thing for colleges and universities to voluntarily create a 

private association that collected the kind of data the feds are considering in 

order to provide them with the data to back their marketing claims. However, the 

Commission is looking at having the data collected by the government.

The very thought of such data collection reveals the current state of affairs for 

higher education. The image of the modern university is rapidly declining from 

one where people with intellectual ability go to pursue an even more learned

understanding of a subject, towards one of a business in competition with other 

businesses offering an ever-expanding menu to try to eek out additional market

share. According to the government, the consumer (student) could use such 

data collected by a private association, policed by industry competitors to make 

sure it is accurate, to choose which business (university) to attend or not attend if 

they did not wish to be part of such data collection. The act of collecting this data 

by government order takes that choice away from students and can use the 

coercive power of law to force them to provide their PII to an entity which is not 

even part of this business sector.

I ask the Commission to consider whether the government can create a data 

clearinghouse and whether it can be self-funding. I am urging the following if 

such a clearinghouse is created:

1. The government must collect only aggregate data, not personally 

identifiable information.

2. Parents and students must have the ability to review and correct records 

and notice as to who can see those records.

3. If such a database is created, it should be funded through the general 

treasury, because this gets at the heart of the decay existent in public 

private partnerships.

The law only considers the cost of the work of maintaining such a database and 

looks for a return of investment. Such a data system will have the intimidating 

power of government behind it. The creation of self-funded enterprises removes 

an important check that the congress provides. Checks and balances are so 

critical when it comes to the collection of data.

I urge the Commission to resist the effort to collect and share personal family and 

student data. I also recommend that any such information that is collected be

excluded from any recommended clearing house and that such a clearing house 

be subject to all the checks and balances that is provided by our Constitution. In 

addition to whether the government should have records in the first place, I 

propose that parents and students have a means to review and correct their 

records and determine who may see them.
722 The Promise of Evidence-Based Policymaking

1333 Broadway, Suite 310 ● Oakland, CA 94612 ● Phone: (510) 763-1499 ● Fax: (510) 763-1599 ● 

www.spra.com

Critical Issues for the Commission on Evidence-Based Policymaking

Founded in 1991, Social Policy Research Associates (SPR) is a nationally recognized research, 

evaluation, and technical assistance firm located in Oakland, California. Our work provides insight 

to government, non-profit and foundation clients on what works in workforce development, 

human services, education and effective philanthropy. After more than 25 years of working with 

dedicated people working on behalf of some of some of our nation's most vulnerable populations, 

we’ve learned that people are better able to make good decisions when they have good 

information. Program directors want to do their best work and create lasting impact on their 

clients. This requires high-quality data, rigorous analysis, and the ability to act on the lessons from 

both. The commission's work represents an excellent opportunity to create infrastructure and 

culture to advance two areas critical to that ability:

 Promoting evaluation requirements for publicly-funded programs that are both rigorous 

and realistic

 Improving the quality and accessibility of administrative data, including by supporting the 

development of integrated state database systems.

We appreciate this opportunity to inform the commission on the importance of the above issues.

#1 Promote evaluation requirements that are both rigorous and realistic

The evidence generated by third-party evaluations can provide critical insight on how best to 

allocate resources and develop policies in order to move our nation's workers and families to 

economic stability. To ensure that such evidence is available, evaluation must be both required by 

the funding entity and held to certain standards of rigor. For example, SPR has been privileged to 

evaluate five different programs funded by the US Department of Labor's Workforce Innovation 

Fund, a grant program which enabled states and local areas to pilot innovative approaches in 

workforce development. Grantees received larger amounts to scale and/or replicate evidencebased approaches and evaluate them using an experimental or quasi-experimental design, and 

smaller amounts to test new approaches using evaluation approaches that measure changes in 

participant outcomes, but do not assess whether the program caused these changes. This 

approach allows both the program and its evaluation to move the field forward, by centering the 

importance of conducting evaluation while accounting for the fact that the most rigorous 

evaluation designs are not always appropriate for the first attempt to learn about a particular 

service design. 

Under the Workforce Innovation Fund, SPR has been able to conduct evaluations of promising 

approaches in workforce development—using a variety of methods appropriate to where each 

approach is situated in its development—to inform the rollout of service delivery under the new 

Workforce Innovation and Opportunity Act, which incorporates many approaches first tested under 
The Promise of Evidence-Based Policymaking 723

2 | Page

the Workforce Innovation Fund. Simply put, encouraging innovation while requiring evaluation 

makes it easier to learn what works and for whom. This approach should be expanded to other 

agencies and to any new funding streams to come under the current administration.

#2: Improving the Quality and Accessibility of Administrative Data

One of the most essential parts of conducting high-quality evaluation to support policy 

development and refinement is being able to track evaluation participants in administrative data: 

for example, using Unemployment Insurance wage records to determine whether a job training 

program increased earnings or using arrest and conviction records to determine whether an 

offender reentry program reduced recidivism. In both cases, to determine whether an investment 

in a program was worthwhile, evaluators need access to high-quality, individual-level data from 

government agencies. SPR has collected employment, education and criminal justice data from 

state and local agencies across the country, and encountered wide variation both across and within 

states in how such data are standardized, stored, shared, and linked depending on the agency in 

question. Key issues encountered include:

 Inconsistent, but often lengthy and cumbersome, processes for accessing data. Each 

agency negotiates its own data sharing agreements, which can add up to significant time 

and cost for national studies that require data from multiple agencies in many states, 

particularly when there may be variations in exactly what the process requires and how 

long it takes. 

 Differences in the collection and completeness of data across agencies. For example, data 

may be stored in different electronic formats or even in hard copy, and evaluators must 

therefore allocate time to merging, standardizing and even keying all of the data prior to 

analysis. Administrative data also do not always include or release the same variables, and 

this limits the extent to which analysis can identify what works for whom. 

In addition to this experience accessing administrative data for evaluation, SPR has also been 

privileged to serve as the technical assistance provider for the Department of Labor's Workforce 

Data Quality Initiative (WDQI), which provided grants to states to develop longitudinal 

administrative databases in order to facilitate a more comprehensive understanding of workers' 

training, employment and earnings outcomes over the course of their careers. Longitudinal 

databases can also help state agencies understand system dynamics over time—how quickly, for 

example, training programs are developed and accessed relative to labor market shifts that require 

new training. Through our work with state agencies as a technical assistance provider, several key 

challenges have been made clear: the difficulty of tracking outcomes for individuals as they move 

across state lines, the need to navigate and adhere to multiple privacy regulations when linking 

data, and the need for improved data systems at the state level. WDQI has helped surface these 

issues and bring agencies together to consider how to address them, but further investment is 

needed to put these ideas into action and replicate successful approaches both across states and 

to other policy areas beyond workforce development and education.
724 The Promise of Evidence-Based Policymaking

3 | Page

Thank you for the opportunity to address the commission during this important time for evidencebased policymaking. Researchers, policymakers and program administrators have a growing 

common understanding of the need to do a better job of understanding what works and why, so 

we can invest public dollars wisely and do work that matters most.

Dr. Andrew Wiegand (Ph.D., Social Psychology, University of California, 

Santa Cruz, M.P.P., University of California, Berkeley) is the President and 

CEO of Social Policy Research Associates. In his nearly two decades of 

work at SPR, Dr. Wiegand has served as Project Director and/or Principal 

Investigator for more than 20 national evaluations, as well as multiple 

state and local evaluations A well-known expert on evaluations of 

workforce, education, and human services programs, particularly those 

focused on youth and justice-involved individuals, Dr. Wiegand has led 

or helped to lead national impact evaluations of the Pell Grant 

Experiments, the Reintegration of Ex-Offenders (RExO) Project, and 

YouthBuild, as well as multiple studies of programs funded under the 

Workforce Investment Act and its successor, the Workforce Innovation and Opportunity Act.
The Promise of Evidence-Based Policymaking 725

DATE: August 24, 2016

TO: Katherine G. Abraham, Chair, Commission on Evidence-Based Policymaking

Ron Haskins, Co-Chair

SUBJECT: Testimony submission to the Commission on Evidence-Based Policymaking

THROUGH: Shelly Martinez, Executive Director for the Commission on Evidence-Based 

Policymaking

VIA EMAIL: rochelle.wilkie.martinez@census.gov

On behalf of the American Evaluation Association (AEA), we are writing in support of the EvidenceBased Policymaking Commission and to thank you for your efforts on such an important and challenging 

undertaking. AEA formally supported this legislation and worked with Congressional staff during its 

drafting to advance awareness and use of evaluation in government decision making. Both Congress and 

executive branch agencies have critical needs for credible assessments of the relevance, efficiency, and 

effectiveness of government programs, policies, and activities. AEA applauds the work of the 

Commission to help Congress embed evaluation into program design and to ensure that quality data are 

available for evaluation of federal programs.

AEA looks forward to addressing the specific questions and issues that arise during the Commission’s 

deliberations. We are prepared to provide support and advice in any manner that best suits the needs of the 

Commission. This could include (1) in-person testimony during which AEA can engage in a dialogue with 

Commission Members, (2) suggestions about possible members for any evaluation-focused advisory 

committees, (3) interaction with various Commission subcommittees, and (4) additional, more detailed, 

documents. 

At the outset of your work, and drawing on its collective extensive evaluation expertise and over 30 years 

of practical experience, AEA wishes to bring the following recommendations to your attention regarding a 

portion of the charges to the Commission: 

Under section 4 (a), the Commission is charged with making “recommendations on how best to 

incorporate outcomes measurement, institutionalize randomized controlled trials, and rigorous impact 

analysis into program design.”

We agree that program design should include an appropriate evaluation framework to guide data 

collection and use over the life of a program. Such early evaluation planning also enables the use of 

rigorous impact evaluation methods that require data collection prior to implementation, such as 

randomized control trials. However, the opportunity to capitalize on early successes or to make midcourse corrections is also critical, so it is essential also to conduct ongoing formative evaluation through 

the program’s life cycle. For example, evaluation can address questions that arise during implementation 

of the program, such as the validity of assumptions that underlie program design, or challenges to 

implementation in the field. Early in the program’s history, relatively simple information may be needed 

quickly (e.g., regarding obstacles to participation in the program). Evaluators should match the 

Other Public Input–Hearing Testimony
726 The Promise of Evidence-Based Policymaking

methodology to the stage of program development and to information needs, which may call for a range of 

methods over time, including targeted data collection that may not always include outcomes measurement. 

Thus, the optimal integration of outcomes measurement, randomized controlled trials, and rigorous impact 

analysis should include room for other methods as well. 

Such a framework will also include several other components, including: (a) a sound procedure for 

establishing annual and multi-year evaluation agendas and timetables, (b) consultation with appropriate 

congressional committees, OMB, and other external program stakeholders on their information needs, (c) 

a dissemination plan, preferably with public access, (d) resources needed for evaluation, and (e) formal 

expectations regarding how the findings and conclusions of evaluations shall be considered in subsequent 

program design, program management, and decisions regarding program reform, expansion or 

termination. 

Organizational structure and staffing are also important. Thus, we suggest it important to ensure 

independence of the evaluation function from program offices with regard to evaluation design, conduct, 

and reporting. Consultation is needed to ensure relevance, but independence is needed to ensure 

impartiality. Adequate staffing of evaluation units and support for professional development are also 

necessary if the Commission’s work is to achieve the kind of benefits foreseen by Congress. 

Under section 4 (b), the Commission is charged to ‘consider whether a clearinghouse for program and 

survey data should be established and how to create such a clearinghouse.’

We encourage the Commission to engage in this important task while trying to avoid unintended negative 

side effects. While recognizing the value and importance of large-scale extant archives and datasets, they 

should not reduce the capacity to gather targeted data as needed to address program evaluation questions. 

Measures of a program's key processes and outcomes should be established while the program is being 

developed, and should be put into place by the time program implementation begins. The most relevant 

measures might not exist in an extant clearinghouse. 

We also ask the Commission to consider whether the proposed clearinghouse could serve as a repository 

for evaluation reports, serving as an archive capacity for the collection, dissemination, and preservation of 

knowledge and lessons learned from evaluation studies. This would provide an enhanced base for guiding 

future program design and management, which often requires a critical mass of knowledge to properly 

comprehend and address the complexity of program processes and influences. It would also be a great 

benefit for future meta-analyses of evaluation findings. 

The American Evaluation Association has been the professional organization devoted to the application 

and exploration of evaluation in all its forms since 1986. It now has 6,000 members across all 50 states, as 

well as 80 other countries. It has developed professional standards for the quality of studies and ethics for 

the multidisciplinary members of the field. AEA has created a community of learning and practice over

the decades, in which its members in academia have worked to develop, refine and teach evaluation 

methods, while its members in practice have served the evaluation needs of agencies across the federal (as 

well as state and local) government. AEA developed the attached An Evaluation Roadmap for a More 

Effective Government (www.eval.org/d/do/472 ) to offer to improve evaluation and the use of study 

results in the federal government. This document defines “evaluation” and addresses management 
The Promise of Evidence-Based Policymaking 727

requirements for ensuring evaluation quality and usefulness, including methods, human resources, 

budgets, independence, transparency, and professional ethics.

We look forward to a dialogue regarding Commission efforts and progress. If we can be of further 

assistance, or if you need more information about evaluation or our organization, please do not hesitate to 

call on us or to contact Dr. Cheryl Oros, our Senior Advisor for evaluation policy 

(EvaluationPolicy@eval.org; 202-367-1166).

Sincerely, 

John Gargani, Ph.D.

President, American Evaluation Association

Kathryn Newcomer, Ph.D.

President-Elect, American Evaluation Association

Stewart Donaldson, Ph.D.

Past President, American Evaluation Association

Attachment: An Evaluation Roadmap for a More Effective Government (www.eval.org/d/do/472)
728 The Promise of Evidence-Based Policymaking

September	9,	2016

Dr.	Katharine	G.	Abraham,	Chair

Mr.	Ron	Haskins,	Co-Chair

Commission	on	Evidence-Based	Policymaking

U.S.	Census	Bureau

4600	Silver	Hill	Road

Suitland,	MD	20746

Dear	Chairwoman Abraham	and	Co-Chairman Haskins:

America	Forward	is	writing	to	thank	you	both	for	your	willingness	to	lead	the	important	effort	of	the	

newly	established	bipartisan	Commission	on	Evidence-Based	Policymaking.		We	were	advocates	of	the	

Evidence-Based	Policymaking	Commission	Act	when	it	was	introduced	by	Speaker	Ryan and	Senator	

Murray	and	supported	efforts	to	ensure	its	passage	given	its	alignment	with	our	belief	that	in	times	of	

greater	demand	for	human	and	social	services	and	tighter	budgets,	we	must	work	together	to	direct	

government	resources	to	programs	that	work	and	that	measurably	improve	people’s	lives.		We	would	

like	to	offer	our	thoughts	on	the	work	before	the	Commission	over	the	next	15	months	and	offer	specific	

insights	and	recommendations	based	on	our	Coalition’s	experiences	for	how	to	ensure	the	promise	of	

providing	“the	nation	with	guidance	on	ways	to	further	expand	our	approaches	to	evidence-building	in	

government”1 is	realized.	

America	Forward is	a	nonpartisan	effort	to	unite	social	innovators	with	policymakers	and	advance	a	

public	policy	agenda	that	fosters	innovation,	rewards	results,	catalyzes	cross-sector	partnerships,	and	

translates	local	impact	into	national	change.		At	America	Forward,	we	champion	innovative,	effective,	

and	efficient	solutions	that	are	helping	to	tackle	our	country’s	most	pressing	social	problems.		We	do	

this	through	a	network	of	more	than	70	social	innovation	organizations,	the	America	Forward	Coalition,

who	are	driving	progress	in	areas	such	as	education,	workforce	development,	early	learning,	public	

health,	pay	for	success,	and	national	service	in	more	than	14,500	communities	nationwide,	touching	the	

lives	of	8	million	Americans	each	year.		Our	work	is	grounded	in	the	real	world,	community-based	

experiences	of	these	organizations	and	those	they	serve.		This	grounding,	we	think, serves	us	extremely	

well	in	not	only	identifying	and	championing	policy	solutions	that	will	have	real	meaningful	impact	but	

our	ability	to	point	to	tangible	examples	of	these	solutions	in	action,	and	to	real	people	who	these	

solutions	have	touched,	sets	us	apart	from	other	advocacy	organizations.

Ensure Focus	on Practice	and On-the-Ground Implementation

The	real	promise	for	the	Commission	for	us	at	America	Forward	is	that	the	focus	on	bringing	evidence	

and	greater	attention	to	outcomes	will	not	only	be	in	service	to	the	creation	of	better	policies	and	

budgeting	practices	but	that	it	will	also	serve	to	inform the	improvement	of	existing	federal	programs,

service	practice	and	delivery.		Better	policies	and	more	effective	allocation	of	resources	only	fixes	a	part	

of	the	equation	of	having	more	evidence-based, government	funded	human	and	social	services.		The	

way	those	services	are	delivered	is	equally	important	to	ensuring	that	we	are	measurably	improving	the	

lives	of	those	who	need	to	access	these	critical	services	at	one	point	or	another	in	their	lives.		Like	our	

																																																																			 1 http://www2.census.gov/about/linkage/meetings/2016-07-22/abraham-haskins.pdf
The Promise of Evidence-Based Policymaking 729

approach	to	policy	advocacy—it	is	not	just	about	a	good	policy,	it	is about	how	that	policy	works	on	the	

ground	and	how	it	impacts	those	the	policy	is	meant	to	help.		

As	with	many	Congressionally	created	commissions,	there	is	the	risk	that	this	Commission	will	fail	to	

include	this	practical	lens	to	its	work.		State	and	local	level	government	leaders	and	service	providers,	

who	are	collecting	data,	using	data,	and	ultimately	serving	the	individuals	who	are	behind	the	data	

points	we	need	to	build	the	evidence	base around,	are	where	the	real	promise	of	the	Commission	lies.		It	

would	be	prudent	for	the Commission	to	include	in	its	outreach	and	information	gathering individuals	

outside	of	the	traditional	academic	and	research	communities as	well	as	to	include	a	focus	on	how	to	

make	data	available	to	inform	real	world	service	program	design,	delivery,	and	indicators	of	impact.	

It	is	in	fact	the	complementary	combination	of	classic	academics,	applied	researchers,	and	program	

facilitators	and	service	providers	that	will	ensure	that	the	work	of	this	Commission	is	as	robust	and	

significant	as	its	promise	holds.		We	think	it	is	imperative	that	all	of	these	perspectives,	from	all	levels—

federal,	state	and	local—as	well	as	across	issue	areas,	are	included	and	that	a	well-rounded	group	of	

experts	is	a	part	of	the	discussion	that	leads	to	this	important	Commission’s	conclusions	and	

recommendations.			

Include	Provider/Practitioner	Perspectives	on	Data	

America	Forward	Coalition	members	share	a	commitment	to	using	data	to	track	progress	and	ensure	

accountability.	That	is	why	we	are particularly	supportive	of	the	focus	of	the	Commission	on	developing	

a	strategy	for	increasing	the	availability	and	use	of	data	in	order	to	build	evidence	about	government	

programs.		The	availability	and	use	of	administrative	data	is	of	particular	interest	because, as	we	know	

from	our	members’	experiences,	it	is	extremely	important	for	engaging	in	meaningful	evaluations	and	

for	driving	down	the	cost	of	randomized	controlled	trials.		

In	addition	to	availability	and	use	of	data,	our	Coalition	members	have	identified	other	challenges	

associated	with	data	in	the	context	of	building	the	evidence	of	what	works.		These	challenges	include:

1. Even	if	data	are	collected	and	available,	antiquated	data	systems particularly	in	the	public	

sector	make	it	difficult	to	efficiently	access	or	engage	in	analyses	of	the	data	to	measure	

outcomes	and	determine	impact.	

2. The	challenges	that	millions	of	Americans	face	every day	cut	cross	agencies	and	sectors.		

Allowing specific	datasets	to	“speak	to”	each	other is	an	important	consideration	for	the	

Commission	to	explore.		In	order	to	adequately	measure	results	and	assess	impact,	data	access	

and	utilization	must	reflect	the	interconnected	nature	of	the	problems faced	by	Americans	and	

the	solutions being	developed	by	organizations	like	those	in	the	America	Forward	Coalition.	

3. The mobility	of	Americans	can	make	it	difficult	to	track	outcomes	absent	a state	or	national	

database that	can	accessed	and	used	for	building	evidence	and	strengthening	efforts	by	

providers	and	governments	to	be	more	evidence-based.		All	of	this	must	be	done	with	an	

assurance	of	privacy	and	confidentiality.	

4. Given	the	long	time	focus	on	compliance	with	rules	rather	than	achievement	of	outcomes,	both	

the	service	delivery	and	government	workforce	are	not	accustomed	to	working	with	data for	

purposes	of	evidence	determination.	
730 The Promise of Evidence-Based Policymaking

Encourage	Federal	Agencies	to	Adopt	Relevant	Data-Driven	Innovations	with	Evidence	of Effectiveness	

Identified	through	Innovation	Fund	and	Tiered	Evidence	Programs	

Tiered	evidence	models	and	innovation	funds	have	been	authorized	across	different	agencies	and	

included	in	various	issue	areas	to	develop	and	build	the	evidence	base	of	what	works	in	education and	

social	services.		One	example,	the	Social	Innovation	Fund (SIF),	makes	grants	to	experienced	

intermediary	organizations	that	are	well	positioned	within	communities	to	identify innovative,	evidencebased	programs	with	potential	for	expansion	in	the	areas	of	economic	opportunity,	healthy	futures,	and	

youth	development.		Since	2010,	the	SIF	has	made	awards	totaling	$243 million	with	more	than	$528

million	in	private	and	non-federal	matching	funds.	In	total, more	than 400 nonprofit	organizations	are	

being	funded	by	the	SIF	to	conduct	diverse	interventions	and	evaluate	results	through	highly	rigorous	

models.

One	challenge	facing	innovations	funds like	SIF	and	tiered	evidence	programs	such	as	the	Investing	in	

Innovation	program	(i3)	at	the	Department	of	Education	is	identifying	a	pathway	to	integrating	and	

expanding	approaches	determined	to	have	evidence	of	effectiveness	under	this	funds	in	a	manner	that	

aligns	with	existing	federal	efforts	to	address	national	and	local	challenges.	In	fact,	despite	the	SIF’s	

widespread	success and	the	bi-partisan	support	of	i3,	many	federal	agencies	are	unaware	of	the	

innovations	supported	by	these	funding	streams	that	could	help	improve	existing	programs,	yielding	

better	outcomes	for	vulnerable	communities. As the	Commission	builds	out	its	priorities	and	scope	of	

work,	our	Coalition	suggests	that	you	encourage	federal	agencies	to	adopt	relevant	data-driven	

programs/strategies	with	evidence	of	effectiveness	currently	supported	by	the	Social	Innovation	Fund,	

the	Investing	in	Innovation	Fund	and	other	similar	programs	whose	evaluations,	data,	and	experience	of	

partner	organizations	are	of	particular	value	to	federal	agencies	tasked	with	providing	effective	services	

in	the	areas	of	economic	opportunity,	healthy	futures,	and	youth	development.

America	Forward	stands	ready	to	work	with	this	important	Commission	to	ensure that	the	concepts	of	

evidence	and	outcomes	are	being	used	to	develop	policies	and	fund	interventions.		We	believe	this	

should	become	common	practice	at	the	federal,	state	and	local	level	to	address	the	current	and	future	

challenges	we	face	as	a	nation.

Thank	you	for	your	leadership	of	the Commission	on	Evidence-Based	Policymaking	and	for	your	effort	to

strengthen	the	government’s	evidence-building	efforts.

Sincerely,	

Nicole	Truhe

Government	Affairs	Director,	America	Forward

1400	Eye	Street,	NW	Ste	400

Washington,	DC	20004
The Promise of Evidence-Based Policymaking 731

To: Commission on Evidence-Based Policymaking

From: Laura and John Arnold Foundation

Date: December 22, 2016

Re: Docket ID USBC-2016-0003

Laura and John Arnold Foundation

Letter to the Commission on Evidence-Based Policymaking 

Dear Commissioners,

As Vice Presidents of the Laura and John Arnold Foundation (LJAF) focused on 

improving evidence-based policymaking by federal, state, and local governments, we 

are submitting this letter in response to the Commission on Evidence-Based 

Policymaking’s request for comments.

We believe the Commission’s statutory charge from Congress presents a critical 

opportunity to reshape the way governments use data and evidence to improve 

citizens’ well-being. By using modern technology and protocols that protect 

individual privacy, it is now possible to create a data infrastructure that enables 

government decision makers, as well as consumers and private sector organizations, 

to obtain reliable, actionable information that can drive their choices on how to use 

resources more effectively. 

Governments in other countries such as Denmark, the Netherlands, the United 

Kingdom, and Estonia are embracing secure data-linkage techniques to help solve 

problems for their citizens. In the United States, where administrative data from

social programs are scattered across programmatic silos in federal, state, and local 

governments, an optimal data infrastructure requires collaboration among multiple 

parties. We believe the government agencies that administer programs and thus 

“own”the associated administrative data should work with research data centers that 

have the existing infrastructure to link and analyze data while protecting individual 

privacy. Since the enactment of the Confidential Information Protection and 

Statistical Efficiency Act of 2002, federal statistical agencies and academic research 

data centers throughout the United States have demonstrated the ability to link and 

manage personally identifiable data for research purposes without breaches of 

privacy. 

We recommend the Commission go beyond offering general recommendations for 

improving data infrastructure and the evaluation of government programs as we 

know them today. Instead, the Commission should seize the opportunity to call for a 

modernized data infrastructure that could lay the essential groundwork for 

transformational reforms in how governments plan, structure, manage, and evaluate

programs to yield substantially higher returns on taxpayer investments. 
732 The Promise of Evidence-Based Policymaking

Recommendations

In recent years, LJAF has supported collaborations between governments, 

researchers, and data scientists across the country in an effort to help to address 

numerous challenges at the federal, state and local levels. In the appendix, we 

highlight some of our most impactful projects involving partnerships between 

governments and universities to enhance evidence-based policymaking. Our projects 

serve as proof points that demonstrate ways that government programs can be 

improved through the use of data and evidence. We offer four overarching 

recommendations for how to strengthen government’s evidence-building efforts: 

1. Develop and apply evidence-focused design principles to all government 

programs. Congress and the administration should embed strategies within 

programs that will increase the share of resources that are allocated to 

evidence-based practices and to rigorous evaluation of promising approaches. 

Initial design principles can be drawn from existing evidence-based initiatives 

such as the Department of Education’s Education Innovation Research 

program, pending legislation for Social Impact Partnerships, Performance 

Partnership Pilots for Disconnected Youth, and waiver demonstrations that 

require rigorous evaluations in welfare programs administered by the 

Administration on Children and Families and the Food and Nutrition Service. 

The principles should guide all new legislation, reauthorizations, regulatory 

reforms, and efforts to modernize agency administrative procedures. 

2. Set an expectation that state and local governments that administer

federal programs should conduct ongoing data analysis and rigorous 

evaluation and require that they make their administrative data 

accessible to the federal government to facilitate research. Governments 

should use existing program funds for this purpose. State and local 

governments are allocated hundreds of billions of dollars every year to 

administer federally funded programs and gather data on outcomes. Yet few 

have strong internal capacity for data analytics or evaluation, and many lack 

incentives to share their data for research and program improvement, which 

often requires a cumbersome process of negotiating agreements to link data 

across jurisdictional boundaries. The federal government should help state 

and local governments better measure their impact and administer their 

programs by creating the capacity to link data across jurisdictional silos and 

providing useful information to state and local governments. The federal 

government should also provide additional funding to facilitate data transfers. 

Such an approach would make it easier for academic institutions and 

innovative government structures to analyze the impact of new programs and 

policies on local populations. 

3. Create a network of government and research institutions that facilitates

secure data linkages while protecting individual privacy. To protect the 

privacy of individuals, there must be a secure and trusted network of access 
The Promise of Evidence-Based Policymaking 733

points to potentially identifiable data. This infrastructure should be built upon 

the existing university-based Federal Statistical Research Data Centers and

include modernized processes to speed approvals of projects that meet 

federally determined standards for accessing data. Together, these 

organizations can provide a highly secure data infrastructure to support 

important information needs of federal, state, and local governments, private 

sector organizations, and the public. Such a network can produce reliable 

published research and aggregated statistics to help governments and other 

decision makers understand the root causes of problems, direct services and 

benefits to where the needs are greatest, measure performance on important 

outcomes, and evaluate the impact of alternative strategies. 

4. Strengthen federal agency evaluation and management capacity. 

Congress and the administration should collaborate to refine and expand the 

successful evaluation practices the Obama administration implemented in 

agencies such as the Department of Labor. In addition, the Paperwork 

Reduction Act (PRA) and the Government Performance and Results 

Modernization Act (GPRMA) should be evaluated to determine how to better 

help federal, state, and local government agencies gather and use information 

to improve outcomes for program beneficiaries. Future efforts to assess and 

rate particular programs—such as a new Program Assessment Rating Tool 

(PART)—should be guided by specific policy goals that cut across program 

labels. Review processes should be designed to identify strategies within and 

across programs that are most effective at achieving overarching policy goals

and developing ways to scale those strategies.

The above recommendations address four core challenges that prevent government 

programs from realizing their full potential to improve lives: 

1. Fragmented federal program structures lack the flexibility and 

incentives to focus state and local resources on improving outcomes. 

Today, vulnerable populations are served by multiple programs run by 

different federal departments and state and local agencies, each with its own 

bureaucratic apparatus focused on a particular set of activities. Local 

governments and providers that deliver services lack the flexibility to combine 

funds and design coordinated interventions that will achieve the best 

outcomes per dollar spent. Only a handful of federal programs launched 

during the Obama administration—such as tiered evidence grantsi and Pay for 

Success initiatives—create strong financial incentives for grantees to use 

existing evidence of what works and participate in rigorous evaluations to 

build new knowledge. 

2. The federal government has poorly communicated the role of data 

analysis and evaluation in program implementation. There is a 

widespread perception across government programs and among grantees 

that data analysis and evaluation are not core elements of effective program 
734 The Promise of Evidence-Based Policymaking

administration, and little or no program funding is allocated for these 

purposes. Government has not signaled to grantees that they should integrate 

data analysis and evaluation into their operations in order to test, learn about, 

and improve the impact of strategies, nor that they should use a portion of 

their program funds for this purpose if other funding is not available. State and 

local governments that seek to test, learn, and improve their public impact are 

stymied when the people they serve move across jurisdictional lines or when 

the agencies that serve them in one location lack the necessary relationships 

to share information. Instead of using reliable data that government already 

collects to address this problem, each federal grant program sets its own 

reporting requirements to ensure accountability. However, this reporting, 

which is often onerous and time-intensive for grantees, rarely generates 

information that is useful for helping grantees or government program 

managers improve their programs. Rather, this approach to accountability 

undermines grantees’ confidence in the ability of the federal government to 

help with data analysis and evaluation.

3. Lack of secure access to high-quality and high-coverage data makes it 

difficult or impossible to measure progress and evaluate impact. 

Authoritative sources at various levels, including both governmental and 

private-sector sources, hold certain data that would be useful to government 

managers and grantees. However, this data is highly dispersed across sectors 

and levels of government. Examples include receipt of public benefits, 

earnings, healthcare usage, criminal justice involvement, and educational 

attainment—many of the key outcomes that public policies aim to improve. As 

people move between states and systems, it is extremely difficult to negotiate 

among all the data “owners” who must cooperate in order to build a complete 

picture of the individuals served by various systems. In the instances where

the federal government currently holds this kind of unified data on outcomes, 

there is little capacity to provide streamlined access to governments and 

researchers. Meanwhile, state and local data owners apply inconsistent 

protections to safeguard the privacy of individuals in their systems. 

4. Federal agencies lack the internal processes for assessing the impact of 

their programs and directing funds to use and build evidence. For the 

most part, agencies lack the internal structures and routines to unite the 

management of programs with the evaluation of strategies implemented 

within programs. Over time, the share of resources allocated to strategies 

proven to be effective through rigorous research—or to efforts to consciously 

build evidence on promising approaches—has remained troublingly low. 

Public outcomes have suffered. Impediments to learning and improvement are 

sometimes embedded in law. For example, the PRA and GPRMA have created 

burdensome compliance processes that divert resources from valuable 

activities. Past efforts, such as the PART, have been overly focused on the 

performance of individual programs rather than the improvement of public 

outcomes through strategies that cut across programs.
The Promise of Evidence-Based Policymaking 735

The attached document, “Detailed Recommendations from the Laura and John Arnold 

Foundation for Improving Government’s Capacity to Use and Build Evidence,” 

provides specific recommendations around each of these four challenges that are 

informed by our collective experience and the knowledge we have gained through 

our network of grantees. We would be happy to provide any additional information 

upon request.

Sincerely,

Kathy Stack, Vice President of Evidence-Based Innovation

Jon Baron, Vice President of Evidence-Based Policy

Josh McGee, Vice President of Public Accountability

Stuart Buck, Vice President of Research Integrity

Appendix: Detailed Recommendations from the Laura and John Arnold Foundation 

for Improving Government’s Capacity to Build and Use Evidence

Attachment: Proposed Social Spending Innovation Research (SSIR) Initiative: 

Harnessing American Entrepreneurial Talent to Solve Major U.S. Social Problems
736 The Promise of Evidence-Based Policymaking

Appendix

Detailed Recommendations from the Laura and John Arnold Foundation for 

Improving Government’s Capacity to Build and Use Evidence

This document builds upon the recommendations in the Laura and John Arnold 

Foundation’s (LJAF’s) December 22, 2016 letter to the Commission on EvidenceBased Policymaking. Under each of our recommendations, we provide examples of 

projects supported by our grantees or activities we carry out directly that serve as 

proof points that government programs can be transformed to generate higher value 

for taxpayers through use of data and evidence. These examples, which we consider 

building blocks for a future evidence infrastructure, have informed our detailed 

recommendations. 

1. Develop and apply evidence-focused design principles to all government 

programs. 

During the Obama administration, a number of significant outcomes-focused, 

evidence-based program initiatives were launched that created strong financial 

incentives for grantees and providers to use existing evidence of what works and to 

participate in rigorous evaluations to build new knowledge. These included tiered 

evidence grants (e.g., the Department of Education’s Investing in Innovation Fund and 

the Department of Health and Human Services’ Home Visiting program), Pay for 

Success pilots, Performance Partnership Pilots for Disconnected Youth, and waiver 

demonstrations in welfare programs. 

LJAF is committed to helping policymakers and researchers refine these models and 

expand their use in order to improve government effectiveness. As examples of this 

work, members of the foundation are having discussions with stakeholders about 

how to enhance the impact and cost-effectiveness of evidence-based program 

initiatives by using the following approaches: 

 Adopting a standard approach to tiered evidence grantmaking in social 

programs modeled on the successful Small Business Innovation Research (SBIR) 

program. The attachment to this appendix is a short concept paper that 

outlines our proposed approach and provides a template that could be used to 

apply the approach across a range of social spending programs. This program 

model was developed for federal competitive grants with a matching 

requirement to incentivize state and local governments to use funds they 

control (including federal formula funds) for evidence-based approaches. A 

similar model could be used by state governments to allocate funds to 

localities and other grantees. 

 Increasing the use of low-cost randomized controlled trials (RCTs). Low-cost 

RCTs are a powerful new tool for building scientific evidence about “what 
The Promise of Evidence-Based Policymaking 737

works” to address major social problems. Well-conducted RCTs are widely 

regarded as the most credible method of evaluating whether an intervention 

model is effective, but they are often assumed to be too expensive and 

burdensome for practical use in most areas. However, researchers have 

recently shown that in many instances, high-quality RCTs can be conducted at 

a low cost and minimal burden, addressing a key obstacle to their widespread 

use. The low cost is achieved by:

 Embedding random assignment in initiatives that are already being 

implemented as part of usual program operations. RCTs can be embedded 

in many new or ongoing programs, for example, by using a lottery process

(i.e., random assignment) to determine who among those eligible will be 

offered a particular service model (since available funds are often 

insufficient to serve everyone who qualifies).

 Measuring key study outcomes with administrative data that are already 

collected for other purposes (e.g., student test scores on state exams, 

criminal arrest records, and health care expenditures), rather than 

engaging in original—and often expensive—data collection through 

interviews or testing. 

Such studies make it possible now as never before for policy officials to use 

scientific evidence about what works to increase government effectiveness. 

LJAF is funding numerous low-cost RCTs (costing between $50,000 and 

$300,000) that have large samples, strong designs, long-term follow-up, and 

outcome measures of self-evident policy importance. For examples, see: 

http://www.arnoldfoundation.org/wp-content/uploads/Request-forProposals-Low-Cost-RCT-Competition-FINAL-9.30.16.pdf. 

The U.S. Department of Education and National Institutes of Health have also 

recently launched low-cost RCT funding initiatives. 

We believe the Commission’s recommendations should include program design 

principles for social programs that aim to increase the share of resources that are 

allocated to evidence-based practices over time. If the incoming administration 

adopts a diagnostic tool such as the Program Assessment Rating Tool (PART) to 

assess program effectiveness, the tool should also assess a program’s capacity to use 

and build evidence about what works. The following program design principles 

should guide all new legislation, reauthorizations, regulatory reforms, and agency 

administrative practices:

 Focus on outcomes. Define the primary and secondary outcomes that programs 

will be accountable for achieving. Avoid prescriptive requirements for how 

those outcomes should be achieved.

 Use and build rigorous evidence. Ensure that programs have requirements and 

incentives for decision makers at every level of a program to:
738 The Promise of Evidence-Based Policymaking

o Use existing evidence about what is likely to work best to improve 

outcomes for program participants;

o Build new evidence using rigorous and cost-effective research methods 

about what works best for different populations in different contexts;

o Adopt consistently high standards for what constitutes rigorous 

evaluation; and

o Enable measurement and evaluation of long-term outcomes after 

participants are no longer receiving services, with a focus on outcomes 

related to education, employment, criminal justice involvement, and 

health.

 Encourage innovation. Reward entrepreneurial innovation, provided there is 

rigorous testing and learning about which strategies are most effective.

 Leverage state and local funds. Require or incentivize state and local 

governments to allocate an increasing share of the funding they control 

(including federal formula funds) to scale-up evidence-based practices. 

 Consolidate programs while targeting services. Reduce the number of programs 

that are serving similar populations by combining funding streams so that 

program administrators and practitioners have greater flexibility to deliver 

the best mix of services using the most effective strategies. Any restructuring 

should include strong safeguards to ensure that services and benefits are 

targeted to vulnerable populations most in need.

 Improve use of high-quality data. Maximize production of, access to, and use of 

high-quality data. Limit reporting to those data elements that are most 

important for assessing progress, measuring outcomes, and evaluating impact. 

Wherever possible, enable state and local governments and service providers 

to measure progress and evaluate results by linking participant data to 

reliable administrative and survey data held by third parties in secure data 

facilities. 

2. Set an expectation that state and local governments that administer federal 

programs should conduct ongoing data analysis and rigorous evaluation and 

require that they make their administrative data accessible to the federal 

government to facilitate research. Governments should use existing program 

funds for this purpose. 

State and local governments administer hundreds of billions of dollars of social 

program funding every year, most of which is provided by the federal government. 

Through partnerships with strong researchers, state and local governments can use 

data, evidence, and innovative experimentation to learn what works and improve 

program delivery. LJAF-funded projects that demonstrate the types of activities that 

federal dollars could support include the following:

 Established in 2015, the Rhode Island Innovative Policy Lab (RIIPL) at Brown 

University built a linked longitudinal database from Rhode Island’s 

government agencies and private vendors. Researchers are using this 
The Promise of Evidence-Based Policymaking 739

database to inform real-time policy decisions, educate policymakers in their 

pursuit of improving economic and social outcomes, and increase the supply 

of experimentation and rigorous evaluation in Rhode Island. RIIPL is currently 

working with seven agencies on 17 projects that span areas such as juvenile 

recidivism; foster care; supplemental nutritional assistance; Medicaid; 

maternal, infant, and early childhood home visitation; labor training; debtrelated incarceration; tax incentives; the Earned Income Tax Credit; and 

policing. 

 The Government Performance Lab at the Harvard Kennedy School of 

Government (GPL) has been working in jurisdictions across the country to 

facilitate Pay for Success projects, results-driven contracting, and 

performance improvement activities. A complete list of projects is available 

here: http://govlab.hks.harvard.edu/our-projects. 

 The Abdul Latif Jameel Poverty Action Lab at the Massachusetts Institute of 

Technology (J-PAL), through its State and Local Innovation Initiative, is 

supporting randomized policy evaluations at the state level in Pennsylvania

and South Carolina, at the city level in Philadelphia and Rochester, N.Y., and at 

the territory level in Puerto Rico. These evaluations aim to address significant 

social problems facing state and local leaders. For example, these evaluations 

examine approaches to increase employment and economic mobility and to 

find more effective treatments for substance use disorders.

The projects described above have demonstrated cost-effective ways for state and 

local governments to partner with research institutions to solve important problems

that could be carried out on a larger scale without legislative change. We believe the 

Commission should recommend that federal agencies take administrative actions 

that: 

 Set clear expectations for state and local grantees regarding data analysis and 

evaluation. The Office of Management and Budget (OMB) and federal agencies 

should clarify that state and local grantees (1) are expected to analyze data 

and evaluate the impact of their program strategies using rigorous methods; 

(2) are encouraged to partner with strong researchers in academia to carry 

out these activities; and (3) are allowed to use programmatic funds for these 

purposes.

 Ensure accessibility of grantee data. The federal government should require 

that states and localities make key data needed to measure progress and 

outcomes accessible to the federal government for research purposes. The 

federal government should collaborate with states and localities to develop 

standard language for procurements that require contractors to make data 

accessible to state and local governments for program analysis and to the 

federal government for research purposes.
740 The Promise of Evidence-Based Policymaking

When legislative barriers prevent federal agencies from implementing these policies 

through administrative action, Congress should remove the barriers through 

legislation. 

The LJAF-funded projects described above rely on high-quality administrative data to 

conduct rigorous low-cost evaluations. However, these projects are limited by their 

lack of access to national datasets, which would allow them to measure outcomes for 

participants who move across state lines. 

To facilitate the development of similar high-value projects, and to improve projects 

like these, the federal government should require that state and local data generated 

for the administration of federally funded programs be accessible for research 

purposes. Once the data from local and state programs is linked with federally held 

data, useful information can be provided to state and local governments in ways that

will greatly enhance what these governments can learn about their programs.

To be scalable and sustainable, partnerships like those described above, or other 

efforts to improve the research capacity of governments, should not rely entirely on 

philanthropic funding. Instead, government should recognize the value of such efforts 

and provide financial support. The federal government could spur this shift by 

clarifying that (1) state and local governments that administer federal programs are 

expected to analyze data and conduct evaluations to improve program impact and (2) 

programmatic funds may be used to finance data infrastructure and evaluation 

activities. To avoid placing a burden on states and localities without providing 

commensurate resources, the federal government should provide additional funding 

for data sharing activities. 

3. Create a network of government and research institutions that facilitate 

secure linkage of data while protecting individual privacy. LJAF supports a 

number of projects with research institutions that are pioneering innovative 

strategies for linking personally identifiable information while protecting privacy. 

Most of these projects benefit state and local governments that are seeking to link 

data across programs or levels of government in order to answer important policy 

questions. 

 Actionable Intelligence for Social Policy (AISP) is an initiative that focuses on 

the development, use, and innovation of integrated data systems (IDS) for 

policy analysis and program evaluation. The goal is to make it easier for state 

and local governments to establish and evaluate effective programs by linking 

privacy-protected data across agencies and standardizing IDS practices. In 

2009, AISP formed a network of county agencies, city agencies, state agencies, 

and universities around the country to address the lack of national standards 

for IDS. The AISP network now has 13 sites that engage in multi-site research 

projects and share best practices for maintaining and developing an IDS. 
The Promise of Evidence-Based Policymaking 741

 In the summer of 2016, LJAF provided support to Chapin Hall at the University 

of Chicago to release an RFP titled “Using Linked Data to Advance EvidenceBased Policymaking,” which solicited research projects that could benefit from 

securely linking state or local-level data with federal data held at the U.S. 

Census Bureau. The data linking services offered by Chapin Hall could support 

a range of analyses including impact evaluations, long-term follow-up studies 

of prior RCTs, outcome measurement studies, needs assessments and 

descriptive studies, and multi-generation studies. The projects identified

through the RFP include a long-term follow up of welfare demonstration 

projects in the 1990s, an evaluation of a higher education student aid program, 

and an analysis of the effect of birth weight and perinatal services on a 

population in California. None of these projects could have been accomplished 

using un-linked federal or state data alone.

 LJAF provided a planning grant to the Virginia Tech Social and Decision 

Analytics Lab (SDAL)to develop a strategy to increase the use of data analytics

to answer pressing policy problems in small- and medium-sized jurisdictions 

around the country. The grant resulted in a plan to use the national land-grant 

university system, which is comprised of research universities with a public 

service-based mission, and the associated Cooperative Extension Service, 

which has a presence in most counties across the country, to bring together 

local policymakers and researchers supported by the analytics capacity at the 

university. The collaboration will allow different state university systems to 

focus on different issues. For example, land grant universities in Virginia may 

develop analytic capacity around youth development and opioid use and their 

counterparts in Iowa may focus on early childhood development. All of those 

topics require linking data across agencies and levels of government to 

measure relevant outcomes.

In addition to these grant projects, LJAF staff are investigating new privacypreserving technologies employed in the United States and other countries. A number 

of modern cryptographic and statistical techniques are making it possible for 

researchers to study and learn from data while preserving privacy for individual 

records. Indeed, a relatively recent development known as secure multi-party 

computation allows two sources of data to be merged and analyzed while remaining 

completely encrypted the entire time so that even the researchers themselves never 

see an individual’s data. 

In 2015, researchers in Estonia engaged in a large demonstration project in which 

they matched 500,000 education records with 10 million tax records to study 

whether working during college increased the risk that students would fail to 

graduate on time. Thanks to secure multi-party computation, both the tax and 

education datasets remained cryptographically secure throughout the analysis, thus 

allowing the research to take place even though sharing tax data was otherwise illegal 

under Estonian law. In the United States, secure multi-party computation has been 
742 The Promise of Evidence-Based Policymaking

largely confined to defense and intelligence applications funded by DARPA, but could 

prove transformational if extended to other government programs’ data. 

We believe the Commission recommendations should include:

 Create an intergovernmental network of data hubs. The U.S. Census Bureau 

should become a central hub in a federated network of state and local 

governments and other research entities that hold high-value data. The U.S. 

Census Bureau should lead efforts to develop common standards and 

protocols for managing and linking data in ways that protect privacy and 

produce rigorous statistical and evaluation products. 

 Harness new privacy-preserving technologies. Standards and protocols for 

data-linkage should be updated periodically to take advantage of evolving 

technology such as secure multi-party computing, which safeguards the 

privacy and security of data during analysis.

 Finance data-linkage and analysis. The federal government should create 

permanent, predictable funding for data-linkage and analysis activities carried 

out by federal statistical agencies and research data centers. Legislation and 

administrative procedures should clarify that a portion of these costs can be 

paid from appropriations for the agency programs that are benefiting from the 

data analysis. 

4. Strengthen federal agency evaluation and management capacity. A number of 

federal agencies have made significant advances in evaluation capacity in recent 

years, setting examples that are highlighted by Results for America’s What Works 

Index, the Evidence-Based Policymaking Collaborative, and Andy Feldman’s 

GovInnovator podcast. These successful efforts pave the way for the next 

administration to refine these approaches and expand them at other agencies. In 

addition, there are other opportunities to tackle issues that the Obama administration 

has not adequately addressed. We believe the Commission recommendations should 

include:

 Encourage coordinated agency learning agendas. Agencies should implement 

learning agendas in coordination with other actors pursuing knowledge on the 

same subject. For example, learning agendas and evaluation strategies can be 

enhanced through collaboration with philanthropies and academic

researchers. Coordinating investments on shared priorities will ensure that 

separate studies build toward common, policy-relevant aims. Maximizing 

access to and use of administrative data to produce high quality studies at low 

cost is a key component of widespread learning. 

 Create Chief Evaluation Officers across federal agencies. Federal agencies 

should establish or designate Chief Evaluation Officers to lead the 
The Promise of Evidence-Based Policymaking 743

development of learning agendas reflecting important research questions and 

to help agencies improve their effectiveness. Evaluation activities can be 

financed through line-item appropriations or through set-asides, such as those 

at the Department of Labor. Agencies should adopt and continually update 

rigorous standards for their research and evaluation activities, taking into 

account the findings of the National Academy of Sciences Workshop on 

“Principles and Practices for Federal Program Evaluation.”

 Promote purposeful testing of ways to increase savings and cost-effectiveness.

Federal agencies, as well as state and local partners, can improve their return 

on investment by evaluating strategies designed to generate savings and cut 

unnecessary costs. This process should include evaluations that identify what 

activities can be stopped or streamlined without diminishing program impact. 

 Create interagency working groups on cross-cutting evaluation issues. Federal 

agencies that operate similar programs, such as regulatory enforcement and 

federal credit programs, should collaborate to develop evaluation strategies 

and tools that can benefit multiple agencies. These working groups should 

collaborate with outside researchers and philanthropy to leverage nongovernmental expertise and funding to conduct evaluations. 

 Strengthen human capital for evaluation activities. Agencies should strengthen 

their internal evaluation expertise through training of federal executives and 

staff. For example, the United Kingdom has developed competency standards 

related to evidence and evaluation that senior officials are expected to meet, 

and outside organizations such as Nesta offer training based on those 

standards. Federal agencies should create more opportunities for highly 

qualified researchers to work inside government to help agencies design and

conduct evaluations, and clean and produce data sets that can be reused to 

answer multiple questions. To accomplish this, the administration should 

consider establishing an Evidence-Based Policy Fellows program to recruit 

top researchers to work at agencies for one or two years on high-priority 

research projects.

 Reevaluate the Government Performance and Results Modernization Act

(GPRMA). In its current form, Government Performance and Results 

Modernization Act (GPRMA) creates a significant reporting burden for federal 

agencies without generating reliable and meaningful information to evaluate 

and improve programs. To meet federal accountability requirements for 

specific programs, state and local governments and non-profit grantees devote 

substantial resources to reporting data of dubious quality on inputs, outputs, 

and processes without receiving useful information in return. GPRMA should 

be reevaluated to minimize unnecessary reporting by agencies and grantees 

and to maximize the use of high-quality administrative data, leveraging new 

capacities to link data across programs. GPRMA should make state and local 
744 The Promise of Evidence-Based Policymaking

governments central partners in helping to develop streamlined 

accountability systems focused on outcomes, which can help all levels of 

government produce useful data for performance measurement and rigorous 

program evaluation.

 Integrate performance management and evaluation activities to generate 

reliable, actionable information. In too many federal agencies, including OMB, 

performance management and evaluation activities are poorly coordinated. 

This creates confusion for senior agency officials and program managers who 

need useful information about what is working and what should be improved 

or stopped. OMB should improve its internal coordination of these functions 

and call upon agencies to focus performance management activities on 

improving data quality and access; using rigorous evaluation to learn what 

strategies are most effective within and across programs; and fine-tuning 

performance metrics to ensure they are correlated with programmatic impact. 

In recent years, the Department of Labor has been an exemplar of this 

approach. The President’s management agenda, which might include an 

updated Program Assessment Rating Tool, could set standards that encourage 

better coordination of these functions. 

 Implement data-use agreements that allow long-term follow-up studies.

Currently, most data-use agreements for federal evaluations do not provide an 

option to re-use the data for long-term follow-up studies after the initial 

evaluation ends. This makes it very difficult to learn about long-term impacts 

of government policies. Federal agencies should update their standard datause agreements to facilitate long-term follow-up studies. Federal agencies 

should further standardize any individual consent forms such that study 

subjects are routinely asked to consent to the preservation and sharing of data 

(e.g., test scores) for future research purposes, rather than be limited only to 

one particular purpose or particular set of researchers.

 Update the Paperwork Reduction Act (PRA). OMB’s clearance requirements 

under the Paperwork Reduction Act (PRA) create long delays and bottlenecks 

for agencies seeking approval of evaluations and have stymied efforts by 

agencies to increase the number of high-quality studies of important research 

questions. Congress and OMB should work to identify ways to streamline the 

PRA approval process for evaluations when agencies have other means to 

ensure evaluation studies use rigorous designs, address important questions, 

and minimize burden on the public by using administrative and survey data 

when feasible. 

 Research clearinghouses and other resources on evidence-based practices.

Congress and the administration should expand federal research 

clearinghouses and their capacity to deliver information in user-friendly ways 

that help state and local decision makers learn which strategies are most likely 
The Promise of Evidence-Based Policymaking 745

to be effective for different communities. To improve transparency and 

reproducibility of research and evaluation findings, clearinghouses should 

adopt standards requiring researchers to preregister studies and make the 

underlying data available, in de-identified form, to other researchers. 

i Tiered Evidence Grants: Opportunities Exist to Share Lessons from Early Implementation and Inform 

Future Federal Efforts, Government Accountability Office, GAO-16-818, September 2016.

 
746 The Promise of Evidence-Based Policymaking

Proposed Social Spending Innovation Research (SSIR) Initiative:

Harnessing American Entrepreneurial Talent to Solve Major U.S. Social Problems

The SSIR proposal seeks to replicate, in social spending, the great success of the Small Business Innovation 

Research (SBIR) program in technology development.

 The SBIR program funds technology development by entrepreneurial small companies. Under SBIR, 

created by Congress in 1982, 11 federal agencies allocate a small percentage of their annual research and 

development (R&D) budgets, for funding awards to small companies to develop and test innovative new 

technologies. The goal is to reach beyond the usual federal R&D grantees (e.g., universities, large defense 

contractors) to fund a new set of entrepreneurs. The program has spawned breakthrough technologies in 

diverse areas such as computer chip production, commercial satellite communications, and medical 

imaging; and has received consistently favorable reviews in assessments by the National Academy of 

Sciences1 and Government Accountability Office.

2 Congress reauthorized and expanded SBIR with 

overwhelming bipartisan support in 1992, 2000, and 2011. It is now funded at over $2 billion per year.

 The proposed initiative, SSIR, would apply the successful SBIR approach in a different (non-technology) 

field—social spending—as discussed below. A version of SSIR, focused on K-12 education, was recently 

enacted into law as part of the Every Student Succeeds Act of 2015—see attachment, sponsored by 

Senators Orrin Hatch (R-UT), and Michael Bennet (D-CO). The concept is also a key recommendation of 

House Speaker Paul Ryan’s recent blueprint for addressing poverty, opportunity, and economic mobility.3

U.S. social spending critically needs an SBIR-like infusion of entrepreneurial new ideas and rigorous testing, 

because:

 Many activities/strategies (“interventions”) funded by government social programs are found to 

produce weak or no positive effects when rigorously evaluated. When evaluated in rigorous randomized 

controlled trials, social interventions in K-12 education, employment and training, crime prevention, and 

other areas are too often found ineffective or marginally effective. Interventions that produce sizable

effects on important life outcomes do exist, as discussed below, but tend to be the exception. This pattern 

occurs not just in social spending, but in other fields where rigorous evaluations are conducted, such as 

medicine and business.

4

 Meanwhile, the United States has failed to make significant progress in key areas such as:

̵ Poverty: The U.S. poverty rate now stands at 13.5%, and has shown little overall change (whether by 

official or alternative National Academy measures) since the late 1970s.5

̵ K-12 education: Reading and math achievement of 17-year-olds—the end product of our K-12 education 

system—is virtually unchanged over the past 40 years, according to official measures,6 despite a 90% 

increase in public spending per student (adjusted for inflation).7

̵ Well-being of low to moderate income Americans: The average yearly income of the bottom 40% of 

U.S. households, now at $22,500, has changed little since the 1970s.8

 Yet, entrepreneurs in the research, nonprofit, and for-profit sectors have developed a few interventions 

found highly effective in rigorous testing, illustrating what is possible. Examples, evaluated in wellconducted randomized controlled trials, include:
The Promise of Evidence-Based Policymaking 747

̵ Nurse-Family Partnership – a nurse visitation program for low-income, first-time mothers during 

pregnancy and children’s infancy (reduced child abuse/neglect and injuries by 20-50% over 2-15 years, 

compared to the control group).

̵ Per Scholas Job Training – a program for low-income, low-skilled workers that provides training in 

information technology (21⁄2 years after program entry, increased workers’ earnings by 31%, or $5,200 

per year, compared to the control group).

̵ New York City’s Small Schools of Choice – small public high schools created citywide in mostly highpoverty communities to replace large, low-performing high schools (4 years later, produced a 6-10 

percentage point increase in the four-year high school graduation rate, versus the control group).

 Such examples are rare because federal social spending has no systematic mechanism—analogous to 

SBIR—to fund and test innovative field-initiated ideas. Federal agency evaluation funds generally focus on

programs selected for testing by Congress or the agency, rather than initiated by entrepreneurs in the field. 

Agency research funding—such as that of the Institute of Education Sciences and National Institutes of 

Health—supports field-initiated ideas but is primarily focused on academic researchers and rarely funds

entrepreneurial practitioners in nonprofit, for-profit, and state/local government organizations.

SSIR would use a streamlined, three-phase process—modeled on SBIR—to fund the development and rigorous 

testing of innovative social interventions. Specifically:

 SSIR programs would be established at federal agencies using existing federal funds. For example, 

Congress could direct federal agencies to restructure their existing discretionary social programs to 

incorporate the SSIR grantmaking process outlined below. Alternatively, as in SBIR, Congress could direct 

each agency to allocate a small percentage of its discretionary social spending (e.g., 0.5%) to fund a new 

SSIR program at that agency.

 Each agency SSIR program would focus on a broad area (e.g., job training, crime prevention, health care 

delivery), and award grants through a competitive process. Applicants could include nonprofit, for-profit, 

research, or state/local government organizations, with a priority for organizations that obtain a partial 

match of funds from other sources to help ensure the project’s sustainability. Grants would include:

̵ Early-phase grants (e.g., $50,000-$300,000) to fund the development and feasibility testing of an 

intervention which has promising prior research, for the purpose of determining whether the 

intervention can be successfully implemented in real-world settings (e.g., public schools, 

unemployment insurance offices, community health clinics);

̵ Mid-phase grants (e.g., $0.5-3.0 million) to fund implementation and a rigorous evaluation of an 

intervention that has been successfully implemented under an early-phase grant (or other effort 

meeting similar criteria), for the purpose of measuring the intervention’simpact on important outcomes, 

such as employment and earnings, high school graduation, criminal arrests, or health; and

̵ Expansion grants (e.g., $3-7 million) to fund implementation and a rigorous replication evaluation of 

an intervention found to produce sizable, important impacts under a mid-phase grant (or other effort 

meeting similar criteria), for the purposes of delivering the intervention on a larger scale and 

determining whether its sizable impacts can be successfully reproduced and sustained over time.

Conclusion: Modeled on the successful SBIR program, the proposed initiative—SSIR—would infuse U.S. social 

spending with a critically-needed supply of entrepreneurial new ideas, shown in rigorous testing to produce 

important improvements in people’s lives. 
748 The Promise of Evidence-Based Policymaking

SEC. 4611. GRANTS FOR EDUCATION INNOVATION AND RESEARCH 

(a) PROGRAM AUTHORIZED—

(1) IN GENERAL: From funds reserved under section 4601(b)(2)(A),1 the Secretary shall make grants to

eligible entities to enable the eligible entities to—

(A) create, develop, implement, replicate, or take to scale entrepreneurial, evidence-based, fieldinitiated innovations to improve student achievement and attainment for high-need students; and

(B) rigorously evaluate such innovations, in accordance with subsection (di).

(2) DESCRIPTION OF GRANTS: The grants described in paragraph (1) shall include—

(A) early-phase grants to fund the development, implementation, and feasibility testing of a program,

which prior research suggests has promise, for the purpose of determining whether the program can

successfully improve student achievement or attainment for high-need students;

(B) mid-phase grants to fund implementation and a rigorous evaluation of a program that has been

successfully implemented under an early-phase grant described in subparagraph (A) or other effort

meeting similar criteria, for the purpose of measuring the program’s impact and cost effectiveness, if

possible using existing administrative data; and

(C) expansion grants to fund implementation and a rigorous replication evaluation of a program that

has been found to produce sizable, important impacts under a mid-phase grant described in

subparagraph (B) or other effort meeting similar criteria, for the purposes of—

(i) determining whether such impacts can be successfully reproduced and sustained over

time; and

(ii) identifying the conditions in which the program is most effective.

(b) ELIGIBLE ENTITY—In this subpart, the term "eligible entity" means any of the following:

(1) a local educational agency;

(2) a State educational agency;

(3) the Bureau of Indian Education;

(4) a consortium of State educational agencies or local educational agencies;

(5) a nonprofit organization;

(6) a State educational agency, a local educational agency, a consortium described in

paragraph (4), or the Bureau of Indian Education, in partnership with—

(A) a nonprofit organization;

(B) a business;

(C) an educational service agency; or

(D) an institution of higher education.

(c) RURAL AREAS—

(1) IN GENERAL: In awarding grants under subsection (a), the Secretary shall ensure that not less than 25

percent of the funds made available for any fiscal year are awarded for programs that meet both of the

following requirements:

(A) The grantee is—

(i) a local educational agency with an urban-centric district locale code of 32, 33, 41, 42, or

43, as determined by the Secretary;

1 This is $70.5 million in FY 2017-18 and $90.6 million in FY 2019-20.

APPENDIX 
The Promise of Evidence-Based Policymaking 749

(ii) a consortium of such local educational agencies;

(iii) an educational service agency or a nonprofit organization in partnership

with such a local educational agency; or

(iv) a grantee described in clause (i) or (ii) in partnership with a State educational agency.

(B) A majority of the schools to be served by the program are designated with a locale code of 32,

33, 41, 42, or 43, or a combination of such codes, as determined by the Secretary.

(2) EXCEPTION: Notwithstanding paragraph (1), the Secretary shall reduce the amount of funds made

available under such paragraph if the Secretary does not receive a sufficient number of applications of

sufficient quality.

(d) MATCHING FUNDS—In order to receive a grant under subsection (a), an eligible entity shall demonstrate that the

eligible entity will provide matching funds, in cash or through in-kind contributions, from Federal, State, local, or

private sources in an amount equal to 10 percent of the funds provided under such grant, except that the Secretary

may waive the matching funds requirement, on a case-by-case basis, upon a showing of exceptional circumstances,

such as:

(1) the difficulty of raising matching funds for a program to serve a rural area;

(2) the difficulty of raising matching funds in areas with a concentration of local educational agencies or

schools with a high percentage of students aged 5 through 17—

(A) who are in poverty, as counted in the most recent census data approved by the Secretary;

(B) who are eligible for a free or reduced price lunch under the Richard B. Russell National School

Lunch Act (42 U.S.C. 1751 et seq.);

(C) whose families receive assistance under the State program funded under part A of title IV of the

Social Security Act (42 U.S.C. 601 et seq.); or

(D) who are eligible to receive medical assistance under the Medicaid program; and

(3) the difficulty of raising funds on tribal land.

(di) EVALUATION—Each recipient of a grant under this section shall conduct an independent evaluation of the

effectiveness of the program carried out under such grant.

(dii) TECHNICAL ASSISTANCE—The Secretary may reserve not more than 5 percent of the funds appropriated

under section 4601(b)(2)(A) for each fiscal year to:

(1) provide technical assistance for eligibility entities, which may include pre-application workshops, webbased seminars, and evaluation support; and

(2) to disseminate best practices.
750 The Promise of Evidence-Based Policymaking

References

1 An Assessment of the Small Business Innovation Research Program, Charles W. Wessner, Editor, Committee on Capitalizing 

on Science, Technology, and Innovation, National Research Council, National Academies, 2008. 

2 The Government Accountability Office has conducted numerous review of SBIR since the program’s establishment in 1982, 

the results of which are summarized in Federal Research: Observations on the Small Business Innovation Research Program, 

Government Accountability Office, GAO-05-861T, June 2005.

3 A Better Way: Our Vision for a Confident America – Poverty, Opportunity, and Upward Mobility, recommendation to 

“Incorporate Tiered-Funding Structures to Build Evidence,” abetterway.speaker.gov, June 7, 2016, p. 20. 

4

Illustrative examples include:

 Education: Of the 90 interventions evaluated in randomized controlled trials (RCTs) commissioned by the Institute of 

Education Sciences (IES) since 2002, approximately 90% were found to have weak or no positive effects. (Coalition 

for Evidence-Based Policy, Randomized Controlled Trials Commissioned by the Institute of Education Sciences Since 

2002: How Many Found Positive Versus Weak or No Effects, July 2013, linked here.)

 Employment/training: In Department of Labor-commissioned RCTs that have reported results since 1992, about 75% 

of tested interventions were found to have found weak or no positive effects. (This is based on a count of results from 

the Department of Labor RCTs that have reported results since 1992, as identified through the Department’s research 

database, linked here.)

 Medicine: Reviews have found that 50-80% of positive results in initial clinical studies are overturned in subsequent, 

more definitive RCTs. (John P. A. Ioannidis, “Contradicted and Initially Stronger Effects in Highly Cited Clinical 

Research,” Journal of the American Medical Association, vol. 294, no. 2, July 13, 2005, pp. 218-228. Mohammad I. 

Zia, Lillian L. Siu, Greg R. Pond, and Eric X. Chen, “Comparison of Outcomes of Phase II Studies and Subsequent 

Randomized Control Studies Using Identical Chemotherapeutic Regimens,” Journal of Clinical Oncology, vol. 23, no. 

28, October 1, 2005, pp. 6982-6991. John K. Chan et. al., “Analysis of Phase II Studies on Targeted Agents and 

Subsequent Phase III Trials: What Are the Predictors for Success,” Journal of Clinical Oncology, vol. 26, no. 9, March 

20, 2008. Michael L. Maitland, Christine Hudoba, Kelly L. Snider, and Mark J. Ratain, “Analysis of the Yield of Phase II 

Combination Therapy Trials in Medical Oncology,” Clinical Cancer Research, vol. 16, no. 21, November 2010, pp. 

5296-5302. Jens Minnerup, Heike Wersching, Matthias Schilling, and Wolf Rüdiger Schäbitz, “Analysis of early phase 

and subsequent phase III stroke studies of neuroprotectants: outcomes and predictors for success,” Experimental & 

Translational Stroke Medicine, vol. 6, no. 2, 2014.)

 Business: Of 13,000 RCTs of new products/strategies conducted by Google and Microsoft, 80-90% have reportedly 

found no significant effects. (Jim Manzi, Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, 

and Society, Perseus Books Group, New York, 2012, pp. 128 and 142. Jim Manzi, Science, Knowledge, and Freedom, 

presentation at Harvard University’s Program on Constitutional Government, December 2012, linked here.)

5 Proctor, Bernadette D., Jessica L. Semega, and Melissa A. Kollar, U.S. Census Bureau, Current Population Reports, P60-

256(RV), Income and Poverty in the United States: 2015, U.S. Government Printing Office, Washington, DC, 2016. U.S. Census 

Bureau, Official and National Academy of Sciences (NAS) Based Poverty Rates: 1999 to 2011, 2012. Kathleen Short, U.S. 

Census Bureau, HHES Division, Estimating Resources for Poverty Measurement, 1993 – 2003, 2005. Panel on Poverty and 

Family Assistance, National Academy of Sciences, Measuring Poverty: A New Approach, 1995, pp. 31-36. Christopher Wimer, 

Liana Fox, Irv Garfinkel, Neeraj Kaushal, and Jane Waldfogel, Trends in Poverty with an Anchored Supplemental Poverty 

Measure, December 2013.

6 The Nation’s Report Card: Trends in Academic Progress 2012, NCES 2013-456, National Center for Education Statistics, 

Institute of Education Sciences, U.S. Department of Education, 2013.

 
The Promise of Evidence-Based Policymaking 751

 7 Cornman, S.Q., and Zhou, L. Revenues and Expenditures for Public Elementary and Secondary Education: School Year 2013-

14 (Fiscal Year 2014) (NCES 2016-301). U.S. Department of Education. Washington, DC: National Center for Education 

Statistics, 2016. Richard H. Barr, Revenues and Expenditures for Public Elementary and Secondary Education, 1973-74 (NCES76-140). U.S. Department of Health, Education & Welfare, National Institute of Education. Washington, DC: National Center 

for Education Statistics, 1976.

8 U.S. Census Bureau, Current Population Reports, 2016, op. cit., no. 5. This refers to inflation-adjusted income. It includes 

income from the economy (such as earnings) but not government transfers (such as Food Stamps). However, the evidence 

suggests that the overall story of income stagnation for the bottom 40% of households changes little even when one adjusts 

income for government transfers and other items that affect household living standards. Specifically, the Census Bureau’s 

alternative, National Academy of Sciences-based poverty measures make adjustments for government transfers, as well as 

factors such as state and local taxes, work expenses such as child care, out-of-pocket medical expenses, and geographic 

differences in housing costs. These adjustments change the poverty rate in any given year, as well as the composition of 

those in poverty, but do not change the overall trend in the poverty rate over time – i.e., little overall progress since the 

late 1970s. (The relevant citations are in endnote 5.) Although the National Academy-based poverty measures only apply 

to a subset of the bottom 40% of U.S. households, their corroboration of no meaningful improvement for that key subset 

suggest that similar findings would be obtained for the larger group.
752 The Promise of Evidence-Based Policymaking

2

AAG Mission

Mission: Provide analytical services for the Army’s senior leadership and

innovative application and data integration across the Enterprise.

Scope:

a. Provides problem solving capabilities that involve massive enterprise data 

integration and analysis coupled with the most advanced information 

technology (IT) solutions.

b. Provides research and analysis facilitation for Army organizations and other 

DoD organizations when approved by DUSA – network Army experts, their 

processes, data, and analysis tools within a secure platform environment.

Resources:

a. 16 Civilians, 3 Military Officers, 82 Contractors

b. Research Facilitation Laboratory (RFL), Medical Analysis Team (MAT)

c. Partners: HRC, CIO-G6, G2, INSCOM, CECOM, OTSG, USAPHC, AMC, JMC,

2RCC, ANC, DITMAC, NPS, TRAC, FNMOC, DMDC

1

March 2017

daniel.c.Jensen.civ@mail.mil

Army Analytics Group (AAG)
The Promise of Evidence-Based Policymaking 753

3

Person-Event Data Environment

PDE is an environment 

which links People, Data, 

and Tools for problem 

solving.

The Data in PDE (over 600 data assets and growing) Many Tools available:

Using a Governance Portal people

create projects, request data from a data 

catalog, get linked up to tools & 

connected to data for research and 

quick study projects – Done Right and 

Right Now!

PDE ensures Human Subject Protection 

Regulations, and all PII/PHI rules are 

followed and audited. 

PDE is an Army operated system designed by 

the Army, Navy, DMDC, and OSD P&R.

PDE-Apps: take the research to end users

Human Capital Big Data

4

Human Capital Big Data (HCBD) Initiative:

• Enterprise supervision of the HCBD analysis platform (PDE) and governance framework 

to ensure ethical and legal use, security and interpretation of human capital data

• Consistent analytical framework (descriptive statistics, policy analysis, and research) links 

data to strategic outcomes such as readiness and senior leader priorities

• More research and answers per dollar - 40-60% better utilization through automation of 

data delivery to analysis & decision making

Enabling Factors:

• Omni-bus Use Agreements

• Streamline Governance Workflows

• Enterprise Approach to Data Knowledge Sharing & Quality Reporting

• Secure Ecosystem
754 The Promise of Evidence-Based Policymaking

PDE-Enclaves

PDE

Staging

PDE 

Analysis

The Insider Threat Stress Rating Study will be conducted in the Person-Event 

Data Environment (PDE) – Development by a partnership between DMDC, 

Navy and Army.

PDE consists of 2 separate enclaves with separate teams:

PDE Staging:

Extract, Load, and Transformation specialist at 

location A.

Receive and load data and run transformation 

scripts to create double encoded de-identified 

data for the PDE analysis enclave.

Two Person Control – CAC card and 2 people 

required at all times to login together and work.

PDE data loaders sign an Acceptable Use 

Policy (AUP) form that outlines acceptable 

behavior while using the PDE Staging enclave.

PDE Analysis:

Researchers at location B use analytic tools to 

run analysis on the double-encoded deidentified data.

Researchers have no access to encoding keys

Researchers sign a PDE Acceptable Use 

Policy form (AUP) that states they will never 

attempt to link data to identify a person.

Researchers can only work within PDE

There is a multi-step process to move analysis 

reports out of PDE.

6

Backup Charts

5
The Promise of Evidence-Based Policymaking 755

PDE Staging - Transforming

PDE Transformation Activities in PDE Staging Enclave:

PDE Transformations:

Run Scripts to change location related data(UIC, Parent UIC, ZIP Code) into sequence or grouping 

codes. 

Change Rank to Rank-Group and GS-Grade-Step to GS-Group and Birthday to Age-Group

Run transport script to send data to PDE analysis replacing PDE-ID with a random STUDY-ID. The

Study encoding key is stored under two person control and can not be accessed by the researchers.

Run scripts on free-text to transform to binary codes – free text is not allowed in PDE analysis

Study 

Encoding 

Key

PDE

Staging

PDE 

Analysis

8

PDE Staging- Loading

PDE

Staging

PDE Loading Activities in PDE Staging Enclave:

PDE Loading:

Data from sources is loaded into PDE Staging using scrubber programs on an SSH server.

Scrubbers remove SSNs and replace with 12 digit PDE-ID code.

The Encoding Key (SSN to PDE-ID) is stored in separate database under 2 person control

Run scripts to identify unique columns, remove name columns if present

Remove DEERS Person-ID, Postal Addresses, Telephone Numbers, Fax Numbers, Med Record 

Numbers, Health Plan Numbers, Account Numbers, Device Identifiers, URLs, IP addresses, 

Biometric Data, Photos, weapons identifiers

Personnel 

Security Data SSH

SSN 

Encoding 

Key

7
756 The Promise of Evidence-Based Policymaking

PDE Free Text Processing

Free Text Transformations

Free Text can be converted to a binary code.

Presence of 

words and 

phrases 

“alone”, 

“loner”, 

“keeps to 

her/himself”

Presence of 

words and 

phrases 

“social”, 

“outgoing”, 

“lots of 

friends”

Matrix of binary coded variables

STUDY-ID Variable jan06 feb06 mar06 apr06 may06 jun06 jul06

1000001 Is Male 1 1 1 1 1 1 1

1000001 Is Female 0 0 0 0 0 0 0

1000001 Is Married 0 0 1 1 1 1 1

1000001 Is Age Group4 0 0 0 0 0 0 0

1000001 Is Age Group5 1 1 1 0 0 0 0

1000001 Is Age Group6 0 0 0 1 1 1 1

1000001 Is Potential Lone/Sad 1 1 0 0 0 0 0

1000001 Is Potential Outgoing/Happy 0 0 1 1 1 1 1

1000001 Has Dependants 0 0 0 0 0 1 1

1000001 Has GED 0 0 0 0 0 0 0

1000001 Graduated High School 1 1 1 1 1 1 1

1000001 Continued Education 0 0 0 1 1 1 1

1000001 Security Incidents 0 0 0 0 0 0 0

1000001 Deployed 0 0 0 0 0 0 0

1000001 Transition UIC 0 0 1 0 0 0 0

10

PDE Analysis

PDE Research Activities in PDE Analysis Enclave:

Research Teams:

CAC Login to a virtual Desktop inside PDE Analysis

Use analysis tools to conduct research (SAS, SPSS, R, Risk Rating Tool configuration)

Can not save to local workstations or drives

Cut and Paste has been disabled

Can place analysis results in a drop-directory that is then reviewed by AAG HPA for approval to 

transfer out of the PDE analysis enclave.

PDE 

Analysis

Virtual 

Desktop Researchers

9
The Promise of Evidence-Based Policymaking 757

2 Serving Those Who Serve Our Country

• What is DMDC?

• A solution provider for all things people (enterprise)

• Why us?

• We have the Person Data Repository (PDR) and Local 

Population Databases

• We know the data

• What makes us stand out?

• We care about the people, their benefits, and their data

• We deliver quality products on time and at lower cost than 

most because we are highly leveraged

• We are passionate about our customers’ business and 

making them successful

• We continuously innovate

Defense Manpower Data Center 

Overview

Presented by:

Mark Breckenridge

Deputy Director, DMDC
758 The Promise of Evidence-Based Policymaking

4 Serving Those Who Serve Our Country

• Personal service employees, visitors, and delivery vendors at military bases*

• Contract personnel in Afghanistan and Iraq

• Persons evacuated during crisis

Data Repositories

DMDC data includes the Person Data Repository (PDR), Local Population 

Databases and several flat files

The PDR holds information on 50 million people

•Wounded, ill and injured

•Disabled American Veterans

•DoD and Non-DoD Civilians

•Contractors & Non-Appropriated 

Funded

•7 Uniformed Services

•Retirees

•Medal of Honor

•ROTC

•Foreign Military

•Family Members

•Veterans

•Patients

Local Population Databases hold an additional 2 million people

The PDR includes: 

3 Serving Those Who Serve Our Country

D

H

R

A

A Global Organization

Office Locations

• DoD Center Monterey Bay, CA

• Metro Washington, DC 

• Boyers, PA

• San Antonio, TX

• Germany

• South Korea

• SW Asia

• Fort Knox, KY

• Dayton, OH

Personnel

• 444 DoD Civilians

• 1450+ Contractors

• 10 Service Component Liaisons

Director

Defense Human 

Resource Activity

Under Secretary of Defense

(Personnel & Readiness)

Michael 

Sorrento

Director

Mark Breckenridge

Deputy

DMDC Organizational Structure

Budget FY16

• O&M Direct: $358,000,000

• Procurement & R&D Direct: $18,200,000
The Promise of Evidence-Based Policymaking 759

6 Serving Those Who Serve Our Country

The Persistent Question

“How do we leverage what we already have to 

enhance capability in such a way that the value 

offered is greater than the costs incurred?”

5 Serving Those Who Serve Our Country

Conceptual Data Model

4854 data elements 

plus 2096 data 

elements not yet 

mapped.

Person 319

Person Identifier158

Person Identifier 

History 4

Security 17

Biometric 69

Person Entitlements 15

Activation 23

Affiliation634

Application Control 1478

Application Roles 142

Citizenship 23

Civilian Employment 

Occupation 11

Deployment 57

Deployment 

Location 26

Education Benefits 147

Family Advocacy38

Family Affiliation200

ID Card 99

Language 104

Life Insurance 99

Medical 

Benefits 143

Medical Enrollment 388

Medical Services25

Occupation 22

Organization137

Organization Billet90

Organization 

Contact 156

Other Benefits 24

Other Government 

Programs 37

Other Health 

Insurance 91

Pay 583

Person Contact197

Personnel Security 233

Primary Care 

Manager 138

Purchase Card Bank 

Account 160

Training 16

System Transaction 

Data 1182

Work Contact 33

Wounded Warrior/

LOD 70

XOR

XOR

XOR

XOR

XOR
760 The Promise of Evidence-Based Policymaking

8 Serving Those Who Serve Our Country

Data Requests

IC

LE

35% of data requests are Law Enforcement 

and Intelligence Community

DMDC provides operational and research support for over 35 federal agencies.

DMDC handles 

about 10,000 data 

requests annually.

• Identification

• Physical Location

• Skills Profile

• Affiliation Status

• Biometrics

• Contact Info

7 Serving Those Who Serve Our Country

Who receives DMDC’s products and services?

Customers

CIO

ABAC

CAC

Identity 

for Enterprise email

Mobility 

SIPR token (maybe) 

IA Workforce

AT&L

Purchase Card

Contracts/Grants

SPOT

Acquisition

workforce

USD(I)

JPAS/DISS

IMESA

CE

Security research

Security workforce

SWFT

Joint Staff

GFM-DI support

Joint Duty Assignments

NTS

JPERSTAT

COCOM Personnel Data

Uniformed Services (7)

(including retirees/

family members)

Benefits (letters/call center)

ID cards/RAPIDS

Data

P&R

ID cards

Transition Assistance

Personnel Accountability

Benefits (MHS, Tricare, 

SGLI, Education , 

Commissary, Exchange)

Funeral Honors

SCRA

ASVAB

DLPT

Personnel Surveys

DRRS to come

DTS (Legacy & Next Gen)

PDE

Law Enforcement/IC

Physical Security system

Vietnam Wall names

Personnel research

ECMRA

HSPD-12

TSA

Pre-check

VA

Benefits

Data sharing

DS-Logon (self service)

Identity

e-Benefits portal portlets

State VAs

HHS

ACA/MEC

Medicare data sharing

State benefits data sharing

Labor

Data Sharing

Occupational 

Matches

IRS, Census Bureau, SSA

Data

Data matching

Comptroller, OMB, CBO, Congress, RAND, CNA, IDA, PA

Data requests
The Promise of Evidence-Based Policymaking 761

10 Serving Those Who Serve Our Country

• DMDC is only one piece – We collaborate with many other 

organizations within the Federal Government to ensure 

that the mission needs are met.

• Collaborating with other agencies and industry partners to 

work toward interoperable capabilities and innovative 

solutions

Partnership and Teaming

Success through teamwork

• Serving our customers, helping them do their jobs more effectively 

and more efficiently

The heavy lifting has been done

How can we help?

9 Serving Those Who Serve Our Country

Call Center/Beneficiary Support

• Self-service – milConnect – CAC or DS-logon

• 85,000 calls per month overall / 60,000 calls per month 

beneficiary

• 7.1M letters annually – moving to email

• Real-time authentication for pharmacy scripts (wholesale, retail, 

military, commercial)

o Sub-second response

• Customer Care Teams for exceptional concerns/issues
762 The Promise of Evidence-Based Policymaking

“fpsyg-04-00934” — 2013/12/11 — 17:27 — page 1 — #1

TECHNOLOGY REPORT ARTICLE

published: 13 December 2013

doi: 10.3389/fpsyg.2013.00934

The Person-Event Data Environment: leveraging big data for

studies of psychological strengths in soldiers

Loryana L. Vie1,2 *, Kevin N. Griffith2 , Lawrence M. Scheier1,2 , Paul B. Lester 2 and Martin E. P. Seligman1

1 Positive Psychology Center, University of Pennsylvania, Philadelphia, PA, USA

2 Research Facilitation Team, Army Analytics Group, Monterey, CA, USA

Edited by:

Steven E. Mock, University of

Waterloo, Canada

*Correspondence:

Loryana L. Vie, Positive Psychology

Center, c/o 20 Ryan Ranch Road,

Suite 290, Monterey, CA 93940, USA

e-mail: lvie@sas.upenn.edu

The Department of Defense (DoD) strives to efficiently manage the large volumes of

administrative data collected and repurpose this information for research and analyses

with policy implications. This need is especially present in the United States Army, which

maintains numerous electronic databases with information on more than one million ActiveDuty, Reserve, and National Guard soldiers, their family members, and Army civilian

employees. The accumulation of vast amounts of digitized health, military service, and

demographic data thus approaches, and may even exceed, traditional benchmarks for Big

Data. Given the challenges of disseminating sensitive personal and health information,

the Person-Event Data Environment (PDE) was created to unify disparate Army and DoD

databases in a secure cloud-based enclave. This electronic repository serves the ultimate

goal of achieving cost efficiencies in psychological and healthcare studies and provides a

platform for collaboration among diverse scientists. This paper provides an overview of the

uses of the PDE to perform command surveillance and policy analysis for Army leadership.

The paper highlights the confluence of both economic and behavioral science perspectives

elucidating empirically-based studies examining relations between psychological assets,

health, and healthcare utilization. Specific examples explore the role of psychological assets

in major cost drivers such as medical expenditures both during deployment and stateside,

drug use, attrition from basic training, and low reenlistment rates. Through creation of

the PDE, the Army and scientific community can now capitalize on the vast amounts of

personnel, financial, medical, training and education, deployment, and security systems

that influence Army-wide policies and procedures.

Keywords: big data, psychological strengths, cost analysis, healthcare utilization, personnel data

As the quantity of data collected on members of the Army community (soldiers, family members, and civilian employees) continues

to increase, researchers and information technology professionals are faced with a daunting task; effective management and

analysis of increasingly larger pools of information with potentially stagnant or dwindling resource levels. Since 2006, the Army

has incrementally developed the Person-Event Data Environment

(PDE), a secure, collaborative research environment, to warehouse

and study health, military service, and demographic information that is regularly collected on Army Active-Duty, Reserve, and

National Guard soldiers, their family members, and Army civilian

employees. This unique resource enables vetted researchers worldwide to mine this trove of data with a scientific mindset that can

answer questions benefitting both the Army and wider scientific

community.

This paper examines three aspects of the PDE. The first section

focuses on the operating environment that gave rise to the PDE,

primarily emphasizing challenges researchers have faced when

working with sensitive but unclassified Department of Defense

(DoD) data. The second aspect focuses on the structural design

and operational capabilities of the PDE. This discussion includes

the PDE’s physical and logical structure, the data contained within,

and the statistical tools that are made available to researchers.

Special attention is given to data sources that are most applicable

to studies of psychological functioning as well as soldier outcomes

that have important cost implications for the Army. The third

section focuses on the types of research that may be conducted

within the PDE. Where relevant, the authors have cited active

research projects and multi-site collaborations. The paper concludes with a discussion of how the PDE can inform both science

and policy.

OPERATING ENVIRONMENT

The PDE was created to address several challenges that researchers

face when working with DoD health, military service, and demographic data. These challenges include centralization of disparate

data sources into an electronic repository, providing effective data

security, and making the data accessible to researchers worldwide.

The subsections below address each challenge in turn and how

they have influenced the PDE’s development.

BIG DATA OVERVIEW

The DoD digitizes an increasing amount of soldier information

every day. Within the DoD, the Defense Manpower Data Center warehouses the largest archive of soldier performance data

covering more than 43 million soldiers, veterans, their family

www.frontiersin.org December 2013 | Volume 4 | Article 934 | 1
The Promise of Evidence-Based Policymaking 763

“fpsyg-04-00934” — 2013/12/11 — 17:27 — page 2 — #2

Vie et al. Army, big data, soldier fitness

members, and DoD civilian personnel (Defense Manpower Data

Center Overview, 2012). The scope and size of these records are

both steadily increasing over time as additional data elements are

added and tracked. The accumulation of trillions of cells of information approaches, and may even exceed, traditional benchmarks

for Big Data.

CHALLENGES WITH DATA ACCESS

The sheer existence of such a repository does not necessarily

imply that the data are accessible. The Army faces the same

challenges that many large organizations face – stove-piped data

collected for a specific purpose but not easily repurposed; specific policies that intentionally protect privacy, restrict access,

and specify uses; and hesitancy to form collaborative relationships with outsiders (Landsbergen and Wolken, 2002). With this

in mind, the burden falls on individual researchers to formulate

data use agreements with each data source provider. This is, at

best, a tedious and lengthy process and in other instances, this

can represent an insurmountable barrier to research, particularly if a data asset provider resists sharing their data. The PDE

aids in this effort by making as much data as possible available

without the need for project-specific data use agreements. Furthermore, data providers are becoming increasingly familiar with

the PDE and previously vetted, standardized data use agreement

contract language may be used to help expedite the data request

process.

SECURITY OF DATA

The accumulation of large pools of data provides countless

research possibilities and necessitates proper governance to prevent abuses and privacy violations (United States Government

Accountability Office, 2008). For example, one can easily imagine

a scenario in which a data-mining application classifies individuals on some basis of risk, grouping them into unique clusters,

for instance, based on higher risk for suicide, drug use, violent crime, or some other sensitive objective criteria. Although

individuals may be assigned into these higher risk groups, the

assignment method will include a number of false positives due

to the low base-rate of these outcomes. An accidental or intentional release of these classification results could have devastating

effects on an individual’s career or social standing even though no

transgression of Army regulations has been (or may ever be) committed. The PDE incorporates several security and governance

features to maximize data security and prevent harm to study

subjects.

UNDER THE HOOD

The PDE is a secure, cloud-based research environment that provides data and software as a service. This section briefly describes

the PDE system architecture, continues with a discussion of

features that enhance privacy and security, and closes with an

overview of the data and tools that are currently contained within

the PDE.

SYSTEM AND DATA ARCHITECTURE

Figure 1 illustrates the separate security and analysis enclaves

within the PDE. Only PDE administrative staff has access to the

PDE-Security Enclave, a back-end server that houses the full data

resource holdings. Data is not transferred from the PDE-Security

Enclave until researchers submit and receive Federal approval for

a study. At this time, data (acquired from a military data source

provider) is then pushed to the PDE-Analysis Enclave, which

researchers can access through a remote, secure connection. The

architecture detailed in Figure 1 incorporates numerous security

and privacy features to protect the data, and these features are

described below.

SECURITY AND PRIVACY FEATURES

The traditional academic research model requires agencies to provide data to researchers. This creates duplication as many “ivory

tower” researchers tend to work independently without drawing

on shared institutional and personnel resources, let alone common data archives. Alternatively, the PDE brings researchers to the

data. This is achieved through Citrix XenDesktop software, which

allows users to securely access the data and software through an

Advanced Encryption Standard 256-bit connection. Researchers

are required to download and install a free copy of Citrix Receiver

in order to use this service.

Data loaded into the PDE may not be extracted in any shape

or form by researchers. Physical hardware and software limitations are in place to prevent breach of security protocols (i.e., data

loss), such as no inter- or intra-net connections and no ability

to copy and paste from the PDE to a user’s desktop. Users may

request extraction of certain allowable non-data file types (e.g.,

Microsoft Word® documents), at which point PDE staff will conduct a compliance review to ensure the requested files do not

contain personally identifiable or protected health information. A

similar vetting process is in place for uploading program or syntax

files into the PDE.

Personally identifiable and protected health information is

securely transformed in order to minimize the risk of identifying individuals. Social security numbers (used to merge soldiers

across time and data sources) are replaced with a random string

of 12 digits that are unique to each individual and to each study.

The algorithm “key” mapping this transformation is destroyed

at the conclusion of the study. In addition to replacing social

security numbers with a scrambled identifier, additional transformations of other personal identifiers are performed. This further

reduces the risk of an individual being identified by an analyst, while maintaining enough information for standard analysis

and reporting of aggregate data. For example, ranks and pay

grades are condensed into groups, in some cases Julian dates are

converted into YYYYMM format (or recorded only as YYYY),

and unit identification codes are also scrambled. Figure 2 diagrams the PDE procedure for transforming unit identification

codes.

REGULATORY FEATURES

Regulatory and compliance oversight of the PDE is provided

by the Army Human Research Protections Office, the Army’s

in-house equivalent of an “Institutional Review Board.” The

Army Human Research Protections Office provides assurances

that cover protection of the rights, welfare, and well-being for

human subject research conducted or supported by the U.S. Army.

Frontiers in Psychology | Developmental Psychology December 2013 | Volume 4 | Article 934 | 2
764 The Promise of Evidence-Based Policymaking

“fpsyg-04-00934” — 2013/12/11 — 17:27 — page 3 — #3

Vie et al. Army, big data, soldier fitness

FIGURE 1 | PDE system architecture.

These assurances are guided by the ethical principles set forth

in the Belmont Report (Ethical Principles and Guidelines for

the Protection of Human Subjects of Research) and the minimum standards set by the Department of Health and Human

Services, Office for Human Research Protection regulations (45

CFR 46). In addition, Army Human Research Protections Office

staff members conduct periodic assessments of current research

projects to monitor compliance. More extensive compliance assurances are required for use of personally identifiable and protected

health information involving medical data and covered under the

standard Privacy Act regulations (Pub. L. No. 93-579) and subsequent amendments (USC Sec. 552a, Title 5, Part I, Chapter 5,

Subchapter II). These assurances are further augmented by the

Army human subjects policy provided in DoD 6025.18-R (DoD

Health Information Privacy Regulation). Additionally, PDE staff

members facilitate research by conducting “first-pass” reviews of

proposals and assisting researchers with compliance questions and

documentation.

www.frontiersin.org December 2013 | Volume 4 | Article 934 | 3
The Promise of Evidence-Based Policymaking 765

“fpsyg-04-00934” — 2013/12/11 — 17:27 — page 4 — #4

Vie et al. Army, big data, soldier fitness

FIGURE 2 | Unit identification code (UIC) transformation.

STATISTICAL TOOLS

The PDE remote desktop contains a suite of statistical, computational, and word processing tools that are made available to

researchers. Currently these include Stata®, SPSS®, Mplus®, SAS®

Enterprise Guide®, ToadTM for Oracle®, Qlikview®, R, and the

Microsoft Office® suite. Thus, in addition to extensive statistical modeling of Army data, analysis findings can be compiled

for publication (peer-review articles or technical reports) entirely

within the PDE. Some software packages have a limited number of “seat” licenses available, while others such as the freely

available software program, R, allow unlimited use. Additional

researcher-provided software and licenses may be loaded into the

PDE if the software has been issued a Certificate of Networthiness

from the Army Networthiness Program. The certificate procedures are designed to protect government information systems

from security threats posed by commercial and other types of

enterprise software. The Army’s current approved software list is

not available on the web, but is publicly available through the PDE

staff.

DATA AVAILABILITY

The PDE’s data holdings are large and continually growing as new

data assets are loaded. The holdings, currently totaling more than

six terabytes (i.e., 6,000 gigabytes), are expected to double over

the coming months. Currently, the PDE contains Army data on

corrections and legal issues, physical fitness tests, military service

information, deployments, demographics, training records, health

assessments, medical visits, medical evacuations, injuries, deaths,

psychological functioning, accessions, recruiting waivers, pay, and

educational attainment. In the ensuing discussion we highlight

datasets that can help illuminate the role of psychological assets

in major cost drivers such as medical expenditures during deployment and stateside, drug use, attrition from basic training, and

low reenlistment rates.

The Global Assessment Tool (GAT) is a primary tool used to

assess psychosocial functioning in soldiers. Launched Army-wide

in 2009, the GAT is a computerized, self-report questionnaire taken

annually by all soldiers as part of the Comprehensive Soldier and

Family Fitness (CSF2) program (Fravell et al., 2011). Soldiers may

take the GAT up to four times a year, if desired. Upon completing

the GAT, soldiers are provided normative feedback (based on a

peer equivalent comparison group) on their relative strengths and

weaknesses across the various psychosocial fitness dimensions.

Then, as part of the CSF2 program, soldiers receive personalized recommendationsfor web-based trainings designed tofurther

develop their psychological fitness. Since its inception, the GAT

has been taken over 3.2 million times by over 1.3 million unique

individuals (Office of CSF2, personal communication, August 26,

2013). The GAT assesses a range of psychological strengths, including self-management and coping skills, flexible thinking, positive

affect, meaning, optimism, character strengths, social support, and

social engagement, to name a few (Peterson et al., 2011).

Medical expenditures are major drivers of avoidable personnel costs for the U.S. Army. Tricare Management Activity is

the health insurance provider for Active-Duty, National Guard,

and Reserve soldiers, their families, survivors, and other eligible

individuals. Tricare’s data holdings represent the largest collection of electronic medical records for the military. The Medical

Data Repository is a Tricare-managed database housing electronic

health records for all aspects of the medical treatment process

conducted stateside. The repository includes medical encounters

(e.g., physician office visits), surgical procedures, prescription use,

and illness diagnoses (ICD-9 codes), to list a few examples. In

addition, the repository houses a number of cost variables that

are familiar to health economists, such as relative value units and

medical expense performance and reporting system codes. Tricare’s Theater Medical Data Store maintains similar health records

for services rendered while soldiers are deployed, although cost

variables are not captured (but can be computed from standard

economic healthcare valuation tables). Additionally, the quality and timeliness of the Theater Medical Data Store records

varies as a function of the deployment environment in which

the medical services are provided. For example, some medical services may be conducted informally during deployments

and, as a result, are never formally documented in the data

system.

Drug and alcohol abuse are also cost drivers for the Army.

Costs due to substance use are not limited to direct medical

care; costs are also incurred due to disciplinary action, outpatient

treatment and prevention programs such as the Army Substance

Frontiers in Psychology | Developmental Psychology December 2013 | Volume 4 | Article 934 | 4
766 The Promise of Evidence-Based Policymaking

“fpsyg-04-00934” — 2013/12/11 — 17:27 — page 5 — #5

Vie et al. Army, big data, soldier fitness

Abuse Program, and discharges from the Army. Furthermore, past

research has demonstrated a link between psychological functioning and substance abuse. As a group, soldiers with positive

urinalysis or breathalyzer tests for illicit drug use score significantly lower on the GAT survey instrument, even after controlling

for demographic differences (Lester et al., 2011a). The Medical

Data Repository and Theater Medical Data Store contain records

on soldiers’ doctor visits and hospital services related to drug

and alcohol abuse, but not all substance abuse problems result

in medical treatment. Researchers may also request data from

the Army’s Drug and Alcohol Management Information System,

which tracks positive breathalyzer and urinalysis drug tests. Additionally, the Drug and Alcohol Management Information System

captures when a unit commander makes a referral for a soldier to

seek treatment and whether behavioral consultation referrals are

consummated.

Attrition represents another major driver of personnel costs

for the Army. Information pertaining to recruits who do not complete basic training is stored in the Medical Entrance Processing

Command database, accessible through the PDE. After completing basic training, thousands of soldiers leave the Army or do

not reenlist for a second term at the end of their military service

contract. Information on soldiers who successfully complete basic

training, and then subsequently attrit from the Army, is tracked in

“Transaction Files” by the Defense Manpower Data Center, which

are also accessible through the PDE.

Many of these assets are still only available to fully vetted PDE

users with valid data use agreements. However, researchers are

currently able to request access to military service information,

deployments, demographics, accessions, pay, and waivers without

a data use agreement. The PDE staff continues to work toward

wider data availability, which is expected to grow as data providers

become more comfortable with the PDE.

RESEARCH FACILITATED BY THE PDE

This section describes three research efforts currently utilizing the

strengths of the PDE. The first is a series of analyses conducted

in support of the CSF2 mission, which evaluate the psychometric properties of the GAT and the efficacy of the CSF2 program.

Next, we briefly discuss military-civilian collaboration between

the Army, the University of Pennsylvania, and the Robert Wood

Johnson Foundation. This breakthrough project seeks to identify positive psychosocial assets that enhance mental and physical

well-being among U.S. Army soldiers. The third project represents collaboration between the Naval Postgraduate School and

the Research Facilitation Team, which seeks to determine the cost

implications of poor psychological health to the U.S. Army.

U.S. ARMY COMPREHENSIVE SOLDIER AND FAMILY FITNESS

In support of CSF2, a series of research efforts have examined the

normative relations between resilience (i.e., psychological health)

and a wide range of negative and positive outcomes. For example,

one line of inquiry has found that soldiers who receive low GAT

scores have a significantly higher propensity for suicide, psychological illness, drug use, and criminal behavior (Lester et al., 2011a;

Harms et al., 2013). A different line of inquiry has linked higher

GAT scores to accelerated promotions, promotion to brigadier

general, selections for command assignments, and Army career

fields that require professional degrees (Lester et al., 2011b).

Additionally, scientists have evaluated the Master Resilience

Training (MRT) program, a “train the trainer” intervention developed to foster psychological resilience in soldiers. Exposure to

MRT was associated with statistically significant increases in

resilience and psychological health, and a slightly lower likelihood

(11% decrease, based on the corresponding odds ratio) of being

diagnosed with a mental health problem, such as post-traumatic

stress disorder (PTSD; Harms et al., 2013). Furthermore, soldiers

exposed to the MRT program displayed a 58% lower rate of substance abuse problem diagnoses (based on the corresponding odds

ratio); significantly lower than their counterparts who were not

exposed to MRT. The PDE was instrumental in facilitating access

to the MRT program evaluation data assets. Furthermore, consistent with the instrumental goals of the PDE, the corresponding

reductions in mental health and substance abuse problem diagnoses could represent tremendous cost-saving potential for the

Army.

MILITARY-CIVILIAN COLLABORATION

In 2011, the Robert Wood Johnson Foundation awarded the

University of Pennsylvania (Penn) a grant to examine predictors of positive health in the Army. A major component of this

grant revolved around creating a foundation for future militarycivilian collaboration. Toward this end, the Penn team began

by re-examining the psychometric structure of the GAT. Confirmatory factor analysis models posited a lower-order factor

structure including positive cognitions, self-management skills,

positivity (affect), character strengths (e.g., warmth), and social

engagement (e.g., social network; Vie et al., 2013). In addition, a higher-order model of psychological strengths adequately

accounted for the moderate statistical relations among the primary factors. This more refined and parsimonious representation

of GAT strengths enables more precise examination of predictors

of positive health and is well suited for examining criterion-related

validity. A chronometric model demonstrated that psychological

assets increase in a linear and positive manner over time. Subsequent analyses have also conditioned growth in psychological

assets on various deployment indices and demographic factors

(e.g., gender, age). In addition, measurement invariance is being

tested across various demographic subgroups, in order to identify whether GAT subscales have unique meaning for different

subgroups (e.g., gender, age, education, and marital status).

The Penn team is currently studying the impact of combat deployments on psychological health. Deployment represents

an important, and highly relevant, potentially stressful event

that many Army soldiers experience at some point during their

service. Past research has examined development of PTSD symptoms following deployment. However, rather than taking a

deficit approach (i.e., assessing ill-being), Penn researchers are

capitalizing on the data resources in the PDE by examining

associations between deployment experiences and soldiers’ levels

of psychosocial strengths over time. The team is also examining relations between pre-deployment psychosocial strengths and

post-deployment depressive symptoms and negative affect. Preliminary analyses reveal that having fewer psychosocial strengths

www.frontiersin.org December 2013 | Volume 4 | Article 934 | 5
The Promise of Evidence-Based Policymaking 767

“fpsyg-04-00934” — 2013/12/11 — 17:27 — page 6 — #6

Vie et al. Army, big data, soldier fitness

is associated with more depressive symptoms and negative affect

post-deployment (controlling for early symptom levels), whereas

having more psychosocial strengths is protective and associated with fewer depressive symptoms and less negative affect

post-deployment.

ECONOMIC ANALYSES

Economists at the Naval Postgraduate School are working to

determine the costs of poor resilience and psychological health

to the Army. In addition, efforts are also underway to identify

the potential economic benefits that could accrue to the Army

through recruiting, training, and maintaining a psychologically

healthy and resilient force. Poor psychological functioning places

a significant burden on the Army (e.g., through costs related

to attrition, healthcare utilization, and psychological illness),

and the struggle for armed forces mental health professionals to

treat illnesses following combat has been considered by some the

Army’s “third front” (Thompson, 2010). On this issue, Former

Defense Secretary Robert Gates once declared, “health care costs

are eating the Defense Department alive” (Gates, 2010). The initial phase of this research effort emphasizes four cost elements

related to poor psychological functioning – attrition during initial military training, attrition before completion of first-term

enlistments, increased medical usage during deployments, and

a higher likelihood of receiving a diagnosis for PTSD or major

depression post-deployment. Preliminary results have demonstrated a significantly higher risk of attrition for recruits that

are in the bottom 10% of GAT scores, the latter treated as four

dimensions assessing spiritual, emotional, social, and family fitness. Furthermore, the results include the first-ever estimate of

the annual costs to the Army and DoD for treatment of PSTD

and major depression. Future phases may include additional

cost elements such as increased usage of medical care stateside,

heightened risk of drug and alcohol abuse, increased likelihood

of committing a criminal offense, and other negative soldier

outcomes.

Estimates of annual Army expenditures related to the treatment of PTSD alone range from $1.54 to $2.69 billion (Cesur

et al., 2011). However, these are merely estimates based on either

simulations or extrapolated treatment costs obtained from civilian medical facilities. The current research program utilizes actual

patient encounter data provided by Tricare to determine the actual

annual expenditures that are directly attributable to PTSD and

major depression. Additionally, costs of basic training are estimated at $73,000 per soldier (U.S. Army Recruiting Command,

2011). Considering that tens of thousands of Army recruits attrit

during basic training and thousands more soldiers attrit before

completion of their first-term enlistments, the cost of attrition

quickly escalates into hundreds of millions of dollars per annum.

Even small reductions in the attrition rate could result in potential

savings of tens of millions of dollars each year.

Once completed, the findings from this project will enable

decision-makers to determine the potential savings to the Army

and DoD through increases in force-wide psychological health.

Currently, researchers may only observe the costs of behavioral

health programs and the subsequent changes in soldier outcomes,

without knowing the true monetary value of any positive changes

observed. These savings, once known, allow for more inclusive

program evaluation that includes cost-benefit analysis of CSF2 or

other behavioral health interventions. Military leaders responsible for making decisions regarding instituting force-wide training

will then be able to observe the return on investment for such

programs.

DISCUSSION

The PDE is currently set to expand significantly in the near future.

This expansion entails both procuring additional data assets and

physical hardware to increase PDE bandwidth. Once fully operational, the PDE represents a tremendous opportunity for scientific

advancement via research and careful analysis. Those working diligently toward meeting this goal often characterize the promise

of the PDE as potential energy that, if properly focused, could

lead to significant breakthroughs benefitting science and humanity. Promise aside, there are a few basic tenets that must hold true

in order for the PDE to become the standard for the governance

of, security for, access to, and analysis of Army and DoD data.

First, we must carefully attend to the notion that the purpose

of the PDE is to bring scientists to the data, rather than pushing

data to scientists. Put succinctly, the data in the PDE describe one

of the U.S.’s most valuable resources – the men and women of our

military community – so proper governance is warranted. Second,

we must continue to expand the data held within the PDE in order

to keep it relevant. While the PDE likely represents the broadest

range of DoD personnel data, there are many other datasets that

have yet to be integrated into the PDE; such integration will be

iterative as new data emerge, and it will likely take years to bring

in the lion’s share of existing data. Third, the PDE must be easy

to use and continuously improve based on user feedback, if we

expect the research community to utilize this platform. Finally,

while we must be mindful of our security and privacy concerns,

we must also trust our processes and governance systems if we

are to meet our ultimate goal – providing access to the broader

scientific community.

Keeping true to these tenets falls to the Army’s Research Facilitation Team. The Research Facilitation Team – whose staff consists

of research psychologists, economists, and Big Data information

technology experts – was established in 2013 to not only provide

governance over the PDE, but also to assist researchers in using the

PDE. Given the metadata needed to navigate the vast data holdings within the PDE, this team is invaluable in shepherding new

users and their projects through the data request, review, and analysis process. The Research Facilitation Team fosters collaboration

within the PDE user community so that participating government organizations and researchers may survey the wide variety

of work done within the PDE. This is all done with an eye toward

leveraging efficiencies and reducing redundancies across research

projects.

The PDE represents a rare opportunity to connect the military

with the wealth of knowledge and experience contained within

the wider scientific community. Working side-by-side, military

and academic researchers will collaborate and examine the confluence of psychological health, soldier performance, economics,

and more. The results of this research may inform Army-wide

policy decisions regarding recruitment, prevention and treatment

Frontiers in Psychology | Developmental Psychology December 2013 | Volume 4 | Article 934 | 6
768 The Promise of Evidence-Based Policymaking

“fpsyg-04-00934” — 2013/12/11 — 17:27 — page 7 — #7

Vie et al. Army, big data, soldier fitness

programs, job assignments, manpower training, and budgeting.

The PDE represents a growing opportunity to unify science, data,

and technology under one umbrella in order to bring the best

information to bear on issues which have widespread implications

for the DoD.

ACKNOWLEDGMENTS

Order of authorship was determined by a coin flip. Loryana L. Vie,

Lawrence M. Scheier, and Martin E. P. Seligman receive support

from the University of Pennsylvania, Positive Psychology Center

with funding provided by the Robert Wood Johnson Foundation

(Grant #68709, Martin Seligman, PI). Kevin N. Griffith and Paul B.

Lester receive support from the Research Facilitation Team, Army

Analytics Group, Office of the Deputy Under Secretary of the

Army.

REFERENCES

Cesur, R., Sabia, J. J., and Tekin, E. (2011). The Psychological Costs of War: Military

Combat and Mental Health. The National Bureau of Economic Research. Working Paper No. 16927. Available at: http://www.nber.org/papers/w16927 (accessed

August 5, 2013).

Defense Manpower Data Center Overview. (2012). Powerpoint Slides. Available at: https://www.dmdc.osd.mil/appj/dwp/getfile.do?fileNm = DMDC

Overview.ppt&filePathNm = about&appId = 567&app_key_id = fh94ne8932er

(accessed August 22, 2013).

Fravell, M., Nasser, K., and Cornum, R. (2011). The soldier fitness tracker:

global delivery of comprehensive soldier fitness. Am. Psychol. 66, 73–76. doi:

10.1037/a0021632

Gates, R. (2010). Remarks as delivered by Secretary of Defense Robert M. Gates,

Abilene, KS. Speech presented at the Eisenhower Presidential Library. Available at:

http://www.defense.gov/speeches/speech.aspx?speechid=1467 (accessed August

13, 2013).

Harms, P. D., Herian, M. N., Krasikova, D. V., Vanhove, A., and Lester, P. B. (2013).

The Comprehensive Soldier and Family Fitness Program Evaluation Report #4:

Evaluation of Resilience Training and Mental and Behavioral Health Outcomes.

United States Army Comprehensive Soldier Fitness Program. Washington, DC: U.S.

Government Printing Office.

Landsbergen, D., and Wolken, G. (2002). Realizing the promise: government information systems and the fourth generation of information technology. Public Adm.

Rev. 61, 206–220. doi: 10.1111/0033-3352.00023

Lester, P. B., Harms, P. D., Bulling, D. J., Herian,M. N., and Spain, S.M. (2011a). Evaluation of Relationships between Reported Resilience and Soldier Outcomes Report

#1: Negative Outcomes. United States Army Comprehensive Soldier Fitness Program.

Washington, DC: U.S. Government Printing Office.

Lester, P. B., Harms, P. D., Bulling, D. J., Herian, M. N., Beal, S. J., and

Spain, S. M. (2011b). Evaluation of Relationships between Reported Resilience

and Soldier Outcomes Report #2: Positive Outcomes. United States Army Comprehensive Soldier Fitness Program. Washington, DC: U.S. Government Printing

Office.

Peterson, C., Park, N., and Castro, C. A. (2011). Assessment for the U.S. Army Comprehensive Soldier Fitness program: the Global Assessment Tool. Am. Psychol. 66,

10–18. doi: 10.1037/a0021658

Thompson, M. (2010). Invisible wounds: mental health and the military.

TIME Magazine. Available at: http://content.time.com/time/magazine/article/0,

9171,2008886,00.html (accessed August 5, 2013).

United States Government Accountability Office. (2008). Information Security:

Protecting Personally Identifiable Information (GAO publication GAO-08-343).

Washington, DC: U.S. Government Printing Office.

U.S. Army Recruiting Command. (2011). Frequently Asked Questions about Recruiting. Available at: http://www.usarec.army.mil/support/faqs.htm (accessed August

21, 2013).

Vie, L. L., Scheier, L. M., Lester, P. B., and Seligman, M. E. P. (2013). Psychometric structure of a self-report assessment of psychological health in young Army

Soldiers. Paper Presented at the 3rd World Congress of the International Positive

Psychology Association, Los Angeles, CA.

Conflict of Interest Statement: The authors declare that the research was conducted

in the absence of any commercial or financial relationships that could be construed

as a potential conflict of interest.

Received: 07 November 2013; accepted: 26 November 2013; published online: 13

December 2013.

Citation: Vie LL, Griffith KN, Scheier LM, Lester PB and Seligman MEP (2013)

The Person-Event Data Environment: leveraging big data for studies of psychological

strengths in soldiers. Front. Psychol. 4:934. doi: 10.3389/fpsyg.2013.00934

This article was submitted to Developmental Psychology, a section of the journal

Frontiers in Psychology.

Copyright © 2013Vie, Griffith, Scheier, Lester and Seligman. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY).

The use, distribution or reproduction in other forums is permitted, provided the original

author(s) or licensor are credited andthatthe original publication inthis journal is cited,

in accordance with accepted academic practice. No use, distribution or reproduction is

permitted which does not comply with these terms.

www.frontiersin.org December 2013 | Volume 4 | Article 934 | 7
The Promise of Evidence-Based Policymaking 769

PERSEREC’s Mission

2

Improve the Effectiveness, Efficiency, 

and Fairness of DoD Personnel 

Security and Suitability Programs

Defense Personnel and Security Research Center

Defense Personnel and Security 

Research Center (PERSEREC): Overview

Office of People Analytics (OPA)

Eric L. Lang, Ph.D., Director, PERSEREC

Defense Personnel and Security Research Center (PERSEREC)

Office of People Analytics (OPA)

March 24, 2017 

Any opinions, findings and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect 

the views of the Department of Defense, the Office of People Analytics, or the Defense Personnel and Security Research Center
770 The Promise of Evidence-Based Policymaking

PERSEREC Six Core Capabilities (1-3)

4

1

2

3

Automated 

Record 

Checks

Emerging 

Indicators

Special Policy 

Initiatives

Support Development of Systems for Using Automated Data 

Sources to Improve the Vetting and Continuing Evaluation 

of Personnel.

Assess the Relevance, Value, and Ethical Use of Social 

Media for Suitability and Security Vetting and Informing 

Health and Wellness Programs (e.g., Suicide Prevention).

Inform Senior Decision-makers with Quick Response Studies 

and Analyses in Support of Improved Policy and Processes, 

e.g., on security risks associated with certain mental health 

conditions.

PERSEREC – Who We Are

3

1986 Since 2011

In the wake of catastrophic 

espionage by John Walker and 

other spies, our mission was to 

improve the effectiveness, 

efficiency, and fairness of 

personnel security in the DoD

In the wake of events like 9-11, Ft. 

Hood, and the Washington Navy 

Yard shootings, we expanded our 

mission to address a broader 

range of human capital, insider 

threat, workplace safety, and 

security issues

Throughout our history, we have:

• Conducted long-term programmatic research for the military, 

security, and intelligence communities 

• Provided quick-response analyses and studies in support of 

policy makers, systems operations, military leaders, and 

security practitioners 

• Developed innovative tools and job aids
The Promise of Evidence-Based Policymaking 771

Insider Risks, Motivations & Behaviors Vary

6

“Insider risk”: Security & safety risks associated with trusted 

government employees, military personnel and contractors

• Espionage/leaks

e.g., cleared employee provides classified info to a foreign group

• Terrorism/violence

e.g., radicalized military personnel turns violent 

• Vandalism/disruption

e.g., vengeful employee harms his organization’s computer system

• Reliability problems and gross negligence 

e.g., employee with alcohol/substance abuse, psych problems, or 

gross incompetence fails to protect sensitive info/systems

PERSEREC Six Core Capabilities (4-6)

5

4

5

6

Security and 

Suitability 

Vetting 

Support

Insider 

Studies

Data Science 

& Advanced 

Analytics

Improve Suitability and Personnel Security Vetting 

Processes through Development of Requirements, 

Standards, and Processes that Comply with Policy and Meet 

DoD Needs.

Be an Authoritative Source of Information on Espionage, 

Terrorism, Workplace Violence, and Other Insider Threats 

and Develop More Effective Systems, Tools, and 

Interventions to Prevent, Detect, and Mitigate Activities 

that may Cause Harm to National Security.

Apply Advanced and Big Data Analytics to Human Capital, 

Security, and Cost Issues
772 The Promise of Evidence-Based Policymaking

Evaluation of Foreground Factors in the Shaping 

of Insider Threats

8

Issue: Insider threat prevention and research efforts have 

focused on identifying early warning signs of individual 

behavior rather than taking into account factors in the 

organizational environment which may influence insider 

threat behaviors, particularly workplace violence

Context: Despite 79 recommendations from the Fort Hood 

review board 2010 report, dangerous incidents at military 

installations have continued

• 2012 Fort Bragg, North Carolina – 1 dead

• 2013 Washington Navy Yard – 4 wounded, 13 dead

• 2014 Fort Hood, Texas – 12 wounded, 4 dead

7

Examples of FY16 Studies 

Conducted at PERSEREC
The Promise of Evidence-Based Policymaking 773

Incorporating PAEI into Continuous Evaluation (CE)

10

Issue: Is Publicly Available Electronic Information (PAEI),

e.g., social media, a viable data source for CE? 

Context: Previous research found that PAEI can provide 

uniquely relevant information that can help inform the 

personnel security decision-making process (Rose & 

Whiteley, 2014). 

• Individuals disclose information on social media platforms that 

they may not share anywhere else

• FBI checks are limited but information about arrests, charges, 

and convictions can be found online

• Social media data can provide additional contextual information

9

Method: Conduct a comparative case study of ~10 lethal 

targeted workplace violence cases by military, DoD civilian 

or contractor personnel within the last 5 yrs.: 

• Describe the background and environmental factors

• Describe the effectiveness of efforts to intervene

Outcome: The current effort will provide policy makers at 

OUSDI with greater understanding of how foreground 

factors shape insider threat behavior. Pending 

success/interest PERSEREC could then:

• Recommend policies and/or follow on research

• Develop training and communication materials
774 The Promise of Evidence-Based Policymaking

Policy, Process, and Adjudication of Mental Health 

Information in Personnel Security Vetting

12

Issue: Question 21 (a series of mental health Qs) on the 

Personnel Security Questionnaire (SF-86) is being revised and 

DoD needs to understand the extent to which the current 

system is capable of handling the new information solicited by 

the revised question.

Context: The revised Q21 will allow for a more relevant 

collection of mental health information from the subject

• OMB preparing for public comment on Q21 in Federal Register

• New Q21 is based on PERSEREC’s “Relevant Risk Approach”

• Goal is to obtain more productive and verifiable information for 

investigation and adjudication of mental health issues

11

Method: Obtain a random sample of 4,000 subjects from 

the CE population and conduct baseline PAEI checks. 

Select a sub-sample of 1,250 subjects and conduct CE 

PAEI checks.

Outcome: Identify the type and frequency of 

adjudicatively-relevant behaviors that occur online, 

evaluate the timeliness and uniqueness of the information, 

and assess the workload impact to determine if PAEI is a 

suitable data source for CE. 
The Promise of Evidence-Based Policymaking 775

Data Science Exploration of DMDC Data Sources for 

Modelling and Improving Personnel Systems

14

Issue: DMDC houses a variety of data that could be useful for 

understanding suitability issues, Continuous Evaluation (CE)

and military attrition but research is needed to evaluate it. 

Context: Data Science technologies and advanced analytics 

can be used to develop algorithms for analysis of data across 

data sources to identify key indicators and predictors of 

behaviors of security, suitability and manpower concern. 

13

Method: Use of a combination of literature and policy 

reviews, and interviews with experts, to:

• Outline current investigative and adjudicative policies 

and procedures

• Evaluate if current processes are adequate for 

handling info collected under the revised Q21

• Use experts and literature to identify best practices for 

addressing gaps

Outcome: The current effort will provide policy makers 

with recommendations for handling information 

collected under the revised Q21

• Changes to standard operating procedures for 

investigators, adjudicators, and evaluating 

psychologists
776 The Promise of Evidence-Based Policymaking

A Few Insider Threat Overarching Insights

16

• Indicators of personnel concern are often ambiguous. 

Appropriate reporting and follow-up are critical

• Initial screening is important, but continuous 

evaluation/monitoring is more important

• Need to fuse relevant, appropriate and legal indicators 

from multiple sources, e.g., classic record checks (such as 

FBI, travel, credit), social media, employee work network 

monitoring, local/HR records, & coworker/supervisor 

reporting

• Policies and technologies are necessary but not sufficient. 

Effective training, personalization and organizational 

climate are key to managing risk 

15

Method: Gather data on a sample of DoD personnel 

and conduct analysis. 

• Code and extract data

• Analyze data using data science technologies and 

advanced analytics

• Develop and test algorithms

Outcome: This effort could provide evidence-based 

recommendations for policy makers and operations 

managers at OUSD(I) and OUSD(P&R) regarding how 

to incorporate DMDC data into Systems related (for 

example) to CE, suitability and military attrition. 
The Promise of Evidence-Based Policymaking 777

Success Stories

18

• Production and revision of the National Adjudicative Guidelines specified in E.O. 

12968

• Phased Periodic Reinvestigation (PPR) method for continuing TOP SECRET access 

eligibility

• Electronic adjudication (“eAdjudication”) of clean SECRET cases

• Tools for evaluating the quality of investigations and adjudications

– RAISE: Rapid Assessment of Incomplete Security Evaluations

– RADAR: Review of Adjudication Documentation Accuracy and Rationales

• Automated Continuous Evaluation System (ACES) for clearances holders

• Tools for security and intel professionals (e.g., “Adjudication Desk Reference,” 

“Customizable Security Guide,” and “Insider Risk Evaluation and Audit Tool”)

• Espionage trend analyses (e.g., “Changes in Espionage by Americans 1947-2007”)

• DIRE: “Dispositional Indicators of Risk Exposure” [DIRE] tool for assessing 

personality characteristics that are security and safety risks

• DoD adjudicator certification standards and procedures

Success = Implementation of PERSEREC recommendations and tools 

Examples: The PPR and eAdjudication have saved the DoD more than $250M

PERSEREC Customers and Stakeholders

17

PERSEREC’s customers and stakeholders include policy-makers at:

• OUSD(P&R), e.g., Civilian & Military Personnel Policy • Sexual Assault 

Prevention and Response Office • Defense Suicide Prevention Office

• OUSD(Intelligence), e.g., Security Policy and Oversight Directorate

• OUSD(Policy), e.g., foreign student vetting

• DoD component Personnel Security and CI policy offices

• DoD Joint Central Adjudication Facility 

Outside of DoD, PERSEREC’s customers/stakeholders include:

• Office of Director of National Intelligence—Federal Security Executive Agent

• Office of Personnel Management—Federal Suitability Executive Agent

• Others (e.g., DOE, DOS, DHS, CIA, FBI, OMB, GAO, PAC)
778 The Promise of Evidence-Based Policymaking

Using Big Data to Study and Improve 

Military Health and Manpower Management 

Yu-Chu Shen, PhD

Professor of Economics

Graduate School of Business and Public Policy

Naval Postgraduate School

Faculty Research Fellow

National Bureau of Economic Research (NBER)

More Information and Contact

19

Selected reports, products and additional 

information are available on our website:

http://www.dhra.mil/perserec/

Or by contacting PERSEREC:

Dr. Eric L. Lang (Director, PERSEREC)

Eric.L.Lang6.civ@mail.mil

perserec@mail.mil
The Promise of Evidence-Based Policymaking 779

Example 1: Evaluating PTSD

 Evaluating GWOT’s effects on PTSD 

incidences

» What are the rates of PTSD among all active duty 

enlisted personnel and how do the rates differ by 

service and deployment location?

» How do deployment location and length of 

deployment affect the probability of being 

diagnosed with PTSD?

My Research Area

 Linking large scale databases to address 

economic issues related to health:

 Economic analysis of mental health issues in 

the US military. 

 Evaluating system-level changes in the 

health care markets and their effects on 

provider behaviors and patient care

 Public health infrastructure and population 

health outcomes
780 The Promise of Evidence-Based Policymaking

Data 

• Defense Enrollment Eligibility Reporting System

– Identify the active duty personnel population and 

obtain demographic and service information

• TRICARE 

– Standard Inpatient Data Record, Standard 

Ambulatory Data Record, Encounter Data.

– Identify the date that PTSD was first diagnosed and 

related health information 

• Contingency Tracking System

– Identify OEF/OIF deployment characteristics and 

military occupational specialty (MOS) codes

Background

 Post-traumatic stress disorder (PTSD) has 

risen steadily since the start of GWOT.

 Understanding the risk factors would allow 

focused preventative measures for those at a 

higher risk. 

 Could aid planning optimal tour length and tour 

rotation to minimize the risks of developing 

PTSD. 
The Promise of Evidence-Based Policymaking 781

Rate of PTSD By Deployment 

Location

0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

9.00

10.00

Army Marines Navy Air Force

Not deployed under OEF/OIF Afghanistan or Iraq Other countries under OEF/OIF

Outcome of Interest

 The dependent variable in our analysis is 

whether an enlisted person was diagnosed with 

PTSD anytime between 2001 and 2006 

 Binary variable: =1 if the ICD-9 code of the 

principal diagnosis is 309.81
782 The Promise of Evidence-Based Policymaking

Challenges

 Extremely length DUA review

 2 years for initial approval, not possible to 

do timely research

 Rule changes on yearly basis

 Linking critical database to capture 

complete medical history

 Most likely miss more severe cases of 

PTSD because there is no linkage to VA

 Many critical health related databases do 

not share common scrambled ID

Logistic Regression Results:

Odds Ratio of PTSD By Location

0.36

3.51

3.97

0.54

1.25

4.57

3.96

9.06

1

1

1

1

0 2 4 6 8 10

Air Force

Marines

Army

Navy

Not Deployed under OEF/OIF (reference)

Deployed to Afghanistan or Iraq

Deployed to other countries under OEF/OIF
The Promise of Evidence-Based Policymaking 783

Data

 Use Census data at the ZIP code level

 Combine with survey and administrative data to 

capture emergency department locations

 Link Census with individual patient records to 

analyze changes in ED access on patient 

outcomes.

 Take advantage of Geographical Information 

System (GIS) to create a heat map.

Example 2: Spatial Analysis of 

Public Health Infrastructure

 Evaluating Emergency Care Access at the 

Population Level

 Critical in Disaster Management
784 The Promise of Evidence-Based Policymaking

Applications in Manpower 

Management

 Use Census and heat map to analyze

 Geographical variation in civilian labor 

market

 Recruitment potentials and attrition 

likelihood

 More targeted recruitment tools based on 

micro area’s demographic composition.

No Data

No increase in driving time

Up to 5min increase

Up to 10min increase

Up to 30min increase

More than 30min increase

Change in driving time to nearest ED 1996-2005

Figure. Distribution of ED Access 

Change on US Map 
The Promise of Evidence-Based Policymaking 785

Example 3: Analyzing the Cost 

of Poor Psychological Health

 Estimating costs to the military that can be 

attributed to psychological health issues.

 Due to attrition at various stages in the first 

term

 Due to increased usage of medical care 

while in theater

 Due to increased likelihood of costly mental 

health conditions post deployment, such as 

PTSD. 

Challenges

 Geocoding data require special license

 Time consuming if doing spatial analysis 

for large area

 Subject to nontrivial measurement errors 

using available map engine (depending 

on time of the day that the search is 

performed)
786 The Promise of Evidence-Based Policymaking

Challenges

 Impossible to obtain complete medical history—

no agreement across agencies to have a 

common research resource

 Even within administrative data, lots of 

unstructured information that needs processing

 PHDA specialty field is text based.

 Privacy issue vs. quality of research product 

Data

 Psychological profile (Army specific)

 Global Assessment Test scores

 DMDC: Personnel records

 DMDC: Detailed separation information

 DMDC: Deployed information 

 CTS

 AFHSC: Combat exposure during deployment

 PDHA, PDHRA

 Health records
The Promise of Evidence-Based Policymaking 787

Data

 CDC: National death index

 Track deaths even after person left military

 DMDC: master personnel file from 2001-

20011; CTS

 TRICARE: clinical encounters to identify mental 

health history

Example 4: Analyzing stages of 

suicide behaviors in the military

 Identify risk factors at different stages of 

suicides across different services

 Suicide ideation, suicide attempts, suicide 

deaths

 examining stressful life events, mental health 

histories, quality of recruits

 Inform suicide prevent effort on high risk groups 

for better or more targeted interventions

 Big data especially necessary for research on 

rare events, such as suicides
788 The Promise of Evidence-Based Policymaking

Model

 Dependent variable 

 Death as a result of suicide

 Statistical model (survival analysis)

 Cox proportional hazard model

 Risk factors

 Mental health history

 Stressful life events (demotion, divorce)

 Recruit quality (moral waivers, test scores)

 Deployment (during and after)

Challenges

 Administrative hurdle to combine 

database

 Missing information from civilian life and 

VA, except for death

 especially for reserve component

 Classical measurement errors
The Promise of Evidence-Based Policymaking 789

Multivariate results on 

hazard of suicides

5.86

1.25

2.39

2.27

1.45

1.46

0.54

1.15

2.31

0 1 2 3 4 5 6

Self inflicted injuries

PTSD

Major depression

Substance abuse

Divorce

Demotion

During deployment

After deployment

After leaving military

Mental health

Life events

Deployment

Career status

All hazard ratio here are 

significant at p<0.01

Crude suicide trend
790 The Promise of Evidence-Based Policymaking

Editorial

The U.S. Army Person-Event Data Environment:

A Military–Civilian Big Data Enterprise

Loryana L. Vie,1,2,* Lawrence M. Scheier,1,2 Paul B. Lester,2 Tiffany E. Ho,1,2

Darwin R. Labarthe,3 and Martin E.P. Seligman1

Abstract

This report describes a groundbreaking military–civilian collaboration that benefits from an Army and Department of Defense (DoD) big data business intelligence platform called the Person-Event Data Environment

(PDE). The PDE is a consolidated data repository that contains unclassified but sensitive manpower, training, financial, health, and medical records covering U.S. Army personnel (Active Duty, Reserve, and National Guard),

civilian contractors, and military dependents. These unique data assets provide a veridical timeline capturing

each soldier’s military experience from entry to separation from the armed forces. The PDE was designed to afford unprecedented cost-efficiencies by bringing researchers and military scientists to a single computerized repository rather than porting vast data resources to individual laboratories. With funding from the Robert Wood

Johnson Foundation, researchers from the University of Pennsylvania Positive Psychology Center joined forces

with the U.S. Army Research Facilitation Laboratory, forming the scientific backbone of the military–civilian collaboration. This unparalleled opportunity was necessitated by a growing need to learn more about relations between psychological and health assets and health outcomes, including healthcare utilization and costs—issues

of major importance for both military and civilian population health. The PDE represents more than 100 times the

population size and many times the number of linked variables covered by the nation’s leading sources of population health data (e.g., the National Health and Nutrition Examination Survey). Following extensive Army vetting procedures, civilian researchers can mine the PDE’s trove of information using a suite of statistical packages

made available in a Citrix Virtual Desktop. A SharePoint collaboration and governance management environment

ensures user compliance with federal and DoD regulations concerning human subjects’ protections and also

provides a secure portal for multisite collaborations. Taking similarities and differences between military and civilian populations into account, PDE studies can provide much more detailed insight into health-related questions of broad societal concern. Finding ways to make the rich repository of digitized information in the PDE

available through military–civilian collaboration can help solve critical medical and behavioral issues affecting

the health and well-being of our nations’ military and civilian populations.

Key words: electronic medical records; epidemiology; military; Person-Event Data Environment; physical health;

population health; soldier; well-being

Introduction to the Person-Event Data Environment

The U.S. military has compiled a vast, secure computer

repository of digitized information that documents the

full breadth of a service member’s military experience.

Modern record keeping of military experience—in computerized formats easily usable today—began in the late

1990s. Since then, the U.S. military has collected extensive service member information obtained from entrance

exams (e.g., personality and aptitude), required annual

physicals, pre- and post-deployment health assessments,

medical and hospitalization treatment records, and

periodic assessments of psychological functioning, job

1

Positive Psychology Center, University of Pennsylvania, Philadelphia, Pennsylvania.

2

Research Facilitation Laboratory, Army Analytics Group, Monterey, California.

3

Feinberg School of Medicine, Northwestern University, Evanston, Illinois.

*Address correspondence to: Loryana L. Vie, Research Facilitation Laboratory, 20 Ryan Ranch Road, Suite 170, Monterey, CA 93940, E-mail: lvie@sas.upenn.edu

Big Data

Volume 3 Number 00, 2015

a Mary Ann Liebert, Inc.

DOI: 10.1089/big.2014.0055

1
The Promise of Evidence-Based Policymaking 791

performance, and military service qualification tests.

The Person-Event Data Environment (PDE) business

intelligence platform is a cloud-based virtual data repository for housing this digitized information. The

PDE was established by the U.S. Department of the

Army to facilitate research and analysis of issues and

policies that affect the military workforce—Active

Duty, Reserve, National Guard, civilians, and contractors. Though research and analysis projects from all

services are emerging in the PDE now, the system is primarily used by the Army, and therefore we will limit the

current discussion to that service. In its current form, the

PDE informs senior military and civilian government

leadership and policy makers about a variety of human

resource-related issues, including the health of the

force, training program efficacy, and return on investment, selection, and attrition. Furthermore, the PDE addresses a critical need as the Army modernizes and

engages novel behavioral health interventions to improve

Army military service. Researchers can use the PDE to

conduct program evaluation, cost-effectively pooling military data assets across numerous facets of a study. Findings of such studies can be highly informative for policy

development and evaluation regarding health and healthcare issues in the civilian population as well.

Functionally, the PDE serves two central purposes: (1)

acquire, integrate, and securely store data for Armyapproved research projects, and (2) provide a secure,

virtual workspace where approved researchers can access ‘‘sensitive’’ although unclassified Army military service, performance, manpower, and health data. Figure 1

depicts a computer screenshot of the PDE SharePoint

environment, which serves as both a governance portal

and a collaborative environment for researchers (both

within and across projects).

The PDE was initiated in 2006 as a business intelligence platform with an initial goal emphasizing command workforce, critical skill resource assessment,

and outcome studies. Other government organizations

soon recognized the power of a collaborative ‘‘commons’’ offered by the PDE technology. Specifically,

the Defense Manpower Data Center began contributing data and computational resources to the PDE,

and other armed forces branches began approaching

the Army for PDE access. Following its rapid expansion and utilization, the Army Human Research Protections Office (AHRPO) and the U.S. Army Public

Health Command were recruited to help configure

standardized governance procedures to ensure human

subjects’ protection and regulatory oversight. The Army

has applied additional resources through the Research

Facilitation Laboratory, a behavioral science unit created

to help the commons repurpose PDE data for operational studies and to promote scientific advancement.

Opportunity

In 2011, the Robert Wood Johnson Foundation (RWJF)

funded the University of Pennsylvania’s Positive Psychology Center to create a sustainable military–civilian

collaboration accessing PDE data assets. This proof-ofconcept research project examines the role that psychological health assets (e.g., optimism and positive affect)

FIG. 1. Screenshot of the PDE SharePoint environment. Source: Reproduced with permission from the U.S.

Department of the Army, Army Analytics Group, 2014.

2 VIE ET AL.
792 The Promise of Evidence-Based Policymaking

play in various health outcomes and healthcare utilization.1–3 The project research team consists of a steering

committee involving both Army and academic scholars,

a cadre of senior scientists, and project management

on-site accessing data through secure Army portals.

Collaboration between all these components has

resulted in eight fundamental areas of scholarly inquiry:

(1) mental health, (2) physical health outcomes (e.g.,

cardiovascular functioning), (3) attrition, (4) substance

use, (5) deployment-related events (i.e., trauma and

concussion-related traumatic brain injuries), (6) posttraumatic stress disorder, (7) criminal behavior, and

(8) healthcare costs.

In addition to cost-efficiencies, the PDE provides useful alternatives to the many pressing challenges faced by

behavioral and medical researchers. For one thing, national and longitudinal research studies are logistically

cumbersome, relatively expensive, and time-consuming.

Soldiers provide a vast amount of behavioral and medical data on a routine basis as part of regular monitoring,

as well as ongoing personnel and manpower data for

practical force considerations. The Army can use this information to detect mission-critical issues, including the

preparedness of its forces, their suitability for combat,

and the effects of Army service on soldier functioning,

including manifestations of prolonged deployment.

The ability to analyze Army data on soldier performance under one computational ‘‘roof’’ is attractive for

several other reasons. Cohorts are often limited to a particular geographic region or specific occupational group

(e.g., nurses), limiting their overall generalizability. Soldiers, on the other hand, ostensibly come from all

walks of life, from geographically diverse neighborhoods,

represent a heterogeneous sociodemographic profile, and

present for medical care with different behavioral and

health maladies influenced by many factors, including

family history, gender, race, age, occupation, and training

regimen, to name a few. This variety alone provides a veritable-rich environment for studying human behavior

and aggregates this information at the population level.

A good deal of the knowledge gained from using the

PDE will find application in comparable populations

and, as a result, the collaborations will provide a costeffective means through which scientists can create lasting solutions to new and continuing health problems.

Other opportunities arise from the continued monitoring of soldier performance, health, and psychosocial

functioning. Disease registries typically recruit participants after a medical condition or ailment of interest

is present, obscuring prospective markers with etiologic

importance. Also, stressful or traumatic events are difficult to anticipate and, as a result, are often studied only

retrospectively, resulting in inconsistent data collection

efforts. Data stored in the PDE provide a plethora of information from initial accession through eventual separation from the Army, providing a means to monitor

soldier functioning from emerging adulthood through

later life for career soldiers. A bulk of the Army is relatively young (17–30 is the largest single demographic

group and in 2014 constituted 64% of Active Duty soldiers) providing a unique trove of data that can inform

social policy for years to come. From a life course perspective, this is a critical age group experiencing transition from adolescent to adult health, a point at which

trends of unfavorable or continued favorable trajectories of adult health diverge. Policies for health promotion and disease prevention are therefore especially

likely to be informed by new knowledge generated by

investigations based on this unique data resource.

How Big Is Big?: Data Complexity

and Survey Frequency

One pressing question is whether the Army PDE is really

managing big data in the usual sense. The census of Army

strength on an annual basis hover around 1.2 million soldiers including all three major components (Active Duty,

Reserve, and National Guard). The PDE, which maintains an ongoing cumulative record of military personnel,

is several times this size due to retention of some and constant recruitment of new members to the population. By

any stretch of the imagination, and when compared to

federal government data bases (e.g., Treasury Department, Social Security, and Labor Statistics), social networking sites (e.g., Twitter, Google, and Facebook), or

commercial data processors of financial information

(e.g., Heartland Payment Systems or Global Payment

Systems), the PDE is not necessarily large. However, additional considerations perhaps qualify Army data as

‘‘big.’’ For one thing, soldiers’ data in the PDE is longitudinal following their military service from point of entry

to discharge or normal termination (the latter including

injury or death). Owing to the digitization of Army records from 1990 forward, this means that several million

soldiers can be examined longitudinally in the PDE. Second, as we outline below, soldiers provide a wealth of data

on many different facets of their training, health, and

functioning. The health records alone include a vast

array of information pertaining to doctor’s visits, hospitalization, medication dispensed, and associated insurance data recording, among other items, the medical

PERSON-EVENT DATA ENVIRONMENT 3
The Promise of Evidence-Based Policymaking 793

reason for the visit, the diagnosis reported for billing purposes, and the cost of the visit to the medical treatment

facility. Thus, the PDE can house upward of 10,000

or more variables collected on a single soldier accumulated through the duration of military service. This

contrasts with the more limited financial and address

information that commercial data processors possess

on any particular individual. Finally, soldiers operate

as individuals embedded in units as small as squads

and as large as brigades or divisions. This ‘‘unit of observation’’ results in nested or hierarchically clustered

data, owing to the similarity of soldier behavior within

as opposed to across different intact social units.

Nested data provide an opportunity to examine social

influence processes and also whether certain behavioral training programs or leadership styles are more

or less beneficial based on aggregate soldier profiles.

Another data complexity arises given the great variability when soldiers enter and leave the Army,

deploy, and receive routine and nonroutine health

assessments. To illustrate, PDE data assets consist of

assessments gathered routinely (e.g., weapons qualifications), on a periodic basis (e.g., annual health assessments), as well as event-based (e.g., pre- and

post-deployment health assessments). Although periodic health assessments (PHAs; annual physicals) are

typically evenly spaced, routine assessments can be

delayed based on the timing, location, and duration

of a soldier’s deployment. The lack of equal time intervals in the spacing between assessments can be methodologically challenging, particularly for researchers

invested in modeling developmental change using

time-structured assessments, although these challenges can be addressed statistically using alternative

random coefficient modeling procedures.

Imposition of Sample Selection and Gating Criteria

The tremendous variability that exists in the timing of

military health and behavioral assessments necessitates

careful consideration of how to manage such large

amounts of complex data. One solution has been to

create replicable ‘‘gating criteria’’ that delimit samples.

These take shape as Oracle Structured Query Language

(SQL) queries to establish precisely which soldiers meet

the time (during which military-related events may

transpire) and assessment windows (which assessments are mandatory during a specified time frame).

There are also service criteria, for instance, the precise

component of service (Active Duty, Reserve, or

National Guard) and whether a soldier deployed to

Iraq or Afghanistan or some other region or combat

zone. The ultimate objective of gating is to create a

uniquely definable ‘‘cohort.’’ These cohorts can then

be examined using traditional longitudinal data mining

approaches, including survival analysis, growth modeling, fixed-effect structural equation modeling, and ordinary least squares regression to ascertain effects

over time. The gating criteria offer a way of managing

the variability that characterizes soldier service records,

while at the same time offering a rigorous means to detect periodicity and regularity often encountered with

longitudinal data.

To illustrate, a recent study employed SQL queries to

elucidate the contribution of deployment combat stress

to alcohol consumption, medical symptoms, and a positive screen for posttraumatic stress disorder (PTSD).

These are important areas for scientific inquiry as well

as major costs drivers for the Army. Understanding

these cost drivers can be explored through intelligent

querying of the database, which involves specifying selective gating criteria that reveal interesting data patterns related to health and cost for such criteria.

As an example, the following gating criteria were applied sequentially to the database: (1) deployed between

September 2012 and December 2014 (n = 154,340),

(2) completed the required pre- and post-deployment

health assessments (n = 46,176), (3) had taken a selfreport survey assessing psychosocial functioning

roughly a year prior to deployment (n = 14,294), and

(4) were between the ages of 18 and 30 (n = 10,058).

This netted a cohort of soldiers that had deployed during a specific time frame, were young adults, and had

completed the required psychosocial and health assessments. The study was framed by learned helplessness

theory, which states that depression and related mental

illnesses result from perceived absence of control over

the outcome of a situation and an exaggerated sense

of helplessness or negativity.* In this study, an attributional tendency termed ‘‘negative explanatory style’’

was assessed using a self-report measure of ‘‘catastrophic thinking,’’ which can be thought of as ruminating about worst case outcomes.{

To test linkages between combat stress and poor

health outcomes among the panel soldiers we assessed

the relative contribution of deployment combat-related

stress, catastrophic thinking, and three demographic

control measures (age, gender, and rank). Soldiers

*http://en.wikipedia.org/wiki/Learned_helplessness

{

https://www.psychologytoday.com/blog/in-the-face-adversity/201103/catastrophicthinking

4 VIE ET AL.
794 The Promise of Evidence-Based Policymaking

who reported any one of seven combat-related stressors

(e.g., ‘‘encountered dead bodies,’’ ‘‘discharged a weapon,’’ and experienced a blast’’) were coded accordingly.

A measure of catastrophic thinking was obtained from

a self-awareness tool soldiers take annually. Sample

items included ‘‘when bad things happen to be, I expect

more bad things to happen’’ and ‘‘when bad things happen to me, I cannot stop thinking about how much

worse things will get,’’ which were scored on a fivepoint Likert-type scale.

Outcome assessments were based on a 17-item PTSD

symptom index4 (cut-point of 30 or more indicated ‘at

risk’), a three-item alcohol misuse screener5 (ranging

from 0–12), and a 29-item index of general health concerns (ranging from 0–29). Controlling for age, rank,

and gender the findings indicated that combat-related

stress (trauma) was significantly related to a positive

screen for PTSD (odds ratio [OR] = 3.11, 95% confidence interval [CI] = 2.37–4.08, p < 0.0001), alcohol

misuse (b = 0.07, p < 0.0001), and more health concerns

(b = 0.21, p < 0.0001), respectively. Catastrophic thinking also placed soldiers at risk for all three outcomes

(PTSD: OR = 1.18, 95% CI = 1.02–1.37, p < .05; alcohol

misuse: b = 0.03, p < .01; and health concerns: b = .07,

p < .0001). Structured queries like the one used here:

1) reinforce the necessity of modeling heterogeneous

measures of military experiences; and 2) highlight the

potential need for tailoring current health promotion

programs to address possible subgroup differences in

military-specific outcomes.

Measures

From their initial point of entry into the Army (accession), soldiers provide continued (semiroutine) information on their health, psychologicalfunctioning, vocational

aptitude, personality, fitness, and training qualification.

There are ancillary data sources that track mandatory

officer evaluations, military and civilian education, and

soldiers who seek alternative training through special operations training and aviation schools. This wealth of data

can provide a composite picture of a soldier’s life and be

used for operational studies or research purposes. Table

1 includes an overview of several key Army health data

assets that are fundamental to the RWJF military–civilian

collaboration. The table illustrates the basic content for

each asset, the primary source of the health data (e.g.,

self-rated reports), and the administration sequence

for each assessment (periodic or event-based).

As an illustration of how resourceful the PDE can be,

we now present a detailed overview of one of the health

data assets, the PHA. Active component and select

Reserve personnel are required to receive an annual

PHA. The PHA is a standardized preventive screening

tool designed to improve the reporting and visibility of

the individual medical readiness describing each soldier’s physical ability to deploy. Specifically, this assessment consists of three integrated steps: (1) an online

Health Risk Assessment (e.g., family history, medical

conditions, and current medication use) with referrals

made for laboratory studies and immunizations, (2)

support staff review the personal medical information

(e.g., height, weight, and medications), and (3) a

healthcare provider reviews each soldier’s statement

of health, evaluates any required laboratory results,

performs a medical symptom-focused exam, rates

body system functioning using the medical physical

profile serial qualification system, and provides referrals for additional medical services as indicated.

Over 600 data elements are collected during this

three-step assessment process, approximately 107 of

which contain administrative (e.g., ‘‘date PHA form

approved’’) or personal information (e.g., soldier’s

telephone number). As we discuss in the following

sections, personally identifying information can never

be examined for research purposes. The health data

in the PHA database can be categorized as follows: allergy information, 43 variables (e.g., reports allergy to

iodine); behavioral health information, 43 variables

(e.g., reports feeling down); clinical evaluation, 119 variables (e.g., diastolic blood pressure); overall health, 87

variables (e.g., soldier has chronic pain); family history,

115 variables (e.g., father had cancer); medications, 18

variables (e.g., ‘‘class of drug’’); preventive health,

74 variables (e.g., frequency of alcohol use); and functional capacity, 18 variables (e.g., score for physical

capacity or stamina). Across a five-year career, this represents over 2,500 soldier health data elements that can

be gathered and made available to researchers as deidentified data. Even more impressive is the fact that

these data elements can be merged with other longitudinal databases housed in the PDE, enabling researchers to examine contextual factors (e.g., psychosocial

strengths, deployments, years of service, and job performance, to name a few) that may relate to health at

specific points in time.

Utility of the PDE

Current use of the PDE generally falls into three categories: novel research, organizational analysis, and program evaluation. In terms of novel research, the PDE is

PERSON-EVENT DATA ENVIRONMENT 5
The Promise of Evidence-Based Policymaking 795

being used for a wide variety or research projects examining the health- and work-related behaviors of members of the military. For example, a recently published

study using PDE data assets analyzed data from the

Army’s Global Assessment Tool (GAT) and found

that high-performing soldiers tended to report relatively higher GAT scores on measures of psychosocial

functioning (i.e., optimism), whereas soldiers with behavioral problems tended to have relatively lower

GAT scores6

; this study reinforced the practical utility

of the GAT as a psychometric instrument. Other research studies in the PDE are underway now, including

research designed to model military family resilience,

feasibility studies for the use of social media data for

suicide prevention, the development of risk algorithms

for a range of behavioral problems, and research examining how leadership behaviors can influence follower

psychological health.

There are also questions about population cardiovascular health and the different approaches to investigating these important medical considerations that offer

another example of the utility and significance of the

PDE for conducting novel research. As previously stated, the Army is a large, racially and ethnically heterogeneous population with diverse age groups from as

young as 17 through later middle life (ages 60–70).

As such, the extensive inventory of personal-level

data on cardiovascular health status, health behaviors,

psychological factors, and social determinants of health

provides an exceptionally rich existing data set

Table 1. Person-Event Data Environment health data assets

Database Description of contents Primary source Admin. Pop.

Deployment Health Assessments Health before and after deployment (e.g., self-rated

health, alcohol and tobacco use, PTSD, depression,

combat exposure, injury and concussion risk, health

concerns, major life stressors, Rx use, environmental

and exposure concerns, suicide ideation, violence or

potential for self-harm)

Self-rated

& objective

Event-based S

Digital Training Management System Comprehensive training records (e.g., marksmanship

training, predeployment training), physical fitness

metrics (e.g., push-ups, sit-ups, two-mile run,

participation in a weight control program)

Objective Event-based

& periodic

S

Drug & Alcohol Management

Information System

Positive drug and alcohol screens (e.g., urinalysis,

breathalyzer), referrals and enrollment in treatment,

patient follow-up, and progress

Objective Event-based S

Electronic Physical Evaluation Board Physical Evaluation Board key dates (e.g., date started,

referral date, approval date, date placed on TDRL),

disposition of the board, overall percentage of

disability, description of condition

Objective Event-based S

Medical Data Repository Electronic health records (e.g., appointment dates, Rx

medications, procedures and surgeries, vitals [e.g.,

blood pressure], healthcare costs, pathology

laboratory results)

Objective Event-based S, D

Periodic Health Assessment Yearly physical assessments (e.g., overall health,

clinical evaluation, medications, family history,

behavioral health, preventive health, physical profile,

deployability)

Self-rated

& objective

Periodic S

Social Security Admin. Death File Death date, death verification code, last residence

(state)

Objective Event-based S

ArmyFit—Global Assessment Tool Psychological strengths (e.g., adaptability, positive/

negative coping, catastrophizing, social engagement,

optimism, meaning, character, depression, positive &

negative affect, family satisfaction, family support,

work engagement, friendship, loneliness,

organizational trust), health, health-related behaviors

(e.g., cigarette smoking)

Self-rated Periodic S, D, C

Theater Medical Data Store Electronic health records during deployment (e.g.,

appointment dates, medications, procedures, injury

type, illness diagnostic categories, symptoms, blood

pressure, pulse rate, temperature)

Objective Event-based S

Admin., administration; C, DoD civilian; D, dependent; Pop., population; PTSD, posttraumatic stress disorder; Rx, prescription medication; S, soldier;

TDRL, temporary disability retired list.

6 VIE ET AL.
796 The Promise of Evidence-Based Policymaking

regarding relations among these health factors and potential strategies for cardiovascular disease prevention.

Record linkage for individuals throughout their military careers provides opportunities for longitudinal as

well as cross-sectional examination of these relationships. Health-related policy changes within the Army

can be proposed, developed, implemented, and evaluated through ongoing investigations with these data.

The applicability of findings to the civilian population

and the opportunity to make direct statistical comparisons to the civilian population add further value to this

exceptional health data asset.

For example, one line of effort has recently examined

PHA data in the PDE to contrast measures of health

among Army Active Duty, Reserve, and National

Guard soldiers, and civilians from the National Health

and Nutrition Examination Survey. Appendix I provides a case study that showcases comparisons on several

health metrics that are major cost drivers for both the

Army and general population.

Additionally, the PDE is used to perform a range of

organizational analyses to answer questions posed by

the Department of Defense’s (DoD’s) senior leadership

as well as members of the U.S. Congress. The PDE was

recently used to respond to inquiries from Congress

about demographic characteristics of service members

within the DoD. In addition, the PDE has great utility

for the development and pilot testing of new standardized assessment forms and behavioral instruments.

Researchers can follow standard psychometric procedures and examine different forms of validity, including concurrent and factorial (using existing data),

predictive (with prospective data), and convergent

and discriminant (with other PDE data assets).

Finally, the PDE is used to perform large-scale program evaluation on a range of DoD-related military

programs. For example, there have been calls in the literature for more rigorous evaluation of the Army’s

Comprehensive Soldier & Family Fitness (CSF2) program—a psychological health and resilience training

program.7,8 The PDE has made it possible to securely

bring together data on a range of soldier outcomes,

begin evaluating the CSF2 Master Resilience Training

(MRT) program, and respond to these clarion calls

with evidence-based findings. Specifically, researchers

have examined longitudinally participation in the

MRT program and subsequent ratings of psychosocial

strengths and assets.9 Follow-up work examined associations between participation in the MRT program

and its effect on the prevalence of diagnoses for mental

health or substance abuse problems.10 Subsequent research

will extend to other outcomes, including health ratings,

job performance, and healthcare costs, to name a few.

The PDE has also been used to evaluate other Army programs, including the Army Surgeon General’s Performance Triad—a program designed to promote healthy

sleep, physical activity, and nutrition behaviors in soldiers.

For researchers in the PDE evaluating military programs

using observational studies or quasi-experimental designs,

use of the propensity score method (which assesses the

likelihood of being assigned to the treatment group

based solely on one’s demographic or covariate information) strengthens the ability to make causal inferences

even in the absence of a randomized control group.11

Compliance with Human Subjects’ Protections

Extensive measures are taken to protect the confidentiality and personal identity of soldiers whose information is part of the digitized resources housed in

the PDE. As part of protecting personally identifying

information (PII), social security numbers undergo a

two-step transformation and encoding process, which

results in the assignment of a random 12-character

alphanumeric ‘‘key’’ to each soldier. Data in the PDE

can then be merged and linked via the randomly generated ‘‘keys’’ in order to create linked files from multiple databases and time points in support of Army- and

DoD-approved research. The governance process that

creates identification keys relies on physically and logically separate computer systems with secure Army and

DoD firewalls using a VPN connection. In addition,

personnel responsible for the extraction, transfer, and

load of de-identified data are federally approved contractors who undergo extensive Health Insurance Portability and Accountability Act (HIPAA) training and

also work in a secure environment.

Additional transformations for limited data sets containing protected health information (PHI) include

truncating birth dates so that only year or month and

year are available. A soldier’s unit identification code,

rank, and pay grade are also transformed for PDE research studies. As outlined below, each transformation

of PII is designed to reduce the risk of a soldier being

re-identified by a researcher, while maintaining enough

information for standard aggregate statistical analysis

and longitudinal record linkage.

Accessing Medical and Health Data

Access to medical and health data is covered under the

HIPAA (45 CFR Subpart 46, PL 104-191) and the

PERSON-EVENT DATA ENVIRONMENT 7
The Promise of Evidence-Based Policymaking 797

Privacy Act of 1974 (Pub. L. No. 93-579, and its subsequent amendments USC Sec. 552a, Title 5, Part I,

Chapter 5, Subchapter II). Both statutes carefully delineate the safeguarding of PHI and the manner in which

‘‘limited data set’’ identifiers can be disclosed (e.g.,

birthdates are transformed to MMYYYY).

All preparation of personal health and medical data

in the PDE must comport with the Standards of DeIdentification of Protected Health Information (Section 164.514[b][2] of the 1974 Privacy Act). The latter

requirement involves establishing compliance with

either the safe harbor or the expert determination

method. Both methods ensure compliance with federal standards that essentially mitigate privacy risks

pertaining to sharing PHI between covered entities

(i.e., health insurers) and outside parties. The former

procedure requires removal of 18 limited data set

identifiers (e.g., name, address, e-mail, driver license,

or other unique identifiers) from PHI in order to reduce the potential of ‘‘re-identification.’’ The latter

method relies on scientifically valid statistical audit

procedures designed to evaluate the potential risk

for re-identification (disclosure) given the proposed

de-identification procedures.

Projects deemed research involving human subjects

must also undergo an external scientific review using an

Institutional Review Board and vetting by the AHRPO,

which provides regulatory oversight for human subjects’

protection of soldiers. All of these assurances and regulatory requirements are detailed in DoD Information

(3216.02 ‘‘Protection of Human Subjects and Adherence

to Ethical Standards in DoD-Supported Research’’) and

Department of the Army (e.g., AR 70-25 ‘‘Use of Volunteers as Subjects of Research’’) guidelines. Applications for

AHRPO human subjects and regulatory approval rely on

the Force Health Protection and Readiness IRBNet portal,

which is part of the Defense Medical Research Network.

The Medical Research and Materiel Command website

provides documentation of the Army procedures and applicable DoD regulatory requirements for human subjects’ protection.

Conclusions

The RWJF military–civilian collaboration paves the

way to methodically and incrementally open the PDE

access aperture over time, thereby melding the enormous data assets of the Army with top research scientists

from private commercial ventures and university-based

settings. This scale-up requires blending the needs of researchers with the operational features of the PDE, all

the while ensuring the protection and confidentiality

of personal and health information obtained from the

individuals tasked with defending our country. The

PDE offers an unprecedented resource to the scientific

community, and it is quickly becoming the most extensive collection of digitized information on this important population or any other population we know of,

given its tremendous breadth and depth.

In addition, the PDE is also moving in the direction

of creating metadata resources to document the various

DoD and Army data assets. This will include archived

institutional history that describes the evolution of soldier assessment forms, version and content changes in

surveys, data management concerns (e.g., variable coding and transformation), and details on data collection

methods and parameters describing test administration.

Future SharePoint capabilities will enable members of

the PDE research community to record comments on

data assets and their elements (e.g., indicate whether

data fields are incomplete or have different values than

expected). This information will prove quite valuable

to subsequent researchers and will help build a more

efficient ‘‘commons’’ research process. The PDE can

also draw upon resources provided by a DoD Metadata

Registry, managed by the Defense Information Systems

Agency, a web-based repository that promotes interoperability and reuse of computer technology (e.g., data

models, symbologies, transformations, and schemas)

among military department and defense agencies.

What the Future Holds

Given its relatively recent inception, the PDE has yet to

reach full operational capability. Rapid growth of the

PDE requiring greater bandwidth, procurement of sufficient ‘‘seat’’ licenses for commercial statistical packages, and the computing power required to manage

and analyze large complex data structures currently

limit the number of users the PDE can host. Resolving

these bandwidth and related operational limitations

will be part of getting the PDE to full operational capability. There is also an effort underway to ‘‘automate’’

much of the governance of the PDE, including security

procedures for vetting end users, conducting background checks, and ensuring that study research

goals are compliant with data use agreements. Every

study in the PDE has to maintain current documentation of individual researchers’ human subjects’ protections compliance, adherence to PDE governance, and

DoD Information Assurance certification. As essential

as this process is for operational logistics, it can also

8 VIE ET AL.
798 The Promise of Evidence-Based Policymaking

be cumbersome. In addition, data assets are not released

unless checked manually to ensure release comports

with Army Data Use Agreements and, in the case of

medical data, meets the requirements specified in Data

Sharing Agreements. This process is crucial to maintaining soldier confidentiality as well as regulatory compliance. Furthermore, the Army is also undergoing rapid

changes in the types of platforms used to gather soldier

data. As an example, a new online platform ‘‘ArmyFit’’ is

now gathering data on soldier (and spouse) fitness, including nutritional information, sleep, and physical exercise. This is part of the Army Resilience Directorate’s

mandate to include emotional, social, spiritual, family,

and physical fitness dimensions as part of routine assessments of soldier functioning. The advent of new webaccessible platforms collecting routine information on

almost 50,000 soldiers each month means new soldier data (with unique coding formats) are constantly

streaming into the PDE, broadening the capability of researchers to track emerging epidemiological trends.

We specifically note that the military–civilian collaboration will reap untold opportunities for researchers, who

will gain access to unique and very extensive, prospective

data on a very large population of Army soldiers. This

will enable them to examine population-based trends

in a wide range of health-related behaviors and conditions with important implications for society at large.

Likewise, the Army will benefit from the expertise of

leading behavioral and medical scientists interested in

measuring and improving soldier performance and

health, with insights of great potential value for population health more generally.

Acknowledgments

We would like to thank the numerous Army, Department of Defense, and University of Pennsylvania entities that have worked tirelessly to ensure the success

of the military–civilian collaboration. It is hoped that

the fruits of this initial effort can have positive and lasting effects on soldiers’ lives and eventually find ways to

inform Army leadership of critical health issues. In particular, Jenny L. English, health statistician, U.S. Army/

PASBA-MEDCOM, and Audrey L. Luken, MEDPROS

program manager, G3-7 Medical Readiness Division,

U.S. Army Medical Command, provided critical information on the data assets described in this profile.

Funding

The PDE is funded by the Departments of Defense,

Army, Navy, and Air Force. Support for this publication is provided in part by the Robert Wood Johnson

Foundation through a grant to the Positive Psychology

Center of the University of Pennsylvania, Martin E.P.

Seligman, principal investigator. Paul B. Lester receives

support from the Research Facilitation Laboratory,

Army Analytics Group, Office of the Deputy Under

Secretary of the Army.

Author Disclosure Statement

The authors had no conflicts of interest with respect to

the authorship or the publication of this article. The

University of Pennsylvania has a proprietary interest

in the Master Resilience Training (MRT) program.

Loryana L. Vie, Lawrence M. Scheier, Tiffany E. Ho,

and Martin E.P. Seligman did not analyze MRT data

in the PDE or take part in discussions of intervention

data related to the MRT program. Paul B. Lester is

the Federal Government Sponsor for research activities

in the PDE. He provides governance oversight as well

as technical assistance obtaining various U.S. Army

data assets, including the GAT data from the ArmyFit

platform. The views expressed in this article are those

of the authors and do not reflect the official policy or

position of the Department of the Army, Department

of Defense, or the U.S. government.

References

1. Seligman MEP, Csikszentmihalyi M. Positive psychology. An introduction.

Am Psychol 2000; 55:5–14.

2. Seligman MEP. Positive health. Appl Psychol Int Rev 2008; 57:3–18.

3. Peterson C, Bossio LM. Optimism and physical well-being. In: Chang EC,

editor. Optimism & Pessimism: Implications for Theory, Research, and

Practice. Washington, DC: American Psychological Association, 2001, pp.

127–145.

4. Blanchard EB, Jones-Alexander J, Buckley TC, Forneris CA. Psychometric

properties of the PTSD checklist (PCL). Behavioral Research & Therapy

1996; 34:669–673.

5. Bush K, Kivlahan DR, McDonell MB, Fihn SD, Bradley KA. The AUDIT

alcohol consumption questions (AUDIT-C): An effective brief screening

test for problem drinking. Archives of Internal Medicine 1998;

158:1789–1795.

6. Lester PB, Harms PD, Herian MN, et al. A force of change: Chris Peterson

and the US Army’s Global Assessment Tool. J Posit Psychol 2014; 10:

7–16.

7. Smith SL. Could Comprehensive Soldier Fitness have iatrogenic

consequences? A commentary. J Behav Health Ser R 2013; 40:242–

246.

8. Steenkamp MM, Nash WP, Litz BT. Post-traumatic stress disorder: Review

of the Comprehensive Soldier Fitness program. Am J Prev Med 2013;

44:507–512.

9. Lester PB, Harms PD, Herian MN, et al. The Comprehensive Soldier

Fitness Program Evaluation Report #3: Longitudinal Analysis of the

Impact of Master Resilience Training on Self-Reported Resilience and

Psychological Health Data. United States Army Comprehensive Soldier

Fitness Program. Washington, DC: U.S. Government Printing Office,

2011.

10. Harms PD, Herian MN, Krasikova DV, et al. The Comprehensive Soldier

and Family Fitness Program Evaluation Report #4: Evaluation of Resilience

Training and Mental and Behavioral Health Outcomes. United States

PERSON-EVENT DATA ENVIRONMENT 9
The Promise of Evidence-Based Policymaking 799

Army Comprehensive Soldier Fitness Program. Washington, DC: U.S.

Government Printing Office, 2013.

11. Scheier LM. Methods for approximating random assignment: Regression

discontinuity and propensity scores. In: Baker E, Peterson PP, McGaw B,

editors. International Encyclopedia of Education (3rd ed.). London:

Elsevier, 2010, pp. 104–110.

Cite this article as: Vie LL, Scheier LM, Lester PB, Ho TE, Labarthe DR,

Seligman MEP (2015) The U.S. Army Person-Event Data Environment:

a military–civilian big data enterprise. Big Data 3:xx, 1–10, DOI:

10.1089/big.2014.0055.

Abbreviations Used

AHRPO 1⁄4 Army Human Research Protections Office

CSF2 1⁄4 Comprehensive Soldier & Family Fitness

DoD 1⁄4 Department of Defense

GAT 1⁄4 Global Assessment Tool

HIPAA 1⁄4 Health Insurance Portability and Accountability Act

MRT 1⁄4 Master Resilience Training

PDE 1⁄4 Person-Event Data Environment

PHA 1⁄4 Periodic Health Assessment

PHI 1⁄4 protected health information

PII 1⁄4 personally identifying information

RWJF 1⁄4 Robert Wood Johnson Foundation

SQL 1⁄4 Structured Query Language

(Appendix follows/)

10 VIE ET AL.
800 The Promise of Evidence-Based Policymaking

Appendix I: Case Study

Below we present one example of how Person-Event Data

Environment (PDE) data can be used to generate a ‘‘report

card’’ on several major cost drivers for the military as well

as the U.S. civilian population. In this project, researchers

from the Robert Wood Johnson Foundation military–

civilian collaboration contrasted five metrics obtained

from Active Duty and Reserve/National Guard soldiers

with data from the 2012 National Health and Nutrition

Examination Survey (NHANES).A1 The NHANES is

one of several nationally representative general population studies that provide valid and reliable measures of

health and psychosocial functioning in the United States,

and it represents the largest ongoing individual-level

health examination survey (other examples include the

Behavioral Risk Factor Surveillance SystemA2 or the Midlife in the United States StudyA3).

The five metrics selected for illustration are heavy cigarette use (transforming number of cigarettes in the past

30 days into the equivalent of a pack or more: ‘‘21 or

more cigarettes per day’’); heavy alcohol consumption

(three AUDIT-CA4 items assessing alcohol frequency

[‘‘How often did you have a drink with alcohol?’’], intensity [‘‘How many drinks did you have on a typical day

when you were drinking?’’] and binge drinking [‘‘How

often did you have 6 or more drinks on one occasion?’’]);

depression (using the 9-item PHQ-9,A5 a general population depression screener assessing depressed mood or irritability, decreased interest or pleasure, significant weight

change or change in appetite, change in sleep, change in

activity, fatigue or loss of energy, feelings of guilt or

worthlessness, diminished concentration, and suicidal

tendencies); physician care (seeing a medical practitioner

over the past year); and hospitalization (whether the respondent had been hospitalized within the past year or,

for soldiers, since their last annual Army physical).

Table A1 shows the demographic comparison between

the Army and civilian samples. The sample size in the

Table A1. Demographic characteristics of military personnel and civilians (2012)

Active Duty soldiers

(n = 265,525)

Reserve/National Guard

soldiersa (n = 398,240)

Civiliansb

(n = 4,854)

Characteristic n (%) n (%) n (%)

Gender

Male 224,767 (84.65) 327,074 (77.51) 2,403 (49.51)

Female 40,758 (15.35) 71,166 (17.87) 2,451 (50.49)

Age group, years

17–29 150,700 (56.76) 209,745 (52.67) 1,441 (29.69)

30–39 78,125 (29.42) 95,467 (23.97) 963 (19.84)

40–49 33,312 (12.55) 70,044 (17.59) 899 (18.52)

50–65 3,388 (1.28) 22,989 (5.77) 1,551 (31.95)

Race/ethnicity

Hispanic 28,553 (10.75) 40,239 (10.10) 1,062 (21.88)

White 157,773 (59.42) 268,861 (67.51) 1,566 (32.26)

Black 58,111 (21.89) 65,742 (16.51) 1,331 (27.42)

Asian 13,687 (5.15) 14,257 (3.58) 742 (15.29)

Other, including multiracial 7,401 (2.79) 9,133 (2.29) 153 (3.15)

Education

No high school diploma 1,245 (0.47) 10,371 (2.63) 1,160 (23.90)

High school diploma or equivalent 195,187 (74.04) 286,588 (72.66) 1,013 (20.87)

Some college 11,941 (4.53) 17,671 (4.48) 1,491 (30.72)

College degree and higher 55,249 (20.96) 79,802 (20.23) 1,190 (24.52)

Marital status

Never married 89,403 (33.68) 186,554 (46.88) 2,515 (57.12)

Married 159,616 (60.14) 182,438 (45.85) 1,136 (25.80)

Separated/divorced/widowed 16,397 (6.18) 28,932 (7.27) 752 (17.08)

Length of service (years)

0–3 90,348 (34.03) 97,493 (24.48) N/A

4–8 64,907 (24.44) 115,742 (29.06) N/A

9–15 57,496 (21.65) 74,841 (18.79) N/A

‡ 16 52,774 (19.88) 110,169 (27.66) N/A

a

Army Reserve and National Guard soldiers differ only in the source of their pay. (The National Guard receives pay compensation from the federal

budget, but they are organized and run by the individual states. Army Reservists receive compensation directly from the federal budget.) Otherwise

their standards of performance and required training programs are identical and both service branches can deploy if needed. Therefore, for the purpose of this article, these two groups were combined. b

Civilians are composed of a nationally representative sample from the NHANES (2011–2012).

PERSON-EVENT DATA ENVIRONMENT 11
The Promise of Evidence-Based Policymaking 801

Table A2. Adjusted and weighted* means and period prevalence for health metrics among military personnel and civilians (2012)

Heavy cigarette use{ (% yes) Heavy alcohol consumption (AUDIT-C score) Depression severity (PHQ-9 score) Visited healthcare provider{{ (% yes) Hospitalized{ (% yes)

AD Res./NG Civ. AD Res./NG Civ. AD Res./NG Civ. AD Res./NG Civ. AD Res./NG Civ.

Age

17–29 3.25a 3.57b 0.40c 2.11a 1.94a 2.96b 1.38a 0.95b 3.33c 29.88a 25.35b 80.35c 11.64a 8.21b 8.37c

30–39 4.82a 5.81b 1.88c 2.32a 2.14a 2.97b 2.41a 1.90a 4.03b 48.34a 37.25b 74.63c 16.60a 10.30b 10.01c40–49 7.08a 8.64a 4.06b 1.79a,b 1.66a 2.54b 3.21a,b 2.52a 4.10b 61.96a 48.48b 83.14c 19.27a 12.06b 9.84c50–65 7.77a 9.62a 3.04b 1.10a 1.11a 1.81a 3.08a 2.13a 3.78a 71.84a 56.36b 87.08c 21.64a 14.71b 10.39c

Gender

Male 4.22a 5.34a 3.44b 2.49a 2.33a 3.40b 1.50a 0.99b 2.96c 37.26a 30.79b 75.54c 12.96a 8.37b 6.71c

Female 1.88a 2.67a 1.24b 1.60a 1.52a 1.86a 2.05a 1.39a 4.47 b 54.23a 49.07b 88.25c 20.90a 16.18b 12.50c

Race/ethnicity

Hispanic 1.84a 1.88b 0.33c 1.63a 1.62a 2.87b 2.01a 1.57a 3.81b 35.32a 28.91b 70.74c 13.49a 8.99b 10.34c

White 5.06a 5.92b 3.17c 2.38a 2.33a 3.21b 1.99a 1.42a 3.67b 40.64a 35.12b 84.83c 14.28a 9.97b 9.15cBlack 1.71a 1.60a 0.66b 1.93a 1.93a 2.72b 2.27a 1.87a 3.67b 40.46a 34.43b 83.58c 14.61a 9.92b 12.85cAsian 1.93a 2.62b 0.14c 2.29a 2.15a 1.83a 1.27a 0.76a 2.50b 36.74a 26.85b 78.99c 12.64a 7.33b 5.39cOther 2.43a 2.80a 5.69b 3.13a 2.74a 3.59a 3.46a 2.69a 5.95b 42.15a 34.30b 80.66c 14.32a 9.85a 11.51b

Education

No HSD 5.73a 4.20b 5.62c 1.83a 1.85a 2.84a 3.86a 5.56a 2.47a 34.38a 26.69b 72.76c 14.25a 8.10b 12.97c

HSD or equiv. 4.04a 5.05b 1.70c 1.92a 1.76a 2.64b 2.27a 1.50b 4.07c 36.18a 30.91b 78.39c 13.88a 9.44b 8.83cSome college 3.77a 5.08a 2.41b 1.91a 1.63a 2.67a 1.90a 1.33a 3.43a 51.52a 42.75b 82.55c 16.66a 11.56b 9.39cCollege degree or greater 2.63a 3.95a 0.77b 1.98a 1.88a 2.73b 1.21a 0.92a 2.37b 50.56a 44.40b 89.00c 14.74a 10.77b 8.62c

Marital status

Never married 4.58a 6.28a 2.73b 1.82a 1.67a 2.40b 1.64a 1.35a 2.85b 45.76a 41.05b 82.78c 16.37a 11.56b 10.08c

Married 2.80a 3.57b 1.04c 1.45a 1.41a 2.91b 1.60a 1.32a 4.57b 27.01a 25.61b 79.22c 9.39a 7.60b 8.28cSep./Div./Widow 5.04a 7.12a 3.34b 2.23a 1.89a 2.51a 2.35a,b 1.95a 4.04b 52.59a 44.50b 82.47c 19.03a 12.40b 11.13c

*Means are adjusted for all variables in the table. Means and period prevalence rates are weighted for NHANES complex sampling design and nonresponse. Comparisons were conducted using ANCOVA and

logistic regression with linear combinations.

{Heavy cigarette smoking was assessed for those who currently smoke and is characterized by smoking at least 21 cigarettes a day. {{Soldiers were asked if they had seen a healthcare provider since their last military examination. Participants in NHANES were asked if they had seen a healthcare provider in the last year. {Soldiers were asked if they had been hospitalized or had surgery since their last military examination (55% had conducted their prior periodic health assessment within a 15-month window, 30% between up to 2

years, 13% between 2 and 3 years, and 2% beyond this window). Participants in NHANES were asked if they had been a patient in a hospital overnight in the past year. a,b,c,dRaised superscript letters that are different indicate statistically significant differences ( p < 0.05) of adjusted and weighted means. Same letters indicate no statistically significant differences in means. Tukey’s

honest significance test method was used to adjust for multiple comparisons and the increased probability of making false-positive type I errors.

AD, Active Duty soldiers; Civ., Civilian (NHANES); HSD, high school diploma; NHANES, National Health and Nutrition Examination Survey; Res./NG, Reserve and National Guard soldiers; Sep./Div./Widow, Separated/

Divorced/Widowed.

12
802 The Promise of Evidence-Based Policymaking

PDE is more than 100 times that of NHANES, most conspicuously so for race-ethnic groups other than white.

The military population is younger, less highly educated,

and more often married than the NHANES population

sample. Notably, the Active Duty and Reserve/National

Guard Soldier populations are generally quite similar demographically, supporting their potential pooling for

many research purposes. Table A2 contains the results

of group mean comparisons for the five metrics outlined

above. A careful inspection shows that, adjusted for all

other covariates in the model, soldiers consistently

reported higher rates of heavy cigarette smoking compared to civilians, and this pattern held for gender,

race, education, and marital status subgroups. Civilians

reported heavier alcohol consumption, and this pattern

held with few exceptions across the different subgroups

(Asian and other race groups were not different). Up

through age 40–49, higher rates of depression were

reported by the civilian population. This pattern held

across most of the demographic subgroups with the exception of less educated civilians who reported fewer

symptoms. Active Duty and Reserve/National Guard soldiers were less likely to have visited a healthcare provider

during the study period compared to civilians; Active

Duty and Reserve/National Guard soldiers were more

likely to have had surgery or been hospitalized in the

last year compared to civilians; the latter may reflect

the occupational risks experienced by this group.

Table A3 contains odds ratios and confidence intervals

obtained from a multinomial logistic regression model.

This model provides information on the relative efficiency of the demographic and health-related predictors

to differentiate group membership (reference group is

the NHANES civilian population). Odds ratios less

than 1.0 indicate higher likelihood of being a member

of the civilian population. These results, adjusted for

the other health behaviors and demographics, show a

fairly consistent pattern reinforcing the lower rates of

heavy alcohol consumption, depressive symptoms, and

utilization of healthcare providers, and the much higher

rates of being hospitalized among all soldiers compared

to civilians. Overall, this illustration demonstrates the applicability of the PDE to questions of population health

important for both the military and civilians. Analysis

of epidemiologic patterns within the military can clearly

inform health issues, such as those shown, for military

health policy. Findings from studies of military samples

can have tremendous bearing on the knowledge of civilian health, particularly when variable definitions are

closely aligned and variables can be compared directly.

Appendix References

A1. Ferketich AK, Schwartzbaum JA, Frid DJ, Moeschberger ML. Depression as

an antecedent to heart disease among women and men in the NHANES I

study. Arch Intern Med 2000; 160:1261–1268.

A2. Pierannunzi C, Hu SS, Balluz L. A systematic review of publications

assessing reliability and validity of the Behavioral Risk Factor Surveillance

System (BRFSS), 2004–2011. BMC Med Res Methodol 2013; 13:49.

A3. Mroczek DK, Kolarz CM. The effect of age on positive and negative affect:

A developmental perspective on happiness. J Pers Soc Psychol 1998;

75:1333–1349.

A4. Bush K, Kivlahan DR, McDonell MZB, et al. The AUDIT alcohol consumption questions (AUDIT-C): An effective brief screening test for problem

drinking. Arch Intern Med 1998; 158:1789–1795.

A5. Kroenki K, Spitzer RL, Williams JBW. The PHQ-9: Validity of a brief depression severity measure. JGIM 2001; 16:606–613.

Table A3. Multinomial logistic regression predicting group membership from demographic and health metrics

Active Duty vs. civiliana Reserve/National Guard vs. civilian Overall likelihood-ratio testb

Metric OR CI OR CI p

Age 0.918 (0.918–0.918) 0.948 (0.948–0.948) < 0.0001

Genderc 5.475 (5.412–5.539) 3.302 (3.273–3.331) < 0.0001

Racec 1.722 (1.708–1.737) 2.521 (2.504–2.539) < 0.0001

Educationc 0.223 (0.221–0.225) 0.199 (0.198–0.201) < 0.0001

Marital statusc 2.222 (2.203–2.242) 0.898 (0.891–0.904) < 0.0001

Cigarette used 0.840 (0.835–0.844) 0.825 (0.821–0.829) < 0.0001

Heavy alcohol consumption 0.808 (0.807–0.810) 0.818 (0.817–0.819) < 0.0001

Depression severity 0.824 (0.822–0.826) 0.747 (0.745–0.749) < 0.0001

Seen healthcare providerc 0.228 (0.226–0.230) 0.175 (0.174–0.176) < 0.0001

Hospitalizedc 4.768 (4.709–4.827) 3.150 (3.114–3.186) < 0.0001

a

The multinomial regression simultaneously tests two logit models comparing one of the three groups against the reference category, which is the

Civilian-NHANES group. b

The overall-likelihood test is analogous to testing whether or not all two separate ORs are significantly different from an OR of 1.0, which shows

that the predictor does not efficiently discriminate group membership. c

Reference categories: female; non-white; high school diploma or less; non-married; did not see healthcare provider; not hospitalized.

d

Variable transformed for this analysis to include nonsmokers (0 = 0 cigarettes/day, 1 = £ 10, 2 = 11–20, 3 = 21–30, 4 = 31 + ).

OR, odds ratio; CI, 95% confidence interval.

PERSON-EVENT DATA ENVIRONMENT 13
The Promise of Evidence-Based Policymaking 803

Employer Data Matching Workgroup

White Paper

January 2017
804 The Promise of Evidence-Based Policymaking

1

CONTENTS

Executive Summary...........................................................................................................................................2

Introduction........................................................................................................................................................5

What is an “employer?”......................................................................................................................................8

Matching Employer Data ...................................................................................................................................9

Best Practices .................................................................................................................................................16

Next Steps.......................................................................................................................................................23

Conclusion.......................................................................................................................................................24

Appendices......................................................................................................................................................25
The Promise of Evidence-Based Policymaking 805

2

EXECUTIVE SUMMARY1

Matching and reusing data on employers across Federal government agencies can have multiple and significant 

benefits, but it is currently very difficult to do. In order to begin to address these issues, OMB convened an 

interagency Employer Data Matching Workgroup. The Workgroup was co-chaired by senior members of the 

Census Bureau and the Bureau of Labor Statistics, and included over 40 expert Federal staff across 14 

Agencies, representing 29 program, evaluation, or statistical offices. The scope of this project included 

examining current and potential future methods of matching data at multiple levels (establishment, and firm or 

enterprise), matching parent/child relationships within firms or enterprises, and capturing the dynamic nature of 

these relationships as they change over time. Key tasks included:

• Documenting issues Federal agencies face related to matching and uniquely identifying establishments 

and firms within and between data sets and over time;

• Identifying current successful strategies and/or policies used by Federal agencies to address matching

challenges in the context of analyzing data, conducting evaluations, producing statistics, and identifying 

where additional strategies may be needed to further facilitate this work; and 

• Summarizing potential future steps Federal agencies can take to improve the Federal government’s

ability to identify and match unique firms and establishments (and the relationship between the two) 

within and across Federal data sets, for the purposes of analyzing data, conducting evaluations, and 

producing statistics. 

Matching Employer Data

Many Federal administrative and statistical activities require a matching process. In general, matching activities 

fall into one of the following fundamental types of activities: 

• Finding data on the same entity within a single data set: agencies are de-duplicating and aggregating

data, within the same business level (for example, at the establishment or enterprise level) and within the 

same data set, to find all observations related to a single legal entity. 

• Aggregating data within a corporate structure: agencies aggregate data to the enterprise level of a 

corporate structure in order to group all observations related to a single enterprise.

• Matching microdata at the same business level between two or more data sets for:

o Statistical purposes (including program evaluation): For example, agencies add variables to existing 

data sets to enhance quantitative analyses of firm behavior.

o Programmatic purposes: For example, agencies may use linked data sets to support decision making 

from merged data that better defines market activity, and resulting risks.

• Matching between different business levels in two or more data sets: for example, agencies link 

different business levels to more fully understand corporate structures in the context of successorship, 

franchising, and multisite employers, at a given point or over a period of time. 

Types of matches

There are two types of matching: deterministic and probabilistic:

• Deterministic, or exact, matching, looks for an exact match between two pieces of data. 

1 This white paper is intended to provide the Commission on Evidence-Based Policymaking with background information on topics 

relevant to the Commission’s work. The paper was prepared by staff from OMB and staff from other Federal agencies.
806 The Promise of Evidence-Based Policymaking

3

• Probabilistic, or “fuzzy,” matching uses a statistical approach to assess the probability that two records 

represent the same entity.

Data quality is a key factor in determining which method to use for matching. If data are well-curated, 

deterministic matching is the simpler, more accurate, and faster method, when the two data sets contain the 

same unique identifiers to perform the match. Often, such identifiers are not available, or the identifiers present 

within the data sources do not uniquely identify the entities to be matched. In such cases, deterministic 

matching may still be possible, but only with painstaking research for each case. Probabilistic matching is more 

complex than deterministic matching, but it provides an approach for matching when deterministic matching is 

not feasible. It is often difficult and resource-intensive to evaluate the quality of probabilistic matches.

Challenges in matching employer data

There are two primary issues that drive the vast majority of the challenges in matching data: the lack of a 

common universal identifier for employer units and poor quality of the underlying identifying data.

The greatest barrier to matching data on employers across data sets is the lack of a common, or universal, 

business identifier. Eliminating this obstacle by developing a Federal system to create and manage a universal 

identifier could result in cost savings in matching but would require a major investment of time and Federal 

resources to create and maintain such an infrastructure. Assuming that the identifier could be created, it would 

be a challenge to enforce consistent use of such an identifier by all employers on the domestic and international 

fronts. This identifier would need to capture various corporate/industry levels and change over time (in other 

words, it should change with firm births, deaths, mergers, acquisitions, etc.), and no Federal entity has the 

authority, staff, or resources to collect and manage such information.

There are examples of voluntary, widespread adoption of important taxonomies, such as the North American 

Industry Classification System (NAICS). Given that the creation and use of a universal identifier is likely in the 

best interest of taxpayers and will likely reduce Federal and enterprise burden, it would be worth exploring 

whether a voluntary means of adoption is viable. Such voluntary adoption may also be complicated by the 

nature of the global economy and the domestic and international structure of some employers.

Because there is no universal employer identifier which meets cross-agency needs, agencies often have to 

expend significant resources to research each case for deterministic matching or to obtain data for probabilistic 

matching. That is, agencies can and do combine different data fields to match employer data, but the 

effectiveness of this approach varies based on data fields available and data quality in those fields. Common 

issues include: missing important data fields, inconsistent data formats, and change over time in critical 

matching fields.

Best Practices

There is substantial potential to achieve efficiencies in matching U.S. employer data across Federal data sets for 

data analysis, evaluations, and statistical activities, based on common needs across agencies. To this end, there 

are immediate steps agencies can take to adopt best practices for data collection and matching, which are 

illustrative of the nature of practices Federal agencies have developed to deal with difficult, entrenched 

matching challenges. There are also ways to maximize the use of existing authorities and data sets to enable 

matching. In particular, leveraging data elements common among Federal agencies which, in combination, 

constitute a universal unique identifier, would facilitate efficiencies for matching Federal data sources. 
The Promise of Evidence-Based Policymaking 807

4

Notably, the utility of Federal data sources can be increased by including as many cross-agency identifiers as 

possible. While these fields are useful for matching, it is important to note that identifiers have confidentiality,

privacy, and proprietary concerns. 

Moving forward, there are topics that could benefit from further investigation from the Employer Data 

Matching Workgroup, such as the development of a roadmap for implementing common data elements in 

Federal data sources, the development of an authoritative source of business existence and characteristics for 

Federal agencies, the possible role and value of a centralized data sharing “referee,” and the establishment of a

Federal community of practice for matching and entity resolution.
808 The Promise of Evidence-Based Policymaking

5

INTRODUCTION

The Federal Government currently collects data from employers and enterprises in the United States for a wide 

range of purposes, including administering small business loan programs, administering regulatory 

requirements, and producing valuable economic statistics. While these data collections are valuable and 

frequently necessary, in some cases they can result in the collection of duplicative information from employers.

For example, both the Bureau of Labor Statistics and Census Bureau collect duplicative information on 

businesses with multiple locations.

2

Several agencies have taken steps to attempt to reduce this burden while also increasing the amount and 

usefulness of the information generated from these data collections.3 For example, since the 1990’s, agencies 

within the Federal Statistical System have taken advantage of electronic data and web-based reporting to 

minimize respondent burden and significantly lower data processing costs.

4 Additionally, the Environmental 

Protection Agency (EPA) implemented an Application Programming Interface (API) that allows regulated 

establishments and facilities to take advantage of previously-reported information for querying, retrieving and 

pre-populating future required reports, which not only improved the data quality and the ability to match data 

across multiple programs, but also reduced the annual reporting burden by over 140,000 hours for a single EPA 

program.5

2 The BLS conducts its quarterly Multiple Worksite Report (MWR), which asks most multi-location employers to provide monthly 

employment and quarterly wage data for all of their establishments covered under one Unemployment Insurance (UI) account in a

state. Most multi-location employers with a total of 10 or more employees combined in their secondary locations are required or 

requested to complete the MWR. The Multiple Worksite Report is designed to collect information showing the distribution of the 

monthly employment and quarterly wages of business establishments, by industry and geographic area. Information on the MWR 

form is used to more accurately classify employment and wage data of multiple establishment employers by industry and by location 

within a State. By collecting and storing employment and wage data by worksite, states and the BLS can disaggregate these data 

below the county level for more extensive and detailed analysis of business and economic conditions within their state, including local 

and regional employment totals. These data are used to ensure an equitable distribution of federal funds through grant programs that 

use county economic indicators as a basis for allocations. No other sources are available to obtain this information.

The Census Bureau conducts its annual Company Organization Survey (COS) to obtain similar information on multi-establishment 

firms in order to maintain its Business Register (BR). Annual data collection for the COS begins in late December of the reference 

year for the pay period of March 12th. Reported data are for activities taking place during the reference year. An annual mail-out 

survey of selected companies is conducted for large multi-establishment companies with 500 or more employees are selected with 

certainty. Small multi-establishment and single-establishment companies are selected based on administrative data indicating a 

probable organizational change. All selected companies are identified from those maintained on the Business Register. Survey results 

are available to the Census Bureau about 8 months after each reference year and are used throughout Census Bureau economic data 

program operations, as a major source of information for County Business Patterns reports, and as a resource in responding to requests 

for a variety of special reports and reimbursable tabulations.

Recently, BLS and Census have been sharing their multi-unit data which accrues cost efficiencies and improves the comparability and 

accuracy of Federal economic statistics by improving the consistency of multi-location data and reducing respondent burden.

3 The examples noted represent the current range of burden reduction activities occurring among Federal agencies. By achieving

efficiencies in matching through the white paper’s suggested approaches, future burden reduction is possible (e.g. avoiding new 

surveys).

4 For example, see:

https://www.census.gov/history/www/innovations/data_collection/counting_the_population.html

http://stats.bls.gov/opub/mlr/2016/article/one-hundred-years-of-current-employment-statistics-data-collection.htm 5 EPA Toxic Release Inventory TRI-MEWeb 2.0 Reporting Burden Estimate
The Promise of Evidence-Based Policymaking 809

6

At the same time, there is a growing interest in identifying more efficient and effective ways to help employers 

succeed and comply with Federal requirements. One cost-effective method to generate these insights is by 

analyzing and evaluating the effectiveness of existing programs using the data that government already collects.

Frequently the programs that are being evaluated are designed to help firms grow, ease compliance with Federal 

requirements, or promote innovation. However, the data on the outcomes of interest, whether it is employment 

growth, regulatory compliance, economic growth, or competitiveness, frequently reside in different government 

data sets, across multiple agencies. To capture this outcome information, agencies can either collect the 

information again—at additional cost and burden on employers, or they can get the data from a data set

elsewhere in government.

Similarly, statistical agencies may be able to more cost-effectively and efficiently collect and generate 

economic statistics by re-using already-collected data. Using such data, statistical agencies can establish 

sampling frames which are more cost-effective by accounting for additional information on the sampling units. 

Additionally, improvements in administrative data can reduce burden associated with statistical collection by 

reducing the need for duplicate reporting of information. If statistical agencies can gain access to key data via 

the administrative data that employers already report as a regular part of their business activities, there will be 

less need for the statistical agencies to ask employers for those data again through surveys. Further, combining 

these data can lead to new, improved, and valuable data products and evidence that can benefit employers and

help government develop smarter policies6

.

It is currently very difficult for agencies to access and use this information for statistical, evaluation, or other 

purposes7

. In order for these data to be useful, researchers, statisticians, and evaluators must be able to match 

data on individual employers within the same data set and between different data sets, and they must be able to 

do so reliably and with little error. Access to the data may also be limited by law, such as by Title 26 or the 

Confidential Information Protection and Statistical Efficiency Act of 2002 (CIPSEA). This type of matching 

requires the ability to accurately identify the same employer in two or more different data sets. When 

conducting this type of match for individuals, frequently a combination of unique identifiers is used, including 

the social security number (SSN), name, and date of birth (DOB). However, for employers, there is no standard 

equivalent to SSN, name, and DOB. In some cases this variation in employer identification approach is the 

byproduct of different program rules and requirements. In other cases it is an artifact of the way government 

data systems have developed over time. In all cases, this variation is a large barrier to determining if Federal 

programs and policies affecting employers and their employees are effective, especially when answering such 

questions requires data from multiple data sets. Just a few of the complicating factors involved include:

• Lack of a unique identifier. There is no government-wide policy on which, or even whether, any unique 

identifier should be used across all government data sets that affect employers. Data sets may include 

different identifiers for legal, policy, or programmatic reasons, such as to maintain consistency with 

historical data or to minimize burden on respondents.

• Inconsistent level of identification. Businesses have multiple levels of organization, including the 

establishment (physical location), enterprise (which may consist of multiple establishments), and parent 

company (which may consist of multiple enterprises). There may also be differing needs for ultimate 

domestic corporate parent versus global corporate parent. Depending on the purpose of a program or 

statistical collection, Federal agencies track employer information at different levels of identification, 

and matching across these levels can be difficult.

6 See also: “Using Administrative and Survey Data to Build Evidence.” July15, 2016. 

https://www.whitehouse.gov/sites/default/files/omb/mgmt-gpra/using_administrative_and_survey_data_to_build_evidence_0.pdf 7 See also: “Barriers to Using Administrative Data for Evidence-Building.” July 15, 2016.

https://www.whitehouse.gov/sites/default/files/omb/mgmt-gpra/barriers_to_using_administrative_data_for_evidence_building.pdf
810 The Promise of Evidence-Based Policymaking

7

• Data quality. Data quality issues, such as respondent reporting error, or lack of formatting consistency 

among fields for matching, prevents or complicates the matching processes.

• Timing. The timing for when the data are collected may complicate the matching process as more 

current information may be available for the same unit of observation.

For all of these reasons, it is often highly labor intensive and expensive to match and reuse these data. Matching 

across data sets can take months or even years, and still may not fully serve the intended goals. These problems

occur across statistical, evaluation, and program functions in Federal agencies. While the purpose of matching 

may differ, the matching and analytical challenges are common across these functions.

Overview of the Workgroup

In order to begin to address these issues, OMB convened an interagency Employer Data Matching Workgroup. 

The Workgroup was co-chaired by senior members of the Census Bureau and the Bureau of Labor Statistics, 

and included over 40 expert Federal staff across 14 Agencies, representing 29 program, evaluation, and

statistical offices. The scope of this project included matching at multiple levels (establishment, and firm or 

enterprise), matching parent/child relationships within firms or enterprises, and capturing the dynamic nature of 

these relationships as they change over time.

8

The Workgroup was charged with developing strategies to make it easier to match data on U.S. employers 

across Federal data sets for statistical purposes. Statistical purposes refer to the use of data to better 

understand the characteristics, behavior, or needs of groups.9 Program evaluation falls within this definition. 

Statistical purposes exclude uses of data that affect the rights, benefits, or privileges of individual entities: 

indeed one of the defining characteristics of statistical uses is that data about an individual entity are never made 

public and are never used to make decisions about that entity. But statistical purposes include a wide range of 

analytic uses, where only aggregated and de-identified data are made public.

While the Workgroup focused its activities on these statistical purposes, it became clear that many of the best 

practices identified could have benefits for non-statistical purposes. One of the primary ways this work could 

reduce burden on employers is by facilitating the re-use of individual-level employer identifying data from one 

program to another (e.g., by allowing an employer to select itself from a pre-populated list or auto-loaded list of 

establishments or entities rather than entering the data anew each time). This type of activity requires 

identification and release of information at the individual identity level, so even though the information may be 

otherwise publicly available (e.g., name and street address), it falls outside of the definition of a statistical 

purpose.

Workgroup members have spent many years finding ways to improve matching for their specific purposes. The 

goal of this white paper is to share knowledge and best practices the Workgroup believes would yield 

potentially significant benefits in reducing agency workloads, burden associated with statistical collection, and 

reporting burden for employers.

8 See Appendix D for a detailed description of participating Federal agencies and the Workgroup methodology. 9 Note that a statutory definition of “statistical purpose” exists in section 502(9) of CIPSEA:

“The term 'statistical purpose'-

“(A) means the description, estimation, or analysis of the characteristics of groups, without identifying the individuals or organizations 

that comprise such groups; and

“(B) includes the development, implementation, or maintenance of methods, technical or administrative procedures, or information 

resources that support the purposes described in subparagraph (A).”
The Promise of Evidence-Based Policymaking 811

8

WHAT IS AN “EMPLOYER?”

In order to identify the barriers and potential best practices for matching and reusing data on employers, it is 

first essential to identify what we mean by “employer.” This task is somewhat more complicated for businesses 

than for individuals, given that the identification of a person is fairly stable regardless of context. However, 

there are multiple levels of business that may be of interest. For example, one could be interested in the effect 

of a technical assistance grant on a single business location, or the effect of a small business loan on a firm that 

has multiple locations. 

Terms such as “employer,” “firm,” or “establishment” are often defined differently in statutes and regulations 

across Federal agencies and programs. Many of these terms are based on commonly shared concepts, so while 

precise matching of terms across agencies is not possible, some generalization is both useful and practical. For 

clarity, this white paper uses an internal taxonomy to account for the definitional differences among Federal 

agencies:

• Establishment: a single physical workplace or facility. An establishment is commonly understood as a 

single economic unit, such as a farm, a mine, a factory, or a store, that produces goods or services, for 

which payroll and employment records are kept. Establishments are typically at one physical location 

and engaged in one, or predominantly one, type of economic activity for which a single industrial 

classification may be applied.10

• Legal Entity: a legal person or structure that is organized under the laws of any jurisdiction.11

• Enterprise: Alternative terms for enterprise are firm and company. An enterprise is a legal business 

entity that may consist of one or more establishments. Each establishment may or may not participate in 

a different economic activity. The establishments may have different physical addresses. If they do, the 

physical address of record should be for the headquarters or main office of the enterprise.12

• Parent Company: An enterprise that owns all or the majority (51% +) of another enterprise so that the 

latter stands in relation to the former as a subsidiary.13

• Employer: a legal entity or individual identified as a worker’s employer either nominally (for example, 

on the workers’ paychecks or tax forms) or through an employment relationship. This term can pertain 

to establishments, enterprises, and parent companies.14

Federal data sources and related matching efforts often capture relationships among establishments, enterprises, 

and parent companies, for mission-related analysis and reporting. For example, an establishment-based 

enforcement program might collect all of their data at the establishment level, but in order to analyze trends by 

enterprise or industry, need to be able to aggregate those data. As another example, a statistical agency may 

collect data at the establishment level, but publish tabulations of these data by the size of the parent company.

10 Portions of this definition are derived from the definition of “establishment” within the Bureau of Labor Statistics’ Quarterly Census 

of Employment and Wages (QCEW) Business Register (BR), and the Census Bureau’s Business Register. 11 This definition is derived from the International Organization for Standardization’s definition of Legal Entity Identifier. For further 

information, see: http://www.iso.org/iso/catalogue_detail?csnumber=59771 12 Portions of this definition are derived from the definition of “firm” or “company” within the Bureau of Labor Statistics’ Quarterly 

Census of Employment and Wages (QCEW) Business Register (BR), and the Census Bureau’s Business Register’s definition of 

“enterprise.”

13 From: https://www.dol.gov/vets/contractor/main.htm#20 14 The definition of “employer” under the Fair Labor Standards Act informs this definition.

See also: https://www.dol.gov/whd/regs/compliance/whdfs13.pdf
812 The Promise of Evidence-Based Policymaking

9

MATCHING EMPLOYER DATA

Why is matching conducted?

Many Federal administrative and statistical activities require a matching process. For example, if an agency 

wants to avoid having duplicate observations for an establishment within an administrative database, then it has 

to identify all reports in its data that apply to a specific establishment, and then look to see if there are duplicates 

for certain phenomena. In general, matching activities fall into one of the following fundamental types of 

activities:

• Finding data on the same entity within a single data set: agencies are de-duplicating and aggregating

data, within the same business level (for example, at the establishment or enterprise level) and within the 

same data set, to find all observations related to a single legal entity. For example, agencies regularly 

undertake projects, involving searches within administrative data sources, to identify all instances of 

transactions with particular legal entities. A regulatory agency may search its administrative data to 

determine the compliance history of a particular employer.

• Aggregating data within a corporate structure: agencies aggregate data to the enterprise level of a 

corporate structure in order to group all observations related to a single enterprise. For example, 

agencies will nest companies within larger, related aggregates in order to measure economic activity 

over time. Statistical agencies may aggregate data (e.g. employment counts) for branch locations of an 

enterprise, in order to measure the size of the enterprise.

• Linking/Matching microdata at the same business level between two or more data sets for:

o Statistical purposes (including program evaluation): For example, agencies add variables to existing 

data sets to enhance quantitative analyses of firm behavior. A statistical agency may integrate data 

sources to augment one source with fields not initially contained within the source, such as firm 

age.15

o Programmatic purposes: For example, agencies may merge data sets to provide fuller information on 

market activity and risks to support decision making. A regulatory agency may take advantage of 

merged employer data sources to get a more complete picture of firms and sources of financial risks 

in an industry.

• Linking/matching between different business levels in two or more data sets: for example, agencies 

link different business levels to more fully understand corporate structures in the context of 

successorship, franchising, and multisite employers, at a given point or over a period of time. An agency 

may match data on corporate hierarchies with another employer data set, to further understand the 

employer’s relationships with other legal entities. Also, an agency may tabulate data collected at the 

establishment level by characteristics of parent companies. 

15 Additionally, statistical agencies link microdata at the same business level to develop new data products. For example, BLS’s

QCEW program matched publicly available IRS data on nonprofits with its business register to develop new data on the non-profit 

sector. This initiative is meeting the needs of data users with no new resources while imposing no new respondent burden on 

businesses. These new combined research data covering 2007–2012 were released in September 2014, meeting a longstanding need 

for recent, detailed industry and geographic detail on this large sector of the economy. The nonprofit sector covers about 10% of 

employment and has higher than average wages, making this segment of the economy important to understand. Another BLS 

matching project overlays hurricane flood zones over geocoded business locations. Maps and tables are published and available on 

the BLS website showing the number of establishments, and the accompanying employment and wages that are exposed to potential 

damage under hurricane conditions of varying strengths.
The Promise of Evidence-Based Policymaking 813

10

Types of matches

There are two types of matching: deterministic and probabilistic:

• Deterministic, or exact, matching, looks for an exact match between two pieces of data. In order for 

this method to be effective, the data being matched should uniquely identify the entity of interest, and 

the same data field should be present and formatted in the same way in both records. This method is 

ideal if the data sets of interest are reliably and accurately collecting unique identifiers, and the unique 

identifiers are well-matched to the purpose of the analysis.

• Probabilistic matching uses a statistical approach to assess the probability that two records represent the 

same entity. In order to accomplish this, a set of data fields are compared between two records and the 

closeness of the match between two record pairs is assessed.

Data quality is a key factor in determining which method to use for matching. If data are well-curated, 

deterministic matching is the simpler, more accurate, and faster method, when the two data sets contain the 

same unique identifiers to perform the match.

16 Often, such identifiers are not available, or the identifiers 

present within the data sources do not uniquely identify the entities to be matched. In such cases, deterministic 

matching may still be possible, but only with painstaking research for each case. Probabilistic matching is more 

complex than deterministic matching, but it provides an approach for matching when deterministic matching is 

not feasible. It is often difficult and resource-intensive to evaluate the quality of probabilistic matches.

Deterministic and probabilistic matching methods have the potential to produce data structures that give 

additional insights the original data could not have provided. Several agencies and offices reported the use of 

matching methods to turn administrative data sources, with multiple discrete observations of employers or 

establishments, into quasi-longitudinal or time series data, by linking observations over time. Similarly, there 

are matching applications that can link subsidiaries to parents or nest companies within larger, related 

aggregates.

Current unique identifiers used across Federal Agencies

Several entities have created unique identifiers (IDs) in order to improve matching and identification of 

employers, which are used across multiple Federal Agencies. However, none of these IDs are universally 

collected, and none of them uniformly identify the level of business or the relationship between the levels of 

business. There are four primary unique identifiers currently in use by Federal Agencies: the Employer 

Identification Number (EIN), Data Universal Numbering System (DUNS©) numbers, Commercial and 

Government Entity (CAGE) codes, and the Legal Entity Identifier (LEI). To apply for a Federal contract or 

grant, an entity must have an EIN, DUNS© number, and CAGE code. However, these identifiers are not 

currently used throughout all Federal data sets as only a small percentage of U.S. enterprises register annually 

for Federal contracts or grants. These identifiers enable clear identification of unique entities seeking Federal 

dollars and are used to identify exclusions, past performance history, and business integrity. Additionally, the 

LEI, as required under Public Law 111-203 (commonly known as the Dodd-Frank Wall Street Reform and

Consumer Protection Act), is heavily used within the federal financial regulatory community, and by nonfinancial regulatory agencies.17

16 See also: https://www.healthit.gov/archive/archive_files/FACA%20Hearings/2010/2010-12-

09%20Patient%20Linking/Probabilistic%20Versus%20Deterministic%20Data%20Matching.pdf 17 For a complete list of Federal agencies currently using LEI, see: https://www.gleif.org/en/about-lei/regulatory-use-of-the-lei.
814 The Promise of Evidence-Based Policymaking

11

• The Employer Identification Number (EIN) is issued by the IRS. Under the Internal Revenue Code 

(IRC), every US entity is required to have an EIN for tax purposes regardless of whether they have 

Federal contracts or grants. An enterprise subject to Federal income tax will file using this EIN on its 

own separate tax return or file under the Parent Company’s EIN of a consolidated return if it elects to 

file with an affiliated group of other enterprises.18 Furthermore, establishments can be associated with 

multiple EINs or an enterprise could use the same EIN for all its establishments. 

• A Data Universal Numbering System (DUNS©) number is a uniform and unique nine-digit number 

administered by Dun and Bradstreet (D&B). The number is assigned by D&B and is currently collected 

by the Federal government as part of the registration process for grants and contracts.

• A Commercial and Government Entity (CAGE) code is a uniform and unique five-character alphanumeric identifier for entities in the U.S. CAGE codes are used internationally as part of the North 

Atlantic Treaty Organization Codification System (NCS). Management of CAGE codes in the United 

States is done by the Department of Defense. If an employer has not applied for Federal contracts or 

grants, the entity would not need a CAGE code and thus may not have one.

• The Legal Entity Identifier (LEI) is a 20-digit, alpha-numeric code based on the ISO 17442 standard 

developed by the International Organization for Standardization (ISO). It connects to key reference 

information that enables clear and unique identification of legal entities participating in financial 

transactions, such as those participating in financial transactions or when used in regulatory and 

supervisory reporting.

19

There are a number of other intra-Agency identifiers which Federal Agencies use for internal databases, or for 

limited cross-agency coordination. For example, the State Unemployment Insurance Account Number (UI 

number) is issued by state unemployment insurance agencies. The state identification number is assigned by 

each state to identify employers covered by State UI laws or to identify federal government installations 

covered by Unemployment Compensation for Federal Employees (UCFE) provisions. UI account numbers are

utilized at the federal level to identify establishments maintained in the BLS’s Quarterly Census of Employment 

and Wages (QCEW) files. This field is consistent from quarter to quarter and allows for identification of the 

same unit over time.

Challenges in matching employer data

There are two primary issues that drive the vast majority of the challenges in matching data: the lack of a 

common universal identifier for employer units, and poor quality of the underlying identifying data.

Universal Employer Identifiers

Agencies lack a single, universal identification system for establishments, firms and other types of employer 

units in Federal and non-Federal data. In the ideal system, such an identifier would be hierarchical such that 

each employer unit, at each level (i.e., establishment, enterprise, parent company) would have its own unique 

ID, and the set of identifiers would be used together to identify the relationships among the levels over time.

18 Further, some foreign entities that are US-owned have an EIN for Federal tax purposes. 19 “Introducing the Legal Entity Identifier (LEI).” Global Legal Entity Identifier Foundation. <https://www.gleif.org/en/aboutlei/introducing-the-legal-entity-identifier-lei>. Accessed 12/9/2016. See also: http://www.leiroc.org/lei.htm.
The Promise of Evidence-Based Policymaking 815

12

Thus a single establishment could be tracked not only across multiple data sets, but also as it is sold from one 

enterprise to another.20

If such an infrastructure existed, linking would be simple, as deterministic matching with a single data element 

would be feasible, and connections among different business levels over time would be implicit in the 

identification system. Developing such an infrastructure could result in cost savings over time for both 

statistical and program agencies, though more work needs to be done to compare the costs and benefits of 

developing this infrastructure.

21 Such a system would also enable an employer to provide a piece of data to 

government only once, rather than multiple times, subject to any legal limitations on sharing of the data. A

primary challenge with such a system is maintaining the accuracy and currency of the underlying firm or 

employer data. While matching would be simple, verifying the mergers, acquisitions, incorporations, and 

continuous changes would require significant domestic and international resources or connections. Absent such 

due diligence, employers could change incorporation or legal structure, and obtain new identifiers. Assigning 

improper activity, sanctions, or other such determinations based on inaccurate identification could have serious 

repercussions.

There is great variability among statutes, regulations, agency policies, and reporting definitions in how 

employer units are identified. This variation largely comes from differences in laws and policies as they apply 

to different subsectors of business. For example, the Fair Labor Standards Act of 1938 (FLSA), which 

“establishes minimum wage, overtime pay, recordkeeping, and child labor standards affecting full-time and 

part-time workers in the private sector and in Federal, State, and local governments,”22 defines “employer” to 

include “any person acting directly or indirectly in the interest of an employer in relation to an employee and 

includes a public agency, but does not include any labor organization (other than when acting as an employer) 

or anyone acting in the capacity of officer or agent of such labor organization.”23 In comparison, the 

Occupational Safety and Health Act of 1970 (OSH Act), which “assure[s] safe and healthful working conditions 

for working men and women,” defines “employer” to mean “a person engaged in a business affecting 

commerce who has employees, but does not include the United States (not including the United States Postal 

Service) or any State or political subdivision of a State.”24

The statutes have additional differences, such as differences in coverage. A 2002 GAO study noted when 

comparing the FLSA and OSH Act that: “Coverage under the OSH Act is broader. All employees of a particular 

employer are covered if the employer is engaged in a business affecting commerce. Coverage under the OSH 

Act does not depend on the specific activities of the employee or the volume of the employer’s business.”25

This variability has obvious ramifications for what can reasonably be expected from matching data sources 

from very different programs and approaches to defining “employment,” “employer,” and “employee,” as well 

as corporate structures such as “firm,” “establishment,” and “enterprise.” 26 Many of these inconsistencies 

would still present obstacles to matching even if the government were to develop a common conceptual 

standard and definition for the unique identifiers. 

20 More specifically, databases with such a universal unique identifier would ideally contain a history for each establishment of its 

relationships over time (i.e. its history of hierarchical identifiers). That database would allow for tracking of the establishment over 

time even if it changed owners and relationships.

21 Costs could include, for example, staff to research the incorporation or legality of an entity, build corporate family trees, and track 

changes among corporations. Benefits could include elimination of costs related to poorly matched or unmatched data. 22 See: https://www.dol.gov/whd/regs/compliance/hrg.htm 23 See: https://www.dol.gov/whd/regs/statutes/FairLaborStandAct.pdf 24 See also: https://www.osha.gov/pls/oshaweb/owadisp.show_document?p_table=OSHACT&p_id=2743 25 From: http://www.gao.gov/assets/240/235612.pdf 26 The paper contains an internal taxonomy to ensure that there is a common set of definitions to support its description of best 

practices.
816 The Promise of Evidence-Based Policymaking

13

In addition, considering the variations in statutes, policies, and uses, creation of a new Federal infrastructure 

that would establish, maintain, and assign identifiers to commercial and Federal entities would require 

considerable effort. Such a change would require the establishment of a Federal program to manage the 

identifiers. In addition, such a change could require changes in statute to harmonize definitions of different 

employer units as well as modifications to policy and numerous data collection and maintenance systems across 

Federal government agencies. Such changes in program definitions would need to be handled in a manner so as 

not to disrupt the continuity of longitudinal data that is already being collected. Also, regulatory agencies could 

not leverage confidential data sources within Federal statistical agencies even if those sources contain a

universal identifier.

Beyond cost and conceptual challenges, it would also be difficult to enforce consistent use of such an identifier 

by all employers within current authorities. There is no single Federal agency that would be a natural owner to 

enforce the use of such an identifier because no Federal entity currently collects all of this information. There 

are examples of voluntary, widespread adoption of important taxonomies, such as the North American Industry 

Classification System, or NAICS. However, this is a very broad standard that is applied to classes of 

enterprises and employers rather than used to uniquely and uniformly identify individual employers. Given that 

the creation and use of a universal identifier is in the interest of businesses and taxpayers, it would be worth 

exploring whether a voluntary means of adoption of a universal identifier is viable.27

Moreover, there are ways to maximize the use of existing authorities and data sets to enable matching. In 

effect, these probabilistic methods rely on a variety of existing, widely collected data fields, which when 

combined can effectively create a flexible “universal identifier” that can be adapted to different applications.

These methods generally require the data to be of high quality and sufficiently standardized in order to be 

effective.

Because there is no universal employer identifier which meets cross-agency needs, agencies often have to 

expend significant resources to obtain data for probabilistic matching. Agencies currently deal with challenges 

related to:28

• Resource and capacity constraints. Acquiring data housed by other agencies can be costly and difficult 

for Federal agencies. The process of developing interagency agreements can be time-intensive, and 

agencies require employees with the proper skill set and time available to reconcile data with different 

definitions. Also, because many agencies do not have a complete and current data inventory, they may not

be fully aware of data that is available to support their particular needs. After receiving the data, agencies 

may not know the usefulness of the available data until after significant efforts to review the source,

depending on the availability of relevant documentation and metadata. Additionally, infrastructure may be a

barrier to obtaining data. For example, if the construction of an agency’s own or preferred data center is 

considered sufficient for one survey but not another (e.g., differences in requirements for physical security 

and data access protocols between two data sources might prohibit linkage), an agency may have to expend 

significant resources to determine an alternative strategy for accessing data. Lastly, the cost of purchasing 

external data sources can serve as a practical barrier for agencies with limited funding available for such 

purchases.

27 A review of voluntary adoption would require the selection or scoping of a prospective identifier. The Workgroup puts forward a 

framework for Federal data sources to include existing, widely collected data fields, which when combined can effectively create a 

flexible “universal identifier” that can be adapted to different applications. 28 See also: https://www.whitehouse.gov/sites/default/files/omb/mgmtgpra/barriers_to_using_administrative_data_for_evidence_building.pdf
The Promise of Evidence-Based Policymaking 817

14

• Legal barriers. While there are Federal data sets containing information that can be valuable to regulatory 

agencies, limitations on the use of such data frequently restrain access. For example, under CIPSEA, data, 

including business data, acquired by an agency under a pledge of confidentiality and for exclusively 

statistical purposes can only be used by officers, employees, or agents of the agency exclusively for 

statistical purposes. Lacking access to an authoritative source on business existence and structure is a 

critical issue for correctly matching data, because agencies have no way to know if the data they are 

collecting is fully accurate. An authoritative source would be useful to serve as a “spine” to match data 

against.

• Policy and legal interpretations. At the agency level, there can be confusion in the interpretation of 

statutes such as CIPSEA as well as other policies on data sharing, which may create additional barriers for 

agencies trying to access and share Federal data sources for matching purposes. Agencies may spend 

months or years coming to agreement on the proper interpretation of a particular statute or policy, and to 

develop interagency agreements to allow matching. OMB’s M-14-06, Guidance for Providing and Using 

Administrative Data for Statistical Purposes29, encourages Federal departments and agencies to promote the

use of administrative data for statistical purposes and provides guidance in addressing legal and policy 

requirements for such uses. It creates “a presumption in favor of openness to the extent permitted by law 

and subject to privacy, confidentiality, security, or other valid restrictions.”

Data quality

Agencies can and do combine different data fields to match employer data, but the effectiveness of this 

approach varies based on data fields available and data quality in those fields. These issues can prevent 

deterministic matching for a number of reasons:

• Missing important data fields: Agencies frequently do not collect enough of the most important pieces 

of data to enable matching, and so they do not engage in data matching for those data sets. Federal 

agencies have generally built data collections for narrow and specific purposes, designing definitions, 

content, and formats to suit the immediate needs of the program performing the collection. While most 

sources with employer data include common data elements such as name, address, and basic 

characteristics such as NAICS code, these fields are not sufficient to match in all cases. For example, a

match can be extremely difficult if not impossible to complete when data in the first data set is at the

corporate level, but the matched data set is at the establishment level and does not include fields linking 

the establishments to their corporate hierarchies, such that neither data set has sufficient fields to create a 

cross-walk between the two. In some cases, agencies may have sufficient data to match on a 

deterministic or probabilistic basis, but may not have sufficient supplementary data to assess how good 

the match is.

• Inconsistent data formats: Federal agencies often have to deal with data quality issues arising from 

inconsistent data format standards. To cite a fairly common example, establishment names can vary 

considerably within and between data sources. Most data sources collect this information in free-text 

fields without edit checks, and allow variation. In extreme cases, establishment names can vary 

considerably, with a single Post Office branch being identified as “USPS”, “US Postal Service”, “United 

States Postal Service”, and “USPS - MAIN STREET SOUTHBEND.” In this case, deterministic

matching of data sources by establishment name will not work, and even probabilistic matching methods 

may not work completely. Also, there is significant variation in data quality due to a lack of internal 

checks or naming conventions.

29 https://www.whitehouse.gov/sites/default/files/omb/memoranda/2014/m-14-06.pdf
818 The Promise of Evidence-Based Policymaking

15

• Change over time in critical matching fields: With both deterministic matching and probabilistic 

matching, a common assumption is that the fields are static over time—that is the field is collecting the 

same information in year one as it is in year ten. However, there are a number of instances in which this 

is not the case, where fields are redefined over time. The business universe is dynamic, and issues such 

as relocation, mergers, successorship, and firm births/deaths produce a mismatch between data 

collection and changes in business status. For example, this issue arises in analyses of young, small 

businesses which may not have been in existence long enough to meet annual reporting requirements for 

collections. Evolving classification systems, such as NAICS and LEI, also present a unique challenge in 

matching data sources where observations occurred over a period of many years.
The Promise of Evidence-Based Policymaking 819

16

BEST PRACTICES

The Workgroup identified a number of practices that agencies could adopt within current authorities that have 

the potential to reduce burden on employers and agency staff while also facilitating better use of information. 

The Workgroup prioritized best practices based on the following criteria:

• Efficiency or Cost-Effectiveness: Examine the levels of effort and pecuniary costs of implementing

each solution, and select the most efficient and cost-effective options.

• Applicability: Select the most useful methods, fields, standards, and data sets. 

• Relevance to Scope: Use the methods, standards and data sets that are the most relevant to improving 

the matching process.

• Alignment: The Workgroup emphasizes best practices which best align to existing guidance and data 

standards, including guidance from, but not limited to, the National Information Exchange Model 

(NIEM), the International Organization for Standardization (ISO), and OMB memo M-13-13, “Open 

Data Policy – Managing Information as an Asset.” 

The Workgroup identified four practices that Federal programs and agencies may find useful in making it easier 

to match employer data:

Best Practices for Matching Data

The Workgroup identified a set of collection and algorithmic approaches that constitute the best practices in 

matching data on employers. Individuals conducting matches could use these methods rather than attempting to 

create a new method each time a match is to be conducted. The specific methods that produce the best results

vary depending on the purpose of the match and the data fields that are available. These methods are described 

in greater detail in Appendices B and E, and a related bibliography is available in Appendix C.

It is worthwhile to note that the quality of a match depends largely on the quality of the underlying data in each 

data set. Before conducting a match, individuals should engage in data cleaning and standardization. In 

addition, employing a range of matching algorithms is critical to obtaining optimal matching results. Finally, 

individuals conducting a match should take advantage of data quality analysis; improvements in business rules,

which improve data quality (by permitting data to be entered into a system if they meet certain criteria); and 

improved use and creation of data documentation to improve matching outcomes.

Data Inventory

One of the common challenges to matching employer data is the identification of data sets that are available for 

matching and what those data sets contain. To ease this identification process, the Workgroup identified a

representative sample of the most frequently desired or most valuable data sets that the government currently 

collects. These data sets contain information on individual employers, firms, and/or establishments that have the 

widest coverage, greatest use, or which agencies are most interested in matching to other data sets. This data 

inventory is included as an attachment to the white paper. The data inventory includes information on the data 

sources’ coverage of U.S. businesses, collection methodology, access restrictions, and information related to 

common fields for matching. CIPSEA limits the use of many of these data sources exclusively to statistical 

purposes.
820 The Promise of Evidence-Based Policymaking

17

Common Data Fields

Federal agencies can benefit from adopting the common data fields for matching purposes detailed in Table 1.30

These fields are most commonly used in the methods described above and in Appendix B. Table 1 also 

accounts for existing guidance from other data-sharing initiatives (such as for Federal spending transparency 

data standards, and the National Information Exchange Model, or NIEM), and lessons learned from prior 

studies linking administrative and statistical data sources.31 32 33 These fields can provide an exact match, and 

aid in matching even if an exact match is not possible due to variance amongst data sources.

Table 1 proposes a set of “Tier 1” fields, which are the most essential fields for matching and entity resolution.

Agency data sources lacking these fields are very difficult to match, and the Workgroup believes Federal 

Agencies can achieve potential cost savings if their data sources universally incorporated Tier 1 fields. It is 

possible to compensate for missing fields by creating a matching profile from a combination of Tier 1 and nonTier 1 data elements. Please note the following considerations for the Tier 1 fields:

• Establishment-Focused: Employer data is collected at various organization levels. The proposed 

framework emphasizes establishment names and addresses to enable matching with potential 

authoritative sources described in Appendix A. However, many Federal employer data sources do not 

always use establishment as the unit of analysis. Placeholders for other fields to describe organization 

level are included as Tier 2 fields.

• Treatment of Identifiers: Generally, the utility of Federal data sources in regards to matching, linkage, 

and reuse can be increased by including as many common identifiers as possible. Table 1 therefore 

includes a number of data elements to reflect this prioritization. Additionally, while these fields are 

useful for matching, it is important to note that identifiers have confidentiality, privacy, and proprietary 

concerns. Where possible, Federal agencies should include non-sensitive and non-proprietary 

identifiers. Ideally, Federal agencies would institute policies and processes that support consistent 

identification of employers within an agency. This would minimize the data cleansing necessary to 

identify the same employer involved in programs across Federal agencies. 

These fields are generally common among 39 data sources in the data inventory. Federal data sources within 

this categorization commonly have multiple Tier 1 data fields available for matching, including: at least one 

identifier, legal and trade names, establishment physical location and mailing addresses, country codes, and data 

timestamps. Table 1 also shows web sites and e-mail addresses as Tier 1 fields.

Additional helpful fields include: information on the ultimate parent company, NAICS codes, latitude and 

longitude, and phone number. A number of these fields are proposed as Tier 2 fields in Table 1, as they inform 

30 Due to the variation in Federal agencies and programs, this Workgroup is not currently prescribing a process for agencies to use to 

implement the Common Data Elements, but a future effort could look into this. 31 For further information, see “Federal Spending Transparency Data Standards” at MAX.gov, available at: 

https://max.gov/maxportal/assets/public/offm/DataStandardsFinal.htm

32 NIEM is “a community-driven, standards-based approach to exchanging information.” The U.S. Department of Homeland Security 

the U.S. Department of Justice, and the U.S. Department of Health and Human Services are the stewards of NIEM. or add a qualifier 

and data source for “majority”

See: https://www.niem.gov/aboutniem/Pages/niem.aspx https://www.niem.gov/aboutniem/Pages/history.aspx 33 For example, see:

Krizan, C. J., Statistics on the International Trade Administration's Global Markets Program (September 1, 2015). US Census Bureau 

Center for Economic Studies Paper No. CES-WP-15-17. Available at SSRN: http://ssrn.com/abstract=2661478 or 

http://dx.doi.org/10.2139/ssrn.2661478 
The Promise of Evidence-Based Policymaking 821

18

and enhance matching. A number of agencies do not have information on the ultimate parent or intermediate 

corporate entities in their data sources, but having this information would allow for hierarchical analysis (e.g. 

having all establishment identifiers each parent company uses). A third set of data elements are Tier 3 fields as 

they further support validation: the age of the firm in years, and the number of individuals employed by the 

firm.

Table 1: Initial Proposal for Common Data Fields

Priority Field(s) Field Definition

Examples of Relevant 

NIEM Core Type/SubProperties34

Tier 1 Identifiers

As many of the following elements as possible or 

feasible:

• Employer Identification Number (EIN)

• D-U-N-S Numbers

• Legal Entity Identifier (LEI)

• Commercial and Government Entity 

(CAGE) codes

• Other cross-agency identifiers (e.g. UI 

account number)

• Other Non-Confidential, Non-Proprietary 

Identifiers 

nc:OrganizationType

nc:OrganizationIdentificati

on

nc:OrganizationOtherIdenti

fication

nc:IdentificationType

nc:IdentificationID

Tier 1 Legal Name Legal Name of establishment.

nc:OrganizationType

nc:OrganizationName

nc:OrganizationBranchNa

me

Tier 1 Trade Name Trade Name, “Operating As” Name, or DBA of 

establishment.

nc:OrganizationType

nc:OrganizationDoingBusi

nessAsName

Tier 1 Establishment Physical 

Location Address

The address is made up of six standardized 

components: Street Number, Street Name, and 

Building/Suite, City, State Code, and ZIP+4 or 

Postal Code. The address should follow the United 

States Postal Service’s standardized address format 

(fully spelled out, abbreviated by using the Postal 

Service standard abbreviations or as shown in the 

current Postal Service ZIP+4 file). See also: 

http://pe.usps.gov/text/pub28/28c2_001.htm

nc:OrganizationType 

nc:OrganizationLocation

nc:LocationType

nc:Address

nc:AddressType

nc:AddressFullText

Tier 1 Establishment Mailing 

Address

The address is made up of six standardized 

components: Street Number, Street Name, and 

Building/Suite, City, State Code, and ZIP+4 or 

Postal Code. The address should follow the United 

States Postal Service’s standardized address format 

(fully spelled out, abbreviated by using the Postal 

Service standard abbreviations or as shown in the 

current Postal Service ZIP+4 file). See also: 

http://pe.usps.gov/text/pub28/28c2_001.htm

nc:OrganizationType 

nc:OrganizationLocation

nc:LocationType

nc:Address

nc:AddressType

nc:AddressDeliveryPoint

Tier 1 Establishment Physical 

Location County Code

County codes from US Census and the American 

National Standards Institute (ANSI INCITS 

31:2009). Available at:

https://www.census.gov/geo/reference/codes/cou.ht

ml

nc:AddressType

nc:LocationCounty

census3.0.1:USCountyCodeType

34 From NIEM 3.2 (current release). Available at: https://www.niem.gov/technical/Pages/current-release.aspx

“NIEM core consists of data elements that are commonly understood and defined across domains, such as person, activity, document, 

location, and item. It’s governed jointly by all NIEM domains.”

From: https://www.niem.gov/technical/Pages/The-Model.aspx
822 The Promise of Evidence-Based Policymaking

19

Priority Field(s) Field Definition

Examples of Relevant 

NIEM Core Type/SubProperties34

Tier 1 Country Code

Country Code as defined by ISO 3166 (codes for 

the countries, dependent territories and special 

areas of geographical interest), FIPS 10-4, or 

Geopolitical Entities, Name and Codes (GENC).

nc:AddressType

nc:LocationCountry

nc:LocationCountryFIPS10

-4Code

nc:LocationCountryGENC

Code

nc:LocationCountryISO316

6Alpha2Code

Tier 1

Time Stamp of Collection,

Time Stamp when 

observation Last Edited, and

Lag Time.

Date and Time of collection,

Date and Time when an observation was last edited, 

and

Amount of Time between the reference period and 

time the data are available. 

nc:DateRepresentation

nc:MetadataType

nc:ReportedDate

nc:LastUpdatedDate

nc:DateRangeType

Tier 1 E-mail Address A corporate E-mail address for respondent.

nc:OrganizationType

nc:OrganizationPrimaryCo

ntactInformation

nc:ContactInformationTy

pe

nc:ContactEmailID

Tier 1 Web Site URL of web site for entity.

nc:OrganizationType

nc:OrganizationPrimaryCo

ntactInformation

nc:ContactInformationTy

pe

nc:ContactWebsiteURI

Tier 2 Telephone Number Telephone number for entity.

nc:OrganizationType

nc:OrganizationPrimaryCo

ntactInformation

nc:ContactInformationTy

pe

nc:ContactTelephoneNumb

er

Tier 2 Other Intermediate Entity 

Legal Name (If Applicable)

Legal Name of Other Intermediate Corporate 

Entity.

nc:OrganizationType

nc:OrganizationName

Tier 2 Other Intermediate Entity 

Identifier

Identifiers of Other Intermediate Corporate Entity 

(see above guidance on Identifiers)

nc:OrganizationType

nc:OrganizationName

nc:OrganizationType

nc:OrganizationIdentificati

on

nc:OrganizationOtherIdenti

fication

nc:IdentificationType

nc:IdentificationID

Tier 2 Ultimate Parent Entity Legal 

Name Legal Name of Ultimate Parent.

nc:OrganizationType

nc:OrganizationParent

nc:OrganizationParentAffili

ate

nc:OrganizationParentOrga

nization

nc:OrganizationType

nc:OrganizationName
The Promise of Evidence-Based Policymaking 823

20

Priority Field(s) Field Definition

Examples of Relevant 

NIEM Core Type/SubProperties34

Tier 2 Ultimate Parent Identifiers Identifiers of Ultimate Parent (see above guidance 

on Identifiers)

nc:OrganizationType

nc:OrganizationParent

nc:OrganizationType

nc:OrganizationIdentificati

on

nc:OrganizationOtherIdenti

fication

nc:IdentificationType

nc:IdentificationID

Tier 2 NAICS Code Six-Digit NAICS Code. 2017 NAICS revision.35 --

Tier 2

Latitude and Longitude of 

Establishment Physical 

Location Address

Latitude and Longitude of Establishment Physical 

Location Address, aligned to ISO 6709 and Federal 

Geographic Data Committee (FGDC.gov) 

established standards.

nc:OrganizationType 

nc:OrganizationLocation

nc:LocationType

nc:LocationGeospatialCoor

dinate

Tier 3 Establishment Age Age of establishment in years.

nc:DateRangeType

nc:OrganizationType

nc:OrganizationIncorporati

onDate

Tier 3 Employment

Count of employees who are on the payroll, ideally 

for the pay period including March 12 of each 

year.36 37

--

The fields in Tier 1, Tier 2 and Tier 3 vary in importance for individual offices. The Tiers shown in Table 1 

reflect consensus on prioritized fields, but offices should account for variations. For example:

• Tier 2 elements become Tier 1 elements in establishment-based surveys. For example, phone numbers 

become an important Tier 1 field for not only identifying individual stores within an enterprise with one 

EIN, but also, having appropriate contact information for each store for future data collection activities. 

When matching on a city, there are many instances where the postal address is different from the 

municipal location. For example, New Jersey townships do not relate to what the postal service calls the 

location. Having additional fields, such as the County Code, further assist in matching.

• Fields in Tiers 2 and 3 may take on particular importance for regulatory activities. For example, it may 

be critical for a program to have information on intermediate and parent entities in order to engage the 

proper stakeholders for a compliance activity. Or, if an agency undertakes an investigation of a business 

that does not have an office, fields related to contact information may become critical for identification 

purposes.

• Also, some Tier 1 elements, such as the corporate e-mail address (following the “@” section) and/or 

Web site can be used to some extent as a self-identified definition of the “firm”.38

35 http://www.census.gov/eos/www/naics/federal_register_notices/notices/fr08au16.pdf 36 Derived from: http://www.census.gov/programs-surveys/cbp/about/glossary.html 37 In instances where an entity has no employees, this field is equal to zero. 38 For more information, see:

Krizan, C. J., Statistics on the International Trade Administration's Global Markets Program (September 1, 2015). US Census Bureau 

Center for Economic Studies Paper No. CES-WP-15-17. Available at SSRN: http://ssrn.com/abstract=2661478 or 

http://dx.doi.org/10.2139/ssrn.2661478
824 The Promise of Evidence-Based Policymaking

21

Federal agencies also want standardization and validation approaches for the Common Data Elements shown in 

Table 1, to maximize the quality of the data and ease of matching across Federal data sources.

• The table references NIEM Core data elements to take advantage of existing data standards, to which a 

number of Federal agencies have already agreed, for the Common Data Fields.39

• Table 1 shows Establishment Physical Location Address, and Establishment Mailing Address, with data 

entry of the street number, street name, and building/suite in separate variables, to expedite data 

cleaning. Table 1 also recommends use of the United States Postal Service’s standardized address 

format for these fields.

• The data elements Legal Name, Trade Name, Other Intermediate Entity Legal Name, and Ultimate 

Parent Entity Legal Name should follow common formatting and validation standards to facilitate 

matching. For example, agency processes should account for legal entities which have a slightly 

different legal name in each state. Due to the range of approaches, which depend on project goals, the 

workgroup is not able to recommend a single standard or method. Rather, agencies should take 

advantage of existing methods for standardization and validation, which are further discussed in 

Appendix B. Also, Federal agencies should explore use of resources that help delve into correct names 

as well as legal filings.40

Data Collection Improvements

Federal agencies can achieve potential cost savings in data matching and reuse when they improve the quality 

of data at the point of entry. They accomplish this with changes in collection methods, subject to any legal 

requirements (e.g., the Paperwork Reduction Act of 1995), or with validation steps at the time of input. There is

a range of methods for improving the quality of fields at the point of collection, resulting in higher quality data, 

fewer resources expended to clean data, and burden reduction for the regulated community.

First, agencies benefit from ensuring that they have clear reporting guidance for data collection activities to 

minimize variability in data quality. Robust and clear guidance, combined with a collection tool which 

reinforces the guidance, ensures that agencies minimize efforts in cleaning data.

Additionally, Application Programming Interfaces (APIs) for validating data have been proven to significantly 

improve data quality for matching while simultaneously reducing reporting burden. APIs on the front and back 

ends of data entry are highly useful for validating and standardizing the common data elements shown in Table 

1. For example, the U.S. Environmental Protection Agency’s Facility Registry Service (FRS), which integrates 

data on over 4 million establishments and places of interest from across 90 different systems, built an API 

which took advantage of existing, previously-collected data from internal data sources for entity resolution at 

the point of data collection.

41 The API allows reporters to identify establishments by searching and retrieving 

information that was previously reported via other systems from which the FRS ingests data. When a user 

searches for a particular firm, the application then allows newly reported information to be associated with 

existing, known establishment information and identifiers. This interface has been used for other EPA 

programs, such as the Toxic Release Inventory (TRI).

42 Within the TRI program, the EPA has seen significant 

data quality improvements and has estimated a burden reduction of 140,269 hours for reporters.

39 See also: NIEM 3.2 (current release). Available at: https://www.niem.gov/technical/Pages/current-release.aspx 40 One example of this is OpenCorporates, which scrapes the legal filings from the various State Departments of State. See: 

https://opencorporates.com/. 41 “Facility Registry Service,” US Environmental Protection Agency, Office of Environmental Information, Retrieved 09 Sep 2016

https://epa.gov/frs 42 US EPA Toxic Release Inventory https://www.epa.gov/toxics-release-inventory-tri-program
The Promise of Evidence-Based Policymaking 825

22

Lastly, agencies benefit from having a strong understanding of respective data sources to be matched, 

anticipating data quality or consistency issues, and estimating the likely overlap in records for a point of 

comparison. Many agencies also provide points of contact who can discuss unusual or anomalous results, as 

well as steps they routinely use to optimize any attempted comparisons using their data. Common practices 

include:

• accounting for context in databases (such as knowing the context of the letters you are matching in a 

name), 

• ensuring metadata is sufficiently descriptive to differentiate fields, and

• using forms with specialized functionalities (e.g. SmartForm) to validate and standardize data entered.

Authoritative Source

An authoritative source is a data source that provides current, accessible, and authoritative information on all 

data of a certain type. An authoritative source for data on business existence and characteristics would greatly 

improve the matching process, and could aid in reducing burden on employers. Such a source would contain 

validated information confirming existing or closed businesses, and characteristics for businesses such as 

geographic location, contact information, size, and industry. Additionally, this source would contain data 

capturing relationships among different levels of corporate and industry structures, and would be able to provide 

information on at least a quarterly basis on changes to these relationships. Making an authoritative source 

accessible would drastically improve the ability to validate data and conduct matches. It could also serve as the 

basis of a universal identifier in the future. Having an authoritative source would be useful for a variety of

purposes including, but not limited to:

• Entity resolution, including identity verification: Having an authoritative source for cleaning or 

matching existing data, both during data entry (for example, confirming the identity of a reporting 

entity) and after (for example, when trying to match two data sets, executing resolution algorithm to 

show that two entities are actually the same), would reduce the time needed to reconcile data sources

when matching them.

• Sampling: An authoritative source could enable Federal agencies to build valid sampling frames for 

surveys or program evaluations without having to rely on proprietary data sources.

• Administrative purposes: Authoritative sources could also assist with statistical processes within 

regulatory agencies, such as creating more accurate or precise estimates of various administrative or 

economic measures used as inputs for assessing agency performance (for example, examining 

compliance trends by normalizing for the size of an industry in a local area).

Currently there is no Federal data source that fits this description precisely. Appendix A describes a set of four 

potential data sets that could be the base for an “authoritative source.” There are advantages and drawbacks to 

each of these sources, and further work would need to be done to determine how to enable Federal agencies to 

use these as authoritative sources. They include: the Business Register (Census Bureau), the Quarterly Census 

of Employment and Wages (BLS), the Business Master File (IRS), and the GLEIF Concatenated File (Global 

Legal Entity Identifier Foundation). While the three government data sources offer the greatest coverage, and 

are of the highest quality for identified uses, they have many legal and practical requirements and restrictions.

43

44 The GLEIF Concatenated File has the advantage of being publicly available, but at this moment does not 

43 U.S. Census: Title 13. https://www.census.gov/history/www/reference/privacy_confidentiality/title_13_us_code.html 44 IRS: Title 26. https://www.irs.gov/tax-professionals/tax-code-regulations-and-official-guidance
826 The Promise of Evidence-Based Policymaking

23

cover a range of industries and establishments in a manner similar to the other three sources. Please see 

Appendix A for additional information.

The Workgroup also suggests further investigation towards creating federal protocols to support data sharing,

such as the creation of centralized data sharing guidance. For example, some statistical agencies can allow 

programs with approved projects (including program evaluations) to detail employees to overcome impediments 

to cross-agency data sharing. Additionally, some agencies might adopt the model set by the State of Illinois’s 

Department of Innovation and Technology, which in 2016 developed a government-wide MOU that defined a 

common data taxonomy, standardized internal controls, streamlined data sharing, and established an arbitration 

process. As a result, data requests are no longer “ad-hoc.” They are processed in a matter of days—like FOIA 

requests—through a process described by the Department’s General Counsel as “safe, quick, and transparent.”45

Several agencies, notably the Social Security Administration, have strong internal data sharing protocols that 

could be replicated on a larger scale. It may also be possible for the Federal government to follow aspects of 

the Illinois model without requiring changes to statute.

In light of the significant legal barriers, or policy and legal interpretations associated with the confidential

sources noted in Appendix A,

46 agencies can instead maximize the use of their existing data by setting up 

validation and entity resolution checks at the point of data entry, or determine how to improve data sharing 

practices accounting for the models just described.

Over the long term, Federal Agencies may consider a range of options for additional best practices. For 

example, it would be beneficial to look into the possibility of combining the authoritative sources noted in 

Appendix A to create a new mapping table that only contains data necessary for entity resolution, and which 

could be available to a wider audience of Federal agencies. This could be treated as distinct from the other 

sources, may not have the same access restrictions as the underlying source, and would be similar to data sets

where the data are confidential but can be shared in a masked or aggregated form. Alternatively, it would be 

useful to further examine to what extent an authoritative source could be constructed that does not contain 

confidential data, and to examine the quality of such a source relative to confidential sources. 

NEXT STEPS

The Workgroup has developed additional recommended topics for investigation that could help improve the 

sharing of employer data:

• The establishment of an interagency community of practice and repository for sharing methods, code, 

and approaches to data collection and matching. Federal agencies might benefit from continuing to 

share knowledge in a structured manner, to ensure that Federal analysts take advantage of the most 

efficient matching and data collection approaches for a variety of applications. Offices within the 

Workgroup are able to assist in the establishment and maintenance of the group so that it would continue 

to be meaningful and useful for participants. 

• Consultation with a broader Federal and external audience to gain additional insights from external 

experts and front-line statisticians.

• The establishment of a centralized data sharing “referee.” Federal agencies have expressed an interest in 

finding ways to expedite and facilitate data sharing, and have identified a centralized office as key to 

achieving this. Such an office could develop common data sharing protocols, serve as a library of data 

45 Testimony of Michael Basil, General Counsel, Department of Innovation and Technology, State of Illinois before the Second 

Meeting of the Commission on Evidence-Based Policymaking, September 9, 2016.
The Promise of Evidence-Based Policymaking 827

24

sharing agreements, and otherwise serve as a home for knowledge and process for interagency data 

sharing. 

CONCLUSION

There is substantial potential to achieve efficiencies in matching U.S. employer data across Federal data sets for 

data analysis, evaluations, and statistical activities, based on common needs across agencies. The greatest

barrier to matching data on employers across data sets is the lack of a common, or universal, business identifier.

Eliminating this obstacle by developing an infrastructure to create, assign, and manage a universal identifier

could result in cost savings in matching but would require a major investment of time and taxpayer resources.

Assuming that the identifier could be created, it would be a challenge to enforce consistent use of such an 

identifier by all employers without statutory changes. This identifier would need to capture various 

corporate/industry levels and change over time, but no Federal entity collects all of this information.

Nevertheless, there are immediate steps agencies can take to adopt best practices for data collection and 

matching, and there are ways to maximize the use of existing authorities and data sets to enable matching. In 

particular, leveraging data elements common among Federal agencies which, in combination, can constitute a

universal unique identifier, would facilitate efficiencies for matching Federal data sources. The utility of 

Federal data sources can be increased by including as many cross-agency identifiers as possible. While these 

fields are useful for matching, it is important to note that identifiers have confidentiality, privacy, and 

proprietary concerns. 

Moving forward, there are potential places to go further to realize long-term improvements, such as developing 

a roadmap for implementing common data elements in Federal data sources, developing an authoritative source 

of business existence and characteristics for Federal agencies, developing a Federal community of practice for 

matching and entity resolution, and establishing a centralized data sharing “referee.” 
828 The Promise of Evidence-Based Policymaking

25

APPENDICES

Appendix A: Potential Authoritative Sources

The workgroup identified four existing data sets that could serve as the basis for potential authoritative sources

of business existence and characteristics across agency missions and program functions (see Table A1 for 

additional information):

Business Register, U.S. Census Bureau 

The Census Bureau’s Business Register contains establishments of all domestic businesses (except agriculture, 

forestry, fishing, hunting, rail transportation, the U.S. Postal Service, elementary and secondary schools, 

colleges and universities, labor organizations, political organizations, religious organizations, public 

administration, and private households) and organizational units of multi-establishment businesses. A singleunit enterprise’s primary identifier is its Employer Identification Number (EIN). A unique employer unit 

identification number identifies each establishment owned by a multi-unit enterprise on the Business Register.

Advantages: The Business Register (BR) covers more than 160,000 multi-establishment companies, 

representing 1.8 million affiliated establishments, 5 million single establishment companies, and nearly 21 

million non-employer businesses (note, Census maintains a separate register for employers and nonemployers).47 The Business Register is updated continuously, and the update frequency varies by its sources. 

Lags also vary, by the reference period of the sources. The Business Register is one of the most complete, 

current, and consistent source of establishment- based information about U.S. businesses, and is essential to 

assuring full coverage and high quality in Federal economic statistics programs.48

Considerations: Users should note that the source excludes a significant number of industries, including, most

notably, agriculture, education, and public sector. Additionally, due to the lags in the data, users should 

exercise care in analyses of industries and sectors with high turnover or conversion rates. This data set contains 

links between establishments and their parent firms, but these links are sometimes recorded a year or more 

before or after the reference date. 49 Also, the Census Bureau’s Business Register is constructed using 

comingled confidential tax information and non-tax data from various sources, including Census surveys. The 

fact that the Census Bureau’s Business Register contains confidential tax information prevents the Census 

Bureau from completely sharing its Business Register with other agencies not authorized to receive the 

confidential tax information under Title 26.

Business Master File, Internal Revenue Service

The Business Master File (BMF) contains data for all Federal business tax returns that meet IRS filing 

requirements. The data set consists of individually filed returns for a single establishment and consolidated 

filed returns consisting of a group of related (affiliated) establishments. BMF data are updated continuously but 

usually become available weekly. The unit of analysis is tax return-based, as filed by the taxpayer; enterprises 

are not aggregated by the IRS. 

Advantages: The data set consists of total population data based on taxpayer filings. Also, the availability of 

Employer Identification Numbers allows for direct linkages. 

47 See also: https://www.census.gov/econ/overview/mu0600.html 48 Ibid. 49 See also: www.bls.gov/osmr/pdf/st140030.pdf
The Promise of Evidence-Based Policymaking 829

26

Considerations: The unit of observation is a tax return, which is a higher, and different, level than establishment 

or physical location for multi-establishment businesses. This differs from both the Census Bureau’s Business 

Register and Quarterly Census of Employment and Wages Business Register. Even for businesses that are not 

filing within a consolidated group, a return may not represent an establishment, for example in cases where a 

single business has operations located at more one physical address. Entity information such as addresses is 

based on taxpayer-reported information and may not necessarily be the actual physical address for matching 

purposes.

U.S. Bureau of Labor Statistics-Office of Employment and Unemployment Statistics, Quarterly Census of 

Employment and Wages (QCEW) Business Register (BR)

The QCEW-BR contains employment, wages, and administrative data (name, location, etc.) for over 9.5 million 

establishments, covering approximately 98% of all employment. The Quarterly Census of Employment and 

Wages (QCEW) provides data for establishments on monthly employment, total quarterly wages, the number of 

business establishments, and other business identification information such as address, industry, and federal 

employer identification number, etc. In addition to being very comprehensive and accurate data, the QCEW-BR 

is timely: data are available 6 months after the reference quarter making this the most current source of 

comprehensive business establishment data available.

The QCEW data are the product of a federal-state cooperative program. The data are derived from summaries of 

employment and total pay of workers covered by state and federal unemployment insurance (UI) legislation and 

provided by State Workforce Agencies (SWAs). States prepare a microdata file each quarter and submit that to 

BLS within 15 weeks of the end of the quarter. QCEW data are developed for the 50 states, the District of 

Columbia, Puerto Rico, and the U.S. Virgin Islands. The summaries are a result of the administration of state 

unemployment insurance programs that require most employers to pay quarterly taxes based on the employment 

and wages of workers covered by UI. Employment and wage data for workers covered by state UI laws are 

compiled from quarterly contribution reports submitted to the SWAs by employers. For federal civilian workers 

covered by the Unemployment Compensation for Federal Employees (UCFE) program, employment and wage 

data are compiled from quarterly reports submitted by four major federal payroll processing centers on behalf of 

all federal agencies.

BLS sets quality standards in state cooperative agreements and provides conceptual, technical, and procedural 

guidance to the states and uses standardized procedures to process these data and ensure consistent quality 

across states. State workforce agencies are responsible for collecting the administrative records from their state

unemployment insurance system and transform these records into meaningful economic data. In addition to 

state data quality improvements, BLS conducts additional data review at its regional and national offices to 

ensure quality. 

BLS also enhances the data with two supplemental surveys: 

• The Annual Refiling Survey, which allows BLS and the states to collect updated North American 

Industry Classification System (NAICS) industry codes, geographic county codes, and address 

information for business establishments. 

• The Multiple Worksite Report, which allows BLS to collect detailed monthly employment and total 

wages each quarter for businesses with more than one location. This allows the program to capture 

business births and deaths in a timely and frequent manner and accurately capture changes in ownership 

as a result of mergers and acquisitions.

BLS links the microdata for each business establishment across quarters to create a longitudinal record. These 

data are available starting in 1990 through the most recent data available: second quarter 2016. This linked 
830 The Promise of Evidence-Based Policymaking

27

microdata file serves as the sampling frame for BLS establishment-based surveys such as the Current 

Employment Statistics (CES), a key survey used for the publication of the monthly Employment Situation. 

Other BLS programs that use the QCEW microdata for sampling purposes include the Job Openings and Labor 

Turnover Survey (JOLTS), Occupational Employment Statistics (OES), Producer Price Index (PPI), Survey of 

Occupational Injuries and Illnesses (SOII), National Compensation Survey (NCS), including the Employment 

Cost Index (ECI) and Employer Benefits Survey (EBS), and the new Occupational Requirements Survey 

(ORS). The Local Area Unemployment Statistics (LAUS) program also uses the QCEW as its source of 

employment when CES estimates are not available.

In addition, BLS publishes Business Employment Dynamics (BED) statistics drawn from this linked microdata 

set. BED statistics are created from the linked individual business establishment records that are tabulated to 

create aggregate time series for national and local business establishment openings, closings, expansions, and 

contractions, all by industry. Over the years, additional variables have also been created, including 

establishment age, survival rates, and firm size.

The QCEW program publishes tabular data for the nation, states, metropolitan statistical areas, and counties at a 

detailed NAICS 6-digit industry level about 6 months after the end of the reference quarter. The release date for 

QCEW data has been moved up by a total of 3 weeks since 2012. The data are valued for their 

comprehensiveness, accuracy, relevance, and timeliness.

Business Employment Dynamics statistics are published in the month following the release of the QCEW 

tabular data. These statistics have gained a wide user base as economists and other analysts continue to examine 

the role of employment dynamics in the U.S. economy. Key user groups of BED statistics include the Federal 

Reserve System, the Small Business Administration, and academics.

Key users of tabular QCEW data include Congress, state and local economic development agencies, state 

revenue forecasters, and numerous Federal agencies. Over $300 billion in public funds are allocated based on 

QCEW data. 

Among the many users of QCEW data, four demonstrate some of the data’s varied roles:

• The Employment and Training Administration (ETA) uses QCEW data to measure the solvency of 

unemployment insurance trust funds and to develop the statistical adjustment models for measuring 

quarterly performance of Workforce Innovation and Opportunity Act (WIOA) funded core programs.

• The Bureau of Economic Analysis (BEA) uses the quarterly QCEW data to develop county, state, 

regional, and national personal income estimates, a component of the gross domestic product, and to 

conduct related statistical research and analysis. In 2015, covered workers received $7.385 trillion in 

pay, representing 94.0 percent of the wage and salary component of personal income and 40.9 percent of 

the gross domestic product.

• Census Bureau

o Since 1990, BLS has shared NAICS codes with the Census Bureau to improve industry coding 

consistency and reduce respondent burden and costs. The Census Bureau uses these industry data 

in its Business Register, which serves as a source of sampling frames for frequent business 

surveys (such as the Annual Survey of Manufacturers) and as a basis for statistical tabulations. 

The most important benefits of this data-sharing project are relieving American businesses of 

unnecessary response burden, improving industry coding for the Census Bureau, improving 

usability and promoting consistency between federal statistical products, and reducing 

redundancy between agency statistical programs, to the exceptional benefit of the American 

taxpayer. 
The Promise of Evidence-Based Policymaking 831

28

o In addition, after states and BLS have edited and curated the microdata, the QCEW data file is 

sent to the Census Bureau, where the data serve as a primary input for the Longitudinal 

Employer-Household Dynamics (LEHD) program.

o QCEW data are also used to calibrate the joint BLS-Census Current Population Survey after each 

decennial census. 

• Outside researchers also apply for access to the QCEW microdata in a protected environment for 

projects that are relevant to the mission and scope of BLS.

By publishing QCEW and BED data, and by sharing these data amongst our statistical partners, BLS provides 

informational infrastructure that enhances the ability of the public and private sectors to make evidence-based 

decisions.

Advantages: The QCEW-BR contains data for all industries (including government), and coverage is 

mandatory, therefore compliance is high. In addition, the data set has consistent terms, is accurate, timely, 

relevant, and has a strong validation process. The QCEW-BR is sharable, subject to state data sharing 

restrictions. It is also a high frequency data set with quarterly data collection for monthly employment values. 

The QCEW data set is sustainable and scalable. There are a number of matching projects and new data products

that can be developed with little or no new response burden at little or no additional cost (i.e., nonprofit data, 

etc.).

Considerations: 

Exclusions from QCEW include self-employed workers, most agricultural workers on small farms, all members 

of the Armed Forces, elected officials in most states, most employees of railroads, some domestic workers, most 

student workers at schools, and employees of certain small nonprofit organizations.

There are some states who do not agree to share their data. A law change allowing for full data sharing would 

enhance the usability of QCEW data.

Global Legal Entity Identifier Foundation, Concatenated Data File

The Global Legal Entity Identifier Foundation (GLEIF) publishes the updated GLEIF Concatenated File daily. 

This file contains the content of the individual files, published by the Legal Entity Identifier (LEI) issuing 

organizations, which list all LEIs issued to legal entities and related LEI reference data. The data provides 

information on a legal entity identifiable with an LEI. 

Advantages: The key advantage of the GLEIF Concatenated File, relative to the other sources noted, is that it is 

publicly available. The Global Legal Entity Identifier Foundation notes on their web site, “The drivers of the 

LEI initiative, i.e. the Group of 20, the [Financial Stability Board] and many regulators around the world, have 

emphasized the need to make the LEI a broad public good. The Global LEI Index, made available by GLEIF, 

greatly contributes to meeting this objective. It puts the complete LEI data at the disposal of any interested 

party, conveniently and free of charge.”50 The GLEIF Foundation will also be collecting information on parent

and subsidiary entities during the annual re-registration for LEI numbers in 2017, and plans to continue this data 

collection in future years.

51

50 “Introducing the Legal Entity Identifier (LEI).” Global Legal Entity Identifier Foundation. <https://www.gleif.org/en/aboutlei/introducing-the-legal-entity-identifier-lei>. Accessed 12/9/2016. 51 https://www.gleif.org/en/lei-data/access-and-use-lei-data/level-2-data-who-owns-whom
832 The Promise of Evidence-Based Policymaking

29

Considerations: While the LEI is intended to be a universal and open identifier, coverage is currently low. As 

of the time writing this there are fewer than 500,000 LEIs issued globally, though this number is expected to 

increase over time as government’s and institutions mature their processes using this identifier. Additional 

considerations regarding application of the LEI can be found in the November 2015 Progress report

by the Legal Entity Identifier Regulatory Oversight Committee (LEI ROC).

52

52 Available at: https://www.leiroc.org/publications/gls/lou_20151105-1.pdf
The Promise of Evidence-Based Policymaking 833

30

Table A1: Options for Authoritative Sources

Component 

Agency or 

Office

Data set

Name

Purpose of the 

Data Collection

Access 

Restrictions, 

Update 

Frequency and 

Lags

Coverage Definition of 

"Company"

Unit of Analysis,

Corporate Structure 

and Relationships

Quality of Fields for Matching

U.S. Census 

Bureau

Business 

Register

To provide a 

current and 

comprehensive 

database of U.S. 

business 

establishments 

and companies for 

statistical program 

use.

The Business 

Register 

information is 

confidential [Title 

13 and Title 26, 

US Code]. Access 

is restricted to 

persons specially 

sworn to uphold 

the confidentiality 

provisions of Title 

13 and Title 26.

Data are updated 

continuously, 

update frequency 

varies by sources; 

lags vary by the 

reference period of 

the sources.

Establishments of all 

domestic businesses 

(except agriculture, 

forestry, fishing, 

hunting, rail 

transportation, the U.S. 

Postal Service, 

elementary and 

secondary schools, 

colleges and 

universities, labor 

organizations, political 

organizations, religious 

organizations, public 

administration, and 

private households) and 

organizational units of 

multi-establishment 

businesses. The 

Business Register (BR) 

covers more than 

160,000 multiestablishment 

companies, representing 

1.8 million affiliated 

establishments, 5 

million single 

establishment 

companies, and nearly 

21 million nonemployer businesses.

An establishment is a 

single physical 

location where 

business transactions 

take place and for 

which payroll and 

employment records 

are kept. Groups of 

one or more 

establishments under 

common ownership or 

control are enterprises. 

A single-unit 

enterprise owns or 

operates only one 

establishment. A 

multi-unit enterprise 

owns or operates two 

or more 

establishments. 

Establishment-based. The Business Register is one of

the most complete, current, and 

consistent source of 

establishment- based 

information about U.S. 

businesses, and is essential to 

assuring full coverage and high 

quality in Federal economic 

statistics programs. Examples of 

quality considerations for this 

source include:

- The annual Company 

Organization Survey covers 30 

percent of multi-unit companies 

and a small sample of firms that 

were single-unit firms in the 

most recent quinquennial 

Economic Census, so 

establishment openings and 

closings in the firms not covered 

may not be reflected in the 

business register until after the 

next Economic Census (though 

the Census Bureau takes 

measures to address this).

- The business register is divided 

into employer and non-employer 

business registers based on 

payroll employment. Some firms 

lease their employees from 

Professional Employer 

Organizations (PEOs) or use 

independent contractors. Such 

firms may appear in the nonemployer business register 

despite having large revenues 

and many leased and/or contract 

employees. 

(https://www.census.gov/econ/o

verview/mu0600.html)
834 The Promise of Evidence-Based Policymaking

31

U.S. Bureau 

of Labor 

StatisticsOffice of 

Employment 

and 

Unemployme

nt Statistics 

(OEUS)

Quarterly 

Census of 

Employment 

and Wages 

(QCEW) 

Business 

Register 

(BR)

To provide a 

quarterly census 

of all 

establishments 

under State 

unemployment 

insurance 

programs, 

representing about 

98 percent of 

employment on 

nonfarm payrolls. 

This database of 

U.S. business 

establishments 

serves as the basis 

for multiple 

statistical 

programs; 

Sampling frame 

& 

benchmark(CES, 

JOLTS, PPI, 

OES, LAUS,

SOII, and NCS, 

which includes 

the ECI, EBS and 

ORS), labor 

market research, 

Business 

Employment 

Dynamics (BED) 

data.

All microdata are

confidential 

subject to BLS 

non-disclosure 

standards.

The QCEW BR is 

updated quarterly, 

and the data 

become available 6 

months after the 

reference cycle. 

Employment, wages, 

and administrative data 

(name, location, etc.) 

for over 9.5 million 

establishments covering 

approximately 98% of 

all employment.

An economic unit that 

produces goods or 

services, usually at a 

single physical 

location, and engages 

in one or 

predominantly one 

activity. --Potential lay 

synonyms: business, 

worksite, brick& 

mortar, site, storefront. 

Lay users my use 

“establishment” 

interchangeably with 

“firm” In the QCEW 

BR, however, there is 

a significant 

distinction between the 

two terms.

Establishment-based. 

Multi-unit enterprises 

may use multiple 

EINs, and they may 

use different ones 

when reporting to 

different Federal 

agencies, complicating 

the matching process. 

The data set indicates 

whether an 

establishment is a 

single or multilocation establishment.

In order to ensure the highest 

possible quality of data from the 

QCEW program, BLS and the 

States verify and update, if 

necessary, the NAICS, location, 

and ownership classifications of 

all units on a 3–year cycle. 

Government units in public 

administration are not reviewed 

routinely. 
The Promise of Evidence-Based Policymaking 835

32

Internal 

Revenue 

Service -

Statistics of 

Income

Business 

Master File

IRS: The purpose 

of data collection 

is mainly for 

determining 

Federal tax 

liability for 

businesses 

required to file. 

SOI: The data are

used to produce 

statistics on 

income, 

deductions, 

credits and other 

taxes, as reported 

by businesses. 

The current 

design is a 

probability 

sample stratified 

by Business type 

(as indicated by 

the IRS form 

filed) and either 

by size of total 

assets alone or 

size of total assets 

and a measure of 

income.

Federal Tax 

Information (FTI) 

is confidential 

[Title 26, US 

Code] and shared 

with other 

government 

agencies under 

IRC 6103(j) 

provisions.

Data are updated 

continuously; Data 

becomes available 

weekly.

IRS: All Federal 

business tax returns that 

meet IRS filing 

requirements. 

SOI: Selected active 

Federal business tax 

returns based on SOI's 

sample design.

Definition of 

"company” is based on 

Title 26 requirement.

Tax Return.

The data set consists 

of individually filed 

returns which can 

represent: corporations 

(a single 

establishment; 

subsidiary 

establishment or 

consolidated filed 

returns representing a 

group of 

establishments);

partnerships and other 

pass-through entities;

or sole proprietorships. 

Most matching is accomplished 

using the EINs provided by 

entities as reported on Federal 

tax returns. SOI uses exact 

matching with EINs and data 

processing begins with 

information already extracted for 

IRS administrative purposes. 

SOI performs limited internal 

"data cleaning" for statistical 

purposes. This includes

organizing data to make it 

structurally consistent, coding 

data items to make them 

analytically useful, and 

validating values to ensure 

mathematical consistency. 

Contact information is validated 

as part of routine administrative 

processing of returns at the time 

they are received by the IRS.

Global Legal 

Entity 

Identifier 

Foundation

GLEIF 

Concatenate

d File

The Global Legal 

Entity Identifier 

Foundation 

(GLEIF) 

publishes the 

updated GLEIF 

Concatenated File 

daily. This file 

contains the 

content of the 

individual files, 

published by the 

Legal Entity 

Identifier (LEI) 

issuing 

organizations, 

which list all LEIs 

issued to legal 

Publicly available 

without restrictions

All legal entities are 

eligible to receive an 

LEI. As of the time 

writing this there are 

465,397 LEIs issued 

globally.

As defined in ISO 

17442, the standard 

underlying the Legal 

Entity Identifier (LEI), 

the term ‘legal entity’ 

includes, but is not 

limited to, unique 

parties that are legally 

or financially 

responsible for the 

performance of 

financial transactions 

or have the legal right 

in their jurisdiction to 

enter independently 

into legal contracts, 

regardless of whether 

they are incorporated 

Legal entity based Very high quality where 

available. There is a validation 

effort done whenever an entity 

applies for an LEI. This ensures 

a distinct unique record exists 

for each identifier (). The LEI 

identifies an entity across 

multiple data sets where required 

by regulation. See the following 

report for additional 

information: 

https://www.leiroc.org/publicati

ons/gls/lou_20151105-1.pdf
836 The Promise of Evidence-Based Policymaking

33

entities and 

related LEI 

reference data. 

The data provides 

information on a 

legal entity 

identifiable with 

an LEI

or constituted in some 

other way (e.g. trust, 

partnership, 

contractual). It 

excludes natural 

persons, but includes 

governmental 

organizations and 

supranationals.
The Promise of Evidence-Based Policymaking 837

34

APPENDIX B: Best Practices for Matching Data

The Workgroup’s Methods Inventory yielded a variety of strategies in data collection and data integration 

which Federal agencies use to work around issues in matching and entity resolution. See Appendix E for 

specific code examples.

DATA COLLECTION

Federal agencies maximize the use of identification approaches during the data collection phase to improve 

matching to prevent downstream challenges in matching and entity resolution. These approaches focus on 

identifying the right entity and industry or corporate relationships, and validating geographic and industry 

information.

Identifying the right entity, right level and right relationship

Relationships

Federal agencies often encounter complex industry or corporate structures, and work to identify the correct 

entities, corporate and industry levels, and corporate and industry relationships at the point of data collection to 

avoid challenges in matching or entity resolution later.

• For example, a facility may be owned by one company but another may control its operation (this was 

the case with the Deepwater Horizon disaster – the rig was owned by Transocean, but leased by British 

Petroleum (BP) and operated by Transocean and other contractors under BP’s direction53). 

• There may also be cases where a facility has one or more contractors who each have their own 

compliance, regulatory and reporting requirements co-located within the same facility, such as a steel 

mill. 

• Another scenario exists in the case of joint ventures, where a facility may be owned and/or operated by 

multiple parties. Additionally, there may be an owner or operator entity being captured at the facility 

level, but also some degree of org parent information, whether an ultimate global parent or domestic 

parent. 

• Lastly, Figure B1 shows an agency example where Facility 1 is owned by Organization B, but operated 

by Organization A; Organization B also owns Facility 2 and is part owner (via Joint Venture or other 

vehicle) of Facility 3, Organization B lists as its ultimate parent Organization E, however there may be 

intermediate entities not captured; Organization B also retains Organization C as a consultant to aid in 

preparation of regulatory documentation for Facility 3. Facility 3’s remainder owner is Organization D

which lists Organization F as its parent, however Organization F might not be its ultimate parent.

53 See also: Final Report of the National Commission. http://oscaction.org/resource-center/commission-reports-papers/
838 The Promise of Evidence-Based Policymaking

35

Figure B1: Example of a Complex Industry Structure

at the point of Data Collection

Facility 1 Facility 2 Facility 3

Org A Org B

Operates: Owns: Owns: OwnsPart:

Consult: Org C Org D

Preparer: OwnsPart:

Org F

Parent:

Org E

UltParent:

It is due to cases such as these that applications and data owners need to be cognizant of the importance of very 

robust and clear guidance, and good data structures for capturing this information, paying close attention to the 

contextual relationships. Agencies have found that relational, NoSQL, RDF triple stores and/or graph 

databases work well for capturing these types of complex relationships in the data. 

Establishment Facility Name

Often, difficulties in matching stem from inconsistent approaches and guidance for providing establishment 

names, for example providing only one organization name for a campus which comprises multiple operations 

and facilities which report to agencies separately from each other (for example a large university which may 

have physical plant and utilities versus labs and other facilities each with independent permitting, compliance 

and reporting responsibilities). In some cases, one may need sufficient information in order to disambiguate, 

requiring enough information and consistency in reporting. In other instances, one may need to merge and deduplicate data, and a lack of consistency can also be an issue in achieving this.

Table B1: Variations in Establishment Name

Establishment Name

Widgetco

WidgetCo Plant 2

Plant 2

Second Unit Widget Co

Contact Information

Data elements such as email addresses or telephone numbers can also be used to identify relationships in data 

and assist in entity resolution; for example one single point of contact is associated with over 4,000 chain 

drugstores in EPA’s Resource Conservation and Recovery Act data set. Similarly, telephone numbers can also 

be a useful tool in entity resolution. As with the prior cases, data entry and reporting interfaces should include 
The Promise of Evidence-Based Policymaking 839

36

validators to ensure proper formatting and values that appear valid. A number of JavaScript form validators 

exist 54 which can aid in these tasks.

Web sites, and e-mail addresses (after the “@” section) can not only serve as potentially valuable linking 

variables, but also as firm identifiers. Web sites and e-mail addresses are not subject to the same sorts of 

variation or mis-spellings that corporate names are. Similarly, while a corporation may have many regional 

telephone numbers, it will only have one domain name. Finally, companies will self-sort their economic 

activities into the appropriate Web domain names, likely based on activity, in cases where they undertake 

multiple, disparate business activities.

Geography-Related Fields

Address fields

As with entity reporting, for address fields, clear guidance and documentation needs to be implemented for 

collection interfaces, as well as providing the appropriate data structures for capturing separate values for 

physical and mailing/administrative address. 

Ideally, systems should provide geocoding capabilities which can validate and standardize entered street 

addresses, i.e. 528 South Fourth Street standardized to 528 S 4th St per USPS standard to aid in disambiguation. 

For large factories, in particular, it is also useful to specify where on the establishment the latitude and longitude 

will be established. Geocoding APIs typically also provide an effective mechanism for parsing and 

standardizing street address elements, such as house number (528) / street directional (S) / street name (4th St). 

Additionally, a geocoding API55 can provide latitude/longitude values which can easily be used to display a 

web map view for additional visual verification of site locations in reporting interfaces.

There are also cases where establishments might not have conventional street addresses, for example remote 

facilities in oil and gas sectors, or many Puerto Rico addresses which are linear addresses by distance marker 

along a route. Some typical cases may involve public works infrastructure where often the City Hall address is 

provided, or a PO Box is provided in place of the physical address, when quite likely both are wanted. Many 

systems also struggle to differentiate adequately in the case of multi-establishment addresses, whether office 

buildings, suites, incubators or industrial parks sharing an address, or other similar cases. Foreign addresses are 

also often problematic in many systems. Table B2 contains examples of commonly-encountered but difficultto-resolve addresses.

Table B2: Examples of Common, Difficult-to-Resolve Addresses:

Street Address 1 City State ZIP

PR 181, KM. 18.2, BO. ESPINO

SAN 

LORENZO PR 00754

NW1/4 S27 T48N R77W

JOHNSON 

COUNTY WY 99999

H.C. 64 BOX 204 MCFADDEN WY 82083

54 “Validators,” formvalidation.io, Retrieved 09 Sep 2016

http://formvalidation.io/validators/

55 “Geocoding”, Wikipedia, Retrieved 09 Sep 2016

https://en.wikipedia.org/wiki/Geocoding
840 The Promise of Evidence-Based Policymaking

37

47 MI W OF LARAMIE ON I-80

EXIT 267 LARAMIE WY 82070

43 32' 11" N 108 49' 29"W KINNEAR WY 82516

90TH SUPPORT GROUP, DEV 

BLVD 320 CHEYENNE WY 82005

When conventional street addresses cannot be provided, where appropriate and feasible, a clear mechanism 

should be provided for capturing alternative addressing schemes to improve how these can be dealt with and 

resolved. For example, WGS84 lat/long values are relatively easy to capture and collect with the advent of 

ubiquitous GPS technology on mobile devices, as well as the ease of embedding web based mapping 

applications which can capture a coordinate. When capturing data via latitude/longitude, existing data standards 

should be implemented, such as ISO 6709:200856 for entry and representation of latitude/longitude values. 

Additionally, consideration should be given to decimal degrees and precision for entry of latitude/longitude.

Public Land Survey System (PLSS) references such as NW1/4 S27 T48N R77W are tied to Bureau of Land 

Management Survey Grid and BLM has web services for determining location based on the descriptive 

elements57. Guidance and consistency in entry of the elements to facilitate parsing for a web service would be 

another necessary consideration in implementing PLSS entry.

As a final caveat, it should also be noted that postal municipality might not be the same as the jurisdictional 

municipality where the establishment is located. 

Latitude / Longitude Fields

Latitude/Longitude, where available, is a useful data element to use for proximity matching. Degree of 

precision of the entered latitude/longitude data impacts spatial resolution - and while the distance spanned by a 

unit of measure will typically be consistent for north/south parallels of latitude, it will vary for east/west 

meridian values as a function of varying latitude due to meridian convergence at the poles, as shown in Figure 

B2:

56 “Catalog: ISO 6709:2008”, International Standards Organization, Retrieved 09 Sep 2016 

http://www.iso.org/iso/catalogue_detail.htm?csnumber=39242

57 “GeoCommunicator Services,” Bureau of Land Management, Retrieved 09 Sep 2016 

http://www.geocommunicator.gov/geocomm/services.htm
The Promise of Evidence-Based Policymaking 841

38

Figure B2: Meridian Convergence

Image source: National Ocean and Atmospheric Administration)

Typical distance values corresponding with various levels of angular precision are as follows:

Table B3: Typical Distance Values Corresponding with Various Levels of Angular Precision

Decimal

places

Decimal

degrees

DMS Qualitative scale 

that can be 

identified

N/S or 

E/W

at equator

E/W at

23N/S

E/W at

45N/S

E/W at

67N/S

0 1.0 1° 00′ 0′′ country or large 

region

111.32 km 102.47 km 78.71 km 43.496 km

1 0.1 0° 06′ 0′′ large city or district 11.132 km 10.247 km 7.871 km 4.3496 km

2 0.01 0° 00′ 36′′ town or village 1.1132 km 1.0247 km 787.1 m 434.96 m

3 0.001 0° 00′ 3.6′′ neighborhood, 

street

111.32 m 102.47 m 78.71 m 43.496 m

4 0.0001 0° 00′ 

0.36′′

individual street, 

land parcel

11.132 m 10.247 m 7.871 m 4.3496 m

5 0.00001 0° 00′ 

0.036′′

individual trees 1.1132 m 1.0247 m 787.1 mm 434.96 mm

6 0.000001 0° 00′ 

0.0036′′

individual humans 111.32 mm 102.47 mm 78.71 mm 43.496 mm

7 0.0000001 0° 00′ 

0.00036′′

practical limit of 

field surveying

11.132 mm 10.247 mm 7.871 mm 4.3496 mm

Latitude/Longitude comparison will be discussed later in this document under Haversine Distance matching.
842 The Promise of Evidence-Based Policymaking

39

Industry Fields

NAICS/SIC Codes

North American Industry Classification (NAICS)58 and Standard Industrial Classification (SIC)59 Codes, may 

also be a helpful asset toward matching or disambiguating establishment records. For example, certain NAICS 

and SIC codes are relatively rare, such as Nuclear Electric Power Generation - NAICS: 221113. Additionally, 

groupings of related and unrelated NAICS/SIC codes could be used in business rules, such as codes which 

typically do not appear within the same establishment. SIC codes have technically been retired, however some 

agencies and data sets still use SIC, i.e. SEC, OSHA and others. One thing to note is that NAICS codes are 

revised on a regular cycle of 5 years, the most recent revisions being 2012 and 2007. As such, in capturing 

NAICS data, it would be useful to also reference the NAICS version being used, i.e. NAICS:2012. The U.S.

Census Bureau provides reference data on NAICS codes, which can be used to populate API-based entry or 

picklists as opposed to allowing manual entry of codes, to reduce data entry errors.

Hierarchy Field

Knowing which establishments are part of which parent companies (updated quarterly) is incredibly useful for 

matching efforts at different employer levels.

58 “North American Industry Classification System, ” US Census Bureau, Retrieved 09 Sep 2016 http://www.census.gov/cgibin/sssd/naics/naicsrch?chart=2012

59 “SIC Division Structure,” Occupational Safety and Health Administration, Retrieved 09 Sep 2016

https://www.osha.gov/pls/imis/sic_manual.html
The Promise of Evidence-Based Policymaking 843

40

COLLECTION METHODS

Leveraging Existing Data via API for Entity Resolution: Example case at US 

EPA

Existing, previously-collected data may be one of the best tools available for entity resolution at the point of 

data entry. It is critical to highlight examples that enable agency employees to make an internal case for 

changes. Agencies should note that EPA has begun leveraging its Facility Registry Service (FRS)60 for that 

purpose. FRS integrates data on over 4 million establishments and places of interest from across 90 different 

systems via master data management and a combination of algorithmic methods and manual data steward 

curation. The FRS team developed an API to improve integration at the point of data collection, by allowing 

reporters to identify establishments by searching and retrieving information that was previously reported via the 

various systems that FRS ingests data from.

This capability is illustrated in the following screen shots, in which a user searches for “Finch Paper” on Glen 

Street in Glen Falls, NY:

60 “Facility Registry Service,” US Environmental Protection Agency, Office of Environmental Information, Retrieved 09 Sep 2016

https://epa.gov/frs
844 The Promise of Evidence-Based Policymaking

41

Figure B3: Entity Search to call API
The Promise of Evidence-Based Policymaking 845

42

Figure B4: Entity API Search Results

The application would then allow newly reported information to be associated with existing, known 

establishment information and identifiers.

Initially this API was a read-only API for data retrieval, however it has since evolved to provide additional 

capabilities, including:

• The ability to submit a suggested update (for example, if an establishment had a typo in its information 

or if it was acquired by a new firm and the establishment’s name needs to change), or 

• if a new facility is being reported, the API will allow a master record and unique identifier to be 

generated in real-time, which will then subsequently be available for other queries.
846 The Promise of Evidence-Based Policymaking

43

An example of creating a new facility is shown in the following figure:

Figure B5: Entity Creation via API - Search failure enables creation of new record

On clicking the “Create New Facility” button, the user is then presented with a view that presents the 

information that has been entered, which is standardized and geocoded.
The Promise of Evidence-Based Policymaking 847

44

Figure B6: Entity Creation via API - Validation, Standardization and Geocoding of New Record

This reporting interface component is built on an API and has been componentized for reuse across EPA 

programs, and in the last two years has been integrated into multiple reporting systems, such as the Toxic 

Release Inventory (TRI), TSCA Chemical Data Reporting (CDR), Compliance Emissions Data Reporting 

Interface (CEDRI) and others. Within the TRI program, they have seen significant data quality improvements 

and have estimated substantial burden reductions (as further described in the white paper).

Data Preparation

Prior to attempting matching, some basic exploratory data analysis is recommended to identify the types of data 

quality challenges noted in the main body of the white paper.

Data sets to be matched may be scattered across different systems, different architectures, and disparate tables, 

and that preprocessing may need to take place in order to extract the data for analysis, via Extract-TransformLoad (ETL) processes or in the case of loading to systems designed for big data analytics such as 

HDFS/MapReduce, Extract-Load-Transform as appropriate61.

61 Davenport, Robert J. (2008), "ETL vs. ELT: A Subjective View", Retrieved Sep 12 2016

http://www.dataacademy.com/files/ETL-vs-ELT-White-Paper.pdf
848 The Promise of Evidence-Based Policymaking

45

It should also be considered that within a data set, there may be substantial turnover of establishments (20% or 

more within some sectors over a 5-year timespan), via acquisitions, mergers, business startups and closures 

which can affect name matching; as such, timestamps and windowing within timeframes may be necessary.

Cleansing to Address Inconsistent Data Formats

In cleansing records for entity matching, some key processes include converting all characters to the same case, 

and removing special characters, extraneous punctuation and extraneous white space between words.

Stop word lists are often used to allow algorithms to ignore what may be extraneous noise within the corpus of 

entities to be matched, such as common words which may not add value like "The" or values that are often 

associated with establishments like "Inc." or "LLC." Preprocessing can aid in stripping out these types of stop 

words. Alternatively, one may look to algorithmically replace these values, mapping to standardized values, 

such as "Limited Liability Company" "LLC" "L.L.C." "L L C" standardized as "LLC."

In applying cleansing steps, care needs to be taken in terms of what order different processing steps are taken, 

along with replacement rules. For example, if stop-word lists result in the omitted word being treated as a space 

rather than null, it may result in failed matches. 

Data Enrichment to Mitigate the Effects of Missing Fields

Toward entity resolution, it may be useful to generate derived attributes to assess data quality, identify potential 

data quality issues, and to standardize data fields for improved matching. This includes use of 

geocoding/reverse geocoding engines to generate standardized address fields and latitude/longitude values, 

along with spatial indexing or other processes for comparing locations to other geographies such as county 

polygons or jurisdictional boundaries. 

Enrichment via geocoding can be a powerful tool to aid in entity matching, however it should be noted that 

modern geocoding algorithms still suffer from some limitations. As noted in the data elements section, address 

fields may include non-standard, descriptive types of values, particularly relative values, such as "35 miles 

north of Gunstock on Highway 17" or "Across from the Empire State Building." A geocoding algorithm would 

need to be sophisticated enough to be able to identify the features and relationships in the descriptive value. 

While progress continues to be made in this area, most current geocoding algorithms cannot handle these types 

of relative values. Geocoders can however typically handle references to street intersections.

Geocoders are most adept at handling absolute values, such as standard address types. As noted in the data 

collection section, geocoding engines incorporate algorithms for address parsing, normalization and 

standardization, for example parsing and standardizing "One South Riverton Avenue" as House Number: "1" 

Prefix Directional: "S" Street Name: "Riverton" Street Type: "Ave". It should however be noted that geocoders 

function optimally when input parameters have already been identified as discrete data elements, such as 

Address / City / State / ZIP. Geocoders also generate match values, returning values corresponding to whether a 

known match was made at the house number level, street level, street intersection, city or ZIP, providing 

varying degrees of confidence. Geocoding engines rely on external data sets for matching and comparison, 

such as Census TIGER data62 and others, and as such will leverage these data sets to provide a 

latitude/longitude value, which is typically interpolated along a street segment, with the geocoding data set

providing data on address ranges as shown in the figure below:

62 "Census TIGER data", US Census, Retrieved Sep 12 2016

https://www.census.gov/geo/maps-data/data/tiger.html
The Promise of Evidence-Based Policymaking 849

46

Figure B7: Street Segment Interpolation

Image source: US Census

Note that the interpolated value calculated will be directly on the street segment centerline. In order to provide 

a more realistic value, an offset may either be supplied as a default, or may need to be supplied as a parameter 

to the geocoding engine in order to provide a coordinate offset from the street centerline in the appropriate 

direction, based on even/odd address number. Some commercial providers such as HERE may also provide 

point addresses, which attempt to provide actual rooftop or parcel centroid coordinates for addresses. Where 

these types of match values are available, they will be displayed in the match results.

Geocoding engines can also typically perform reverse geocoding, which accepts a latitude/longitude value, and 

returns the closest matching street address. While caution should be exercised in using at the street address 

level, this capability can potentially be useful for enriching data which has incomplete or unreliable 

city/state/ZIP data.

Data to be matched can also be augmented via spatial indexing - this entails testing a location value against 

polygonal values63, such as comparing entity latitude/longitude to a county polygon boundary. Spatial 

databases and GIS tools can provide this capability. Some useful geographies for analysis may include county 

polygons (note: it is more reliable to use county FIPS codes64 rather than county names, as county names are 

63 "Point-In-Polygon", Wikipedia, Retrieved Sep 12 2016, 

https://en.wikipedia.org/wiki/Point_in_polygon

64 "2010 FIPS Codes for Counties and County Equivalent Entities", US Census Bureau, Retrieved Sep 12 2016

https://www.census.gov/geo/reference/codes/cou.html
850 The Promise of Evidence-Based Policymaking

47

not unique) as well as other Census geographies such as Place65 or Block66 (as appropriate, and if other data 

elements reference these identifiers).

ALGORITHMIC MATCHING METHODS

Deterministic Methods:

The deterministic method is a process of data linkage that requires two records to agree on a pre-determined set 

of variables to conclude the pair as a link. A match is defined as records from two files that are truly the same 

unit. A link is defined as two records that are designated as the same unit based on their characteristics and 

decision rules. While matching on employer name or EIN may seem simple, great caution should be taken to 

ensure that the data elements from the matched sources are truly comparable and the resulting comparisons are 

meaningful.

The variables to be used are usually established by subject matter experts and require a significant amount of 

human review. Deterministic matching works best when there are unique IDs to match, or when there are 

reliable rules and high quality data elements that can be used for matching, such as email addresses, telephone 

numbers, dates of birth, NAICS or SIC codes - ideally with multiple elements used in combination - for 

example, a validated ZIP code by itself might not be sufficient for deterministic matching and would need to be 

combined with other data elements. It would also potentially be useful in deterministic processing to 

disambiguate from other similarly-named entities at different locations. Similarly, one could look at 

relationships within values such as related industrial classifications within NAICS or NAICS-SIC crosswalks67.

In using deterministic approaches, consider business rules that may affect relationships between entities (for 

example, a common email or phone number that is at the HQ office but is shared across multiple entities, or one 

unique email address that shows up across 4,000 chain pharmacy locations).

Relationships between each individual establishment to the HQ office can be established but do not conflate 

individual entities. Also, as noted in the data elements discussion, consider that there may be third parties 

involved in the data submission process, such as consultants who are doing preparation on behalf of an 

establishment, as such, roles, wherever available should be considered. For example, the same law firm may 

prepare filings for many small companies and may be listed as the “contact person” and “contact mailing 

address” for all these employers. Linking on this field will create many erroneous links. The usefulness of 

company websites and Wikipedia pages in deterministic matching of parent companies with subsidiaries should 

not be understated.

In deterministic matching, it is much more efficient to have automated matching methods err on the side of 

matching too much, and having an analyst remove erroneous matches than vice-versa. It is much less timeconsuming for an analyst to remove erroneous matches than to search out true matches.

65 "Geographic Terms and Concepts - Place," US Census Bureau, retrieved Sep 12 2016, 

https://www.census.gov/geo/reference/gtc/gtc_place.html

66 "2010 Census - Census Block Maps," US Census Bureau, retrieved Sep 12 2016,

https://www.census.gov/geo/maps-data/maps/block/2010/ 67 "NAICS to SIC Crosswalk", NAICS Association, Retrieved Sep 14 2016

https://www.naics.com/naics-to-sic-crosswalk/
The Promise of Evidence-Based Policymaking 851

48

Probabilistic Methods

Probabilistic methods are used to assign a score to two record pairs then set criteria determining link status. 

Many methods use three ranges of scores, a link with a high score, a non-link with a low score, and a possible 

link if the score falls in an indeterminate range. The possible links are then reviewed to determine match status. 

In attempting to match across large data sets, blocking is an additional strategy that can be used when doing 

probabilistic record linkage in order to reduce the size of the comparison space. This strategy is used to group 

similar records before performing the comparison. For example, a full Cartesian join of two files with 450,000 

records would yield over 200 billion comparison pairs. It can be seen as these data sets grow the number of 

comparison pairs can be computationally prohibitive or processing would take an unfeasible amount of time.

There are other strategies that can be used for the probabilistic matching such as standardizing names or fields 

prior to matching. In some cases edit distance measures can be used to rescale the component weights on a 

given field instead of giving a field a binary agree/disagree designation. Considerations also need to be made 

on how to handle missing data when using these methods. In some cases two missing fields should be 

considered a match while in others they should be non-deterministic or even a non-match. Other ways have 

been suggested to refine the m- and u-probabilities described previously using a frequency scaling such as the 

one described in Matching and Record Linkage by William Winkler68.

Considerations and Clarifications in Probabilistic Entity Resolution Algorithms

1. Entity Resolution is not Classification. Simply identifying whether or not two records match or do not 

match is classification while entity resolution develops a dynamic entity using metadata. 

2. Entity Resolution is not Clustering. The goal of clustering is to identify similar groupings of entities; 

the goal of entity resolution is to reconcile different iterations of the same entity down to their common 

iteration.

Algorithms for Comparing Latitude/Longitude

Comparisons of latitude/longitude pairs for proximity assume that coordinates contain valid values. Wherever 

possible, a prior data QA step should be taken to ensure that values pass reasonable checks. Common issues 

specific to entered lat/long values include reversal of lat/long values to long/lat, omission of sign for 

hemisphere, and use of placeholder values, such as 0,0 or 1,1. Additionally, data sets may use differing 

standards, such as sexagesimal degrees-minutes-seconds versus decimal degrees. These will need to be parsed 

and converted to decimal values. One way to perform a validation is to use a reverse geocoder on the address 

and compare the reverse geocoding result to the provided latitude/longitude value.

Haversine Distance: The Haversine Distance formula69 can be used to compare spatial latitude/longitude tuples 

by computing Great Circle surface distance70 between locations. 

68 Winkler, W. (2000). Frequency-Based Matching in Fellegi-Sunter Model of Record Linkage 69 Williams, Ed (2011), "Aviation Formulary V1.46," Accessed Sep 12 2016 

http://williams.best.vwh.net/avform.htm#Dist

70 "Great Circle," Wikipedia, Accessed Sep 12 2016

https://en.wikipedia.org/wiki/Great-circle_distance
852 The Promise of Evidence-Based Policymaking

49

dddd = 2 sin−1 ��sin �

φφφφ1 − φφφφ2

2 �

2

+ cos(φφφφ1) cos(φφφφ2) sin �

λλλλ1 − λλλλ2

2 �

2

�

where:

• φ1 is first coordinate latitude in radians, φ2 is second coordinate latitude in radians 

• λ1 is first coordinate longitude in radians, λ2 is second coordinate longitude in radians

• d is computed angular distance between the points

Text Matching Algorithms for Entity Resolution

The following methods address challenges related to missing important data fields, or having inconsistent data 

formats in text fields, such as name and address fields. Depending on which data element is being matched 

across data sets, different algorithms can be applied.

Objective Description Examples

String Matching Quantify permutations needed to convert one string to 

another

Edit Distance, Alignment, 

Phonetic

Distance Metrics Apply physical distance measures to abstract concept of data 

objects

Similarity, Text Analytics

Relational 

Matching

Conjunctive view reliant on one data object’s relationship to 

other objects

Set Based, Aggregate

String Matching

String matching algorithms are concerned with whether or not two strings say the same thing. Outlined in the 

table below, there are four essential approaches. They may, however, be further subdivided into exact elementby-element character or phonetic comparison.

Boolean Matching, is easily understood as a Yes or No, 0 or 1, match or non-match between two strings. It is

the most simplistic of the group and is the core logic on which the subsequent algorithms operate.

Method Description Examples

Edit Distance Quantified permutations to convert one textual string into another Levenshtein, JaroWinkler

Jaccard 

Coefficient

Ratio of existence or absence of one entity’s individual attributes 

in another

Jaccard

Phonetic 

Similarity

Pronunciation of letters are phonetically related on a 0 to 1 

similarity scale, aka, fuzzy matching

Soundex, Translation

In the heavily-studied Edit Distance, similarity is quantified by physically measuring the permutations needed 

to convert one string into other. The core implementation of edit distance is Levenshtein, which penalizes for 

insertions, deletions and substitutions. Over time Levenshtein has been modified with additional costs for gaps 

(Sellers), transpositions (Smith-Waterman), and affine gaps, i.e., weighted costs per each of the actions or the 

location of where the permutation must be made (Gotoh).

Jaro Distance is a hybrid version of edit distance whose more popular counterpart, Jaro-Winkler, is considered 

a hybrid algorithm. Practically, Jaro slides along two strings, comparing nGrams along the way to quantify the 

number of characters appearing in the same position and the number of transpositions required for coincident 
The Promise of Evidence-Based Policymaking 853

50

characters which must be reordered in one string to match the other. Its best application is with short strings, 

also it disobeys triangle inequality.

The Jaccard Coefficient is an element-by-element measure of intersection. Stated otherwise, it is the ratio of 

the intersecting set to the union set. The Jaccard Coefficient satisfies triangle inequality. One frequentlyconfused issue: the similarity version of Jaccard and Tanimoto Coefficients are identical, but their dissimilarity 

coefficients diverge due to triangle inequality. While this justifies the need for two separate algorithms, they are 

frequently credited as the Jaccard-Tanimoto Coefficient as both mathematicians independently published this

ratio unbeknownst of each other.

Phonetic Similarity algorithms result in Soundex encodings, which sidestep misspellings and variations, by 

indexing a table of language-specific homophones for a string’s Soundex encoding rather than searching the 

string itself. Two critical inputs to phonetic similarity are (1) discerning which language the string is written in 

and (2) knowing the context of the letters you are matching. The crucial former prerequisite is accomplished by 

matching pronunciation rules of letter sequences using their location in the string (“sch” in German vs. “sz” in 

Polish at beginning of a string). The latter is accomplished by parsing the string into a sequence of phonetic 

tokens according to pronunciation rules in that language. The International Phonetic Alphabet (IPA) is 

popularly used to identify tokens with corresponding sounds, though frequently criticized for being too fine of 

match.

Distance Metrics

While string matching compares strings element-wise, distance metrics incorporate a spatial element, measuring 

the literal distance between two entities using algorithms seen in the table below. The first three are interrelated, easy visualized by plotting the entities to be reconciled on a preference space with x and y axes. A 

discerning eye anticipates the obvious limitation of these, that only a certain number of attributes is practical.

Method Description Examples

Euclidean ‘As-the-crow-flies’ distance L2-Norm, Ruler, Spearman

Mahalanobis Matching for centered and standardized distances

Manhattan Distance if following a grid-like path, turning 

corners

L1-Norm, Taxicab, City-Block, 

Footruler, Rectilinear

Minkowski Generic edit distance, of which Euclidean, 

Manhattan and Chebyshev are instances

Soundex, Translation

Chebyshev Distance along axis on which the objects show 

greatest absolute difference

Lmax-Norm, Chessboard

Text Analytics Pearson Coefficient, Jaccard 

Similarity Coefficient

Vector 

Similarity

Cosine Similarity, TFIDF

Minkowski Distance71 is the generalized distance between two points in a plane. Specialized forms include 

Euclidean, Manhattan and the less-common Chebyshev.

Mathematically, Euclidean Distance is Minkowski Distance squared. Practically, it is the equivalent of the 

bishop in chess in that it moves diagonally, or as-the-crow-flies. The Euclidian Squared Distance Metric is a 

variation with quicker processing time since it does not take the square root.

71 "Minkowski Space," Wikipedia, accessed Sep 12 2016

https://en.wikipedia.org/wiki/Minkowski_space
854 The Promise of Evidence-Based Policymaking

51

Manhattan Distance is mathematically the Minkowski Distance raised to 1; it is the same as Euclidean, except 

for the requirement of absolute value since it is not squared. Practically it is the equivalent of a knight, which 

makes L-shape moves. Its name is coined after the great borough of New York City, where pedestrians and cars 

must obey the laws of street corners. 

Text Analytics

In contrast to the Minkowski distances, which scale similarity on a scale of 0 to 1, Pearson's Coefficient scales 

from –1 to 1, in other words fitting similarity along a line, making it a better choice for non-normalized data and 

when attributes' scales are undefined. Mathematically, it is the ratio between two points' covariance and 

standard deviation.

The Jaccard Similarity Coefficient is mathematically the size, i.e., the existence of defined attributes using a 

binary 0/1, of the intersection of two points divided by the size of the union of the points.

Vector Similarity

First, a quick introduction to Vector Similarity. We construct a VSM (Vector Space Model) as a series of 

vectors quantifying frequency of a selected attribute inside a document. These vectors are subsequently 

assembled into a matrix, allowing easy algebraic manipulation. Two vector similarity functions are of particular 

note:

The widely-known bag of words model is enhanced to a 'bag of terms' with TF-IDF. Weighted TF-IDF 

incorporates local and global parameters, applying a logarithmic scale to account for a term's relative 

importance versus frequency of appearance. This allows the algorithm emphasize less-frequent terms' 

importance. TF-IDF normalizes any bias introduced into the vectors by keyword spanning, most commonly 

with the L2 (Euclidean) Norm. The equation is the row-wise multiple of two matrices: TF (Term Frequency, the 

local parameter): matrix of vectors of selected terms' frequencies of appearance in each document IDF (Inverse 

Document Frequency, the global parameter): diagonal matrix version of vector containing, for each term, the 

log of the number of documents divided by the number of documents in which the selected term appears

Cosine Similarity is most useful when it is known that two points have a high proportion of non-shared 

attributes. Mathematically, the attributes are presented in a vector, allowing the algorithm to find the dot 

product of the two points. It measures the angle of the vector rather than the magnitude. Theoretically this 

results in the angle between the two points' attributes; a 90° angle is perfect dissimilarity.

Relational Matching

Relational Matching algorithms retain many commonalities with Jaccard and Euclidean, but are mathematically 

differentiated since as a group they do not satisfy triangular inequality. Practically speaking, while the 

aforementioned algorithms measure similarity between two documents, relational algorithms broaden the 

playing field, incorporating a third document's attributes into the mix.

While the Tanimoto (Jaccard) Similarity Coefficient is the same as the Jaccard Similarity Coefficient, the 

dissimilarity coefficient is where these two algorithms diverge. This is to say that Tanimoto is a proper 

similarity metric but its distance metric is not mathematically legal since it allows the two points to share 

commonality with a third point, causing it to disprove triangular inequality. In application, Tanimoto is 

preferred over Jaccard in cases when we want to allow the two points, themselves very different, to share 

commonalities with a third point. Mathematically, Tanimoto is the number of intersecting elements divided by 

the number of elements in either point.

Dice's Coefficient is mathematically the number of intersecting attributes divided into the total population of 

attributes, thus, as with Tanimoto, it shares a definition in its similarity metric version but Dice's dissimilarity 
The Promise of Evidence-Based Policymaking 855

52

coefficient is not the same as it does not satisfy triangle equality. Compared to Markowski's, Dice's coefficient 

is sensitive to heterogeneity in data sets and less sensitive to outliers.

A simplistic similarity measure is Common Neighbors, which predicts the likeness between two documents in 

terms of the number of common attributes each of those two documents independently shares with other 

documents.

Adamic/Adar Weighted modifies Common Neighbors to weight attributes that are shared infrequently relatively 

higher than those which are more common across all documents. Mathematically, this is accomplished by 

weighting a shared attribute's vector value with 1/ log of the number of times the attribute is shared across all 

documents.

Hybrid Metrics

Experts inevitably merge foundational seminal concepts together. Thus, this section explains some well 

executed hybrid metrics derived from the ones above.

Method Description

Jaro-Winkler Jaro Distance modified to favor common prefixes

Monge-Elkan Atomic Strings matching with Gotoh

Soft-TFIDF A forgiving version of Cosine & Monge-Elkan

Jaro-Winkler is a hybrid algorithm with its roots in Jaro Distance, edit distance, but incorporates Cosine 

Similarity’s approach towards strings with high degree of dissimilarity and TF-IDF’s concept of applying a 

weight to certain elements. It improves on the basic Jaro Distance by accommodating for strings with a common 

prefix, effectively biasing its matching to favor similarity between two otherwise-dissimilar strings who share a 

common prefix.

Monge-Elkan is sometimes considered synonymously with Smith-Waterman Edit Distance, but the two are 

differentiated as Monge-Elkan uses the Gotoh Distance. The confusion is understandable, as Gotoh amends 

Smith-Waterman distance by accommodating affine gaps. Practically, it applies the combined power of 

Levenshtein and Jaro Similarity Measures to n-Gram subsets of strings (called atomic strings). Mathematically, 

Monge-Elkan uses Gotoh edit distance to evaluate atomic strings against each other. Before deciding on 

Monge-Elkan you should understand how sensitive your matching is to the symmetry of your strings, i.e., if one 

string is longer than the other. It has quadratic time complexity due to its recursive calculations.

Soft-TFIDF adds a forgiveness factor to Cosine Similarity and Monge-Elkan, which are intolerant of spelling 

errors as they roll along atomic strings in the order of appearance by incorporating TF-IDF’s concept of matrix 

of terms (i.e., letters) to develop an internal frequency per Atomic string. Soft-TFIDF calculates an inner score 

comparator, thus allowing partial matches.

Fellegi-Sunter Method and the Expectation-Maximization (EM) Algorithm

The Fellegi-Sunter is a common and well established method of probabilistic record linkage. This method is 

used to develop the scores for determining link status. The Fellegi-Sunter model sums the weights the log 

likelihood for each component to determine a match score. The log likelihoods are developed by taking the log 

of the ratio of the m- and u-probabilities. The m- probability is the probability of agreement on a field between 

two records that are a true match. The u-probability is the probability of agreement on a field between two 

records that are not a true match. The single component weight is log �

mmmm

uuuu �, if the field agrees and 

log �

1−mmmm

1−uuuu�, otherwise. The fields with more distinguishing power will have lower u-probabilities and therefore 

yield a large weight if the fields agree. There are a few different ways to estimate these probabilities. The u 
856 The Promise of Evidence-Based Policymaking

53

probabilities can be estimated as a ratio of the frequency of the values divided by the number of pairs in the 

comparison space. The m probabilities can be estimated by taking samples of pairs and calculating a match rate 

based on human review or can be based on prior knowledge. A common method would be to use the EM 

algorithm to estimate both the m- and u-probabilities using the observed agreement patterns in the data. The EM 

algorithm starts with a sometimes arbitrary estimate of the m- and u-probabilities along with an estimate of the 

true match rate. The first step or expectation step is to use these initial parameters to estimate the probability of 

observing an agreement pattern among all the components given they are a true match for each record pair. 

Using these probabilities and the estimated match rate, the probability of a true match given the observed 

agreement pattern is calculated. Next the complete log-likelihood is separated into three maximization problems 

to solve for the new estimates of the m- and u- probabilities along with the true match rate. This process is 

repeated until some convergence criteria is met. See Using the EM Algorithm for Weight Computation in the 

Fellegi-Sunter Model of Record Linkage72 by William Winkler to read this process is detail.

Naive Bayes Machine Learning: Naive Bayes Machine Learning methods have also been explored for entity 

resolution73.

Ensemble Methods

In addition to many of the individual techniques described above, ensemble or composite approaches may help 

to enhance the quality of matching efforts. Given that many of the matching tasks are on text fields or text 

within fields, and string distance measures are often used to quantify the degrees of similarity and dissimilarity, 

leveraging the benefit of several measures may yield better results than relying on any one measure. Previous 

work (see Tejada et. al. 2001, Cohen and Richman 2002, and Bilenko and Mooney, 2002 in the supplemental 

bibliography) has empirically shown that compositing individual measures may offer better performance. A 

simple example of this method is provided with the code examples, and represents one of many possible 

approaches that could be taken in looking at finding establishments in data with a high degree of variability in 

how the establishment names and addresses are coded.

Evaluating Results of Matching and Entity Resolution Approaches

Cutoff Scores

When linking data, there could be an unfeasible amount of records that could be flagged for review given the 

size of the data sets being linked. Given consideration to the amount of resources that would be needed to 

review the links there are additional methods that reduce the amount of human review. Dusetzina, Tyree, 

Meyer, et al.74 suggests a single cutoff developed by Cook75 that uses a single cutoff which allows an 

acceptable distance between the starting weight and the desired weight. The acceptable distance is determined 

by the researcher’s business need on the desired specificity and sensitivity. To increase the number of true 

matches at the cost of introducing more false positives the researcher would use a more liberal cutoff. If the 

72 Winkler, W. (2002). Using the EM Algorithm for Weight Computation in the Fellegi-Sunter Model of Record Linkage 73 Yun Zhou , Minlue Wang, Valeriia Haberland, John Howroyd, Sebastian Danicic, and J. Mark (2015) Improving Record Linkage 

Accuracy with Hierarchical Feature Level Information and Parsed Data

https://research.gold.ac.uk/17342/1/Yun_AMBN_2015_journal.pdf

74 Dusetzina, SB., Tyree, S., Meyer, AM., et al. (2014). Linking Data for Health Services Research: A framework and Instructional 

Guide

75 Cook LJ, Olson LM, Dean JM. (2001) Probabilistic record linkage: relationships between file sizes, identifiers, and match weights.

Methods Information Med. 2001;40:196-203. PMID: 11501632.

http://www.ncbi.nlm.nih.gov/books/NBK253313/pdf/Bookshelf_NBK253313.pdf
The Promise of Evidence-Based Policymaking 857

54

goal is to only keep record pairs with high probabilities of a match the researcher would set a more conservative 

cutoff at the risk of increasing false negatives. 

Other considerations for cutoffs must be made on whether the matches should be one-to-one, one-to-many, or 

many-to-many. In the case of one-to-one matches the “greedy” strategy can be employed. The greedy strategy 

accepts the best match therefore it can only be used for one-to-one matching. This involves taking the pair that 

has the highest score for a given record. This can be employed without a cutoff if the assumption is a match 

exists for every record in a given file. In addition a combination of using the cutoff with the greedy method can 

be used when the assumption is a match may or may not exist and if it does there will only be one.

Performance Evaluation

A common problem with the probabilistic method is a way to evaluate the quality of the links. In most cases the 

sensitivity, specificity, positive predictive value and negative predictive value are analyzed. There are different 

ways to calculate these such as taking samples of the pairs and manual reviewing the pairs and characterizing 

them as true/false positive and true/false negative. However, this process is very resource intensive. Another 

option would be to use a training set to evaluate the method using known matches and non-matches but this 

scenario is usually not available. 

Machine Learning in Evaluation

Another evaluation option would be to use a training set to evaluate the method using known matches and nonmatches but this scenario is usually not available, however if a curated data set already exists, this may be an 

option. A machine learning exercise can help in evaluating weighting and thresholds to be applied in tuning 

some of the other matching strategies, where a data set containing verified matches would be used to train the 

algorithm and then be used to analyze values returned by the match algorithms to aid in determining what 

combinations of algorithms are effective; what thresholds and tunings should be used for each algorithm; and

weightings and approaches, whether hierarchical or otherwise should be used in using multiple algorithms in 

combination.
858 The Promise of Evidence-Based Policymaking

55

APPENDIX C: Best Practices - Bibliography

Bengfort, B. (2008). “Entity Resolution for Big Data: A Summary of the KDD 2013 Tutorial Taught by Dr. 

Lise Getoor and Dr. Ashwin Machanavajjhala.” http://www.datacommunitydc.org/blog/2013/08/entityresolution-for-big-data

Bengfort, B. (2015). “A Primer on Entity Resolution.” http://www.slideshare.net/BenjaminBengfort/a-primeron-entity-resolution

Bilenko, M and Mooney, R. (2003). “Adaptive duplicate detection using learning string similarity measures.” 

Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.

Bilenko, M. (2006). “Learnable Similarity Functions and Their Application to Record Linkage and Clustering.” 

http://www.cs.utexas.edu/~ml/papers/marlin-dissertation-06.pdf

Bilgic, M., Licamele, L., Getoor, L. and Shneiderman, B. (2006). D-Dupe: An interactive tool for entity 

resolution in social networks. Proceedings of IEEE Symposium on Visual Analytics Science and Technology.

BYU Data Mining Lab - Record Linkage Resources 

https://facwiki.cs.byu.edu/DML/index.php/Record_Linkage_Resources

Chen, Z., Kalashnikov, D., and Mehrotra, S. (2009). “Exploiting context analysis for combining multiple entity 

resolution systems.” Proceedings of the 2009 ACM SIGMOD International Conference on Management of 

Data.

Christen, P. (2008). “Automatic record linkage using seeded nearest neighbour and support vector machine 

classification.” Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and 

Data Mining.

Cochinwala, M. (2001). “Efficient data reconciliation.” Information Sciences 137.1, 1-15.

Cohen, William W., and Jacob Richman (2002). “Learning to Match and Cluster Large High-Dimensional Data 

Sets for Data Integration.” In KDD 2002: 475-480. http://www.cs.cmu.edu/~wcohen/postscript/kdd-2002.pdf

Daggy, J., Xu, H., et al. (2013). A practical approach for incorporating dependence among fields in 

probabilistic record linkage

“D-Dupe: A Novel Tool for Interactive Data Deduplication and Integration.” 

http://linqs.cs.umd.edu/projects/ddupe/

Elmagarmid, Ahmed K., Ipeirotis, P., Verykios, V. (2007) “Duplicate Record Detection: A Survey.” 

https://www.cs.purdue.edu/homes/ake/pub/TKDE-0240-0605-1.pdf

Grannis, S., Overhage, M., et al. (2003). Analysis of a Probabilistic Record Linkage Technique without Human

Review
The Promise of Evidence-Based Policymaking 859

56

Hadjieleftheriou, M. and Srivastava, D. (2010). “Weighted Set-Based String Similarity.” 

http://sites.computer.org/debull/A10mar/divesh-paper.pdf

Herzog, T., Scheuren, F. and Winkler, W. (2007). “Data quality and record linkage techniques.” Springer 

Science & Business Media.

Fellegi, I. and Sunter, A. (1969). "A theory for record linkage." Journal of the American Statistical Association 

64:328. http://courses.cs.washington.edu/courses/cse590q/04au/papers/Felligi69.pdf

Gupta, R. and Sarawagi, S. (2009). “Answering table augmentation queries from unstructured lists on the web.” 

Proceedings of the VLDB Endowment 2.1, 289-300.

Kang, H., Getoor, L. and Singh, L. (2007). C-Group: A visual analytic tool for pairwise analysis of dynamic 

group membership. Proceedings of IEEE Symposium on Visual Analytics Science and Technology.

Kang, H., Sehgal, V. and Getoor, L. (2007). GeoDDupe: A novel interface for interactive entity resolution in 

geospatial data. Proceedings of Information Visualisation, 489-496.

Kang, H., Getoor, L., Shneiderman, B., Bilgic, M. and Licamele, L. (2008). “Interactive entity resolution in 

relational data: A visual analytic tool and its evaluation.” IEEE Transactions on Visualization and Computer 

Graphics, Volume 14, Number 5, 999-1014.

Kardes, H., Konidena, D., Agrawal, S., Huff, M., and Sun, A. (2013). “Graph-based Approaches for 

Organization Entity Resolution in MapReduce.” http://www.aclweb.org/anthology/W13-5010

Navarro, G., Baeza-Yates, R., Sutineny, E., and Tarhioz, J. (2001). “Indexing Methods for Approximate String 

Matching.” http://www.dcc.uchile.cl/~gnavarro/ps/deb01.pdf

Raffoa, J. and Lhuillery, S. (2009). “How to Play the ‘Names Game’: Patent Retrieval Comparing Different 

Heuristics.” http://infoscience.epfl.ch/record/161961/files/Lhuillery%20RP2009b.pdf

Ravikumar, P. and Cohen, W. (2004). “A hierarchical graphical model for record linkage.” Proceedings of the 

20th Conference on Uncertainty in Artificial Intelligence. AUAI Press.

“Record Linkages.” https://en.wikipedia.org/wiki/Record_linkage

Sarawagi, S and Bjamidipaty. (2002). “Interactive deduplication using active learning.” Proceedings of the 

Either ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.

Steorts, R., Hall, R., Fienberg, S. (2015). "A Bayesian Approach to Graphical Record Linkage and Deduplication." http://arxiv.org/abs/1312.4645

Tejada, S., Knoblock, C., and Minton, S. (2001). “Learning object identification rules for information 

integration.” Information Systems 26.8, 607-633.

Winkler, W. (2006). “Overview of record linkage and current research directions.” Bureau of the Census.

Winkler, W. and Thibaudeau, Y. (1990). “An Application of the Fellegi-Sunter Model of Record Linkage to the 

1990 U.S. Decennial Census.” https://www.census.gov/srd/papers/pdf/rr91-9.pdf
860 The Promise of Evidence-Based Policymaking

57

Winkler, W. (2014). Matching and Record Linkage

Zhu, V., Overhage, M., et al. (2009). An Empiric Modification to the Probabilistic Record Linkage Algorithm 

Using Frequency-Based Weight Scaling
The Promise of Evidence-Based Policymaking 861

58

APPENDIX D: Workgroup Methodology

Drawing on strong interest among Federal agencies to achieve efficiencies in matching U.S. employer data 

across Federal data sets for data analysis, evaluations, and statistical activities, based on common needs across 

agencies, OMB convened the Employer Data Matching Workgroup in 2016 to complete the following tasks:

• Document pain-points related to matching and uniquely identifying establishments and firms within and 

between data sets and over time. For example, how are agencies classifying employers in their data and 

what terms are interchangeable across data sets (e.g., establishment level firm, or enterprise)? What 

challenges exist in the data for creating matches? 

• Identify current successful strategies used by agencies to address this challenge in the context of 

analyzing data, conducting evaluations, producing statistics, and identifying where additional strategies 

may be needed to further facilitate this work. This process focuses on coming to agreement on one or a 

few methods that will be effective for multiple agencies.

• Along with OMB staff, work to develop a white paper identifying best practices, and high-level 

implementation steps, on how Federal agencies can achieve efficiencies in identifying and matching 

unique firms and establishments (and the relationship between the two) within and across Federal data 

sets, for the purposes of analyzing data, conducting evaluations, and producing statistics. 

The Workgroup has representative and cross-functional mixture of:

• statistical agencies reporting data on employers,

• evaluation offices examining employers, and

• agencies with Federal programs affecting employers whose data are prime for research, evaluation, and 

data analysis. 

See Table D1 for a complete list of participating offices and component agencies. OMB has also provided 

strong support for this work. The Social Security Administration’s Office of Data Exchange also provided 

subject matter expertise to help inform the Workgroup's development of best practices.
862 The Promise of Evidence-Based Policymaking

59

Table D1: Employer Data Matching Workgroup, Participating Offices and Component Agencies

Department/Agency Component Agency or Office

Department of Agriculture Economic Research Service

Department of Commerce Bureau of Economic Analysis

Department of Commerce Census Bureau

Department of Commerce Commerce Data Service

Department of Commerce International Trade Administration

Department of Commerce Minority Business Development Agency

Department of Commerce National Oceanic and Atmospheric Administration

Department of Commerce

Office of the Secretary, Office of Performance, 

Evaluation, and Risk Management

Department of Education

Institute of Education Sciences, National Center for 

Education Statistics

Department of Health and Human 

Services Agency for Healthcare Research and Quality

Department of Health and Human 

Services

Centers for Medicare and Medicaid Services, Center 

for Medicare & Medicaid Innovation

Department of Housing and Urban 

Development Office of Policy Development and Research

Department of Labor Bureau of Labor Statistics

Department of Labor Chief Evaluation Office

Department of Labor Employee Benefits Security Administration

Department of Labor Occupational Safety and Health Administration

Department of Labor Wage and Hour Division

Department of the Treasury

Internal Revenue Service, Statistics of Income 

Division

Department of the Treasury Office of Financial Research

Department of the Treasury Office of Tax Analysis

Department of Transportation Chief Data Officer

Environmental Protection Agency National Center for Environmental Economics

Environmental Protection Agency Office of Environmental Information

Equal Employment Opportunity 

Commission Office of Information Technology

Equal Employment Opportunity 

Commission Program Research and Surveys Division

General Services Administration Federal Acquisition Service

Securities and Exchange Commission Division of Economic and Risk Analysis

Small Business Administration Office of Performance Management

Figure D1 provides an overview of the Workgroup’s approach. The Workgroup first prioritized agency pain 

points, and potential categories of best practices, to determine where to focus its efforts. Through this work, it 

became clear that there was consensus to review best practices for matching, and to review of long-term, highvalue changes that agencies could implement to improve matching.
The Promise of Evidence-Based Policymaking 863

60

Given the unique scoping and makeup of the Workgroup, members recommended conducting a Workgroupspecific Data Inventory to get information on representative data sets (at a more detailed level than information 

provided through preexisting efforts, such as Data.gov), and a Workgroup-specific Methods Inventory to obtain 

information on representative methods for record linkage and entity resolution, to inform best practices.

• The Workgroup relied primarily on the Data Inventory to generate sections in the white paper related to

long-term best practices. In August and September of 2016, Workgroup members provided information 

on a representative sample of data sets from their agencies contain information on individual employers, 

firms, and/or establishments that have the widest coverage, greatest use, or which they were most 

interested in matching to other data sets.

• The Workgroup relied primarily on its Methods Inventory to generate Best Practices for Matching and 

Data Collection Improvements. The Methods Inventory collected and disseminated best practices for

matching. Specifically, the Workgroup asked for information on successful methods and tools agencies 

use for achieving efficiencies in matching employer, establishment or firm data. 

The Workgroup then supplemented synthesized findings with iterative feedback from member agencies, a 

literature review, and an interagency clearance process. This approach was deemed analytically sufficient by 

the Workgroup’s co-chairs, OMB, and Workgroup members.

Figure D1: Methodology

The Data Inventory produced information on administrative and statistical data sources from a subset of 

participating Agencies, as shown in Table D2:

Come to Consensus Using Common Needs Across Agencies

Refinements for AgencySpecific Issues Finalize Common Taxonomy

Review of Literature to Round Out Best Practices

Existing Standards Methods

Conduct Inventories to Inform Best Practices

Data Inventory Methods Inventory

Determine Categories of Best Practices

Pain Points Analysis Prioritzation of Related Best 

Practices
864 The Promise of Evidence-Based Policymaking

61

Table D2: Agencies and Offices Represented in Data Inventory

Department/Agency

of Data Source

Component Agency or Office

of Data Source

Department of Commerce

Bureau of Economic Analysis, Balance of 

Payments Division

Department of Commerce

Bureau of Economic Analysis, Direct 

Investment Division

Department of Commerce Census Bureau

Department of Commerce

Census Bureau (Survey sponsored by the 

Agency for Healthcare Research and Quality)

Department of Education

Institute of Education Sciences, National 

Center for Education Statistics

Department of Education

Office for Civil Rights and National Center for 

Education Statistics

Department of Health and Human 

Services

Administration for Children and Families, 

Office of Child Support Enforcement

Department of Labor

Bureau of Labor Statistics, Office of 

Employment and Unemployment Statistics

Department of Labor Employee Benefits Security Administration

Department of Labor Occupational Safety & Health Administration

Department of Labor Wage and Hour Division

Department of Transportation Federal Aviation Administration

Department of Transportation Federal Motor Carrier Safety Administration

Department of Transportation Federal Railroad Administration

Department of Transportation

National Highway Traffic Safety 

Administration

Department of Transportation

Pipeline and Hazardous Materials 

Administration, 

Office of Hazmat Safety

Department of Treasury

Statistics of Income Division, Internal Revenue 

Service

Equal Employment Opportunity 

Commission

Program Research and Surveys Division,

Office of Research, Information and Planning

General Services Administration Integrated Award Environment

Securities and Exchange 

Commission (Multiple Offices)

Small Business Administration Office of Capital Access

Small Business Administration Office of Disaster Assistance

Small Business Administration Office of Entrepreneurial Development

72% of the data sources were administrative data sources and not subject to CIPSEA’s requirements, but are, or 

could be, confidential under other statutes or policies; 28% of the data sources were from statistical agencies 

and subject to CIPSEA’s requirements. Among the administrative data sources, the main unit of analysis is at 

the transactional level, and employers or firms referenced in these sources can be at either the establishment or 
The Promise of Evidence-Based Policymaking 865

62

enterprise level. Among the statistical data sources, six (55%) were identified as at, or generally at, the 

establishment level; three (27%) were at the enterprise level; the remaining sources were at another type of level

(e.g. government unit). The Workgroup examined commonalities among fields available in the example 

sources flagged from these agencies to develop common data elements. For example, approximately one-half 

of the sources in the inventory noted at least one identifier which could be used across agencies (e.g. EIN). 

Approximately one-half of the sources had an agency-specific identifier. As about half of the example sources 

included at least one interagency identifier, the Workgroup proceeded to include it in the Common Data 

Elements. 87% of the sources included physical and/or mailing addresses, and nearly all the sources included 

name fields. 

As further described in Appendix A, the Workgroup also reviewed the Data Inventory for suggestions on 

potential authoritative sources, and considerations for reviewing legal barriers, policy and legal interpretations, 

and capacity and resource constraints.

To gather sufficient information to describe best practices in employer record linkage and entity resolution, the 

Workgroup conducted a methods inventory: a questionnaire that was sent to a broad spectrum of Federal 

agencies and offices in July 2016. Responses were obtained from 21 individuals representing 14 different 

Federal agencies or offices. The results are further detailed in Appendix B.

In order to round out the findings from the Data and Methods Inventories, the Workgroup also made note of

relevant sources, and reviewed them in compiling best practices. These sources are listed in Appendix C.

Future work would include further refinement of best practices. Member agencies agreed that the approaches 

we capture are useful and agencies stand to gain from implementing them. There is however, an open question 

on how best to achieve the benefits of the best practices. A second phase of the work would focus on launching 

a methods community for matching employer data to develop refined methods by application, and further 

vetting of long-term best practices for consideration in future policy documents.
866 The Promise of Evidence-Based Policymaking

63

Appendix E: Best Practices Code Examples

Agencies submitted these code examples to support a better understanding of current methods and best 

practices, and illustrate approaches used in data remediation, canonicalization and data matching. This 

appendix provides the code samples in full so that individuals seeking to match data on employers can see the 

specific steps that are taken. Specifically:

• Code Example 1: Census SAS code to canonicalize/standardize string content to facilitate matching

• Code Example 2: Code from Census in SAS to demonstrate matching with Business Register (This code 

requires the SAS Data Quality Server)

• Code Example 3: Code from CEO/DOL to implement probabilistic matching, EM algorithm (This code 

does NOT require the SAS Data Quality Server)

• Code Example 4: Stata code from EBSA/DOL to remediate data quality issues and match data, using 

regular expressions

• Code Example 5: Code from OSHA/DOL to normalize/canonicalize/remediate data quality issues. This 

uses regular expressions, and makes use of information from some fields that code example 1 deletes.

• Code Example 6: Code from CEO/DOL to composite string distance measures

• Code Example 7: Code compiled by Rebecca Bilbro, in Python, to test string matching functions

• Code Example 8: Module used by the Office of Foreign Labor Certification (OFLC), ETA/DOL to 

remediate data quality issues 

Note that many of the code examples that deal with data quality issues search for very specific patterns which 

can cause issues in both exact matching and inexact matching applications, as opposed to searching for 

generally unexpected or anomalous values. The exact code syntax is less important, rather the examples that 

warrant modification illustrate the kinds of actual agency data issues and problems that impede or limit the 

effectiveness of matching. The goal of these code sections is typically to standardize strings for comparison so 

that exact matching can occur, or to increase the accuracy and precision for inexact matching processes by 

reducing the rates of false positives and false negatives. 

Code example 1: Census SAS code to canonicalize/standardize string content to facilitate matching

/* Cleaning up the NETS - Standardizing the NETS */ 

options compress=yes;

libname NETS ''; *where the data is located;

run;

libname output '';

run; * output files libname;

%let state=CA; *CHOOSE THE STATE

/*///// RAW NETS DATA /////*/

proc sort data=nets.nets_&state.2007; * RAW DC/CA FULL DATA FROM 1990 TO 2007 ;

by Company;

run;

/*///// NETS ONLY HQ /////*/
The Promise of Evidence-Based Policymaking 867

64

* Keeping only a subset (headquarters) of the observations; 

data nets_&state.2007hq;

set nets.nets_&state.2007(where=(category='Standalone' or category='Headquarters')) ;

run;

/* Only keeping some NETS variables */

data Nets_&state.2007no (keep=Company Address City State ZipCode); 

set Nets_&state.2007hq(drop= xxx);

run;

proc sql;

create table Nets_&state.2007nodups as

select distinct *

from Nets_&state.2007no;

quit;

/* Keeping only firms in STATE */

data Nets_&state.2007nodups;

set Nets_&state.2007nodups;

if state^="&state." then delete; 

run;

/*//////////// Further Cleaning and Pairing ///////////////*/

/* IF THE DATA IS NOT CAPITALIZED WE NEED TO CAPITALIZED IT BEFORE */

%DQLOAD(DQLOCALE=(ENUSA), DQSETUPLOC="sas pathname");

%macro standard(n,s,file);

data &file._st;

set &file;

&n=tranwrd(&n,".","");

&n=tranwrd(&n,",","");

&n=tranwrd(&n,"& ","");

&n=tranwrd(&n,"&","");

&n=tranwrd(&n,"'","");

&n=tranwrd(&n,"#","");

&n=tranwrd(&n,"-","");

&n=tranwrd(&n,"/","");

&n=tranwrd(&n," "," ");

&n=tranwrd(&n,' INC ', '');

&n=tranwrd(&n,'LLC ', '');

&s=tranwrd(&s,".","");
868 The Promise of Evidence-Based Policymaking

65

&s=tranwrd(&s,",","");

&s=tranwrd(&s,"& ","");

&s=tranwrd(&s,"&","");

&s=tranwrd(&s,"'","");

&s=tranwrd(&s,"#","");

&s=tranwrd(&s,"-","");

&s=tranwrd(&s,"/","");

&s=tranwrd(&s," "," ");

&s=tranwrd(&s,"P O ", "PO ");

&n=tranwrd(&n,' THE ','');

&n=tranwrd(&n,'THE ','');

&n=tranwrd(&n,' OF ','');

&n=tranwrd(&n,","," ");

&n=tranwrd(&n,"'","");

&n=tranwrd(&n,"-"," ");

&n=tranwrd(&n,"&"," ");

&n=tranwrd(&n,' CO ','');

&n=tranwrd(&n,' COMPANY ','');

&n=tranwrd(&n,' CORP ','');

&n=tranwrd(&n,' INC ','');

&n=tranwrd(&n,' L P ','');

&n=tranwrd(&n,' LP ','');

&n=tranwrd(&n,'LLC ','');

&n=tranwrd(&n,'L L C ','');

&n=tranwrd(&n,' PL ','');

&n=tranwrd(&n,' P L ','');

&n=tranwrd(&n,'PLLC ','');

&n=tranwrd(&n,'P L L C ','');

&n=tranwrd(&n,'GRP ','');

&n=tranwrd(&n,'LTD ','');

&n=tranwrd(&n,'LLP ','');

&n=tranwrd(&n,' AND ','');

&n=tranwrd(&n,' ASSOC ','');

&n=tranwrd(&n,' ASSOCS ','');

&n=tranwrd(&n,' DBA ','');

&n=tranwrd(&n,' D B A ','');

&n=tranwrd(&n,' MD ','');

&n=tranwrd(&n,' M D ','');

&n=tranwrd(&n,' DMD ','');

&n=tranwrd(&n,' D M D ','');

&n=tranwrd(&n,' PS ','');

&n=tranwrd(&n,' P S ','');

&n=tranwrd(&n,' MSD ','');
The Promise of Evidence-Based Policymaking 869

66

&n=tranwrd(&n,' M S D ','');

&n=tranwrd(&n,' DVM ','');

&n=tranwrd(&n,' D V M ','');

&n=tranwrd(&n,' ESQ ','');

&n=tranwrd(&n,' DDS ','');

&n=tranwrd(&n,' D D S ','');

&n=tranwrd(&n,' DR ','');

&n=tranwrd(&n,' D R ','');

&n=tranwrd(&n,' OD ','');

&n=tranwrd(&n,' O D ','');

&n=tranwrd(&n,' PC ','');

&n=tranwrd(&n,' P C ','');

&n=tranwrd(&n,' PA ','');

&n=tranwrd(&n,' P A ','');

&n=tranwrd(&n,' S C ','');

&n=tranwrd(&n,' CPA ','');

&n=tranwrd(&n,' CPAS ','');

&n=tranwrd(&n,' C P A ','');

&n=tranwrd(&n,' PARTNERS ','');

&n=tranwrd(&n,' PARTNER ','');

&n=tranwrd(&n,' SYSTS ','');

&n=tranwrd(&n,' SVC ','');

&n=tranwrd(&n,' SVCS ','');

&n=tranwrd(&n,' SLTNS ','');

&n=tranwrd(&n,' HOLDINGS ','');

&n=tranwrd(&n,' HOLDING ','');

&n=tranwrd(&n,' INTRNTL ','');

&n=tranwrd(&n,' U S ',' US ');

&n=tranwrd(&n,' PC ','');

&n=tranwrd(&n,' PTR ','');

&n=tranwrd(&n,' GEN ','');

&n=tranwrd(&n,' MBR ','');

&n=tranwrd(&n,' MGR ','');

&n=tranwrd(&n,' MEMBER ','');

&n=tranwrd(&n,' ET AL ','');

&n=tranwrd(&n,' SOLE ','');

&n=tranwrd(&n,' SINGLE ','');

&n=tranwrd(&n,' SNGL ','');

&n=tranwrd(&n,' OWNER ','');

a=index(&n,"INC ");if a=1 then &n='';drop a;

if scan(&n,1)='U' and scan(&n,2)='S' then &n=tranwrd(&n,'U S ',' US ');

if strip(&n)='PL' then &n='';

if strip(&n)='P L' then &n='';
870 The Promise of Evidence-Based Policymaking

67

if strip(&n)='PA' then &n='';

if strip(&n)='P A' then &n='';

if strip(&n)='HOLDINGS' then &n='';

if strip(&n)='ASSN' then &n='';

if strip(&n)='ASSOCS' then &n='';

if strip(&n)='LAW' then &n='';

if strip(&n)='SVCS' then &n='';

if strip(&n)='A PROFESSIONAL' then &n='';

if strip(&n)='FNDTN' then &n='';

if strip(&n)='L P' then &n='';

if strip(&n)='LP' then &n='';

if strip(&n)='CORP' then &n='';

if strip(&n)='INST' then &n='';

if strip(&n)='MGMT' then &n='';

if strip(&n)='LIABILITY' then &n='';

if strip(&n)='CO' then &n='';

if strip(&n)='INCOPORATED' then &n='';

if strip(&n)='INCORPARATED' then &n='';

if strip(&n)='INCORPERATED' then &n='';

if strip(&n)='INCORPORACTED' then &n='';

if strip(&n)='INCORPORADO' then &n='';

if strip(&n)='INCORPORATION' then &n='';

if strip(&n)='INCORPORAID' then &n='';

if strip(&n)='INCORPORATD' then &n='';

if strip(&n)='INCORPORATE' then &n='';

if strip(&n)='INCORPORATOR' then &n='';

if strip(&n)='INCORPORRATED' then &n='';

if strip(&n)='INCORPORTED' then &n=''; *strip all leading and trail blanks removed;

&n=strip(compbl(&n)); *reduce the spaces between words to one space;

&s=tranwrd(&s,".","");

&s=tranwrd(&s,",","");

&s=tranwrd(&s,"& ","");

&s=tranwrd(&s,"&","");

&s=tranwrd(&s,"'","");

&s=tranwrd(&s,"#","");

&s=tranwrd(&s,"-","");

&s=tranwrd(&s,"/","");

&s=tranwrd(&s," "," ");

&s=tranwrd(&s,"P O ","PO ");

&s=tranwrd(&s," ZERO "," 0 ");

&s=tranwrd(&s," ONE "," 1 ");

&s=tranwrd(&s," TWO "," 2 ");

&s=tranwrd(&s," THREE "," 3 ");
The Promise of Evidence-Based Policymaking 871

68

&s=tranwrd(&s," FOUR "," 4 ");

&s=tranwrd(&s," FIVE "," 5 ");

&s=tranwrd(&s," SIX "," 6 ");

&s=tranwrd(&s," SEVEN "," 7 ");

&s=tranwrd(&s," EIGHT "," 8 ");

&s=tranwrd(&s," NINE "," 9 ");

&s=tranwrd(&s," TEN "," 10 ");

&s=tranwrd(&s," ELEVEN "," 11 ");

&s=tranwrd(&s," TWELVE "," 12 ");

&s=tranwrd(&s," THIRTEEN "," 13 ");

&s=tranwrd(&s," FOURTEEN "," 14 ");

&s=tranwrd(&s," FIFTEEN "," 15 ");

&s=tranwrd(&s," SIXTEEN "," 16 ");

&s=tranwrd(&s," SEVENTEEN "," 17 ");

&s=tranwrd(&s," EIGHTEEN "," 18 ");

&s=tranwrd(&s," NINETEEN "," 19 ");

&s=tranwrd(&s,"ZERO "," 0 ");

&s=tranwrd(&s,"ONE "," 1 ");

&s=tranwrd(&s,"TWO "," 2 ");

&s=tranwrd(&s,"THREE "," 3 ");

&s=tranwrd(&s,"FOUR "," 4 ");

&s=tranwrd(&s,"FIVE "," 5 ");

&s=tranwrd(&s,"SIX "," 6 ");

&s=tranwrd(&s,"SEVEN "," 7 ");

&s=tranwrd(&s,"EIGHT "," 8 ");

&s=tranwrd(&s,"NINE "," 9 ");

&s=tranwrd(&s,"TEN "," 10 ");

&s=tranwrd(&s,"ELEVEN "," 11 ");

&s=tranwrd(&s,"TWELVE "," 12 ");

&s=tranwrd(&s,"THIRTEEN "," 13 ");

&s=tranwrd(&s,"FOURTEEN "," 14 ");

&s=tranwrd(&s,"FIFTEEN "," 15 ");

&s=tranwrd(&s,"SIXTEEN "," 16 ");

&s=tranwrd(&s,"SEVENTEEN "," 17 ");

&s=tranwrd(&s,"EIGHTEEN "," 18 ");

&s=tranwrd(&s,"NINETEEN "," 19 ");

&s=tranwrd(&s," FIRST "," 1ST ");

&s=tranwrd(&s," SECOND "," 2ND ");

&s=tranwrd(&s," THIRD "," 3RD ");

&s=tranwrd(&s," FOURTH "," 4TH ");

&s=tranwrd(&s," FIFTH "," 5TH ");

&s=tranwrd(&s," SIXTH "," 6TH ");

&s=tranwrd(&s," SEVENTH "," 7TH ");
872 The Promise of Evidence-Based Policymaking

69

&s=tranwrd(&s," EIGHTH "," 8TH ");

&s=tranwrd(&s," NINTH "," 9TH ");

&s=tranwrd(&s," TENTH "," 10TH ");

&s=tranwrd(&s," ELEVENTH "," 11TH ");

&s=tranwrd(&s," TWELFTH "," 12TH ");

&s=tranwrd(&s," THIRTEENTH "," 13TH ");

&s=tranwrd(&s," FOURTEENTH "," 14TH ");

&s=tranwrd(&s," FIFTEENTH "," 15TH ");

&s=tranwrd(&s," SIXTEENTH "," 16TH ");

&s=tranwrd(&s," SEVENTEENTH "," 17TH ");

&s=tranwrd(&s," EIGHTEENTH "," 18TH ");

&s=tranwrd(&s," NINETEENTH "," 19TH ");

&s=tranwrd(&s," TWENTY "," 20 ");

&s=tranwrd(&s," THIRTY "," 30 ");

&s=tranwrd(&s," FORTY "," 40 ");

&s=tranwrd(&s," FIFTY "," 50 ");

&s=tranwrd(&s," SIXTY "," 60 ");

&s=tranwrd(&s," SEVENTY "," 70 ");

&s=tranwrd(&s," EIGHTY "," 80 ");

&s=tranwrd(&s," NINETY "," 90 ");

&s=tranwrd(&s,"TWENTY "," 20 ");

&s=tranwrd(&s,"THIRTY "," 30 ");

&s=tranwrd(&s,"FORTY "," 40 ");

&s=tranwrd(&s,"FIFTY "," 50 ");

&s=tranwrd(&s,"SIXTY "," 60 ");

&s=tranwrd(&s,"SEVENTY "," 70 ");

&s=tranwrd(&s,"EIGHTY "," 80 ");

&s=tranwrd(&s,"NINETY "," 90 ");

&s=tranwrd(&s," TWENTIETH "," 20TH ");

&s=tranwrd(&s," THIRTIETH "," 30TH ");

&s=tranwrd(&s," FORTIETH "," 40TH ");

&s=tranwrd(&s," FIFTIETH "," 50TH ");

&s=tranwrd(&s," SIXTIETH "," 60TH ");

&s=tranwrd(&s," SEVENTIETH "," 70TH ");

&s=tranwrd(&s," EIGHTIETH "," 80TH ");

&s=tranwrd(&s," NINETIETH "," 90TH ");

&s=tranwrd(&s," EXT ","");

&s=tranwrd(&s," STREET ","");

&s=tranwrd(&s," ST ","");

&s=tranwrd(&s," STE ","");

&s=tranwrd(&s," RD ","");

&s=tranwrd(&s," ST ","");

&s=tranwrd(&s,"BLVD","");
The Promise of Evidence-Based Policymaking 873

70

&s=tranwrd(&s," LN ","");

&s=tranwrd(&s," PL ","");

&s=tranwrd(&s,"PKWY","");

&s=tranwrd(&s,"PRKWY","");

&s=tranwrd(&s,"PWY","");

&s=tranwrd(&s,"FREEWAY","");

&s=tranwrd(&s,"FRWY","");

&s=tranwrd(&s,"FWY","");

&s=tranwrd(&s,"HIGHWAY","");

&s=tranwrd(&s,"HWY","");

&s=tranwrd(&s," STATE ","");

&s=tranwrd(&s," ROAD ","");

&s=tranwrd(&s," RR ","");

&s=tranwrd(&s," COUNTY ","");

&s=tranwrd(&s," CNTY ","");

&s=tranwrd(&s," US ","");

&s=tranwrd(&s," EXPRESSWAY ","");

&s=tranwrd(&s," EXPWY ","");

&s=tranwrd(&s," EXPY ","");

&s=tranwrd(&s," DR ","");

&s=tranwrd(&s," PIKE ","");

&s=tranwrd(&s," TERRACE ","");

&s=tranwrd(&s," TERR ","");

&s=tranwrd(&s," TER ","");

&s=tranwrd(&s," CT ","");

&s=tranwrd(&s," CIR "," ");

&s=tranwrd(&s," CTR ","");

&s=tranwrd(&s," WAY ","");

&s=tranwrd(&s," AVE ","");

&s=tranwrd(&s," PLZ ","");

&s=tranwrd(&s," PLAZA ","");

&s=tranwrd(&s," CROSSING ","");

&s=tranwrd(&s," XING ","");

&s=tranwrd(&s," FLOOR ","");

&s=tranwrd(&s," FLOO ","");

&s=tranwrd(&s," FLR ","");

&s=tranwrd(&s," FL ","");

&s=tranwrd(&s," LOT ","");

&s=tranwrd(&s," APT ","");

&s=tranwrd(&s," SUITE ","");

&s=tranwrd(&s," ROOM ","");

&s=tranwrd(&s," RM ","");

&s=tranwrd(&s," UNIT ","");
874 The Promise of Evidence-Based Policymaking

71

&s=tranwrd(&s," UNITS ","");

&s=tranwrd(&s," NUM ","");

&s=tranwrd(&s,"#","");

&s=tranwrd(&s," NO ","");

&s=tranwrd(&s," BLDG ","");

&s=tranwrd(&s," BLD ","");

&s=compbl(&s);

&s=strip(&s); *all leading and trailing blanks remove;

run;

%mend standard;

/*/ Do the loop /*/

%standard(Company, Address,Nets_&state.2007nodups)

Code Example 2: Code from Census in SAS to demonstrate matching with Business Register (This code 

requires the SAS Data Quality Server)

/* PROGRAM THAT MERGES the BUSINESS REGISTER (BR) for a particular year WITH NETS DATA */

/* The Business Register data set is establishment-based and includes business location, organization type (e.g., 

subsidiary or parent), industry classification, and operating data (e.g., receipts and employment).*/ 

/* The nets data has financial information (credit score, financial stress score) at the firm level. The only issue is 

that the majority of the observations in the nets data don't have an identifier that directly links the nets data with 

the business register database. In order to merge these two data sets we will do name and address matching */

/* THIS PROGRAM USES THE CLEAN BR (ONLY FOR CA AND DC) FROM THE 

PROGRAM BR_CLEANUP AND CLEAN NETS DATA FROM THE PROGRAM NETS_CLEANUP */

options compress=yes;

libname NETS ''; *path for the input files ;

run; 

libname NETS2 ''; * path for the output files - the merge data;

run;

LIBNAME br ''; *the BR clean files; 

run;

/*/////////////////////////////////////////////////////////////////

/// Matching NETS to the BR by EIN (EMPLOYER IDENTIFICATION NUMBER) 

//////////////////////////////////////////////////////////////////*/
The Promise of Evidence-Based Policymaking 875

72

*defining macros;

%let yr=2005; *BR year to merge ;

%let year=05; *BR year to merge TWO DIGITS;

%let base=2005; *NETS year to merge;

%let st=CA; *state to merge;

%let statenum=06;

data netsfile;

set nets.nets_&st._&base.; *nets file used in the merge;

run;

data brfile;

set br.br&yr.(where=(sstate="&st")); *br file used in the merge;

run;

data &mergefile &miss1(keep=nname nstreet ncity nzip nstate ) &miss2(keep=sname sstreet scity szip sstate);

format nname nstreet ncity nzip sname sstreet scity szip;

merge &out1(in=a) &out2(in=b);

by ein;

if a and b then output &mergefile;

if a and not b then output &miss1;

if b and not a then output &miss2;

run;

/* ///////////////////////////////////////////////////////////

// NAME AND ADRESS MATCHING PART I 

///////////////////////////////////////////////////////////// */

/* 

Step 1: We first match the NETS in year 1 with the BR in year 1, then we save the obs that we have not match 

from this first step and called this file unmatched1. 

Step 2: We merge unmatched1 with the BR in year 0, then we save the obs we have not match from this second 

step and called this file unmatched2.

Step 3: We merge unmatched2 with the BR in year 2. 

We proceed with these 3 steps to maximize the number of matches.

*/

%macro merge_current_post_pre(pre,current,post,statefips,statetwodigits);

%do iter=1 %to 3;
876 The Promise of Evidence-Based Policymaking

73

*re-defining macros;

%let base=¤t.; *NETS year to merge;

%let st=&statetwodigits.; *state to merge;

%let statenum=&statefips.;

/* This files have been obtained from Nets_cleanup.sas and BR_cleanup.sas; */

* First iteration,merging BR year 1 with NETS year 1

%if &iter.=1 %then %do;

%let yr=&base.; *BR year to merge ;

%let year=%substr(&base.,3,2); *BR year to merge TWO DIGITS;

data netsfile;

set nets.nets_&st._&base.; *nets file use in the merge;

run;

%end;

*Second iteration, merging (unmatched)NETS year 1 with BR year 0 

%else %if &iter.=2 %then %do;

%let yr=&pre.; *BR year to merge ;

%let year=%substr(&pre.,3,2); *BR year to merge TWO DIGITS;

data netsfile;

set nets2.miss_nets10_nets&base._br&base._&st; * missing nets obs used in the previos merge;

run;

%end;

* Third iteration, merging (unmatched)NETS year 1 with BR year 2

%else %if &iter.=3 %then %do;

%let yr=&post.; *BR year to merge ;

%let year=%substr(&post.,3,2); *BR year to merge TWO DIGITS;

data netsfile;

set nets2.miss_nets10_nets&base._br&pre._&st; * missing nets obs used in the previous merge;

run;

%end;

data brfile;

set br.br&yr.(where=(sstate="&st")); *br file use in the merge;

run;

/* ///////////////////////////////////////////////////////////

// NAME AND ADRESS MATCHING PART II - DQmatching

///////////////////////////////////////////////////////////// */
The Promise of Evidence-Based Policymaking 877

74

*Match everything using Proc DQ;

%DQLOAD (DQLOCALE=(ENUSA), DQSETUPLOC='sas pathname');

%macro match_name_address(match_number,s1,s2,input1,out1,matchcode,input2,out2,mergefile,miss1,miss2);

*two most important files here are input1 and input2 (the two tables that you want to merge);

*the rest are the outputs;

***********************First match - Name and Address *************************;

proc dqmatch data=&input1 out=&out1 matchcode=&matchcode;

criteria

var=nname matchdef='organization' sensitivity=&s1;

criteria

var=nstreet matchdef='address' sensitivity=&s2;

run;

proc dqmatch data=&input2 out=&out2 matchcode=&matchcode;

criteria

var=sname matchdef='organization' sensitivity=&s1;

criteria

var=sstreet matchdef='address' sensitivity=&s2;

run;

proc sort data=&out1;

by &matchcode;

run;

proc sort data=&out2;

by &matchcode;

run;

data &mergefile &miss1(keep=nname nstreet ncity nzip nstate) &miss2(keep=sname sstreet scity szip sstate);

format nname nstreet ncity nzip sname sstreet scity szip;

merge &out1(in=a) &out2(in=b);

by &matchcode;

match=&match_number;

if a and b then output &mergefile;

if a and not b then output &miss1;

if b and not a then output &miss2;

run;

%mend match_name_address;

options nomprint; *mprint;

*Pass1 ; 
878 The Promise of Evidence-Based Policymaking

75

%match_name_address(1,80,80,netsfile,nets2.nets_match1,match_cd1,brfile,nets2.br_match1,nets2.mergepass1

,miss_nets,miss_br)

*Pass2 - The inputs for the second pass will be the missing matches from the first pass;

%match_name_address(2,80,55,miss_nets,nets2.nets_match2,match_cd2,miss_br,nets2.br_match2,nets2.merge

pass2,miss_nets2,miss_br2)

*Pass3 ;

%match_name_address(3,78,78,miss_nets2,nets2.nets_match3,match_cd3,miss_br2,nets2.br_match3,nets2.mer

gepass3,miss_nets3,miss_br3)

/* we could also chose to do more passes with different sensitivity, but in our case they were not good 

matches*/

* Deleting data sets;

proc datasets library=nets2;

delete Br_match: Nets_match: ;

quit;

***********************Second match - Name and Zip*************************;

%macro match_name_zip(match_number,s1,s2,input1,out1,matchcode,input2,out2,mergefile,miss1,miss2);

* Using what we did not match from nets or br we proceed;

proc dqmatch data=&input1 out=&out1 matchcode=&matchcode;

criteria

var=nname matchdef='organization' sensitivity=&s1;

criteria

var=nzip matchdef='text' sensitivity=&s2;

run;

proc dqmatch data=&input2 out=&out2 matchcode=&matchcode;

criteria

var=sname matchdef='organization' sensitivity=&s1;

criteria

var=szip matchdef='text' sensitivity=&s2;

run;

proc sort data=&out1;

by &matchcode;

run;

proc sort data=&out2;

by &matchcode;

run;

data &mergefile &miss1(keep=nname nstreet ncity nzip nstate) &miss2(keep=sname sstreet scity szip sstate);

format nname nstreet ncity nzip sname sstreet scity szip; *ordering the variables in the data set;

merge &out1(in=a) &out2(in=b);
The Promise of Evidence-Based Policymaking 879

76

by &matchcode;

match=&match_number;

if a and b then output &mergefile;

if a and not b then output &miss1;

if b and not a then output &miss2;

run;

%mend match_name_zip;

*Pass4 ;

%match_name_zip(4,93,95,miss_nets3,nets2.nets_match4,match_cd4,miss_br3,nets2.br_match4,nets2.mergepa

ss4,nets2.miss_nets10,nets2.miss_br10)

data nets2.miss_nets10_nets&base._br&yr._&st;

set nets2.miss_nets10;

run;

data nets2.miss_br10_nets&base._br&yr._&st;

set nets2.miss_br10;

run;

proc datasets library=nets2;

delete Br_match: Nets_match: miss_nets10 miss_br10 miss_nets8 miss_br8;

quit;

*****************Organize all of the Match files into One 

File***************************************;

proc datasets library=work;

delete nets_br_match0;

quit;

proc append base=nets_br_match0 data=nets2.mergepass1 force;

run;

proc append base=nets_br_match0 data=nets2.mergepass2 force;

run;

proc append base=nets_br_match0 data=nets2.mergepass3 force;

run;

proc append base=nets_br_match0 data=nets2.mergepass4 force;

run;

/*
880 The Promise of Evidence-Based Policymaking

77

proc append base=nets_br_match0 data=nets2.mergepass5 force;

run;

proc append base=nets_br_match0 data=nets2.mergepass6 force;

run;

proc append base=nets_br_match0 data=nets2.mergepass7 force;

run;

proc append base=nets_br_match0 data=nets2.mergepass8 force;

run;

proc append base=nets_br_match0 data=nets2.mergepass9 force;

run;

proc append base=nets_br_match0 data=nets2.mergepass10 force;

run;

*/

proc datasets library=nets2;

delete mergepass:;

quit;

/* This is the end of the matching, then we can merge our merge data sets to other data sets as the lbd */ 

%merge_current_post_pre(2002,2003,2004,06,CA);

Code Example 3: Code from CEO/DOL to implement probabilistic matching, EM algorithm (This code 

does NOT require the SAS Data Quality Server)

/* Select states and their corresponding ETA Regions*/

proc sql;

select state, reg into :st1-:st&sysmaxlong, :rg1-:rg&sysmaxlong

from ((select distinct(state), '6' as reg from pudf.WP_PY2013Q4_PUBSEEKERS_REG6)

union

(select distinct(state), '5' as reg from pudf.WP_PY2013Q4_PUBSEEKERS_REG5)

union

(select distinct(state), '4' as reg from pudf.WP_PY2013Q4_PUBSEEKERS_REG4)

union

(select distinct(state), '3' as reg from pudf.WP_PY2013Q4_PUBSEEKERS_REG3)

union

(select distinct(state), '2' as reg from pudf.WP_PY2013Q4_PUBSEEKERS_REG2)

union
The Promise of Evidence-Based Policymaking 881

78

(select distinct(state), '1' as reg from pudf.WP_PY2013Q4_PUBSEEKERS_REG1));

quit;

%let n=&sqlobs;

/* Selecting variables to be excluded for data linkage*/

proc sql;

select name into :var1-:var&sysmaxlong

from dictionary.columns

where memname="WP_PY2013Q4_PUBSEEKERS_REG1"

and upcase(name) not in ('ID','FILETYPE','STATE','PROGRAMYEAR','OBS','QUARTER', 

'BIRTH_DT', 'REG_DT','WIBNAME','GENDER','WIB','WHITE','INDIAN','ASIAN','BLACK',

'MULTI','HAWAIIAN','HISPANIC','VET','VETELIG','VET911','VETCAMP','VETDIS',

'VETTAP','VETRECENT','VETHOMELESS','VETTRANS','I','FIRST_SERVICE')

and libname='PUDF';

quit;

%let varnum=&sqlobs;

%macro EM;

/*Intializing p, u- and m-probabilities for the first iteration of the EM algorithm*/

/*Start state loop*/

%do s=1 %to &n;

data kfs.em0&&ST&s;

iter=0;

p=.005;

%do i=1 %to &varnum;

m&&var&i=.9 ; u&&var&i=.1; 

%end;

run;

/*Start iteration loop*/

%let j=0; %let stop=1;

%do %until (&j=50 or &stop=0);

%let j=%eval(&j+1);

/*EM Algorithm*/

data kfs.em&&ST&s;
882 The Promise of Evidence-Based Policymaking

79

sysecho "Vector creation Iteration &j &&st&s &s of &n";

set pudf.WP_PY2013Q4_PUBSEEKERS_REG&&RG&s (obs=1);

declare hash ob(data set:%unquote(%bquote(')pudf.WP_PY2013Q4_PUBSEEKERS_REG&&RG&s 

(where=(state="&&ST&s"))%bquote(')) , multidata:"Y", hashExp:16);

ob.definekey('reg_dt','gender');

ob.definedata(all:'Y');

ob.definedone();

set kfs.em0&&ST&s (where=(iter=%eval(&j-1)));

n=0; n1=0; gm=0;

DO UNTIL (eof);

set PUDF.WP_PY2013Q3_SEEKERS_REG&&RG&s (where=(state="&&ST&s") 

rename=(id=aid %do i=1 %to &varnum; &&var&i=a&&var&i %end;)) end=eof;

n1+1;

call missing(id %do i=1 %to &varnum; ,&&var&i %end;);

rc=ob.find(key:reg_dt, key:gender);

if rc=0 then do;

m=1;u=1;

%do i=1 %to &varnum;

if m&&var&i>0 and u&&var&i>0 and m&&var&i>u&&var&i then do;

if missing(&&var&i)=1 or missing(a&&var&i)=1 or &&var&i^=a&&var&i 

then do;

m=m*(1-m&&var&i); u=u*(1-u&&var&i);

end;

else if &&var&i=a&&var&i then do;

m=m*m&&var&i;

u=u*u&&var&i;

end;

end;

%end;

g=round(p*m/(p*m+(1-p)*u),.00001);

n+1;

gm + g;

%do i=1 %to &varnum;
The Promise of Evidence-Based Policymaking 883

80

if &&var&i=a&&var&i and missing(&&var&i)=0 and missing(a&&var&i)=0 

then do;

g&&var&i+1;

if m&&var&i>0 and u&&var&i>0 and m&&var&i>u&&var&i then do;

gm&&var&i + g;

gu&&var&i + (1-g);

end;

end;

%end;

rc=ob.has_next(RESULT: idother);

do while (idother ne 0);

rc=ob.find_next(key:reg_dt, key:gender);

rc=ob.has_next(result: idother);

m=1;u=1;

%do i=1 %to &varnum;

if m&&var&i>0 and u&&var&i>0 and m&&var&i>u&&var&i then do;

if missing(&&var&i)=1 or missing(a&&var&i)=1 or 

&&var&i^=a&&var&i then do;

m=m*(1-m&&var&i); u=u*(1-u&&var&i);

end;

else if &&var&i=a&&var&i then do;

m=m*m&&var&i;

u=u*u&&var&i;

end;

end;

%end;

g=round(p*m/(p*m+(1-p)*u),.00001);

n+1;

gm + g;

%do i=1 %to &varnum;

if &&var&i=a&&var&i and missing(&&var&i)=0 and missing(a&&var&i)=0 

then do;

g&&var&i+1;
884 The Promise of Evidence-Based Policymaking

81

if m&&var&i>0 and u&&var&i>0 and m&&var&i>u&&var&i then do;

gm&&var&i + g;

gu&&var&i + (1-g);

end;

end;

%end;

end;

end;

/*Calculating new p,u- and m-probability estimates based on EM algorithm*/

if eof=1 then do;

state="&&st&s";

iter=&j;

n2=ob.num_items;

p=min(gm,n1,n2)/n;

%do i=1 %to &varnum;

if m&&var&i>0 and u&&var&i>0 and m&&var&i>u&&var&i then do;

if round(gm&&var&i/gm,.00001)=0 then m&&var&i=.00001;

else if round(gm&&var&i/gm,.00001)=1 then m&&var&i=1-.00001;

else m&&var&i = round(gm&&var&i/gm,.00001);

if round(gu&&var&i/(n-gm),.00001)=1 then u&&var&i=1-.00001;

else if round(gu&&var&i/(n-gm),.00001)=0 then u&&var&i=.00001;

else u&&var&i = round(gu&&var&i/(n-gm),.00001);

end;

u&&var&i = min(u&&var&i,round(g&&var&i/n,.00001));

%end;

keep state iter p n n1 n2 gm %do i=1 %to &varnum; m&&var&i u&&var&i %end;;

output kfs.em&&ST&s;

end;

end;

stop;

run;

%if &syserr ne 0 %then %do;
The Promise of Evidence-Based Policymaking 885

82

%put ERROR: EM for &&st&s was not created;

%let stop=0;

%goto exit;

%end;

/*Checking for convergence of p, u- and m-probabilities 

between current and prior iterations*/

proc sql noprint;

select case when round(abs(a.m&var1-b.m&var1),.00001) 

%do i=2 %to &varnum; +round(abs(a.m&&var&i-b.m&&var&i),.00001) %end; =0

or b.gm>min(b.n1,b.n2) then 0 else 1 end into :stop

from kfs.em0&&ST&s a, kfs.em&&ST&s b

where a.iter=%eval(&j-1);

quit;

%put For iteration &j stop was &stop;

data kfs.em0&&ST&s;

set kfs.em0&&ST&s kfs.em&&ST&s ;

run;

%exit:

%end;

%end;

%mend;

%EM;

Code Example 4: Stata code from EBSA/DOL to remediate data quality issues and match data, using 

regular expressions

drop if accountant_firm_name=="" | accountant_firm_name=="0" | accountant_firm_name=="A"

drop if accountant_firm_ein=="" | accountant_firm_ein=="123456789" | accountant_firm_ein=="111111111" | 

accountant_firm_ein=="000000000" | accountant_firm_ein=="999999999"

*Drop plans with no-names

drop if regexm(accountant_firm_name,"PLEASE") & regexm(accountant_firm_name,"ATTACHMENT")

drop if regexm(accountant_firm_name,"SEE ATTACHMENT") | regexm(accountant_firm_name,"SEE 

ATTACHED") 

drop if regexm(accountant_firm_name,"TBD") 

drop if regexm(accountant_firm_name,"TO BE DETERMINED") 

drop if regexm(accountant_firm_name,"IN PROCESS")

drop if regexm(accountant_firm_name,"ACCOUNTANTS NAME")

drop if regexm(accountant_firm_name,"ABCDEFGHI")

drop if regexm(accountant_firm_name,"PDFDOC")
886 The Promise of Evidence-Based Policymaking

83

/*Cut down by EIN first, then assign the most common name in the data to that EIN*/

/*initial ID is the audit firm name*/

bysort accountant_firm_ein accountant_firm_name: gen num_names=_N

gsort accountant_firm_ein -num_names ack_id

bro accountant_firm_name accountant_firm_ein num_names

bysort accountant_firm_ein: gen num_ein=_N

bysort accountant_firm_ein: keep if _n==1

count

/*generate a shorter variable name*/

gen preproc=accountant_firm_name /*Preprocessed name*/

gen proccessed_name=accountant_firm_name

/*Standardize the Accountant Name*/

quietly replace proccessed_name = trim(proccessed_name)

quietly replace proccessed_name = upper(proccessed_name)

/*Remove funky characters*/

quietly replace proccessed_name=subinstr(proccessed_name,"(","",.)

quietly replace proccessed_name=subinstr(proccessed_name,")","",.)

quietly replace proccessed_name=subinstr(proccessed_name,"%","",.)

quietly replace proccessed_name=subinstr(proccessed_name,"#","",.)

quietly replace proccessed_name=subinstr(proccessed_name,"'","",.)

quietly replace proccessed_name=subinstr(proccessed_name,"`","",.)

quietly replace proccessed_name=subinstr(proccessed_name,","," ",.)

quietly replace proccessed_name=subinstr(proccessed_name,"{","",.)

quietly replace proccessed_name=subinstr(proccessed_name,"}","",.)

quietly replace proccessed_name=subinstr(proccessed_name,"[","",.)

quietly replace proccessed_name=subinstr(proccessed_name,"]","",.)

quietly replace proccessed_name=subinstr(proccessed_name,"\"," ",.)

quietly replace proccessed_name=subinstr(proccessed_name,"/"," ",.)

quietly replace proccessed_name=subinstr(proccessed_name,"&"," AND ",.)

quietly replace proccessed_name=subinstr(proccessed_name,"*","",.)

quietly replace proccessed_name=subinstr(proccessed_name,"!","",.)

quietly replace proccessed_name=subinstr(proccessed_name,"?","",.)

quietly replace proccessed_name=subinstr(proccessed_name,".","",.)

quietly replace proccessed_name=subinstr(proccessed_name,"-","",.)

quietly replace proccessed_name=subinstr(proccessed_name,"_","",.)

/*Doing this to remove any double spaces we will encounter from the / \ & replacements*/

quietly replace proccessed_name = itrim(proccessed_name)

/*Eliminate the most common words*/

quietly replace proccessed_name=subinstr(proccessed_name," AND ","",.)

quietly replace proccessed_name=subinstr(proccessed_name,"ASSOCIATES","",.)
The Promise of Evidence-Based Policymaking 887

84

quietly replace proccessed_name=subinstr(proccessed_name," LLC","",.)

quietly replace proccessed_name=subinstr(proccessed_name," CPAS","",.)

quietly replace proccessed_name=subinstr(proccessed_name," PLLC","",.)

quietly replace proccessed_name=subinstr(proccessed_name," INC","",.)

quietly replace proccessed_name=subinstr(proccessed_name," LTD","",.)

quietly replace proccessed_name=subinstr(proccessed_name," LLP","",.)

quietly replace proccessed_name=subinstr(proccessed_name," CPA","",.)

quietly replace proccessed_name=subinstr(proccessed_name," PC","",.)

quietly replace proccessed_name=subinstr(proccessed_name,"COMPANY","",.)

quietly replace proccessed_name=subinstr(proccessed_name,"ASSOC","",.)

quietly replace proccessed_name=subinstr(proccessed_name,"FIRM","",.)

/*Get rid of the spacings*/

gen test=subinstr(proccessed_name," ","",.)

/*to eliminate the common letter combos as the end of strings, only take them off if they are the last two letters 

of the string*/

gen str_length=length(test)

gen lasttwo=substr(test,str_length-1,2)

replace test=substr(test,1,str_length-2) if inlist(lasttwo, "PA", "CO", "PC")

/*Only test on the first 20 characters of the auditor names*/

gen substring=substr(test,1,20)

/*This is the grouping substring function. I have set the matching threshold to be two edits for the longest string 

(2/20=.1)*/

/*Also prevents anything from shorter than 9 (1/9=.1111) to be matched if there are any differences*/ 

strgroup substring, gen(strgroupid) threshold(.175) first force

sort strgroupid

bro accountant_firm_ein accountant_firm_name strgroup

bysort strgroupid: gen flag_auditor_id=_n

count if flag_auditor_id==1

rename strgroupid summit_auditor_id

keep summit_auditor_id proccessed_name substring accountant_firm_ein

sort accountant_firm_ein

Code Example 5: Code from OSHA/DOL to normalize/canonicalize/remediate data quality issues. This 

uses regular expressions, and makes use of information from some fields that code example 1 deletes.

/*Name Standardization*/

data kfs.osha_reg5_HI;

set KFS.OSHA_REG5_HI;

If _n_=1 then do;

namekey= prxparse('s/\b(STORES?)? #?\d+$|\-|\\|\/|\.|\,|\bINC\b|\bINCORP[A-Z]+?\b|^THE\b|

\bCORP([A-Z]+)?\b|\bLLC\b|\bCOMPANY\b|\bCO\b|\bD.?B.?A.?\b|\bLTD\b|\(.+\)/ /I');
888 The Promise of Evidence-Based Policymaking

85

end;

retain namekey;

name=strip(compbl(compress(upcase(estab_name),"'")));

name = prxchange(namekey,-1,strip(name));

name = compbl(prxchange('s/ ?& ?|\sAND\s/ & /I',-1,name));

name = prxchange('s/\bUNITED STATES\b|\bU S\b/US/I',-1,name);

name = prxchange('s/\b(UNITED STATES|US) POSTAL SERVICE.*\b/USPS/I',-1,name);

name = prxchange('s/\bUNITED PARCEL SERVICE\b|\bU S\b/UPS/I',-1,name);

name = prxchange('s/\bSERVICES?\b|\bSRVCS\b/SRVC/I',-1,name);

name = prxchange('s/\bCENTERS?\b/CTR/I',-1,name);

name = prxchange('s/\bDEPARTMENT\b/DEPT/I',-1,name);

name = prxchange('s/\bHTLH/HEALTH/I',-1,name);

name = prxchange('s/\bHEALTH CARE/HEALTHCARE/I',-1,name);

name = prxchange('s/\b(SUPER)? ?MARKETS?\b/MARKET/I',-1,name);

name = prxchange('s/^USDOL OSHA.*/USDOL OSHA/I',-1,strip(name));

name = prxchange('s/^WAL.?MART.*/WALMART/I',-1,strip(name));

name = prxchange('s/^TYSON (\w+)? ?(FARMS?|FRESH|FOODS?).*/TYSON FOOD/I',-1,strip(name));

name = prxchange('s/^LOWES\b(OF|HOME|MILLWORK).*/LOWES HOME IMPROVEMENT/I',-

1,strip(name));

name = prxchange('s/^HOME ?DEPOT.*/HOME DEPOT/I',-1,strip(name));

name = prxchange('s/^FED( |-)EX.*/FEDEX/I',-1,strip(name));

name = prxchange('s/\bMANUF[A-Z]+/MFG/I',-1,strip(name));

name = prxchange('s/\bPACK[A-Z]+/PKG/I',-1,strip(name));

name = prxchange('s/.*\bA ?T & T\b.*/AT&T/I',-1,strip(name));

name = compbl(strip(name));

run;

/*Street Standardization*/

/*The majority of the following comes from a paper from a SAS Users Group meeting*/

data kfs.OSHA_REG5_HI;

set KFS.OSHA_REG5_HI;

If _n_=1 then do;

streetkey= prxparse('s/\sST\b|\sSTREET |\sAVE\b|\sAV\b|\sAVENUE\b|\sDR\b|\sDRIVE\b|

\sLN\b|\sLANE\b|\sRD\b|\sROAD\b|\sPKWY\b|\sPARKWAY\b|\sBLVD\b|\sBOULEVARD\b|\sPL\b|

\sPLACE\b|\sPLAZA\b|\sCT\b|\sCRT\b|\sCOURT\b|\sCIR\b|\sCIRCLE\b|\.|\-|\,|\(.+\)/ /I'); end;

retain streetkey;

street=strip(compbl(compress(upcase(site_address),"'")));

street = TRANWRD(street,'NORTH ','N ');

street = TRANWRD(street,'EAST ','E ');

street = TRANWRD(street,'WEST ','W ');

street = TRANWRD(street,'SOUTH ','S ');
The Promise of Evidence-Based Policymaking 889

86

street = prxchange('s/\bNORTHWEST\b|\bN W\b/NW/I',-1,street);

street = prxchange('s/\bNORTHEAST\b|\bN E\b/NE/I',-1,street);

street = prxchange('s/\bSOUTHWEST\b|\bS W\b/SW/I',-1,street);

street = prxchange('s/\bSOUTHEAST\b|\bS E\b/SE/I',-1,street);

street = prxchange(streetkey,-1,strip(street));

street = compbl(prxchange('s/ ?& ?|\sAND\s/ & /I',-1,street));

street = PRXCHANGE('s/#|STE|SUITE|BUILDING|BLDG/ ZTE /',-1,street);

street = PRXCHANGE('s/FRWY|FREEWAY|FWY/ FWY /',-1,street);

street = PRXCHANGE('s/EXPRWY|EXPRESSWAY|EXPWY|EXPY/ EXPY /',-1,street);

street = PRXCHANGE('s/HIWAY|HIGHWAY/ HWY /',-1,street);

street = PRXCHANGE('s/P ?(0|O) ?BOX\b|\bPMB\b|\bP ?O DRAWER\b|

\bPOST OFFICE DRAWER\b|\bDRAWER\b/ ZPB /',-1,street);

street = PRXCHANGE('s/ RM | ROOM / ZRM /',-1,street);

street = TRANWRD(street,'FIRST ','1ST ');

street = TRANWRD(street,'SECOND ','2ND ');

street = TRANWRD(street,'THIRD ','3RD ');

street = TRANWRD(street,'FOURTH ','4TH ');

street = TRANWRD(street,'FIFTH ','5TH ');

street = TRANWRD(street,'SIXTH ','6TH ');

street = TRANWRD(street,'SEVENTH ','7TH ');

street = TRANWRD(street,'EIGHTH ','8TH ');

street = TRANWRD(street,'NINTH ','9TH ');

street = TRANWRD(street,'TENTH ','10TH ');

street = TRANWRD(street,'ONE ','1 ');

street = TRANWRD(street,'TWO ','2 ');

street = TRANWRD(street,'THREE ','3 ');

street = TRANWRD(street,'FOUR ','4 ');

street = TRANWRD(street,'FIVE ','5 ');

street = TRANWRD(street,'SIX ','6 ');

street = TRANWRD(street,'SEVEN ','7 ');

street = TRANWRD(street,'EIGHT ','8 ');

street = TRANWRD(street,'NINE ','9 ');

street = TRANWRD(street,'TEN ','10 ');

street = PRXCHANGE('s/TWENTY.?(?=[0-9])/2/I',-1,STREET);

run;

/*This macro is not mine but thought I'd share. It likely needs extensive customization and review for individual 

needs*/

%MACRO BREAKUP_ADD (PATTERN=,VAR=,NUM=,NEWVAR=);

IF _N_=1 THEN DO;

RETAIN ExpID&NUM;
890 The Promise of Evidence-Based Policymaking

87

PATTERN="/&PATTERN/I";

ExpID&NUM=PRXPARSE(PATTERN);

END;

CALL PRXSUBSTR(ExpID&NUM, &VAR, POSITION&NUM);

/*The first half of the macro creates a DO Loop that looks for the patterns in the addresses that were created in 

the

first macro, e.g. ZRM. When the pattern is found, it outputs the starting position of the pattern.*/

IF POSITION&NUM = 1 THEN DO;

MATCH = SUBSTR(&VAR,POSITION&NUM);

&NEWVAR=MATCH;

END;

IF INDEX(&VAR,"&PATTERN") THEN &NEWVAR=SUBSTR(&VAR,POSITION&NUM);

IF INDEX(&VAR,"&PATTERN") THEN SUBSTR(&VAR,POSITION&NUM)='';

%MEND BREAKUP_ADD;

data kfs.OSHA_REG5_HI (DROP=POSITION1 POSITION2 POSITION3 POSITION4 POSITION5

ExpID1 ExpID2 ExpID3 ExpID4 ExpID5 MATCH PATTERN);

set KFS.OSHA_REG5_HI;

%BREAKUP_ADD (PATTERN =ZPB,VAR=STREET,NUM=1,NEWVAR=PO_BOX_STREET);

%BREAKUP_ADD (PATTERN=ZTE,VAR=STREET,NUM=2,NEWVAR=SUITE_STREET);

%BREAKUP_ADD (PATTERN=ZTE,VAR=PO_BOX_STREET,NUM=3,NEWVAR=SUITE_STREET);

%BREAKUP_ADD (PATTERN=ZRM,VAR=STREET1,NUM=4,NEWVAR=RM_STREET);

%BREAKUP_ADD (PATTERN=ZRM,VAR=SUITE_STREET,NUM=5,NEWVAR=RM_STREET);

PO_BOX_STREET = STRIP(TRANWRD(PO_BOX_STREET,'ZPB',''));

SUITE_STREET = STRIP(TRANWRD(SUITE_STREET,'ZTE',''));

RM_STREET = STRIP(TRANWRD(RM_STREET,'ZRM',''));

PO_BOX_STREET=COMPRESS(PO_BOX_STREET,'#');

SUITE_STREET=COMPRESS(SUITE_STREET,'#');

RUN;

Code Example 6: Code from CEO/DOL to composite string distance measures

Code file in R, along with a sample data file, for evaluating the relative effectiveness and contrasts between 

common string distance measures. The code also demonstrates composite string distance measurement, and 

using high levels of similarity and dissimilarity to identify data quality issues. The code and data file can be 

found at the following address: 

https://github.com/dullandboring/employer-data-matching

Code Example 7: Code compiled by Rebecca Bilbro, in Python, to test string matching functions

The following page contains a variety of tools and code examples to demonstrate and test different string

distance measures. The resources include native capabilities in the base distribution, as well as DIFFLIB, 

FuzzyWuzzy, Jaccard and Jellyfish modules. The code for these varying examples, libraries and functions can 

be found at the following address: 

https://github.com/DruidSmith/Python-Matching-Algorithms/blob/master/String%20Comparison.ipynb
The Promise of Evidence-Based Policymaking 891

88

Code Example 8: Module used by OFLC, ETA/DOL to remediate data quality issues 

The following code is used by the ETA Office of Foreign Labor Certification (OFLC) to standardize string 

content in order to minimize the amount of manual de-duplicating required when trying to match or aggregate 

data.

Option Compare Database

Sub ModifyData()

DoCmd.SetWarnings False

'Convert string to proper case: UPDATE PW SET PW.EMPLOYER_LEGAL_BUSINESS_NAME = 

StrConv([EMPLOYER_LEGAL_BUSINESS_NAME],3)

DoCmd.OpenQuery "UPDATE_Employer_Name_Proper_Case"

'Remove Special Characters from a string: UPDATE PW SET PW.EMPLOYER_LEGAL_BUSINESS_NAME 

= Replace([EMPLOYER_LEGAL_BUSINESS_NAME],"enter special character between quotes"," ")

DoCmd.OpenQuery "UPDATE_Remove_Ampersands"

DoCmd.OpenQuery "UPDATE_Remove_Periods"

DoCmd.OpenQuery "UPDATE_Remove_Commas"

DoCmd.OpenQuery "UPDATE_Remove_Semicolons"

DoCmd.OpenQuery "UPDATE_Remove_Colons"

DoCmd.OpenQuery "UPDATE_Remove_Dashes"

DoCmd.OpenQuery "UPDATE_Remove_Underscores"

DoCmd.OpenQuery "UPDATE_Remove_Open_Paren"

DoCmd.OpenQuery "UPDATE_Remove_Closed_Paren"

DoCmd.OpenQuery "UPDATE_Remove_Apotrophe"

DoCmd.OpenQuery "UPDATE_Remove_Additional_Spaces"

DoCmd.SetWarnings True

MsgBox "Complete"

End Sub
892 The Promise of Evidence-Based Policymaking

Employer Data Matching Workgroup

Data Inventory

Page 1 of 5

Department/

Agency

Component Agency 

or Office

Dataset Name Dataset Type: 

Owned by Data 

Inventory 

Respondent, or 

Desired by Data 

Inventory 

Respondent?

Data Update 

Frequency and Lags

Contact E-mail Address Link to Data Collection 

Tool

Data.gov Link Other Relevant Link(s) Purpose of the Data Collection Access Restrictions Coverage Unit of Analysis Definition of "Company" Corporate Structure and Relationships Quality of Fields for Matching Unique Identifier Fields Name Fields Address Fields Country fields FIPS Code Fields Industry Code Fields Geocoding Other Fields Additional Notes

U.S. Department of 

Commerce

U.S. Census Bureau Business Register Owned Continuously, 

update frequency 

varies by sources; 

lags vary by the 

reference period of 

the sources.

The Business Register is 

updated using information 

from several federal 

agencies, including the IRS, 

the Bureau of Labor 

Statistics (BLS), the Social 

Security Administration 

(SSA) and the Census 

Bureau. There are data 

collection tools linked to 

the sources. For example, 

The Company Organization 

Survey (COS), also known 

as the Report of 

Organization Survey, is an 

annual survey performed 

specifically for the purpose 

of maintaining BR 

information about multiunit establishments’ 

current organization and 

operating information.

https://www.census.gov/e

con/overview/mu0700.ht

ml

http://www.reginfo.gov/pu

blic/do/PRAViewICR?ref_n

br=201502-0607-004

Not Applicable www.census.gov/econ/ove

rview/mu0600.html

To provide a current and comprehensive 

database of U.S. business 

establishments and companies for 

statistical program use.

The Business Register information is 

confidential [Title 13 and Title 26, US 

Code]. Access is restricted to persons 

specially sworn to uphold the 

confidentiality provisions of Title 13 and 

Title 26.

Establishments of all domestic 

businesses (except agriculture, forestry, 

fishing, hunting, rail transportation, the 

U.S. Postal Service, elementary and 

secondary schools, colleges and 

universities, labor organizations, political 

organizations, religious organizations, 

public administration, and private 

households) and organizational units of 

multi-establishment businesses. The 

Business Register (BR) covers more than 

160,000 multi-establishment companies, 

representing 1.8 million affiliated 

establishments, 5 million single 

establishment companies, and nearly 21 

million non-employer businesses.

Establishment-based; An establishment is a single physical 

location where business transactions 

take place and for which payroll and 

employment records are kept. Groups of 

one or more establishments under 

common ownership or control are 

enterprises. A single-unit enterprise 

owns or operates only one 

establishment. A multi-unit enterprise 

owns or operates two or more 

establishments.

Multi-unit enterprises may use multiple 

EINs, and they may use different ones 

when reporting to different federal 

agencies, complicating the matching 

process. The Census Bureau has linked 

many, but not all of the EINs to 

enterprises.

The Business Register is one of the most 

complete, current, and consistent source 

of establishment- based information 

about U.S. businesses, and is essential to 

assuring full coverage and high quality in 

Federal economic statistics programs. 

Examples of quality considerations for 

this source include:

- The annual Company Organization 

Survey covers 30 percent of multi-unit 

companies and a small sample of firms 

that were single-unit firms in the most 

recent quinquennial Economic Census, 

so establishment openings and closings 

in the firms not covered may- The 

business register is divided into 

employer and non-employer business 

registers based on payroll employment. 

Some firms lease their employees from 

Professional Employer Organizations 

(PEOs) or use independent contractors. 

Such firms may appear in the nonemployer business register despite 

having large revenues and many leased 

and/or contract employees. not be 

reflected in the business register until 

after the next Economic Census (though 

the Census Bureau takes measures to 

address this). 

(www.census.gov/econ/overview/mu06

A single-unit enterprise’s primary 

identifier is its Employer Identification 

Number (EIN). 

A unique employer unit identification 

number identifies each establishment 

owned by a multi-unit enterprise on the 

Business Register.

(https://www.census.gov/econ/susb/me

thodology.html)

Yes; Primary and secondary business 

names, permitting identification of both 

legal and trade designations for 

businesses that support them (e.g., 

franchises); character

Yes; Physical location, mailing address, 

post office (city or town) name, postal 

state abbreviation. ZIP+4; character

This field is not currently populated in 

the BR, but it may be possible to link in 

this information from the Company 

Organization Survey

Yes; County and State; character 6-digit NAICS codes;

(https://www.census.gov/econ/susb/me

thodology.html)

Yes; latitude and longitude to six digits 

after the decimal; numeric

Census Tract (character), Census Block 

(character), 4-digit SIC codes (character)

Please also reference the working paper, 

"Documenting the Business Register and 

Related Economic Business Data"

papers.ssrn.com/sol3/papers.cfm?abstr

act_id=2755723 

U.S. Department of 

Labor

U.S. Bureau of 

Labor StatisticsOffice of 

Employment and 

Unemployment 

Statistics (OEUS)

Quarterly Census of 

Employment and 

Wages (QCEW) 

Business Register 

(BR)

Owned The QCEW BR is 

updated quarterly, 

and the data 

become available 6 

months after the 

reference cycle.

QCEWinfo@bls.gov, 

BDMinfo@bls.gov

www.reginfo.gov/public/d

o/PRAViewICR?ref_nbr=20

1406-1220-001

www.reginfo.gov/public/d

o/PRAViewDocument?ref_

nbr=201406-1220-001

catalog.data.gov/dataset/q

uarterly-censusemployment-and-wagearea

www.bls.gov/bdm, 

www.bls.gov/cew

To provide a quarterly census of all 

establishments under State 

unemployment insurance programs, 

representing about 98 percent of 

employment on nonfarm payrolls. This 

database of U.S. business 

establishments serves as the basis for 

multiple statistical programs; Sampling 

frame & benchmark(CES, JOLTS, PPI, 

OES, LAUS, SOII, and NCS, which includes 

the ECI, EBS and ORS), labor market 

research, Business Employment 

Dynamics (BED) data.

All microdata are confidential subject to 

BLS non-disclosure standards.

Employment, wages, and administrative 

data (name, location, etc.) for over 9.5 

million establishments covering 

approximately 98% of all employment.

Establishment-based An economic unit that produces goods or 

services, usually at a single physical 

location, and engages in one or 

predominantly one activity. --Potential 

lay synonyms: business, worksite, brick& 

mortar, site, storefront. Lay users my use 

“establishment” interchangeably with 

“firm” In the QCEW BR, however, there is 

a significant distinction between the two 

terms.

Multi-unit enterprises may use multiple 

EINs, and they may use different ones 

when reporting to different Federal 

agencies, complicating the matching 

process. The dataset indicates whether 

an establishment is a single or multilocation establishment.

In order to ensure the highest possible 

quality of data from the QCEW program, 

BLS and the States verify and update, if 

necessary, the NAICS, location, and 

ownership classifications of all units on a 

3–year cycle. Government units in public 

administration are not reviewed 

routinely. 

EIN Yes; Legal Name and Trade Name, 

character. 

Yes; Physical location and mailing 

address; 

Yes; County and State; character 6-digit NAICS codes; Yes; latitude and longitude to six digits 

after the decimal; numeric

See also: 

www.bls.gov/opub/hom/pdf/homch5.pd

f

U.S. Department of 

Labor

Wage and Hour 

Division

Wage and Hour 

Investigative 

Support and 

Reporting Database 

(WHISARD)

Owned Continuously; Not 

Applicable.

brown.brandon@dol.gov Not Applicable. catalog.data.gov/dataset/

wage-and-hour-divisioncompliance-action-data 

ogesdw.dol.gov/views/dat

a_catalogs.php

WHISARD is an automated data 

processing system that enables 

investigators, managers, and assistants 

in the Wage and Hour Division (WHD) to 

process complaints; assign and 

investigate cases; manage and close 

cases; assist with outreach; record and 

monitor investigator time; track case 

history through narratives and diary 

entries; process FOIA and publication 

requests; and report to management. 

(www.dol.gov/oasam/ocio/programs/PI

A/WHD/WHD-WHISARD.htm)

Publicly available data is available at: 

ogesdw.dol.gov/views/data_catalogs.ph

p

Otherwise, procedures are in place that 

must be followed before allowing users 

access to the system. The process is 

designed to comply with the principles of 

least privilege and separation of duties.

(www.dol.gov/oasam/ocio/programs/PI

A/WHD/WHD-WHISARD.htm)

WHISARD contains WHD compliance 

actions. Publicly available data show all 

compliance actions from FY 2005 - 

onward.

Compliance action; WHISARD includes data on employers, 

following the definitions of "employer" 

within the statutes administered and 

enforced by WHD. The WHD is 

responsible for administering and 

enforcing some of our nation’s most 

comprehensive labor laws, including: the 

minimum wage, overtime, and child 

labor provisions of the Fair Labor 

Standards Act (FLSA); the Family and 

Medical Leave Act (FMLA); the Migrant 

and Seasonal Agricultural Worker 

Protection Act (MSPA); worker 

protections provided in several 

temporary visa programs; and the 

prevailing wage requirements of the 

Davis-Bacon Act (DBA) and the Service 

Contract Act (SCA).

WHISARD has limited functionality for 

corporate and industry structures. It is 

possible to infer linkages from other 

fields (for example, Trade Name, Legal 

Name, or EIN).

“...WHISARD has been improved as an 

information system...Quality control 

procedures have been introduced that 

improve the accuracy and reliability of 

key fields. The ability to run analyses for 

specific employers, industries, or 

geographic areas has been enhanced.”

(https://www.dol.gov/whd/resources/str

ategicenforcement.pdf)

EIN Employer Trade Name; Employer Legal 

Name; character

Employer Street Address, City Name and 

State Abbreviation, Zip Code; character

-- -- NAICS Code; character -- -- --

U.S. Department of 

Commerce

U.S. Census Bureau Longitudinal 

Employer 

Household 

Dynamics (LEHD)

Owned Updated quarterly. CES.Local.Employment.Dyna

mics@census.gov

See QCEW entry. catalog.data.gov/dataset/l

ehd-origin-destinationemployment-statisticslodes

catalog.data.gov/dataset/l

eds-onthemap-foremergency-management

www.census.gov/ces/data

products/lehddata.html

www2.census.gov/ces/wp/

2014/CES-WP-14-26.pdf

The purpose is to provide high quality 

local labor market information and to 

improve the Census Bureau's economic 

and demographic data programs. 

The data are confidential [Title 13 and 

Title 26, US Code]. Access is restricted to 

persons specially sworn to uphold the 

confidentiality provisions of Title 13 and 

Title 26.

Employment, wages, and administrative 

data (name, location, etc.) for over 9.5 

million establishments covering 

approximately 98% of all employment.

Job-level (worker-firm combinations) An economic unit that produces goods or 

services, usually at a single physical 

location, and engages in one or 

predominantly one activity. --Potential 

lay synonyms: business, worksite, brick& 

mortar, site, storefront. Lay users my use 

“establishment” interchangeably with 

“firm” In the QCEW, however, there is a 

significant distinction between the two 

terms.

The data are reported at the firm-state 

level and for reporting units 

(establishments) of the firm-state. The 

data have been linked to the Census 

Business Register to attach a firm 

identification number, firm age, and firm 

size.

The worker-level LEHD data are the 

earnings records used for the 

administration of the Unemployment 

Insurance program. The LEHD 

establishment-level data are 

downstream of the Quarterly Census of 

Employment and Wages. In addition to 

benefitting from the data quality 

monitoring and improvements made by 

these upstream programs, the LEHD 

program also maintains a data quality 

staff to ensure the quality of the 

microdata. 

Firm identifier is state employer 

identification number (SEIN), reporting 

unit ("establishment") identifier is 

SEINUNIT id. 

Legal Name and Trade Name, character. Yes; Physical location and mailing 

address of establishments

Yes; County and State; character 6-digit NAICS codes; Yes; latitude and longitude to six digits 

after the decimal; numeric

Census Tract (character), Census Block 

(character), 4-digit SIC codes (character)

See also: lehd.ces.census.gov/

Equal Employment 

Opportunity 

Commission

Surveys Division EEO-1 Desired Annually benita.marsh@eeoc.gov www.eeoc.gov/employers/

eeo1survey/index.cfm

www.eeoc.gov/employers/

eeo1survey/2007instructio

ns.cfm

www.eeoc.gov/employers/

eeo1survey/upload/eeo1-2-

2.pdf

The EEO-1 Report is a compliance survey 

mandated by federal statute and 

regulations. The survey requires 

company employment data to be 

categorized by race/ethnicity, gender 

and job category. The EEO-1 Report is 

used by the EEO-1 Joint Reporting 

Committee comprised of the Equal 

Employment Opportunity Commission 

(EEOC) and the Department of Labor's 

Office of Federal Contract Compliance 

Programs (OFCCP) to collect data from 

private employers and government 

contractors about their women and 

minority workforce. The agencies also 

use the EEO-1 Report data to support 

civil rights enforcement and to analyze 

employment patterns, such as the 

representation of women and minorities 

within companies, industries or regions.

All reports and information from 

individual reports will be kept 

confidential, as required by Section 

709(e) of Title VII of the Civil Rights Act 

of 1964. 42 U.S.C. 2000e-8(e). Only data 

aggregating information by industry or 

area, in such a way as not to reveal any 

particular employers statistics, will be 

made public.

All Private employers who are (1) subject 

to Title VII of the Civil Rights Act of 1964, 

as amended with 100 or more 

employees, or (2) private employers 

subject to Title VII who have fewer than 

100 employees if the company is owned 

or affiliated with another company, or 

there is centralized ownership, control or 

management (such as central control of 

personnel policies and labor relations) so 

that the group legally constitutes a single 

enterprise, and the entire enterprise 

employs a total of 100 or more 

employees. All federal contractors 

(private employers), who (1) are not 

exempt as provided for by 41 C.F.R. § 60-

1.5; (2) have fifty or more employees 

and (a) have a federal contract or firsttier subcontract worth $50,000 or more, 

or (b) act as depositories of federal funds 

in any amount, or (c) act as issuing and 

paying agents for U.S. savings bonds and 

notes. Single-establishment employers 

submit only one EEO-1 report, while 

employers whose business was 

conducted at more than one location 

submitted a company-wide consolidated 

report, a headquarters report, and 

individual reports for each establishment 

with fifty or more employees.

At the firm level for all covered firms. 

Also at the establishment level for 

establishments with at least 50 

employees. 

"Employer" under Section 701(b), Title 

VII of the Civil Rights Act of 1964, as 

amended, means a person engaged in an 

industry affecting commerce who has 

fifteen or more employees for each 

working day in each of twenty or more 

calendar weeks in the current or 

preceding calendar year, and any agent 

of such a person, but such term does not 

include the United States, a corporation 

wholly owned by the government of the 

United States, an Indian tribe, or any 

department or agency of the District of 

Columbia subject by statute to 

procedures of the competitive service (as 

defined in section 2102 of Title 5 of the 

United States Code), or a bona fide 

private membership club (other than a 

labor organization) which is exempt from 

taxation under Section 501(c) of the 

Internal Revenue Code of 1954;

Multi-establishment employers, i.e., 

employers doing business at more than 

one establishment, must complete 

online: (1) a report covering the principal 

or headquarters office; (2) a separate 

report for each establishment employing 

50 or more persons; and (3) a separate 

report (Type 8 record) for each 

establishment employing fewer than 50 

employees, OR an Establishment List 

(Type 6 record), showing the name, 

address, and total employment for each 

establishment employing fewer than 50 

persons, including a Type 6 employment 

data grid that combines all employees 

working at establishments employing 

fewer than 50 employees by race, sex, 

and job category. For the EEO-1 online 

application, keyed employment data 

automatically transfers to the overall 

Consolidated Report.

The total number of employees indicated 

on the headquarters report, PLUS the 

establishment reports, PLUS the list of 

establishments employing fewer than 50 

employees, MUST equal the total 

number of employees shown on the 

Consolidated Report.

 

EIN. Government contractors are also 

asked to provide their DUNS number.

Parent company name, establishment 

name. 

Yes, number and street address, city, 

state, and zip code for parent company 

and establishment. Also county for 

establishment.

No. Verbal description of major activity of 

the establishment.

Employment definition includes leased 

employees, unlike the Census Business 

Register.

OR any person or entity subject to 

Executive Order 11246 who is a federal 

government prime contractor or 

subcontractor at any tier (including a 

bank or other establishment serving as a 

depository of federal government funds, 

or an issuing and paying agent of U.S. 

Savings Bonds and Notes, or a holder of 

a federal government bill of lading) or a 

federally-assisted construction prime 

contractor or subcontractor at any tier.

Securities and 

Exchange 

Commission

Integrated Award 

Environment

SEC Company Filings Desired Continuously, daily 

updates

www.sec.gov/info/edgar.s

html

www.sec.gov/dera/data/fi

nancial-statement-datasets.html

www.sec.gov/edgar/search

edgar/companysearch.htm

l

EDGAR, the Electronic Data Gathering, 

Analysis, and Retrieval system, performs 

automated collection, validation, 

indexing, acceptance, and forwarding of 

submissions by companies and others 

who are required by law to file forms 

with the U.S. Securities and Exchange 

Commission (SEC). Its primary purpose is 

to increase the efficiency and fairness of 

the securities market for the benefit of 

investors, corporations, and the 

economy by accelerating the receipt, 

acceptance, dissemination, and analysis 

of time-sensitive corporate information 

filed with the agency.

Publicly available through EDGAR. All public companies must file. Private 

companies also make filings when 

issuing unregistered securities. 

EIN, CIK Business name Business and mailing addresses. Includes 

number and street, city, state, 9-digit zip 

code.

4-digit SIC code

Department of 

Health and Human 

Services

Administration for 

Children and 

Families, Office of 

Child Support 

Enforcement

National Directory 

of New Hires 

(NDNH)

Desired Quarterly linda.boyer@acf.hhs.gov www.acf.hhs.gov/css/reso

urce/a-guide-to-thenational-directory-of-newhires

The NDNH is a national repository of 

employment, earnings, and 

unemployment insurance (UI) 

information to help state child support 

agencies locate noncustodial parents, 

establish child support orders, and 

enforce child support orders.

Researchers can only receive access to 

data without personal identifiers, and 

only for research found by the Secretary 

of Health and Human Services to be 

likely to contribute to achieving the 

purposes of Part A or Part D of the Social 

Security Act. There has been discussion 

about new legislation to include 

statistical purposes in the list of 

authorized uses.

Similar to QCEW? Establishment EIN Employer name Business address, including number, 

street name, city, state, and zip code (5-

digit, and second field for 4-digit 

extension if provided)

Foreign country code

General Services 

Administration

Integrated Award 

Environment

System of Award 

Management (SAM) 

Desired Continuously, in real 

time

www.sam.gov/portal/SAM

/#1

www.sam.gov select 

"help" and "user guide"

Registration is required to apply for a 

Federal govt. contract or grant.

Searchable database and recent filings 

publicly available on sam.gov.

All firms doing business with the Federal 

government must file.

Establishment in accordance with the Federal 

Acquisition Regulation

Additional information available for 

authorized users

Data are matched to authoritative 

sources.

DUNS number, CAGE code Business name Business and mailing street address, city, 

state, 5-digit zip code, and 4-digit zip 

code extension

Country abbreviation. NAICS code company website
The Promise of Evidence-Based Policymaking 893

Employer Data Matching Workgroup

Data Inventory

Page 2 of 5

Department/

Agency

Component Agency 

or Office

Dataset Name Dataset Type: 

Owned by Data 

Inventory 

Respondent, or 

Desired by Data 

Inventory 

Respondent?

Data Update 

Frequency and Lags

Contact E-mail Address Link to Data Collection 

Tool

Data.gov Link Other Relevant Link(s) Purpose of the Data Collection Access Restrictions Coverage Unit of Analysis Definition of "Company" Corporate Structure and Relationships Quality of Fields for Matching Unique Identifier Fields Name Fields Address Fields Country fields FIPS Code Fields Industry Code Fields Geocoding Other Fields Additional Notes

U.S. Department of 

Commerce

U.S. Census Bureau Governments 

Intergrated 

Directory(GID). A 

migration from the 

GID to the 

Governments 

Master Address File 

(GMAF) is planned 

for October 2017.

Owned The GID universe is 

updated 

continuously, 

although a formal 

and comprehensive 

update occurs once 

every five years 

during the Census 

of Governments. 

There are also 

annual surveys, 

including the 

Boundary and 

Annexation Survey, 

the Annual Finance 

Survey and the 

Annual Survey of 

Personnel and 

Payroll, which 

provide periodic 

updates. From the 

annual survey's 

collection period, 

the data are 

reviewed and edited 

as necessary, and 

the GID unverse is 

updated 1.5-2 years 

following initial 

collection cycle.

esmd.gus.psfcb@census.gov www.census.gov/govs/cog

/

N/A www.meps.ahrq.gov for 

published statistics from 

this dataset

In order to measure government activity, 

various surveys collect statistics for state 

and local governments and their 

dependent

agencies. A complete list of local

governments is needed to sample for 

these surveys. This list of governments is 

found in the Governments Integrated 

Directory (GID).

Public Sector Statistics are considered 

public information, which is available 

online through AFF. We also provide 

additional statistics, not available in our 

publications, upon data user request in 

the form of a Special Tabulation file.

The GID includes all governments in the 

United States, along with their 

dependent agencies. This includes all 

states and dependent agencies, counties 

and dependent agencies, cities, towns 

and townships, municipalities, special 

districts, authorities, boards and 

commissions.

Government Unit A governmental units is defined as an 

organized entity which, in addition to 

having governmental character, has 

sufficient discretion in the management 

of its own affairs to distinguish it as 

separate from the administrative 

structure of any other governmental 

unit. An entity must possess all three of 

the critical attributes to be regarded as a 

government: Existence as an Organized 

Entity, Governmental Character, and 

Substantial Autonomy.

Both parent and dependent agencies as 

well as independent agencies.

Each unit has a 14 digit GID ID: SS T CCC 

UUU SUP SB

where:

SS = State. Numeric sequence assigned 

to the states alphabetically – Alabama is 

01, Alaska 02, and so forth, until 

Wyoming at 51. Note the District of 

Columbia is assigned code 09 and the 

U.S. Federal Government is assigned 

code 00.

T = type of government. As defined by 

the Census Bureau. Government type 

codes are

0 = state governments

1 = county governments

2 = municipal governments

3 = township governments

4 = special district governments

5 = school district governments

6 = Federal Government

CCC = county area within the state. The 

county area codes, are assigned 

alphabetically within each state area, 

although there are some exceptions 

where county consolidations have taken 

place or new counties have been 

created.

Yes - complete governmental unit name Yes physical location and mailing address N/A N/A N/A No Although this information is specific to 

the current GID, most of the information 

also applies to the GMAF with the 

exception of the ID format.

UUU = unit code within the county area. 

The unit code identifies a specific 

government within the state, county, 

and type of government sequence. They, 

too, are generally alphabetical within the 

county area and type of government 

combination, but exceptions exist 

(especially for special districts). The unit 

code for a county government is usually 

the same as its county area code.

SUP = supplemental unit code, for 

dependent agencies of the parent unit. 

The supplemental code identifies 

different dependent agencies of the 

same parent state or local government. 

Every supplement must have a parent 

government. The first nine digits of a 

supplement and parent code always are 

the same.

SB = subsidiary code for dependent 

agencies that require separate data 

collection / canvassing. The subsidiary 

code is used for component units of 

dependent agencies that require a 

separate data collection. A good 

example is found in state dependent 

university systems, where separate 

campuses exist for a dependent 

university, each campus having its own 

subsidiary code.

U.S. Census Bureau Sponsored by the 

Agency for 

Healthcare 

Research and 

Quality

Medical 

Expenditure Panel 

Survey- Insurance 

Component - Public 

Sector Employer 

Sample

Owned Fielded annually. Karen.Davis@ahrq.hhs.gov www.meps.ahrq.gov has 

information on survey 

questionnaires and 

methodology reports.

N/A www.meps.ahrq.gov for 

published statistics from 

this dataset

To collect information about employersponsored health insurance. The MEPSIC collects information on whether an 

employer offers coverage, details of 

plans and information on establishment 

and workforce characteristics. For the 

government sample, information is 

collected on all plans that are offered to 

active employees.

Public Sector Statistics are considered 

public information, which is available 

online through AFF(American Fact 

Finder: https://factfinder.census.gov/).

We also provide additional statistics, not 

available in our publications, upon data 

user request in the form of a Special 

Tabulation file.

A sample drawn from the Governments 

Integrated Directory (GID) of 3,000 

government units

Government Unit A governmental units is defined as an 

organized entity which, in addition to 

having governmental character, has 

sufficient discretion in the management 

of its own affairs to distinguish it as 

separate from the administrative 

structure of any other governmental 

unit. An entity must possess all three of 

the critical attributes to be regarded as a 

government: Existence as an Organized 

Entity, Governmental Character, and 

Substantial Autonomy.

Both parent and dependent agencies as 

well as independent agencies.

Each unit has a 14 digit GID ID: SS T CCC 

UUU SUP SB

where:

SS = State. Numeric sequence assigned 

to the states alphabetically – Alabama is 

01, Alaska 02, and so forth, until 

Wyoming at 51. Note the District of 

Columbia is assigned code 09 and the 

U.S. Federal Government is assigned 

code 00.

T = type of government. As defined by 

the Census Bureau. Government type 

codes are

0 = state governments

1 = county governments

2 = municipal governments

3 = township governments

4 = special district governments

5 = school district governments

6 = Federal Government

CCC = county area within the state. The 

county area codes, are assigned 

alphabetically within each state area, 

although there are some exceptions 

where county consolidations have taken 

 

Yes - complete governmental unit name Yes physical location and mailing address N/A N/A Yes; latitude and longitude to six digits 

after the decimal; numeric

Although this information is specific to 

the current GID, most of the information 

also applies to the GMAF with the 

exception of the ID format.

UUU = unit code within the county area. 

The unit code identifies a specific 

government within the state, county, 

and type of government sequence. They, 

too, are generally alphabetical within the 

county area and type of government 

combination, but exceptions exist 

(especially for special districts). The unit 

code for a county government is usually 

the same as its county area code.

SUP = supplemental unit code, for 

dependent agencies of the parent unit. 

The supplemental code identifies 

different dependent agencies of the 

same parent state or local government. 

Every supplement must have a parent 

government. The first nine digits of a 

supplement and parent code always are 

the same.

SB = subsidiary code for dependent 

agencies that require separate data 

collection / canvassing. The subsidiary 

code is used for component units of 

dependent agencies that require a 

separate data collection. A good 

example is found in state dependent 

university systems, where separate 

campuses exist for a dependent 

university, each campus having its own 

subsidiary code.

U.S. Census Bureau Sponsored by the 

Agency for 

Healthcare 

Research and 

Quality

Medical 

Expenditure Panel 

Survey- Insurance 

Component - 

Private Sector 

Employer Sample

Owned Fielded annually. Karen.Davis@ahrq.hhs.gov www.meps.ahrq.gov has 

information on survey 

questionnaires and 

methodology reports.

N/A www.meps.ahrq.gov for 

published statistics from 

this dataset

To collect information about employersponsored health insurance. The MEPSIC collects information on whether an 

employer offers coverage, details of 

plans and information on establishment 

and workforce characteristics. For the 

private-sector sample, information is 

collected on up to 4 plans that are 

offered to active employees.

The Business Register information is 

confidential [Title 13 and Title 26, US 

Code]. Access is restricted to persons 

specially sworn to uphold the 

confidentiality provisions of Title 13 and 

Title 26. Micro-data can be used at a 

Census Research Data Center.

A sample drawn from the Business 

Register of approximately 40,000 

establishments each year.

Private-sector establishments An establishment is a single physical 

location where business transactions 

take place and for which payroll and 

employment records are kept. Groups of 

one or more establishments under 

common ownership or control are 

enterprises. A single-unit enterprise 

owns or operates only one 

establishment. A multi-unit enterprise 

owns or operates two or more 

establishments. The treatment of 

establishments on the Business Register 

differs according to whether the 

establishment is part of a single-unit or 

multi-unit enterprise. 

(https://www.census.gov/econ/susb/me

thodology.html)

Multi-unit enterprises may use multiple 

EINs, and they may use different ones 

when reporting to different federal 

agencies, complicating the matching 

process. The Census Bureau has linked 

many, but not all of the EINs to 

enterprises.

The Business Register provides the most 

complete, current, and consistent source 

of establishment- based information 

about U.S. businesses, and is essential to 

assuring full coverage and high quality in 

Federal economic statistics programs. 

Examples of quality considerations for 

this source include:

- The annual Company Organization 

Survey covers 30 percent of multi-unit 

companies and a small sample of firms 

that were single-unit firms in the most 

recent quinquennial Economic Census, 

so establishment openings and closings 

in the firms not covered may not be 

reflected in the business register until 

after the next Economic Census. 

- The business register is divided into 

employer and nonemployer business 

registers based on payroll employment. 

Some firms lease their employees from 

Professional Employer Organizations 

(PEOs) or use independent contractors. 

Such firms may appear in the 

nonemployer business register despite 

having large revenues and many leased 

and/or contract employees. 

www.census.gov/econ/overview/mu060

0.html

A single-unit enterprise’s primary 

identifier is its Employer Identification 

Number (EIN). 

A unique employer unit identification 

number identifies each establishment 

owned by a multi-unit enterprise on the 

Business Register.

(https://www.census.gov/econ/susb/me

thodology.html)

Yes; Primary and secondary business 

names, permitting identification of both 

legal and trade designations for 

businesses that support them (e.g., 

franchises); character

Yes; Physical location, mailing address, 

post office (city or town) name, postal 

state abbreviation. ZIP+4; character

This field is not currently populated in 

the BR, but it may be possible to link in 

this information from the Company 

Organization Survey

Yes; County and State; character 6 digit NAICS codes Yes; latitude and longitude to six digits 

after the decimal; numeric

Census Tract (character), Census Block 

(character), 4-digit SIC codes (character)

U.S. Department of 

Education

National Center for 

Education Statistics

Common Core of 

Data (CCD)

Owned Annual Collection. 

LAG: Data on 2014-

15 school year are 

set to be released in 

September

mark.glander@ed.gov eden.ed.gov/EDENPortal/ catalog.data.gov/dataset/e

dfacts-general-information201112

nces.ed.gov/ccd CCD is a program of the U.S. Department 

of Education's National Center for 

Education Statistics that annually collects 

fiscal and non-fiscal data about all public 

schools, public school districts and state 

education agencies in the United States. 

The data are supplied by state education 

agency officials and include information 

that describes schools and school 

districts, including name, address, and 

phone number; descriptive information 

about students and staff, including 

demographics; and fiscal data, including 

revenues and current expenditures.

Publicly available data is available 

through nces.ed.gov/ccd/ccddata.asp 

Otherwise, restricted use datasets can 

be requested by licensees of the Institute 

for Education Science's Restricted Use 

Data License

Public, K-12 local education agencies. 

Universe collection performed from all 

state education agencies each year.

State, Local Education Agency and School N/A: Collection is focused upon state and 

local education agencies

All schools are reported as members of 

local education agencies/school districts. 

NCES assigns each school district a 

unique identifer (LEAID) and each school 

it's own unique identifier (SCHNO). 

These IDs are combined into a 12-digit 

identifier (NCESSCH) which is unique for 

each school

LEAID, NCESSCH, FIPST, SCHNO, STID, 

SEASCH

LEANM, SCHNAM MSTREE, MCITY, MSTATE, MZIP, MZIP4, 

LSTREE, LCITY, LSTATE, LZIP, LZIP4

N/A FIPST N/A For individual schools only: LATCOD, 

LONCOD

CONUM (County Number), CONAME 

(County Name), CDCODE (Congressional 

Distrcit code)
894 The Promise of Evidence-Based Policymaking

Employer Data Matching Workgroup

Data Inventory

Page 3 of 5

Department/

Agency

Component Agency 

or Office

Dataset Name Dataset Type: 

Owned by Data 

Inventory 

Respondent, or 

Desired by Data 

Inventory 

Respondent?

Data Update 

Frequency and Lags

Contact E-mail Address Link to Data Collection 

Tool

Data.gov Link Other Relevant Link(s) Purpose of the Data Collection Access Restrictions Coverage Unit of Analysis Definition of "Company" Corporate Structure and Relationships Quality of Fields for Matching Unique Identifier Fields Name Fields Address Fields Country fields FIPS Code Fields Industry Code Fields Geocoding Other Fields Additional Notes

U.S. Department of 

Education

National Center for 

Education Statistics

Integrated 

Postsecondary 

Education Data 

System (IPEDS)

Owned Annual Collection. 

LAG: Institutional 

Characteristics data 

on 2015-16 was 

released in July 

2016

ipedstools@rti.org nces.ed.gov/ipeds catalog.data.gov/dataset/2

01314-integratedpostsecondary-educationdata-system

IPEDS provides basic data needed to 

describe — and analyze trends in — 

postsecondary education in the United 

States, in terms of the numbers of 

students enrolled, staff employed, 

dollars expended, and degrees earned. 

Congress, federal agencies, state 

governments, education providers, 

professional associations, private 

businesses, media, students and 

parents, and others rely on IPEDS data 

for this basic information on 

postsecondary institutions.

IPEDS forms the institutional sampling 

frame for other NCES postsecondary 

surveys, such as the National 

Postsecondary Student Aid Study and the 

National Survey of Postsecondary 

Faculty

Public data is available through: 

nces.ed.gov/ipeds/Home/UseTheData

Postsecondary Institutions. Universe 

collection performed every year

Postsecondary Institutions N/A: Collection is focused upon 

postsecondary istitutions

Several insitutions are recognized as 

"multi-instituion" or "multi-campus" 

organizations, otherwise each institution 

is reported as an independent entity

UNITID (Unique identificaiton number of 

the institution); EIN (Employer 

Identification Number); OPEID (Office of 

Postsecondary Education (OPE) ID 

Number); NEWID (UNITID for merged 

schools); F1SYSCOD (ID number of multiinstition or multi-campus organization)

INSTNM (Institution [entity] name); 

FYSYSNAM (Name of multi-institution or 

multi-campus organization)

ADDR, CITY, STABBR, ZIP N/A FIPS N/A LONGITUD; LATITUDE

U.S. Department of 

Education

National Center for 

Education Statistics

Private School 

Universe Survey 

(PSS)

Owned Biennial (most 

recent publicly 

available data: 2011-

12)

stephen.broughman@ed.gov N/A catalog.data.gov/dataset/2

01516-private-schooluniverse-survey

The purposes of this data collection 

activity are; a) to generate biennial data 

on the total number of private schools, 

teachers, and students; and b) to build 

an accurate and complete list of private 

schools to serve as a sampling frame for 

NCES surveys of private schools. The PSS 

began with the 1989-90 school year and 

has been conducted every two years 

since.

Public data available at: 

nces.ed.gov/surveys/pss/pssdata.asp

All private schools Individual private schools N/A N/A PPIN (Permanent Indentification 

Number)

PINST (School Name) PADDRS; PCITY; PSTABB; PZIP; PZIP4; 

PCNTY; PCNTNM; PL_ADD; PL_CIT; 

PL_STABB; PL_ZIP; PL_ZIP4

N/A PSTANSI (ANSI State Code) N/A LATITUDE##; LONGITUDE## (where ## = 

year of collection)

REGION (Census region)

U.S. Department of 

Education

Office for Civil 

Rights / National 

Center for 

Education Statistics

Civil Rights Data 

Collection

Owned Biennial Collection. 

LAG: Data on 2013-

14 school year were 

released in Spring 

2016

stephanie.miller@ed.gov 

(NCES); 

mary.schifferli@ed.gov (OCR)

Not currently online 

(informational link: 

www2.ed.gov/about/office

s/list/ocr/data.html)

catalog.data.gov/dataset/c

ivil-rights-data-collection2013-14 

ocrdata.ed.gov/ Since 1968, the U.S. Department of 

Education (ED) has conducted the Civil 

Rights Data Collection (CRDC) to collect 

data on key education and civil rights 

issues in our nation's public schools. The 

collection was formerly administered as 

the Elementary and Secondary School 

Survey (E&S Survey). 

The CRDC collects a variety of 

information including, student 

enrollment and educational programs 

and services, most of which is 

disaggregated by race/ethnicity, sex, 

limited English proficiency and disability. 

The CRDC is a longstanding and 

important aspect of the ED Office for 

Civil Rights (OCR) overall strategy for 

administering and enforcing the civil 

rights statutes for which it is responsible.

Publicly available data is available 

through ocrdata.ed.gov

Otherwise, restricted use datasets can 

be requested by licensees of the Institute 

for Education Science's Restricted Use 

Data License

Public, K-12 local education agencies. 

Universe collection from all agencies 

every two years

Primary unit of anlysis: individual public 

schools Secondary unit of analysis: 

Individual school districts/local education 

agencies

N/A: Collection is focused upon school 

districts

There are three main identifiers (IDs) 

used in the data files: one to identify the 

LEA, one to identify the school (in schoollevel files), and a combined LEA/school 

ID:

LEAID is the 7-digit district identification 

code

SCHID is the 5-digit school identification 

code

COMBOKEY is the 7-digit LEAID + the 5-

digit SCHID

Districts or schools that had an existing 

LEAID or SCHID kept that ID. In most 

cases, the LEAID and SCHID match the 

National Center of Education Data (NCES) 

ID used in EDFacts, although there are 

some cases that differ because of 

differences in definitions and procedures 

between EDFacts and the CRDC. Districts 

that did not have an existing ID were 

issued an NCES ID if there was one, or a 

new 7-digit LEAID, which consisted of the 

2-digit state FIPS code followed by a 5-

digit ID assigned during collection. 

Schools that did not have an existing ID 

were issued an NCES ID if there was one, 

or a new ID which consisted of the LEAID 

and a 5-digit school code assigned during 

collection. 

LEAID, SCHID, COMBOKEY LEA_NAME LEA_ADDRESS; LEA_CITY; LEA_ZIP N/A NOTE: The first two digits of LEAID are 

the FIPS Code for that district, however 

the FIPS code is not separately reported 

within its own field

N/A N/A

U.S. Department of 

Commerce - Bureau 

of Economic 

Analysis

Direct Investment 

Division (DID)

BEA surveys on U.S. 

direct investment 

abroad

Owned quarterly, annual, 

and every five years 

depending on the 

data item and on 

the size of the 

reporting unit

ricardo.limes@bea.gov

www.bea.gov/surveys/dias

urv.htm

Not Applicable Statistics:

http://www.bea.gov/intern

ational/direct_investment_

multinational_companies_

comprehensive_data.htm

BEA produces comprehensive statistics 

on U.S. direct investment abroad that 

are essential to the compilation of the 

U.S. economic accounts and for the 

analysis of multinational enterprises 

(MNEs).

Aggregates are public; microdata are nonpublicStatistics on U.S. multinational 

enterprises (MNEs). U.S. direct 

investment abroad (USDIA) is defined as 

ownership by a U.S. investor of at least 

10 percent of a foreign business. The 

direct investor is known as a U.S. parent, 

and the U.S.-owned foreign business is 

known as a foreign affiliate. The 

combined global operations of a U.S. 

parent company and its foreign affiliates 

constitute a U.S. MNE. BEA produces 

two broad sets of statistics on U.S. direct 

investment abroad: (1) Direct 

investment transactions and positions 

statistics (which are used in the U.S. 

International Transaction and 

International Investment Position 

Accounts), and (2) statistics on the 

activities of U.S. MNEs.

Enterprise level; data are available on 

U.S. parents and each of their foreign 

affiliates. For U.S. parents, data are 

avilable for the fully consolidated U.S. 

domestic business enterprise. 

The U.S. reporter is the U.S. person that 

has direct investment in a foreign 

business enterprise, including a branch. 

If the U.S. person is a business 

enterprise, the U.S. parent is the fully 

consolidated U.S. domestic business 

enterprise (see corporate structure and 

relationships for consolidation rules). 

The foreign affiliate is a business 

enterprise located in one country which 

is directly or indirectly owned or 

controlled by a U.S. person to the extent 

of 10 percent or more. A business 

enterprise is any organization, 

association, branch, or venture which 

exists for profit making purposes or to 

otherwise secure economic advantage, 

and any ownership of any real estate.

The fully consolidated U.S. domestic 

business enterprise is defined as: 1) the 

U.S. business enterprise whose voting 

securities are not owned more than 50 

percent by another U.S. business 

enterprise, and 2) proceeding down each 

ownership chain from that U.S. business 

enterprise, any U.S. business enterprise 

(including Foreign Sales Corporations 

located in the United States) whose 

voting securities are more than 50 

percent owned by the U.S. business 

enterprise above it.

The industry codes reported reflect BEA's 

International Surveys Industry 

classification system, which is based on 4-

digit NAICS codes 

(www.bea.gov/surveys/iftcmat.htm).

The data are collected and reviewed 

according to documented procedures 

and best practice standards and ongoing review by the appropriate 

supervisor. The data are validated using 

a battery of edit checks to detect 

potential errors and to otherwise ensure 

that the data are accurate, reliable, and 

relevant for the estimates being made. 

Data are routinely revised as more 

complete source data become available. 

 EINs are collected for U.S. parents. Name of U.S. parent, name of contact 

person for U.S. parent, name of 

certifying official, and name of each 

foreign affiliate. (all character)

Address of U.S. parent and address of 

contact person for U.S. parent. 

(Character)

Each foreign affiliate record includes a 

field indicating which country it is located 

in. All U.S. parents are located in the 

United States. 

No International Surveys

Industry (ISI) classification system, which 

is based on 4-digit NAICS codes 

(http://www.bea.gov/surveys/iftcmat.ht

m). Each reporting unit is classified into a 

single industry for publication, but BEA 

collects data for up to 10 industries in 

which they have revenue. This more 

detailed industry breakdown 

approximates the industry of the 

establishments that comprise the 

enterprise. 

No Telephone and fax numbers for contact 

person and certifying official. Email 

address of contact person.

U.S. Department of 

Commerce - Bureau 

of Economic 

Analysis

Direct Investment 

Division (DID)

BEA surveys on 

foreign direct 

investment in the 

United States 

Owned quarterly, annual, 

and every five years 

depending on the 

data item and on 

the size of the 

reporting unit

ricardo.limes@bea.gov www.bea.gov/surveys/fdiu

surv.htm

Not Applicable Statistics:

www.bea.gov/internationa

l/direct_investment_multin

ational_companies_compr

ehensive_data.htm

BEA produces comprehensive statistics 

on foreign direct investment in the 

United States that are essential to the 

compilation of the U.S. economic 

accounts and for the analysis of 

multinational enterprises (MNEs).

Aggregates are public; microdata are nonpublicStatistics on U.S. affiliates of foreign 

multinational enterprises (MNEs). 

Foreign direct investment in the United 

States is defined as the ownership by a 

foreign investor of 10 percent or more of 

a U.S. business. The direct investor is 

known as a foreign parent, and the 

foreign-owned U.S. business is known as 

a U.S. affiliate. BEA produces three broad 

sets of statistics on foreign direct 

investment in the United States: (1) 

Direct investment transactions and 

positions statistics (which are used in 

the U.S. International Transaction and 

International Investment Position 

Accounts), (2) statistics on the activities 

of U.S. affiliates of foreign MNEs, and (3) 

new foreign direct investment statistics.

Enterprise level; data are available for 

the consolidated U.S. affiliate.

A U.S. affiliate is a business enterprise 

located in the United States which is 

directly or indirectly owned or controlled 

by a person of another country to the 

extent of 10 percent or more. A 

business enterprise means any 

organization, association, branch, or 

venture which exists for profit making 

purposes or to otherwise secure 

economic advantage, and any ownership 

of any real estate.

The consolidated U.S. affiliate consists of 

the U.S. business enterprise which is 

directly owned by a foreign person and 

all U.S. business enterprises proceeding 

down each ownership chain whose 

voting securities are more than 50 

percent owned by the U.S. business 

enterprise above. The fully consolidated 

entity is considered one U.S. affiliate. 

Data on the industry of sales for U.S. 

affiliates are classified under the 

International Surveys Industry 

classification system, which is based on 4-

digit NAICS codes 

(http://www.bea.gov/surveys/iftcmat.ht

m). 

The data are collected and reviewed 

according to documented procedures 

and best practice standards and ongoing review by the appropriate 

supervisor. The data are validated using 

a battery of edit checks to detect 

potential errors and to otherwise ensure 

that the data are accurate, reliable, and 

relevant for the estimates being made. 

Data are routinely revised as more 

complete source data become available. 

 EINs are collected for the U.S. business 

enterprise which is foreign-owned and 

for each U.S. business enterprise 

consolidated with it.

Name of U.S. affiliate, name of contact 

person for U.S. affiliate, name of 

certifying official, and name of each U.S. 

business enterprise consolidated. (all 

character)

 Address of U.S. affiliate and address of 

contact person for U.S.affiliate. 

(Character)

Each U.S. affiliate is located in the United 

States. Each U.S. affiliate record includes 

a field indicating the country of the 

foreign parent and a field with the 

country of the ultimate beneficial owner 

(which may be a foreign or U.S. person or 

entity).

No International Surveys

Industry (ISI) classification system, which 

is based on 4-digit NAICS codes 

(http://www.bea.gov/surveys/iftcmat.ht

m). The consolidated affiliate is classified 

into a single industry for publication, but 

BEA collects data for up to 10 industries 

in which they have revenue. This more 

detailed industry breakdown 

approximates the industry of the 

establishments that comprise the 

enterprise. 

No Telephone and fax numbers for contact 

person and certifying official. Email 

address of contact person. Number of 

employees associated with each 

industry. Number of employees, number 

of manufacturing employees, gross 

propety, plant, and equipment (PP&E), 

and commercial property by state.

U.S. Department of 

Commerce - Bureau 

of Economic 

Analysis

Balance of 

Payments Division

BEA surveys on U.S. 

international trade 

in services

Owned Quarterly and 

annually depending 

on the level of 

component detail; 

more granular data 

by country and by 

type of service are 

published annually 

than quarterly

christopher.stein@bea.gov www.bea.gov/surveys/iuss

urv.htm

Not Applicable Statistics:

www.bea.gov/iTable/iTabl

eHtml.cfm?reqid=62&step

=2&isuri=1&6210=1

BEA combines data from the trade in 

services surveys with data from other 

sources to compile the official statistics 

on trade in services, a component of the 

U.S. international transactons accounts.

Other government agencies use these 

statistics to formulate U.S. trade policy, 

to analyze the impact of that policy and 

the policies of foreign countries on 

international trade in services, and to 

support trade promotion activities. The 

trade statistics are also used by 

companies involved in international 

trade, researchers, international 

organizations, and the general public.

Aggregates are public; microdata are nonpublicBEA's international trade in services 

surveys collect data on U.S. exports and 

imports of transportation services, 

financial services, insurance services, 

other business services, and charges for 

the use of intellectual property.

Enterprise level; data are available for 

the consolidated U.S. enterprise.

Survey respondents are required to 

identify the primary Employer 

Identification Number (EIN) used to file 

U.S. income or payroll taxes and the 

industry code that best describes the 

major activity of the U.S. reporter.

Respondents include U.S. persons 

engaged in international trade in services 

where person means any individual, 

branch, partnership, associated group, 

association, estate, trust, corporation, or 

other organization (whether or not 

organized under the laws of any State), 

and any government (including a foreign 

government, the United States 

Government, a State or local 

government, and any agency, 

corporation, financial institution, or 

other entity or instrumentality thereof, 

including a government sponsored 

agency). If the U.S. person is a business 

enterprise, the U.S. parent is the fully 

consolidated U.S. domestic business 

enterprise (see corporate structure and 

relationships for consolidation rules). A 

business enterprise is any organization, 

association, branch, or venture which 

exists for profit making purposes or to 

otherwise secure economic advantage, 

and any ownership of any real estate.

The fully consolidated U.S. domestic 

business enterprise is defined as: 1) the 

U.S. business enterprise whose voting 

securities are not owned more than 50 

percent by another U.S. business 

enterprise, and 2) proceeding down each 

ownership chain from that U.S. business 

enterprise, any U.S. business enterprise 

(including Foreign Sales Corporations 

located in the United States) whose 

voting securities are more than 50 

percent owned by the U.S. business 

enterprise above it.

The industry codes reported reflect BEA's 

International Surveys Industry 

classification system, which is based on 4-

digit NAICS codes 

(http://www.bea.gov/surveys/iftcmat.ht

m).

The data are collected and reviewed 

according to documented procedures 

and best practice standards and ongoing review by the appropriate 

supervisor. The data are validated using 

a battery of edit checks to detect 

potential errors and to otherwise ensure 

that the data are accurate, reliable, and 

relevant for the estimates being made. 

Data are routinely revised as more 

complete source data become available. 

EINs of survey respondents are collected. Name of U.S. "person" responding to the 

survey, name of contact person, and 

name of certifying official; all character 

fields.

Address of the U.S. "person" responding 

to the survey and address of contact 

person; both character fields.

All survey respondents are U.S. 

"persons."

No BEA's International Surveys

Industry (ISI) classification system, which 

is based on 4-digit NAICS codes 

(http://www.bea.gov/surveys/iftcmat.ht

m).

No Telephone and fax numbers for contact 

person and certifying official. Email 

address of contact person.

U.S. Department of 

Labor

Occupational Safety 

& Health 

Administration

OSHA Information 

System (OIS)

Owned Continuously; Not 

Applicable.

Moore, Joshua 

(Moore.Joshua@dol.gov)

Not Applicable. catalog.data.gov/dataset?

publisher=U.S. Department 

of Labor Occupational 

Safety and Health 

Administration

enforcedata.dol.gov/views

/data_catalogs.php

To track, document and analyze OSHA's 

Enforcement and Consultation 

programs.

Only users in the OSHA family have 

access to the system. Some data is 

available via osha.gov and the data.gov

OIS contains all Enforcement and 

Consultation information starting in FY 

11 in a staggered role out. All Federal 

enforcement was on the system starting 

in FY13.

Compliance actions, Outreach and 

Consultation services

"Employer" means a person engaged in a 

business affecting commerce who has 

employees, but does not include the 

United States or any State or political 

subdivision of a State. OSHA does 

inspect Federal agencies but does not 

issue citations with penalties. OSHA 

collects data on the establishment 

(business location) level; employers are 

the "parents" of establishments.

OIS does not have this functionality at 

this time.

All data is user entered. We have edit 

checks on the number of characters on 

fields such as the FEIN but we have no 

way to verify if the numbers entered are 

accurate.

Federal EIN/TIN, DUNS, DUNS plus4 and 

CAGE Code. All of these are optional in 

the system.

Establishment Name and Doing Business 

As Name

Full address of Mailing Business and Site 

Address

Business and Mailing address contain 

Country, International City, International 

Division and International Postal Code

Yes; County and State; character 6 digit NAICS No

U.S. Department of 

Labor

Occupational Safety 

& Health 

Administration

OSHA Legacy Data 

(OLD) replaces the 

Legacy IMIS system 

(Aug 22, 2016)

Owned Continuously; Not 

Applicable.

Legasse, Gizaw 

(Legesse.Gizaw@DOL.GOV)

Not Applicable. catalog.data.gov/dataset?

publisher=U.S. Department 

of Labor Occupational 

Safety and Health 

Administration

enforcedata.dol.gov/views

/data_catalogs.php 

To track, document and analyze OSHA's 

Enforcement and Consultation 

programs. NO new Case data is being 

collected but maintained for historical 

records. [New case data is being 

collected in OIS)

Only users in the OSHA family have 

access to the system. Some data is 

available via osha.gov and the data.gov

The Integrated Management 

Information System (IMIS) contained all 

Enforcement and Consultation 

information starting in FY 1970, The IMIS 

data was migrated over a new database 

called OSHA Legacy Data (OLD)

Compliance actions, Outreach and 

Consultation services

"Employer" means a person engaged in a 

business affecting commerce who has 

employees, but does not include the 

United States or any State or political 

subdivision of a State. OSHA does 

inspect Federal agencies but does not 

issue citations with penalties. OSHA 

collects data on the establishment 

(business location) level; employers are 

the "parents" of establishments.

OLD does not have this functionality at 

this time.

All data is user entered. We have edit 

checks on the number of characters on 

fields such as the FEIN but we have no 

way to verify if the numbers entered are 

accurate.

Federal EIN/TIN, DUNS, All of these are 

optional in the system.

Establishment Name and Doing Business 

As Name

Full address of Mailing Business and Site 

Address

Business and Mailing address contain 

Country, International City, International 

Division and International Postal Code

Yes; County and State; character 6 digit NAICS No
The Promise of Evidence-Based Policymaking 895

Employer Data Matching Workgroup

Data Inventory

Page 4 of 5

Department/

Agency

Component Agency 

or Office

Dataset Name Dataset Type: 

Owned by Data 

Inventory 

Respondent, or 

Desired by Data 

Inventory 

Respondent?

Data Update 

Frequency and Lags

Contact E-mail Address Link to Data Collection 

Tool

Data.gov Link Other Relevant Link(s) Purpose of the Data Collection Access Restrictions Coverage Unit of Analysis Definition of "Company" Corporate Structure and Relationships Quality of Fields for Matching Unique Identifier Fields Name Fields Address Fields Country fields FIPS Code Fields Industry Code Fields Geocoding Other Fields Additional Notes

U.S. Department of 

Labor

Occupational Safety 

& Health 

Administration

Whistleblower 

Program Retaliation 

Data

Owned Continuously; Not 

Applicable.

Garrahan, MaryAnn - OSHA 

<Garrahan.MaryAnn@dol.gov

>

www.osha.gov/whistleblo

wer/WBComplaint.html 

(public) 

Not Applicable Not Applicable To track and administer the OSHA 

whistleblower protection provisions that 

include more than twenty whistleblower 

protection statutes, including Section 

11(c) of the Occupational Safety and 

Health (OSH) Act, which prohibits any 

person from discharging or in any 

manner retaliating against any employee 

because the employee has complained 

about unsafe or unhealthful conditions 

or exercised other rights under the Act. 

Whistleblower protection provisions 

administered by OSHA also protect 

employees from retaliation for reporting 

violations of various airline, commercial 

motor carrier, motor vehicle safety, 

consumer product, environmental, 

consumer finance, food safety, health 

insurance reform, nuclear, pipeline, 

public transportation agency, railroad, 

maritime and securities laws. 

Only pertinent OSHA -DWPP employees The Integrated Management 

Information System (IMIS) contains 

Whistleblower Complaint and 

Investigation information starting in 

FY1982.

Whistleblower complaints "Employer" is statutorily defined by the 

numerous statutes which contain the 22 

different whistleblower provisions 

administered by OSHA. Under the 

whistleblower provisions, "employer" 

may include public and private sector 

employers, including the U. S. Postal 

Service, as well as other federal, state, 

and tribal entities. OSHA collects data on 

the establishment (business location) 

level; employers are the "parents" of 

establishments.

Whistleblower database does not have 

this functionality 

All data is user entered. We have edit 

checks on the number of characters on 

fields such as the FEIN but we have no 

way to verify if the numbers entered are 

accurate.

Federal EIN/TIN, DUNS, All of these are 

optional in the system.

Establishment/Employer Name and 

Doing Business As Name

Full address of Mailing Business and Site 

Address

Business and Mailing address contain 

Country, International City, International 

Division and International Postal Code

Yes; County and State; character 6 digit NAICS No

U.S. Small Business 

Administration

Office of 

Entrepreneurial 

Development

Entrepreneurial 

Development 

Management 

Information System 

(EDMIS)

Owned Updated Quarterly, 

no lag

edmis@sba.gov N/A N/A www.sba.gov/edmis Data are collected for program 

administration, performance, and 

evaluation purposes. 

Client (entrepreneur or small business 

owner) level data is available for all those 

who have sought services from an OED 

Resource Partner or online Learning 

Center course. There are approximately 

1 M records annually across all 

programs.

Client (clients may or may not own a 

business)

The client self-identifies if they have an 

active business

N/A There is no client level PII in the system None Client names are not required Client address is not required Not required No Industry Category (not NAICS) No No N/A

U.S. Department of 

Commerce

US Census Bureau Survey of Business 

Owners

Owned Annually ewd.survey.of.business.owne

rs@census.gov

www.census.gov/program

s-surveys/sbo/technicaldocumentation/questionn

aires.html

www.census.gov/econ/ove

rview/mu0200.html

The Survey of Business Owners (SBO) 

provides the only comprehensive, 

regularly collected source of information 

on selected economic and demographic 

characteristics for businesses and 

business owners by gender, ethnicity, 

race, and veteran status.

The data are confidential [Title 13 and 

Title 26, US Code]. Access is restricted to 

persons specially sworn to uphold the 

confidentiality provisions of Title 13 and 

Title 26.

Included are all nonfarm businesses filing 

Internal Revenue Service tax forms as 

individual proprietorships, partnerships, 

or any type of corporation, and with 

receipts of $1,000 or more. The SBO 

covers both firms with paid employees 

and firms with no paid employees. The 

SBO is conducted on a company or firm 

basis rather than an establishment basis. 

A company or firm is a business 

consisting of one or more domestic 

establishments that the reporting firm 

specified under its ownership or control.

Firm, usually single unit A firm is a business organization or 

entity consisting of one domestic 

establishment (location) or more under 

common ownership or control. All 

establishments are included as part of 

the owning or controlling firm. For the 

economic census, the terms "firm" and 

"company" are synonymous.

NA Outside data must first be matched to 

the Business Register and then linked to 

the SBO using the BR's business 

identifiers.

Internal Census identifiers such as the 

LBDNUM

NA (available through Business Register 

link)

NA (available through Business Register 

link)

Yes; County and State; character 6-digit NAICS NA (available through Business Register 

link)

U.S. Department of 

Commerce

US Census Bureau Longitudinal 

Business Database

Owned Annually ces.contacts@census.gov The LBD is built by linking 

consecutive years of the 

Census Bureau's Business 

Register

www.census.gov/ces/data

products/datasets/lbd.htm

l

The Business Register information is 

confidential [Title 13 and Title 26, US 

Code]. Access is restricted to persons 

specially sworn to uphold the 

confidentiality provisions of Title 13 and 

Title 26.

The data are compiled by combining data Establishment-based; An establishment is a single physical 

location where business transactions 

take place and for which payroll and 

employment records are kept. Groups of 

one or more establishments under 

common ownership or control are 

enterprises. A single-unit enterprise 

owns or operates only one 

establishment. A multi-unit enterprise 

owns or operates two or more 

establishments.

Multi-unit enterprises may use multiple 

EINs, and they may use different ones 

when reporting to different federal 

agencies, complicating the matching 

process. The Census Bureau has linked 

many, but not all of the EINs to 

enterprises.

Outside data must first be matched to 

the Business Register and then linked to 

the LBD using the BR's business 

identifiers.

A Cenus-constructed "LBDNUM" field 

links establishments over time.

NA NA NA Yes, county, state FIPS, character. 6-digit NAICS codes; Yes; latitude and longitude to six digits 

after the decimal; numeric

U.S. Department of 

Transportation

NHTSA Manufacturer 

Information 

Database (MID)

Owned Continuously manufacturerinfo@dot.gov vpic.nhtsa.dot.gov/mid catalog.data.gov/dataset/n

htsa-product-informationcatalog-and-vehicle-listingvpic-mid-8ee79

vpic.nhtsa.dot.gov/

Per CFR 49 Part 566, CFR 49 Part 565, 

CFR 49 Part 574

Publicly available data is available at: 

vpic.nhtsa.dot.gov/mid

Manufacturer informatin database 

contains the following items:

- World Manufacturer Identifiers (WMIs) 

assigned to a manufacturer assigned by 

SAE and referenced in vPIC. 

- Motor vehicle manufacturer 

information (Address, Products 

Manufactured, Makes, etc.).

- Motor vehicle identification information 

(vehicle VIN) 

- Vehicle equipment plant information

establishment Motor vehicle modifiers for handicapped MID is designed to store parent/child 

relationship, but most of the fields are 

not populated. 

When data are collected, there is a group 

of staff reviewing the information before 

they are accepted into the system. Email 

are sent out to the entities to confim the 

contact information submitted. For 

vehicle identification informatin, quality 

check run are regularly performed to 

compare with the results from other 

databases.

manufacturerID (System generated) Common Name, Previous Legal Name

URL

All Company DBAs

Trade/Brand Names

Full company mailing address Contains country fields for non-US 

companies

NO no

U.S. Department of 

Transportation

NHTSA Manufacturer 

Information 

Database (MID) - 

Modifiers

Owned Continuously manufacturerinfo@dot.gov vpic.nhtsa.dot.gov/mid/ho

me/ModifierSearch

TBD vpic.nhtsa.dot.gov/ CFR 49 Part 595 Publicly available data is available at: 

vpic.nhtsa.dot.gov/mid/home/ModifierS

earch

Manufacturer informatin database 

contains the following items:

- Vehicle modifier (for handicapped) 

registration information

establishment Motor vehicle industry manufacturers 

and alterers of vehicles, motor vehicle 

equipment plants, motor vehicle 

modifiers for handicapped

MID is designed to store parent/child 

relationship, but most of the fields are 

not populated. 

When data are collected, there is a group 

of staff reviewing the information before 

they are accepted into the system. Email 

are sent out to the entities to confim the 

contact information submitted. For 

vehicle identification informatin, quality 

check run are regularly performed to 

compare with the results from other 

databases.

manufacturerID (System generated) Common Name, Previous Legal Name

URL

All Company DBAs

Trade/Brand Names

Full company mailing address Contains country fields for non-US 

companies

no NO

U.S. Department of 

Transportation

Pipeline and 

Hazardous 

Materials 

Administration(PH

MSA)

Office of Pipeline 

Safety

pipeline operator 

registry data is 

contained in 

PHMSA'sPipeline 

DataMart(PDM)

Owned Continuous update 

by operator for 

registration and 

notification 

changes.

william.mccarthy@dot.gov Not publicly available catalog.data.gov/dataset?c

ollection_package_id=1a13

d07b-a2e4-4acc-b194-

cf4dc330e350

primis.phmsa.dot.gov/com

m/reports/operator/Opera

torlist.html

The pipeline national operator registry is 

a data collection tool PHMSA developed 

to provide pipeline operators an 

automated portal in which they can 

register and update changes to their 

pipelines. This data is combined with 

various regulatory applications to 

perform inspections and enforcements 

of all interstate gas and liquid pipelines.

this data is collected and conformed with 

inspections, enforcements,incident and 

annual reporting in a PHMSA data 

warehouse that is available to PHMSA 

and state regulatory agencies

registration based Companies may be either an owner of a 

pipeline or an operator for another 

owner.

Corporate pipeline operator may 

structure themselves as a single 

corporate entity or sub corporate 

entities. 

an operator identification number is 

assigned by PHMSA upon operator 

registration in the PHMSA appliation 

portal.

Company name Physical address headquarters country none none none company contacts(name, 

phone,emailaddress)

U.S. Department of 

Transportation

Pipeline and 

Hazardous 

Materials 

Administration(PH

MSA)

Office of Hazmat 

Safety

Hazardous 

Materials 

Information Poral 

(HIP)

Owned Updaed dailly by 

various PHMSA and 

FMCSA sources

adrian.carter@dot.gov Not publicly available Special Permits: 

catalog.data.gov/dataset/h

azmat-special-permitsdata-mining-tool-d7acf

Approvals: 

catalog.data.gov/dataset/h

azmat-approvals-datamining-tool-9e63b

Hazmat Incident reports: 

catalog.data.gov/dataset/h

azmat-incident-reportsdata-mining-tool-e6800

Hazmat Registratoin: 

catalog.data.gov/dataset/h

azmat-registrationstatistics-2011-f2e80

www.phmsa.dot.gov/hazm

at/e-services

The purpose of gathering this data is to 

help regulate those that manufacture, 

transport, ship, test and package 

hazardous matierials.

This information is restricted to US DOT 

hazmat personnel

The data includes hazardous materials 

special permits, approvals, registrations, 

incidents, RAM certications and 

enforcemetn case details.

All records are transactional Compnaines include hazmat carriers, 

shippers, manufacturers, packagers and 

testers.

Corporate relatonship exists between 

locations that share a legal liability

Company information is maintained in 

the HIP master compnay hub (MCH). 

The fields include company name, 

address, city, state zip and D&B number. 

Any compnay that does business with 

PHMSA trhough the various transactions 

gets validated by D&B and is added to 

the MCH if they don't already exist. 

The D&B number is used as the unique 

identifier in the master company hub. If 

no D&B number exists, the system 

generates a unique number.

Company name Physical address physical country FIPS country code DOT Mode of transportation, US DOT #, 

D&D number

N/A N/A

U.S. Department of 

Transportation

Federal Aviation 

Administration

Air Operators Owned Weekly Tim.Perez@faa.gov avinfo.faa.gov/data/AirOpera

tors/tab/airopera.txt

The FAA regulates the Air Operators. The dataset is publicly available. Contains Air Operator name, certificate 

number, and types of certificates.

Contains 5210 records.

Establishment-based The company is an employer. This is a 

designator of an Air Operator or Air 

Agency by a governmental registration 

authority. They are certificate holders. 

The designator is an identifying code that 

identifies an organization as holding one 

or more certificates for conducting 

aviation operations. Air Operator is often 

referred to as an "Air carrier" which is a 

organization that undertakes directly by 

lease, or other arrangement, to engage 

in air transportation.

Air Operator Designation Code 

(Degn_Code)

Air Operator Name None No, does not contain this field. No, does not contain this field. None None None The file layout is: avinfo.faa.gov/data/AirOperators/AirOpera

.doc

U.S. Department of 

Transportation

Federal Aviation 

Administration

Contractors for Air 

Operators

Owned Weekly Tim.Perez@faa.gov avinfo.faa.gov/data/AIROPER

ATORS/fix/contract.txt

The FAA regulates the Air Operators. The dataset is publicly available. Contains Air Operator name, address, 

and type.

Contains 3075 records

Establishment-based The company is an employer. The 

company provides contracting services 

to Air Operators.

Contractor Designation Code 

(Contractor_Dsgn_Code)

Air Operator Contractor Name Employer Street Address, City Name and 

State Abbreviation, Zip Code

Contains the full address. 

Yes, the dataset does include a Country 

field and contains data of countries 

outside of the United States, such as 

Canada.

No, does not contain this field. None None Air Operator Designation Code 

(Dsgn_Code) can be used to tie this 

Contractor record to an associated Air 

Operator record in the various datasets 

containing Air Operator.

The file layout is: avinfo.faa.gov/data/AIROPERATORS/Contr

act.doc

U.S. Department of 

Transportation

Federal Aviation 

Administration

Air Operator CEO 

Name

Owned Weekly Tim.Perez@faa.gov avinfo.faa.gov/data/AirOpera

tors/tab/operceo.txt

The FAA regulates the Air Operators. The dataset is publicly available. Contains Air Operator CEO's name, 

address, phone

Contains 5910 records

The company is an employer. This is a 

designator of an Air Operator or Air 

Agency by a governmental registration 

authority. They are certificate holders. 

The designator is an identifying code that 

identifies an organization as holding one 

or more certificates for conducting 

aviation operations. Air Operator is often 

referred to as an "Air carrier" which is a 

organization that undertakes directly by 

lease, or other arrangement, to engage 

in air transportation.

Air Operator Designation Code 

(Degn_Code)

Name of the Chief Executive Officer of an 

Air Operator 

Chief Executive Officer Street Address, 

City Name and State Abbreviation, Zip 

Code

Contains the full address. 

Yes, the dataset does include a Country 

field and contains data of countries 

outside of the United States, such as 

Canada.

No, does not contain this field. Air Operator Designation Code 

(Dsgn_Code) can be used to tie this 

Contractor record to an associated Air 

Operator record in the various datasets 

containing Air Operator.

The file layout is: avinfo.faa.gov/data/AIROPERATORS/OperC

EO.doc

U.S. Department of 

Transportation

Federal Aviation 

Administration

Air Operator Doing 

Business As Name 

Owned Weekly Tim.Perez@faa.gov avinfo.faa.gov/data/AirOpera

tors/tab/operdba.txt

The FAA regulates the Air Operators. The dataset is publicly available. Contains Air Operator Doing Business As 

Authorized name

Contains 1784 records

Establishment-based The company is an employer. This is a 

designator of an Air Operator or Air 

Agency by a governmental registration 

authority. They are certificate holders. 

The designator is an identifying code that 

identifies an organization as holding one 

or more certificates for conducting 

aviation operations. Air Operator is often 

referred to as an "Air carrier" which is a 

organization that undertakes directly by 

lease, or other arrangement, to engage 

in air transportation.

Air Operator Designation Code 

(Degn_Code)

Authorized Doing Business As Name for 

an Air Operator

None No, does not contain this field. No, does not contain this field. None None Air Operator Designation Code 

(Dsgn_Code) can be used to tie this 

Contractor record to an associated Air 

Operator record in the various datasets 

containing Air Operator.

The file layout is: avinfo.faa.gov/data/AIROPERATORS/Oper

DBA.doc

U.S. Department of 

Transportation

Federal Motor 

Carrier Safety 

Administration

Motor Carrier 

Census

Owned Continuously Norma.Ott@dot.gov ai.fmcsa.dot.gov/SMS/Tool

s/Downloads.aspx

The FMCSA regulates carriers engaged in 

the transportation of people and goods 

across state lines.

The dataset is publicly available. Is a Census of interstate motor carriers Establishment-based Per the Federal Motor Carrier Safety 

Regulations. See guidance 

www.fmcsa.dot.gov/registration/unifiedregistration-system

Only headquarters element. No 

relationships maintained.

Company Name, address, and EIN/TIN 

(TIN in the case of small owneroperators)

US DOT Number Company name Employer Street Address, City Name and 

State Abbreviation, Zip Code

Contains the full address.

Yes, if a Mexican/Canadian domiciled 

company.

No, does not contain this field. None None USDOT Number is an agency-assigned 

code.

See download link for file layout.
896 The Promise of Evidence-Based Policymaking

Employer Data Matching Workgroup

Data Inventory

Page 5 of 5

Department/

Agency

Component Agency 

or Office

Dataset Name Dataset Type: 

Owned by Data 

Inventory 

Respondent, or 

Desired by Data 

Inventory 

Respondent?

Data Update 

Frequency and Lags

Contact E-mail Address Link to Data Collection 

Tool

Data.gov Link Other Relevant Link(s) Purpose of the Data Collection Access Restrictions Coverage Unit of Analysis Definition of "Company" Corporate Structure and Relationships Quality of Fields for Matching Unique Identifier Fields Name Fields Address Fields Country fields FIPS Code Fields Industry Code Fields Geocoding Other Fields Additional Notes

U.S. Department of 

Transportation

Federal Railroad 

Administration

List of railroads and 

companies

Owned Continuously Andrew.Martin@dot.gov safetydata.fra.dot.gov/Offi

ceofSafety/publicsite/down

loads/RailroadAndCompan

yFile/RailroadAndCompany

File.aspx

The FRA regulates railroad companies. The dataset is publicly available. Establishment-based See 

safetydata.fra.dot.gov/OfficeofSafety/pu

blicsite/ReportingLevelDefinitions.aspx

See 

safetydata.fra.dot.gov/OfficeofSafety/pu

blicsite/ReportingLevelDefinitions.aspx

Company Name, address Railroad code Company name Employer Street Address, City Name and 

State Abbreviation, Zip Code

Contains the full address. 

No, does not contain this field. No None none Railroad identifier is industry-assigned See download link for file layout.

U.S. Small Business 

Administration

Office of Disaster 

Asistance

Disaster Credit 

Management 

System

Owned Updated Annually N/A www.sba.gov/loansgrants/see-what-sbaoffers/sba-loanprograms/disasterloans/disaster-loan-dataN/A www.sba.gov/loansgrants/see-what-sbaoffers/sba-loanprograms/disasterloans/disaster-loan-dataData are collected for program general 

public knowledge and evaluation 

purposes. 

None 2004 - 2015 Individual Disaster Loan Approval Individuals / companies are not revealed N/A No PII is within files Declaration ID / zip code / state / county 

/ business or home loan

N/A N/A Not required No No zip code No N/A

U.S. Small Business 

Administration

Office of Capital 

Access

EIS Owned Updated weekly run N/A N/A N/A N/A E-tran - system for electronically 

capturing loan application information 

from lenders; Data collected to track 7(a) 

& 504 loan approvals

None All loans approved, information updated 

in real-time with weekly database 

refreshes.

Individual Loan Approval Borrower Firm Name, TIN N/A High: Full details of loan approval 

process, including the lender and bank, 

vetted by bank before the approval

TIN, Demographic, location, details on 

loan program

Borrower & Lender Names captured Borrower & Lender Addresses captured Yes No NAICS Code; character Zip code No N/A

U.S. Small Business 

Administration

Office of Capital 

Access

MPERS Owned Updated weekly run N/A N/A N/A N/A MPERS - system for Microloan 

intermediaries to self-report Microloan 

loan information; Data collected from 

Microloan intermediaries on loans to 

Microborrowers

None Self-reported Microborrower 

information, reported up to 3 months 

after Microloan approval

Individual Microloan Approval Microborrower Name, SS# N/A Medium: Data is self-reported from 

lender-intermediaries; there is no QA on 

the data

SSN, Demographic, location Microborrower and Microlender 

Intermediary Names captured

Microborrower and Microlender 

Intermediary Addresses captured

Yes No NAICS Code; character Zip code No N/A

U.S. Department of 

Labor

Employee Benefits 

Security 

Administration 

(EBSA)

Form 5500 Data 

Sets

Owned The Form 5500 

Data is updated on 

the website first 

business day of 

each month. 

foiarequest@dol.gov www.dol.gov/agencies/ebs

a/about-ebsa/ouractivities/publicdisclosure/foia/form-5500-

datasets

catalog.data.gov/dataset/f

orm-5500-foia-datasets

www.efast.dol.gov/portal/

app/disseminate?executio

n=e2s1

Data is used for compliance, 

enforcement, and research. The Internal 

Revenue Service (IRS), Department of 

Labor (DOL), and Pension Benefit 

Guaranty Corporation (PBGC) have 

consolidated certain returns and report 

forms to reduce the filing burden for plan 

administrators and employers. 

Employers and administrators who 

comply with the instructions for the 

Form 5500 generally will satisfy the 

annual reporting requirements for the 

IRS, and DOL, and PBGC.

The data sets are shared by IRS, PBGC, 

and DOL. Some data containing Personal 

Identifying Information (such as SSN) will 

not be available.

Over 700,000 Pension and Welfare 

filings for a given plan year. Specific plan 

information are attached (participant, 

asset, etc.). A return/report must be 

filed every year for every pension benefit 

plan, welfare benefit plan, and for every 

entity that files as a DFE as specified 

below (pursuant to Code section 6058 

and ERISA sections 104 and 4065).

Plan based unit of analysis Usually based on entire firm, not on a 

single establishment. Firms will be 

reported in data base as plan sponsors 

or plan service providers.

Sponsor to benefit plan(s) structure. 

Same sponsor may provide multiple 

plans. Each company may be 

represented in various plans. (If any 

connection can be identified, it's usually 

through the EIN)

Name fields are not very dependable. 

Numerous different names can be 

spelled, shortened, etc for 1 given plan 

or sponsor. Ack_ID and EIN fields are 

primarily used. Other variables such as 

participant, asset, and plan characteristic 

fields are used to ensure quality. Various 

edit checks are performed to ensure no 

duplicate filings are tabulated. 

EIN; Ack_ID Yes; Plan name, Sponsor name, Plan 

administrator name. 

Yes, all are simple mailing addresses and 

not where the plans are established; 

Sponsor mailing address and Plan 

administrator mailing address; 

Theoretically, the data can contain nonUS address. If outside US, enter the 

foreign routing code, if applicable. Leave 

U.S.state and zip code blank if entering a 

foreign routing code and country name.

No 6-digit NAICS codes; No Possibly match name fields and EINs to 

other sources. Not sure about the 

resulting quality.

Page 35 from the following pdf has data 

variables: 

www.dol.gov/agencies/ebsa/researchers

/data/retirement/pension-user-guide2013.pdf

U.S. Department of 

Treasury

Internal Revenue 

Service - Statistics 

of Income

Business Master File Owned Data are updated 

continuously; Data 

becomes available 

weekly.

sis@irs.gov Not applicable Data.gov doesn’t host any 

of our data, it just points at 

our pages on IRS.gov. As 

far as us pointing out to 

Data.gov, there’s only a 

couple places where we 

direct users out to the 

Data.gov site through Tax 

Stats on IRS.gov.

www.irs.gov/uac/tax-stats IRS: The purpose of data collection is 

mainly for determining Federal tax 

liability for businesses required to file. 

SOI: The data are used to produce 

statistics on income, deductions, credits 

and other taxes, as reported by 

businesses. The current design is a 

probability sample stratified by Business 

type (as indicated by the IRS form filed) 

and either by size of total assets alone or 

size of total assets and a measure of 

income.

Federal Tax Information (FTI) is 

confidential [Title 26, US Code] and 

shared with other government agencies 

under IRC 6103(j) provisions.

IRS: All Federal business tax returns that 

meet IRS filing requirements. SOI: 

Selected active Federal business tax 

returns based on SOI's sample design.

Tax Return. Definition of "company” is based on Title 

26 requirement.

The dataset consists of individually filed 

returns which can represent: 

corporations (a single establishment; 

subsidiary establishment or consolidated 

filed returns representing a group of 

establishments); partnerships and other 

pass-through entities; or sole 

proprietorships. 

Most matching is accomplished using 

the EINs provided by entities as reported 

on Federal tax returns. SOI uses exact 

matching with EINs and data processing 

begins with information already 

extracted for IRS administrative 

purposes. SOI performs limited internal 

"data cleaning" for statistical purposes. 

This includes organizing data to make it 

structurally consistent, coding data items 

to make them analytically useful, and 

validating values to ensure mathematical 

consistency. Contact information is 

validated as part of routine 

administrative processing of returns at 

the time they are received by the IRS.

Employer Identification Number (EIN) Yes; Parent Company Name Yes; Tax Filing Address, City, State 

Abbreviation, Zip Code

 - - 6-digit NAICS codes - - The focal point of the answers provided 

is based on data collection for Statistics 

of Income (SOI) unless otherwise noted.
The Promise of Evidence-Based Policymaking 897

From Data to Evidence to Policy

Recommendations for the Commission on Evidence-Based Policymaking

MAY, 2016

The William T. Grant Foundation

The William T. Grant Foundation supports research to improve the lives of young people ages 5–25 in the United 

States. Our goal is to accumulate a body of knowledge that will advance theory, policy, and practice and contribute 

to improved outcomes and opportunities for youth, today and in the future. In 2014, we launched an initiative to 

invest in research to identify effective responses to inequality in its many forms. And since 2009, we have supported 

studies that provide insight into how policymakers, administrators, and service providers acquire, interpret, and use 

research evidence. In 2015, we signaled a new direction in this initiative, calling for studies that identify, create, and 

test strategies to improve the use of research evidence in ways that benefit youth. 

The Forum for Youth Investment

The Forum for Youth Investment is a nonprofit, nonpartisan "action tank" dedicated to helping communities and the 

nation make sure all young people are Ready by 21®: ready for college, work and life. A trusted resource for policy 

makers, advocates, researchers and program professionals, the Forum provides youth and adult leaders with the 

information, connections and tools they need to create greater opportunities and outcomes for young people. The 

Forum manages a number of centers and projects, including Big Picture Approach Consulting, the David P. Weikart 

Center for Youth Program Quality, the Children’s Cabinet Network and SparkAction. The core work of the Forum is 

helping leaders, organizations, partnerships and systems – at the local, state and national levels – assess, improve 

and align their practices and policies.
898 The Promise of Evidence-Based Policymaking

1 | Page

From Data to Evidence to Policy

Recommendations for the Commission on Evidence-Based Policymaking

The Forum for Youth Investment and the William T. Grant Foundation are pleased to submit the 

following recommendations to the Commission on Evidence-Based Policymaking. We are encouraged by 

the Commission’s potential to promote the use of research evidence in policymaking, and we offer these 

insights with the hope of helping the Commission make the most of its historic opportunity. 

These suggestions are drawn from our experience as conveners of a learning group of senior career staff 

and appointees in research offices focused on children, youth, and families within the U.S. Departments 

of Education, Labor, Justice, and Health and Human Services, as well as in the Corporation for National 

and Community Service and the National Science Foundation. These agencies invest in research and 

evaluation to build policy-relevant evidence and will likely be charged with implementing many of the 

Commission’s recommendations. They also have experience responding to similar challenges in the past. 

For these reasons, their insights may guide the Commission in developing a transformational set of 

recommendations.

The Path from Data to Evidence to Policy

A narrow interpretation of the Commission on Evidence-Based Policymaking legislation might suggest 

the Commission’s charge is primarily to determine how the federal government can share and link 

administrative data sets. Accomplishing this aim would be of significant value. But a broader 

interpretation of its charge suggests that the Commission must not stop there. It should also consider 

how the federal government can use data to create the evidence required for smart policy decisions, as 

well as how to create the infrastructure to support the use of evidence in policymaking.

We urge the Commission to prioritize those elements of its charge that point toward these broader 

aims:

 Emphasize how data “may be integrated and made available to facilitate program evaluation, 

continuous improvement, policy-relevant research, and cost-benefit analyses by qualified 

researchers and institutions;” and

 Examine “how data and results of research can be used to inform program administrators and 

policymakers to improve program design.”

Sharing and linking data is necessary but not sufficient to achieve these goals. The data must be 

deployed in research and evaluation to create evidence, which must be then be used to inform 

policymaking. Laying the groundwork for the path from data to evidence to policy is essential to the 

Commission’s charge.

Sharing and 

Linking Data

Using Data to 

Create 

Evidence

Using Evidence 

to Inform 

Policymaking
The Promise of Evidence-Based Policymaking 899

2 | Page

Sharing and Linking Data

The Commission should recommend that data are collected and 

shared in ways that facilitate their use in research and evaluation.

Sharing data is a valuable first step. But the data take on added 

power when they can be used in research and evaluation studies. 

The Commission can fulfill its charge of addressing “how data and 

results can be used to inform program administrators and 

policymakers” by recommending ways that administrative data 

systems can be made ready for use in research and evaluation. The 

Commission could craft recommendations that would help ensure 

that data are linked; are of sufficient quality; and are delivered in 

formats that encourage their use in research that answers

policymakers’ questions.

Linking separate data sets together increases the value of each for 

research and evaluation. One federal staff member in our learning 

group commented, “I am working on linking existing databases to 

conduct research. I need to figure out what kinds of data are being 

collected....Who is tracking relevant outcomes, and how do you 

synthesize that with community-level data from multiple sources to 

tell the impact of interventions across multiple domains?”

Linking existing data can also enable faster and cheaper research studies. As one participant reported, 

“There has been a lot of thinking overall in our department on the opportunities these longitudinal data 

systems have for low cost evaluations. A whole group of people in our program offices are focused on 

this right now.” But using data that are not designed for research can be challenging. Sometimes the 

quality of the data is poor or unknown. As a learning group participant put it, “It is not easy to figure out 

if a given set of administrative data is high-quality enough to be used in a research context.”

Sometimes the ways that the data are shared make it difficult to produce useful research reports. As 

one learning group participant shared, “Only one or two people in each of our agencies know how to 

manipulate specific databases, and those people have a long list of requests from multiple agencies to 

do specific data runs. We are hoping that we can take on the burden of getting the data ready. A lot of 

the data we have, like those that manage case files, were not created to be used for research, so that is 

a heavier lift. We want to give agencies a more realistic view of the data they actually have.”

Using Data to Create Evidence

The Commission should recommend that federal agencies adopt a broad and inclusive view of the types of 

research studies that can and should be produced with administrative data.

The power of data increases when they are used to create research evidence (see sidebar). Too often, 

however, a narrow conceptualization of data as evidence limits the ability of policymakers to gain full 

understanding of an issue. The Commission could add value to the field by advancing a framework that 

delineates the types of research evidence that should be created to guide policymaking.

The William T. Grant Foundation 

defines research evidence as “a 

type of evidence derived from 

applying systematic methods 

and analyses to address a 

predefined question or 

hypothesis. This includes 

descriptive studies, intervention 

or evaluation studies, metaanalyses, and cost-effectiveness 

studies conducted within or 

outside research organizations.”

DEFINITION OF RESEARCH 

EVIDENCE
900 The Promise of Evidence-Based Policymaking

3 | Page

The Commission may wish to reinforce frameworks such as the Institute of Education Sciences/National 

Science Foundation’s Common Guidelines for Education Research Development framework1 and the 

Health and Human Services Administration for Children and Families’ Common Framework for Research 

& Evaluation.2 Adopting these types of frameworks would help ensure that policymakers not only 

receive findings from the full range of types of research evidence, but review this evidence with a clear 

understanding of the level of rigor and quality inherent in each type of study, and what types of 

questions are answerable by each respective research methodology.3

Using Evidence to Inform Policymaking

The Commission should recommend that federal agencies support partnerships between researchers and 

policymakers that inform key research questions and facilitate the use of research evidence.

Partnerships between researchers and policymakers can improve the use of research evidence by 

guiding researchers to ask questions that respond to the needs of policymakers, building stronger 

practice-focused research networks or community-based participatory approaches, and creating a 

culture of learning in which administrators, policymakers and other government leaders include 

research evidence in their deliberations.

The growing literature about how and when research evidence is and is not used in policymaking can

inform the Commission’s work. These studies “complicate the common conception of research users as 

merely rational actors who have questions, go in search of research to answer them, and then apply it to 

their decisions.... In none of their cases does research use easily boil down to a single moment or an 

isolated decision....It is not a simple process whereby research ‘facts’ are passed from researchers to 

research users and then applied in a linear decision making process. Instead, research use is contingent, 

interactive, and iterative. It involves people individually and collectively engaging with research over 

time, bringing their own and their organization’s goals, motivations, routines, and political contexts with 

them.”4 It is often helpful to construct evidence in a process of engagement, in which the needs of 

decision makers help shape research questions and findings are delivered in an accessible and timely 

fashion through relationships of trust and mutual understanding.

The Commission should recommend ways to strengthen the federal infrastructure for producing research 

evidence that can inform policymaking.

To address its charge to examine how data may be “made available to facilitate program evaluation, 

continuous improvement, policy-relevant research, and cost-benefit analyses by qualified researchers 

and institutions,” it is important for the Commission to create recommendations for strengthening the 

federal infrastructure for using data to create evidence, and using that evidence to inform policymaking.

 1 Available at: http://www.nsf.gov/pubs/2013/nsf13126/nsf13126.pdf 2 Available at: http://www.acf.hhs.gov/programs/opre/resource/the-administration-for-children-familiescommon-framework-for-research-and-evaluation 3 While there may not yet be widely accepted quality standards for the all types of research, some of our learning 

group members pointed to the following article as a promising start: Gottfredson, D. C., Cook, T. D., Gardner, F. E. 

M., Gorman-Smith, D., Howe, G. W., Sandler, I. N., & and Zafft, K. M. (2015). Standards of evidence for efficacy, 

effectiveness, and scale-up research in prevention science: Next generation. Prevention Science, 16, 893-926. 4 Tseng, V. & Nutley, S. (2014). Building the Infrastructure to Improve the Use and Usefulness of Research in 

Education. In K.S. Finnigan & A.J. Daly (Eds.). Using Research Evidence in Education: From the Schoolhouse Door to 

Capitol Hill. Springer: New York. Available at: http://wtgrantfoundation.org/resource/building-the-infrastructure-toimprove-the-use-and-usefulness-of-research-in-education
The Promise of Evidence-Based Policymaking 901

4 | Page

An infrastructure could include elements such as formal policies, codified practices, established offices,

and interagency coordinating structures. 

Fortunately, there are existing efforts that the Commission could build upon. The Department of Labor 

and the Department of Health and Human Services, Administration for Children and Families both 

created evaluation policies that can serve as models for other agencies.

5 The Department of Labor 

created a Chief Evaluation Office, and conducts an annual survey to assess its performance meeting the 

research needs of program offices. HHS Administration for Children and Families, Office of Planning 

Research and Evaluation also published a set of principles to guide all of its evaluation work.

6 Further,

the Institute of Education Sciences has protections, granted by Congress in its authorizing language, that 

support scientific integrity and independence from political influence.

Additionally, a subset of learning group participants funded a National Academy of Sciences Roundtable 

to consider an infrastructure for evaluation that parallels the existing infrastructure for statistical 

agencies. (Federal statistical agencies receive support from a Chief Statistician housed at the White 

House Office of Management and Budget; a public-private, interagency Committee on National 

Statistics; and a carefully codified and updated set of Principles and Practices for a Federal Statistical 

Agency.7

) The Commission may wish to be briefed on this effort. The Commission may also wish to be 

briefed on the types of infrastructure that have been created to support federal agencies’ performance 

management functions, such as the role the federal Performance Improvement Council plays in 

fostering widespread and effective use of performance management practices across federal agencies

and sparking cross-cutting performance improvements.8

Conclusion

The creation of the Commission on Evidence-Based Policymaking is well timed. As shared in the William 

T. Grant Foundation’s recent blog series Evidence at the Crossroads, “research evidence can improve 

public policies and programs, but fulfilling that potential will require honest assessments of current 

initiatives, coming to terms with outsized expectations, and learning ways to improve social 

interventions and public systems.”9

The Commission is well positioned to drive this work forward, especially if it focuses on the full 

continuum of activity from sharing and linking data, to using those data to create research evidence, to 

using that evidence to inform policymaking.

 5 See http://www.dol.gov/asp/evaluation/EvaluationPolicy.htm and 

http://www.acf.hhs.gov/programs/opre/resource/acf-evaluation-policy 6 The HHS principles are posted at: http://www.acf.hhs.gov/opre/resource/acf-evaluation-policy 7 Principles and practices for federal statistics agencies are posted at: 

http://sites.nationalacademies.org/DBASSE/CNSTAT/Principles_and_Practices_for_a_Federal_Statistical_Agency/i

ndex.htm

8 On the role of the Performance Improvement Council, see: http://www.gsa.gov/portal/content/133807 9 See: http://wtgrantfoundation.org/tag/evidence-at-the-crossroads
902 The Promise of Evidence-Based Policymaking

1

Memorandum

TO: The Commission on Evidence-based Policy

FROM: Members of the Interagency Council on Evaluation Policy

SUBJECT: “Top-5 List” of issues and solutions related to Federal evaluation activity

DATE: November 22, 2016

Several individual members of the Interagency Council on Evaluation Policy appreciate the 

opportunity to provide input to the Commissioners as you proceed to consider strategies to 

improve data for use in evidence-building. The following are the top issues of concern to 

Federal evaluation offices, along with several solutions for the Commission to consider.

1. ISSUE: Key federal administrative and statistical data sources are inaccessible for 

evaluation purposes or incomplete as a result of statute, policy, or administrative 

practices.

SOLUTION: Establish a mechanism to assist agencies to act upon their M-14-06 

statistical and administrative data priorities:

a. Establish cooperative information technology procedures (e.g., data security, 

privacy, and data maintenance) for Federal agencies that share or exchange data

to do so more efficiently (currently agencies have different informational 

technology and privacy rules and procedures which typically requires a new or 

modified memorandum of agreement)

b. Establish expedited procedures for Federal agencies to more easily and directly 

access data from Federal statistical agencies for evaluations (e.g., develop special 

approval and access procedures for Federal agencies rather than requiring them to 

follow the cumbersome general public procedures for accessing data through

Census Research Data Centers; allow federal agencies to access the data directly 

from secure Federal offices and computers).

c. Require records in some key Federal data bases used for evaluations, such as the 

National Directory of New Hires (NDNH) to be maintained permanently to allow 

data to be linked and tracked for evaluations requiring long-term data (e.g., preprogram and post-program follow up).

2. ISSUE: Federal evaluation capacity and activity is uneven across agencies

SOLUTION: Establish an independent federal evaluation system to 1) articulate the 

specific role and value of evaluation in evidence-building, 2) facilitate the development 

of strong, independent evaluation offices in all agencies, and 3) reinforce the importance 

of evaluation offices to strengthen the support for the offices within their agencies.
The Promise of Evidence-Based Policymaking 903

2

a. Encourage Departments and Agencies to establish independent evaluation 

offices to coordinate evaluation efforts and build evaluation capacity, but 

also acknowledge there is often a complementary and important role for

special independent research institutes, bureaus or offices where they exist 

(e.g., DOJ-NIJ, USDA-ERS, DOL-BLS, Commerce-Census).

b. Develop general principles and practices for Federal evaluation offices (see 

NAS workshop)

c. Clarify that evaluation and evidence-building is a unique activity, not to be 

conflated with data collection or performance measurement and monitoring:

i. Evidence is not reducible to data (see Goldstein/ACF testimony)

ii. Evaluation capacity is not reducible to data collection capacity (see 

Goldstein/ACF testimony)

iii. Rigorous evaluation is not reducible to impact studies using RCTs (see 

Goldstein/ACF testimony)

d. Establish an interagency evaluation coordination structure (see 

Goldstein/ACF testimony)

e. Make the Federal evaluation system a complement and counterpart to the 

Federal statistical system rather than subsumed by the latter (see 

Goldstein/ACF testimony)

f. Strengthen and clarify privacy/confidentiality protections that also allow 

access for Federal evaluations by clarifying statutory provisions in the 

Privacy Act, Freedom of Information Act, and the Family Educational Rights 

and Privacy Act (see O’Regan/HUD testimony) 

3. ISSUE: Often there is no or inadequate funding available to conduct rigorous evaluations

SOLUTION: Ensure adequate evaluation funding

a. Expand flexible evaluation funding set-aside authority to more agencies to 

allow funding for program evaluations

b. Embed evaluation funding and requirements into programs when possible

c. Increase cross-agency transfer authority to allow combining funding for 

evaluations that span multiple programs/agencies when appropriate (see 

Solution 1.b. above).

4. ISSUE: A number of bureaucratic barriers discourage evaluation, create inefficiencies 

and pose additional costs when conducting a Federal evaluation, particularly issues 

related to PRA, IAAs, and procurement.

SOLUTION: Reduce bureaucratic complexity and barriers to evaluation

a. Streamline PRA requirements for Federal evaluations to minimize cost and 

time:

i. Assign PRA responsibility to agencies on collections below some 

threshold such as 1,000 responses or under 500 burden hours (see 

O’Regan/HUD testimony)
904 The Promise of Evidence-Based Policymaking

3

ii. Allow PRA responsibility to be assigned to agency evaluation offices that 

have established formal clearance, peer/technical review, statistical 

expertise, and public notice procedures

iii. Reduce requirements for public comment periods in some cases, e.g. no 

second public comment period if there are no substantive comments 

during the first period.

b. Streamline the Federal Interagency Agreement (IAA) processes to allow 

more efficient collaboration and sharing of funds for cross-agency 

evaluations and data exchanges:

i. Use pre-approved agreement templates for expedited clearance in Federal 

agencies

ii. Standardize the legal procedures and requirements across Departments to 

facilitate and speed up interagency agreements.

c. Allow more flexible procurement strategies for evaluations to improve study 

quality and efficiency (see 2017 President’s Budget)

i. Allow Federal agencies a five year period of funds availability to allow 

agencies to pool funds over multiple years to pay for large, long-term 

evaluations.

ii. Provide Federal evaluation agencies with the authority to recapture, and 

re-obligate for other studies, unused funds not needed to complete a 

particular study (e.g., unused termination costs, or cost savings from 

projects that proceed more quickly than expected).

5. Each agency has unique data requirements for program evaluations. Below are examples 

of specific solutions that relate to evaluations using employment, wages, tax, research 

projects, health, and education data):

a. Allow access to the National Directory of New Hires (NDNH) employment 

and earnings data for Federal evaluations (see FY 2017 President’s Budget

pp. 303-307, 328-330: 

https://www.acf.hhs.gov/sites/default/files/olab/final_cj_2017_print.pdf), 

and maintain the records for longer periods of time to allow for long-term 

follow-up.

b. Allow administrative data with unique identification and collected in 

workforce development, education, research grants, public housing, social 

services and public assistance programs to be linked at the individual level 

and shared among Federal agencies for evaluations (with appropriate security 

and privacy protections). (For WIOA example see President’s Budget Fiscal 

Year 2016: Analytical Perspectives, pp. 69-70: 

https://www.whitehouse.gov/sites/default/files/omb/budget/fy2016/assets/spe

c.pdf).

c. Improve Federal agency access to the Longitudinal Employer and Household 

Dynamics (LEHD) files for evaluations.

d. Improve access to individual level student-level data (e.g., K-12 and postsecondary) for Federal evaluations (with appropriate security and privacy 

protections) (see testimony of multiple witnesses at October public hearing).
The Promise of Evidence-Based Policymaking 905

4

e. Improve the ability of the vital statistics systems to be more interoperable 

with other electronic healthy data systems and foster the use of national 

standards on birth and death statistics to support interoperability for 

evaluation and research.

We would be happy to discuss any of these issues or solutions. Please feel free to contact any of 

the following:

DOL - Molly Irwin Irwin.Molly.E.@dol.gov or Demetra Nightingale 

Nightingale.Demetr@dol.gov

HHS - Naomi Goldstein naomi.goldstein@acf.hhs.govHUD –

Katherine O’Regan Katherine.M.ORegan@hud.gov or Mark Shroder, HUD 

Mark.D.Shroder@hud.gov

USDA-FNS – Richard Lucas Richard.Lucas@fns.usda.gov

DOJ – Howard Spivak Howard.Spivak@usdoj.gov

NSF – Anand Desai adesai@nsf.gov

 
906 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 907
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: September 19, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8rzo-ikhc

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0003

Comment on FR Doc # 2016-22002

Submitter Information

Name: Steve Redburn

Address:

Washington, DC, 20005

Email: s.redburn@icloud.com

General Comment

This is a comment in response to overarching question #1:

One way systematic evidence developed with government data can be deployed to

improve policy choices is through the federal budget process. Paul Posner and I have

proposed a change in that process that would make room each year for deeper, evidencebased, systemic analysis of a handful of major national policy objectives. For each

objective, this alternative, goal-focused "portfolio budgeting" approach would group

together and thus look broadly across the set of related programs, tax provisions,

regulatory, and other federal policies affecting the goal. The relevant policy portfolio

would typically cut across agency boundaries and congressional committee jurisdictions.

For the goal of expanding access to higher education, for example, it would include

loans and grants administered by the Department of Education, GI Bill benefits for

veterans, and various tax provisions that support savings for college and charitable

giving to college endowments used for scholarships.

For each goal, analysis would assess how current federal strategy for using resources -

908 The Promise of Evidence-Based Policymaking

Comments Submitted to the Federal Register
implied by the current portfolio of spending, tax expenditures, and other policies -

incentives and shapes the behavior of other actors in the system. Combining data from

multiple federal administrative data bases would facilitate evaluation of the interactive

and combined effects of federal policies. Analysis would compare the effectiveness of

today's federal strategy with alternative strategies that could make better use of resources

to accelerate progress toward the goal and possibly generate budget savings. Through the

budget process, potential budget savings could either be reinvested in this goal or used

for other priorities, including deficit reduction.

The process now used to develop the federal budget is biased toward the incremental, the

short term, and the familiar. It also is piecemeal and fragmented, considering issues in

narrow programmatic categories and giving most scrutiny to marginal changes in

spending, largely ignoring tax policies. The way budgets are developed is too often blind

to the major shifts sweeping over the nation's economy and social structure. The result is

little change and inadequate focus on national priorities or how to achieve them more

efficiently. That is why a portfolio approach is needed.

Building on growing executive branch experience with strategic analysis and reviews,

portfolio budgeting would use policy makers' time more efficiently by helping them

focus on the biggest opportunities to adjust policies and resources to accelerate the

achievement of major national goals and identifying breakthrough gains in productive

use of resources. For Congress, the House and Senate budget committees could organize

the process by annually identifying a small set of major policy goals for analysis; asking

GAO, CBO, or the congressionally chartered national academies to assess current

strategy relative to alternatives; and in their subsequent budget resolutions, developed in

consultation with congressional leaders, other committees, and the Administration,

include policy guidance and estimates for an alternative that would use budget resources

more productively.

More details on implementing a portfolio budgeting approach can be found at

http://www.brookings.edu/research/papers/2015/09/30-portfolio-budgeting-newapproach-redburn-posner and in the National Budgeting Roundtable working paper on

budgeting for national goals at http://psc.gmu.edu/wp-content/uploads/New-Ideas-forFederal-Budget-Working-Paper-No.-2.pdf . 

Steve Redburn

Professorial Lecturer, Trachtenberg School of Public Policy and Public Administration,

George Washington University and

Director of fiscal studies, Centers on the Public Service, Schar School of Policy and

Government, George Mason University,

Email: s.redburn@icloud.com Phone: 240 643 1935

The Promise of Evidence-Based Policymaking 909
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: September 27, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8s53-6enq

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0004

Comment on FR Doc # 2016-22002

Submitter Information

Name: Jeremy Ayers

Address:

Washington, DC, 20036

Email: jeremy@results4america.org

General Comment

Response to Request for Comments for the Evidence-Based Policymaking Commission

Docket ID USBC-2016-0003

To Whom It May Concern:

Please find attached a response to your request for public comment for the EvidenceBased Policymaking Commission from Results from America and members of our

Invest in What Works coalition. These recommendations are in response to Questions 3,

4, 15, 16, and 19 found in the Notice Document soliciting public comment. Thank you in

advance for considering these recommendations.

Jeremy Ayers

Vice President, Policy

Results for America

910 The Promise of Evidence-Based Policymaking
Attachments

Coalition Letter 9.27.16

The Promise of Evidence-Based Policymaking 911
Commission on Evidence-Based Policymaking Comments: Docket ID USBC-2016-0003

1 

Response to Request for Comments for the Evidence-Based Policymaking 

Commission 

Docket ID USBC-2016-0003 

Name of Institution: Results for America and Invest in What Works Coalition Members 

Address of Institution: 1875 Connecticut Avenue, NW 

11th Floor 

Washington, DC 20036 

Contact Person: Jeremy Ayers 

 Vice President, Policy 

 Results for America 

 1875 Connecticut Ave, NW 

 11th Floor 

 Washington, DC 20036 

 jeremy@results4america.org 

 (202) 288-2186 

912 The Promise of Evidence-Based Policymaking
Commission on Evidence-Based Policymaking Comments: Docket ID USBC-2016-0003

2 

September 27, 2016 

Dr. Katharine G. Abraham, Chair 

Mr. Ron Haskins, Co-Chair 

Evidence-Based Policymaking Commission 

U.S. Census Bureau 

4600 Silver Hill Road 

Suitland, MD 20746

Dear Chairwoman Abraham and Co-Chairman Haskins, 

We are writing to encourage you to consider the attached policy recommendations as a response 

to your request for public comment. These recommendations address Questions 3, 4, 15, 16, and 

19 in the Notice Document requesting public comments. 

We believe that the Commission can help invest taxpayer dollars in what works by assisting 

policymakers at all levels of government in: 

x Building evidence about the practices, policies and programs that will achieve the most 

effective and efficient results so that policymakers can make better decisions; 

x Investing limited taxpayer dollars in practices, policies and programs that use data, evidence 

and evaluation to demonstrate they work; and 

x Directing funds away from practices, policies, and programs that consistently fail to achieve 

measurable outcomes. 

Although the Evidence-Based Policymaking Commission Act of 2016 directs the Commission to 

study and report on several important topics including data privacy and data sharing, our attached 

policy proposals focus on the provision that directs the Commission to “make recommendations on 

how best to incorporate outcomes measurement, institutionalize randomized controlled trials, and 

rigorous impact analysis into program design.” 

We thank you in advance for your consideration of our recommendations. 

Sincerely, 

Actionable Intelligence for Social Policy (AISP) 

America Forward 

Center for Employment Opportunities 

Center for Research and Reform in Education, Johns Hopkins University 

KIPP 

REDF 

Results for America 

Sorenson Center for Impact 

Success for All Foundation 

Sunlight Foundation 

cc: Members of the Evidence-Based Policymaking Commission 

The Promise of Evidence-Based Policymaking 913
Commission on Evidence-Based Policymaking Comments: Docket ID USBC-2016-0003

3 

INVEST IN WHAT WORKS COALITION RECOMMENDATIONS 

ADDRESSING QUESTIONS 3, 4, 15, 16, AND 19 

Data Collection

x Federal Data Infrastructure: The Commission should consider recommending that Congress 

and the Administration provide sufficient funding to help the U.S. Census Bureau accelerate 

the process of acquiring key administrative data-sets from local, state, and federal agencies, 

and strengthen its infrastructure for processing, standardizing, linking, and making data 

available to other government agencies and independent researchers via data use agreements 

with strong privacy protections. As part of this effort, the Census Bureau should develop an 

inventory of data-sets at the local, state, and federal levels and make this inventory accessible 

to government agencies and independent researchers. 

x Federal Data Inventories: The Commission should consider recommending that Congress 

and the Administration codify into law what is already required by the May 2013 Executive 

Order by passing the OPEN Government Data Act. This legislation would mandate that every 

federal agency create an enterprise data inventory of all data sets held by the agency and 

make these lists public in machine-readable formats with strong privacy protections. 

x Federal Data Information Technology: The Commission should consider recommending that 

Congress and the Administration provide sufficient funding to allow every federal agency to 

update and modernize its IT infrastructure that supports data collection, analysis, sharing, and 

usage so that data can be appropriately structured, protected, analyzed and disclosed in line 

with the updated information policy of the United States. A 2016 report by the U.S. General 

Accountability Office highlighted the urgent need for the U.S. government to modernize its 

aging legacy systems. 

x Workforce Data: The Commission should recommend that Congress and the Administration 

allow the linking of workforce datasets (including but not limited to state and federal 

unemployment insurance and new hires data sets) to improve the effectiveness and efficiency 

of publicly-supported workforce development programs, as long as the linking is consistent 

with strong privacy protections. For example, many states cannot determine the impact of their 

job training programs without the ability to link their participant information with information 

about wage earnings across multiple states where participants obtain employment. 

x State Education and Workforce Data Systems: The Commission should recommend that 

Congress and the Administration support the enhancement of the existing State Longitudinal 

Data Systems (SLDS) program administered by the U.S. Department of Education, which 

helps states integrate education and workforce data, and the proposed expansion of the 

Workforce Data Quality Initiative that would help build state and local capacity to track 

employment and educational outcomes of Workforce Innovation and Opportunity Act program 

participants, including those with disabilities, and provide information about job success rates 

and training programs. 

914 The Promise of Evidence-Based Policymaking
Commission on Evidence-Based Policymaking Comments: Docket ID USBC-2016-0003

4 

x Federal Education Data Identifiers: The Commission should consider recommending that 

Congress and the Administration direct federal agencies to standardize the way they collect 

and share student-level identifiers (e.g., de-identified but encrypted) so that researchers can 

more effectively evaluate publicly-supported education and workforce development programs. 

This information should be housed in one federal agency in order to promote appropriate 

sharing and usage of this standardized data. 

x Federal Programmatic Data: The Commission should consider recommending that Congress 

and the Administration authorize every federal agency to set aside 1% of their program funds 

for program evaluations that generate programmatic outcomes data that can help make federal 

programs more effective and efficient. 

Data Analysis

x Data Leadership and Infrastructure: The Commission should consider recommending that 

Congress and the Administration direct every federal agency to have a senior staff member 

(i.e., Chief Evaluation Officer or equivalent position) with the authority, staff, and budget to 

develop important programmatic data through the evaluation of its major programs and to use 

this programmatic data and available administrative data to inform the agency’s policies and 

improve its programs.

Data Sharing 

x Local and State Data Systems: The Commission should consider recommending that 

Congress and the Administration clarify that local and state agencies can invest federal 

program funds in strengthening their data infrastructures for processing, standardizing, linking, 

and making data available to other government agencies and independent researchers via 

data use agreements with strong privacy protections. 

x Federal Education Data Infrastructure: The Commission should consider recommending 

that Congress and the Administration strengthen the U.S. Department of Education’s (ED’s)

data infrastructure, including the hiring and training of key analytic staff, to manage the 

collection, quality, release, and analysis of education data with strong privacy protections and 

the support the proposed InformED initiative that would pull together ED’s diverse array of data 

and studies on a particular topic, and allow open data access to help unlock answers to 

pressing education questions and needs. 

Data Usage

x “What Works” Clearinghouses: The Commission should consider recommending that 

Congress and the Administration direct every federal agency to develop a “What Works” 

clearinghouse or evidence exchange with the purpose of making evaluation reports available 

to the public. 

x Performance Management/Continuous Improvement: The Commission should consider 

recommending that Congress and the Administration direct every federal agency to develop 

The Promise of Evidence-Based Policymaking 915
Commission on Evidence-Based Policymaking Comments: Docket ID USBC-2016-0003

5 

and operate a performance management system with clear and prioritized outcome-focused 

goals and aligned program objectives and that frequently collects, analyzes, and uses 

administrative and programmatic outcomes data to improve outcomes, return on investment, 

and other dimensions of performance. 

x Federal Grant Programs: The Commission should consider recommending that Congress 

and the Administration direct every federal agency to use evidence of effectiveness, including 

impact analysis and other outcomes measurements based on high-quality administrative and 

programmatic outcomes data, when allocating funds from its 5 largest competitive and noncompetitive grant programs.

x Evaluation and Research: The Commission should consider recommending that Congress 

and the Administration direct every federal agency to have an evaluation policy, evaluation 

plan, and research/learning agenda which ensures that the agency has an intentional 

approach to the collection, analysis, sharing, and usage of administrative and programmatic 

data and publicly release the findings of all completed evaluations to improve the effectiveness 

and efficiency of federal programs. 

x Repurpose for Results: The Commission should consider recommending that Congress and 

the Administration direct every federal agency to use its administrative and programmatic data 

to determine when to shift funds away from practices, policies, and programs which 

consistently fail to achieve desired outcomes and toward evidence-based, results-driven 

solutions. 

916 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: October 03, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8s99-hptf

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0005

Comment on FR Doc # 2016-22002

Submitter Information

Name: Karl Jaensch

General Comment

More detailed data about individuals (the sort of information that is vulnerable to privacy

violations) and efforts to attain 100 percent data collection does not lead to better

decision-making and is a waste of effort and money.

The Promise of Evidence-Based Policymaking 917
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: October 07, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sbr-jfhp

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0006

Comment on FR Doc # 2016-22002

Submitter Information

Name: Sander Greenland

General Comment

From: Sander Greenland

Sent: Thursday, October 6, 2016 10:56 PM

To: Nicholas Hart; A Reamer

Subject: Response to Request for Comments for the Commission on Evidence-Based

Policymaking

Dear Drs. Hart and Reamer,

In response to the request and the posting in ASA Connect, the attached articles (The

ASA Statement on statistical significance and P-values, and supplementary article on

misinterpretations) may be of interest, 

especially regarding question 16 (How can data, statistics, results of research, and

findings from evaluation, be best used to improve policies and programs?).

The material may be of particular import given the large number of articles in health and

medicine whose published conclusions are based on elementary misinterpretations of

their own statistical outputs.

918 The Promise of Evidence-Based Policymaking
The attachments are published with open access and may be distributed noncommercially.

Best Regards,

Sander Greenland

Attachments

Greenland Senn Rothman Carlin Poole Goodman Altman.Testing CI

misinterpretation.TAS 2016

Wasserstein Lazar.ASA Statement on P-values.7-2016

The Promise of Evidence-Based Policymaking 919
Statistical Tests, P-values, Confidence Intervals, and Power: A Guide

to Misinterpretations

Sander GREENLAND, Stephen J. SENN, Kenneth J. ROTHMAN, John B. CARLIN, Charles POOLE,

Steven N. GOODMAN, and Douglas G. ALTMAN

Misinterpretation and abuse of statistical tests, confidence intervals, and statistical power have been decried for decades, yet

remain rampant. A key problem is that there are no interpretations of these concepts that are at once simple, intuitive, correct, and foolproof. Instead, correct use and interpretation of

these statistics requires an attention to detail which seems to tax

the patience of working scientists. This high cognitive demand

has led to an epidemic of shortcut definitions and interpretations that are simply wrong, sometimes disastrously so—and

yet these misinterpretations dominate much of the scientific literature.

In light of this problem, we provide definitions and a discussion of basic statistics that are more general and critical than

typically found in traditional introductory expositions. Our goal

is to provide a resource for instructors, researchers, and consumers of statistics whose knowledge of statistical theory and

technique may be limited but who wish to avoid and spot misinterpretations. We emphasize how violation of often unstated

analysis protocols (such as selecting analyses for presentation

based on the P-values they produce) can lead to small P-values

even if the declared test hypothesis is correct, and can lead to

large P-values even if that hypothesis is incorrect. We then provide an explanatory list of 25 misinterpretations of P-values,

confidence intervals, and power. We conclude with guidelines

for improving statistical interpretation and reporting.

KEY WORDS: Confidence intervals; Hypothesis testing; Null

testing; P-value; Power; Significance tests; Statistical testing.

Online supplement to the ASA Statement on Statistical Significance and PValues, The American Statistician, 70. Sander Greenland (corresponding author), Department of Epidemiology and Department of Statistics, University

of California, Los Angeles, CA (Email: lesdomes@ucla.edu). Stephen J. Senn,

Competence Center for Methodology and Statistics, Luxembourg Institute of

Health, Luxembourg (Email: stephen.senn@lih.lu). Kenneth J. Rothman, RTI

Health Solutions, Research Triangle Institute, Research Triangle Park, NC.

John B. Carlin, Clinical Epidemiology and Biostatistics Unit, Murdoch Children’s Research Institute, School of Population Health, University of Melbourne, Victoria, Australia (Email: john.carlin@mcri.edu.au). Charles Poole,

Department of Epidemiology, Gillings School of Global Public Health, University of North Carolina, Chapel Hill, NC (Email: cpoole@unc.edu). Steven

N. Goodman, Meta-Research Innovation Center, Departments of Medicine and

of Health Research and Policy, Stanford University School of Medicine, Stanford, CA (Email: steve.goodman@stanford.edu). Douglas G. Altman, Centre

for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology

& Musculoskeletal Sciences, University of Oxford, Oxford, United Kingdom

(Email: doug.altman@csm.ox.ac.uk). SJS receives funding from the IDEAL

project supported by the European Union’s Seventh Framework Programme

for research, technological development and demonstration under Grant Agreement no 602552. We thank Stuart Hurlbert, Deborah Mayo, Keith O’Rourke,

and Andreas Stang for helpful comments, and Ron Wasserstein for his invaluable encouragement on this project.

Introduction

Misinterpretation and abuse of statistical tests has been decried for decades, yet remains so rampant that some scientific

journals discourage use of “statistical significance” (classifying results as “significant” or not based on a P-value) (Lang et

al. 1998). One journal now bans all statistical tests and mathematically related procedures such as confidence intervals (Trafimow and Marks 2015), which has led to considerable discussion

and debate about the merits of such bans (e.g., Ashworth 2015;

Flanagan 2015).

Despite such bans, we expect that the statistical methods at

issue will be with us for many years to come. We thus think it

imperative that basic teaching as well as general understanding

of these methods be improved. Toward that end, we attempt to

explain the meaning of significance tests, confidence intervals,

and statistical power in a more general and critical way than

is traditionally done, and then review 25 common misconceptions in light of our explanations. We also discuss a few more

subtle but nonetheless pervasive problems, explaining why it

is important to examine and synthesize all results relating to

a scientific question, rather than focus on individual findings.

We further explain why statistical tests should never constitute

the sole input to inferences or decisions about associations or

effects. Among the many reasons are that, in most scientific settings, the arbitrary classification of results into “significant” and

“nonsignificant” is unnecessary for and often damaging to valid

interpretation of data; and that estimation of the size of effects

and the uncertainty surrounding our estimates will be far more

important for scientific inference and sound judgment than any

such classification.

More detailed discussion of the general issues can be found

in many articles, chapters, and books on statistical methods and

their interpretation (e.g., Altman et al. 2000; Atkins and Jarrett

1979; Cox 1977, 1982; Cox and Hinkley 1974; Freedman et al.

2007; Gibbons and Pratt 1975; Gigerenzer et al. 1990, Ch. 3;

Harlow et al. 1997; Hogben 1957; Kaye and Freedman 2011;

Morrison and Henkel 1970; Oakes 1986; Pratt 1965; Rothman

et al. 2008, Ch. 10; Ware et al. 2009; Ziliak and McCloskey

2008). Specific issues are covered at length in these sources and

in the many peer-reviewed articles that critique common misinterpretations of null-hypothesis testing and “statistical significance” (e.g., Altman and Bland 1995; Anscombe 1990; Bakan

1966; Bandt and Boen 1972; Berkson 1942; Bland and Altman

2015; Chia 1997; Cohen 1994; Evans et al. 1988; Fidler and

Loftus 2009; Gardner and Altman 1986; Gelman 2013; Gelman

and Loken 2014; Gelman and Stern 2006; Gigerenzer 2004;

c 2016 S. Greenland, S. Senn, K. Rothman, J. Carlin, C. Poole, S. Goodman, and D. Altman The American Statistician, Online Supplement 1

920 The Promise of Evidence-Based Policymaking
Gigerenzer and Marewski 2015; Goodman 1992, 1993, 1999,

2008; Greenland 2011, 2012ab; Greenland and Poole, 2011,

2013ab; Grieve 2015; Harlow et al. 1997; Hoekstra et al. 2006;

Hurlbert and Lombardi 2009; Kaye 1986; Lambdin 2012; Lang

et al. 1998; Langman 1986; LeCoutre et al. 2003; Lew 2012;

Loftus 1996; Matthews and Altman 1996a; Pocock and Ware

2009; Pocock et al. 1987; Poole 1987ab, 2001; Rosnow and

Rosenthal 1989; Rothman 1978, 1986; Rozeboom 1960; Salsburg 1985; Schmidt 1996; Schmidt and Hunter 2002; Sterne and

Davey Smith 2001; Thompson 1987; Thompson 2004; Wagenmakers 2007; Walker 1986; Wood et al. 2014).

Statistical Tests, P-values, and Confidence Intervals: A

Caustic Primer

Statistical Models, Hypotheses, and Tests

Every method of statistical inference depends on a complex

web of assumptions about how data were collected and analyzed, and how the analysis results were selected for presentation. The full set of assumptions is embodied in a statistical

model that underpins the method. This model is a mathematical

representation of data variability, and thus ideally would capture

accurately all sources of such variability. Many problems arise,

however, because this statistical model often incorporates unrealistic or at best unjustified assumptions. This is true even for

so-called “nonparametric” methods, which (like other methods)

depend on assumptions of random sampling or randomization.

These assumptions are often deceptively simple to write mathematically, yet in practice are difficult to satisfy and verify, as

they may depend on successful completion of a long sequence

of actions (such as identifying, contacting, obtaining consent

from, obtaining cooperation of, and following up subjects, as

well as adherence to study protocols for treatment allocation,

masking, and data analysis).

There is also a serious problem of defining the scope of a

model, in that it should allow not only for a good representation of the observed data but also of hypothetical alternative

data that might have been observed. The reference frame for

data that “might have been observed” is often unclear, for example if multiple outcome measures or multiple predictive factors

have been measured, and many decisions surrounding analysis

choices have been made after the data were collected—as is invariably the case (Gelman and Loken 2014).

The difficulty of understanding and assessing underlying assumptions is exacerbated by the fact that the statistical model is

usually presented in a highly compressed and abstract form—if

presented at all. As a result, many assumptions go unremarked

and are often unrecognized by users as well as consumers of

statistics. Nonetheless, all statistical methods and interpretations are premised on the model assumptions; that is, on an assumption that the model provides a valid representation of the

variation we would expect to see across data sets, faithfully reflecting the circumstances surrounding the study and phenomena occurring within it.

In most applications of statistical testing, one assumption in

the model is a hypothesis that a particular effect has a specific

size, and has been targeted for statistical analysis. (For simplicity, we use the word “effect” when “association or effect”

would arguably be better in allowing for noncausal studies such

as most surveys.) This targeted assumption is called the study

hypothesis or test hypothesis, and the statistical methods used

to evaluate it are called statistical hypothesis tests. Most often,

the targeted effect size is a “null” value representing zero effect

(e.g., that the study treatment makes no difference in average

outcome), in which case the test hypothesis is called the null

hypothesis. Nonetheless, it is also possible to test other effect

sizes. We may also test hypotheses that the effect does or does

not fall within a specific range; for example, we may test the hypothesis that the effect is no greater than a particular amount, in

which case the hypothesis is said to be a one-sided or dividing

hypothesis (Cox 1977, 1982).

Much statistical teaching and practice has developed a strong

(and unhealthy) focus on the idea that the main aim of a study

should be to test null hypotheses. In fact most descriptions of

statistical testing focus only on testing null hypotheses, and

the entire topic has been called “Null Hypothesis Significance

Testing” (NHST). This exclusive focus on null hypotheses contributes to misunderstanding of tests. Adding to the misunderstanding is that many authors (including R.A. Fisher) use “null

hypothesis” to refer to any test hypothesis, even though this usage is at odds with other authors and with ordinary English definitions of “null”—as are statistical usages of “significance” and

“confidence.”

Uncertainty, Probability, and Statistical Significance

A more refined goal of statistical analysis is to provide an

evaluation of certainty or uncertainty regarding the size of an effect. It is natural to express such certainty in terms of “probabilities” of hypotheses. In conventional statistical methods, however, “probability” refers not to hypotheses, but to quantities

that are hypothetical frequencies of data patterns under an assumed statistical model. These methods are thus called frequentist methods, and the hypothetical frequencies they predict are

called “frequency probabilities.” Despite considerable training

to the contrary, many statistically educated scientists revert to

the habit of misinterpreting these frequency probabilities as hypothesis probabilities. (Even more confusingly, the term “likelihood of a parameter value” is reserved by statisticians to refer to

the probability of the observed data given the parameter value;

it does not refer to a probability of the parameter taking on the

given value.)

Nowhere are these problems more rampant than in applications of a hypothetical frequency called the P-value, also known

as the “observed significance level” for the test hypothesis. Statistical “significance tests” based on this concept have been a

central part of statistical analyses for centuries (Stigler 1986).

The focus of traditional definitions of P-values and statistical

significance has been on null hypotheses, treating all other assumptions used to compute the P-value as if they were known

to be correct. Recognizing that these other assumptions are often questionable if not unwarranted, we will adopt a more general view of the P-value as a statistical summary of the compatibility between the observed data and what we would predict

or expect to see if we knew the entire statistical model (all the

2 Online Supplement to the ASA Statement on Statistical Significance and P-values

The Promise of Evidence-Based Policymaking 921
assumptions used to compute the P-value) were correct.

Specifically, the distance between the data and the model prediction is measured using a test statistic (such as a t-statistic

or a chi-squared statistic). The P-value is then the probability

that the chosen test statistic would have been at least as large

as its observed value if every model assumption were correct,

including the test hypothesis. This definition embodies a crucial point lost in traditional definitions: In logical terms, the Pvalue tests all the assumptions about how the data were generated (the entire model), not just the targeted hypothesis it is

supposed to test (such as a null hypothesis). Furthermore, these

assumptions include far more than what are traditionally presented as modeling or probability assumptions—they include

assumptions about the conduct of the analysis, for example that

intermediate analysis results were not used to determine which

analyses would be presented.

It is true that the smaller the P-value, the more unusual the

data would be if every single assumption were correct; but a

very small P-value does not tell us which assumption is incorrect. For example, the P-value may be very small because the

targeted hypothesis is false; but it may instead (or in addition)

be very small because the study protocols were violated, or because it was selected for presentation based on its small size.

Conversely, a large P-value indicates only that the data are not

unusual under the model, but does not imply that the model or

any aspect of it (such as the targeted hypothesis) is correct; it

may instead (or in addition) be large because (again) the study

protocols were violated, or because it was selected for presentation based on its large size.

The general definition of a P-value may help one to understand why statistical tests tell us much less than what many think

they do: Not only does a P-value not tell us whether the hypothesis targeted for testing is true or not; it says nothing specifically related to that hypothesis unless we can be completely

assured that every other assumption used for its computation is

correct—an assurance that is lacking in far too many studies.

Nonetheless, the P-value can be viewed as a continuous measure of the compatibility between the data and the entire model

used to compute it, ranging from 0 for complete incompatibility

to 1 for perfect compatibility, and in this sense may be viewed as

measuring the fit of the model to the data. Too often, however,

the P-value is degraded into a dichotomy in which results are

declared “statistically significant” if P falls on or below a cutoff (usually 0.05) and declared “nonsignificant” otherwise. The

terms “significance level” and “alpha level” (α) are often used

to refer to the cut-off; however, the term “significance level” invites confusion of the cut-off with the P-value itself. Their difference is profound: the cut-off value α is supposed to be fixed

in advance and is thus part of the study design, unchanged in

light of the data. In contrast, the P-value is a number computed

from the data and thus an analysis result, unknown until it is

computed.

Moving From Tests to Estimates

We can vary the test hypothesis while leaving other assumptions unchanged, to see how the P-value differs across competing test hypotheses. Usually, these test hypotheses specify different sizes for a targeted effect; for example, we may test the

hypothesis that the average difference between two treatment

groups is zero (the null hypothesis), or that it is 20 or –10 or

any size of interest. The effect size whose test produced P = 1

is the size most compatible with the data (in the sense of predicting what was in fact observed) if all the other assumptions

used in the test (the statistical model) were correct, and provides

a point estimate of the effect under those assumptions. The effect sizes whose test produced P > 0.05 will typically define a

range of sizes (e.g., from 11.0 to 19.5) that would be considered

more compatible with the data (in the sense of the observations

being closer to what the model predicted) than sizes outside the

range—again, if the statistical model were correct. This range

corresponds to a 1 − 0.05 = 0.95 or 95% confidence interval,

and provides a convenient way of summarizing the results of

hypothesis tests for many effect sizes. Confidence intervals are

examples of interval estimates.

Neyman (1937) proposed the construction of confidence intervals in this way because they have the following property:

If one calculates, say, 95% confidence intervals repeatedly in

valid applications, 95% of them, on average, will contain (i.e.,

include or cover) the true effect size. Hence, the specified confidence level is called the coverage probability. As Neyman

stressed repeatedly, this coverage probability is a property of

a long sequence of confidence intervals computed from valid

models, rather than a property of any single confidence interval.

Many journals now require confidence intervals, but most

textbooks and studies discuss P-values only for the null hypothesis of no effect. This exclusive focus on null hypotheses

in testing not only contributes to misunderstanding of tests and

underappreciation of estimation, but also obscures the close relationship between P-values and confidence intervals, as well

as the weaknesses they share.

What P-values, Confidence Intervals, and Power

Calculations Don’t Tell Us

Much distortion arises from basic misunderstanding of what

P-values and their relatives (such as confidence intervals) do

not tell us. Therefore, based on the articles in our reference

list, we review prevalent P-value misinterpretations as a way

of moving toward defensible interpretations and presentations.

We adopt the format of Goodman (2008) in providing a list of

misinterpretations that can be used to critically evaluate conclusions offered by research reports and reviews. Every one of

the italicized statements in our list has contributed to statistical

distortion of the scientific literature, and we add the emphatic

“No!” to underscore statements that are not only fallacious but

also not “true enough for practical purposes.”

Common Misinterpretations of Single P-values

1. The P-value is the probability that the test hypothesis is

true; for example, if a test of the null hypothesis gave P = 0.01,

the null hypothesis has only a 1% chance of being true; if instead it gave P = 0.40, the null hypothesis has a 40% chance

of being true.—No! The P-value assumes the test hypothesis is

true—it is not a hypothesis probability and may be far from any

The American Statistician, Online Supplement 3

922 The Promise of Evidence-Based Policymaking
reasonable probability for the test hypothesis. The P-value simply indicates the degree to which the data conform to the pattern predicted by the test hypothesis and all the other assumptions used in the test (the underlying statistical model). Thus

P = 0.01 would indicate that the data are not very close to what

the statistical model (including the test hypothesis) predicted

they should be, while P = 0.40 would indicate that the data

are much closer to the model prediction, allowing for chance

variation.

2. The P-value for the null hypothesis is the probability that

chance alone produced the observed association; for example,

if the P-value for the null hypothesis is 0.08, there is an 8%

probability that chance alone produced the association.—No!

This is a common variation of the first fallacy and it is just as

false. To say that chance alone produced the observed association is logically equivalent to asserting that every assumption

used to compute the P-value is correct, including the null hypothesis. Thus to claim that the null P-value is the probability

that chance alone produced the observed association is completely backwards: The P-value is a probability computed assuming chance was operating alone. The absurdity of the common backwards interpretation might be appreciated by pondering how the P-value, which is a probability deduced from a set

of assumptions (the statistical model), can possibly refer to the

probability of those assumptions.

Note: One often sees “alone” dropped from this description

(becoming “the P-value for the null hypothesis is the probability that chance produced the observed association”), so that the

statement is more ambiguous, but just as wrong.

3. A significant test result (P ≤ 0.05) means that the test hypothesis is false or should be rejected.—No! A small P-value

simply flags the data as being unusual if all the assumptions

used to compute it (including the test hypothesis) were correct;

it may be small because there was a large random error or because some assumption other than the test hypothesis was violated (for example, the assumption that this P-value was not

selected for presentation because it was below 0.05). P ≤ 0.05

only means that a discrepancy from the hypothesis prediction

(e.g., no difference between treatment groups) would be as large

or larger than that observed no more than 5% of the time if only

chance were creating the discrepancy (as opposed to a violation

of the test hypothesis or a mistaken assumption).

4. A nonsignificant test result (P > 0.05) means that the test

hypothesis is true or should be accepted.—No! A large P-value

only suggests that the data are not unusual if all the assumptions

used to compute the P-value (including the test hypothesis)

were correct. The same data would also not be unusual under

many other hypotheses. Furthermore, even if the test hypothesis

is wrong, the P-value may be large because it was inflated by a

large random error or because of some other erroneous assumption (e.g., the assumption that this P-value was not selected for

presentation because it was above 0.05). P > 0.05 only means

that a discrepancy from the hypothesis prediction (e.g., no difference between treatment groups) would be as large or larger

than that observed more than 5% of the time if only chance were

creating the discrepancy.

5. A large P-value is evidence in favor of the test

hypothesis.—No! In fact, any P-value less than 1 implies that

the test hypothesis is not the hypothesis most compatible with

the data, because any other hypothesis with a larger P-value

would be even more compatible with the data. A P-value cannot be said to favor the test hypothesis except in relation to

those hypotheses with smaller P-values. Furthermore, a large

P-value often indicates only that the data are incapable of discriminating among many competing hypotheses (as would be

seen immediately by examining the range of the confidence interval). For example, many authors will misinterpret P = 0.70

from a test of the null hypothesis as evidence for no effect, when

in fact it indicates that, even though the null hypothesis is compatible with the data under the assumptions used to compute

the P-value, it is not the hypothesis most compatible with the

data—that honor would belong to a hypothesis with P = 1.

But even if P = 1, there will be many other hypotheses that are

highly consistent with the data, so that a definitive conclusion of

“no association” cannot be deduced from a P-value, no matter

how large.

6. A null-hypothesis P-value greater than 0.05 means that

no effect was observed, or that absence of an effect was shown

or demonstrated.—No! Observing P > 0.05 for the null hypothesis only means that the null is one among the many hypotheses that have P > 0.05. Thus, unless the point estimate

(observed association) equals the null value exactly, it is a mistake to conclude from P > 0.05 that a study found “no association” or “no evidence” of an effect. If the null P-value is less

than 1 some association must be present in the data, and one

must look at the point estimate to determine the effect size most

compatible with the data under the assumed model.

7. Statistical significance indicates a scientifically or substantively important relation has been detected.—No! Especially when a study is large, very minor effects or small assumption violations can lead to statistically significant tests of

the null hypothesis. Again, a small null P-value simply flags

the data as being unusual if all the assumptions used to compute it (including the null hypothesis) were correct; but the way

the data are unusual might be of no clinical interest. One must

look at the confidence interval to determine which effect sizes

of scientific or other substantive (e.g., clinical) importance are

relatively compatible with the data, given the model.

8. Lack of statistical significance indicates that the effect size

is small.—No! Especially when a study is small, even large effects may be “drowned in noise” and thus fail to be detected as

statistically significant by a statistical test. A large null P-value

simply flags the data as not being unusual if all the assumptions

used to compute it (including the test hypothesis) were correct;

but the same data will also not be unusual under many other

models and hypotheses besides the null. Again, one must look

at the confidence interval to determine whether it includes effect

sizes of importance.

4 Online Supplement to the ASA Statement on Statistical Significance and P-values

The Promise of Evidence-Based Policymaking 923
9. The P-value is the chance of our data occurring if the

test hypothesis is true; for example, P = 0.05 means that the

observed association would occur only 5% of the time under

the test hypothesis.—No! The P-value refers not only to what

we observed, but also observations more extreme than what we

observed (where “extremity” is measured in a particular way).

And again, the P-value refers to a data frequency when all the

assumptions used to compute it are correct. In addition to the

test hypothesis, these assumptions include randomness in sampling, treatment assignment, loss, and missingness, as well as

an assumption that the P-value was not selected for presentation based on its size or some other aspect of the results.

10. If you reject the test hypothesis because P ≤ 0.05, the

chance you are in error (the chance your “significant finding”

is a false positive) is 5%.—No! To see why this description is

false, suppose the test hypothesis is in fact true. Then, if you

reject it, the chance you are in error is 100%, not 5%. The 5%

refers only to how often you would reject it, and therefore be

in error, over very many uses of the test across different studies

when the test hypothesis and all other assumptions used for the

test are true. It does not refer to your single use of the test, which

may have been thrown off by assumption violations as well as

random errors. This is yet another version of misinterpretation

#1.

11. P = 0.05 and P ≤ 0.05 mean the same thing.—No!

This is like saying reported height = 2 meters and reported

height ≤ 2 meters are the same thing: “height = 2 meters”

would include few people and those people would be considered

tall, whereas “height ≤ 2 meters” would include most people

including small children. Similarly, P = 0.05 would be considered a borderline result in terms of statistical significance,

whereas P ≤ 0.05 lumps borderline results together with results very incompatible with the model (e.g., P = 0.0001) thus

rendering its meaning vague, for no good purpose.

12. P-values are properly reported as inequalities (e.g., report “P < 0.02” when P = 0.015 or report P > 0.05 when

P = 0.06 or P = 0.70).—No! This is bad practice because

it makes it difficult or impossible for the reader to accurately

interpret the statistical result. Only when the P-value is very

small (e.g., under 0.001) does an inequality become justifiable:

There is little practical difference among very small P-values

when the assumptions used to compute P-values are not known

with enough certainty to justify such precision, and most methods for computing P-values are not numerically accurate below

a certain point.

13. Statistical significance is a property of the phenomenon

being studied, and thus statistical tests detect significance.—

No! This misinterpretation is promoted when researchers state

that they have or have not found “evidence of” a statistically

significant effect. The effect being tested either exists or does

not exist. “Statistical significance” is a dichotomous description

of a P-value (that it is below the chosen cut-off) and thus is a

property of a result of a statistical test; it is not a property of the

effect or population being studied.

14. One should always use two-sided P-values.—No! Twosided P-values are designed to test hypotheses that the targeted

effect measure equals a specific value (e.g., zero), and is neither

above nor below this value. When however the test hypothesis

of scientific or practical interest is a one-sided (dividing) hypothesis, a one-sided P-value is appropriate. For example, consider the practical question of whether a new drug is at least

as good as the standard drug for increasing survival time. This

question is one-sided, so testing this hypothesis calls for a onesided P-value. Nonetheless, because two-sided P-values are the

usual default, it will be important to note when and why a onesided P-value is being used instead.

There are other interpretations of P values that are controversial, in that whether a categorical “No!” is warranted depends

on one’s philosophy of statistics and the precise meaning given

to the terms involved. The disputed claims deserve recognition

if one wishes to avoid such controversy.

For example, it has been argued that P-values overstate

evidence against test hypotheses, based on directly comparing P-values against certain quantities (likelihood ratios and

Bayes factors) that play a central role as evidence measures

in Bayesian analysis (Edwards et al. 1963; Berger and Sellke

1987; Edwards 1992; Goodman and Royall 1988; Royall 1997;

Sellke et al. 2001; Goodman 1992, 2005; Wagenmakers 2007).

Nonetheless, many other statisticians do not accept these quantities as gold standards, and instead point out that P-values summarize crucial evidence needed to gauge the error rates of decisions based on statistical tests (even though they are far from

sufficient for making those decisions). Thus, from this frequentist perspective, P-values do not overstate evidence and may

even be considered as measuring one aspect of evidence (Cox

1977, 1982; Lehmann 1986; Senn 2001, 2002a; Mayo and Cox

2006), with 1 − P measuring evidence against the model used

to compute the P-value. See also Murtaugh (2014) and its accompanying discussion.

Common Misinterpretations of P-Value Comparisons and Predictions

Some of the most severe distortions of the scientific literature produced by statistical testing involve erroneous comparison and synthesis of results from different studies or study subgroups. Among the worst are:

15. When the same hypothesis is tested in different studies

and none or a minority of the tests are statistically significant

(all P > 0.05), the overall evidence supports the hypothesis.—

No! This belief is often used to claim that a literature supports

no effect when the opposite is case. It reflects a tendency of researchers to “overestimate the power of most research” (Hedges

and Olkin 1980). In reality, every study could fail to reach statistical significance and yet when combined show a statistically

significant association and persuasive evidence of an effect. For

example, if there were five studies each with P = 0.10, none

would be significant at 0.05 level; but when these P-values are

combined using the Fisher formula (Cox and Hinkley 1974, p.

80), the overall P-value would be 0.01. There are many real exThe American Statistician, Online Supplement 5

924 The Promise of Evidence-Based Policymaking
amples of persuasive evidence for important effects when few

studies or even no study reported “statistically significant” associations (e.g., Chalmers and Lau 1996; Maheshwari et al. 2007).

Thus, lack of statistical significance of individual studies should

not be taken as implying that the totality of evidence supports

no effect.

16. When the same hypothesis is tested in two different populations and the resulting P-values are on opposite sides of 0.05,

the results are conflicting.—No! Statistical tests are sensitive to

many differences between study populations that are irrelevant

to whether their results are in agreement, such as the sizes of

compared groups in each population. As a consequence, two

studies may provide very different P-values for the same test

hypothesis and yet be in perfect agreement (e.g., may show

identical observed associations). For example, suppose we had

two randomized trials A and B of a treatment, identical except

that trial A had a known standard error of 2 for the mean difference between treatment groups whereas trial B had a known

standard error of 1 for the difference. If both trials observed a

difference between treatment groups of exactly 3, the usual normal test would produce P = 0.13 in A but P = 0.003 in B.

Despite their difference in P-values, the test of the hypothesis

of no difference in effect across studies would have P = 1, reflecting the perfect agreement of the observed mean differences

from the studies. Differences between results must be evaluated

by directly, for example by estimating and testing those differences to produce a confidence interval and a P-value comparing

the results (often called analysis of heterogeneity, interaction, or

modification).

17. When the same hypothesis is tested in two different populations and the same P-values are obtained, the results are

in agreement.—No! Again, tests are sensitive to many differences between populations that are irrelevant to whether their

results are in agreement. Two different studies may even exhibit

identical P-values for testing the same hypothesis yet also exhibit clearly different observed associations. For example, suppose randomized experiment A observed a mean difference between treatment groups of 3.00 with standard error 1.00, while

B observed a mean difference of 12.00 with standard error 4.00.

Then the standard normal test would produce P = 0.003 in

both; yet the test of the hypothesis of no difference in effect

across studies gives P = 0.03, reflecting the large difference

(12.00 − 3.00 = 9.00) between the mean differences.

18. If one observes a small P-value, there is a good chance

that the next study will produce a P-value at least as small for

the same hypothesis.—No! This is false even under the ideal

condition that both studies are independent and all assumptions

including the test hypothesis are correct in both studies. In that

case, if (say) one observes P = 0.03, the chance that the new

study will show P ≤ 0.03 is only 3%; thus the chance the

new study will show a P-value as small or smaller (the “replication probability”) is exactly the observed P-value! If on the

other hand the small P-value arose solely because the true effect exactly equaled its observed estimate, there would be a 50%

chance that a repeat experiment of identical design would have

a larger P-value (Goodman 1992). In general, the size of the

new P-value will be extremely sensitive to the study size and

the extent to which the test hypothesis or other assumptions are

violated in the new study (Senn 2002a); in particular, P may be

very small or very large depending on whether the study and the

violations are large or small.

Finally, although it is (we hope obviously) wrong to do so,

one sometimes sees the null hypothesis compared with another

(alternative) hypothesis using a two-sided P-value for the null

and a one-sided P-value for the alternative. This comparison is

biased in favor of the null in that the two-sided test will falsely

reject the null only half as often as the one-sided test will falsely

reject the alternative (again, under all the assumptions used for

testing).

Common Misinterpretations of Confidence Intervals

Most of the above misinterpretations translate into an analogous misinterpretation for confidence intervals. For example,

another misinterpretation of P > 0.05 is that it means the test

hypothesis has only a 5% chance of being false, which in terms

of a confidence interval becomes the common fallacy:

19. The specific 95% confidence interval presented by a

study has a 95% chance of containing the true effect size.—No!

A reported confidence interval is a range between two numbers.

The frequency with which an observed interval (e.g., 0.72 to

2.88) contains the true effect is either 100% if the true effect

is within the interval or 0% if not; the 95% refers only to how

often 95% confidence intervals computed from very many studies would contain the true size if all the assumptions used to

compute the intervals were correct. It is possible to compute

an interval that can be interpreted as having 95% probability of

containing the true value; nonetheless, such computations require not only the assumptions used to compute the confidence

interval, but also further assumptions about the size of effects in

the model. These further assumptions are summarized in what is

called a prior distribution, and the resulting intervals are usually

called Bayesian posterior (or credible) intervals to distinguish

them from confidence intervals (e.g., see Rothman et al. 2008,

Ch. 13 and 18).

Symmetrically, the misinterpretation of a small P-value as

disproving the test hypothesis could be translated into:

20. An effect size outside the 95% confidence interval has

been refuted (or excluded) by the data.—No! As with the Pvalue, the confidence interval is computed from many assumptions, the violation of which may have led to the results. Thus it

is the combination of the data with the assumptions, along with

the arbitrary 95% criterion, that are needed to declare an effect size outside the interval is in some way incompatible with

the observations. Even then, judgements as extreme as saying

the effect size has been refuted or excluded will require even

stronger conditions.

As with P-values, nave comparison of confidence intervals

6 Online Supplement to the ASA Statement on Statistical Significance and P-values

The Promise of Evidence-Based Policymaking 925
can be highly misleading:

21. If two confidence intervals overlap, the difference between two estimates or studies is not significant.—No! The 95%

confidence intervals from two subgroups or studies may overlap substantially and yet the test for difference between them

may still produce P < 0.05. Suppose for example, two 95%

confidence intervals for means from normal populations with

known variances are (1.04, 4.96) and (4.16, 19.84); these intervals overlap, yet the test of the hypothesis of no difference in

effect across studies gives P = 0.03. As with P-values, comparison between groups requires statistics that directly test and

estimate the differences across groups. It can, however, be noted

that if the two 95% confidence intervals fail to overlap, then

when using the same assumptions used to compute the confidence intervals we will find P < 0.05 for the difference; and

if one of the 95% intervals contains the point estimate from the

other group or study, we will find P > 0.05 for the difference.

Finally, as with P-values, the replication properties of confidence intervals are usually misunderstood:

22. An observed 95% confidence interval predicts that 95%

of the estimates from future studies will fall inside the observed

interval.—No! This statement is wrong in several ways. Most

importantly, under the model, 95% is the frequency with which

other unobserved intervals will contain the true effect, not how

frequently the one interval being presented will contain future

estimates. In fact, even under ideal conditions the chance that

a future estimate will fall within the current interval will usually be much less than 95%. For example, if two independent

studies of the same quantity provide unbiased normal point estimates with the same standard errors, the chance that the 95%

confidence interval for the first study contains the point estimate

from the second is 83% (which is the chance that the difference

between the two estimates is less than 1.96 standard errors).

Again, an observed interval either does or does not contain the

true effect; the 95% refers only to how often 95% confidence

intervals computed from very many studies would contain the

true effect if all the assumptions used to compute the intervals

were correct.

23. If one 95% confidence interval includes the null value

and another excludes that value, the interval excluding the null

is the more precise one.—No! When the model is correct, precision of statistical estimation is measured directly by confidence interval width (measured on the appropriate scale). It is

not a matter of inclusion or exclusion of the null or any other

value. Consider two 95% confidence intervals for a difference

in means, one with limits of 5 and 40, the other with limits of

−5 and 10. The first interval excludes the null value of 0, but is

30 units wide. The second includes the null value, but is half as

wide and therefore much more precise.

In addition to the above misinterpretations, 95% confidence

intervals force the 0.05-level cutoff on the reader, lumping together all effect sizes with P > 0.05, and in this way are as

bad as presenting P-values as dichotomies. Nonetheless, many

authors agree that confidence intervals are superior to tests and

P-values because they allow one to shift focus away from the

null hypothesis, toward the full range of effect sizes compatible with the data—a shift recommended by many authors and

a growing number of journals. Another way to bring attention

to nonnull hypotheses is to present their P-values; for example,

one could provide or demand P-values for those effect sizes

that are recognized as scientifically reasonable alternatives to

the null.

As with P-values, further cautions are needed to avoid misinterpreting confidence intervals as providing sharp answers when

none are warranted. The hypothesis which says the point estimate is the correct effect will have the largest P-value (P = 1

in most cases), and hypotheses inside a confidence interval will

have higher P-values than hypotheses outside the interval. The

P-values will vary greatly, however, among hypotheses inside

the interval, as well as among hypotheses on the outside. Also,

two hypotheses may have nearly equal P-values even though

one of the hypotheses is inside the interval and the other is outside. Thus, if we use P-values to measure compatibility of hypotheses with data and wish to compare hypotheses with this

measure, we need to examine their P-values directly, not simply ask whether the hypotheses are inside or outside the interval. This need is particularly acute when (as usual) one of the

hypotheses under scrutiny is a null hypothesis.

Common Misinterpretations of Power

The power of a test to detect a correct alternative hypothesis

is the pre-study probability that the test will reject the test hypothesis (e.g., the probability that P will not exceed a prespecified cut-off such as 0.05). (The corresponding prestudy probability of failing to reject the test hypothesis when the alternative

is correct is one minus the power, also known as the Type-II

or beta error rate; see Lehmann 1986.) As with P-values and

confidence intervals, this probability is defined over repetitions

of the same study design and so is a frequency probability. One

source of reasonable alternative hypotheses are the effect sizes

that were used to compute power in the study proposal. Prestudy power calculations do not, however, measure the compatibility of these alternatives with the data actually observed,

while power calculated from the observed data is a direct (if

obscure) transformation of the null P-value and so provides no

test of the alternatives. Thus, presentation of power does not

obviate the need to provide interval estimates and direct tests of

the alternatives.

For these reasons, many authors have condemned use of

power to interpret estimates and statistical tests (e.g., Cox 1958;

Smith and Bates 1992; Goodman 1994; Goodman and Berlin

1994; Hoenig and Heisey 2001; Senn 2002b; Greenland 2012a),

arguing that (in contrast to confidence intervals) it distracts attention from direct comparisons of hypotheses and introduces

new misinterpretations, such as:

24. If you accept the null hypothesis because the null Pvalue exceeds 0.05 and the power of your test is 90%, the chance

you are in error (the chance that your finding is a false negative) is 10%.—No! If the null hypothesis is false and you accept

The American Statistician, Online Supplement 7

926 The Promise of Evidence-Based Policymaking
it, the chance you are in error is 100%, not 10%. Conversely, if

the null hypothesis is true and you accept it, the chance you are

in error is 0%. The 10% refers only to how often you would be

in error over very many uses of the test across different studies

when the particular alternative used to compute power is correct

and all other assumptions used for the test are correct in all the

studies. It does not refer to your single use of the test or your error rate under any alternative effect size other than the one used

to compute power.

It can be especially misleading to compare results for two

hypotheses by presenting a test or P-value for one and power

for the other. For example, testing the null by seeing whether

P ≤ 0.05 with a power less than 1−0.05 = 0.95 for the alternative (as done routinely) will bias the comparison in favor of the

null because it entails a lower probability of incorrectly rejecting the null (0.05) than of incorrectly accepting the null when

the alternative is correct. Thus, claims about relative support or

evidence need to be based on direct and comparable measures

of support or evidence for both hypotheses, otherwise mistakes

like the following will occur:

25. If the null P-value exceeds 0.05 and the power of this

test is 90% at an alternative, the results support the null over

the alternative.—This claim seems intuitive to many, but counterexamples are easy to construct in which the null P-value is

between 0.05 and 0.10, and yet there are alternatives whose own

P-value exceeds 0.10 and for which the power is 0.90. Parallel

results ensue for other accepted measures of compatibility, evidence, and support, indicating that the data show lower compatibility with and more evidence against the null than the alternative, despite the fact that the null P-value is “not significant”

at the 0.05 alpha level and the power against the alternative is

“very high” (Greenland, 2012a).

Despite its shortcomings for interpreting current data, power

can be useful for designing studies and for understanding why

replication of “statistical significance” will often fail even under

ideal conditions. Studies are often designed or claimed to have

80% power against a key alternative when using a 0.05 significance level, although in execution often have less power due to

unanticipated problems such as low subject recruitment. Thus,

if the alternative is correct and the actual power of two studies

is 80%, the chance that the studies will both show P ≤ 0.05

will at best be only 0.80(0.80) = 64%; furthermore, the chance

that one study shows P ≤ 0.05 and the other does not (and

thus will be misinterpreted as showing conflicting results) is

2(0.80)0.20 = 32% or about 1 chance in 3. Similar calculations

taking account of typical problems suggest that one could anticipate a “replication crisis” even if there were no publication or

reporting bias, simply because current design and testing conventions treat individual study results as dichotomous outputs

of “significant”/“nonsignificant” or “reject”/“accept.”

A Statistical Model is Much More Than an Equation with

Greek Letters

The above list could be expanded by reviewing the research

literature. We will however turn to direct discussion of an issue

that has been receiving more attention of late, yet is still widely

overlooked or interpreted too narrowly in statistical teaching

and presentations: That the statistical model used to obtain the

results is correct.

Too often, the full statistical model is treated as a simple regression or structural equation in which effects are represented

by parameters denoted by Greek letters. “Model checking” is

then limited to tests of fit or testing additional terms for the

model. Yet these tests of fit themselves make further assumptions that should be seen as part of the full model. For example,

all common tests and confidence intervals depend on assumptions of random selection for observation or treatment and random loss or missingness within levels of controlled covariates.

These assumptions have gradually come under scrutiny via sensitivity and bias analysis (e.g., Lash et al. 2014), but such methods remain far removed from the basic statistical training given

to most researchers.

Less often stated is the even more crucial assumption that

the analyses themselves were not guided toward finding nonsignificance or significance (analysis bias), and that the analysis results were not reported based on their nonsignificance or

significance (reporting bias and publication bias). Selective reporting renders false even the limited ideal meanings of statistical significance, P-values, and confidence intervals. Because

author decisions to report and editorial decisions to publish results often depend on whether the P-value is above or below

0.05, selective reporting has been identified as a major problem

in large segments of the scientific literature (Dwan et al. 2013;

Page et al. 2014; You et al. 2012).

Although this selection problem has also been subject to sensitivity analysis, there has been a bias in studies of reporting and

publication bias: It is usually assumed that these biases favor

significance. This assumption is of course correct when (as is

often the case) researchers select results for presentation when

P ≤ 0.05, a practice that tends to exaggerate associations (Button et al. 2013; Eyding et al. 2010; Land 1980; Land 1981).

Nonetheless, bias in favor of reporting P ≤ 0.05 is not always

plausible let alone supported by evidence or common sense. For

example, one might expect selection for P > 0.05 in publications funded by those with stakes in acceptance of the null

hypothesis (a practice which tends to understate associations);

in accord with that expectation, some empirical studies have

observed smaller estimates and “nonsignificance” more often

in such publications than in other studies (Eyding et al. 2010;

Greenland 2009; Xu et al. 2013).

Addressing such problems would require far more political

will and effort than addressing misinterpretation of statistics,

such as enforcing registration of trials, along with open data

and analysis code from all completed studies (as in the AllTrials initiative, http://www.alltrials.net/). In the meantime, readers are advised to consider the entire context in which research

reports are produced and appear when interpreting the statistics

and conclusions offered by the reports.

8 Online Supplement to the ASA Statement on Statistical Significance and P-values

The Promise of Evidence-Based Policymaking 927
Conclusions

Upon realizing that statistical tests are usually misinterpreted,

one may wonder what if anything these tests do for science.

They were originally intended to account for random variability

as a source of error, thereby sounding a note of caution against

overinterpretation of observed associations as true effects or as

stronger evidence against null hypotheses than was warranted.

But before long that use was turned on its head to provide fallacious support for null hypotheses in the form of “failure to

achieve” or “failure to attain” statistical significance.

We have no doubt that the founders of modern statistical

testing would be horrified by common treatments of their invention. In their first paper describing their binary approach to

statistical testing, Neyman and Pearson (1928) wrote that “it

is doubtful whether the knowledge that [a P-value] was really

0.03 (or 0.06), rather than 0.05 . . . would in fact ever modify

our judgment” and that “The tests themselves give no final verdict, but as tools help the worker who is using them to form

his final decision.” Pearson (1955) later added, “No doubt we

could more aptly have said, ‘his final or provisional decision’.”

Fisher (1956, p. 42) went further, saying “No scientific worker

has a fixed level of significance at which from year to year, and

in all circumstances, he rejects hypotheses; he rather gives his

mind to each particular case in the light of his evidence and

his ideas.” Yet fallacious and ritualistic use of tests continued

to spread, including beliefs that whether P was above or below

0.05 was a universal arbiter of discovery. Thus by 1965, Hill

(1965) lamented that “too often we weaken our capacity to interpret data and to take reasonable decisions whatever the value

of P. And far too often we deduce ‘no difference’ from ‘no significant difference’.”

In response, it has been argued that some misinterpretations are harmless in tightly controlled experiments on wellunderstood systems, where the test hypothesis may have special

support from established theories (e.g., Mendelian genetics) and

in which every other assumption (such as random allocation) is

forced to hold by careful design and execution of the study. But

it has long been asserted that the harms of statistical testing in

more uncontrollable and amorphous research settings (such as

social-science, health, and medical fields) have far outweighed

its benefits, leading to calls for banning such tests in research

reports—again, with one journal banning confidence intervals

as well as P-values (Trafimow and Marks 2015).

Given, however, the deep entrenchment of statistical testing,

as well as the absence of generally accepted alternative methods, there have been many attempts to salvage P-values by detaching them from their use in significance tests. One approach

is to focus on P-values as continuous measures of compatibility,

as described earlier. Although this approach has its own limitations (as described in points 1, 2, 5, 9, 17, and 18), it avoids

misconceptions arising from comparison of P-values with arbitrary cutoffs such as 0.05 (as described in points 3, 4, 6–8,

10–13, 15, 16, 21, and 23–25). Another approach is to teach and

use correct relations of P-values to hypothesis probabilities. For

example, under common statistical models, one-sided P-values

can provide lower bounds on probabilities for hypotheses about

effect directions (Casella and Berger 1987ab; Greenland and

Poole 2013ab). Whether such reinterpretations can eventually

replace common misinterpretations to good effect remains to be

seen.

A shift in emphasis from hypothesis testing to estimation has

been promoted as a simple and relatively safe way to improve

practice (Yates 1951; Rothman 1978; Altman et al. 2000; Poole

2001; Cumming 2011), resulting in increasing use of confidence

intervals and editorial demands for them; nonetheless, this shift

has brought to the fore misinterpretations of intervals such as

19–23 above (Morey et al. 2015). Other approaches combine

tests of the null with further calculations involving both null

and alternative hypotheses (Rosenthal and Rubin 1994; Mayo

and Spanos 2006); such calculations may, however, may bring

with them further misinterpretations similar to those described

above for power, as well as greater complexity.

Meanwhile, in the hopes of minimizing harms of current

practice, we can offer several guidelines for users and readers

of statistics, and re-emphasize some key warnings from our list

of misinterpretations:

a) Correct and careful interpretation of statistical tests demands examining the sizes of effect estimates and confidence limits, as well as precise P-values (not just whether

P-values are above or below 0.05 or some other threshold).

b) Careful interpretation also demands critical examination

of the assumptions and conventions used for the statistical analysis—not just the usual statistical assumptions, but

also the hidden assumptions about how results were generated and chosen for presentation.

c) It is simply false to claim that statistically nonsignificant results support a test hypothesis, because the same

results may be even more compatible with alternative

hypotheses—even if the power of the test is high for those

alternatives.

d) Interval estimates aid in evaluating whether the data are

capable of discriminating among various hypotheses about

effect sizes, or whether statistical results have been misrepresented as supporting one hypothesis when those results are better explained by other hypotheses (see points

4–6). We caution however that confidence intervals are often only a first step in these tasks. To compare hypotheses

in light of the data and the statistical model it may be necessary to calculate the P-value (or relative likelihood) of

each hypothesis. We further caution that confidence intervals provide only a best-case measure of the uncertainty

or ambiguity left by the data, insofar as they depend on an

uncertain statistical model.

e) Correct statistical evaluation of multiple studies requires a

pooled analysis or meta-analysis that deals correctly with

study biases (Whitehead 2002; Borenstein et al. 2009;

Chen and Peace 2013; Cooper et al. 2009; Greenland and

O’Rourke 2008; Petitti 2000; Schmidt and Hunter 2014;

Sterne 2009). Even when this is done, however, all the earlier cautions apply. Furthermore, the outcome of any statistical procedure is but one of many considerations that must

be evaluated when examining the totality of evidence. In

The American Statistician, Online Supplement 9

928 The Promise of Evidence-Based Policymaking
particular, statistical significance is neither necessary nor

sufficient for determining the scientific or practical significance of a set of observations. This view was affirmed

unanimously by the U.S. Supreme Court, (Matrixx Initiatives, Inc., et al. v. Siracusano et al. No. 091156. Argued

January 10, 2011, Decided March 22, 2011), and can be

seen in our earlier quotes from Neyman and Pearson.

f) Any opinion offered about the probability, likelihood, certainty, or similar property for a hypothesis cannot be derived from statistical methods alone. In particular, significance tests and confidence intervals do not by themselves

provide a logically sound basis for concluding an effect

is present or absent with certainty or a given probability.

This point should be borne in mind whenever one sees

a conclusion framed as a statement of probability, likelihood, or certainty about a hypothesis. Information about

the hypothesis beyond that contained in the analyzed data

and in conventional statistical models (which give only

data probabilities) must be used to reach such a conclusion; that information should be explicitly acknowledged

and described by those offering the conclusion. Bayesian

statistics offers methods that attempt to incorporate the

needed information directly into the statistical model; they

have not however achieved the popularity of P-values and

confidence intervals, in part because of philosophical objections and in part because no conventions have become

established for their use.

g) All statistical methods (whether frequentist or Bayesian,

or for testing or estimation, or for inference or decision)

make extensive assumptions about the sequence of events

that led to the results presented—not only in the data generation, but in the analysis choices. Thus, to allow critical evaluation, research reports (including meta-analyses)

should describe in detail the full sequence of events that

led to the statistics presented, including the motivation for

the study, its design, the original analysis plan, the criteria

used to include and exclude subjects (or studies) and data,

and a thorough description of all the analyses that were

conducted.

In closing, we note that no statistical method is immune to

misinterpretation and misuse, but prudent users of statistics will

avoid approaches especially prone to serious abuse. In this regard, we join others in singling out the degradation of P-values

into “significant” and “nonsignificant” as an especially pernicious statistical practice (Weinberg 2001).

References

Altman, D.G., and Bland, J.M. (1995), “Absence of Evidence is not Evidence

of Absence,” British Medical Journal, 311, 485.

Altman, D.G., Machin, D., Bryant, T.N., and Gardner, M.J. (eds.) (2000), Statistics with Confidence (2nd ed.), London: BMJ Books.

Anscombe, F.J. (1990), “The Summarizing of Clinical Experiments by Significance Levels,” Statistics in Medicine, 9, 703–708.

Ashworth, A. (2015), “Veto on the Use of Null Hypothesis Testing

and p Intervals: Right or Wrong?” Taylor & Francis Editor Resources online, http://editorresources.taylorandfrancisgroup.com/

veto-on-the-use-of-null-hypothesis-testing-and-p-intervals-right-or-wrong/,

accessed Feb. 27, 2016.

Atkins, L., and Jarrett, D. (1979), “The Significance of ‘Significance Tests,”’

in Demystifying Social Statistics, Irvine, J., Miles, I., and Evans, J., eds.

London: Pluto Press.

Bakan, D. (1966), “The Test of Significance in Psychological Research,” Psychological Bulletin, 66, 423–437.

Bandt, C.L., and Boen, J.R. (1972), “A Prevalent Misconception About Sample Size, Statistical Significance, and Clinical Importance,” Journal of Periodontal Research, 43, 181–183.

Berger, J.O., and Sellke, T.M. (1987), “Testing a Point Null Hypothesis: The

Irreconcilability of P-values and Evidence,” Journal of the American Statistical Association, 82, 112–139.

Berkson, J. (1942), “Tests of Significance Considered as Evidence,” Journal of

the American Statistical Association, 37, 325–335.

Bland, J.M., and Altman, D.G. (2015), “Best (but oft forgotten) Practices: Testing for Treatment Effects in Randomized Trials by Separate Analyses of

Changes from Baseline in Each Group is a Misleading Approach,” American Journal of Clinical Nutrition, 102, 991–994.

Borenstein, M., Hedges, L.V., Higgins, J.P.T., and Rothstein, H.R. (2009), Introduction to Meta-Analysis, New York: Wiley.

Button, K., Ioannidis, J.P.A., Mokrysz, C., Nosek, B.A., Flint, J., Robinson,

E.S.J., Munafo, M.R. (2013), “Power Failure: Why Small Sample Size Un-  ́

dermines the Reliability of Neuroscience,” Nature Reviews Neuroscience,

14, 365–376.

Casella, G., and Berger, R.L. (1987a), “Reconciling Bayesian and Frequentist

Evidence in the One-Sided Testing Problem,” Journal of the American Statistical Association, 82, 106–111.

Casella, G., and Berger, R.L. (1987b), “Comment,” Stat Sci, 2, 344–417.

Chalmers, T.C., and Lau, J. (1996), “Changes in Clinical Trials Mandated by

the Advent of Meta-analysis,” Statistics in Medicine, 15, 1263–1268.

Chen, D.-G., and Peace, K.E. (2013), Applied Meta-Analysis with R, New York:

Chapman & Hall/CRC.

Chia, K.S. (1997), “‘Significant-itis’An Obsession with the P-Value,” Scandinavian Journal of Work, Environment & Health, 23, 152–154.

Cooper, H., Hedges, L.V., and Valentine, J.C. (2009), The Handbook of Research Synthesis and Meta-Analysis, Thousand Oaks, CA: Sage.

Cohen, J. (1994), “The Earth is Round (p < 0.05),” American Psychology, 47,

997–1003.

Cornfield, J. (1966), “Sequential Trials, Sequential Analysis, and the Likelihood

Principle,” The American Statistician, 25, 617–657.

Cox, D.R. (1958), The Planning of Experiments, New York: Wiley, p. 161.

(1977), “The Role of Significance Tests” (with discussion), Scandinavian Journal of Statistics, 4, 49–70.

(1982), “Statistical Significance Tests,” British Journal of Clinical

Pharmacology, 14, 325–331.

Cox, D.R., and Hinkley, D.V. (1974), Theoretical Statistics, New York: Chapman and Hall.

Cumming, G. (2011), Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis, London: Routledge.

Dickersin, K. (1990), “The Existence of Publication Bias and Risk Factors for

its Occurrence,” Journal of the American Medical Association, 263, 1385–

1389.

Dwan, K., Gamble, C., Williamson, P.R., Kirkham, J.J.; Reporting Bias Group

(2013), “Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias—An Updated Review,” PLoS One,

8:e66844.

Edwards, A.W.F. (1992), Likelihood (2nd ed.), Baltimore: Johns Hopkins University Press.

Edwards, W., Lindman, H., and Savage, L.J. (1963), “Bayesian Statistical Inference for Psychological Research,” Psychological Review, 70, 193–242.

Evans, S.J.W., Mills, P., and Dawson, J. (1988), “The End of the P-value?”

British Heart Journal, 60,177–180.

Eyding, D., Lelgemann, M., Grouven, U., Harter, M., Kromp, M., Kaiser, T.,  ̈

Kerekes, M.F., Gerken, M., and Wieseler, B. (2010), “Reboxetine for Acute

Treatment of Major Depression: Systematic Review and Meta-analysis of

Published and Unpublished Placebo and Selective Serotonin Reuptake In10 Online Supplement to the ASA Statement on Statistical Significance and P-values

The Promise of Evidence-Based Policymaking 929
hibitor Controlled Trials,” British Medical Journal, 341, c4737.

Fidler, F., and Loftus, G.R. (2009), “Why Figures with Error Bars Should Replace p Values: Some Conceptual Arguments and Empirical Demonstrations,” Journal of Psychology, 217, 27–37.

Fisher, R. A. (1956), Statistical Methods and Scientific Inference, Edinburgh,

UK: Oliver & Boyd.

Flanagan, O. (2015), “Journal’s Ban on Null Hypothesis Significance Testing:

Reactions from the Statistical Arena,” Stats Life online, accessed 27 Feb.

2016.

Freedman, D.A., Pisani, R., and Purves, R. (2007), Statistics (4th ed.), New

York: Norton.

Gardner, M.A., and Altman, D.G. (1986), “Confidence Intervals Rather than P

Values: Estimation Rather than Hypothesis Testing,” British Medical Journal, 292, 746–750.

Gelman, A. (2013), “P-Values and Statistical Practice,” Epidemiology, 24, 69–

72.

Gelman, A., and Loken, E. (2014), “The Statistical Crisis in Science: DataDependent Analysis—A ‘Garden of Forking Paths’—Explains why Many

Statistically Significant Comparisons Don’t Hold Up,” American Scientist, 102, 460–465. Erratum at http:// andrewgelman.com/ 2014/ 10/ 14/

didnt-say-part-2/, accessed Feb. 27, 2016.

Gelman, A., and Stern, H.S. (2006), “The Difference Between ‘Significant’ and

‘Not Significant’ is not Itself Statistically Significant,” The American Statistician, 60, 328–331.

Gibbons, J.D., and Pratt, J.W. (1975), “P-Values: Interpretation and Methodology,” The American Statistician, 29, 20–25.

Gigerenzer, G. (2004), “Mindless Statistics,” Journal of Socioeconomics, 33,

567–606.

Gigerenzer, G., and Marewski, J.N. (2015), “Surrogate Science: The Idol of

a Universal Method for Scientific Inference,” Journal of Management, 41,

421–440.

Gigerenzer, G., Swijtink, Z., Porter, T., Daston, L., Beatty, J., and Kruger, L.

(1990), The Empire of Chance: How Probability Changed Science and Everyday Life, New York: Cambridge University Press.

Goodman, S.N. (1992), “A Comment on Replication, p-values and Evidence,”

Statistics in Medicine, 11, 875–879.

(1993), “P-values, Hypothesis Tests and Likelihood: Implications for

Epidemiology of a Neglected Historical Debate,” American Journal of Epidemiology, 137, 485–496.

(1994), Letter to the editor re Smith and Bates, Epidemiology, 5, 266–

268.

(1999), “Towards Evidence-Based Medical Statistics, I: The P-value

Fallacy,” Annals of Internal Medicine, 130, 995–1004.

(2005), “Introduction to Bayesian Methods I: Measuring the Strength

of Evidence,” Clinical Trials, 2, 282–290.

(2008), “A Dirty Dozen: Twelve P-value Misconceptions,” Seminars in

Hematology, 45, 135–140.

Goodman, S.N., and Berlin, J. (1994), “The Use of Predicted Confidence Intervals when Planning Experiments and the Misuse of Power when Interpreting

Results,” Annals of Internal Medicine, 121, 200–206.

Goodman, S.N., and Royall, R. (1988), “Evidence and Scientific Research,”

American Journal of Public Health, 78, 1568–1574.

Greenland, S. (2009), “Dealing with Uncertainty About iInvestigator Bias: Disclosure is Informative,” Journal of Epidemiology and Community Health,

63, 593–598.

(2011), “Null Misinterpretation in Statistical Testing and its Impact on

Health Risk Assessment,” Preventive Medicine, 53, 225–228.

(2012a), “Nonsignificance Plus High Power Does not Imply Support

for the Null over the Alternative,” Annals of Epidemiology, 22, 364–368.

(2012b), “Transparency and Disclosure, Neutrality and Balance: Shared

Values or Just Shared Words?” Journal of Epidemiology and Community

Health, 66, 967–970.

Greenland, S., and O’Rourke, K. (2008), “Meta-analysis,” in Modern Epidemiology (3rd ed.), Rothman, K.J., Greenland, S., Lash, T.L., eds., Philadelphia:

Lippincott-Wolters-Kluwer, pp. 682–685.

Greenland, S., and Poole, C. (2011), “Problems in Common Interpretations of

Statistics in Scientific Articles, Expert Reports, and Testimony,” Jurimetrics,

51, 113–129.

(2013a), “Living with P-values: Resurrecting a Bayesian Perspective

on Frequentist Statistics,” Epidemiology, 24, 62–68.

(2013b), “Living with Statistics in Observational Research,” Epidemiology, 24, 73–78.

Grieve, A.P. (2015), “How to Test Hypotheses if You Must,” Pharmaceutical

Statistics, 14, 139–150.

Hanley, J.A. (1994), Letter to the Editor re Smith and Bates,” Epidemiology, 5,

264–266.

Harlow, L.L., Mulaik, S.A., and Steiger, J.H. (1997), “What if There Were No

Significance Tests?” Psychology Press.

Hauer, E. (2003), “The Harm Done by Tests of Significance,” Accident Analysis

& Prevention, 36, 495–500.

Hedges, L.V., and Olkin, I. (1980), “Vote-Counting Methods in Research Synthesis,” Psychological Bulletin, 88, 359–369.

Hill, A.B. (1965), “The Environment and Disease: Association or Causation?”

Proceedings of the Royal Society of Medicine, 58, 295–300.

Hoekstra, R., Finch, S., Kiers, H.A.L., and Johnson, A. (2006), “Probability as

Certainty: Dichotomous Thinking and the Misuse of p-Values,” Psychological Bulletin Review, 13, 1033–1037.

Hoenig, J.M., and Heisey, D.M. (2001), “The Abuse of Power: The Pervasive

Fallacy of Power Calculations for Data Analysis,” The American Statistician, 55, 19–24.

Hogben, L. (1957), Statistical Theory, London: Allen and Unwin.

Hurlbert, S.H., and Lombardi, C.M. (2009), “Final Collapse of the NeymanPearson Decision Theoretic Framework and Rise of the neoFisherian,” Annales Zoologici Fennici, 46, 311–349.

Kaye, D.H. (1986), “Is Proof of Statistical Significance Relevant?” Washington

Law Review, 61, 1333–1366.

Kaye, D.H., and Freedman, D.A. (2011), “Reference Guide on Statistics,” in

Reference Manual on Scientific Evidence (3rd ed.), Washington, DC: Federal Judicial Center, 211–302.

Kline, R.B. (2013), Beyond Significance Testing: Statistics Reform in the Behavioral Sciences, Washington, DC: American Psychological Association.

Lambdin, C. (2012), “Significance Tests as Sorcery: Science is Empirical—

Significance Tests are Not,” Theory & Psychology, 22, 67–90.

Land, C.E. (1980), “Estimating Cancer Risks from Low Doses of Ionizing Radiation,” Science, 209, 1197–1203.

(1981), “Statistical Limitations in Relation to Sample Size,” Environmental Health Perspectives, 42, 15–21.

Lang, J.M., Rothman, K.J., and Cann, C.I. (1998), “That Confounded P-Value,”

Epidemiology, 9, 7–8.

Langman, M.J.S. (1986), “Towards Estimation and Confidence Intervals,” BMJ,

292, 716.

Lash, T.L., Fox, M.P., Maclehose, R.F., Maldonado, G., McCandless, L.C., and

Greenland, S. (2014), “Good Practices for Quantitative Bias Analysis,” International Journal of Epidemiology, 43, 1969–1985.

Lecoutre, M.-P., Poitevineau, J., and Lecoutre, B. (2003), “Even Statisticians are

not Immune to Misinterpretations of Null Hypothesis Tests,” International

Journal of Psychology, 38, 37–45.

Lehmann, E.L. (1986), Testing Statistical Hypotheses (2nd ed.), New York, Wiley.

Lew, M.J. (2012), “Bad Statistical Practice in Pharmacology (and Other Basic

Biomedical Disciplines): You Probably Don’t Know P,” British Journal of

Pharmacology, 166, 1559–1567.

Loftus, G.R. (1996), “Psychology Will be a Much Better Science When We

Change the Way We Analyze Data,” Current Directions in Psychology, 5,

161–171.

Maheshwari, S., Sarraj, A., Kramer, J., and El-Serag, H.B. (2007), “Oral Contraception and the Risk of Hepatocellular Carcinoma,” Journal Hepatology,

47, 506–513.

Marshall, S.W. (2006), “Commentary on Making Meaningful Inferences About

Magnitudes,” Sportscience, 9, 43–44.

The American Statistician, Online Supplement 11

930 The Promise of Evidence-Based Policymaking
Matthews, J.N.S., and Altman, D.G. (1996a), “Interaction 2: Compare Effect

Sizes not P Values,” British Medical Journal, 313, 808.

(1996b), “Interaction 3: How to Examine Heterogeneity,” British Medical Journal, 313, 862.

Mayo, D.G., and Cox, D.R. (2006), “Frequentist Statistics as a Theory of Inductive Inference,” in Optimality: The Second Erich L. Lehmann Symposium,

Lecture Notes-Monograph Series, J. Rojo (ed.), Hayward, CA: Institute of

Mathematical Statistics (IMS) 49, 77–97.

Mayo, D.G., and Spanos, A. (2006), “Severe Testing as a Basic Concept in a

Neyman-Pearson Philosophy of Induction,” British Journal of Philosophical Science, 57, 323–357.

Morey, R.D., Hoekstra, R., Rouder, J.N., Lee, M.D., and Wagenmakers, E.-

J. (in press), “The Fallacy of Placing Confidence in Confidence Intervals,”

Psychological Bulletin Review.

Morrison, D.E., and Henkel, R.E. (eds.) (1970), The Significance Test Controversy , Chicago: Aldine.

Murtaugh, P.A. (2014), “In Defense of P-Values” (with discussion), Ecology,

95, 611–653.

Neyman, J. (1937), “Outline of a Theory of Statistical Estimation Based on the

Classical Theory of Probability,” Philospohical Transactions of the Royal

Society of London A, 236, 333–380.

Neyman, J., and Pearson, E.S. (1928), “On the Use and Interpretation of Certain

Test Criteria for Purposes of Statistical Inference: Part I,” Biometrika, 20A,

175–240.

Oakes, M. (1986), Statistical Inference: A Commentary for the Social and Behavioural Sciences, Chichester: Wiley.

Page, M.J., McKenzie, J.E., Kirkham, J., Dwan, K., Kramer, S., Green, S., and

Forbes, A. (2014), “Bias Due to Selective Inclusion and Reporting of Outcomes and Analyses in Systematic Reviews of Randomised Trials of Healthcare Interventions,” Cochrane Database System Reviews, 10:MR000035.

Peace, K. (1988), “Some Thoughts on One-Tailed Tests,” Biometrics, 44, 911–

912.

Pearson, E.S. (1955), “Statistical Concepts in the Relation to Reality,” Journal

fo the Royal Statistical Society, Series B, 17, 204–207.

Petitti, D.B. (2000), Meta-Analysis, Decision Analysis, and Cost-Effectiveness

Analysis: Methods for Quantitative Synthesis in Medicine (2nd ed.), New

York: Oxford University Press.

Phillips, C.V. (2004), “Publication Bias In Situ,” BMC Medical Research and

Methodology, 4, 20.

Pocock, S.J., Hughes, M.D., and Lee, R.J. (1987), “Statistical Problems in the

Reporting of Clinical Trials,” New England Journal of Medicine, 317, 426–

432.

Pocock, S.J., and Ware, J.H. (2009), “Translating Statistical Findings into Plain

English,” The Lancet, 373, 1926–1928.

Poole, C. (1987a), “Beyond the Confidence Interval,” American Journal of Public Health, 77, 195–199.

(1987b), “Confidence Intervals Exclude Nothing,” American Journal of

Public Health, 77:, 492–493.

(2001), “Low P-Values or Narrow Confidence Intervals: Which are

More Durable?”, Epidemiology, 12, 291–294.

Pratt, J.W. (1965), “Bayesian Interpretation of Standard Inference Statements,”

Journal of the Royal Statistical Society, Series B, 27, 169–203.

Rosenthal, R., and Rubin, D.B. (1994), “The Counternull Value of an Effect

Size: A New Statistic,” Psychological Science, 5, 329–334.

Rosnow, R.L., and Rosenthal, R. (1989), “Statistical Procedures and the Justification of Knowledge in Psychological Science,” American Psychologist,

44, 1276–1284.

Rothman, K.J. (1978), “A Show of Confidence,” New England School of

Medicine, 299, 1362–1363.

(1986), “Significance Questing,” Annals of Internal Medicine, 105,

445–447.

Rothman, K.J., Greenland, S., and Lash, T.L. (2008), Modern Epidemiology

(3rd ed.), Philadelphia, PA: Lippincott-Wolters-Kluwer.

Royall, R. (1997), Statistical Evidence, New York: Chapman and Hall.

Rozeboom, W.M. (1960), “The Fallacy of Null-Hypothesis Significance Test,”

Psychological Bulletin, 57, 416–428.

Salsburg, D.S. (1985), “The Religion of Statistics as Practiced in Medical Journals,” The American Statistician, 39, 220–223.

Schervish, M. (1996), “P-Values: What They are and What They are Not,” The

American Statistician, 50, 203–206.

Schmidt, F.L. (1996), “Statistical Significance Testing and Cumulative Knowledge in Psychology: Implications for Training of Researchers,” Psychological Methods, 1, 115–129.

Schmidt, F.L., and Hunter, J.E. (2014), Methods of Meta-Analysis: Correcting

Error and Bias in Research Findings (3rd ed.), Thousand Oaks, CA: Sage.

Sellke, T.M., Bayarri, M.J., and Berger, J.O. (2001), “Calibration of p Values

for Testing Precise Null Hypotheses,” The American Statistician, 55, 62–71.

Senn, S.J. (2001), “Two Cheers for P?Values,” Journal of Epidemiology and

Biostatistics, 6, 193–204.

(2002a), Letter to the Editor re: Goodman 1992, Statistics in Medicine,

21, 2437–2444.

(2002b), “Power is Indeed Irrelevant in Interpreting Completed Studies,” BMJ, 325, 1304.

Smith, A.H., and Bates, M. (1992), “Confidence Limit Analyses Should Replace Power Calculations in the Interpretation of Epidemiologic Studies,”

Epidemiology, 3, 449–452.

Sterne, J.A.C. (2009), Meta-Analysis: An Updated Collection from the Stata

Journal, College Station, TX: Stata Press.

Sterne, J.A.C., and Davey Smith, G. (2001), “Sifting the Evidence—What’s

Wrong with Significance Tests?” British Medical Journal, 322, 226–231.

Stigler, S.M. (1986), The History of Statistics, Cambridge, MA: Belknap Press.

Thompson, B. (2004), “The ‘significance’ Crisis in Psychology and Education,”

The Journal of Socio-Economics, 33, 607–613.

Thompson, W.D. (1987), “Statistical Criteria in the Interpretation of Epidemiologic Data,” American Journal of Public Health, 77, 191–194.

Trafimow, D., and Marks, M. (2015), Editorial, Basic and Applied Social Psychology, 37, 1–2.

Wagenmakers, E.-J. (2007), “A Practical Solution to the Pervasive Problem of

p Values,” Psychonomic Bulletin Review, 14, 779–804.

Walker, A.M. (1986), “Reporting the Results of Epidemiologic Studies,” American Journal of Public Health, 76, 556–558.

Ware, J.H., Mosteller, F., and Ingelfinger, J.A. (2009), “p-Values,” in Medical

Uses of Statistics (3rd ed.), Bailar, J.C. and Hoaglin, D.C. (eds.), Hoboken,

NJ: Wiley, pp. 175–194.

Weinberg, C.R. (2001), “Its Time to Rehabilitate the P-Value,” Epidemiology,

12, 288–290.

Whitehead, A. (2002), Meta-Analysis of Controlled Clinical Trials, New York:

Wiley.

Wood, J., Freemantle, N., King, M., and Nazareth, I. (2014), “Trap of

Trends to Statistical Significance: Likelihood of Near Significant P

Value Becoming More Significant with Extra Data,” BMJ, 348, g2215,

doi:10.1136/bmj.g2215.

Xu, L., Freeman, G., Cowling, B.J., and Schooling, C.M. (2013), “Testosterone

Therapy and Cardiovascular Events Among Men: A Systematic Review and

Meta-analysis of Placebo-Controlled Randomized Trials,” BMC Med., 11,

108.

Yates, F. (1951), “The Influence of Statistical Methods for Research Workers

on the Development of the Science of Statistics,” Journal of the American

Statistical Association, 46, 19–34.

You ,B., Gan, H.K., Pond, G., and Chen, E.X. (2012), “Consistency in the Analysis and Reporting of Primary End Points in Oncology Randomized Controlled Trials from Registration to Publication: A Systematic Review,” Journal of Clinical Oncology, 30, 210–216.

Ziliak, S.T., and McCloskey, D.N. (2008), The Cult of Statistical Significance:

How the Standard Error Costs Us Jobs, Justice and Lives, Ann Arbor: U

Michigan Press.

12 Online Supplement to the ASA Statement on Statistical Significance and P-values

The Promise of Evidence-Based Policymaking 931
>%		"'
?
%:?%


#&22666
?	"22<%	0?"@<%	'
A%

	






 !"

!	#
$#%%&&&'
()!'!	%!%"


*
	!#+
",!-.
!.


("
#!

/!
(0'1

2!'0
3



!

	

	 ! "

#$	%&'()*)
*%#)" )+&),--)./0&

12--3341

!4!

#&22
(
5212--3341

$6%##	"7"	

#
%8#
	&+

9

*%:	
	&,;%

 %:"7%	<%	

	86&-43

$6	
	

$6'"=


'5	&-$65	

932 The Promise of Evidence-Based Policymaking
THE AMERICAN STATISTICIAN

, VOL. , NO. , –

http://dx.doi.org/./..

EDITORIAL

The ASA’s Statement on p-Values: Context, Process, and Purpose

In February 2014, George Cobb, Professor Emeritus of Mathematics and Statistics at Mount Holyoke College, posed these

questions to an ASA discussion forum:

Q: Why do so many colleges and grad schools teach p = 0.05?

A: Because that’s still what the scientific community and journal

editors use.

Q: Why do so many people still use p = 0.05?

A: Because that’s what they were taught in college or grad school.

Cobb’s concern was a long-worrisome circularity in the sociology of science based on the use of bright lines such as p < 0.05:

“We teach it because it’s what we do; we do it because it’s what

we teach.” This concern was brought to the attention of the ASA

Board.

The ASA Board was also stimulated by highly visible discussions over the last few years. For example, ScienceNews

(Siegfried 2010) wrote: “It’s science’s dirtiest secret: The ‘scientific method’ of testing hypotheses by statistical analysis stands

on a flimsy foundation.” A November 2013, article in Phys.org

Science News Wire (2013) cited “numerous deep flaws” in null

hypothesis significance testing. A ScienceNews article (Siegfried

2014) on February 7, 2014, said “statistical techniques for testing

hypotheses ...have more flaws than Facebook’s privacy policies.”

A week later, statistician and “Simply Statistics” blogger Jeff Leek

responded. “The problem is not that people use P-values poorly,”

Leek wrote, “it is that the vast majority of data analysis is not

performed by people properly trained to perform data analysis” (Leek 2014). That same week, statistician and science writer

Regina Nuzzo published an article in Nature entitled “Scientific

Method: Statistical Errors” (Nuzzo 2014). That article is now one

of the most highly viewed Nature articles, as reported by altmetric.com (http://www.altmetric.com/details/2115792#score).

Of course, it was not simply a matter of responding to some

articles in print. The statistical community has been deeply concerned about issues of reproducibility and replicability of scientific conclusions. Without getting into definitions and distinctions of these terms, we observe that much confusion and even

doubt about the validity of science is arising. Such doubt can lead

to radical choices, such as the one taken by the editors of Basic

and Applied Social Psychology, who decided to ban p-values (null

hypothesis significance testing) (Trafimow and Marks 2015).

Misunderstanding or misuse of statistical inference is only one

cause of the “reproducibility crisis” (Peng 2015), but to our community, it is an important one.

When the ASA Board decided to take up the challenge of

developing a policy statement on p-values and statistical significance, it did so recognizing this was not a lightly taken step.

The ASA has not previously taken positions on specific matters of statistical practice. The closest the association has come

to this is a statement on the use of value-added models (VAM)

for educational assessment (Morganstein and Wasserstein

2014) and a statement on risk-limiting post-election audits

(American Statistical Association 2010). However, these were

truly policy-related statements. The VAM statement addressed

a key educational policy issue, acknowledging the complexity of

the issues involved, citing limitations of VAMs as effective performance models, and urging that they be developed and interpreted with the involvement of statisticians. The statement on

election auditing was also in response to a major but specific

policy issue (close elections in 2008), and said that statistically

based election audits should become a routine part of election

processes.

By contrast, the Board envisioned that the ASA statement

on p-values and statistical significance would shed light on an

aspect of our field that is too often misunderstood and misused

in the broader research community, and, in the process, provides the community a service. The intended audience would be

researchers, practitioners, and science writers who are not primarily statisticians. Thus, this statement would be quite different

from anything previously attempted.

The Board tasked Wasserstein with assembling a group of

experts representing a wide variety of points of view. On behalf

of the Board, he reached out to more than two dozen such people, all of whom said they would be happy to be involved. Several

expressed doubt about whether agreement could be reached, but

those who did said, in effect, that if there was going to be a discussion, they wanted to be involved.

Over the course of many months, group members discussed

what format the statement should take, tried to more concretely visualize the audience for the statement, and began

to find points of agreement. That turned out to be relatively

easy to do, but it was just as easy to find points of intense

disagreement.

The time came for the group to sit down together to hash

out these points, and so in October 2015, 20 members of the

group met at the ASA Office in Alexandria, Virginia. The 2-day

meeting was facilitated by Regina Nuzzo, and by the end of the

meeting, a good set of points around which the statement could

be built was developed.

The next 3 months saw multiple drafts of the statement,

reviewed by group members, by Board members (in a lengthy

discussion at the November 2015 ASA Board meeting), and

by members of the target audience. Finally, on January 29,

2016, the Executive Committee of the ASA approved the

statement.

The statement development process was lengthier and more

controversial than anticipated. For example, there was considerable discussion about how best to address the issue of multiple

potential comparisons (Gelman and Loken 2014). We debated

at some length the issues behind the words “a p-value near

0.05 taken by itself offers only weak evidence against the null

©  American Statistical Association

Downloaded by [University of California, Los Angeles (UCLA)] at 09:36 06 July 2016 

The Promise of Evidence-Based Policymaking 933
130 EDITORIAL

hypothesis” (Johnson 2013). There were differing perspectives

about how to characterize various alternatives to the p-value and

in how much detail to address them. To keep the statement reasonably simple, we did not address alternative hypotheses, error

types, or power (among other things), and not everyone agreed

with that approach.

As the end of the statement development process neared,

Wasserstein contacted Lazar and asked if the policy statement

might be appropriate for publication in The American Statistician (TAS). After consideration, Lazar decided that TAS would

provide a good platform to reach a broad and general statistical readership. Together, we decided that the addition of an

online discussion would heighten the interest level for the TAS

audience, giving an opportunity to reflect the aforementioned

controversy.

To that end, a group of discussants was contacted to provide

comments on the statement. You can read their statements in

the online supplement, and a guide to those statements appears

at the end of this editorial. We thank Naomi Altman, Douglas Altman, Daniel J. Benjamin, Yoav Benjamini, Jim Berger,

Don Berry, John Carlin, George Cobb, Andrew Gelman, Steve

Goodman, Sander Greenland, John Ioannidis, Joseph Horowitz,

Valen Johnson, Michael Lavine, Michael Lew, Rod Little, Deborah Mayo, Michele Millar, Charles Poole, Ken Rothman, Stephen

Senn, Dalene Stangl, Philip Stark and Steve Ziliak for sharing

their insightful perspectives.

Of special note is the following article, which is a significant

contribution to the literature about p-values and statistical

significance.

Greenland, S., Senn, S.J., Rothman, K.J., Carlin, J.B., Poole, C., Goodman, S.N. and Altman, D.G.: “Statistical Tests, P-values, Confidence

Intervals, and Power: A Guide to Misinterpretations.”

Though there was disagreement on exactly what the statement should say, there was high agreement that the ASA should

be speaking out about these matters.

Let us be clear. Nothing in the ASA statement is new. Statisticians and others have been sounding the alarm about these matters for decades, to little avail. We hoped that a statement from

the world’s largest professional association of statisticians would

open a fresh discussion and draw renewed and vigorous attention to changing the practice of science with regards to the use

of statistical inference.

Guide to the Online Supplemental Material to the ASA

Statement on P-Values and Statistical Significance

Many of the participants in the development of the ASA statement contributed commentary about the statement or matters

related to it. Their comments are posted as online supplements

to the statement. We provide here a list of the supplemental

articles.

Supplemental Material to the ASA Statement on

P-Values and Statistical Significance

 Altman, Naomi: Ideas from multiple testing of high dimensional data provide insights about reproducibility and false

discovery rates of hypothesis supported by p-values

 Benjamin, Daniel J, and Berger, James O: A simple alternative to p-values  Benjamini, Yoav: It’s not the p-values’ fault  Berry, Donald A: P-values are not what they’re cracked up

to be

 Carlin, John B: Comment: Is reform possible without a

paradigm shift?

 Cobb, George: ASA statement on p-values: Two consequences we can hope for

 Gelman, Andrew: The problems with p-values are not just

with p-values  Goodman, Steven N: The next questions: Who, what, when,

where, and why?

 Greenland, Sander: The ASA guidelines and null bias in

current teaching and practice

 Ioannidis, John P.A.: Fit-for-purpose inferential methods: abandoning/changing P-values versus abandoning/changing research

 Johnson, Valen E.: Comments on the “ASA Statement on

Statistical Significance and P-values" and marginally significant p-values  Lavine, Michael, and Horowitz, Joseph: Comment  Lew, Michael J: Three inferential questions, two types of

P-value  Little, Roderick J: Discussion  Mayo, Deborah G: Don’t throw out the error control baby

with the bad statistics bathwater

 Millar, Michele: ASA statement on p-values: some implications for education

 Rothman, Kenneth J: Disengaging from statistical significance

 Senn, Stephen: Are P-Values the Problem?  Stangl, Dalene: Comment  Stark, P.B.: The value of p-values  Ziliak, Stephen T: The significance of the ASA statement on

statistical significance and p-values

References

American Statistical Association (2010), “ASA Statement on RiskLimiting Post Election Audits.” Available at http://www.amstat.org/

policy/pdfs/Risk-Limiting_Endorsement.pdf. [129]

Gelman, A., and Loken, E. (2014), “The Statistical Crisis in Science

[online],” American Scientist, 102. Available at http://www.americanscientist.org/issues/feature/2014/6/the-statistical-crisis-in-science. [129]

Johnson, V. E. (2013), “Uniformly Most Powerful Bayesian Tests,” Annals of

Statistics, 41, 1716–1741. [130]

Leek, J. (2014), “On the Scalability of Statistical Procedures: Why the

p-Value Bashers Just Don’t Get It,” Simply Statistics Blog, Available

at http://simplystatistics.org/2014/02/14/on-the-scalability-of-statisticalprocedures-why-the-p-value-bashers-just-dont-get-it/. [129]

Morganstein, D., and Wasserstein, R. (2014), “ASA Statement on ValueAdded Models,” Statistics and Public Policy, 1, 108–110. Available at

http://amstat.tandfonline.com/doi/full/10.1080/2330443X.2014.956906.

[129]

Nuzzo, R. (2014), “Scientific Method: Statistical Errors,” Nature, 506,

150–152. Available at http://www.nature.com/news/scientific-methodstatistical-errors-1.14700. [129]

Peng, R. (2015), “The Reproducibility Crisis in Science: A Statistical Counterattack,” Significance, 12, 30–32. [129]

Phys.org Science News Wire (2013), “The Problem With p Values: How

Significant are They, Really?” Available at http://phys.org/wire-news/

Downloaded by [University of California, Los Angeles (UCLA)] at 09:36 06 July 2016 

934 The Promise of Evidence-Based Policymaking
EDITORIAL 131

145707973/the-problem-with-p-values-how-significant-are-they-really.

html. [129]

Siegfried, T. (2010), “Odds Are, It’s Wrong: Science Fails to Face the

Shortcomings of Statistics,” Science News, 177, 26. Available at https://

www.sciencenews.org/article/odds-are-its-wrong. [129]

Siegfried, T. (2014), “To Make Science Better, Watch out for Statistical Flaws,” Science News Context Blog, February 7, 2014. Available at https://www.sciencenews.org/blog/context/make-science-betterwatch-out-statistical-flaws. [129]

Trafimow, D., and Marks, M. (2015), “Editorial,” Basic and Applied Social

Psychology 37, 1–2. [129]

Ronald L. Wasserstein and Nicole A. Lazar

ron@amstat.org

American Statistical Association, 732 North Washington Street,

Alexandria, VA 22314-1943.

ASA Statement on Statistical Significance and P-Values

1. Introduction

Increased quantification of scientific research and a proliferation of large, complex datasets in recent years have expanded the

scope of applications of statistical methods. This has created new

avenues for scientific progress, but it also brings concerns about

conclusions drawn from research data. The validity of scientific

conclusions, including their reproducibility, depends on more

than the statistical methods themselves. Appropriately chosen

techniques, properly conducted analyses and correct interpretation of statistical results also play a key role in ensuring that

conclusions are sound and that uncertainty surrounding them

is represented properly.

Underpinning many published scientific conclusions is the

concept of “statistical significance,” typically assessed with an

index called the p-value. While the p-value can be a useful statistical measure, it is commonly misused and misinterpreted. This has led to some scientific journals discouraging

the use of p-values, and some scientists and statisticians recommending their abandonment, with some arguments essentially

unchanged since p-values were first introduced.

In this context, the American Statistical Association (ASA)

believes that the scientific community could benefit from a formal statement clarifying several widely agreed upon principles

underlying the proper use and interpretation of the p-value. The

issues touched on here affect not only research, but research

funding, journal practices, career advancement, scientific education, public policy, journalism, and law. This statement does

not seek to resolve all the issues relating to sound statistical practice, nor to settle foundational controversies. Rather, the statement articulates in nontechnical terms a few select principles

that could improve the conduct or interpretation of quantitative science, according to widespread consensus in the statistical

community.

2. What is a p-Value?

Informally, a p-value is the probability under a specified statistical model that a statistical summary of the data (e.g., the sample

mean difference between two compared groups) would be equal

to or more extreme than its observed value.

3. Principles

1. P-values can indicate how incompatible the data are

with a specified statistical model.

A p-value provides one approach to summarizing

the incompatibility between a particular set of data and

a proposed model for the data. The most common

context is a model, constructed under a set of assumptions, together with a so-called “null hypothesis.” Often

the null hypothesis postulates the absence of an effect,

such as no difference between two groups, or the absence

of a relationship between a factor and an outcome. The

smaller the p-value, the greater the statistical incompatibility of the data with the null hypothesis, if the underlying assumptions used to calculate the p-value hold. This

incompatibility can be interpreted as casting doubt on

or providing evidence against the null hypothesis or the

underlying assumptions.

2. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data

were produced by random chance alone.

Researchers often wish to turn a p-value into a statement about the truth of a null hypothesis, or about the

probability that random chance produced the observed

data. The p-value is neither. It is a statement about data

in relation to a specified hypothetical explanation, and is

not a statement about the explanation itself.

3. Scientific conclusions and business or policy decisions

should not be based only on whether a p-value passes

a specific threshold.

Practices that reduce data analysis or scientific inference to mechanical “bright-line” rules (such as “p <

0.05”) for justifying scientific claims or conclusions can

lead to erroneous beliefs and poor decision making. A

conclusion does not immediately become “true” on one

side of the divide and “false” on the other. Researchers

should bring many contextual factors into play to derive

scientific inferences, including the design of a study,

the quality of the measurements, the external evidence

for the phenomenon under study, and the validity of

assumptions that underlie the data analysis. Pragmatic

considerations often require binary, “yes-no” decisions,

but this does not mean that p-values alone can ensure

that a decision is correct or incorrect. The widespread

use of “statistical significance” (generally interpreted as

“p  0.05”) as a license for making a claim of a scientific

finding (or implied truth) leads to considerable distortion of the scientific process.

4. Proper inference requires full reporting and

transparency

P-values and related analyses should not be reported

selectively. Conducting multiple analyses of the data

and reporting only those with certain p-values (typically those passing a significance threshold) renders the

Downloaded by [University of California, Los Angeles (UCLA)] at 09:36 06 July 2016 

The Promise of Evidence-Based Policymaking 935
132 EDITORIAL

reported p-values essentially uninterpretable. Cherrypicking promising findings, also known by such terms as

data dredging, significance chasing, significance questing, selective inference, and “p-hacking,” leads to a

spurious excess of statistically significant results in the

published literature and should be vigorously avoided.

One need not formally carry out multiple statistical tests

for this problem to arise: Whenever a researcher chooses

what to present based on statistical results, valid interpretation of those results is severely compromised if

the reader is not informed of the choice and its basis.

Researchers should disclose the number of hypotheses

explored during the study, all data collection decisions,

all statistical analyses conducted, and all p-values computed. Valid scientific conclusions based on p-values and

related statistics cannot be drawn without at least knowing how many and which analyses were conducted, and

how those analyses (including p-values) were selected for

reporting.

5. A p-value, or statistical significance, does not measure

the size of an effect or the importance of a result.

Statistical significance is not equivalent to scientific, human, or economic significance. Smaller p-values

do not necessarily imply the presence of larger or

more important effects, and larger p-values do not

imply a lack of importance or even lack of effect. Any

effect, no matter how tiny, can produce a small p-value

if the sample size or measurement precision is high

enough, and large effects may produce unimpressive

p-values if the sample size is small or measurements

are imprecise. Similarly, identical estimated effects will

have different p-values if the precision of the estimates

differs.

6. By itself, a p-value does not provide a good measure of

evidence regarding a model or hypothesis.

Researchers should recognize that a p-value without

context or other evidence provides limited information.

For example, a p-value near 0.05 taken by itself offers only

weak evidence against the null hypothesis. Likewise, a

relatively large p-value does not imply evidence in favor

of the null hypothesis; many other hypotheses may be

equally or more consistent with the observed data. For

these reasons, data analysis should not end with the calculation of a p-value when other approaches are appropriate and feasible.

4. Other Approaches

In view of the prevalent misuses of and misconceptions concerning p-values, some statisticians prefer to supplement or even

replace p-values with other approaches. These include methods that emphasize estimation over testing, such as confidence,

credibility, or prediction intervals; Bayesian methods; alternative measures of evidence, such as likelihood ratios or Bayes Factors; and other approaches such as decision-theoretic modeling

and false discovery rates. All these measures and approaches rely

on further assumptions, but they may more directly address the

size of an effect (and its associated uncertainty) or whether the

hypothesis is correct.

5. Conclusion

Good statistical practice, as an essential component of good

scientific practice, emphasizes principles of good study design

and conduct, a variety of numerical and graphical summaries

of data, understanding of the phenomenon under study, interpretation of results in context, complete reporting and proper

logical and quantitative understanding of what data summaries

mean. No single index should substitute for scientific reasoning.

Acknowledgments

The ASA Board of Directors thanks the following people for sharing their

expertise and perspectives during the development of the statement. The

statement does not necessarily reflect the viewpoint of all these people, and

in fact some have views that are in opposition to all or part of the statement.

Nonetheless, we are deeply grateful for their contributions. Naomi Altman,

Jim Berger, Yoav Benjamini, Don Berry, Brad Carlin, John Carlin, George

Cobb, Marie Davidian, Steve Fienberg, Andrew Gelman, Steve Goodman, Sander Greenland, Guido Imbens, John Ioannidis, Valen Johnson,

Michael Lavine, Michael Lew, Rod Little, Deborah Mayo, Chuck McCulloch, Michele Millar, Sally Morton, Regina Nuzzo, Hilary Parker, Kenneth

Rothman, Don Rubin, Stephen Senn, Uri Simonsohn, Dalene Stangl, Philip

Stark, Steve Ziliak.

Edited by Ronald L.Wasserstein, Executive Director

On behalf of the American Statistical Association

Board of Directors

A Brief p-Values and Statistical Significance Reference

List

Altman D.G., and Bland J.M. (1995), “Absence of Evidence is not Evidence

of Absence,” British Medical Journal, 311, 485.

Altman, D.G., Machin, D., Bryant, T.N., and Gardner, M.J. (eds.) (2000),

Statistics with Confidence (2nd ed.), London: BMJ Books.

Berger, J.O., and Delampady, M. (1987), “Testing Precise Hypotheses,” Statistical Science, 2, 317–335.

Berry, D. (2012), “Multiplicities in Cancer Research: Ubiquitous and Necessary Evils,” Journal of the National Cancer Institute, 104, 1124–1132.

Christensen, R. (2005), “Testing Fisher, Neyman, Pearson, and Bayes,” The

American Statistician, 59, 121–126.

Cox, D.R. (1982), “Statistical Significance Tests,” British Journal of Clinical

Pharmacology, 14, 325–331.

Edwards, W., Lindman, H., and Savage, L.J. (1963), “Bayesian Statistical

Inference for Psychological Research,” Psychological Review, 70, 193–

242.

Gelman, A., and Loken, E. (2014), “The Statistical Crisis in Science

[online],” American Scientist, 102. Available at http://www.americansc

ientist.org/issues/feature/2014/6/the-statistical-crisis-in-science

Gelman, A., and Stern, H.S. (2006), “The Difference Between ‘Significant’

and ‘Not Significant’ is not Itself Statistically Significant,” The American

Statistician, 60, 328–331.

Gigerenzer, G. (2004), “Mindless Statistics,” Journal of Socioeconomics, 33,

567–606.

Goodman, S.N. (1999a), “Toward Evidence-Based Medical Statistics 1: The

P-Value Fallacy,” Annals of Internal Medicine, 130, 995–1004.

——— (1999b), “Toward Evidence-Based Medical Statistics. 2: The Bayes

Factor,” Annals of Internal Medicine, 130, 1005–1013.

——— (2008), “A Dirty Dozen: Twelve P-Value Misconceptions,” Seminars

in Hematology, 45, 135–140.

Greenland, S. (2011), “Null Misinterpretation in Statistical Testing and its

Impact on Health Risk Assessment,” Preventive Medicine, 53, 225–228.

——— (2012), “Nonsignificance Plus High Power Does Not Imply Support

for the Null Over the Alternative,” Annals of Epidemiology, 22, 364–368.

Downloaded by [University of California, Los Angeles (UCLA)] at 09:36 06 July 2016 

936 The Promise of Evidence-Based Policymaking
EDITORIAL 133

Greenland, S., and Poole, C. (2011), “Problems in Common Interpretations of Statistics in Scientific Articles, Expert Reports, and Testimony,”

Jurimetrics, 51, 113–129.

Hoenig, J.M., and Heisey, D.M. (2001), “The Abuse of Power: The Pervasive

Fallacy of Power Calculations for Data Analysis,” The American Statistician, 55, 19–24.

Ioannidis, J.P. (2005), “Contradicted and Initially Stronger Effects in Highly

Cited Clinical Research,” Journal of the American Medical Association,

294, 218–228.

——— (2008), “Why Most Discovered True Associations are Inflated” (with

discussion), Epidemiology 19, 640–658.

Johnson, V.E. (2013), “Revised Standards for Statistical Evidence,”

Proceedings of the National Academy of Sciences, 110(48),

19313–19317.

——— (2013), “Uniformly Most Powerful Bayesian Tests,” Annals of Statistics, 41, 1716–1741.

Lang, J., Rothman K.J., and Cann, C.I. (1998), “That Confounded P-value”

(editorial), Epidemiology, 9, 7–8.

Lavine, M. (1999), “What is Bayesian Statistics and Why Everything Else is

Wrong,” UMAP Journal, 20, 2.

Lew, M.J. (2012), “Bad Statistical Practice in Pharmacology (and Other

Basic Biomedical Disciplines): You Probably Don’t Know P,” British

Journal of Pharmacology, 166, 5, 1559–1567.

Phillips, C.V. (2004), “Publication Bias In Situ,” BMC Medical Research

Methodology, 4, 20.

Poole, C. (1987), “Beyond the Confidence Interval,” American Journal of

Public Health, 77, 195–199.

——— (2001), “Low P-values or Narrow Confidence Intervals: Which are More Durable?” Epidemiology, 12,

291–294.

Rothman, K.J. (1978), “A Show of Confidence” (editorial), New England

Journal of Medicine, 299, 1362–1363.

——— (1986), “Significance Questing” (editorial), Annals of Internal

Medicine, 105, 445–447.

———- (2010), “Curbing Type I and Type II Errors,” European Journal of

Epidemiology, 25, 223–224.

Rothman, K.J., Weiss, N.S., Robins, J., Neutra, R., and Stellman, S. (1992),

“Amicus Curiae Brief for the U. S. Supreme Court, Daubert v. Merrell Dow Pharmaceuticals, Petition for Writ of Certiorari to the United

States Court of Appeals for the Ninth Circuit,” No. 92-102, October

Term, 1992.

Rozeboom, W.M. (1960), “The Fallacy of the Null-Hypothesis Significance

Test,” Psychological Bulletin, 57, 416–428.

Schervish, M.J. (1996), “P-Values: What They Are and What They Are Not,”

The American Statistician, 50, 203–206.

Simmons, J.P., Nelson, L.D., and Simonsohn, U. (2011), “False-Positive

Psychology: Undisclosed Flexibility in Data Collection and Analysis

Allows Presenting Anything as Significant,” Psychological Science, 22,

1359–1366.

Stang, A., and Rothman, K.J. (2011), “That Confounded P-value Revisited,”

Journal of Clinical Epidemiology, 64, 1047–1048.

Stang, A., Poole, C., and Kuss, O. (2010), “The Ongoing Tyranny of Statistical Significance Testing in Biomedical Research,” European Journal of

Epidemiology, 25, 225–230.

Sterne, J. A. C. (2002). “Teaching Hypothesis Tests—Time for Significant

Change?” Statistics in Medicine, 21, 985–994.

Sterne, J. A. C., and Smith, G. D. (2001), “Sifting the Evidence—What’s

Wrong with Significance Tests?” British Medical Journal, 322, 226–

231.

Ziliak, S.T. (2010), “The Validus Medicus and a New Gold Standard,” The

Lancet, 376, 9738, 324–325.

Ziliak, S.T., and McCloskey, D.N. (2008), The Cult of Statistical Significance:

How the Standard Error Costs Us Jobs, Justice, and Lives, Ann Arbor, MI:

University of Michigan Press.

Downloaded by [University of California, Los Angeles (UCLA)] at 09:36 06 July 2016 

The Promise of Evidence-Based Policymaking 937
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: October 11, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sef-wigq

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0007

Comment on FR Doc # 2016-22002

Submitter Information

Name: Adam Gamoran

Address:

New York, NY, 10022

Email: agamoran@wtgrantfdn.org

General Comment

Thank you for this opportunity to address the challenges faced by the Commission on

Evidence-Based Policymaking and to share my views on what the Commission may

realistically hope to accomplish. My comments, attached here, draw on the statement

that my colleagues and I prepared, "From Data to Evidence to Policy," as well as on my

long experience as an analyst of large-scale data sets from administrative records, survey

studies, and field experiments.

Attachments

Goals for the Commission + Data to Evidence packet_WTG

938 The Promise of Evidence-Based Policymaking
October 11, 2016

Dr. Katharine Abraham

Chair, Commission on Evidence-Based Policymaking

University of Maryland

3114 Tydings Hall, 7343 Preinkert Dr.

College Park, MD 20742

Ron Haskins 

Co-Chair, Commission on Evidence-Based Policymaking

The Brookings Institution

1775 Massachusetts Ave., NW

Washington, DC 20036

Dear Katharine and Ron,

Thank you for this opportunity to address the challenges faced by the Commission on Evidence-Based 

Policymaking and to share my views on what the Commission may realistically hope to accomplish. My 

comments draw on the statement that my colleagues and I prepared, From Data to Evidence to Policy, as 

well as on my long experience as an analyst of large-scale data sets from administrative records, survey 

studies, and field experiments.

In our statement, we offer four recommendations to the Commission:

x Adopt a broad reading of the Commission’s mandate to include not just linking and sharing 

administrative data, but preparing data for use by researchers inside and outside of government.

x Advance a framework that recognizes different forms of evidence and varied uses of linked 

administrative data.

x Recommend ways to strengthen the federal infrastructure for producing research evidence that can 

inform policymaking.

x Recommend to federal agencies that they support partnerships between researchers and 

policymakers to enhance the use of research evidence for smart policy decisions.

In this letter, I will identify three specific goals to which the Commission may aspire that would respond to 

these recommendations and the Commission’s mandate.

1. Devise a standardized process for linking administrative data across federal 

government agencies, and between federal agencies and willing state partners, that 

protects the privacy of individuals while facilitating research.

Many proof cases exist that demonstrate the feasibility of producing valuable research evidence by 

linking federal administrative data to surveys, experimental data, or other administrative records. 

Moreover, we know that the data can be linked in secure facilities using protected identity codes that 

ensure privacy. At present, however, each researcher using the data must start the process anew, which 

poses nearly impassible barriers for researchers and unreasonable burdens on federal agencies and 

state partners. The Commission can address these challenges by identifying an approach, or a set of 

approaches, that allows researchers to access and use data routinely, in secure facilities, while also 

satisfying the technical demands of privacy protection.

570 Lexington Avenue, 18th Floor New York, NY 10022-6837 · T: 212.752.0071 · F: 212.752.1398 · wtgrantfoundation.org 

The Promise of Evidence-Based Policymaking 939
Page 2 

One example that may guide the Commission in this regard comes from the experience of the National 

Center for Education Statistics, which planned a new longitudinal survey with oversamples in ten states 

whose education agencies had agreed in principle to link their administrative data to the surveys. This 

would have created a valuable new resource for addressing critical education policy questions, yet 

bureaucratic and legal barriers ultimately prevented data sharing by any of the ten states. The 

Commission can overcome such barriers with a framework that articulates the technical requirements,

legal standing, and accepted procedures for linking and sharing data across federal agencies and with 

willing state partners. 

In providing a framework to standardize data sharing, the Commission can specify the different types of 

data that may be linked—including administrative, survey, and experimental data—and the different 

purposes of such linkages, including program continuation decisions, program improvement plans, and 

obtaining accurate understanding of the nature of challenges we face as a nation (such as the link 

between geography and economic mobility as demonstrated in recent testimony to the Commission by 

Raj Chetty).

The Commission might further recommend a pilot that would begin with specific agencies that, based 

on the investigations of the Commission, seem best prepared to implement a standardized procedure 

for linking data and making them accessible to researchers inside and outside of government. For 

instance, Chapin Hall at the University of Chicago, with support from the Laura and John Arnold 

Foundation and in partnership with the U.S. Census Bureau, is sponsoring a series of studies to 

demonstrate innovative approaches to linking administrative data, and the Commission might look to 

these studies as examples.

2. Recommend specific infrastructure elements that the federal government may adopt 

to support the use of evidence in policymaking.

The Commission’s mandate includes not only a directive to recommend ways to link and share 

administrative data, but also a call for guidance on ways that data may be used to create evidence that 

can guide policy. The Commission can respond to this challenge by examining infrastructure supports 

that are found in various agencies, identifying best practices, and recommending their wide adoption. 

Examples of such infrastructure elements include:

x leadership positions focused on evidence use, such as the special advisor for evidence-based 

policy in OMB and the chief technology officer in OSTP;

x interagency collaborative bodies, such as the federal chief information officers council and the 

Committee on National Statistics of the National Academies;

x a dedicated office for research and evaluation within each federal agency, such as the office of 

the assistant secretary for planning and evaluation within HHS, and the office of policy 

development and research within HUD;

x a codified set of principles and practices for evidence use, analogous to Principles and Practices 

for a Federal Statistical Agency and Common Guidelines for Education Research and 

Development; and

x policies and legislation to prioritize programs that strengthen the use of evidence and 

innovation (as modeled by OMB in its FY15 budget memorandum) and facilitate data-sharing 

across agencies to assess and improve programs.

3. Examine models of partnerships that support the use of research evidence in 

policymaking, and recommend federal support for effective partnerships.

570 Lexington Avenue, 18th Floor New York, NY 10022-6837 · T: 212.752.0071 · F: 212.752.1398 · wtgrantfoundation.org

940 The Promise of Evidence-Based Policymaking
Page 3 

All too often, evidence about effective or ineffective policies or programs has little bearing on decisions, 

even when the evidence is rigorous, timely, and accessible. But sustained partnerships between 

researchers and policymakers can improve the use of research evidence by allowing for ongoing 

dialogue between researchers and decision makers. Indeed, partnerships that offer a basis for trust, 

incentivize researchers to address questions that really matter, and create a culture of evidence in the 

decision-making body can improve the use of research evidence in policy and practice.

The Commission can improve such use by identifying effective partnership models and promoting them 

across the federal government. Many models of partnerships exist, such as those outlined by the 

William T. Grant Foundation and the Forum for Youth Investment in their online guide for building 

and sustaining partnerships in education, located here: rpp.wtgrantfoundation.org.

Of course, these goals will not be achieved simply. But the Commission on Evidence-Based Policymaking is 

a rare and unique opportunity with tremendous potential, and we believe that the scale of this potential 

warrants equally ambitious goals. I would be pleased to discuss these recommendations and the specific 

strategies for achieving them, individually or with the Commission as a whole, if that would be helpful.

Sincerely,

Adam Gamoran, President

570 Lexington Avenue, 18th Floor New York, NY 10022-6837 · T: 212.752.0071 · F: 212.752.1398 · wtgrantfoundation.org

The Promise of Evidence-Based Policymaking 941
 

From Data to Evidence to Policy 

Recommendations for the Commission on Evidence-Based Policymaking 

MAY, 2016 

The William T. Grant Foundation

The William T. Grant Foundation supports research to improve the lives of young people ages 5–25 in the United 

States. Our goal is to accumulate a body of knowledge that will advance theory, policy, and practice and contribute 

to improved outcomes and opportunities for youth, today and in the future. In 2014, we launched an initiative to 

invest in research to identify effective responses to inequality in its many forms. And since 2009, we have supported 

studies that provide insight into how policymakers, administrators, and service providers acquire, interpret, and use 

research evidence. In 2015, we signaled a new direction in this initiative, calling for studies that identify, create, and 

test strategies to improve the use of research evidence in ways that benefit youth. 

The Forum for Youth Investment 

The Forum for Youth Investment is a nonprofit, nonpartisan "action tank" dedicated to helping communities and the 

nation make sure all young people are Ready by 21®: ready for college, work and life. A trusted resource for policy 

makers, advocates, researchers and program professionals, the Forum provides youth and adult leaders with the 

information, connections and tools they need to create greater opportunities and outcomes for young people. The 

Forum manages a number of centers and projects, including Big Picture Approach Consulting, the David P. Weikart 

Center for Youth Program Quality, the Children’s Cabinet Network and SparkAction. The core work of the Forum is 

helping leaders, organizations, partnerships and systems – at the local, state and national levels – assess, improve 

and align their practices and policies.

942 The Promise of Evidence-Based Policymaking
 

1 | Page

From Data to Evidence to Policy 

Recommendations for the Commission on Evidence-Based Policymaking 

The Forum for Youth Investment and the William T. Grant Foundation are pleased to submit the 

following recommendations to the Commission on Evidence-Based Policymaking. We are encouraged by 

the Commission’s potential to promote the use of research evidence in policymaking, and we offer these 

insights with the hope of helping the Commission make the most of its historic opportunity. 

These suggestions are drawn from our experience as conveners of a learning group of senior career staff 

and appointees in research offices focused on children, youth, and families within the U.S. Departments 

of Education, Labor, Justice, and Health and Human Services, as well as in the Corporation for National 

and Community Service and the National Science Foundation. These agencies invest in research and 

evaluation to build policy-relevant evidence and will likely be charged with implementing many of the 

Commission’s recommendations. They also have experience responding to similar challenges in the past. 

For these reasons, their insights may guide the Commission in developing a transformational set of 

recommendations. 

The Path from Data to Evidence to Policy 

A narrow interpretation of the Commission on Evidence-Based Policymaking legislation might suggest 

the Commission’s charge is primarily to determine how the federal government can share and link 

administrative data sets. Accomplishing this aim would be of significant value. But a broader 

interpretation of its charge suggests that the Commission must not stop there. It should also consider 

how the federal government can use data to create the evidence required for smart policy decisions, as 

well as how to create the infrastructure to support the use of evidence in policymaking. 

We urge the Commission to prioritize those elements of its charge that point toward these broader 

aims: 

x Emphasize how data “may be integrated and made available to facilitate program evaluation, 

continuous improvement, policy-relevant research, and cost-benefit analyses by qualified 

researchers and institutions;” and 

x Examine “how data and results of research can be used to inform program administrators and 

policymakers to improve program design.” 

Sharing and linking data is necessary but not sufficient to achieve these goals. The data must be 

deployed in research and evaluation to create evidence, which must be then be used to inform 

policymaking. Laying the groundwork for the path from data to evidence to policy is essential to the 

Commission’s charge. 

 

Sharing and 

Linking Data

Using Data to 

Create 

Evidence

Using Evidence 

to Inform 

Policymaking

The Promise of Evidence-Based Policymaking 943
 

2 | Page

Sharing and Linking Data 

The Commission should recommend that data are collected and 

shared in ways that facilitate their use in research and evaluation. 

Sharing data is a valuable first step. But the data take on added 

power when they can be used in research and evaluation studies. 

The Commission can fulfill its charge of addressing “how data and 

results can be used to inform program administrators and 

policymakers” by recommending ways that administrative data 

systems can be made ready for use in research and evaluation. The 

Commission could craft recommendations that would help ensure 

that data are linked; are of sufficient quality; and are delivered in 

formats that encourage their use in research that answers 

policymakers’ questions. 

Linking separate data sets together increases the value of each for 

research and evaluation. One federal staff member in our learning 

group commented, “I am working on linking existing databases to 

conduct research. I need to figure out what kinds of data are being 

collected....Who is tracking relevant outcomes, and how do you 

synthesize that with community-level data from multiple sources to 

tell the impact of interventions across multiple domains?” 

Linking existing data can also enable faster and cheaper research studies. As one participant reported, 

“There has been a lot of thinking overall in our department on the opportunities these longitudinal data 

systems have for low cost evaluations. A whole group of people in our program offices are focused on 

this right now.” But using data that are not designed for research can be challenging. Sometimes the 

quality of the data is poor or unknown. As a learning group participant put it, “It is not easy to figure out 

if a given set of administrative data is high-quality enough to be used in a research context.” 

Sometimes the ways that the data are shared make it difficult to produce useful research reports. As 

one learning group participant shared, “Only one or two people in each of our agencies know how to 

manipulate specific databases, and those people have a long list of requests from multiple agencies to 

do specific data runs. We are hoping that we can take on the burden of getting the data ready. A lot of 

the data we have, like those that manage case files, were not created to be used for research, so that is 

a heavier lift. We want to give agencies a more realistic view of the data they actually have.” 

Using Data to Create Evidence 

The Commission should recommend that federal agencies adopt a broad and inclusive view of the types of 

research studies that can and should be produced with administrative data. 

The power of data increases when they are used to create research evidence (see sidebar). Too often, 

however, a narrow conceptualization of data as evidence limits the ability of policymakers to gain full 

understanding of an issue. The Commission could add value to the field by advancing a framework that 

delineates the types of research evidence that should be created to guide policymaking. 

The William T. Grant Foundation 

defines research evidence as “a 

type of evidence derived from 

applying systematic methods 

and analyses to address a 

predefined question or 

hypothesis. This includes 

descriptive studies, intervention 

or evaluation studies, metaanalyses, and cost-effectiveness 

studies conducted within or 

outside research organizations.” 

DEFINITION OF RESEARCH 

EVIDENCE 

944 The Promise of Evidence-Based Policymaking
 

3 | Page

The Commission may wish to reinforce frameworks such as the Institute of Education Sciences/National 

Science Foundation’s Common Guidelines for Education Research Development framework1

 and the 

Health and Human Services Administration for Children and Families’ Common Framework for Research 

& Evaluation.2

 Adopting these types of frameworks would help ensure that policymakers not only 

receive findings from the full range of types of research evidence, but review this evidence with a clear 

understanding of the level of rigor and quality inherent in each type of study, and what types of 

questions are answerable by each respective research methodology.3

 

Using Evidence to Inform Policymaking 

The Commission should recommend that federal agencies support partnerships between researchers and 

policymakers that inform key research questions and facilitate the use of research evidence. 

Partnerships between researchers and policymakers can improve the use of research evidence by 

guiding researchers to ask questions that respond to the needs of policymakers, building stronger 

practice-focused research networks or community-based participatory approaches, and creating a 

culture of learning in which administrators, policymakers and other government leaders include 

research evidence in their deliberations. 

The growing literature about how and when research evidence is and is not used in policymaking can 

inform the Commission’s work. These studies “complicate the common conception of research users as 

merely rational actors who have questions, go in search of research to answer them, and then apply it to 

their decisions.... In none of their cases does research use easily boil down to a single moment or an 

isolated decision....It is not a simple process whereby research ‘facts’ are passed from researchers to 

research users and then applied in a linear decision making process. Instead, research use is contingent, 

interactive, and iterative. It involves people individually and collectively engaging with research over 

time, bringing their own and their organization’s goals, motivations, routines, and political contexts with 

them.”4

 It is often helpful to construct evidence in a process of engagement, in which the needs of 

decision makers help shape research questions and findings are delivered in an accessible and timely 

fashion through relationships of trust and mutual understanding. 

The Commission should recommend ways to strengthen the federal infrastructure for producing research 

evidence that can inform policymaking. 

To address its charge to examine how data may be “made available to facilitate program evaluation, 

continuous improvement, policy-relevant research, and cost-benefit analyses by qualified researchers 

and institutions,” it is important for the Commission to create recommendations for strengthening the 

federal infrastructure for using data to create evidence, and using that evidence to inform policymaking. 

 

1

 Available at: http://www.nsf.gov/pubs/2013/nsf13126/nsf13126.pdf 

2 Available at: http://www.acf.hhs.gov/programs/opre/resource/the-administration-for-children-familiescommon-framework-for-research-and-evaluation

3

 While there may not yet be widely accepted quality standards for the all types of research, some of our learning 

group members pointed to the following article as a promising start: Gottfredson, D. C., Cook, T. D., Gardner, F. E. 

M., Gorman-Smith, D., Howe, G. W., Sandler, I. N., & and Zafft, K. M. (2015). Standards of evidence for efficacy, 

effectiveness, and scale-up research in prevention science: Next generation. Prevention Science, 16, 893-926. 4

 Tseng, V. & Nutley, S. (2014). Building the Infrastructure to Improve the Use and Usefulness of Research in 

Education. In K.S. Finnigan & A.J. Daly (Eds.). Using Research Evidence in Education: From the Schoolhouse Door to 

Capitol Hill. Springer: New York. Available at: http://wtgrantfoundation.org/resource/building-the-infrastructure-toimprove-the-use-and-usefulness-of-research-in-education

The Promise of Evidence-Based Policymaking 945
 

4 | Page

An infrastructure could include elements such as formal policies, codified practices, established offices, 

and interagency coordinating structures. 

Fortunately, there are existing efforts that the Commission could build upon. The Department of Labor 

and the Department of Health and Human Services, Administration for Children and Families both 

created evaluation policies that can serve as models for other agencies.5

 The Department of Labor 

created a Chief Evaluation Office, and conducts an annual survey to assess its performance meeting the 

research needs of program offices. HHS Administration for Children and Families, Office of Planning 

Research and Evaluation also published a set of principles to guide all of its evaluation work.6

 Further, 

the Institute of Education Sciences has protections, granted by Congress in its authorizing language, that 

support scientific integrity and independence from political influence. 

Additionally, a subset of learning group participants funded a National Academy of Sciences Roundtable 

to consider an infrastructure for evaluation that parallels the existing infrastructure for statistical 

agencies. (Federal statistical agencies receive support from a Chief Statistician housed at the White 

House Office of Management and Budget; a public-private, interagency Committee on National 

Statistics; and a carefully codified and updated set of Principles and Practices for a Federal Statistical 

Agency.7

) The Commission may wish to be briefed on this effort. The Commission may also wish to be 

briefed on the types of infrastructure that have been created to support federal agencies’ performance 

management functions, such as the role the federal Performance Improvement Council plays in 

fostering widespread and effective use of performance management practices across federal agencies 

and sparking cross-cutting performance improvements.8

Conclusion 

The creation of the Commission on Evidence-Based Policymaking is well timed. As shared in the William 

T. Grant Foundation’s recent blog series Evidence at the Crossroads, “research evidence can improve 

public policies and programs, but fulfilling that potential will require honest assessments of current 

initiatives, coming to terms with outsized expectations, and learning ways to improve social 

interventions and public systems.”9

The Commission is well positioned to drive this work forward, especially if it focuses on the full 

continuum of activity from sharing and linking data, to using those data to create research evidence, to 

using that evidence to inform policymaking. 

 

5

 See http://www.dol.gov/asp/evaluation/EvaluationPolicy.htm and 

http://www.acf.hhs.gov/programs/opre/resource/acf-evaluation-policy 6

 The HHS principles are posted at: http://www.acf.hhs.gov/opre/resource/acf-evaluation-policy 7

 Principles and practices for federal statistics agencies are posted at: 

http://sites.nationalacademies.org/DBASSE/CNSTAT/Principles_and_Practices_for_a_Federal_Statistical_Agency/i

ndex.htm

8

 On the role of the Performance Improvement Council, see: http://www.gsa.gov/portal/content/133807 9

 See: http://wtgrantfoundation.org/tag/evidence-at-the-crossroads

946 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: October 13, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sfr-rwyy

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0008

Comment on FR Doc # 2016-22002

Submitter Information

Name: Stephen Crawford

Address:

Washington, MD, 20052

Email: crawford@gwu.edu

General Comment

One simple way to vastly improve evidence-based policymaking regarding higher

education would be to repeal the existing ban on a federal student unit record system.

Lifting that ban would enable EXISTING data systems to communicate with one another

and thereby answer critical questions that can't be answered now, especially questions

about the OUTCOMES of higher education programs in which the federal

government/taxpayers invest billions of dollars. These questions include: how part-time

and older students fare; what happens to the large number of students who transfer from

one college to another; how many and which students complete their programs at

specific colleges, especially students receiving federal Pell grants ($30 billion per year);

and whether graduates are able to obtain jobs that pay enough to meet their student loan

obligations.

Supporters of the existing ban express say it is needed to protect student privacy.

However, major higher education associations that once supported that position, notably

the Association of Public and Land-grant Universities and the National Association of

Student Financial Aid Administrators, now back creating a Student Unit Record. A

The Promise of Evidence-Based Policymaking 947
recent statement to CEP by New America's Rachel Fishman shows how the privacy

concerns can be addressed without a ban: "Creating a Student Unit Record wold not

require the collection of additional student data, but would allow the connecting of

exiting data already held by a variety of federal and state agencies. Protecting these data

al all points of the lifecycle is crucial, and it is worth considering housing such a system

in the Department of Education's National Center for Education Statistics, which is

classified as a statistical agency and therefore subject to strict privacy and security

requirements..."

Some states (IN, TX) and universities are now developing their own workarounds to the

ban, for they feel a keen need to obtain and share better data on student outcomes. Such

initiatives are commendable, but underline the obsolescence of the ban on a federal

student unit record and raise concerns about data consistency and comparability across

states.

In sum, the federal government could realize and promote a much higher return on its

and the country's enormous investments in higher education if there were a federal

student unit record system that enabled existing data systems to realize their potential for

informing more evidence-based policymaking.

948 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: October 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sge-2upn

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0009

Comment on FR Doc # 2016-22002

Submitter Information

Name: Stephanie Shipman

Address:

Washington, DC, 20548

Email: shipmans@gao.gov

General Comment

See attached file(s) from the Federal Evaluators network.

Attachments

FEDEVAL_COMMENTS_ON_COMMISSION_SOLICITATION

The Promise of Evidence-Based Policymaking 949
Submitted 10/14/2016 

1 

Docket # USBC-2016-0003: Commission on Evidence-Based Policymaking Comments 

from Federal Evaluators 

On September 15, 2016, members of the Federal Evaluators network met with the Executive 

Director and Policy and Research Director for the Commission on Evidence-Based 

Policymaking at the Government Accountability Office (GAO) headquarters to discuss the 

Commission’s planned work. (Federal Evaluators is an informal network of employees in the 

executive and legislative branches of the U.S. federal government.) Participants came from the 

Departments of Agriculture, Commerce, Education, Health and Human Services, Homeland 

Security, Housing and Urban Development, Labor, State, Transportation, and Veterans Affairs; 

GAO, the Congressional Research Service; Corporation for National and Community Service; 

Environmental Protection Agency; Holocaust Museum; National Endowment for the Arts; 

National Science Foundation; Office of Management and Budget (OMB); Office of Personnel 

Management (OPM); Peace Corps; and Small Business Administration. To encourage candid 

discussion, we agreed that participants would speak for themselves, not their agencies. These 

comments represent the opinions of individual attendees, not necessarily all. 

This summary, prepared by GAO staff, organizes the issues raised around the questions posed 

by the Commission in a Request for Comments posted in the Federal Register on September 

14, 2016 https://www.regulations.gov/document?D=USBC-2016-0003-0001. 

This summary was shared with the full Federal Evaluators membership (about 1500) for 

additional comments, before being submitted to the Commission. The Commission staff 

announced the Commission’s plans to conduct additional outreach to federal agencies and hold 

additional public hearings, and invited members to sign up for their listserv. 

#1. Are there successful frameworks, policies, practices and methods to overcome challenges 

related to evidence-building from state, local and/or international governments the Commission 

should consider...?

- See the following examples for lessons learned in using state datasets: Research Data 

Assistance Center for access to Medicare and Medicaid program claims data 

http://www.resdac.org/, Education’s Race to the Top state grant program 

http://www2.ed.gov/programs/racetothetop/index.html , GAO analysis of states’ use of 

American Recovery and Reinvestment Act grants. http://www.gao.gov/recovery/ 

- The Institute of Medicine set up a framework for quality of care that has helped health 

care agencies focus their efforts. One aspect of that is identifying clear policy questions 

to focus on. Linking data is most successful when there is a strong policy question to 

answer. 

- The Inter-University Consortium for Political and Social Research (ICPSR) at the 

University of Michigan maintains a data archive of social science research files 

(including federal databases) since the 1960s and provides leadership and training in 

data access, curation, and analysis for the research community. 

http://www.icpsr.umich.edu/icpsrweb/ 

- The Patient-Centered Outcomes Research Institute (PCORI) has established a national 

network for conducting comparative clinical effectiveness research by establishing a 

resource of clinical data gathered and stored in standardized formats. They have worked 

on the legal and ethical issues involved in data release, sensitivity, and security. 

www.pcori.org 

 

950 The Promise of Evidence-Based Policymaking
Submitted 10/14/2016 

2 

#4. What data-sharing infrastructure should be used to facilitate data merging, linking, and 

access for research, and analysis purposes? 

- While the formally recognized statistical agencies have principles and practices in place 

to facilitate data sharing, most of the Departments and smaller agencies do not, so this 

would require a lot of training and rulemaking for the program offices who hold the data 

of interest. 

- Despite a congressional mandate for all foreign assistance agencies to supply data to a 

foreign assistance dashboard, agencies lack the money required to change their 

financial systems to produce about 40 percent of the required reporting elements. 

- Agencies will need to standardize or at least loosen restrictions on data access. 

#5. What challenges currently exist in linking state and local data to federal data?

- In social programs it is often local grantees that collect program performance data that 

are then aggregated at the state level for federal reporting. Issues of capacity to collect 

reliable data are compounded when sub-grantees are involved. 

- Local communities often need small area estimates for which there may be significant 

quality issues. 

#7a. What data should be included in a potential US government data clearinghouse(s)? 

[The Commission is aware that, under Open Data, agencies have inventoried over 70,000 

datasets, with a potential additional 70,000 non-public datasets. This would be prohibitively 

difficult to work with so the Commission is considering creating a list of the top 20 to work with.] 

- Some of the most useful data for assessing program outcomes (health, education, 

employment and income status) are actually state data, not federal data. Accessing 

them requires separate negotiation with the individual states as well as addressing data 

compatibility across states. 

- Policymakers’ needs should be more prominent in this discussion. GPRA’s lesson was 

that supply was not as important as demand for use of evidence in decision making. 

What is the Hill’s agenda and timing? What is the agency’s evaluation agenda? 

- Agencies struggle with prioritizing evaluation questions, so the Commission could help 

by developing a framework for prioritizing evaluation questions. 

#7b. What are the current legal and administrative barriers to including such data in a 

clearinghouse or linking the data?

- Several databases contain highly sensitive national security data that do not seem 

appropriate for this use, or would require different security protocols. 

- Some programs (e.g., child welfare) explicitly prohibit releasing data with the personally 

identifiable information that is specifically required to link with other datasets. 

- Private sector companies lack a common identifier that can be used to link data on their 

contacts with various federal bureaus. 

- Data definitions, etc. vary tremendously across cash and food assistance programs. 

Data quality also varies across datasets, so what mechanism could we use to grade 

data quality, and to enforce data standardization? 

- The Department of Education requires the evaluator to demonstrate a valid educational 

purpose to combine their data with data from another federal program. This constrains 

the ability to examine cross-cutting issues. 

- The length of the Paperwork Reduction Act review process at OMB (for example, 2 

years) is a considerable barrier to conducting evaluations in a timely manner. 

- Weak design and implementation of data quality controls; e.g., GAO recent assessment 

of the reliability of OPM payroll data. http://www.gao.gov/products/GAO-17-127 

The Promise of Evidence-Based Policymaking 951
Submitted 10/14/2016 

3 

#8. What factors or strategies should the Commission consider for how a clearinghouse could 

be self-funded?

- Medicaid charges a federal agency $25,000 for access their claims data; nongovernment users pay more. Researchers consider this a barrier to use. 

 

#10. How should the Commission define “qualified researchers and institutions”?

- The Centers for Medicare and Medicaid Services provide researchers with training and 

technical assistance and access to Medicare and Medicaid data through the Research 

Data Assistance Center (ResDAC) located at the University of Minnesota. 

http://www.resdac.org/ It has a “qualified entity program” defined by the Patient 

Protection and Affordable Care Act. 

- Would government and non-government researchers and entities be treated alike? 

#12. If a clearinghouse were created, what types of restrictions ought to be placed on the uses 

of the data by “qualified researchers and institutions”?

- Note that databases are typically created for specific uses and may not be appropriate 

for other uses. People outside the program can easily misinterpret data leading to 

misuse of data. 

- See the procedures used by ICPSR (and other organizations described above) for 

managing access to restricted use datasets. 

https://www.icpsr.umich.edu/icpsrweb/content/ICPSR/access/restricted/index.html 

#16. How can data, statistics, results of research, and findings from evaluation be best used to 

improve policies and programs?

- We appreciate the Commission’s broad definition of “evidence”, but it would also be 

useful to define ”rigor” and “quality” of evidence, which involves data source, reliability, 

the rigor of the evaluation, internal and external validity, use of mixed methods. Note that 

program administrative data only pertain to program participants, and exclude 

nonparticipants which might form a proper comparison group with which to assess 

program effects. 

- Beware that, in combining administrative data with other data, the quality of the data 

may be compromised. 

- Please recognize that for data to become evidence there first needs to be a research 

question that determines how the data will be analyzed. 

- There is very little guidance out there about how to use data or evidence to inform policy 

– how to analyze and use it. 

- The intermediary step between having data and creating policy is the learning process. 

We use a variety of studies, both summative - to answer whether an approach works - 

and formative - to understand how an approach is implemented in practice. 

- Local communities and federal agencies may have different questions and uses for data. 

#17. To what extent can or should program and policy evaluation be addressed in program 

design?

- The best-known example is the tiered-evidence grants which provide more funding to 

expand approaches that have stronger evaluation evidence of effectiveness and less to 

programs with less or weaker evaluative evidence. There is concern that requiring prior 

evaluation evidence for funding could stifle program innovation. 

- There is also a concern that these tiered-evidence programs tend to rely too narrowly on 

impact evaluations using experimental research designs. Such evidence of internal 

952 The Promise of Evidence-Based Policymaking
Submitted 10/14/2016 

4 

validity is important but not sufficient for replication, scale-up (external validity 

assessment), and learning how the program works, and is often not appropriate for 

some types of prevention programs. 

The Promise of Evidence-Based Policymaking 953
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: October 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sgj-iuzd

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0010

Comment on FR Doc # 2016-22002

Submitter Information

Name: Christel Swasey

Address:

Pleasant Grove - Utah, 84062

Email: 212christel@gmail.com

General Comment

Dear Commission on Evidence-Based Policymaking,

I love the American concept of voter-based, Constitution-based, elected representativebased, policymaking. It's why I live in America. In contrast to voter-based policymaking

there is evidence-based policymaking, which I don't love because it implies that one

entity's "evidence" trumps individuals' consent to new policy changes.

Former Secretary of Agriculture Ezra Taft Benson said something about education that

also applies to educational data and policymaking:

"The best way to prevent a political faction or any small group of people from capturing

control of the nation's educational system is to keep it decentralized into small local

units... This may not be as efficient as one giant super educational system (although

bigness is not necessarily efficient, either) but it is far more safe. There are other factors,

too, in favor of local and independent school systems. First, they are more responsive to

the needs and wishes of the parents and the community. The door to the school

954 The Promise of Evidence-Based Policymaking
superintendent's office is usually open to any parent [or teacher]... But the average

citizen would be hard pressed to obtain more than a form letter reply from the national

Commissioner of Education in Washington, D.C."

Local control, and consent of the governed, are two foundational principles in our great

nation.

Because the CEP is not an elected body, it does not hold authority to collect, or to

recommend collection, of student-level evidence, or of any evidence, without written

consent; and, for the same reasons, neither does the Department of Education.

Because the fifty, federally-designed, evidence-collecting, State Longitudinal Database

Systems never received any consent from the governed in any state to collect data on

individuals (as the systems were put into place not by authority, but by grant money) it

follows that the idea of having CEP study the possible removal of barriers to federal

access of those databases, is an egregious overstep that even exceeds the overstep of the

State Longitudinal Database Systems.

Because federal FERPA regulations altered the original protective intent of FERPA, and

removed the mandate that governments must get parental (or adult student) consent for

any use of student level data, it seems that the idea of having CEP study and possible

influence removal of additional "barriers" to federal use of data, is another egregious

overstep.

As a licensed teacher in the State of Utah; as co-founder of Utahns Against Common

Core (UACC); as a mother of children who currently attend public, private and home

schools; as acting president of the Utah Chapter of United States Parents Involved in

Education (USPIE); as a patriot who believes in "consent of the governed" and in the

principles of the U.S. Constitution; and, as a current tenth grade English teacher, I feel

that my letter represents the will of many who stand opposed to the study of the removal

of protective barriers on student-level data, which the CEP's website has outlined it will

do.

I urge this commission to use its power to strengthen local control of data, meaning

parental and teacher stewardship over student data, instead of aiming to broaden the

numbers of people with access to personally identifiable student information to include

government agencies and/or educational sales/research corporations such as Pearson,

Microsoft, or the American Institutes for Research.

To remove barriers to federal access of student-level data only makes sense to a socialist

who agrees with the Marc Tucker/Hillary Clinton 1998 vision of a cradle-to-grave nanny

state with "large scale data management systems" that dismiss privacy as a relic in

subservience to modern government. It does not make sense to those who cherish local

control.

The Promise of Evidence-Based Policymaking 955
It is clear that there is a strong debate about local control and about consent of the

governed, concerning data and concerning education in general. NCEE Chair Mark

Tucker articulated one side of the debate when he said: "the United States will have to

largely abandon the beloved emblem of American education: local control. If the goal is

to greatly increase the capacity and authority of the state education agencies, much of the

new authority will have to come at the expense of local control."

Does that statement match the philosophical stand of this commission? I hope not. Local

control means individual control of one's own life. How would an individual control his

or her own destiny if "large scale data management systems" in a cradle-to-grave system,

like the one that Tucker and Clinton envisioned, override the right to personal privacy

and local control? It is not possible.

I urge this commission to use any influence that it has to promote safekeeping of unitrecord data at the parental and teacher level, where that authority rightly belongs.

Sincerely,

Christel Swasey

956 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: October 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sgr-pcca

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0011

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

I see that you've 'redacted' the previous six comments! At least you're reading them, but

they're hidden which is censorship.

The constitution is still in effect at the moment, just a reminder.

Well, the short of what I have to say is, NO. I refuse the government agencies, state

agencies, any agencies to gather data and / or ask my child questions relating to

anything.

You do not have the authority to do so. The American public does not want you to.

What you're doing is illegal.

Here comes Donald Trump to the rescue, Thank GOD.

The Promise of Evidence-Based Policymaking 957
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: October 16, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8shw-6jt0

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0012

Comment on FR Doc # 2016-22002

Submitter Information

Name: Lynne Taylor

General Comment

As an avid anti federal legislation in education researcher and writer, I find that this is

not only, once again, a violation of our U.S. Constitution, but, frankly, a violation of

ethics, morals, a our privacy. This Panel is too closely tied to the newly created publicprivate Artificial Intelligence Council of which you will find some of the same backers

involved in this as well. 

ESSA (Every Student Achieves Act) also rubber stamps MORE federally dictated/led

education policy. Have we learned NOTHING in the past? It is not right, legal, or

binding that the U.S. is making such global strides to not only data track and mine our

citizens via education or our workforce! This moves are being made to rather satisfy an

agreement made with the United Nations. We the people did not get to voice that we

wish to remain American, not global citizens. Why do I bring this up? Education for all

is among the Goals in the US/UN agreement. Again, no voice, no vote by the citizens.

958 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: October 19, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sjs-bwo9

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0013

Comment on FR Doc # 2016-22002

Submitter Information

Name: JUdy Bergfield

General Comment

Dear Commission on Evidence-Based Policymaking,

I love the American concept of voter-based, Constitution-based, elected representativebased, policymaking. It's why I live in America.

In contrast to voter-based policymaking there is evidence-based policymaking, which I

don't love because it implies that one entity's "evidence" trumps individuals' evidence, or

trumps individuals' consent to policy changes.

Former Secretary of Agriculture Ezra Taft Benson said something about education that

also applies to educational data and policymaking:

"The best way to prevent a political faction or any small group of people from capturing

control of the nation's educational system is to keep it decentralized into small local

units, each with its own board of education and superintendent. This may not be as

efficient as one giant super educational system (although bigness is not necessarily

efficient, either) but it is far more safe. There are other factors, too, in favor of local and

independent school systems. First, they are more responsive to the needs and wishes of

the parents and the community. The door to the school superintendent's office is usually

open to any parent who wishes to make his views known. But the average citizen would

be hard pressed to obtain more than a form letter reply from the national Commissioner

of Education in Washington, D.C."

Local control, and consent of the governed, are two foundational principles in our great

The Promise of Evidence-Based Policymaking 959
nation.

Because the CEP is not an elected body, it does not actually hold representative authority

to collect, or to recommend collection, of student-level evidence, or of any evidence,

without written consent; and, for the same reasons, neither does the Department of

Education.

Because the fifty, federally-designed, evidence-collecting, State Longitudinal Database

Systems never received any consent from the governed in any state to collect data on

individuals (as the systems were put into place not by authority, but by grant money) it

follows that the idea of having CEP study the possible removal of barriers to federal

access of those databases, is an egregious overstep that even exceeds the overstep of the

State Longitudinal Database Systems.

Because federal FERPA regulations altered the original protective intent of FERPA, and

removed the mandate that governments must get parental (or adult student) consent for

any use of student level data, it seems that the idea of having CEP study and possible

influence removal of additional "barriers" to federal use of data, is another egregious

overstep.

As a licensed teacher in the State of Utah; as co-founder of Utahns Against Common

Core (UACC); as a mother of children who currently attend public, private and home

schools; as acting president of the Utah Chapter of United States Parents Involved in

Education (USPIE); as a patriot who believes in "consent of the governed" and in the

principles of the U.S. Constitution; and, as a current tenth grade English teacher, I feel

that my letter represents the will of many who stand opposed to the "study" of the

protective barriers on student-level data, which the CEP's website has outlined it will do.

I urge this commission to use its power to strengthen local control of data, meaning

parental and teacher stewardship over student data, instead of aiming to broaden the

numbers of people with access to personally identifiable student information to include

government agencies and/or educational sales/research corporations such as Pearson,

Microsoft, or the American Institutes for Research.

To remove barriers to federal access of student-level data only makes sense to a socialist

who agrees with the Marc Tucker/Hillary Clinton 1998 vision of a cradle-to-grave nanny

state with "large scale data management systems" that dismiss privacy as a relic in

subservience to modern government. It does not make sense to those who cherish local

control.

It is clear that there is a strong debate about local control and about consent of the

governed, concerning data and concerning education in general. NCEE Chair Mark

Tucker articulated one side of the debate when he said: "the United States will have to

largely abandon the beloved emblem of American education: local control. If the goal is

to greatly increase the capacity and authority of the state education agencies, much of the

new authority will have to come at the expense of local control."

Does that statement match the philosophical stand of this commission? I hope not. Local

control means individual control of one's own life. How would an individual control his

or her own destiny if "large scale data management systems" in a cradle-to-grave system,

like the one that Tucker and Clinton envisioned, override the right to personal privacy

and local control? It is not possible.

I urge this commission to use any influence that it has to promote safekeeping of unitrecord data at the parental and teacher level, where that authority rightly belongs.

960 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: October 21, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sl5-1fww

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0014

Comment on FR Doc # 2016-22002

Submitter Information

Name: Laurel Haak

Address:

Bethesda, 20817

Email: l.haak@orcid.org

General Comment

We all would like to better understand how innovation works, the better to support it.

That requires both better access to information about R&D and a shift in our thinking

about the role of researchers -- those at the heart of the innovation process. We need

linked data infrastructures on the education, affiliations, funding, activities and outputs

of researchers. ORCID identifiers provide the basis for such linking at a federal and

international level. This can revolutionize our understanding of innovation and lead to

more effective R&D policies and better government investments.

I am attaching a short document that provides more detail.

Attachments

ORCID comments to EBPC - 20161021

The Promise of Evidence-Based Policymaking 961
1 

To: The Commission on Evidence-based Policy Making 

Re: Docket Number 160907825–6825–01 

From: Laurel Haak, Executive Director, ORCID, l.haak@orcid.org

Date: 21 October 2016 

Abstract: We all would like to better understand how innovation works, the better to support 

it. That requires both better access to information about R&D and a shift in our thinking about 

the role of researchers -- those at the heart of the innovation process. We need linked data 

infrastructures on the education, affiliations, funding, activities and outputs of researchers. 

ORCID identifiers provide the basis for such linking at a federal and international level. This can 

revolutionize our understanding of innovation and lead to more effective R&D policies and 

better government investments. 

Statement: There is broad international consensus that innovation and R&D matter for longterm economic growth and that innovation underpins many improvements in social, health and 

environmental outcomes. Governments play a key role in innovation by influencing framework 

conditions, setting policies, and providing funding assistance. The exact mechanisms, however, 

through which research, science and innovation lead to socioeconomic outcomes are not well 

understood. We need to better understand these interconnections and develop a robust 

theory of change, which can underpin government policies and investments. 

Delivering on this aspiration requires governments to build local, national, and international 

data infrastructures for science and innovation. International standards, in particular, are 

essential given the global nature of science and the fact that the research community is highly 

mobile. Linked data on the funding of R&D and the outputs generated is a first step to building 

this infrastructure. 

Data oversight and quality of federal support for R&D is poor. A recent paper by the National 

Science Foundation1 highlights the poor state of federal data. This needs to be urgently 

rectified by developing standards for funding of R&D and federal agencies agreeing to 

implement these standards. Obtaining an accurate picture of federal investment in R&D is 

critical to developing understanding of how the innovation system works and generating 

evidence on return on investment. It is beholden on the federal government to present linked 

data on the investments it is making in the science and innovation system to the public. 

Second, the outputs of the research and innovation system need to be systematically collected 

and linked back to funding sources. It is only when the key inputs and outputs are clearly linked 

that we can begin to understand how government policies and investments impact innovation. 

Aggregated data from across funding mechanisms and the various federal funding agencies will 

 

1 Pece, 2016, Putting the Cart Before a Lame Horse: A Case Study for Future Initiatives to Automate the Use of 

Administrative Records for Reporting Government R&D. OECD Blue Sky Forum, Ghent, 19-21 September, 2016. 

962 The Promise of Evidence-Based Policymaking
2 

enable policymakers to determine the relative effectiveness and impacts of various investments 

and policy settings. 

At the heart of the innovation process rests people. It is researchers who generate both 

incremental and transformative innovations. The researcher needs to therefore be at the heart 

of data infrastructures for research and innovation. ORCID identifiers provide that foundation. 

In 2012, ORCID launched a registry where researchers can obtain a unique identifier that they 

can use as they complete their theses, publish, apply for grants or submit reports as a student 

or employee at a research organization. These research “transactions” involve obtaining the 

consent of the researcher to both collect their identifier and share information about the 

activity or affiliation (Figure 1). 

Figure 1. The ORCID vision starts with researchers, and involves the entire research community in 

collecting persistent identifiers, connecting them to activities and affiliations as researchers engage in 

these activities, and making public and verified assertions about relationships. 

The ORCID registry has already issued more than 2.6 million identifiers around the world and 

uptake is rising fast. The ORCID system is now integrated with all of the world’s major 

publishers, hundreds of universities and research organizations and more and more funding 

agencies. Some funding agencies, including the Portuguese Fundacao Ciencias e Technologia, 

the Swedish Research Council, Science Foundation Ireland, the UK National Institute for Health 

The Promise of Evidence-Based Policymaking 963
3 

Research and the US Department of Transportation have already mandated the use of ORCID 

identifiers. 

ORCID is being adopted at a national level in countries across Europe and Oceania, and by 

funding organizations and publishers around the world. It is being used by funders and 

government agencies in the United States including NIH, DOE, EPA, and has been recognized by 

Science Europe - an association of European research funding organizations - as a key 

component of the research infrastructure.2 In Australia, Europe, and the UK, ORCID is a 

component of their grant management programs.3 In New Zealand ORCID will be a critical 

underpinning feature of the national data strategy for science and innovation which contains a 

vision of creating a national research information system which links data on funding, 

researchers, outputs and end user collaborations.4 

Widespread use of ORCID identifiers around the world is paving the way for a global R&D data 

infrastructure. Unfortunately, federal agencies have been slow to see the potential of ORCID 

identifiers. ORCID is registered in the United States, yet few federal funding agencies have 

integrated ORCID identifiers into their systems. This needs to change if the United States 

government wishes to generate evidence on the impacts of its R&D investments. 

Data specifically about researchers, for example detailed survey files managed by the National 

Science Foundation, or actual grant application and award data held by the National Institutes 

of Health5, or administrative data held by universities relating to federally-funded projects6, 

makes it possible to gain a much deeper and more nuanced understanding of innovation – and 

the impact of policies. 

However, at present, it is very difficult to gain access to these individual-level data, as the 

researchers involved have not given permission for it to be disclosed beyond the agency or 

employer. This has necessitated the implementation of secure data centers, stringent access 

policies, and has restricted the types of analysis that can be performed. 

However, it is possible to envision an open data infrastructure that would enable sharing of 

much of this data that is locked within agencies. Such an infrastructure is indeed becoming a 

 

2 Science Europe, 2016. Draft proposal for a Science Europe position statement on research information systems.

OECD Blue Sky Forum, Ghent, 19-21 September. 

3 Australian Government, 2015. NHMRC and ARC Statement on Open Researcher and Contributor ID (ORCID); 

Townsend, 2015. We’re now a member of ORCID. Research Councils UK Blog; Ministry of Business, Innovation, 

and Employment, 2016. ORCID Joint Statement of Principle; European Commission, 2016, H2020 Programme 

guidelines on open access to scientific publications and research data in Horizon 2020. 4 Ministry of Business, innovation, and Employment, 2016, Research, Science, and Innovation Domain Plan. 5 Haak et al., 2011, The electronic Scientific Portfolio Assistant: Integrating scientific knowledge databases to 

support program impact assessment. Science and Public Policy 39:464; Ginther et al, 2011, Race, Ethnicity, and NIH 

Research Awards. Science 333:1015; NIH, 2011, Individual Mentored Career Development Awards Program 

Evaluation; Haak et al., 2011, The electronic Scientific Portfolio Assistant: Integrating scientific knowledge 

databases to support program impact assessment. Science and Public Policy 39:464.

6 Largent and Lane, 2012, STAR Metrics and the Science of Science Policy. Research Policy 29:431. 

964 The Promise of Evidence-Based Policymaking
4 

reality at an international scale,7 supported by an evolving community consensus on FAIR data 

management principles.8 It is based on persistent identifiers, community participation, and the 

consent of individuals in sharing their information.9 ORCID has emerged as the global standard 

for identifiers for researchers. Persistent identifiers (digital object identifiers ) for research 

articles and datasets have been in place for over a decade. It is now a standard practice for 

DOIs to be assigned to articles and datasets as they are published, and these identifiers 

underpin search and discovery of tens of millions of items in the research literature. DOI 

assignment is managed by two non-profit organizations, CrossRef (crossref.org) and DataCite 

(datacite.org). 

With my colleagues at Crossref and DataCite, I ask that the Committee consider the role that 

persistent identifiers can play in improving data on the inputs and outputs of the research and 

innovation ecosystem. I strongly encourage you to consider the essential role of researchers: 

they must be included as partners in any initiative to share research information. I also 

encourage the Committee to open up the funding of R&D by federal agencies and develop data 

standards to enable data aggregation. Naturally, we believe that all federal agencies should 

integrate their systems with ORCID. 

As Robert Oppenheimer said, “The best way to send information is to wrap it up in a person.” 

As we use person-level data to understand innovation, we can demonstrate the verity of this 

statement.10 The more ways that individual researchers can connect their unique identifier to 

activities and affiliations, the more we will understand the flow of knowledge underpinning 

innovation and develop appropriate policies to support it. 

 

7 Porter, 2016, Digital Science White Paper: A New ‘Research Data Mechanics’ 8 Wilkinson et al., 2016, The FAIR guiding principles for scientific data management and stewardship. Scientific 

Data 3:16008. 9 Haak et al., 2012, Information science. Standards and infrastructure for innovation data exchange. Science 

338:196 10 Zolas et al., 2016, Wrapping it up in a person: Examining employment and earnings outcomes for Ph.D. 

recipients. Science 350-1367.

The Promise of Evidence-Based Policymaking 965
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: October 24, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sn7-pcs3

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0015

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

Dear Commission on Evidence-Based Policymaking,

I love the American concept of voter-based, Constitution-based, elected representativebased, policymaking. It's why I live in America.

In contrast to voter-based policymaking there is evidence-based policymaking, which I

don't love because it implies that one entity's "evidence" trumps individuals' evidence, or

trumps individuals' consent to policy changes.

Former Secretary of Agriculture Ezra Taft Benson said something about education that

also applies to educational data and policymaking:

"The best way to prevent a political faction or any small group of people from capturing

control of the nation's educational system is to keep it decentralized into small local

units, each with its own board of education and superintendent. This may not be as

efficient as one giant super educational system (although bigness is not necessarily

efficient, either) but it is far more safe. There are other factors, too, in favor of local and

independent school systems. First, they are more responsive to the needs and wishes of

the parents and the community. The door to the school superintendent's office is usually

966 The Promise of Evidence-Based Policymaking
open to any parent who wishes to make his views known. But the average citizen would

be hard pressed to obtain more than a form letter reply from the national Commissioner

of Education in Washington, D.C."

Local control, and consent of the governed, are two foundational principles in our great

nation.

Because the CEP is not an elected body, it does not actually hold representative authority

to collect, or to recommend collection, of student-level evidence, or of any evidence,

without written consent; and, for the same reasons, neither does the Department of

Education.

Because the fifty, federally-designed, evidence-collecting, State Longitudinal Database

Systems never received any consent from the governed in any state to collect data on

individuals (as the systems were put into place not by authority, but by grant money) it

follows that the idea of having CEP study the possible removal of barriers to federal

access of those databases, is an egregious overstep that even exceeds the overstep of the

State Longitudinal Database Systems.

Because federal FERPA regulations altered the original protective intent of FERPA, and

removed the mandate that governments must get parental (or adult student) consent for

any use of student level data, it seems that the idea of having CEP study and possible

influence removal of additional "barriers" to federal use of data, is another egregious

overstep.

As a licensed teacher in the State of Utah; as co-founder of Utahns Against Common

Core (UACC); as a mother of children who currently attend public, private and home

schools; as acting president of the Utah Chapter of United States Parents Involved in

Education (USPIE); as a patriot who believes in "consent of the governed" and in the

principles of the U.S. Constitution; and, as a current tenth grade English teacher, I feel

that my letter represents the will of many who stand opposed to the "study" of the

protective barriers on student-level data, which the CEP's website has outlined it will do.

I urge this commission to use its power to strengthen local control of data, meaning

parental and teacher stewardship over student data, instead of aiming to broaden the

numbers of people with access to personally identifiable student information to include

government agencies and/or educational sales/research corporations such as Pearson,

Microsoft, or the American Institutes for Research.

To remove barriers to federal access of student-level data only makes sense to a socialist

who agrees with the Marc Tucker/Hillary Clinton vision of a cradle-to-grave nanny state

with "large scale data management systems" that dismiss privacy as a relic in

subservience to modern government. It does not make sense to those who cherish local

control.

The Promise of Evidence-Based Policymaking 967
It is clear that there is a strong debate about local control and about consent of the

governed, concerning data and concerning education in general. NCEE Chair Mark

Tucker articulated one side of the debate when he said: "the United States will have to

largely abandon the beloved emblem of American education: local control. If the goal is

to greatly increase the capacity and authority of the state education agencies, much of the

new authority will have to come at the expense of local control."

Does that statement match the philosophical stand of this commission? I hope not. Local

control means individual control of one's own life. How would an individual control his

or her own destiny if "large scale data management systems" in a cradle-to-grave system,

like the one that Tucker and Clinton envisioned, override the right to personal privacy

and local control? It is not possible.

Authority regarding the management of data belongs at the local level - not the federal

level.

968 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: October 25, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8snu-w4tu

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0016

Comment on FR Doc # 2016-22002

Submitter Information

Name: Sandra Hofferth

Address:

College Park, MD, 20742

Email: hofferth@umd.edu

General Comment

See attached file(s)

Attachments

Comments_SOCN_Oct25_2016

The Promise of Evidence-Based Policymaking 969
Commission on Evidence-Based Policymaking Comments 

Sandra Hofferth, PhD, Professor Emerita and Research Professor 

Principal Investigator, Social Observatories Coordinating Network* 

Maryland Population Research Center, University of Maryland, College Park 20742 

301-405-6403, Hofferth@umd.edu

October 25, 2016 

Overarching Questions 

1. Are there successful frameworks, policies, practices, and methods to overcome challenges 

related to evidence-building from state, local, and/or international governments the 

Commission should consider when developing findings and recommendations regarding Federal 

evidence-based policymaking? If so, please describe. 

There are several ongoing privately sponsored community-based programs that link data across local, 

state, and national levels. An ongoing project at the University of Chicago (Goerge, 2013) provides an 

example of evidence-building that could be advanced more effectively and scaled to greater impact by 

facilitating data sharing. For the past 30 years, researchers have been laboriously compiling a database 

with anonymized information on Chicago's children from information housed at multiple state and local 

agencies, including schools, social services, and criminal justice. Combined with contextual information 

(e.g., census tracts, city administrative districts), geographical areas where resources could be targeted 

to improve family well-being and reduce dependence on public services can be identified. Parallel work 

at the University of Chicago's Urban Center for Computation and Data shows how such data could be 

acquired, anonymized, and georeferenced more efficiently by synthesizing digital data streams from 

diverse administrative databases. The Chicago work shows the potential utility of such efforts for 

providing services to citizens. 

A second effort is entitled the National Neighborhood Indicators Partnership (Kingsley & Pettit, 2011). 

In this collaborative effort between the Urban Institute and 37 cities, all partners build and operate 

neighborhood level information systems, drawing upon administrative data from multiple sources. Data 

on employment, Births, deaths, Crimes, TANF, Food Stamps, Child care, Health, and schools are merged 

with property, tax, and infrastructure quality information. Activities consist of continuous indicator 

review and dissemination of information over the web. Indicators at the city or metro-wide level can be 

used to change laws and policies; for geographic targeting/coordination of resources for programs and 

investments; for individual neighborhood improvement initiatives; for performance management; and 

for program evaluation. The types of data include national data files –e.g., American Community 

Survey; open data – government administrative data; and integrated data systems. One important 

substantive area studied was the effect of foreclosures on children. One program linked foreclosures in 

970 The Promise of Evidence-Based Policymaking
2 

neighborhoods to the number of children in those areas. They did this by linking parcel level data on 

properties and neighborhood level data. 

2. Based on identified best practices and existing examples, what factors should be considered 

in reasonably ensuring the security and privacy of administrative and survey data? 

The two critical factors are potential risk of disclosure and degree of harm that could follow. First, how 

likely is it that someone could be identified given the data available? Second, how serious is the harm 

that would be occasioned? 

In a book Privacy, Big Data, and the Public Good: Frameworks for Engagement, Julia Lane and colleagues 

(2014), describe the importance of balancing data utility for the public good against disclosure risk. One 

of the most important potential risks is that of disclosing the geographic location of individuals whose 

data are held, and, therefore, potentially disclosing their actual identity. This risk collides head on with 

our understanding of the critical part that residence plays in access to quality public and private services, 

governance, and community amenities, and risk of weather-related and other events. 

A second area of potential risk could occur when data sets are linked or combined, depending on the 

types and level of information contained in the different data sets. However, the main advantage of 

linking is that of efficiency; once the possibility of linking is available, data collectors need to add only 

the data elements not otherwise available. 

Data Infrastructure and Access 

3. Based on identified best practices and existing examples, how should existing government data 

infrastructure be modified to best facilitate use of and access to administrative and survey data?

Major types of structures for data sharing include release of public use data, access to a research data 

center in which the researcher can access but not remove data, provision of a license or contract to 

obtain restricted files, release of synthetic files in which some data are altered, access to a government 

employee who actually runs analysis software on the data, and a data enclave in which data are 

accessed remotely. At the present time the National Center for Health Statistics (NCHS) uses several 

models: data are divided by level of risk. Low risk data are released on-line as public data files. 

Descriptors of geographic location are not released in these public data files. To obtain geographic 

information on state of birth, for example, an application needs to be submitted to NCHS. The NCHS will 

release the geographic data after the application is approved. Because only data at the state level are 

released, and the state numbers are large, the risk of individual identification is low. 

Data with greater risk of individual identification or disclosure, such as geographic location at the county 

or census tract level, have a higher bar for sharing. For them, the NCHS uses the Census Bureau 

Research Data Center (RDC) model, a physical location in which researchers can securely access 

confidential data. For example, researchers can access securely the geographic location of participants 

and merge onto their data files information about that geographic location. The actual geographic 

location can be removed from files once this other information is attached. The data on the file retain 

The Promise of Evidence-Based Policymaking 971
3 

confidentiality. Other types of data can be linked as long as there are restrictions on how data sets can 

be linked and what types of information can be made available to researchers. 

Given new technology, a large dedicated physical structure becomes less necessary. Instead, the data 

enclave model seems to be the most feasible and practical. The enclave does not need to actually have 

possession of the data, simply a secure connection to the organization that does. 

 4. What data-sharing infrastructure should be used to facilitate data merging, linking, and access 

for research, evaluation, and analysis purposes? 

The federal government has legal restrictions on data sharing. An efficient way to improve access would 

be for the federal government to contract with a private data center to oversee and monitor the process 

of reviewing requests for data access, setting up specialized data, and reviewing and monitoring data 

users. A number of such data centers are currently in place. The Research Data Centers (RDC) of the 

U.S. Census Bureau are examples; however, in this case employees need to be Census Bureau 

employees, a very restrictive management model and quite expensive. In addition, data are actually 

accessed in the RDC. 

As an alternative model, access could be provided through a private data center, but the data sets could 

be housed with their original proprietors. In this model, a center would help identify data needed by a 

prospective researcher. The center would be charged with having a record of the type of confidentiality 

protection required by the data provider and would broker access to the data. Access could be obtained 

remotely through the data center. The researcher may wish to have access to data with differential 

restrictions on access; the data center could handle this process so that the individual could access, 

merge the data and analyze the data for scientific purposes with approved access protocols 

administered by this single organization. 

5. What challenges currently exist in linking state and local data to federal data? Are there 

successful instances where these challenges have been addressed? 

Because such initiatives are so important, why have they not happened more broadly? This is partly 

because agencies, levels of government, professionals, and advocates work in silos that prevent putting 

together comprehensive data in a format conducive to research and analysis. Integration and 

collaboration are critical for putting the information together. With increased concern about 

confidentiality and identity theft, some of the biggest obstacles are in putting together complex data 

sharing agreements. More resources could be spent to develop model examples of such agreements 

that could be widely used. A network of regional data centers that utilizes and shares these models 

would be an enormous contribution. At the moment there is substantial expertise in different areas of 

the country, but no concerted effort to learn and share experiences, in spite of the increasing interest in 

addressing the obstacles to data sharing and encouragement by many federal agencies in increasing the 

usage of administrative and other data. 

972 The Promise of Evidence-Based Policymaking
4 

6. Should a single or multiple clearinghouse(s) for administrative and survey data be established 

to improve evidence-based policymaking? What benefits or limitations are likely to be 

encountered in either approach? 

In a series of meetings conducted by the Social Observatories Coordinating Network over the past three 

years (see materials at socialobservatories.org), the network settled on a challenge for the future— to 

design a national network of regional data centers that could be coordinated through common 

objectives, sharing of protocols, and data sharing (Moran et al. 2014). The network concluded that 

behavior is so situation- and place-specific that it is practically impossible to use widely dispersed 

national samples of populations to draw conclusions about processes in any one place. Based upon 

extensive discussion and consultation with statisticians, our model would embed in a set of 20-25 

regional observatories or data centers a nationally representative population-based sample from about 

400 census tracts that would enable the observatory data to be aggregated in such a way as to produce 

a nationally representative picture of the United States on an ongoing basis while also fully capturing the 

diversity that characterizes local places. Each would be an entity, whether physical or virtual, that is 

charged with collecting, curating, and disseminating data from people, places, and institutions in the 

United States. Each of these data centers would be responsible for gathering information from a set of 

tracts that are located in proximity to their collaborating scientists and other information from their 

local community. The individuals, institutions and communities in which these census tracts are 

embedded can be systematically studied over time and space by regional data centers and the data 

would represent the U.S. as a whole. 

7. What data should be included in a potential U.S. government data clearinghouse(s)? What are 

the current legal or administrative barriers to including such data in a clearinghouse or linking 

the data? 

Each data center would be charged with cataloging and providing access to data collected and archived 

from the local area as well as data that may be national in scope. They are also intended to stimulate 

the development of new directions and modes of inquiry. They will do so through the use of diverse 

complementary methods and data sources including ethnography, experiments, administrative data, 

social media, biomarkers, and financial, land, environment, and public health records as well as survey 

research. These observatories will work closely with local and state governments to gain access to 

administrative records that provide extensive data on the population in those tracts, thereby providing a 

depth of understanding and integration of knowledge that is less invasive and less subject to declining 

response rates than survey-derived data. Starting with a single such data center charged with setting up 

confidential data agreements and facilitating access would be a pathway to such a network. This initial 

national data center, as test of the framework, can serve as a pilot for the larger network of centers 

needed to meet national needs. 

Data are collected by private parties such as Twitter, eBay, Netflix, LinkedIn, Facebook, and others. 

Many of these companies make their data available to researchers under contract. When combined 

with other private data or other public surveys and administrative data sets, these data could be very 

useful but their use poses some very challenging privacy and security issues. 

The Promise of Evidence-Based Policymaking 973
5 

Traditional forensic lab data along with emergency calls, and Google Trend or Twitter data can be used 

for emerging disease or drug trends. 

8. What factors or strategies should the Commission consider for how a clearinghouse(s) could 

be self-funded? What successful examples exist for self-financing related to similar purposes? 

One such effort is located in the Urban Center for Computation and Data at the University of Chicago; 18 

different organizations and multiple individuals are involved (Goerge, 2013). Data come from 311 

service requests, 911 calls, and other public and private data bases. This group is taking the data and 

making them accessible to the community. Existing software programs extract data coming out of the 

continuously updated grid to create analysis files. With additional resources the data bases currently 

developed could be continuously updated and transferred to a data portal that multiple partners could 

access in real time. This would permit city officials to use their administrative data to better target 

resources. Social media data could be geographically linked so that police and social workers would be 

aware of potential problems at locations in the city even prior to or during adverse environmental and 

social events. With additional resources they would be able to follow children from different 

environmental locations into adolescence, linking data from public school and education records and 

then adulthood, linking employment and wage data from national data bases. Such centers could also 

develop and archive data from evidence-based program evaluations that are increasingly required by 

federal agencies. 

A data clearinghouse is not initially self-fundable but requires a consortium of private and public sources 

of support on behalf of the national statistical system and scientific enterprise similar to the city-focused 

model represented by Chicago. This would most likely require substantial federal seed money through 

grants to get it off the ground, but the model could eventually be integrated into regional statistical 

systems, local governmental budgets, and private funding partnerships. 

The National Neighborhood Indicators Project (NNIP) started in 1995 with 6 institutions that had put 

together data sharing arrangements in their cities. Today this neighborhood indicators partnership has 

expanded to 37 cities and 12 more are talking about joining. These include community and social 

service nonprofits, university, metropolitan planning agencies, and foundations; all have a substantial 

community involvement component, i.e. they are close to the stakeholders. Data are used for 

performance measurement, management and policy analysis. 

9. What specific administrative or legal barriers currently exist for accessing survey and 

administrative data? 

These barriers will vary across administrative areas such as states, counties, and cities, thus fitting with 

the proposed regional focus of such data centers. Each center will be attuned to local issues and 

administrative concerns. In addition to the confidentiality issue, an underlying barrier, and justification 

for regional focus, is that formats are often incompatible. Whereas a regional center can link at the local 

level, such centers will have the resources to figure out how to link across centers to provide a national 

picture. 

974 The Promise of Evidence-Based Policymaking
6 

10. How should the Commission define “qualified researchers and institutions?” To what extent 

should administrative and survey data held by government agencies be made available to 

“qualified researchers and institutions?” 

Most public access data shops request researchers to fill out a form that describes the research project 

for which they are requesting access to data. In 25 words or so, it is fairly straightforward to describe 

and then determine whether this is a legitimate project. Plus contact information from the potential 

user is required. We have had requests from clearly illegitimate “bots”. Those are easy to sort out. 

Adding the requirement to fill in a randomly generated passcode is also helpful. 

11. How might integration of administrative and survey data in a clearinghouse affect the risk of 

unintentional or unauthorized access or release of personally-identifiable information, 

confidential business information, or other identifiable records? How can identifiable 

information be best protected to ensure the privacy and confidentiality of individual or business 

data in a clearinghouse? 

Because we are recommending that this data center not actually hold the data, but rather serve as a 

portal to access data, we imagine that the organization/entity will be charged with implementing a 

variety of different types of data confidentiality provisions and licenses. Having a single entity charged 

with implementing a set of confidentiality protections for access to data through a single portal is 

efficient. There is sufficient expertise in the social science community to put those protections in place. 

There has never been a data breach in the existing data centers to date. 

12. If a clearinghouse were created, what types of restrictions should be placed on the uses of 

data in the clearinghouse by “qualified researchers and institutions?” 

This would depend upon the organization providing access to the actual data. 

13. What technological solutions from government or the private sector are relevant for 

facilitating data sharing and management? 

A set of private data centers could involve private industry such as Google, Facebook, Twitter, etc. who 

have already been addressing these issues. Participation from these sophisticated technology companies 

is essential but they need the participation of qualified researchers and institutions in universities to 

better identify the important questions about our society and contextual variables to include. 

14. What incentives may best facilitate interagency sharing of information to improve 

programmatic effectiveness and enhance data accuracy and comprehensiveness? 

The best incentive would be the actual productive use of data they generate. Too much of the data 

collected by local, state and federal agencies remains archived and unused. Making these data available 

would constitute a more efficient use of resources and may obviate the need for other forms of data 

collection which may be more invasive. However, additionally, having a private non-government entity 

The Promise of Evidence-Based Policymaking 975
7 

funded and charged with the proper protection of the data and facilitating access to approved 

individuals, would also be an incentive. 

Data Use in Program Design, Management, Research, Evaluation, and Analysis 

15. What barriers currently exist for using survey and administrative data to support program 

management and/or evaluation activities? 

Barriers to entry in terms of knowing how to access data and then gaining such access are substantial in 

both time and money. Census Bureau access through a Research Data Center requires a full proposal 

and then about $15,000 in research funds unless the investigator has access to an RDC at his/her 

institution. 

16. How can data, statistics, results of research, and findings from evaluation, be best used to 

improve policies and programs? 

A national framework for studying local contexts may be the best way forward. The best neighborhood 

studies to date feature specific cities such as Chicago or Los Angeles. The observatories will facilitate 

comparison across such communities. With a regional data center network, we can contemplate the 

possibility of a national sample of neighborhood and other local contexts that can be studied at multiple 

levels and in multiple ways. For example, because of the national sample frame of census tracts 

embedded in them, these observatories become ideal locales to examine how proximal social contexts 

(a) directly affect individuals’ and families’ social, behavioral and economic functioning or (b) alter social 

and behavioral relationships, and (c) how individuals and families, in turn, change their environments. 

The community context will enable the study of social networks, which cannot be studied within the 

current survey research model in which clustering of observation is minimal. It also increases the 

chance of linking climate change and other natural phenomena with behavioral responses. Naturally 

occurring exogenous shocks to some tracts but not all tracts will provide the opportunity for scientists to 

conduct comparative studies of adaptation and change. Similarly, a set of experimental trials situated in 

diverse contexts will afford researchers the opportunity to conduct rigorous comparative studies of 

interventions. 

17. To what extent can or should program and policy evaluation be addressed in program 

designs? 

One limitation of program evaluations is that they may be tested in one region of the country and not in 

others. The proposed infrastructure permits comparisons across communities. Program evaluation 

data should be part of this new set of regional data centers. The advantage of having a network is that 

projects will be encouraged to replicate their evaluations on programs spread across the U.S. This 

would enable researchers to see whether program success varies across the U.S. Evaluations of 

responsible fatherhood initiatives varied across program locations, for example. 

976 The Promise of Evidence-Based Policymaking
8 

18. How can or should program evaluation be incorporated into program designs? What specific 

examples demonstrate where evaluation has been successfully incorporated in program 

designs? 

No comment. 

19. To what extent should evaluations specifically with either experimental (sometimes referred 

to as “randomized control trials”) or quasi-experimental designs be institutionalized in 

programs? What specific examples demonstrate where such institutionalization has been 

successful and what best practices exist for doing so? 

No comment. 

*The Social Observatories Coordinating Network 

The Social Observatories Coordinating Network (SOCN) is funded by the National Science Foundation to 

work with scientific communities in the development and planning of a set of observatories for the 

Social, Behavioral, and Economic (SBE) Sciences that will be transformative. This network follows up on a 

set of 4 workshops prior to 2010 and 4 from 2010 to 2012 that engaged the scientific community in 

initial discussions about the proposed approach. (Reports from the 2010-2012 meetings are available at 

the website below.) Dissemination of what we produce will play an important role, particularly at 

national professional meetings and other scientific conferences. The final product will be a report to 

NSF outlining the views of the various scientific communities and the consensus emerging from these 

discussions. 

Members of the NSF Social Observatories Coordinating Network include: 

J. Lawrence Aber, New York University 

Henry E. Brady, University of California, Berkeley 

Dalton Conley, New York University 

Susan Cutter, University of South Carolina 

Catherine Eckel, Texas A & M University 

Barbara Entwisle, University of North Carolina 

Sandra Hofferth, University of Maryland 

Klaus Hubacek, University of Maryland 

Emilio Moran, Michigan State University 

A complete list of earlier contributors to this project can be found at the website below. 

For additional information contact: 

Sandra Hofferth: hofferth@umd.edu or 

Emilio Moran: moranef@msu.edu 

The Promise of Evidence-Based Policymaking 977
9 

Our objectives and reports from previous meetings are available at: 

http://socialobservatories.org/

References 

Goerge, Robert. 2013. Presentation to Social Observatories Coordinating Network, May 8-9. Summary 

report available on http://Socialobservatories.org/reports. 

Kingsley, G. Thomas & Kathryn L.S. Pettit, 2011. Quality of Life at a Finer Grain: The National 

Neighborhood Indicators Partnership, Chapter 4 in M.J. Sirgy et al. (eds.), Community Quality-of-Life 

Indicators: Best Cases V, Community Quality-of-Life Indicators 3, DOI 10.1007/978-94-007-0535-7_4. 

Lane, Julia, Victoria Stodden, Stefan Bender, & Helen Nissenbaum. 2014. Privacy, Big Data, and the 

Public Good: Frameworks for Engagement, Cambridge, UK: Cambridge University Press. 

Moran, E.F., et al. 2014. Building a 21st century infrastructure for the social sciences. Proceedings of 

the National Academy of Sciences, November 11, vol. 111, no. 45, 15855–15856. 

(www.pnas.org/cgi/doi/10.1073/pnas.1416561111). 

978 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: October 25, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8so0-9csa

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0017

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

I am AGAINST the government collecting private information from students and others

without the consent of someone who is of legal age to give said consent. Citizens have

the right to keep their information private.

The Promise of Evidence-Based Policymaking 979
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: October 27, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8spb-eawr

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0018

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

NO, NEVER, NOT AT ALL. DEFINITELY NOT!

980 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: October 28, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8spr-moot

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0019

Comment on FR Doc # 2016-22002

Submitter Information

Name: Doug Schofield

Address:

Rochester, NY, 14607

Email: Doug@SexAndVirginity.com

General Comment

Hi,

We are a non profit that shows the advantages of saving sex for a commitment. The

women that we have shared our information with either waited to have sex until they

were engaged to be married or are currently in a long term relationship but still saving

sex for a major commitment like marriage/engagement. This was achieved by sharing

with them a few surveys that we conducted that shows that...

1 Most men prefer virginity in the woman they marry.

2 Having sex with a man doesn't help him fall in love, make a woman more special to

him or even guarantee a commitment will follow.

3 If a man is using a woman for sex, it is very unlikely that she will get a commitment.

4 When a man has serious intentions for a woman he will wait significantly longer for

sex than if he intends to just use her for sex.

The Promise of Evidence-Based Policymaking 981
My wife develops scientific studies for Neilsen. And together we put together 3 small

surveys that asks men questions that women want to know. For example, We just

completed using Survey Monkey with 110 men, ages 18 to 29 in USA and found that...

3 out of 4 heterosexual men answered either Somewhat good or Very Good to the

question "How would you feel if the woman you marry previously had never had sexual

intercourse with another man" with the majority answering Very Good.

78% of heterosexual men said that it would bother them when asked "How would you

feel if the woman you marry had sexual intercourse quickly with the men that she

previously dated" with the majority saying that it would bother them a lot.

2 out of 3 heterosexual men answered Zero to the question "Would you prefer the

woman you marry to have previously had zero sex partners or more than zero sex

partners"

We had to pay for this out of our own pocket ($770). Zogby said they would replicate

this study for $8,950. If we could just get funding, this information would have national

visibility. On our small scale this worked. But we need help to take this to the next step.

-Doug Schofield

www.SexAndVirginity.com

982 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: October 30, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sre-oopm

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0020

Comment on FR Doc # 2016-22002

Submitter Information

Name: Lisa Vavrik

General Comment

I am not comfortable with everything about a student and their family being loaded on a

data base right down to a students social media comments and social security number.

This is new terrain and way to soon to run experiments with children's lives. We know

all the data breaches that have taken place and can only imagine how putting personal

information on a data base could turn out.

The Promise of Evidence-Based Policymaking 983
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: October 31, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8srg-o6pi

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0021

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

The government is getting out of hand. We pay taxes for you to serve us not control us.

PEOPLE are waking up to your tyranny and we are not happy.

984 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: October 31, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8srn-8zxp

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0022

Comment on FR Doc # 2016-22002

Submitter Information

Name: Christine Runquist

General Comment

We Oppose the National Student Tracking Database.

The Promise of Evidence-Based Policymaking 985
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: October 31, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8srs-4las

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0023

Comment on FR Doc # 2016-22002

Submitter Information

Name: Jennifer Klein

General Comment

I am against the over-reaching collection of data on our children. When is enough

enough? Never for these groups advocating for more and more data. Why? Are they

really concerned about our children or more concerned about profiling them and making

a lot of money off this sensitive data. The SLDS are too intrusive the way they are now.

Why expand the information in them. I don't want my kids tracked from cradle to grave.

Please start protecting them as this data is never safe from breaches and the more data

you have on everyone, the more dangerous it is for our kids safety. Time to start

protecting people's privacy.

986 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: October 31, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8srt-asni

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0024

Comment on FR Doc # 2016-22002

Submitter Information

Name: Susan Bella

General Comment

I am wholly against creating a national database made up of the combination of many

federal agencies for use in student data tracking. it is a violation of our privacy, and it

can lead to massive misuse of data through unauthorized and unethical studies. It goes

against the Institutional Review Board guidelines for human studies. Unscrupulous

researchers can use data to create outcomes that suit their agenda. Algorithms are often

wrong, and can be skewed to produce a specific goal and there is no oversight.

I respectfully ask that the ban on a national database on our citizens continue to be

banned and not allowed under ANY circumstances.

The Promise of Evidence-Based Policymaking 987
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: October 31, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8ss0-tapi

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0025

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

Don't do this. Do the right thing.

988 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 01, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8ssi-bd7k

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0026

Comment on FR Doc # 2016-22002

Submitter Information

Name: Cheri Kiesecker

Address: United States,

Email: ckiesecker@gmail.com

General Comment

See attached file(s)

Please see attached file for comment: parent comment to CEP_ck

Thank you

Attachments

Parent comment to CEP_ck

The Promise of Evidence-Based Policymaking 989
November 1, 2016 

To the Comments to Commission on Evidence-Based Policy. 

https://www.regulations.gov/document?D=USBC-2016-0003-0001 

I am a parent, only a parent, but I hope the commission will consider my testimony with the same 

weight you give others who have a louder, perhaps more "important" voice. Parents deserve to have a 

voice in matters involving children. 

In the rush to collect and report student data, WHO is auditing the data for accuracy? 

There are cases of KNOWN data erros being submitted. The US Dept of Ed, Office of Civil 

Rights admits there is not sufficient protocol to verify data is correct and according to State 

Dept of Ed, Office of Civil Rights, it is too burdensome to correct known data errors. See 

news article and email exchange with US OCR: http://tinyurl.com/zc9blg4

How often do data errors like this happen? How would students know if incorrect data were 

being reported or inferred about them? What decisions are being made based on inaccurate 

data? The FTC has noted the lack of transparency / laws governing algorithms and data 

brokers. 

NO transparency on algorithms. 

WHO is able to verify that analysis of data is not biased, when algorithms are kept secret, 

proprietary? Unless students can see data being collected or inferred about them, the data 

should not be collected or shared. (FIPS, Nuremberg) 

Ethics and IRB's. 

Several presenters at the Oct. 21 Commission Education-based Policy hearing mentioned that 

IRB reviews are hampering their data access. IRBs are essential in ensuring ethical use of 

data. Are those wanting access to data hoping to avoid independent analysis of their 

legitimate need and use of children's data? 

Security concerns abound. 

There are no enforceable penalties, no parent right of action for misuse or breach of data, 

and the federal government has a shocking track record on data security. (OPM breach, 

http://www.reuters.com/article/us-cybersecurity-usa-idUSKCN0PJ2M420150709 and the 

Vitara rating of F, over 400 repeat incidents in US Dept of Ed 2015 audit 

https://oversight.house.gov/hearing/u-s-department-of-education-information-security-review/ 

.) Allowing the federal government even more access to student data, knowing these 

substantial deficiencies exist, would be negligent. 

990 The Promise of Evidence-Based Policymaking
The data collected and shared from ed records or SLDS is NOT just grades. 

Under the School Exemption Clause of FERPA, access to pii data contained in a student's 

education record or state longitudinal data base (SLDS) can be shared. For a snapshot of pii 

data shared with researchers, see examples from the Colorado Dept. of Education research 

contracts here http://tinyurl.com/he8z2yv

You will need to scroll through contracts to find data sharing section. ie: see this contract with 

AIR, (Appendix A) which allows access to pii educational records: 

http://tinyurl.com/gnqt5ae

SEL and 21st Century Skills. 

Teachers are required to upload ever increasing amounts of data, including behavioral and 

survey data, examples of how a student thinks, how a student feels (21st century social 

emotional skills.) See one such survey here: 

https://drive.google.com/open?id=0B7epgdVXe0gKRm94TEt2aDVsQjA This is subjective data 

at best, not to mention highly personal. IS IT ETHICAL to allow behavioral and emotional data 

to be measured and shared outside of the classroom? HOW will this information be used to 

profile or predict a child? Considering that students' emotions will be measured and 

standardized / ranked in the near future, access to personal student records needs to be 

better constrained, not broadened. 

http://blogs.edweek.org/edweek/rulesforengagement/2016/09/commission_formed_to_advan

ce_schools_focus_on_social_emotional_development.html 

A national Unit Record will Link k12-to Higher ed data 

Unless you explictly prohibit in future statute, opening the higher ed student unit record will lead 

to access to k12 ed record and K12 data, linking data between agencies and programs. The WDQI 

US Dept of Labor plans to access individual, pii, K12 student data, stored in state SLDS databases, 

starting in pre-school. SEE HERE: 

The Promise of Evidence-Based Policymaking 991
http://tinyurl.com/ho9m9gq https://www.doleta.gov/performance/workforcedataquality.cfm 

Many panelists testifying in support of the unit record at the October 21, 2016 CEP hearing, also 

propose accessing this K12 data. For example, but not limited to, Rachel Zinn from the Workforce 

Data Quality Campaign, David Medina from the Results for America, Actionable Intelligence for Social 

Policy (AISP) America Forward Center for Employment Opportunities Center for Research and Reform in 

Education, Johns Hopkins University, KIPP, REDF, Sorenson Center for Impact Success for All Foundation 

Sunlight Foundation: 

Medina testimony: "State Education and Workforce Data Systems: The Commission 

should recommend that Congress and the Administration support the enhancement 

of the existing State Longitudinal Data Systems (SLDS) program administered by the 

U.S. Department of Education, which helps states integrate education and workforce 

data, and the proposed expansion of the Workforce Data Quality Initiative* that 

would help build state and local capacity to track employment and educational outcomes" 

CEP Hearing- Oct 21, 2016

Linking higher ed and K12 student data (K12 SLDS data systems) has long been the 

992 The Promise of Evidence-Based Policymaking
goal: Student data systems unite!-2010

https://www.insidehighered.com/news/2010/02/16/data but has always been rejected due 

to privacy or cost concerns.

Bill Gates' recently released 2016 Priorities Report, also stresses goal of linking K12 - Higher 

Ed data, in a national database: 

Bill Gates: "Continue to develop robust state data systems that connect disparate higher 

education systems within and across states, including non-public institutions, and 

improve linkages between higher education, K-12, and workforce data to facilitate 

the timely and safe exchange of data for decision-making by educators and policymakers.

Develop a comprehensive national data system or exchange that would expand 

coverage and quality by collecting a key set of performance metrics for all students in all 

institutions. Such a system or exchange would also alleviate reporting burden and reduce 

duplication by leveraging existing state and national data collections and would require 

revising data privacy and security protocols to ensure compliance with state and federal 

laws, as well as accepted standards and practices in the field. Options range from 

improving IPEDS to creating a federal student-level data system." 

http://postsecondary.gatesfoundation.org/wp-content/uploads/2016/09/PostsecondarySuc

cessAdvocacyPriorities2016.pdf

There is a reason this national student database has been banned for years.

Some data should not be mixed. Data collection is much more pervasive – and much less 

controlled – than people realize. Algorithms are often biased and wrong. There is no way for 

a student to know how his/her data is being used against him/her or to know what opportunities 

were denied based on this shared data. 

http://www.fastcoexist.com/3057514/your-data-footprint-is-affecting-your-life-in-ways-you-cant

-even-imagine

"I think the opportunity is a rich one. At the same time, the ethical considerations need to 

be guiding us," says Jesse Russell, chief program officer at the National Council on 

Crime and Delinquency, who has followed the use of predictive analytics in child 

protective services. Officials, he says, are treading carefully before using data to make 

decisions about individuals, especially when the consequences of being wrong—such 

as taking a child out of his or her home unnecessarily—are huge. And while 

caseworker decision-making can be flawed or biased, so can the programs that 

humans design. When you rely too much on data—if the data is flawed or incomplete, as 

could be the case in predictive policing—you risk further validating bad decisions or 

existing biases. There’s this danger we lose our identity as people and we become 

categories."

The Promise of Evidence-Based Policymaking 993
Until there are LAWS and enforceable PENALTIES that restrict how data is collected, how it is 

shared, mixed, and transparency around algorithms and ways to audit and regulate the data

analytics used to profile and predict a person, there should be no expansion of data collection.

Rather, people should own their own data and as in Europe, large agencies, nonprofits and 

corporations should not have access to children's personal data without consent.

Thank you for NOT creating a national student database that would increase risk for all 

stakeholders, especially the children you are meant to protect. 

Thank you, 

Cheri Kiesecker, parent 

994 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 01, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8ssl-yfq2

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0027

Comment on FR Doc # 2016-22002

Submitter Information

Name: Paul Lingenfelter

Address:

Westminster, CO, 80234

Email: paul@sheeo.org

General Comment

See attached file(s)

Attachments

Comments Offered to the Commission on Evidence

The Promise of Evidence-Based Policymaking 995
COMMENTS OFFERED TO THE COMMISSION ON EVIDENCE- BASED POLICY

November 1, 2016 

Paul E. Lingenfelter 

President Emeritus, State Higher Education Executive Officers 

paul@sheeo.org

11581 Eliot Court, Westminster, Colorado 80234 

(303) 748-7425 

The first section of these comments addresses the “Overarching Questions,” and 

questions concerning “Data Use in Program Design, Management, Research, 

Evaluation, and Analysis” posed by the Commission. These comments are more fully 

elaborated in my book, “Proof,” Policy, and Practice: Understanding the Role of 

Evidence in Improving Education. (2016. Stylus Publishing, Sterling, VA.) The second 

section will address specific questions concerning “Data Infrastructure and Access.” 

Although representing only my own views, my comments reflect my work 

experience and service in the following roles: Deputy Director, Illinois Board of 

Higher Education, Director of Program Evaluation and Vice President for Human 

and Community Development, John D. and Catherine T. MacArthur Foundation, 

President, State Higher Education Executive Officers association, and governing or 

advisory board member for the National Student Clearinghouse, National Survey of 

Student Engagement, National Institute for Learning Outcomes Assessment, and 

Parcc, Inc. 

OVERARCHING QUESTIONS AND DATA USE IN PROGRAM DESIGN, 

MANAGEMENT, RESEARCH, EVALUATION, AND ANALYSIS 

1. The proper use of evidence is essential for improving both policy and 

practice. But the complexity and variety of most situations confronted by 

policy makers and practitioners, pose serious challenges for the proper use 

of evidence. These challenges must be well-understood in order to avoid 

wasteful, unproductive effort and superficial “evidence,” which can be 

ineffective, misleading, or even harmful. 

Chapter One, of “Proof,” Policy, and Practice sets the stage for subsequent 

chapters of the book by discussing the difficulty of generating authoritative 

scientific knowledge in the situations faced by policy makers and 

practitioners. It draws largely on Consilience by E.O. Wilson, “Science and 

Complexity” by Warren Weaver, Useful Knowledge by Charles Lindblom and 

David Cohen, and Unsimple Truths: Science, Complexity, and Policy, by Sandra 

Mitchell. 

2. Experimental methods, such as randomized clinical trials (RCT), are excellent 

tools for addressing problems with a relatively simple cause, which can be 

ameliorated with a straightforward, relatively simple intervention. 

996 The Promise of Evidence-Based Policymaking
Unfortunately, few problems in social policy (especially perhaps, in 

education) are “conceptually neat,” a term coined by Donald Berwick, a 

leader in the application of improvement science to the delivery of health 

care. Nearly half a century of efforts to use experimental methods in program 

evaluation have demonstrated most forcefully the inadequacy, rather than 

the utility of experimental methods to provide clear guidance for evaluating 

and improving complex programs to address complex problems. 

Chapter Two summarizes major contributions to and debates within the field 

of program evaluation and offers a critical analysis of the Institute for 

Education Science’s “What Works Clearinghouse,” and work by MDRC.i

 It 

considers the particular case of Success for All, a multifaceted and thoroughly 

designed educational program, which has been positively evaluated, but has 

proved difficult to replicate widely. It provides examples of “conceptually 

neat” problems under which RCTs can make important contributions, such 

as: a study of the effects of providing guidance in completing Pell Grant 

applications in the course of tax preparation assistance, and studies of the 

effectiveness of making good choices the “default” option for policies such as 

retirement savings. But the number of “conceptually neat” problems are few. 

Chapter Two also draws on Realistic Evaluation by Ray Pawson and Nick 

Tilley who advocate evaluating the effectiveness of individual interventions 

in particular situations, rather than mounting experiments to discover large 

scale interventions that “work” across varying situations. As discussed in 

Chapter Five, improvement scientists in industry and health care employ this 

technique. 

3. Proper attention to the problem of measurement is essential for using 

evidence to improve policy and practice. E.O. Wilson asserts that 

measurement using “universally accepted scales” is a fundamental 

requirement for science. And the essential role of measurement is universally 

recognized by the discipline of “improvement science” in industry, health 

care, and most recently in the Networked Improvement Communities of the 

Carnegie Foundation for the Improvement of Teaching. But getting 

measurement “right” is challenging. 

Chapter Three offers a discussion of the potential and the challenges of 

measurement focusing on four initiatives in education: a) Measuring Up, a 

ten year initiative of the National Center for Public Policy in Higher 

Education, which graded the states on their performance in key dimensions 

of higher education policy and outcomes; b) the Data Quality Campaign, 

which has successfully promoted the development of more robust,

longitudinal data systems in the states and helped inspire the Common 

Education Data Standards initiative; c) the Common Core Standards for 

Career and College Readiness in mathematics and English language; and d) 

Assessing Higher Education Learning Outcomes (AHELO), a recent initiative 

The Promise of Evidence-Based Policymaking 997
of the Organization for Economic Cooperation and Development. 

These four initiatives have provided opportunities to learn about the 

challenges of creating valid, useful measurement, and the difficulties of 

translating useful measurement into action for improvement. None of them 

solved all the relevant problems, but all of them have made important 

contributions in increasing our understanding of measurement challenges 

and our capabilities for using measurement productively. 

4. Useful evidence must be fit for the purposes for which it is employed, it must 

serve the purposes of the stakeholders involved in policy and practice, it 

must acknowledge the individual rights of stakeholders, including their right 

to privacy and the (limited but legitimate) right to protect their own 

interests, and it must find a way to navigate the shoals of accountability while 

working for improvement. 

The fitness of evidence for different purposes must recognize the distinction 

between policy and practice, and the capabilities and limitations of policy 

and practice. Policy is generally limited to two blunt instruments: law and 

regulation associated with law; and money applied to different purposes and 

under the conditions specified by policy. The inherent inflexibility of policy is 

its greatest weakness in solving complex, multi-faceted problems. In Seeing 

Like a State: How Certain Schemes to Improve the Human Condition Have 

Failed, James C. Scott analyses the factors that lead to policy over-reach and 

provides numerous examples of failed policy efforts to solve social problems. 

Policy especially fails when it over-specifies practice and constrains the 

abilities of practitioners to learn and take adaptive actions in the situations in 

which they work. 

In the policy realm evidence can and should be used to identify the scope, 

intensity, and geographical incidence of social problems where the tools 

most readily available to government (the investment of money and enacting 

of laws and regulations) might make an important contribution. Government 

can productively support research to increase the understanding of a 

problem, government can support efforts to increase the capabilities of 

practitioners, and government can influence or enable others to act by 

providing direct or indirect investments where money by itself is a 

significant factor in a problem. Evidence can and should be used to test 

whether policies are effective in making such contributions. 

While good policies are essential for good practice, governments and coarsegrained governmental policies and programs by themselves are incapable of 

solving complex problems. Complex problems require the work of skillful 

practitioners, whether they are health care or social service professionals, 

teachers, scientists, military leaders, or administrators. Evidence-based 

policy and evidence-based practice (some prefer practice-based evidence) in 

998 The Promise of Evidence-Based Policymaking
partnership are needed to improve the human condition. “Evidence-based 

policy” will fail if it seeks to discover and then faithfully implement at scale 

practices or programs to solve complex problems. Complexity and faithful 

implementation at scale are inherently in conflict. 

Chapter Four of the book explores these issues in some depth, including an 

analysis of issues related to student and institutional privacy in a proposal to 

create student unit record data system for higher education. The four 

measurement initiatives discussed in the previous chapter are evaluated in 

terms of five questions: a) How does the initiative serve policy makers, the 

public, and practitioners? b) Is the division of labor between policy and 

practice clear and productive? c) Are the uses of the data well suited to the 

capabilities of the users? d) Are privacy rights an issue? And e) Are the data 

accurate, reliable, valid, and fit for purpose? 

5. A productive partnership between evidence-based policy and evidencebased practice requires practitioners to employ the disciplined use of 

evidence to improve outcomes. The practitioners of “improvement science” 

in industry, in health care, and more recently in education employ qualitative 

research techniques (which some call “reflective practice”) to develop causal 

theories about problems and to design small-scale interventions to 

ameliorate the problems. They then employ quantitative techniques to test 

interventions quickly, study the results, learn, and adapt their strategies to 

obtain better results. Tony Bryk of the Carnegie Foundation for the 

Improvement of Teaching characterizes this approach as “small change with 

fast learning” as opposed to the traditional strategy of “big change and slow 

learning” through traditional research studies of large scale interventions. 

Practitioners can learn from “laboratory” research and research based in 

other practice settings, but the first step in “evidence-based practice” is to 

consider whether and to what extent findings in a different context and 

setting may apply in the immediate situation. Experimentation and 

adaptation in the practice setting are essential for improving outcomes. 

Chapter Five considers evidence-based practice in health care, education 

(focusing on the Networked Improvement Communities of the Carnegie 

Foundation for the Advancement of Teaching), and social work. It concludes, 

“In the end, the effective use of evidence to improve practice will require 

respect for the complexity of practice and the wisdom of practitioners 

combined with respect for scholarship—systemic analysis, the development 

and pursuit of theory (or hunches) about causal relationships, measurement, 

experimentation, and adaptation. Advances in data systems, in measurement, 

in basic science, and in creative ways of marrying the wisdom of practice 

with systemic inquiry are increasing the potential for significant progress.” 

The growing need to increase the breadth and quality of educational 

The Promise of Evidence-Based Policymaking 999
attainment has recently led to important advances in the use of measurement 

to improve educational practice. The National Institute for Learning 

Outcomes Assessment http://www.learningoutcomeassessment.org has 

documented and promoted the increasing use of assessment to improve 

teaching and student achievement in higher education. In both K-12 and 

higher education, new tools have been developed for increasing the focus 

and clarity of learning objectives and for assessing student learning in order 

to improve instruction and achievement. 

These tools include: 

a. The Common Core State Standards Initiative which has added clarity 

and focus to K-12 learning objectives http://www.corestandards.org 

and the summative and formative assessments it inspired have 

advanced the quality and sophistication of tools available to improve 

teaching and learning. 

b. The Degree Qualifications Profile http://degreeprofile.org and the 

quite similar Essential Learning Objectives published by the 

Association of American Colleges and Universities 

(AAC&U) https://www.aacu.org/leap/essential-learning-outcomes

which have given colleges and universities clear targets for student 

learning. 

c. The VALUE rubrics https://www.aacu.org/value-rubrics, also 

developed by AAC&U which have provided tools for assessing levels of 

student learning based on actual student work, rather than 

unavoidably superficial standardized tests. Twelve states are 

participating in a demonstration project to employ VALUE rubrics in 

the Multi-State Collaborative, a collaboration of the State Higher 

Education Executive Officers and AAC&U. http://www.sheeo.org/msc

6. Public policy is not shaped simply by evidence. It is also shaped by competing 

interests and by varying views about the causes of and solutions to social 

problems. Evidence can inform and shape the views of policy makers and 

how they perceive their interests, but the powers of evidence are limited. In 

democracies, competition for influence and negotiation among competing 

perspectives powerfully shape policy. It is unlikely that evidence will ever 

become the final arbiter between competing interests and different 

worldviews. 

But as discussed in Chapter Six, evidence can play a constructive role. 

Establishing widely accepted measures of important social conditions can be 

a powerful way for evidence to shape policy. If policymakers can agree on 

ways to measure facts on the ground, especially facts that are directly 

1000 The Promise of Evidence-Based Policymaking
relevant to the communities they represent, evidence can foster more 

effective policies. 

Research can also inform the shared understanding of the systems that 

produce undesirable outcomes and the factors associated with improved 

outcomes. For example, over several decades research by the University of 

Chicago Consortium on School Research has helped policy makers and 

practitioners better understand and more effectively create the conditions 

associated with more effective schools and higher levels of student 

achievement. Such understanding is the first step in developing policies and 

practices to yield improvement. 

7. In summary, the use of evidence has great potential to improve social policies 

and the effectiveness of specific programs and interventions. To realize this 

potential, however, the distinctive roles of policy and practice must be 

observed. Policy and practice have different capabilities, as well as different 

needs and uses for data and evidence. If policy seeks to improve practice by 

substituting policy for practice or by intruding too forcefully into the domain 

of practice, it will fail. 

Perhaps most importantly, improvement in practice, rather than policy 

accountability should be the first purpose of measurement. The halting and 

controversial implementation of the Common Core State Standards

exemplifies the consequences of failing to observe this principle. Although it 

is quite helpful, almost essential for practitioners to have a common 

yardstick for assessing results, if measurement is predominantly employed 

as a tool for threat-based accountability, more energy will be devoted to 

avoiding or reducing threat than to improving performance. 

Instead, policy should work to create the conditions necessary for effective 

practice, and accept in partnership with practitioners shared accountability 

for improvement. Practitioners need to accept and embrace their

responsibilities for using evidence to improve practice and for identifying 

and advocating policies that can help them become more successful. 

Improved outcomes require an evidence-based partnership between policy 

and practice. 

 DATA INFRASTRUCTURE AND ACCESS 

During the past fifteen years a number of projects discussed in the book -- the 

Common Education Data Standards project, a proposed student unit record system 

for higher education, federal funding for State Longitudinal Data System, and the 

Data Quality Campaign -- confronted the questions enumerated by the Commission 

in this section. These observations address specific issues that I observed while 

participating in these projects. 

The Promise of Evidence-Based Policymaking 1001
1. More data are required for guiding practice than are required for guiding 

policy. It is close to impossible to achieve common data definitions and 

common practices in complying with those definitions in a wide range of 

domains. The more data are collected, the more difficult the challenge, and 

the more likely the data will not be useful. It is easier to obtain useful data for 

policy if the data set is coarse-grained, focusing on high-level indicators. The 

practice setting normally requires more data elements, but absolute fidelity 

to common data definitions is needed for only a few important outcomes. 

The solution to this problem is to collect the data that are needed for policy 

and for practice as separate activities. Both policy and practice data sets 

should employ the same definitions for the comparatively few data elements 

needed for policy analysis, but there is no need to have perfectly aligned data 

for the many thousands of data elements that will be employed in different 

practice settings. (The scope of the Common Education Data Standards is an 

example of data system overreach.) 

2. In some, but not all respects, the issues involved in protecting individual 

privacy apply also to protecting institutional or organizational privacy. This 

was the underlying issue that prompted congressional action forbidding the 

Department of Education from establishing a student unit record system that 

would include data on financial assistance provided by private colleges and 

universities. Some situations limit the rights to institutional privacy, just as 

some situations limit the right to individual privacy. These need to be defined 

as a matter of policy, informed by the legitimate interests of all stakeholders. 

3. The National Student Clearinghouse (NSC) has collected individual student 

enrollment and completion data in higher education for more than a decade 

while protecting both individual privacy and institutional privacy. These 

data, analyzed by the NSC Research Center, have been used to improve 

available information about graduation and transfer rates in higher 

education. 

NSC is largely supported by fees generated by outside clients for degree and 

enrollment verification and other uses of the data, which are approved by the 

students and institutions involved. A public/private partnership, employing 

entities like NSC could be a mechanism for financing data collection and 

utilization, governed by privacy policies developed by consensus among 

stakeholders. 

4. The Patient Centered Outcomes Research Institute (PCORI)ii is an effort to 

develop a data sharing platform for improving medical practice. While 

consistent, reliable data collection is complex even in medicine, a large scale 

data set is likely to be more useful in improving medical practice than in 

fields with more complicated problems of measurement. This initiative 

might, however, provide experience useful in sorting through appropriate 

1002 The Promise of Evidence-Based Policymaking
boundaries between practice and policy and the privacy issues in other 

domains. 

CONCLUSION 

I would be pleased to elaborate on these comments, respond to questions, or 

provide additional information on any topics of interest to the Commission. 

Respectively submitted, 

Paul E. Lingenfelter 

 

i

 See also: Ginsberg, A. and Smith, M.S. (2016) Do Randomized-Controlled Trials Meet 

the Gold Standard? Washington, DC: American Enterprise Institute. 

ii Collins, F. S., Hudson, K. L., Briggs, J. P., & Lauer, M. S. (2014). PCORnet: Turning a 

dream into reality. Journal of the American Medical Informatics Association: 

JAMIA, 21(4), 576–577. doi:10.1136/amiajnl-2014-002864

The Promise of Evidence-Based Policymaking 1003
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 02, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8ssz-vvbg

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0028

Comment on FR Doc # 2016-22002

Submitter Information

Name: t g

General Comment

we do not need such database. NO

1004 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 02, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8ssz-sp77

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0029

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

This is an absolute OUTRAGE! Compiling and linking data like this on private citizens -

children, no less! - is something that is done in communist countries, NOT the United

States of America. Mine and my children's lives are PRIVATE. Stop taking money from

Bill Gates, and stay out of our private lives!

The Promise of Evidence-Based Policymaking 1005
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 02, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8st4-qg8u

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0030

Comment on FR Doc # 2016-22002

Submitter Information

Name: Christi Hammons

Address:

Hugo, OK, 74538

Email: chammons@choctawnation.com

General Comment

See attached file(s)

Attachments

CEP_Comments_for_TPP_Tier_1_CNO

1006 The Promise of Evidence-Based Policymaking
Commission on Evidence-Based Policymaking Comments

Docket ID: USBC-2016-0003

Federal Register Number: 2016-22002

Christi Hammons

Choctaw Nation of Oklahoma 

P.O. Box 88

Hugo, Oklahoma 74743 November 2, 2016

Dear Chair Katharine G. Abraham, Co-chair Ron Haskins, and members of the Commission:

The Choctaw Nation of Oklahoma welcomes the opportunity to provide comments to the 

Commission on Evidence-Based Policymaking (CEP). We offer input for questions 16 and 18 

based on our experience as a grantee of a high-quality evidence-based program, the Office of 

Adolescent Health’s Teen Pregnancy Prevention (TPP) Program. This program has been 

recognized as a pioneering example of tiered evidence-based policymaking, and represents an 

important contribution to building a body of evidence of what works. This includes high quality 

implementation, evaluation, innovation, and learning from results. 

The Choctaw Nation of Oklahoma is a non-reservation-based Native American tribal 

government located in 10 1⁄2 counties in southeast Oklahoma, and was the first tribe to be named 

as a Promise Zone. Youth in the Choctaw Nation face many disparities including high rates of 

poverty, low educational attainment, and high unemployment. An additional disparity, and 

perhaps underlying cause of other disparities, is the alarmingly high rate of teen pregnancy. In 

2015, the Choctaw Nation received a TPP Tier 1B grant to address teen pregnancy in the three 

Choctaw Nation counties with the highest teen birth rates. The goal of the newly implemented 

program is to have a significant impact on reducing rates of teen pregnancy and existing 

disparities by replicating evidence-based teen pregnancy prevention programs to scale in middle 

schools, high schools, and alternative schools in Choctaw, McCurtain, and Pushmataha Counties 

in southeast Oklahoma. 

As a TPP Tier 1 grantee we incorporated evidence into all aspects of our project. The program 

models we implemented were chosen from a list of those that had already been rigorously 

evaluated and demonstrated to change behavior.

We also used evidence to guide and improve our project throughout the grant period. During the 

first year of the grant, the program collected data from over 1,000 community members as well 

as published resources to ensure the program chose the evidence-based models most suited for 

our community. Now that the program has begun implementation, staff collect data on all 

lessons taught to youth to report on performance measures as well as to improve fidelity to the 

evidence-based curricula and improve the quality of the teaching activities. This data is also

used for continuous quality improvement for the program. We look forward to analyzing 

comparison data over the coming years to quantify the impact this program has made.

In closing, we consider the TPP Program to be a prime example of high-quality evidence-based 

policymaking. It is one of the few government programs that use evidence and evaluation 

The Promise of Evidence-Based Policymaking 1007
criteria throughout the grant life cycle. Thank you for considering our input for the CEP. If you 

have any questions or need additional information, please contact me at (580) 326-8304 (ext. 

6058) or chammons@choctawnation.com.

Sincerely,

Christi Hammons

Director

Teen Pregnancy Prevention Program

1008 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 04, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sud-gq16

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0031

Comment on FR Doc # 2016-22002

Submitter Information

Name: S. Stanley Young

Address:

Raleigh, NC, 27607

Email: genetree@bellsouth.net

General Comment

See attached file(s)

Attachments

Data Access Docket Number 160907825-6825-01

The Promise of Evidence-Based Policymaking 1009
Data Availability: 

[Docket Number 160907825-6825-01] 

Request for Comments for the Commission on Evidence-Based Policymaking 

There is an obvious tension between “strategies to increase the availability and use of 

government data” and “protecting the privacy and confidentiality of the data”. But there are 

other tensions as well: a fair building and analysis of a data set versus finding data and building 

an analysis data set to support a preselected policy. The later strategy can lead to publication bias 

whereby results that do not support a policy are simply not published. Fund researchers that 

support your policy versus fund research to answer a question. There is the individual researcher 

incentive versus the common good. 

The Request for Comments is answered in a series of short notes, usually with a pointer to 

relevant literature. I take the perspective of Edward Glaeser that researchers are expected to be 

self-serving and systems of data access and analysis should be in place to take that into account 

and promote the common good. 

Summary: 

Asking scientists to do better, share data, has not and will not work. As a condition of funding 

and publication, it should be a requirement that authors place their analysis data set in a public 

repository. One should assume that authors are self-serving and optimize getting a publication 

and that includes slanting the data set construction and analysis methods to get a publishable 

result or a result that favors a particular policy. 

The analysis data set should be built in a way that personal identity is protected. For example, if 

the data is micro aggregated before the analysis is done, the micro aggregated data can be shared 

as the aggregation step protects personal identity. See “productive and counterproductive 

aggregation of experimental units” below.

 The raw data set and the code used to build the analysis data set should be saved in a way that it 

can be made public or given to a trusted 3rd party for examination. The raw data, the code used to 

convert the raw data to the analysis data set and the analysis data set should go to the agency that 

funded the work. And/or it should go to a public archive, e.g.dryad. Any work that cannot 

provide these things should not be funded with taxpayer money. 

A study should be registered in a public registry and a summary report, results positive or 

negative, should be placed in the registry. 

1010 The Promise of Evidence-Based Policymaking
On important questions funded by government agencies, two studies should be funded, one 

seeking to support the claim being tested and the other seeking to deny the claim being tested. 

More broadly, under what conditions is the claim supported and under what conditions is the 

claim not supported. 

The Promise of Evidence-Based Policymaking 1011
1. Joe Cecil wrote a very perceptive article on legal aspects of data access back in 1985. Here are 

two quotes I like: 

"As an abstract principle, the sharing of research data is a noble goal and 

meets with little opposition. However, when data sharing is attempted in a particular 

circumstance, the conflicting interests of the parties can thwart the exchange. A glance at the 

benefits and obstacles to data sharing discussed by Hedrick (in this volume) reveals the reason: 

few of the benefits and most of the burdens fall to the possessor of a data set." 

and 

"This case suggests that an agency can insulate its actions from public scrutiny by funding a 

grant for controversial research and then basing its action on those findings. As long as the 

agency does not take possession or control of the records, the FOIA will not assist those who 

wish to challenge the findings that underlie the agency action." 

Required data access would be a step in the right direction. There are almost always technical 

and operational ways to share data IF the will is there. 

Cecil JS, Griffin E. 1985. The role of legal policies in data sharing. Pages 148-198. Sharing 

Research Data. National Academy Press. Washington DC.

 

1012 The Promise of Evidence-Based Policymaking
2. The problem will not be solved by admonishing scientist to do reproducible science and make 

their data sets public. Or act symbiotically with those that have the data. Such thinking is 

hopelessly naive. We should expect scientists to act in their own best interest, which includes 

shaping their data set and analysis to produce a claim that is publishable, p-value <0.05, with a 

nice story. 

Glaeser 2006, an economist, wrote against fellow economists that they did not treat themselves 

as they treated others. Economists typically argue that people work to their own self-interest. 

Glaeser said economist should take the position that researchers work to their own self-interest. 

They will fudge here and there. They will provide facts that support their position. They will 

ignore facts that do not support their story. It is not only money, grants, but it is also prestige. 

Glaeser used the term “optimizing researchers” for what would be called science crooks in less 

polite company. Requiring that data be placed in a public repository will make self-serving more 

difficult, but not impossible. 

Some quotes: 

“The solution to this problem is not to expect a mass renunciation of data mining [data dredging 

doing this and that to find some publishable result], selective data cleaning or opportunistic 

methodology selection, but rather to follow Leamer’s lead in designing and using techniques that 

anticipate the behavior of optimizing researchers.”

“After all, the impact of such techniques is invariably to reduce the significance levels of results. 

[get smaller p-values.] The same incentives that induce researchers to data mine will induce them 

to avoid techniques that appropriately correct for that data mining.”

Large data sets “...give researchers new opportunities to use their initiative to enhance their 

results.” 

Discussion Paper Number 2122 Researcher Incentives and Empirical Methods. Edward L. 

Glaeser October 2006 HARVARD UNIVERSITY 

 

The Promise of Evidence-Based Policymaking 1013
3. Lewandowsky, writing in Nature, seems to take the position that Michael Mann of GW fame 

is very unfairly put upon. There is lots of evidence that Mann acted with malice: trying to get an 

editor fired; manipulating data and analysis, etc. Whatever the situation scientifically, open 

access to data would have helped. The climate data held by CRU, Climate Research Unit, was 

both stonewalled and manipulated. See the short book ClimateGate, The CRUTape Letters by 

Steven Mosher and Thomas Fuller. Ultimately Phil Jones of CRU deleted raw data so that it 

could not be gotten through freedom of information. 

Mostly my time is my own. In Nov of 2009, I took two days to follow closely the blogging that 

was the reaction to a 60GB CRU data dump. The discussion was very lively and quite detailed. 

The overall impression was that the CRU and their inside people were science crooks. For a 

rather impartial look at the situation track down Judith Curry of Georgia Tech. Or look at some 

of the work of Steven McIntyre. 

Keep in mind “Anything worth having is worth cheating for.” W. C. Fields 

 

 

1014 The Promise of Evidence-Based Policymaking
4. In the Significance article of Young and Alan Karr, the position is taken that funding agencies 

and editors need to exercise their management responsibility (Demning). You cannot expect 

workers to clean up the lack of data access. As Deming points out, mostly the workers are 

reacting to the incentives in place. They are more or less adjusted to them, if not actually happy. 

Workers, work-a-day-scientists, cannot and will not change the system. Any charge to them to 

behave better is so much wasted ink/breath. Only the managers of the scientific system, funding 

agencies and journal editors, have the power to change it. 

Young SS, Karr A. 2011. Deming, data and observational studies: A process out of control and 

needing fixing. Signicance September, 8:122–126. 

 

The Promise of Evidence-Based Policymaking 1015
5. Air pollution is a good case in point for data access. In 2006 Roger Peng (and others) were 

without good access to environmental epidemiology data; he and others wanted data access. 

When Congress, with oversight over the EPA, wanted data, Peng blogged that Congress was just 

trying to cause trouble and that they should not have access to the data. Peng et al. appear to take 

the position that 

a. it is clearly true that current air quality is a causal killer and 

b. it would be costly to provide access to the data. 

but 

a. It is possible that air quality is not a killer, e.g. Greven et al. (2011), Milojevic et al. 

(2014), among many others 

b. the regulation of air quality is enormously expensive without effective scientific 

oversight. 

NB: Researchers funded by EPA on air pollution are making very serious public health claims. 

Where public health is concerned, data is often/usually made public or provided to a trusted 3rd

party. 

The evidence is that the EPA and their funded researchers stonewall on data access. I have a 

massive data set, Young et al. (2015), that makes the case that current air quality is not killing 

anyone. 

But how can normal science work efficiently without access to data? 

Greven S, Dominici F, Zeger S. 2011. An approach to the estimation of chronic air pollution 

effects using spatio-temporal information. J Amer Stat Assoc. 106:396-406. 

Milojevic A, Wilkinson P, Armstrong B, Bhaskaran K, Smeeth L, Hajat S. 2014. Short-term 

effects of air pollution on a range of cardiovascular events in England and Wales: case-crossover 

analysis of the MINAP database, hospital admissions and mortality. Heart 100:1093-1098. 

Peng RD, Dominici F, Zeger SL. 2006. Reproducible epidemiologic research. American Journal 

of Epidemiology 163 (9), 783-789 

Peng blog. 

http://simplystatistics.org/2014/04/01/this-is-how-an-important-scientific-debate-is-being-usedto-stop-epa-regulation/

1016 The Promise of Evidence-Based Policymaking
Lopiano KK, Smith RL, Young SS. 2015. Air quality and acute deaths in California, 2000-2012. 

https://arxiv.org/abs/1502.03062 

The Promise of Evidence-Based Policymaking 1017
6. The evidence is that well over half of claims made in science papers do not replicate. Now, all 

of the flawed papers are full of statistics. See Begley and Ellis (2012). At some point, the public 

might well say: How could all these flawed claims be made in peer reviewed journals and have 

the blessing of statistical analysis? Why should the taxpayer fund any of this? Also, where were 

the real statisticians to let this go on so long and so widely? 

Note that the White House, OSTP Feb2013, by executive order, require data access to taxpayer 

funded research. So far as I know the EPA is not in compliance. They have a policy of not 

accepting data sets so the FOAI will be unsuccessful. I have a letter from the EPA saying they do 

not accept data sets of the research they fund. 

Baker, M. 2016. 1,500 scientists lift the lid on reproducibility. Nature 533:452-454. 

http://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970

Begley CG, Ellis LM. 2012. Raise standards for preclinical cancer research. Nature 483:531–

533. 

 

1018 The Promise of Evidence-Based Policymaking
7. Learned societies, e.g. The Royal Society, National Academy of Science, etc., take the 

position that researchers make their data sets available. They put the responsibility on scientists. 

Their position is totally without effect. The responsibility lies with funding agencies and journal 

editors. They need to change the incentives. But even giving the problem to the funding 

agencies, e.g. EPA, is not enough. The EPA lets contracts to support their policies and do not 

take possession of the data so that FOIA cannot reach through the agency to get the data. Policies 

need to be in place so that the EPA is required/has the incentive to make data available**. 

Cecil JS, Griffin E. 1985. The role of legal policies in data sharing. Pages 148-198. Sharing 

Research Data. National Academy Press. Washington DC. 

** 

Use of Science Transparency Act 

Any federal agency proposing rule-making or legislation shall specifically name each document 

used to support the proposed rule-making or legislation and provide all data used in said 

document for viewing by the public. 

Federal Study Transparency Act 

If federal funds are provided for a study, all data relating to the reporting of results of said study 

must be provided for scrutiny by the public at the time of publication. 

 

The Promise of Evidence-Based Policymaking 1019
8. No help from judges: 

‘This amount of information that can be presented to the court is huge. “You can appreciate 

what the ‘record’ looks like on review,” Verkuil said. “It is enormous, and the agency has to 

decide what is in and what is out on its own.” “Although courts can scrutinize the research and 

underlying data upon which the agency relied, Verkuil said, they generally defer to the agency 

on technical determinations. Courts recognize that they do not have the technical expertise to 

second-guess an agency’s scientific or technical judgment.’

So the EPA, for example, does not provide the data for the papers used to support their policy 

and judges defer to their judgement on technical matters. Any agency can adopt the position that 

controversial research can be outsourced and so long as they do not take possession of the data, 

FOIA cannot reach through the EPA to get the data. 

This is a perfect recipe for self-dealing, without any general science supervision. 

The EPA, and any government agency, should be required to take possession of every analysis 

data set used in a publication they fund. Ideally they should require the data sets to be place in a 

public repository. 

THE NATIONAL ACADEMIES PRESS 

http://www.nap.edu/21703 

 

1020 The Promise of Evidence-Based Policymaking
9. Micro aggregation to protect personal identity. 

Productive and Counterproductive Aggregation of Experimental Units 

Aggregation of experimental units (usually individual persons or business entities) is quite 

commonly used, especially by governmental entities, in order to (legally and ethically) publicly 

share data / information while preserving personal privacy and/or anonymity. For example, 

summary statistics are computed and available for counties in the US. 

This tactic is applied within Local Control (LC) Strategy [Def. In Local Control analysis 

experimental units are clustered based on the characteristic, characteristics that matter.] for 

statistical analysis in a particularly productive way (i.e. allowing / enabling conditional 

inference) ...to create subsets of experimental units that are relatively well-matched on either one 

or (simultaneously) several different "relevant" x-confounder characteristics. Each LC subset 

(cluster) represents an intersection of publishable x-ranges ...or, at the least, at a given x-vector 

centroid. [Experimental units are within clusters of similar objects.] 

In stark contrast, government entities typically aggregate data for political and/or geographical 

units in a distinctly counterproductive way ...for cities, counties, states and even regions of the 

US. These subsets contain experimental units that are statistically diverse ...corresponding to a 

union of arbitrary (usually un-specified) x-values. 

For example, consider information on smoking and mortality. Individuals supposedly control 

their level of participation in smoking, but city-level data typically report (census or survey data) 

on only the proportions of their adults in various smoking categories. Such information is much 

less potentially informative than, say, mortality rates broken down by smoking-level category. 

Even worse, whatever this initial aggregation is, it usually cannot be subsequently undone by the 

recipient of only aggregated data. While LC strategy can still be applied to analyses of 

counterproductively aggregated data in a remotely productive way, it's already much too late to 

have fully avoided incalculable information loss! 

This committee should make a lot of noise about this key issue. All government data 

administrators (including the census bureau) and health-care data holders really do need to more 

fully appreciate the information-loss penalties they are arbitrarily imposing upon key 

observational data via counterproductive data-reporting practices (geographical aggregation 

and/or fire-walls between databases of competing providers.) 

Obenchain RL, Young SS. 2013. Advancing statistical thinking in observational health care 

research, J Stat Theory Pract 7 (2013), 456–469. 

 

The Promise of Evidence-Based Policymaking 1021
10. A National Academy of Science report was issued on data sharing in the life sciences and the 

principles apply to other areas of research. 

The report comments: 

“Community standards for sharing publication-related data and materials should flow from the 

general principle that the publication of scientific information is intended to move science 

forward. More specifically, the act of publishing is a quid pro quo in which authors receive 

credit and acknowledgment in exchange for disclosure of their scientific findings. An author’s 

obligation is not only to release data and materials to enable others to verify or replicate 

published findings (as journals already implicitly or explicitly require) but also to provide them 

in a form on which other scientists can build with further research. All members of the scientific 

community—whether working in academia, government, or a commercial enterprise—have 

equal responsibility for upholding community standards as participants in the publication system, 

and all should be equally able to derive benefits from it.”

This statement describes the ideal. In practice, the standard must be enforced by those managing 

the scientific process, journal editors and funding agencies. See Young and Karr (2011). 

“Sharing Publication-Related Data and Materials: Responsibilities of Authorship in the Life 

Sciences” http://www.nap.edu/catalog/10613.html. 

Young SS, Karr A. 2011. Deming, data and observational studies: A process out of control and 

needing fixing. Signicance September, 8:122–126. 

 

1022 The Promise of Evidence-Based Policymaking
11. To produce a publishable story, researchers will tweak data and analysis methods. Glaser 

2006 argues that policies need to be in place to “anticipate the behavior of optimizing 

researchers”. Pre-registration of studies and requiring access to data are two steps that “anticipate 

the behavior of optimizing researchers”. Here is the abstract of Glaeser:

“Economists are quick to assume opportunistic behavior in almost every walk of life other than 

our own. Our empirical methods are based on assumptions of human behavior that would not 

pass muster in any of our models. The solution to this problem is not to expect a mass 

renunciation of data mining, selective data cleaning or opportunistic methodology selection, but 

rather to follow Leamer’s lead in designing and using techniques that anticipate the behavior of 

optimizing researchers. In this essay, I make ten points about a more economic approach to 

empirical methods and suggest paths for methodological progress.”

Discussion Paper Number 2122 Researcher Incentives and Empirical Methods. Edward L. 

Glaeser October 2006 HARVARD UNIVERSITY 

 

The Promise of Evidence-Based Policymaking 1023
12. How do you get a statistically significant result? You ask a lot of questions. If 100 questions 

are asked testing at the 5% level, you expect about five chance “nominally statistically 

significant” results, Young and Karr (2011) and Austin et al. (2006). It is well-known that most 

research findings fail to replicate. Austin et al. (2006) provide a humorous exposition of multiple 

testing problems: 

“The purpose of the current study was to demonstrate the pitfalls of multiple hypothesis testing 

and of conducting analyses without prespecified hypotheses.”

“Conclusions: Our analyses illustrate how the testing of multiple, non-prespecified hypotheses 

increases the likelihood of detecting implausible associations. Our findings have important 

implications for the analysis and interpretation of clinical studies.”

Not stating a pre-specified hypothesis is common across all science. 

Austin PC1

, Mamdani MM, Juurlink DN, Hux JE. 2006. Testing multiple statistical hypotheses 

resulted in spurious associations: a study of astrological signs and health. J Clin Epidemiol. 

59:964-969. 

Young SS, Karr A. 2011. Deming, data and observational studies: A process out of control and 

needing fixing. Signicance September, 8:122–126. 

 

1024 The Promise of Evidence-Based Policymaking
13. Place data in a public repository 

Vines et al. (2013) stated 

“...we believe that journal-based mandatory data archiving policies and mandatory data 

availability statements should be more widely adopted.”

In journals that require data sharing, most typically (2/3s of the time) data is not available. 

I systematically requested 50 environmental epidemiology data sets. I got no data sets. Making 

data set available is not happening. 

Vines TH, Andrew RL, Bock DG, Franklin MT. 2013. Mandated data archiving greatly 

improves access to research data. FASEB 

 

The Promise of Evidence-Based Policymaking 1025
14. Personal experience with data access A 

I requested the Chay data set from the authors of the paper. Carlos Dobkin said that he would 

look for the data. My request was in 2014 or so. He found the data and send it to me. We did a 

re-analysis of the data using a new method, Local Control, that we expected to be simpler and 

more robust to covariates. We reached the same conclusion as Chay et al. (2003). We found no 

association between PM2.5 and mortality. 

Chay K, Dobkin C, Greenstone M. 2003. The Clean Air Act of 1970 and adult mortality. J Risk 

Uncertainty 27:279-300. 

Obenchain RL, Young SS. 2013. 2016. Local Control Strategy: Simple Analyses of Air Pollution 

Data can reveal Heterogeneity in Longevity Outcomes. Risk Analysis (in press) 

 

1026 The Promise of Evidence-Based Policymaking
15. Personal experience with data access B, California 

I was unable to get data sets used in air quality/health effect publications. At one point I sent out 

a request to 50 environmental epidemiology researchers. I got no data sets. California has an 

open access policy. I got ~2M e death certificates. The process to get the data took 1-2 months. I 

paid a nominal amount for each year. I had to read and process the data into an analysis data set, 

several weeks of work. I got permission from California to make the analysis data set public. The 

personal identity of the individuals was protected in my analysis and data disclosure by micro 

aggregation. I gave daily deaths only, not individual person records. 

 

The Promise of Evidence-Based Policymaking 1027
16. Personal experience with data access C, London 

London was the site of a massive air quality/mortality event in 1952. There was atemperature 

inversion and air pollution increased dramatically. It is estimated that over 4000 died. We asked 

for London more current daily mortality data. Again there was nominal fee for each year. In 

about two months we received our data. Most interestingly, the government agency also posted 

the data set publicly so that anyone could download the data. We asked for daily data which 

protected individual identity. 

 

1028 The Promise of Evidence-Based Policymaking
17. Personal experience with data access C, CDC NCHS. 

We have an ongoing request for data from the US CDC National Center for Health Statistics. We 

would like to duplicate the work of Zanabett et al. (2009). Zanabetti would not make her analysis 

data set available thereby forcing our research group to duplicate data access and staging work. 

This process of getting access to the analysis data set (protected by micro aggregation) is into its 

2nd year with no end in sight. We do not actually get the data set from NCHS. We must appear at 

a CDC data center and do all our work on CDC computers. We need some rather mainstream 

statistical analysis programs, which they do not have. It remains to be seen if we can use the 

analysis programs we want. All-in-all it is a colossal bureaucratic pain. Typical, the CDC NCHS 

has the data and they can make whatever conditions they like. Hell will freeze over before we get 

access to the data. They don’t appear to care. The analysis data set should be available. The new 

researcher should not have to replicate all the data acquisition and cleaning. 

Our UK supplier of data, vetted the data request and considered personal identity protected. They 

provide us with the data AND put the data up on a web site. They took responsibility for making 

the data public. Multiple legal forms are filled out for CDC/NCHS that make the receiver of the 

data subject to massive fines and imprisonment. Many do not have the time and patience to put 

up with their data access policies. 

If the CDC charged a user fee, they might have the incentive to actually provide data. 

Zanobetti A, Schwartz J. 2009. The effect of fine and coarse particulate air pollution on 

mortality: a national analysis. Environ Health Perspect. 117:898-903. 

 

The Promise of Evidence-Based Policymaking 1029
18. Young-Graham Law 

The clear incentive of a researcher or a regulatory agency is not to make data available. Joe Cecil 

pointed that out in 1986. Effectively nothing has changed in 30 years. Cecil works for the 

Department of Justice and is well worth talking to. Several years ago, I asked if he was still 

involved or would like to be in data access. He said, NO, it is/was all talk and no progress. The 

issue is that there should be an external incentive to share data. For example, researchers should 

promise to share data or they get no funding. The EPA cannot use a paper to support a regulation 

unless the data used in the paper is publicly available. To wit 

Use of Science Transparency 

Any federal agency proposing rule-making or legislation shall specifically name each 

document used to support the proposed rule-making or legislation and provide all data 

used in said document for viewing by the public. 

Federal Study Transparency 

If federal funds are provided for a study, all data relating to the reporting of results of said 

study must be provided for scrutiny by the public at the time of publication.

Following the guidance of Cecil and Griffin an agency can stonewall access to data that is 

government funded.

 

1030 The Promise of Evidence-Based Policymaking
19. Personal experience with data access C, Pope, NEJM 2009 

 Pope made a claim that the decrease in PM2.5 lead to a decrease in human mortality over the 

years ~1980 to ~2000. Of course, many things changed over that time period. Medical science 

improved, smoking declined, income improved, etc. I asked for the data set from Arden Pope 

and he sent me the analysis data set. I was able to show, Young and Xie (2013) that there was no 

effect of PM2.5 in the western US, including in particular California. I was also able to show, 

Young and Xie (2003) that variable most associated with improved mortality was income, 3x 

more important than PM2.5. Later, using the Pope data set we were able to show that if 

covariates are taken into account using Local Control analysis that any effect of PM2.5 on 

mortality varied due to covariates; Young, Obenchain and Lambert (2016). We got similar 

results with a much larger data set, Obenchain and Young (2016). 

The presumed point of doing research is to get valid claims, claims that will replicate. An 

important reason to have access to research data sets is to allow scientists to carefully examine 

the data and methods and importantly subject the data set to additional analysis. For normal 

scientific oversight, there has to be access to data by the public at large or by a trusted 3rd party. 

Young SS, Obenchain RL, Lambert cg. 2016. A problem of bias and response heterogeneity. In 

Standing with Giants, Moghissi and Ross, editors. 

Obenchain RL, Young SS. 2013. 2016. Local Control Strategy: Simple Analyses of Air Pollution 

Data can reveal Heterogeneity in Longevity Outcomes. Risk Analysis (in press) 

 

 

The Promise of Evidence-Based Policymaking 1031
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 04, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sue-91rf

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0032

Comment on FR Doc # 2016-22002

Submitter Information

Name: Terry Greenberg

Address:

Dallas, TX, 75204

Email: tggreenberg@ntarupt.org

General Comment

Commission on Evidence-Based Policymaking Comments

Docket ID: USBC-2016-0003

Federal Register Number: 2016-22002

Ntarupt - North Texas Alliance to Reduce Unintended Pregnancy in Teens

Fund of the Dallas Foundation

624 N. Good-Latimer Expy, Suite 100

Dallas, Texas 75204

November 4, 2016

Dear Chair Katharine G. Abraham, Co-chair Ron Haskins, and members of the

Commission:

Ntarupt welcomes the opportunity to provide comments to the Commission on EvidenceBased Policymaking (CEP). We offer input for questions 16 and 18 based on our

experience as a grantee of a high-quality evidence-based program, the Office of

Adolescent Health's Teen Pregnancy Prevention (TPP) Program. This program has been

1032 The Promise of Evidence-Based Policymaking
recognized as a pioneering example of tiered evidence-based policymaking, and

represents an important contribution to building a body of evidence of what works. This

includes high quality implementation, evaluation, innovation, and learning from results. 

o Our Mission: To increase the quality of life for children, families and communities by

preventing unintended teen pregnancy; and 

o Our Vision: That every young person in our community has the opportunity to set

goals, plan an education and a career, and become self-sufficient before entering into

parenthood.

As a TPP Tier 1 grantee we incorporated evidence into all aspects of our project. The

program model we implemented was chosen from a list of those that had already been

rigorously evaluated and demonstrated to change behavior. 

We also used evidence to guide and improve our project throughout the grant period.

1. We conducted an in depth needs assessment with parents, teens, medical providers

and teachers in the grant area.

2. We selected evidence based interventions, using the Getting to Outcomes framework,

which suited the populations.

3. We have added evaluation of our programs using pre- and post- tests to ensure that the

intervention is doing what is supposed to do.

4. We have delivered to both teens and their parents, reinforcing their knowledge,

attitudes and ability to communicate about delaying sexual initiation. We are finding

that, once we are able to talk to parents, they are very much in favor of their children

being educated.

5. The pilot period has allowed our outreach to be in depth, and we are just now able to

take advantage for recruitment. Cutting funds at this point would not allow the

community and the government to reap the fruits of this investment.

In closing, we consider The TPP Program to be a prime example of high-quality

evidence-based policymaking. It is one of the few government programs that use

evidence and evaluation criteria throughout the grant life cycle. Thank you for

considering our input for the CEP. If you have any questions or need additional

information, please contact me at 214-717-6477or tggreenberg@ntarupt.org

Sincerely,

Terry Goltz Greenberg,

CEO, Ntarupt 

Attachments

CEP_Comments_for_TPP_Tier_1_and_PREP (2)

The Promise of Evidence-Based Policymaking 1033
{Comments are due November 14, 2016. Click here for the submission link. Use this 

version of the letter if you were a TPP Program Tier 1 grantee, a State PREP 

sub/grantee, Competitive, or Tribal PREP grantee.}

Commission on Evidence-Based Policymaking Comments

Docket ID: USBC-2016-0003

Federal Register Number: 2016-22002

Ntarupt – North Texas Alliance to Reduce Unintended Pregnancy in 

Teens

Fund of the Dallas Foundation

624 N. Good-Latimer Expy, Suite 100

Dallas, Texas 75204

November 4, 2016

Dear Chair Katharine G. Abraham, Co-chair Ron Haskins, and 

members of the Commission:

Ntarupt welcomes the opportunity to provide comments to the 

Commission on Evidence-Based Policymaking (CEP). We offer input 

for questions 16 and 18 based on our experience as a grantee of 

a high-quality evidence-based program, the Office of Adolescent 

Health’s Teen Pregnancy Prevention (TPP) Program. This program

has been recognized as a pioneering example of tiered evidencebased policymaking, and represents an important contribution to 

building a body of evidence of what works. This includes high 

quality implementation, evaluation, innovation, and learning 

from results.

o Our Mission: To increase the quality of life for children, 

families and communities by preventing unintended teen 

pregnancy; and 

o Our Vision: That every young person in our community has 

the opportunity to set goals, plan an education and a 

career, and become self-sufficient before entering into 

parenthood.

As a TPP Tier 1 grantee we incorporated evidence into all 

aspects of our project. The program model we implemented was 

chosen from a list of those that had already been rigorously

evaluated and demonstrated to change behavior.

We also used evidence to guide and improve our project 

throughout the grant period.

1. We conducted an in depth needs assessment with parents, 

teens, medical providers and teachers in the grant area.

1034 The Promise of Evidence-Based Policymaking
2. We selected evidence based interventions, using the Getting 

to Outcomes framework, which suited the populations.

3. We have added evaluation of our programs using pre- and

post- tests to ensure that the intervention is doing what 

is supposed to do.

4. We have delivered to both teens and their parents,

reinforcing their knowledge, attitudes and ability to 

communicate about delaying sexual initiation. We are 

finding that, once we are able to talk to parents, they are 

very much in favor of their children being educated.

5. The pilot period has allowed our outreach to be in depth, 

and we are just now able to take advantage for recruitment.

Cutting funds at this point would not allow the community 

and the government to reap the fruits of this investment.

In closing, we consider The TPP Program to be a prime example of 

high-quality evidence-based policymaking. It is one of the few

government programs that use evidence and evaluation criteria 

throughout the grant life cycle. Thank you for considering our 

input for the CEP. If you have any questions or need additional 

information, please contact me at 214-717-6477or

tggreenberg@ntarupt.org

Sincerely,

Terry Goltz 

Greenberg,

CEO, Ntarupt 

The Promise of Evidence-Based Policymaking 1035
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 04, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sul-lbdo

Comments Due: December 14, 2016

Submission Type: API

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0033

Comment on FR Doc # 2016-22002

Submitter Information

Name: Julie Goodman

Address:

Cambridge, MA, 02138

Email: jgoodman@gradientcorp.com

General Comment

See attached file.

Attachments

Gradient_Goodman_Federal _Reg_Comment

1036 The Promise of Evidence-Based Policymaking
 1

9. What specific administrative or legal barriers currently exist for accessing survey and administrative 

data?

Currently, the data application procedures for accessing survey and administrative data are extremely 

time-consuming and cumbersome, and sometimes appear to involve unnecessary steps. For example, the 

national mortality data with individual-level information for specific date and cause of death are 

considered research identifiable data, and the Centers of Disease Control and Prevention (CDC) website 

clearly indicates that such data can only be accessed through a data application to The National Center for 

Health Statistics (NCHS). Before submitting an application to NCHS, one is required to first submit a 

formal data application to the National Association for Public Health Statistics and Information Systems 

(NAPHSIS) for national vital record-based datasets. NAPHSIS only releases aggregated mortality data 

(i.e., at population level), but researchers who explicitly request for individual-level mortality data cannot 

bypass this organization. A review by the NAPHSIS research committee often takes several months to 

complete. It is not clear why it takes this much time, and this often is a major barrier to completing 

research. 

In addition, the application processes and requirements for data access are often unfavorably biased 

towards researchers and institutions in the non-profit sector. For example, the Centers for Medicare & 

Medicaid Services (CMS) collects administrative data on Medicare and Medicaid enrollees, and such data 

can be accessed through formal data applications. However, the CMS does not allow for-profit entities to 

receive physical copies of the data regardless of the purpose of data access whereas such restrictions are 

not applicable to academic institutions. In current CMS rule, for-profit entities can only access the data 

through an online virtual data center for which the costs are often considerably higher than purchasing 

physical data copies. It is not clear how such restrictions towards for-profit entities are justified or 

whether CMS has any legal standing to impose them. 

10. How should the Commission define “qualified researchers and institutions?” To what extent should 

administrative and survey data held by government agencies be made available to “qualified researchers 

and institutions?” 

Qualifications for researchers should be based on education, publications, and past experience. In 

addition, a review of the methods of the proposed work should be informative regarding whether the 

researchers have the qualifications to complete the work (based on their ability to scope the work). 

Whether an institution is non-profit or for-profit should have no bearing on the determining a researcher’s 

qualifications.

All data should be made available to qualified researchers and institutions so long as it is clear the 

research will be beneficial to public health and that it will not violate any person's privacy. 

The Promise of Evidence-Based Policymaking 1037
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 07, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8swf-d2sv

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0034

Comment on FR Doc # 2016-22002

Submitter Information

Name: Martin Slann

General Comment

CEP is orwellian and totalitarian. It enables the federal government to assemble data,

much of them personal, on American citizens from the beginning of their lives. And for

what purpose? If anything, the government already has more than enough information on

each person living here to thoroughly track them throughout their school years and

beyond.

1038 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 07, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8swf-3n7b

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0035

Comment on FR Doc # 2016-22002

Submitter Information

Name: Karen Bracken

Address:

Ocoee, TN, 37361

Email: karen.bracken@reagan.com

General Comment

Are you kidding me? This is totally unconstitutional. When parents find out about this

they will pull their kids from the system as fast as possible. This smells of corporate

fascism. This is an attempt to monitor all human behavior and has nothing to do with

education. We educated the greatest minds in the history of world from the kitchen table

and one room school houses. This is a complete invasion of privacy and parents will no

longer allow this to continue. Please see the enclosed document. Anything other than

what is proposed in this document will be considered an attack on the American people

in an effort to monitor our children and all American citizens. This is a total violation of

the Constitution. You might want to get out a copy and read it.

Attachments

CEP oct 21 2016 written-public-comments

The Promise of Evidence-Based Policymaking 1039
1

September 9, 2016

Dr. Katharine G. Abraham, Chair

Mr. Ron Haskins, Co-Chair

Evidence-Based Policymaking Commission

U.S. Census Bureau

4600 Silver Hill Road

Suitland, MD 20746

Dear Chairwoman Abraham and Co-Chairman Haskins,

We are writing to encourage you to consider including the attached policy recommendations in 

your final report to Congress and the Administration. 

We believe that the Commission can help invest taxpayer dollars in what works by assisting

policymakers at all levels of government in:

x Building evidence about the practices, policies and programs that will achieve the most

effective and efficient results so that policymakers can make better decisions;

x Investing limited taxpayer dollars in practices, policies and programs that use data, evidence

and evaluation to demonstrate they work; and

x Directing funds away from practices, policies, and programs that consistently fail to achieve

measurable outcomes.

Although the Evidence-Based Policymaking Commission Act of 2016 directs the Commission to 

study and report on several important topics including data privacy and data sharing, our attached 

policy proposals focus on the provision that directs the Commission to “make recommendations on 

how best to incorporate outcomes measurement, institutionalize randomized controlled trials, and 

rigorous impact analysis into program design.” 

We thank you in advance for your consideration of our recommendations.

Sincerely,

Actionable Intelligence for Social Policy (AISP)

America Forward

Center for Employment Opportunities

Center for Research and Reform in Education, Johns Hopkins University

KIPP

REDF

Results for America

Sorenson Center for Impact

Success for All Foundation

Sunlight Foundation

cc: Members of the Evidence-Based Policymaking Commission

)@@

0DUPCFS
)FBSJOH

3FTVMUTGPS"NFSJDB

1SFTFOUFS%BWJE.FEJOB

1040 The Promise of Evidence-Based Policymaking
2

INVEST IN WHAT WORKS COALITION RECOMMENDATIONS

Data Collection

x Federal Data Infrastructure: The Commission should consider recommending that Congress

and the Administration provide sufficient funding to help the U.S. Census Bureau accelerate

the process of acquiring key administrative data-sets from local, state, and federal agencies,

and strengthen its infrastructure for processing, standardizing, linking, and making data

available to other government agencies and independent researchers via data use agreements

with strong privacy protections. As part of this effort, the Census Bureau should develop an

inventory of data-sets at the local, state, and federal levels and make this inventory accessible

to government agencies and independent researchers.

x Federal Data Inventories: The Commission should consider recommending that Congress

and the Administration codify into law what is already required by the May 2013 Executive

Order by passing the OPEN Government Data Act. This legislation would mandate that every

federal agency create an enterprise data inventory of all data sets held by the agency and

make these lists public in machine-readable formats with strong privacy protections.

x Federal Data Information Technology: The Commission should consider recommending that

Congress and the Administration provide sufficient funding to allow every federal agency to

update and modernize its IT infrastructure that supports data collection, analysis, sharing, and

usage so that data can be appropriately structured, protected, analyzed and disclosed in line

with the updated information policy of the United States. A 2016 report by the U.S. General

Accountability Office highlighted the urgent need for the U.S. government to modernize its

aging legacy systems.

x Workforce Data: The Commission should recommend that Congress and the Administration

allow the linking of workforce datasets (including but not limited to state and federal

unemployment insurance and new hires data sets) to improve the effectiveness and efficiency

of publicly-supported workforce development programs, as long as the linking is consistent

with strong privacy protections. For example, many states cannot determine the impact of their

job training programs without the ability to link their participant information with information

about wage earnings across multiple states where participants obtain employment.

x State Education and Workforce Data Systems: The Commission should recommend that

Congress and the Administration support the enhancement of the existing State Longitudinal

Data Systems (SLDS) program administered by the U.S. Department of Education, which

helps states integrate education and workforce data, and the proposed expansion of the

Workforce Data Quality Initiative that would help build state and local capacity to track

employment and educational outcomes of Workforce Innovation and Opportunity Act program

participants, including those with disabilities, and provide information about job success rates

and training programs.

x Federal Education Data Identifiers: The Commission should consider recommending that

Congress and the Administration direct federal agencies to standardize the way they collect

The Promise of Evidence-Based Policymaking 1041
3 

and share student-level identifiers (e.g., de-identified but encrypted) so that researchers can 

more effectively evaluate publicly-supported education and workforce development programs. 

This information should be housed in one federal agency in order to promote appropriate 

sharing and usage of this standardized data. 

x Federal Programmatic Data: The Commission should consider recommending that Congress 

and the Administration authorize every federal agency to set aside 1% of their program funds 

for program evaluations that generate programmatic outcomes data that can help make federal 

programs more effective and efficient. 

Data Analysis

x Data Leadership and Infrastructure: The Commission should consider recommending that 

Congress and the Administration direct every federal agency to have a senior staff member 

(i.e., Chief Evaluation Officer or equivalent position) with the authority, staff, and budget to 

develop important programmatic data through the evaluation of its major programs and to use 

this programmatic data and available administrative data to inform the agency’s policies and 

improve its programs.

Data Sharing 

x Local and State Data Systems: The Commission should consider recommending that 

Congress and the Administration clarify that local and state agencies can invest federal 

program funds in strengthening their data infrastructures for processing, standardizing, linking, 

and making data available to other government agencies and independent researchers via 

data use agreements with strong privacy protections. 

x Federal Education Data Infrastructure: The Commission should consider recommending 

that Congress and the Administration strengthen the U.S. Department of Education’s (ED’s)

data infrastructure, including the hiring and training of key analytic staff, to manage the 

collection, quality, release, and analysis of education data with strong privacy protections and 

the support the proposed InformED initiative that would pull together ED’s diverse array of data 

and studies on a particular topic, and allow open data access to help unlock answers to 

pressing education questions and needs. 

Data Usage

x “What Works” Clearinghouses: The Commission should consider recommending that 

Congress and the Administration direct every federal agency to develop a “What Works” 

clearinghouse or evidence exchange with the purpose of making evaluation reports available 

to the public. 

x Performance Management/Continuous Improvement: The Commission should consider 

recommending that Congress and the Administration direct every federal agency to develop 

and operate a performance management system with clear and prioritized outcome-focused 

1042 The Promise of Evidence-Based Policymaking
4 

goals and aligned program objectives and that frequently collects, analyzes, and uses 

administrative and programmatic outcomes data to improve outcomes, return on investment, 

and other dimensions of performance. 

x Federal Grant Programs: The Commission should consider recommending that Congress 

and the Administration direct every federal agency to use evidence of effectiveness, including 

impact analysis and other outcomes measurements based on high-quality administrative and 

programmatic outcomes data, when allocating funds from its 5 largest competitive and noncompetitive grant programs.

x Evaluation and Research: The Commission should consider recommending that Congress 

and the Administration direct every federal agency to have an evaluation policy, evaluation 

plan, and research/learning agenda which ensures that the agency has an intentional 

approach to the collection, analysis, sharing, and usage of administrative and programmatic 

data and publicly release the findings of all completed evaluations to improve the effectiveness 

and efficiency of federal programs. 

x Repurpose for Results: The Commission should consider recommending that Congress and 

the Administration direct every federal agency to use its administrative and programmatic data 

to determine when to shift funds away from practices, policies, and programs which 

consistently fail to achieve desired outcomes and toward evidence-based, results-driven 

solutions. 

The Promise of Evidence-Based Policymaking 1043
CRITERIA Administration 

for Children and 

Families (HHS)

Corporation for 

National and 

Community 

Service

Millennium 

Challenge 

Corporation

U.S. Agency for 

International 

Development

U.S. Department 

of Education

U.S. Dept. of 

Housing & Urban 

Development

U.S. Department 

of Labor

TOTAL SCORE (Out of a possible 100)* 80 72 85 83 80 76 80

1. Leadership: Did the agency have a senior staff member(s) with the authority, 

staff, and budget to evaluate its major programs and inform policy decisions affecting 

them in FY16?

8888889

2. Evaluation and Research: Did the agency have an evaluation policy, evaluation 

SODQDQGUHVHDUFKOHDUQLQJDJHQGDVDQGGLGLWSXEOLFO\UHOHDVHWKHÀQGLQJVRIDOO

completed evaluations in FY16?

9898889

3. Resources: Did the agency invest at least 1% of program funds in evaluations in 

FY16?(Note: Meeting this criteria requires both Agency and Congressional action.) 7 7 10 10 7 6 8

4. Performance Management/Continuous Improvement: Did the agency 

implement a performance management system with clear and prioritized outcomefocused goals and aligned program objectives and measures, and did it frequently 

collect, analyze, and use data and evidence to improve outcomes, return on 

investment, and other dimensions of performance in FY16?

8 7 88899

5. Data: Did the agency collect, analyze, share, and use high-quality administrative 

and survey data - consistent with strong privacy protections - to improve (or help 

other entities improve) federal, state, and local programs in FY16?

9899999

6. Common Evidence Standards/What Works Designations: Did the agency use 

a common evidence framework, guidelines, or standards to inform its research and 

funding decisions and did it disseminate and promote the use of evidence-based 

interventions through a user-friendly tool in FY16?

98889 7 9

7. Innovation: Did the agency have staff, policies, and processes in place that 

encouraged innovation to improve the impact of its programs in FY16? 8 7 9988 7

8. Use of Evidence in 5 Largest Competitive Grant Programs: Did the agency 

use evidence of effectiveness when allocating funds from its 5 largest competitive 

grant programs in FY16?

7 9 81 82 8 7 7

9. Use of Evidence in 5 Largest Non-Competitive Grant Programs: Did the 

agency use evidence of effectiveness when allocating funds from its 5 largest noncompetitive grant programs in FY16?

(Note: Meeting this criteria requires both Agency and Congressional action.)

7 7 N/A N/A 8 7 7

10. Repurpose for Results: In FY16, did the agency shift funds away from any 

practice, policy, or program which consistently failed to achieve desired outcomes?

(Note: Meeting this criteria requires both Agency and Congressional action.)

8 3 8 7776

U S D t f


7KHVHVFRUHVDUHEDVHGRQLQIRUPDWLRQSURYLGHGE\WKHIHGHUDOGHSDUWPHQWVDQGDJHQFLHVLQFOXGHGLQWKLVLQGH[<RXFDQÀQGWKLVEDFNJURXQGLQIRUPDWLRQDVZHOODVDGHVFULSWLRQRIKRZ5)$GHYHORSHGWKHVHVFRUHVDWhttp://results4america.org/policy/invest-in-what-works-indexes/ 1 Since MCC only administers competitive grant programs, its total possible score was 20 for Question #8 and 0 for question #9. 2

6LQFH86$,'RQO\DGPLQLVWHUVFRPSHWLWLYHJUDQWSURJUDPVLWVWRWDOSRVVLEOHVFRUHZDVIRU4XHVWLRQDQGIRUTXHVWLRQ

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

1044 The Promise of Evidence-Based Policymaking
EVIDENCE / EVALUATION CRITERIA

1. Leadership: Did the agency have a senior staff 

member(s) with the authority, staff, and budget 

to evaluate its major programs and inform policy 

decisions affecting them in FY16?

$&):

• ,QODWH)<$&)HVWDEOLVKHGWKHQHZFDUHHUSRVLWLRQRI'HSXW\$VVLVWDQW6HFUHWDU\IRU3ODQQLQJ5HVHDUFKDQG(YDOXDWLRQWRRYHUVHHLWV2IÀFHRI

3ODQQLQJ5HVHDUFKDQG(YDOXDWLRQ235(DQGVXSSRUWHYDOXDWLRQDQGRWKHUOHDUQLQJDFWLYLWLHVDFURVVWKHDJHQF\$&)·VEXGJHWIRUUHVHDUFKDQG

HYDOXDWLRQLQ)<LVDSSUR[LPDWHO\PLOOLRQ$&)·Vevaluation policyJLYHVWKH235('LUHFWRUQRZ'HSXW\$VVLVWDQW6HFUHWDU\ ́DXWKRULW\WR

DSSURYHWKHGHVLJQRIHYDOXDWLRQSURMHFWVDQGDQDO\VLVSODQVDQG«DXWKRULW\WRDSSURYHUHOHDVHDQGGLVVHPLQDWHHYDOXDWLRQUHSRUWVμ235(·VVWDII

RILQFOXGHVH[SHUWVLQUHVHDUFKDQGHYDOXDWLRQPHWKRGVDVZHOODV$&)SURJUDPVDQGSROLFLHVDQGWKHSRSXODWLRQVWKH\VHUYH235(HQJDJHV

LQRQJRLQJFROODERUDWLRQZLWKSURJUDPRIÀFHVWDIIDQGOHDGHUVKLSWRLQWHUSUHWUHVHDUFKDQGHYDOXDWLRQÀQGLQJVDQGWRLGHQWLI\WKHLULPSOLFDWLRQVIRU

SURJUDPPDWLFDQGSROLF\GHFLVLRQV235(DOVRSURYLGHVZULWWHQVXPPDULHVRIHPHUJLQJÀQGLQJVDQGKROGVPRQWKO\PHHWLQJVZLWKDJHQF\DQG

program leadership to discuss their implications.

• :KLOH235(RYHUVHHVPRVWRI$&)·VHYDOXDWLRQDFWLYLW\DQGSURYLGHVRYHUDOOFRRUGLQDWLRQVRPH$&)SURJUDPRIÀFHVDOVRVSRQVRUHYDOXDWLRQV

$&)·VHYDOXDWLRQSROLF\VWDWHV ́,QRUGHUWRSURPRWHTXDOLW\FRRUGLQDWLRQDQGXVHIXOQHVVLQ$&)·VHYDOXDWLRQDFWLYLWLHV$&)SURJUDPRIÀFHVZLOO

FRQVXOWZLWK235(LQGHYHORSLQJHYDOXDWLRQDFWLYLWLHV3URJUDPRIÀFHVZLOOGLVFXVVHYDOXDWLRQSURMHFWVZLWK235(LQHDUO\VWDJHVWRFODULI\HYDOXDWLRQ

TXHVWLRQVDQGPHWKRGRORJLFDORSWLRQVIRUDGGUHVVLQJWKHPDQGDVDFWLYLWLHVSURJUHVV235(ZLOOUHYLHZGHVLJQVSODQVDQGUHSRUWV3URJUDPRIÀFHV

PD\DOVRDVN235(WRGHVLJQDQGRYHUVHHHYDOXDWLRQSURMHFWVRQWKHLUEHKDOIRULQFROODERUDWLRQZLWKSURJUDPRIÀFHVWDIIμ

CNCS:

• &1&6·V2IÀFHRI5HVHDUFKDQG(YDOXDWLRQ'LUHFWRU5	(RYHUVHHVWKHGHYHORSPHQWRIVRFLDOVFLHQFHUHVHDUFKGHVLJQHGWRPHDVXUHWKHLPSDFWRI

CNCS programs and shape policy decisions; encourage a culture of performance and accountability in national and community service programs; 

SURYLGHLQIRUPDWLRQRQYROXQWHHULQJFLYLFHQJDJHPHQWDQGYROXQWHHUPDQDJHPHQWLQQRQSURÀWRUJDQL]DWLRQVDQGDVVLVWLQWKHGHYHORSPHQWDQG

DVVHVVPHQWRIQHZLQLWLDWLYHVDQGGHPRQVWUDWLRQSURMHFWV7KH5	('LUHFWRUZKRLVRYHUVHHLQJ5	(·VPLOOLRQEXGJHWDQGDstaff of 9 in FY16, is 

DPHPEHURI&1&6·V/HDGHUVKLS7HDPDQG3ROLF\&RXQFLO7KH5	('LUHFWRUDOVRPHHWVUHJXODUO\ZLWK&1&63URJUDP'LUHFWRUVWRLGHQWLI\DUHDV

where evidence can be generated and used for various decisions. 

• 7KH5	('LUHFWRUPHHWVDQQXDOO\ZLWKDOO&1&6SURJUDPRIÀFHVWRLGHQWLI\SULRULWLHVDQGQHJRWLDWHZKLFKSRROVRIIXQGVDUHQHHGWRVXSSRUWWKH\HDU·V

priorities. The FY16 plan was developed through a series of formal and informal conversations.

MCC:

• 0&&·V0RQLWRULQJDQG(YDOXDWLRQ0	('LYLVLRQZKLFKIDOOVZLWKLQWKH'HSDUWPHQWRI3ROLF\DQG(YDOXDWLRQ'3(KDVDVWDIIRIDQGDQ)<

budget of $20.6 million in due diligence (DD) funds to be used directly for measuring high-level outcomes and impacts in order to assess the effects of 

its programs and activities. Departments throughout the agency have a total of $75 million in DD funds in FY16. The 0	(0DQDJLQJ'LUHFWRU as well 

as the Departmental 9LFH3UHVLGHQWKDYHWKHDXWKRULW\WRH[HFXWH0	(·VEXGJHWDQGLQIRUPSROLF\GHFLVLRQVDIIHFWLQJLQGHSHQGHQWHYDOXDWLRQV7KH

0	(0DQDJLQJ'LUHFWRUSDUWLFLSDWHVLQWHFKQLFDOUHYLHZVRISURSRVHGLQYHVWPHQWVDVZHOODVLQUHJXODUPRQLWRULQJPHHWLQJVLQRUGHUWRLQIRUPSROLF\

DQGLQYHVWPHQWGHFLVLRQV7KH9LFH3UHVLGHQWVLWVRQWKH$JHQF\·V,QYHVWPHQW0DQDJHPHQW&RPPLWWHHZKLFKH[DPLQHVWKHHYLGHQFHEDVHIRUHDFK

investment before it is approved by the MCC Board and conducts regular oversight over the compact (i.e., grant program) development process.

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

2

The Promise of Evidence-Based Policymaking 1045
EVIDENCE / EVALUATION CRITERIA

1. Leadership: Did the agency have a senior staff 

member(s) with the authority, staff, and budget 

to evaluate its major programs and inform policy 

decisions affecting them in FY16?

86$,':

• 86$,'·V2IÀFHRI/HDUQLQJ(YDOXDWLRQDQG5HVHDUFK/(5 in the %XUHDXIRU3ROLF\3ODQQLQJDQG/HDUQLQJ33/ provides guidance, tools and 

WHFKQLFDODVVLVWDQFHWR86$,'VWDIIDQGSDUWQHUVWRVXSSRUWPRQLWRULQJHYDOXDWLRQDQGOHDUQLQJSUDFWLFHVVRPHRIZKLFKFDQEHIRXQGonline. The 

/(5'LUHFWRURYHUVDZDSSUR[LPDWHO\VWDIIDQGDPLOOLRQEXGJHWLQ)<7KH)<EXGJHWLVHVWLPDWHGWREHFORVHWRWKHVDPHOHYHODVLQ

FY2015.) 

• /(5KROGVVHYHUDOFRQWUDFWVWKDW86$,'PLVVLRQVDQGRIÀFHVFDQXVHIRUEXLOGLQJVWDIIFDSDFLW\LQPRQLWRULQJHYDOXDWLRQDQGOHDUQLQJDQGIRU

FRPPLVVLRQLQJHYDOXDWLRQVDQGPRQLWRULQJVHUYLFHVGLUHFWO\)RUH[DPSOH/(5PDQDJHVWKH0RQLWRULQJDQG(YDOXDWLRQ6HUYLFHV,QGHÀQLWH'HOLYHU\

,QGHÀQLWH4XDQWLW\(9$/0(,',4FRQWUDFWZKLFKDOORZVPLVVLRQVXVLQJWKHLURZQIXQGVWRFRPSHWLWLYHO\ELGVWDWHPHQWVRIZRUNDPRQJSUH

approved companies that have been selected for their monitoring and evaluation capabilities, shortening and simplifying the process for contracting an 

LQGHSHQGHQWHYDOXDWLRQWHDP/(5DOVRPDQDJHVDFODVVURRPWUDLQLQJSURJUDPLQPRQLWRULQJDQGHYDOXDWLRQIRU86$,'VWDII

• 7KH/(5'LUHFWRUSDUWLFLSDWHVLQWKH86$,'$GPLQLVWUDWRU·V/HDGHUVKLS&RXQFLO$/&DVHQLRUOHYHOELZHHNO\PHHWLQJFKDLUHGE\WKH86$,'

$GPLQLVWUDWRUDQGDWWHQGHGE\$VVLVWDQW$GPLQLVWUDWRUVDQGVHOHFW$JHQF\6HQLRU6WDIIZKHQWKHDJHQGDLQFOXGHVLVVXHVUHODWHGWRHYDOXDWLRQ7KH

/(5'LUHFWRUDOVRLQIRUPVSROLF\GHFLVLRQVDFURVVWKHDJHQF\E\SURYLGLQJLQSXWLQWRZRUNLQJJURXSVDQGUHYLHZLQJVWDWHPHQWVGUDIWPHPRVDQGRWKHU

policy products. 

• 2QHRI/(5·VSULPDU\REMHFWLYHVLVWREXLOG86$,'·VFDSDFLW\LQWKHÀHOGRI0RQLWRULQJ(YDOXDWLRQDQG/HDUQLQJ)RUH[DPSOHXQGHUDFRQWUDFWWR

EXLOG0RQLWRULQJDQG(YDOXDWLRQFDSDFLW\DW86$,'0(&DSLQGLYLGXDO86$,'2IÀFHVDQG0LVVLRQVFDQDFFHVV0RQLWRULQJ	(YDOXDWLRQ)HOORZV

DQG/HDUQLQJ)HOORZV7KHVHIHOORZVZRUNZLWKDVSHFLÀFPLVVLRQRURIÀFHIRUPRQWKVWRXSWR\HDUV0(&DSFDQDOVRÀHOGH[SHUWVIRUVKRUW

WHUPWHFKQLFDODVVLVWDQFHIRUDVSHFLÀFPRQLWRULQJRUHYDOXDWLRQUHODWHGWDVNOLNHHYDOXDWLRQGHVLJQRUGHYHORSLQJDPLVVLRQZLGHSHUIRUPDQFH

PDQDJHPHQWSODQ$QRWKHUFRQWUDFWKHOGE\/(5/($51SURYLGHVVXSSRUWWRPLVVLRQVWRPRUHLQWHQWLRQDOO\OHDUQIURPPRQLWRULQJHYDOXDWLRQDQG

H[SHULHQFHDQGDSSO\WKDWOHDUQLQJ7REXLOGVWDIIFDSDFLW\LQGHVLJQLQJRUFRPPLVVLRQLQJLPSDFWHYDOXDWLRQVIXQGHGE\PLVVLRQVRURIÀFHV/(5KDV

KRVWHGFOLQLFVRQ,PSDFW(YDOXDWLRQWRSURYLGH86$,'ÀHOG0LVVLRQVZLWKWRROVUHVRXUFHVDQGKDQGVRQVXSSRUWWRGHVLJQDQLPSDFWHYDOXDWLRQIRUD

future program activity. In addition to providing general capacity-building services in the form of training, clinics, technical assistance, and fellowships, 

/(5VWDIIRFFDVLRQDOO\PDQDJHHYDOXDWLRQVGLUHFWO\RUSDUWLFLSDWHRQHYDOXDWLRQWHDPVIRUHYDOXDWLRQVIXQGHGE\/(5RUIRUWKRVHIXQGHGE\RWKHU

SDUWVRIWKH$JHQF\/(5DOVRFRRUGLQDWHVVHYHUDOFURVVDJHQF\ZRUNLQJJURXSVRUJDQL]HGWRVXSSRUW/HDUQLQJFKDPSLRQVDQGPRQLWRULQJDQG

HYDOXDWLRQVSHFLDOLVWVWKURXJKRXWWKH$JHQF\

86(':

• ('·V,QVWLWXWHRI(GXFDWLRQ6FLHQFHV,(6ZLWKDEXGJHWRIPLOOLRQLQ)<VXSSRUWVUHVHDUFKDQGFRQGXFWVHYDOXDWLRQVRI('·VPDMRU

SURJUDPVLQFOXGLQJLPSDFWHYDOXDWLRQV7KH'LUHFWRURI,(6DQGWKH&RPPLVVLRQHURIWKH1DWLRQDO&HQWHUIRU(GXFDWLRQ(YDOXDWLRQDQG5HJLRQDO

$VVLVWDQFH1&((DUHVXSSRUWHGE\VWDIIZKRRYHUVHHWKHVHHYDOXDWLRQV7KH2IÀFHRI3ODQQLQJ(YDOXDWLRQDQG3ROLF\'HYHORSPHQW·V23(3'

3URJUDPDQG3ROLF\6WXGLHV6HUYLFHV3366KDVDVWDIIRIDQGVHUYHVDVWKH'HSDUWPHQW·VLQWHUQDODQDO\WLFVRIÀFH3366FRQGXFWVVKRUWWHUP

HYDOXDWLRQVWRVXSSRUWFRQWLQXRXVLPSURYHPHQWRISURJUDPLPSOHPHQWDWLRQDQGZRUNVFORVHO\ZLWKSURJUDPRIÀFHVDQGVHQLRUOHDGHUVKLSWRLQIRUP

policy decisions with evidence. While some evaluation funding – such as that for 6SHFLDO(GXFDWLRQ6WXGLHVDQG(YDOXDWLRQV2LVDSSURSULDWHGWR,(6

PLOOLRQLQ)<PRVWHYDOXDWLRQVDUHVXSSRUWHGE\IXQGVDSSURSULDWHGWR('SURJUDPV1&((DQG3366VWDIIZRUNFORVHO\ZLWKSURJUDP

RIÀFHVWRGHVLJQSURJUDPHYDOXDWLRQVWKDWUHÁHFWSURJUDPSULRULWLHVDQGTXHVWLRQV%RWK,(6DQG3366SURYLGHUHJXODUEULHÀQJVRQUHVXOWVWRKHOS

HQVXUHLQIRUPDWLRQFDQEHXVHGE\SURJUDPRIÀFHVIRUSURJUDPLPSURYHPHQW

• %RWK,(6DQG3366VLWRQ('·V(YLGHQFH3ODQQLQJ*URXS(3*ZLWKRWKHUVHQLRUVWDIIIURPWKH('·V2IÀFHRI3ODQQLQJ(YDOXDWLRQDQG3ROLF\

Development23(3'DQGWKH2IÀFHRI,QQRYDWLRQDQG,PSURYHPHQW2,,(3*UHYLHZVDQGDGYLVHVSURJUDPVDQG'HSDUWPHQWOHDGHUVKLSRQKRZ

HYLGHQFHFDQEHXVHGWRLPSURYH'HSDUWPHQWSURJUDPV6HQLRURIÀFLDOVIURP,(62,,DQG3366DUHSDUWRI('·VOHDGHUVKLSVWUXFWXUHDQGZHLJK

LQRQPDMRUSROLF\GHFLVLRQV7KH\SOD\OHDGLQJUROHVLQWKHIRUPDWLRQRIWKH'HSDUWPHQW·VDQQXDOEXGJHWUHTXHVWVUHFRPPHQGDWLRQVDURXQGJUDQW

competition priorities, including evidence, and providing technical assistance to Congress to ensure that evidence informs policy design.

FEDERAL INVEST IN WHAT WORKS INDEX (2016)



1046 The Promise of Evidence-Based Policymaking
EVIDENCE / EVALUATION CRITERIA

1. Leadership: Did the agency have a senior staff 

member(s) with the authority, staff, and budget 

to evaluate its major programs and inform policy 

decisions affecting them in FY16?

USHUD:

• +8'·V2IÀFHRI3ROLF\'HYHORSPHQW	5HVHDUFK3'	5LQIRUPV+8'·VSROLF\GHYHORSPHQWDQGLPSOHPHQWDWLRQE\FRQGXFWLQJVXSSRUWLQJ

DQGVKDULQJUHVHDUFKVXUYH\VGHPRQVWUDWLRQVSURJUDPHYDOXDWLRQVDQGEHVWSUDFWLFHV3'	5DFKLHYHVWKLVPLVVLRQWKURXJKWKUHHLQWHUUHODWHG

core functions: (1) collecting and analyzing national housing market data (including with the Census Bureau); (2) conducting research, program 

HYDOXDWLRQVDQGGHPRQVWUDWLRQVDQGSURYLGLQJSROLF\DGYLFHDQGVXSSRUWWRWKH+8'6HFUHWDU\DQGSURJUDPRIÀFHV3'	5LVOHGE\DQ$VVLVWDQW

6HFUHWDU\ZKRRYHUVHHVVL[RIÀFHVDERXWVWDIILQFOXGLQJDWHDPRIÀHOGHFRQRPLVWVWKDWZRUNLQ+8'·VUHJLRQDORIÀFHVDFURVVWKHFRXQWU\DQG

DEXGJHWRIPLOOLRQLQ)<7KH$VVLVWDQW6HFUHWDU\HQVXUHVWKDWHYLGHQFHLQIRUPVSROLF\GHYHORSPHQWWKURXJKIUHTXHQWSHUVRQDOHQJDJHPHQW

ZLWKRWKHUSULQFLSDOVWDIIWKH6HFUHWDU\DQGH[WHUQDOSROLF\RIÀFLDOV+8'VWDWSHUIRUPDQFHUHYLHZPHHWLQJVVHH4XHVWLRQEHORZIRUDGHVFULSWLRQ

VSHHFKHVWRSROLF\DXGLHQFHVVSRQVRUVKLSRISXEOLFUHVHDUFKEULHÀQJVDQGSROLF\LPSOLFDWLRQVPHPRUDQGD7KH$VVLVWDQW6HFUHWDU\DOVRUHJXODUO\

HQJDJHVZLWKHDFK+8'SURJUDPRIÀFHWRHQVXUHWKDWPHWULFVHYDOXDWLRQVDQGHYLGHQFHLQIRUPSURJUDPGHVLJQEXGJHWLQJDQGLPSOHPHQWDWLRQ

• 3HULRGLF3'	5PHHWLQJVZLWKSURJUDPRIÀFHVHQDEOHNQRZOHGJHVKDULQJDERXWHYDOXDWLRQSURJUHVVDQGHPHUJLQJQHHGVIRUUHVHDUFKHYDOXDWLRQDQG

demonstrations. In recent years, Congress has authorized support for evaluations from program resources through set-asides, transfer authority, and 

supplemental appropriations to implement demonstrations.

86'2/:

• '2/·V&KLHI(YDOXDWLRQ2IÀFHULVDVHQLRURIÀFLDOZLWKUHVSRQVLELOLW\IRUDOODFWLYLWLHVRIWKH&KLHI(YDOXDWLRQ2IÀFH&(2, and coordination of 

HYDOXDWLRQV'HSDUWPHQWZLGH&(2LQFOXGHVIXOOWLPHVWDIIDQGFRQWUDFWRUVSOXVGHWDLOHHVDWDQ\JLYHQWLPH7KH&(2LVUHVSRQVLEOHIRUWKH

DSSURSULDWHGEXGJHWIRUWKH'HSDUWPHQWDO3URJUDP(YDOXDWLRQPLOOLRQLQ)<DQGWKH'HSDUWPHQW·VHYDOXDWLRQVHWDVLGHIXQGVPLOOLRQ

LQ)<,Q)<WKH&(2ZLOOGLUHFWO\RYHUVHHDQHVWLPDWHGPLOOLRQLQHYDOXDWLRQIXQGLQJDQGFROODERUDWHZLWK'2/DJHQFLHVRQDGGLWLRQDO

HYDOXDWLRQVEHLQJFDUULHGRXWWKURXJKDQDGGLWLRQDOPLOOLRQWRHYDOXDWH(PSOR\PHQWDQG7UDLQLQJ$GPLQLVWUDWLRQ(7$SLORWVGHPRQVWUDWLRQVDQG

UHVHDUFKDQGHYDOXDWLRQVRIODUJHJUDQWSURJUDPVLQFOXGLQJWKH3HUIRUPDQFH3DUWQHUVKLS3LORWV3$PHULFDQ$SSUHQWLFHVKLS,QLWLDWLYH$,$WKH

7UDGH$GMXVWPHQW$VVLVWDQFH&RPPXQLW\&ROOHJHDQG&DUHHU7UDLQLQJ7$$&&&7*UDQW3URJUDPDQG5HHQWU\3URJUDPVIRU([2IIHQGHUV7KH&(2

also participates actively in the performance review process during which each operating agency meets with the Deputy Secretary to review progress 

RQSHUIRUPDQFHJRDOVHVWDEOLVKHGIRUWKH\HDUUHTXLUHGXQGHU*RYHUQPHQW3HUIRUPDQFHDQG5HVXOWV$FW*35$7KH&(2·VUROHLVWRLQFRUSRUDWH

HYLGHQFHDQGHYDOXDWLRQÀQGLQJVDVDSSURSULDWHDQGWRLGHQWLI\NQRZOHGJHJDSVWKDWPLJKWEHÀOOHGE\HYDOXDWLRQVRUFRQYH\HYLGHQFHWKDWFDQLQIRUP

policy and program decisions or performance.

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

4

The Promise of Evidence-Based Policymaking 1047
EVIDENCE / EVALUATION CRITERIA

2. Evaluation and Research: Did the agency have 

an evaluation policy, evaluation plan, and research/

learning agenda(s) and did it publicly release the 

ÀQGLQJVRIDOOFRPSOHWHGHYDOXDWLRQVLQ)<"

$&):

• $&)·Vevaluation policyHVWDEOLVKHGLQDGGUHVVHVWKHSULQFLSOHVRIULJRUUHOHYDQFHWUDQVSDUHQF\LQGHSHQGHQFHDQGHWKLFVDQGUHTXLUHV$&)

SURJUDPHYDOXDWLRQDQGUHVHDUFKVWDIIWRFROODERUDWH)RUH[DPSOHWKHSROLF\VWDWHV ́$&)SURJUDPRIÀFHVZLOOFRQVXOWZLWK235(LQGHYHORSLQJ

HYDOXDWLRQDFWLYLWLHVμ$QG ́7KHUHPXVWEHVWURQJSDUWQHUVKLSVDPRQJHYDOXDWLRQVWDIISURJUDPVWDIISROLF\PDNHUVDQGVHUYLFHSURYLGHUVμ

• $&)·V2IÀFHRI3ODQQLQJ5HVHDUFKDQG(YDOXDWLRQ235(SURSRVHVDQHYDOXDWLRQSODQWRWKH$VVLVWDQW6HFUHWDU\HDFK\HDULQDUHDVLQZKLFK

Congress has provided authority and funding to conduct research and evaluation. 

• $&)·VDQQXDOSRUWIROLRUHYLHZVGHVFULEHUHFHQWZRUNDQGRQJRLQJOHDUQLQJDJHQGDVLQWKHDUHDVRIIDPLO\VHOIVXIÀFLHQF\, child and family 

development, and family strengthening, including work related to child welfare, child care, Head Start, (DUO\+HDG6WDUW, strengthening families, teen 

pregnancy prevention and youth development, home visiting, VHOIVXIÀFLHQF\ZHOIDUHDQGHPSOR\PHQW([DPSOHVLQFOXGHÀQGLQJVIURPHead Start 

&$5(6; the %,$6SURMHFWPXOWLSOHUHSRUWVIURPWKHÀUVWQDWLRQDOO\UHSUHVHQWDWLYHstudy of early care and education in over 20 years; HDUO\ÀQGLQJV

RQWKH0DWHUQDO,QIDQWDQG(DUO\&KLOGKRRG+RPH9LVLWLQJSURJUDPDQGa report on challenges and opportunities in using administrative data for 

evaluation. 

• $&)·VHYDOXDWLRQSROLF\UHTXLUHVWKDW ́$&)ZLOOUHOHDVHHYDOXDWLRQUHVXOWVUHJDUGOHVVRIÀQGLQJV(YDOXDWLRQUHSRUWVZLOOSUHVHQWFRPSUHKHQVLYH

ÀQGLQJVLQFOXGLQJIDYRUDEOHXQIDYRUDEOHDQGQXOOÀQGLQJV$&)ZLOOUHOHDVHHYDOXDWLRQUHVXOWVWLPHO\2XVXDOO\ZLWKLQWZRPRQWKVRIDUHSRUW·V

FRPSOHWLRQμ$&)KDVSXEOLFO\UHOHDVHGWKHÀQGLQJVRIDOOFRPSOHWHGHYDOXDWLRQVWRGDWH,Q235(UHOHDVHGQHDUO\120 publications.

CNCS:

• CNCS has an evaluation policyWKDWSUHVHQWVNH\SULQFLSOHVWKDWJRYHUQWKHDJHQF\·VSODQQLQJFRQGXFWDQGXVHRISURJUDPHYDOXDWLRQVULJRU

relevance, transparency, independence, and ethics.

• CNCS has an evaluation plan/learning agenda that is updated annually based on input from agency leadership as well as from emerging evidence 

IURPFRPSOHWHGVWXGLHV7KLVDJHQGDZDVVKDUHGZLWKWKH&1&6%RDUGLQDQGLVUHÁHFWHGLQWKH&1&6&RQJUHVVLRQDO%XGJHW-XVWLÀFDWLRQIRU

Fiscal Year 2016 (pp. 55-56) and Fiscal Year 2017SS&1&6·V5	(2IÀFHLVFXUUHQWO\GHYHORSLQJVFRSHVRIZRUNDQGZLOOPHHWZLWK

SURJUDPRIÀFHUVLQ$SULOWRGLVFXVVWKHP

• CNCS creates four types of reports for public release: research reports produced directly by research and evaluation staff, research conducted by 

WKLUGSDUW\UHVHDUFKÀUPVDQGRYHUVHHQE\UHVHDUFKDQGHYDOXDWLRQVWDIIUHSRUWVSURGXFHGE\&1&6IXQGHGUHVHDUFKJUDQWHHVDQGHYDOXDWLRQ

UHSRUWVVXEPLWWHGE\&1&6IXQGHGSURJUDPJUDQWHHV$OOUHSRUWVFRPSOHWHGDQGFOHDUHGLQWHUQDOO\DUHSRVWHGWRWKH(YLGHQFH([FKDQJH. CNCS 

H[SHFWVWRUHOHDVHDGGLWLRQDOUHSRUWVLQ)<DQGDOOHYDOXDWLRQVDUHH[SHFWHGWREHFOHDUHG

• In FY16 CNCS developed (YDOXDWLRQ&RUH&XUULFXOXP&RXUVHV which are presented to its grantees through a webinar series and is available on the 

CNCS website along with other evaluation resources. The courses are designed to help grantees and other stakeholders easily access materials to 

aid in conducting or managing program evaluations.

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

5

1048 The Promise of Evidence-Based Policymaking
EVIDENCE / EVALUATION CRITERIA

2. Evaluation and Research: Did the agency have 

an evaluation policy, evaluation plan, and research/

learning agenda(s) and did it publicly release the 

ÀQGLQJVRIDOOFRPSOHWHGHYDOXDWLRQVLQ)<"

MCC:

• MCC has developed a 3ROLF\IRU0RQLWRULQJDQG(YDOXDWLRQRI&RPSDFWVDQG7KUHVKROG3URJUDPVLQRUGHUWRHQVXUHWKDWDOOSURJUDPVGHYHORS

DQGIROORZFRPSUHKHQVLYH0RQLWRULQJ	(YDOXDWLRQ0	(SODQVWKDWDGKHUHWR0&&·VVWDQGDUGV7KHPRQLWRULQJFRPSRQHQWRIWKH0	(3ODQOD\V

RXWWKHPHWKRGRORJ\DQGSURFHVVIRUDVVHVVLQJSURJUHVVWRZDUGV&RPSDFWLHJUDQWREMHFWLYHV,WLGHQWLÀHVLQGLFDWRUVHVWDEOLVKHVSHUIRUPDQFH

WDUJHWVDQGGHWDLOVWKHGDWDFROOHFWLRQDQGUHSRUWLQJSODQWRWUDFNSURJUHVVDJDLQVWWDUJHWVRQDUHJXODUEDVLV7KHHYDOXDWLRQFRPSRQHQWLGHQWLÀHV

and describes the evaluations that will be conducted, the key evaluation questions and methodologies, and the data collection strategies that will 

EHHPSOR\HG3XUVXDQWWR0&&·V0	(SROLF\HYHU\SURMHFWPXVWXQGHUJRDQLQGHSHQGHQWHYDOXDWLRQDQGDQDO\VLVWRDVVHVV0&&·VLPSDFW2QFH

HYDOXDWLRQUHSRUWVDUHÀQDOL]HGWKH\DUHSXEOLVKHGRQWKH0&&(YDOXDWLRQ&DWDORJ7RGDWHÀIW\WKUHHLQWHULPDQGÀQDOUHSRUWVKDYHEHHQSXEOLFO\

released, with several additional evaluations expected to be completed and released in the coming months. MCC also produces periodic reports 

for internal and external consumption on results and learning, and holds agency-wide sessions that help to translate evaluation results into lessons 

OHDUQHGIRUIXWXUHFRPSDFWGHYHORSPHQW)LQDOO\LQ)HEUXDU\0&&ODXQFKHG ́1(;7$6WUDWHJ\IRU0&&·V)XWXUH” which outlines new strategic 

directions on how it will invest more in strengthening feedback systems to harness this learning for ongoing adaptation of design and implementation, 

ERWKIRULWVRZQHIIHFWLYHQHVVDQGIRUWKHEHQHÀWRIFRXQWU\SDUWQHUVDQGRWKHUVLQWKHGHYHORSPHQWFRPPXQLW\1(;7LVGHVLJQHGWREHDÀYH\HDU

VWUDWHJLFSODQIRU0&&EXWDOVRLQFOXGHV0&&·VOHDUQLQJDJHQGDE\LQFRUSRUDWLQJDJHQF\ZLGHOHDUQLQJDQGNQRZOHGJHJRDOVWREHSXUVXHGZLWKLQ

that timeframe

86$,':

• 86$,'KDVDQDJHQF\ZLGH(YDOXDWLRQ3ROLF\. The agency just released a reportWRPDUNWKHÀYH\HDUDQQLYHUVDU\RIWKHSROLF\

• 86$,'ÀHOGPLVVLRQVDUHUHTXLUHGWRKDYHDQHYDOXDWLRQSODQDQGDOO86$,'PLVVLRQVDQGRIÀFHVSURYLGHDQLQWHUQDOUHSRUWRQDQDQQXDOEDVLV

RQFRPSOHWHGRQJRLQJDQGSODQQHGHYDOXDWLRQVLQFOXGLQJHYDOXDWLRQVSODQQHGWRVWDUWDQ\WLPHLQWKHQH[WWKUHHÀVFDO\HDUV86$,'SURYLGHVD

3HUIRUPDQFH0DQDJHPHQW3ODQ303 Toolkit to assist missions worldwide. 

• *LYHQ86$,'·VGHFHQWUDOL]HGVWUXFWXUHLQGLYLGXDOSURJUDPVRIÀFHVEXUHDXVDQGPLVVLRQVPD\GHYHORSOHDUQLQJDJHQGDVZKLFKVHYHUDOKDYHGRQH

LQFOXGLQJWKH86$,'·V%XUHDXIRU)RRG6HFXULW\IRUWKH86JRYHUQPHQW·VFeed the Future initiativeDQG86$,'·V'HPRFUDF\+XPDQ5LJKWVDQG

*RYHUQDQFH'5*&HQWHU$OO:DVKLQJWRQ%XUHDXVKDYHDQQXDOHYDOXDWLRQDFWLRQSODQVWKDWORRNDWTXDOLW\DQGXVHDQGLGHQWLI\FKDOOHQJHVDQGWKH

priorities for the year ahead. 

• $OOÀQDO86$,'HYDOXDWLRQUHSRUWVDUHDYDLODEOHRQWKH'HYHORSPHQW([SHULHQFH&OHDULQJKRXVHH[FHSWIRUDSSUR[LPDWHO\ÀYHSHUFHQWRIHYDOXDWLRQV

completed each year that are not public due to principled exceptions to the presumption in favor of openness guided by 20%%XOOHWLQ*XLGDQFH

RQ&ROOHFWLRQRI86)RUHLJQ$VVLVWDQFH'DWD. 

• 86$,'LVFXUUHQWO\XSGDWLQJLWVRSHUDWLRQDOSROLF\IRUSODQQLQJDQGLPSOHPHQWLQJFRXQWU\SURJUDPV$NH\FKDQJHLQWKHSROLF\LVWKDWPLVVLRQVZLOO

LQFOXGHDOHDUQLQJSODQDVSDUWRIWKHLUÀYH\HDUVWUDWHJLFSODQDOVRNQRZQDVWKH&'&67KHSODQZLOORXWOLQHKRZPLVVLRQVZLOOLQFRUSRUDWHOHDUQLQJ

into their programming, including activities like regular portfolio reviews, evaluation tracking and dissemination plans, and other analytic processes to 

better understand the dynamics of their programs and their country contexts.

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

6

The Promise of Evidence-Based Policymaking 1049
EVIDENCE / EVALUATION CRITERIA

2. Evaluation and Research: Did the agency have 

an evaluation policy, evaluation plan, and research/

learning agenda(s) and did it publicly release the 

ÀQGLQJVRIDOOFRPSOHWHGHYDOXDWLRQVLQ)<"

86(':

• ('·V,QVWLWXWHRI(GXFDWLRQ6FLHQFHV,(6VXSSRUWVUHVHDUFKDQGFRQGXFWVHYDOXDWLRQVRI('·VPDMRUSURJUDPV,(6·HYDOXDWLRQSROLFLHVDUHVHWE\

WKH,(66WDQGDUGVDQG5HYLHZ2IÀFHDGGUHVVLQJLVVXHVRIVFLHQWLÀFTXDOLW\LQWHJULW\DQGWLPHO\UHOHDVHRIUHSRUWV5HODWHGWKHNational Board for 

(GXFDWLRQ6FLHQFHV,(6·VDGYLVRU\ERDUGKDVDSSURYHGSROLFLHVIRU3HHU5HYLHZZKLFKDUHLPSOHPHQWHGE\WKH6WDQGDUGVDQG5HYLHZ2IÀFH

• (3*ZRUNVZLWKSURJUDPRIÀFHVDQG('OHDGHUVKLSRQWKHGHYHORSPHQWRI('·VDQQXDOHYDOXDWLRQSODQ7KLVKDSSHQVWKURXJKWKH'HSDUWPHQW·V

DQQXDOVSHQGLQJSODQSURFHVVDQGWKURXJKLGHQWLÀFDWLRQRIKLJKSULRULW\HYDOXDWLRQVIRUXVHRIWKHSRROHGHYDOXDWLRQDXWKRULW\,(6DQG3366ZRUNZLWK

programs to design and share results from relevant evaluations that help with program improvement. 

• ('·VFXUUHQWHYDOXDWLRQVFRQVWLWXWHLWVOHDUQLQJDJHQGD

• ('·VHYDOXDWLRQVDUHSRVWHGRQWKH,(6 website and the 3366 website. See )<$QQXDO3HUIRUPDQFH5HSRUWDQG)<$QQXDO3HUIRUPDQFH3ODQ

IRUDOLVWRI('·VFXUUHQWHYDOXDWLRQV,(6SXEOLFO\UHOHDVHVÀQGLQJVIURPDOORILWVFRPSOHWHGSHHUUHYLHZHGHYDOXDWLRQVRQWKH,(6ZHEVLWH and also in 

the (GXFDWLRQ5HVRXUFHV,QIRUPDWLRQ&OHDULQJKRXVH(5,&

• ('·VVXSSRUWVUHVHDUFKWKURXJK ,(6·V1DWLRQDO&HQWHUIRU(GXFDWLRQ5HVHDUFK1&(5ZKLFKPDNHVJUDQWVIRUSUHNLQGHUJDUWHQWKURXJK

SRVWVHFRQGDU\UHVHDUFKDQG,(6·1DWLRQDO&HQWHUIRU6SHFLDO(GXFDWLRQ5HVHDUFK1&6(5ZKLFKVSRQVRUVDFRPSUHKHQVLYHSURJUDPRIVSHFLDO

HGXFDWLRQUHVHDUFKGHVLJQHGWRH[SDQGWKHNQRZOHGJHDQGXQGHUVWDQGLQJRILQIDQWVWRGGOHUVFKLOGUHQDQG\RXQJDGXOWVZLWKGLVDELOLWLHV,(6DOVR

PDQDJHVWKH5HJLRQDO(GXFDWLRQDO/DERUDWRU\5(/) program, which supports districts, states, and boards of education throughout the United States 

to use research in decision making.

USHUD:

• +8'·Vevaluation policyVHHSS2ZKLFKJXLGHV+8'·V5HVHDUFK5RDGPDSGHVFULEHGEHORZLQFOXGHVUHDFKLQJRXWWRLQWHUQDODQG

external stakeholders through a participatory approach; making research planning systematic, iterative, and transparent; focusing on research 

TXHVWLRQVWKDWDUHWLPHO\IRUZDUGORRNLQJSROLF\UHOHYDQWDQGOHYHUDJH+8'·VFRPSDUDWLYHDGYDQWDJHVDQGSDUWQHUVKLSRSSRUWXQLWLHVDOLJQLQJ

UHVHDUFKZLWK+8'·VVWUDWHJLFJRDOVDQGXVLQJULJRURXVUHVHDUFKPHWKRGVLQFOXGLQJSURJUDPGHPRQVWUDWLRQVZLWKUDQGRPL]HGFRQWUROOHGWULDOVDV

appropriate. 

• +8'·V2IÀFHRI3ROLF\'HYHORSPHQWDQG5HVHDUFK3'	5KDVGHYHORSHGWKH5HVHDUFK5RDGPDS)<)<VHHSSDVWUDWHJLFÀYH\HDU

SODQIRUSULRULW\SURJUDPHYDOXDWLRQVDQGUHVHDUFKWREHSXUVXHGJLYHQDVXIÀFLHQWO\UREXVWOHYHORIIXQGLQJ3'	5DOVRLQWHJUDWHGLWVHYDOXDWLRQSODQ

LQWR+8'·V)<)<6WUDWHJLF3ODQVHHSSWRVWUHQJWKHQWKHDOLJQPHQWEHWZHHQHYDOXDWLRQDQGSHUIRUPDQFHPDQDJHPHQW'XULQJ)<

3'	5LVXVLQJVLPLODUSULQFLSOHVDQGPHWKRGVWRrefreshWKH5RDGPDSWRDGGUHVVHPHUJLQJUHVHDUFKWRSLFV

• HUD also employs its role as convener to help establish frameworks for evidence, metrics, and future research. 

• $FFRUGLQJWRWKH5HVHDUFK5RDGPDS)<)<VHHSDVSDUWRI+8'·VDQQXDOSHUIRUPDQFHUHSRUWUHTXLUHGE\*35$ ́DJHQFLHVVKRXOG

GHVFULEHÀQGLQJVIURPDJHQF\IXQGHGHYDOXDWLRQVRURWKHUUHVHDUFKFRPSOHWHGGXULQJWKHSULRUÀVFDO\HDUμ)XUWKHU ́$JHQFLHVDUHH[SHFWHGWRKDYHD

ZHESDJHRQWKHDJHQF\·VHYDOXDWLRQVRUOLQNVWRRWKHUHYDOXDWLRQVUHOHYDQWWRWKHDJHQF\·VZRUNZLWKVXPPDULHVRIWKHÀQGLQJVDQGVSHFLÀFFLWDWLRQVμ

3'	5SXEOLVKHVDQGGLVVHPLQDWHVHYDOXDWLRQVLQDWLPHO\IDVKLRQWKURXJKWKHVHDQGRWKHUPHDQVDQGDOVRIROORZVDSROLF\RILQFOXGLQJODQJXDJHLQ

research and evaluation contracts that allows researchers to independently publish results, even without HUD approval, after not more than 6 months.

86'2/:

• '2/KDVDIRUPDO(YDOXDWLRQ3ROLF\6WDWHPHQW that formalizes the principles that govern all program evaluations in the Department, including 

PHWKRGRORJLFDOULJRULQGHSHQGHQFHWUDQVSDUHQF\HWKLFVDQGUHOHYDQFH,QDGGLWLRQWKH&KLHI(YDOXDWLRQ2IÀFHSXEOLFO\FRPPXQLFDWHVWKHVWDQGDUGV

DQGPHWKRGVH[SHFWHGLQ'2/HYDOXDWLRQVLQIRUPDOSURFXUHPHQWVWDWHPHQWVRIZRUN

• '2/DOVRGHYHORSVLPSOHPHQWVDQGSXEOLFO\UHOHDVHVDQDQQXDO(YDOXDWLRQ3ODQLH'HSDUWPHQWOHYHOOHDUQLQJDJHQGDDVGRHDFKRI'2/·V

RSHUDWLQJDJHQFLHV7KHDJHQF\OHDUQLQJDJHQGDVIRUPWKHEDVLVIRUWKH'2/·V(YDOXDWLRQ3ODQ7KH(YDOXDWLRQ3ODQ was released for public 

FRPPHQWLQWKH)HGHUDO5HJLVWHUDQGLVSRVWHGRQWKH&(2ZHEVLWH

• $OO'2/UHSRUWVDQGÀQGLQJVDUHSXEOLFO\UHOHDVHGDQGSRVWHGRQWKH&(2ZHEVLWH'2/DJHQFLHVDOVRSRVWDQGUHOHDVHWKHLUreports.

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

7

1050 The Promise of Evidence-Based Policymaking
EVIDENCE / EVALUATION CRITERIA

3. Resources: Did the agency invest at least 1% of 

program funds in evaluations in FY16?

(Note: Meeting this criteria requires both Agency and 

Congressional action.)

$&):

• ,Q)<$&)SODQVWRVSHQGPLOOLRQRQHYDOXDWLRQVUHSUHVHQWLQJRI$&)·VELOOLRQEXGJHWLQ)<LQDGGLWLRQWRLQYHVWPHQWVLQ

HYDOXDWLRQVE\$&)JUDQWHHV

• 7KH$GPLQLVWUDWLRQ·V)<budget request seeks authority for numerous new investments in learning, including set-asides of up to 1.5% of the Social 

6HUYLFHV%ORFN*UDQWSURJUDPPLQ)<LQFOXGLQJPLOOLRQIRUDGHPRQVWUDWLRQDQGHYDOXDWLRQRQVXSSO\LQJGLDSHUVWRORZLQFRPHIDPLOLHV

DQGPLOOLRQIRUUHVHDUFKDQGHYDOXDWLRQLQ)<DQGRIWKH&RPPXQLW\6HUYLFHV%ORFN*UDQWSURJUDPPLOOLRQLQ)<IRUHYDOXDWLRQV

CNCS:

• &1&6SODQVWRVSHQGDWRWDORIPLOOLRQUHSUHVHQWLQJRI&1&6·VELOOLRQEXGJHWLQ)<LQHYDOXDWLRQDQGHYDOXDWLRQFDSDFLW\EXLOGLQJ

DFWLYLWLHV5	(HYDOXDWLRQDQGSURJUDPIXQGVFRPELQHGLQFOXGLQJ

- $1.2 million of Senior Corps funding for the supplemental award, program funding used for evaluation and evidence purposes versus funding 

given to sponsor organizations;

- $400,000 in Senior Corps funds for the longitudinal survey in FY16, an evaluation of the volunteers who participate in Senior Corps programs 

(examining their health and well-being outcomes over time);

LQVXSSRUWLQJUHYLHZVRIJUDQWHHHYDOXDWLRQSODQVDQGUHSRUWVLQFOXGLQJIRUUHVHDUFK	HYDOXDWLRQH[SHUWLVHWRUHYLHZVWXGLHV

submitted by grantees applying for funding; and

PLOOLRQLQHYDOXDWLRQIXQGVDQG6,)IXQGVWRVXSSRUWSURJUDPHYDOXDWLRQVDQGWHFKQLFDODVVLVWDQFHIRUJUDQWHHVWRFRQGXFWHYDOXDWLRQV

MCC:

• ,Q)<0	(LQYHVWHGRYHUPLOOLRQRQPRQLWRULQJDQGHYDOXDWLRQRI&RPSDFWSURMHFWVZKLFKDPRXQWHGWRRI&RPSDFWVSHQGLQJIRU)<

PLOOLRQ&DOFXODWLRQVDUHVWLOORQJRLQJIRU)<+RZHYHU0&&H[SHFWVWRGLVEXUVHDPRXQWVVLPLODUWR)<7KLVLVUHÁHFWHGLQQXPEHUVIRU

4	4LQ)<DVRI0DUFKZKLFKVKRZ0	(LQYHVWPHQWVRIPLOOLRQ

86$,':

• ,Q)<86$,'PLVVLRQVDQGRIÀFHVUHSRUWHGFRPSOHWLQJHYDOXDWLRQVZLWKUHVRXUFHVWRWDOLQJDSSUR[LPDWHO\PLOOLRQDQGPDQDJLQJDQRWKHU

251 ongoing evaluations, many that span more than one year, with total ongoing budgets estimated to reach $168.9 million. Overall spending on 

HYDOXDWLRQVFRPSOHWHGRURQJRLQJLQ)<PLOOLRQUHSUHVHQWVDERXWRI86$,'·VELOOLRQ)<SURJUDPEXGJHW

• 7KLVDPRXQWGRHVQRWLQFOXGHWKH2IÀFHRI/HDUQLQJ(YDOXDWLRQDQG5HVHDUFKEXGJHWZKLFKSULPDULO\IRFXVHVRQHYDOXDWLRQFDSDFLW\EXLOGLQJDQG

technical assistance ($17.5 million FY15) or the investment in the Demographic and Health Surveys'+6PLOOLRQWRWDOLQ)<)<RU

surveys funded by other sector programs that often make up some of the underlying data used in many evaluations.

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

8

The Promise of Evidence-Based Policymaking 1051
EVIDENCE / EVALUATION CRITERIA

3. Resources: Did the agency invest at least 1% of 

program funds in evaluations in FY16?

(Note: Meeting this criteria requires both Agency and 

Congressional action.)

86(':

• 7KHUHDUHDYDULHW\RIZD\VWKDW('JHQHUDOO\VXSSRUWVHYDOXDWLRQVDVZHOODVHYDOXDWLRQWHFKQLFDODVVLVWDQFHDQGFDSDFLW\EXLOGLQJ,Q)<DQG

)<('KDVWKHDXWKRULW\WRUHVHUYHXSWRRI(6($IXQGV2H[FHSW7LWOH,IXQGV7LWOH,,,IXQGVDQGIXQGVIRUSURJUDPVWKDWDOUHDG\KDYHDQ

HYDOXDWLRQSURYLVLRQ2WRHYDOXDWH(6($SURJUDPVZKLFK5)$HVWLPDWHVDWPLOOLRQIRU)<,Q)<('SRROHGPLOOLRQWRFRQGXFW

HYDOXDWLRQVWKDWZLOOEXLOGQHZHYLGHQFHDERXWWKHIROORZLQJSURJUDPV(6($7LWOH,3DUW$WKHPLJUDQWHGXFDWLRQSURJUDPDQGWKH,QGLDQ(GXFDWLRQ

/($*UDQWV3URJUDPDQGDOVRSURYLGHGFRQWLQXHGVXSSRUWIRUSURJUDPHYDOXDWLRQVRQ(6($7LWOH,3DUW$(6($7LWOH,3DUW'DQG(6($7LWOH,,,

ZKLFKEHJDQZLWK)<SRROHGIXQGLQJ7KH(YHU\6WXGHQW6XFFHHGV$FW(66$RIZKLFKUHDXWKRUL]HG(6($FRQWLQXHVWKHSRROLQJDXWKRULW\

DQGLQFOXGHV7LWOH,,,DVDQDOORZDEOHSURJUDPIURPZKLFKWRSRROIXQGV(66$DOVRDXWKRUL]HVIRUDQHYDOXDWLRQRI7LWOH,IRU)<)<('

spent over $60 million on program evaluations in FY15.

• ,QDGGLWLRQPDQ\('TSURJUDPVDUHDXWKRUL]HGWRVXSSRUWQDWLRQDODFWLYLWLHVLQFOXGLQJSURJUDPHYDOXDWLRQVDQGVRPHSURJUDPVHQFRXUDJHWKHLU

JUDQWHHVWRFRQGXFWSURMHFWOHYHOHYDOXDWLRQV2QHRIWKHNH\OHVVRQVIURPLKDVEHHQWKDWKLJKTXDOLW\WHFKQLFDODVVLVWDQFHIRUJUDQWHHVRQSURMHFW

OHYHOHYDOXDWLRQVLVFULWLFDOWRSURGXFLQJFUHGLEOHLQIRUPDWLRQRQSURMHFWRXWFRPHV,Q)<LLQYHVWHGPRUHWKDQPLOOLRQRILWVDSSURSULDWLRQLQ

evaluation technical assistance – virtually no other discretionary grant program has the authority or means to fund such a robust vehicle for technical 

DVVLVWDQFH('ZLWKWKHH[SHUWLVHRI,(6KDVEHJXQWRSLORWOHVVH[SHQVLYHDSSURDFKHVWRHYDOXDWLRQWHFKQLFDODVVLVWDQFHIRUSURJUDPVOLNHFirst in 

the World ($1.5 m), and 6XSSRUWLQJ(IIHFWLYH(GXFDWRU'HYHORSPHQW ($~800,000), which also tasks its grantees with producing rigorous project-level 

evaluations. 

• $FFRUGLQJWR5)$HVWLPDWHVRYHUDOOVSHQGLQJRQHYDOXDWLRQPLOOLRQLQ)<DQGHYDOXDWLRQWHFKQLFDODVVLVWDQFHDQGFDSDFLW\EXLOGLQJ

PLOOLRQLQ)<UHSUHVHQWVRI('·VELOOLRQGLVFUHWLRQDU\EXGJHWLQ)<

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

9

1052 The Promise of Evidence-Based Policymaking
EVIDENCE / EVALUATION CRITERIA

3. Resources: Did the agency invest at least 1% of 

program funds in evaluations in FY16?

(Note: Meeting this criteria requires both Agency and 

Congressional action.)

USHUD:

• For FY16, Congress appropriated $50 million for core research activities; $10 million for research, evaluations, and demonstrations; and $25 million 

IRUWHFKQLFDODVVLVWDQFHLQWKH5HVHDUFK	7HFKQRORJ\DFFRXQW7KLVPLOOLRQWRWDOKDOIRIWKHUHTXHVWHGDPRXQWHTXDOVSHUFHQWRI+8'·V

$45.5 billion of FY16 program budget authorityQHWRI6DODULHVDQG([SHQVHV7KHPLOOLRQGHYRWHGWRUHVHDUFKHYDOXDWLRQVDQGGHPRQVWUDWLRQV

LVDERXWSHUFHQWRIWKHPLOOLRQWRWDO$GGLWLRQDOO\PXFKRIWKHPLOOLRQLVXVHGIRUVXUYH\VHVSHFLDOO\IRUWKH$PHULFDQ+RXVLQJ6XUYH\DQG

RWKHUGDWDDFTXLVLWLRQWKDWLQGLUHFWO\VXSSRUWHYDOXDWLRQRI+8'·VPLVVLRQDFWLYLWLHVLQGRPDLQVVXFKDVDIIRUGDEOHKRXVLQJDQGKRXVLQJÀQDQFH

• In FY10, Congress authorized the transfer of up to 1% of funds from individual HUD program funds to the Transformation Initiative (TI) Fund for: (1) 

UHVHDUFKHYDOXDWLRQDQGSURJUDPPHWULFVSURJUDPGHPRQVWUDWLRQVWHFKQLFDODVVLVWDQFHDQGLQIRUPDWLRQWHFKQRORJ\$IWHU)<+8'QR

longer sought to fund information technology with the TI Fund, and Congress has not provided requested levels of evaluation funding or, since FY14, 

supported transfers to TI

86'2/:

• ,Q)<'2/·V&(2ZLOOGLUHFWO\RYHUVHHDQHVWLPDWHGPLOOLRQLQHYDOXDWLRQIXQGLQJ$GGLWLRQDOO\&(2ZLOOFROODERUDWHZLWK'2/DJHQFLHV

RQDGGLWLRQDOHYDOXDWLRQVEHLQJFDUULHGRXWZLWKDSSUR[LPDWHO\PLOOLRQWRHYDOXDWH(PSOR\PHQWDQG7UDLQLQJ$GPLQLVWUDWLRQ(7$SLORWV

GHPRQVWUDWLRQVDQGUHVHDUFKDQGHYDOXDWLRQVRIODUJHJUDQWSURJUDPVLQFOXGLQJIRUH[DPSOHWKH3HUIRUPDQFH3DUWQHUVKLS3LORWV3$PHULFDQ

$SSUHQWLFHVKLS,QLWLDWLYH$,$WKH7UDGH$GMXVWPHQW$VVLVWDQFH&RPPXQLW\&ROOHJHDQG&DUHHU7UDLQLQJ7$$&&&7*UDQW3URJUDPDQG5HHQWU\

3URJUDPVIRU([2IIHQGHUV7KHFRPELQHGDPRXQWRIPLOOLRQUHSUHVHQWVDSSUR[LPDWHO\RI'2/·V)<GLVFUHWLRQDU\EXGJHWRIELOOLRQ

(For many of the largest programs, however, up to 5% of their budgets is dedicated to program evaluation and related activities).

• '2/·V&KLHI(YDOXDWLRQ2IÀFHGLUHFWO\IXQGVHYDOXDWLRQVDQGDOVRFRPELQHV&(2IXQGVZLWKDJHQF\IXQGVWRMRLQWO\VSRQVRUVRPHHYDOXDWLRQV

The largest discretionary programs can use program funds for evaluations and technical assistance, often up to 5% by statute. For example, three 

separate rounds of grants funded by H1-B worker visa fees totaling about $400 million in FY16 support training particular populations, such as high 

VFKRROVWXGHQWVWUDQVLWLRQLQJWRZRUNORQJWHUPXQHPSOR\HGZRUNHUVDQGDSSUHQWLFHVKLSWUDLQLQJDQGEHWZHHQDQGRIWKHVHJUDQWIXQGV

DWOHDVWPLOOLRQLVH[SHFWHGWREHLQYHVWHGLQHYDOXDWLRQVLQ)<$QRWKHUH[DPSOHLQ)<DQG)<XSWRRIWKHIXQGVDYDLODEOHIRUWKH

workforce innovation activities were used for technical assistance and evaluations related to the projects carried out with these funds. The legislation 

SURYLGHGIXUWKHUWKDWWKH6HFUHWDU\PD\DXWKRUL]HDZDUGHHVWRXVHDSRUWLRQRIDZDUGHGIXQGVIRUHYDOXDWLRQXSRQWKH&KLHI(YDOXDWLRQ2IÀFHU·V

DSSURYDORIDQHYDOXDWLRQSODQ)XUWKHUVHYHUDO'2/DJHQFLHVDOVRKDYHVHSDUDWHHYDOXDWLRQDSSURSULDWLRQV'2/VWXGLHVIXQGHGWKURXJKLQGLYLGXDO

DJHQFLHVDUHDOVRFRRUGLQDWHGZLWK'2/·V&(2

• 7KH$GPLQLVWUDWLRQ·V)<)<EXGJHWUHTXHVWVUHFRPPHQGHGDOORZLQJWKH866HFUHWDU\RI/DERUWRVHWDVLGHXSWRRIDOORSHUDWLQJDJHQFLHV·

EXGJHWVIRUHYDOXDWLRQVFRRUGLQDWHGE\&(2,Q)<V&RQJUHVVDXWKRUL]HGWKH6HFUHWDU\WRVHWDVLGHXSWRRIWKHVHIXQGVIRU

HYDOXDWLRQVLQDGGLWLRQWRWKHVHSDUDWHHYDOXDWLRQIXQGVWKDWH[LVWLQPDQ\'2/DJHQFLHV,Q)<&RQJUHVVDXWKRUL]HG'2/WRVHWDVLGHXSWR

of operating agency budgets in evaluations.

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

10

The Promise of Evidence-Based Policymaking 1053
EVIDENCE / EVALUATION CRITERIA

4. Performance Management/Continuous 

Improvement: Did the agency implement a 

performance management system with clear and 

prioritized outcome-focused goals and aligned 

program objectives and measures, and did it frequently 

collect, analyze, and use data and evidence to improve 

outcomes, return on investment, and other dimensions 

of performance in FY16?

$&):

• $&)·VSHUIRUPDQFHPDQDJHPHQWIUDPHZRUNIRFXVHVRQRXWFRPHVDQGDLPVIRUFRRUGLQDWHGDQGUHVXOWVRULHQWHGPDQDJHPHQWDQGRSHUDWLRQVDFURVV

DOO$&)SURJUDPV$&)·V6WUDWHJLF3ODQHVWDEOLVKHVÀYHSULRULWLHVZKLFKDOLJQZLWKWKH++66WUDWHJLF3ODQ$&)IRUPDOO\UHYLHZVSURJUHVVWRZDUGLWV

6WUDWHJLF3ODQJRDOVHYHU\TXDUWHU$&)·V6WUDWHJLF3ODQHVWDEOLVKHVÀYHSULRULWLHVLQVXSSRUWRIWKHDJHQF\·VPLVVLRQRIIRVWHULQJKHDOWKDQGZHOOEHLQJ

E\SURYLGLQJIHGHUDOOHDGHUVKLSSDUWQHUVKLSDQGUHVRXUFHVIRUWKHFRPSDVVLRQDWHDQGHIIHFWLYHGHOLYHU\RIKXPDQVHUYLFHV7KHÀYHSULRULWLHVDUH

3URPRWHHFRQRPLFKHDOWKDQGVRFLDOZHOOEHLQJIRULQGLYLGXDOVIDPLOLHVDQGFRPPXQLWLHV3URPRWHKHDOWK\GHYHORSPHQWDQGVFKRROUHDGLQHVV

IRUFKLOGUHQHVSHFLDOO\WKRVHLQORZLQFRPHIDPLOLHV3URPRWHVDIHW\DQGZHOOEHLQJRIFKLOGUHQ\RXWKDQGIDPLOLHV6XSSRUWXQGHUVHUYHGDQG

XQGHUUHSUHVHQWHGSRSXODWLRQVDQG8SJUDGHWKHFDSDFLW\RI$&)WRPDNHDGLIIHUHQFHIRUIDPLOLHVDQGFRPPXQLWLHV

• $&)DLPVWRGHYHORSSHUIRUPDQFHPHDVXUHVWKDWDUHPHDQLQJIXODQGFDQEHXVHGE\SURJUDPPDQDJHUVOHDGHUVKLSRXWVLGHVWDNHKROGHUVDQG

&RQJUHVVWRDVVHVVDQGFRPPXQLFDWHSURJUHVV5HVXOWVIRUWKHVHPHWULFVDUHUHSRUWHGDQQXDOO\LQWKH$&)&RQJUHVVLRQDO%XGJHW-XVWLÀFDWLRQ$&)

UHSRUWVRQDWRWDORISHUIRUPDQFHPHDVXUHVRXWFRPHPHDVXUHVDQGRXWSXWPHDVXUHVLQWKH)<&RQJUHVVLRQDO%XGJHW-XVWLÀFDWLRQ$

VHOHFWLRQRI$&)SHUIRUPDQFHPHDVXUHVLVDOVRKLJKOLJKWHGDVSDUWRIWKH)<++6$QQXDO3HUIRUPDQFH3ODQDQG5HSRUWZKLFKGHVFULEHV++6·

progress toward achieving the goals and objectives described in the )<++66WUDWHJLF3ODQ. This report includes the most recent results 

DYDLODEOHDWWKHHQGRI)<IRU++6LQFOXGLQJ$&)

• $VSDUWRIWKH)<3UHVLGHQW·V%XGJHWUHTXHVW++6DQQRXQFHGWKH)<++63ULRULW\*RDOV$&)LVWKHOHDGDJHQF\IRUWKHJRDOWR ́,PSURYH

WKHTXDOLW\RIHDUO\FKLOGKRRGSURJUDPVIRUORZLQFRPHFKLOGUHQμLQFROODERUDWLRQZLWKWKH6XEVWDQFH$EXVHDQG0HQWDO+HDOWK6HUYLFHV$GPLQLVWUDWLRQ

6$0+6$DQGWKH+HDOWK5HVRXUFHVDQG6HUYLFHV$GPLQLVWUDWLRQ+56$2QDTXDUWHUO\EDVLV$&)SURYLGHVXSGDWHVRQWKLV3ULRULW\*RDORQ

3HUIRUPDQFHJRY$&)DOVRSDUWLFLSDWHVLQWKH*35$0$UHTXLUHG6WUDWHJLF2EMHFWLYH$QQXDO5HYLHZSURFHVV++6PDLQWDLQVDQLQWHUQDOSHUIRUPDQFH

GDVKERDUGZKHUH$&)SURYLGHVUHJXODUSHUIRUPDQFHXSGDWHVRQWKHSHUIRUPDQFHPHDVXUHVLQFOXGHGLQWKHDQQXDO$&)%XGJHW5HTXHVW

CNCS:

• &1&6·VSHUIRUPDQFHPDQDJHPHQWIUDPHZRUNLVGHVFULEHGLQWKH&RQJUHVVLRQDO%XGJHW-XVWLÀFDWLRQIRUFiscal Year 2016SDQGFiscal Year 2017

(p.6). 

• &1&6KDVDIRFXVHGVHWRI$JHQF\:LGH3ULRULW\0HDVXUHVGHULYHGIURPWKH6WUDWHJLF3ODQ(YHU\&1&63URJUDPFRQWULEXWHVWRWKH

$JHQF\:LGH3ULRULW\0HDVXUHV7KHUHDUHDOVRVSHFLÀFJUDQWHHVSRQVRUPHDVXUHVWKDWUROOXSLQWRWKH$JHQF\:LGH3ULRULW\0HDVXUHVZKLFKFDQEH

found in the $JHQF\:LGH3ULRULW\0HDVXUHVFKDUW*UDQWHHVDUHUHTXLUHGWRVHOHFWDWOHDVWRQHQDWLRQDOSHUIRUPDQFHPHDVXUHDQGWKH\DUHUHTXLUHG

to report performance measures data annually. CNCS encourages grantees to use these measure for continuous program improvement. CNCS uses 

the agency-wide priority measures to assess its own progress toward attaining the goals and objectives of its strategic plan.

• $GGLWLRQDOO\&1&6SURGXFHVVWDWHSURÀOHUHSRUWV, which provide a picture of agency resources in each state at a given point. These reports contain a 

number of priority indicators, including the number of participants engaged in national service activities as well as the amount of non-CNCS resources 

JHQHUDWHGE\WKHDJHQF\·VSURJUDPV$ORQJZLWKLWVVWDNHKROGHUV&1&6XVHVWKLVLQIRUPDWLRQWRXQGHUVWDQGWKHFDSDFLW\RIVHUYLFHDYDLODEOHLQ

different geographic regions and discuss related implications with key service partners.

• &1&6·V&KLHI2SHUDWLQJ2IÀFHU&22LVFXUUHQWO\SLORWLQJDSURRIRIFRQFHSWSHUIRUPDQFHIUDPHZRUNWKDWDOLJQVZLWK*35$7KH&22LVÀQDOL]LQJ

their objectives, measures, and targets, and they will be conducting quarterly performance reviews starting in the fourth quarter of FY16. The goal is to 

establish an effective performance framework within the COO, work agency-wide to implement a similar process, and have an enhanced performance 

PDQDJHPHQWIUDPHZRUNLQSODFHDV&1&6EHJLQVLWVQHZ6WUDWHJLF3ODQQLQJSURFHVVLQ

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

11

1054 The Promise of Evidence-Based Policymaking
EVIDENCE / EVALUATION CRITERIA

4. Performance Management/Continuous 

Improvement: Did the agency implement a 

performance management system with clear and 

prioritized outcome-focused goals and aligned 

program objectives and measures, and did it frequently 

collect, analyze, and use data and evidence to improve 

outcomes, return on investment, and other dimensions 

of performance in FY16?

MCC:

• 0&&PRQLWRUVSURJUHVVWRZDUGVFRPSDFWUHVXOWVRQDTXDUWHUO\EDVLVXVLQJSHUIRUPDQFHLQGLFDWRUVWKDWDUHVSHFLÀHGLQWKH&RPSDFW0	(3ODQV

7KH0	(3ODQVVSHFLI\LQGLFDWRUVDWDOOOHYHOVSURFHVVRXWSXWDQGRXWFRPHVRWKDWSURJUHVVWRZDUGVÀQDOUHVXOWVFDQEHWUDFNHG(YHU\TXDUWHU

each country partner submits an Indicator Tracking Table (ITT) that shows actual performance of each indicator relative to the baseline level that was 

HVWDEOLVKHGEHIRUHWKHDFWLYLW\EHJDQDQGWKHSHUIRUPDQFHWDUJHWVWKDWZHUHHVWDEOLVKHGLQWKH0	(3ODQ6RPHRIWKHNH\SHUIRUPDQFHLQGLFDWRUV

and their accompanying data by country are publicly available. MCC reviews this data every quarter to assess whether results are being achieved and 

integrates this information into project management decisions.

• 0&&DOVRVXSSRUWVWKHFUHDWLRQRIPXOWLGLVFLSOLQDU\¶compact development teams·WRPDQDJHWKHGHYHORSPHQWDQGLPSOHPHQWDWLRQRIHDFK&RPSDFW

program. Teams usually include the following members: Coordinator, economist, private sector development specialist, social inclusion and gender 

LQWHJUDWLRQVSHFLDOLVWWHFKQLFDOVSHFLDOLVWVSURMHFWVSHFLÀF0	(VSHFLDOLVWHQYLURQPHQWDODQGVRFLDOSHUIRUPDQFHVSHFLDOLVW/HJDODQGÀQDQFLDO

PDQDJHPHQWDQGSURFXUHPHQWVSHFLDOLVWV)URPWKHHDUOLHVWVWDJHVWKHVHWHDPVGHYHORSSURMHFWORJLFVDQG0	(IUDPHZRUNVVXSSRUWHGE\GDWDDQG

evidence, and use them to inform the development of the projects within each Compact program. Teams meet frequently to gather evidence, discuss 

progress, make project design decisions, and solve problems; and they are encouraged to use the lessons from completed evaluations to inform their 

work going forward.

• 0&&KRVWVUHJXODU ́FROOHJHVμLQZKLFK0&&FRXQWHUSDUWVIURPSDUWQHULQJFRXQWULHVDUHLQYLWHGWRDZHHNORQJVHWRIPHHWLQJVDQGZRUNVKRSVWR

discuss best practices, strengthen collaboration, and improve strategies for effectively implementing projects..

86$,':

• 86$,'SDUWQHUVZLWKWKH86'HSDUWPHQWRI6WDWHWRMRLQWO\GHYHORSDQGLPSOHPHQWFOHDUVWUDWHJLFJRDOVDQGREMHFWLYHV86$,'·V3HUIRUPDQFH

,PSURYHPHQW2IÀFHU3,2OHDGV$JHQF\HIIRUWVWRXVHGDWDIRUGHFLVLRQPDNLQJDQGLPSURYHSHUIRUPDQFHDQGRSHUDWLRQDOHIÀFLHQF\DQG

HIIHFWLYHQHVV7KH$VVLVWDQW$GPLQLVWUDWRUIRUWKH0DQDJHPHQW%XUHDX$QJHOLTXH0&UXPEO\DOVRVHUYHVDVWKH3HUIRUPDQFH,PSURYHPHQW2IÀFHU

7KH2IÀFHRI0DQDJHPHQWDQG%XGJHW·VFLUFXODU$ ́3UHSDUDWLRQ6XEPLVVLRQDQG([HFXWLRQRIWKH%XGJHWμ3DUW6L[GHVFULEHVWKHUROHRIWKH3,2

6SHFLÀFDOO\WKH3,2FRRUGLQDWHVWUDFNLQJRI&URVV$JHQF\3ULRULW\&$3DQG$JHQF\3ULRULW\*RDO$3*SURJUHVVOHYHUDJHVVWDWUHYLHZVVXFKDV

3RUWIROLR6WDW+56WDWDQG&\EHU6WDWWRFRQGXFWGHHSGLYHVLQWRHYLGHQFHDQGRYHUVHHVEXVLQHVVSURFHVVUHYLHZVDQGRWKHUDVVHVVPHQWVWRHQVXUH

WKDWWKH$JHQF\PRUHHIÀFLHQWO\DQGHIIHFWLYHO\DFKLHYHVLWVPLVVLRQDQGJRDOV

• 86$,'·VVWUDWHJLFSODQDQQXDOSHUIRUPDQFHSODQDQGUHSRUWDQGRWKHUSHUIRUPDQFHUHSRUWVDUHSXEOLFO\DYDLODEOH

• $JHQF\-RLQW6WUDWHJLF3ODQ-63

• $JHQF\)LQDQFLDO5HSRUW$)5

• $QQXDO3HUIRUPDQFH3ODQDQG5HSRUW$35

• 6XPPDU\RI3HUIRUPDQFHDQG)LQDQFLDO,QIRUPDWLRQ

• 86$,'UHSRUWVRQWKUHH$JHQF\3ULRULW\*RDOVDQGQLQH&URVV$JHQF\3ULRULW\*RDOVRQwww.performance.gov7KHVHJRDOVKHOSWKH$JHQF\LPSURYH

SHUIRUPDQFHDQGFXWFRVWVZKLOHKROGLQJWKH$JHQF\DFFRXQWDEOHWRWKHSXEOLF86$,'DVVHVVHVSURJUHVVDQGFKDOOHQJHVWRZDUGPHHWLQJWKHJRDOV

DQQXDOO\GXULQJGDWDGULYHQUHYLHZVZLWK$JHQF\OHDGHUVKLS86$,'DOVRPHDVXUHVSURJUHVVWRZDUGLWV86$,')RUZDUG reform agenda through eight 

SXEOLFLQGLFDWRUVZKLFKKHOSWKH$JHQF\DGDSWEXVLQHVVSURFHVVHVWRLPSURYHSHUIRUPDQFH

• 86$,'ÀHOGPLVVLRQVGHYHORSCountry Development Cooperation Strategies (CDCS) with clear goals and objectives and a performance management 

planWKDWLGHQWLÀHVH[SHFWHGUHVXOWVSHUIRUPDQFHLQGLFDWRUVWRPHDVXUHWKRVHUHVXOWVSODQVIRUGDWDFROOHFWLRQDQGDQDO\VLVDQGSHULRGLFUHYLHZRI

performance measures to use data and evidence to adapt programs for improved outcomes. 

• ,QDGGLWLRQWRPHDVXULQJSURJUDPSHUIRUPDQFH86$,'PHDVXUHVRSHUDWLRQVSHUIRUPDQFHPDQDJHPHQWWRHQVXUHWKDWWKH$JHQF\DFKLHYHVLWV

development objectives; aligns resources with priorities; and institutionalizes 86$,')RUZDUG reforms.

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

12

The Promise of Evidence-Based Policymaking 1055
EVIDENCE / EVALUATION CRITERIA

4. Performance Management/Continuous 

Improvement: Did the agency implement a 

performance management system with clear and 

prioritized outcome-focused goals and aligned 

program objectives and measures, and did it frequently 

collect, analyze, and use data and evidence to improve 

outcomes, return on investment, and other dimensions 

of performance in FY16?

86(':

• ('GHYHORSVDIRXU\HDUVWUDWHJLFSODQDQGKROGVTXDUWHUO\GDWDGULYHQSURJUHVVUHYLHZVRIWKHJRDOVDQGREMHFWLYHVHVWDEOLVKHGLQWKHSODQDV

UHTXLUHGE\WKH*RYHUQPHQW3HUIRUPDQFHDQG5HVXOWV$FW0RGHUQL]DWLRQ$FW*35$0$RI('·V)<6WUDWHJLF3ODQ includes a goal on 

the continuous improvement of the United States education system with objectives focused on enhancing the use of data, research, evaluation, and 

WHFKQRORJ\VHHSS*350$DOVRUHTXLUHVDJHQFLHVWRGHYHORSDJHQF\SULRULW\JRDOV$3*VDQGVXEPLWLQIRUPDWLRQRQWKRVHJRDOVWR20%

RQDTXDUWHUO\EDVLV$3*VUHÁHFWWKHWRSQHDUWHUPSHUIRUPDQFHSULRULWLHVWKDWDJHQF\OHDGHUVKLSDLPVWRDFFRPSOLVKZLWKLQDWZR\HDUSHULRG('

HVWDEOLVKHGDQ$3*RQHQDEOLQJHYLGHQFHEDVHGGHFLVLRQPDNLQJVHH3HUIRUPDQFHJRYIRUTXDUWHUO\UHSRUWLQJRQWKH$3*VDQGLQ0DUFK

GHFLGHGWRFRQWLQXHLWVZRUNRQWKLV$3*IRU)<2QFHHVWDEOLVKHGWKHPHWULFVIRUWKH$3*VDUHLQFOXGHGLQWKHVWUDWHJLFSODQ)RUH[DPSOH

VWUDWHJLFREMHFWLYHLQWKH'HSDUWPHQW·VFXUUHQWIRXU\HDUVWUDWHJLFSODQZKLFKLVSDUWRIWKHFRQWLQXRXVLPSURYHPHQWJRDOUHIHUHQFHGDERYH

LQFOXGHVWKHPHWULFVIRUWKHHYLGHQFH$3*$OWKRXJKPDQ\RIWKHPHWULFVLQWKHVWUDWHJLFSODQDUHDQQXDOWKH'HSDUWPHQWXVHVWKHTXDUWHUO\UHYLHZV

to discuss data available and milestones achieved.

USHUD:

• +8'FRQGXFWVUHJXODUGDWDGULYHQSHUIRUPDQFHUHYLHZV3 ́+8'6WDWμPHHWLQJV3WKDWIRFXVRQTXDUWHUO\SURJUHVVWRZDUGDFKLHYLQJHDFKRI+8'·V

priority goals. The HUD Secretary and senior leadership from throughout the agency, and sometimes from partner agencies, attend these meetings 

to address challenges, review metrics, improve internal and external collaboration, and increase performance. Strategic goals and two-year priority 

goals are publicly posted. HUD documents alignment between strategic goals and supporting objectives and metrics in the consolidated $QQXDO

3HUIRUPDQFH3ODQ$QQXDO3HUIRUPDQFH5HSRUWDQGLGHQWLÀHVWKHVWDIIDVVLJQHGOHDGUHVSRQVLELOLW\IRUHDFKREMHFWLYH

86'2/:

• '2/·V3HUIRUPDQFH0DQDJHPHQW&HQWHU30&LVUHVSRQVLEOHIRUWKH'HSDUWPHQW·VH[WHQVLYHSHUIRUPDQFHPDQDJHPHQWV\VWHPZKLFKLQFOXGHVRYHU

PHDVXUHVZKRVHUHVXOWVDUHUHYLHZHGTXDUWHUO\E\WKH'HSXW\6HFUHWDU\30&·VDFWLYLWLHVDUHLQWHQGHGWRLPSURYH'2/·VSURJUDPSHUIRUPDQFH

WKURXJKGDWDGULYHQDQDO\VLVVKDULQJEHVWSUDFWLFHVDQGLPSOHPHQWLQJDFWLYLWLHVDVVRFLDWHGZLWKWKH*RYHUQPHQW3HUIRUPDQFHDQG5HVXOWV

0RGHUQL]DWLRQ$FWRI*35$8VLQJD3HUIRUPDQFH6WDWW\SHUHSRUWLQJDQGGDVKERDUGV\VWHP30&FRRUGLQDWHVTXDUWHUO\PHHWLQJVEHWZHHQ

WKH'HSXW\6HFUHWDU\DQGHDFKDJHQF\KHDGWRUHYLHZSHUIRUPDQFHUHVXOWVDQGDQDO\VLVRIWKHSULRULW\SHUIRUPDQFHPHDVXUHVFRQWULEXWLQJWR'2/·V

strategic goals, to make commitments related to performance improvement, and to follow up on the progress of previous performance improvement 

FRPPLWPHQWV30&DOVRRYHUVHHVWKH6WUDWHJLF3ODQQLQJSURFHVV and analyzes performance data in collaboration with agencies to achieve 

FRQWLQXRXVSHUIRUPDQFHLPSURYHPHQW&(2DFWLYHO\SDUWLFLSDWHVLQWKHTXDUWHUO\SHUIRUPDQFHUHYLHZVWRLQFRUSRUDWHÀQGLQJVIURPHYDOXDWLRQVDV

appropriate.

• 2QHRIWKHPRVWLPSRUWDQWUROHVWKDW'2/·V&(2SOD\VLVWRIDFLOLWDWHWKHLQWHUDFWLRQEHWZHHQSURJUDPDQGHYDOXDWLRQDQDO\VWVDQGSHUIRUPDQFH

PDQDJHPHQWDQGHYDOXDWLRQ/HDUQLQJDJHQGDVXSGDWHGDQQXDOO\E\'2/DJHQFLHVLQFROODERUDWLRQZLWK'2/·V&(2LQFOXGHSURJUDPSHUIRUPDQFH

WKHPHVDQGSULRULWLHVIRUDQDO\VLVQHHGHGWRUHÀQHSHUIRUPDQFHPHDVXUHVDQGLGHQWLI\VWUDWHJLHVIRULPSURYLQJSHUIRUPDQFH7KHTXDUWHUO\*35$

PHHWLQJVZLWKWKH'HSXW\6HFUHWDU\URXWLQHO\LQFOXGHVSHFLÀFGLVFXVVLRQVDERXWLPSURYLQJSHUIRUPDQFHDQGÀQGLQJVIURPUHFHQWHYDOXDWLRQVWKDW

suggest opportunities for improvement.

• 7RSURPRWHWKHXVHRIHYLGHQFHEDVHGVWUDWHJLHV'2/·V(PSOR\PHQWDQG7UDLQLQJ$GPLQLVWUDWLRQ(7$DOVRFRQWLQXHVWRPDQDJHWKHWorkforce 

Systems StrategiesZHEVLWHZKLFKLGHQWLÀHVDUDQJHRISRWHQWLDOVWUDWHJLHVLQIRUPHGE\UHVHDUFKHYLGHQFHDQGSHHUH[FKDQJHVWRVXSSRUWJUDQWHHVLQ

providing effective services to customers.

FEDERAL INVEST IN WHAT WORKS INDEX (2016)



1056 The Promise of Evidence-Based Policymaking
EVIDENCE / EVALUATION CRITERIA

5. Data: Did the agency collect, analyze, share, and 

use high-quality administrative and survey data - 

consistent with strong privacy protections - to improve 

(or help other entities improve) federal, state, and local 

programs in FY16?

$&):

• $&)KDVPDGHQXPHURXVDGPLQLVWUDWLYHDQGVXUYH\GDWDVHWVSXEOLFO\DYDLODEOHIRUVHFRQGDU\XVHVXFKDVGDWDIURPWKH1DWLRQDO6XUYH\RI(DUO\

&DUHDQG(GXFDWLRQ, Child Care and Development Fund, 1DWLRQDO6XUYH\RI&KLOGDQG$GROHVFHQW:HOO%HLQJ, and $GRSWLRQDQG)RVWHU&DUH$QDO\VLV

DQG5HSRUWLQJ6\VWHP, among many other examples.

• $&)·VInteroperability Initiative supports data sharing through policies and guidelines to accelerate adoption; standards and tools that are reusable 

DFURVVWKHFRXQWU\ÀHOGEDVHGSLORWVDQGDGGUHVVLQJFRPPRQSULYDF\DQGVHFXULW\UHTXLUHPHQWVWRPLWLJDWHULVNV

• 6HYHUDO$&)GLYLVLRQVKDYHDOVREHHQLQVWUXPHQWDOLQVXSSRUWLQJFURVVJRYHUQPHQWDOHIIRUWVVXFKDVWKH1DWLRQDO,QIRUPDWLRQ([FKDQJH0RGHO

1,(0WKDWZLOOHQDEOHKXPDQVHUYLFHVDJHQFLHVWRFROODERUDWHZLWKKHDOWKHGXFDWLRQMXVWLFHDQGPDQ\RWKHUFRQVWLWXHQFLHVWKDWSOD\DUROHLQWKH

well-being of children and families.

• $&)·V1DWLRQDO'LUHFWRU\RI1HZ+LUHVKDVHQWHUHGLQWRGDWDVKDULQJDJUHHPHQWVZLWKQXPHURXVDJHQFLHV)RUH[DPSOH'2/·V&(2DQG(7$KDYH

LQWHUDJHQF\DJUHHPHQWVZLWK++6$&)IRUVKDULQJDQGPDWFKLQJHDUQLQJVGDWDRQGLIIHUHQWIRUPDOQHWLPSDFWHYDOXDWLRQV7KH1'1+*XLGH for 

'DWD6XEPLVVLRQGHVFULEHVDQDJUHHPHQWZLWKWKH6RFLDO6HFXULW\$GPLQLVWUDWLRQWRXVHLWVQHWZRUNIRUGDWDWUDQVPLVVLRQ$OVR$&)$GPLQLVWHUV

the 3XEOLF$VVLVWDQFH5HSRUWLQJ,QIRUPDWLRQ6\VWHPDSODWIRUPIRUH[FKDQJHRIGDWDRQEHQHÀWVUHFHLSWDFURVV$&)'HSDUWPHQWRI'HIHQVHDQG

9HWHUDQV$IIDLUVSURJUDPV7KLVSODWIRUPHQWDLOVGDWDVKDULQJDJUHHPHQWVEHWZHHQWKHVHWKUHHIHGHUDODJHQFLHVDQGEHWZHHQ$&)DQGVWDWHDJHQFLHV

• 7KH$GPLQLVWUDWLRQ·V)<budget requestLQFOXGHVPLOOLRQRYHUÀYH\HDUVIRUKXPDQVHUYLFHVGDWDLQWHURSHUDELOLW\LQFOXGLQJJUDQWVIRU6WDWHZLGH

Human Services Data Systems and a Systems Innovation Center.

CNCS:

• $VWKHQDWLRQ·VODUJHVWJUDQWPDNHUIRUVHUYLFHDQGYROXQWHHULQJ&1&6FROOHFWVGDWDDERXWVHUYLFHSURJUDPPHPEHUVYROXQWHHUVDQGWKH

organizations in which members and volunteers are placed. Member/volunteer demographic, service experience, and outcome data are collected in a 

variety of ways – both through administrative processes and through surveys: 

• In FY16 data collected from a revised member exit survey allowed CNCS to generate more accurate reports on key experiences and 

anticipated college, career, and civic engagement outcomes, which were shared internally. Survey results are being shared with program 

DQGDJHQF\OHDGHUVKLSLQ)<IRUSURJUDPLPSURYHPHQWSXUSRVHV,Q)<5	(ZLOODOVREHJLQJHQHUDWLQJVWDWHOHYHOUHSRUWVIRULWV6WDWH

&RPPLVVLRQV7KHORQJHUWHUPJRDOLVWRÀQDOL]HUHVSRQVHUDWHVWDQGDUGVDFURVVWKH$PHUL&RUSVSURJUDPVVRWKDWGDWDVHWVFDQEHPDGH

available for public use in FY17.

• $UHSRUWVXPPDUL]LQJFURVVVHFWLRQDOVXUYH\ÀQGLQJVRQ6HQLRU&RUSV)RVWHU*UDQGSDUHQWVDQG6HQLRU&RPSDQLRQ3URJUDPYROXQWHHUV

will be released in FY16. The paper compares health, mobility disability, and life satisfaction between participants in both programs; and 

examines how their health status differs from similar adult volunteers and non-volunteers in the general population (a matched sample of 

YROXQWHHUVDQGQRQYROXQWHHUVIURPWKH+HDOWKDQG5HWLUHPHQW6WXG\+56$ORQJLWXGLQDOVXUYH\RIYROXQWHHUVLQWKHVH6HQLRU&RUSV

SURJUDPVZDVLPSOHPHQWHGLQ)<DQGSUHOLPLQDU\ÀQGLQJVDUHH[SHFWHGLQ)<

• )RUWKHÀUVWWLPHUHVXOWVIURPWKHUHGHVLJQHG$PHUL&RUSVPHPEHUH[LWVXUYH\ZHUHPHUJHGZLWKDGPLQLVWUDWLYHGDWDVHWVRQPHPEHU

GHPRJUDSKLFVSURJUDPFKDUDFWHULVWLFVDQGVHUYLFHORFDWLRQVWRSURGXFHDQHZXQLÀHGGDWDVHWWKDWFXUUHQWO\KDVDOPRVW

REVHUYDWLRQV$QDO\VLVEHJDQLQ)<DQGSUHOLPLQDU\ÀQGLQJVDUHH[SHFWHGE\WKHHQGRI)<

• Findings from an alumni outcome survey pilot were published in FY16.

• ,Q)<&1&6·V5	(2IÀFHLVH[HFXWLQJDQHZDGPLQLVWUDWLYHGDWDPDWFKEHWZHHQDVDPSOHRI$PHUL&RUSVDOXPQLUHFRUGVDQGSRVWVHFRQGDU\

RXWFRPHGDWDIURPWKH1DWLRQDO6WXGHQW&OHDULQJKRXVH5	(DOVRSODQVWRH[HFXWHDVHFRQGDGPLQLVWUDWLYHGDWDPDWFKEHWZHHQDOXPQLUHFRUGVDQG

WKH&HQVXV·/(+'GDWDVHWWRREWDLQHPSOR\PHQWDQGHPSOR\PHQWVHFWRURXWFRPHVIRU$PHUL&RUSVDOXPQL$OWKRXJK5	(FXUUHQWO\UHOLHVRQVXUYH\V

CNCS would prefer to reduce its reliance on this method so that key college and career outcomes can be obtained from more objective sources and 

for less cost.

• &1&6·V2IÀFHRI5HVHDUFKDQG(YDOXDWLRQ5	(PDNHVSXEOLFO\DYDLODEOHVWDWHSURÀOHV that depict national service resources (grant funds, 

members, volunteers, grantees) and program performance metrics across the country and (2) volunteering statistics at the local, state, and national 

levels collected for CNCS by the U.S. Census Bureau through an interagency agreement. ((https://www.volunteeringinamerica.gov/))

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

14

The Promise of Evidence-Based Policymaking 1057
EVIDENCE / EVALUATION CRITERIA

5. Data: Did the agency collect, analyze, share, and 

use high-quality administrative and survey data - 

consistent with strong privacy protections - to improve 

(or help other entities improve) federal, state, and local 

programs in FY16?

MCC:

• 0&&·V0	('LYLVLRQRYHUVHHVWKHXSORDGRIDQRQ\PL]HGHYDOXDWLRQGDWDWR0&&·VSXEOLF(YDOXDWLRQ&DWDORJ. There, partner countries, as well 

as the general public, can access spreadsheets that show economic rates of return calculations, performance indicator tracking tables, results of 

independent evaluationsIRU0&&IXQGHGSURMHFWVDQGSXEOLFXVHYHUVLRQVRIWKHGDWDXVHGLQWKRVHHYDOXDWLRQV$OOHYDOXDWLRQGDWDLVPHWLFXORXVO\

UHYLHZHGE\0&&·VLQWHUQDO'LVFORVXUH5HYLHZ%RDUGSULRUWRSRVWLQJWRHQVXUHWKDWUHVSRQGHQWV·SULYDF\LVSURWHFWHG

• $VSDUWRILWVData2x commitment, MCC and other donors are increasing the amount of gender data released and helping to improve international 

data transparency standards. 

• MCC is also a founding partner of the *RYHUQDQFH'DWD$OOLDQFH, a collaborative effort by governance data producers, consumers, and funders to 

improve the quality, availability, breadth, and use of governance data. 

• 0&&DOVRKDVDSDUWQHUVKLSZLWKWKH3UHVLGHQW·V(PHUJHQF\3ODQIRU$,'65HOLHI3(3)$5ZKLFKLVKHOSLQJWRLQFUHDVHWKHDYDLODELOLW\DQGTXDOLW\RI

GHYHORSPHQWUHODWHGGDWDLQVHOHFWHGFRXQWULHV0&&SDUWQHUHGZLWK3(3)$5WRFUHDWHlocal data hubs that would engage stakeholders around the 

availability, accessibility and analysis of data. The data hubs have a local board drawn from partner country governments, the private sector and civil 

society. The hubs will comprise both a physical space for data analysts and other staff and virtual engagement among such stakeholders as donors, 

IRXQGDWLRQVUHVHDUFKHUVDQG1*2V

• 0&&DOVRKRVWHGDSXEOLFO\DYDLODEOHZHELQDU ́0RQLWRULQJDQG(YDOXDWLRQLQWKH:DWHU6HFWRUμLQZKLFKDSUHVHQWDWLRQZDVJLYHQRQ0&&·VULJRURXV

evidence-based approach to monitoring and evaluation, followed by a closer look at lessons learned in the water sector and a discussion of ways in 

which monitoring and evaluation can contribute to aid effectiveness.

86$,':

• 86$,'KDVDQopen data policy which: 

• (VWDEOLVKHVWKH'HYHORSPHQW'DWD/LEUDU\''/DVWKH$JHQF\·VUHSRVLWRU\RI86$,'IXQGHGPDFKLQHUHDGDEOHGDWDFUHDWHGRUFROOHFWHG

E\WKH$JHQF\DQGLWVLPSOHPHQWLQJSDUWQHUV

• 5HTXLUHV86$,'VWDIIDQGLPSOHPHQWLQJSDUWQHUVYLDDVVRFLDWHGFKDQJHVWRSURFXUHPHQWLQVWUXPHQWVWRVXEPLWGDWDVHWVJHQHUDWHGZLWK

86$,'IXQGLQJWRWKH''/LQPDFKLQHUHDGDEOHQRQSURSULHWDU\IRUPDWV

• ,PSOHPHQWVDGDWDWDJJLQJSURWRFROLQNHHSLQJZLWKWKH3UHVLGHQW·V([HFXWLYH2UGHUDQG2IÀFHRI0DQDJHPHQWDQG%XGJHWSROLF\RQ2SHQ

Data; 

• 'HÀQHVDGDWDFOHDUDQFHSURFHVVWRHQVXUHWKDW86$,'PDNHVDVPXFKGDWDSXEOLFO\DYDLODEOHDVSRVVLEOHZKLOHVWLOODIIRUGLQJDOO

protections for individual privacy, operational and national security, and other considerations allowable by law; and

• (QVXUHVGDWDLVXSGDWHGTXDUWHUO\DWPLQLPXP

• In November 2011, the United States became a signatory to the ,QWHUQDWLRQDO$LG7UDQVSDUHQF\,QLWLDWLYH,$7,,$7,GHYHORSHGDVWDQGDUGIRU

publishing foreign assistance spending data that allows for comparison across publishers. 3XEOLVK:KDW<RX)XQG3:<)D8QLWHG.LQJGRP

EDVHGQRQJRYHUQPHQWDORUJDQL]DWLRQDGYRFDWLQJIRUJUHDWHUDLGWUDQVSDUHQF\DVVHVVHVELODWHUDODQGPXOWLODWHUDOGRQRUV·RYHUDOOFRPPLWPHQW

to aid transparency and the information they publish in an annual $LG7UDQVSDUHQF\,QGH[$7,,Q86$,'UDQNHGVWRXWRIGRQRUVDQG

ZDVDWWKHERWWRPRIWKH ́)DLUμFDWHJRU\,Q-XO\86$,'SURGXFHGDcost management plan&03LQRUGHUWRLPSURYHLWVUHSRUWLQJWR,$7,DQG

WKHUHE\LPSURYHWKH$JHQF\·VVFRUHLQWKH$7,7KHSODQHODERUDWHVRQWKHQHFHVVDU\UHTXLUHPHQWVIRUH[DPSOHSROLWLFDOPRYHPHQWGLVFXVVLRQV

WHFKQLFDOZRUNV\VWHPXSJUDGHVDQGHVWLPDWHGWLPHOLQHIRULPSOHPHQWDWLRQWRDGYDQFHLQWKHVHDUHDV5HFRJQL]LQJWKHOHYHORIHIIRUWLQYROYHGZLWK

WKHLPSURYHPHQWVYDULHVJUHDWO\WKH&03RXWOLQHVDIRXUSKDVHGDSSURDFK86$,'LVDOUHDG\VHHLQJUHVXOWV86$,'·VVFRUHLQ3:<)·V$LG

7UDQVSDUHQF\5HYLHZMXPSHGE\PRUHWKDQSRLQWVSURSHOOLQJ86$,'WRWKH ́*RRGμFDWHJRU\

• 86$,'FRQWLQXHVWRH[SDQGWKHGDWDLWSXEOLVKHVRQ)RUHLJQ$VVLVWDQFHJRY7KH)RUHLJQ$VVLVWDQFH'DVKERDUGDQGWKH,QWHUQDWLRQDO$LG

Transparency Initiative86$,'UHFHQWO\ODXQFKHGWKH)RUHLJQ$LG([SORUHUZKLFKVKDUHV\HDUVRIGDWDWKURXJKDQHDV\WRQDYLJDWHZHEVLWH86$,'

SXEOLVKHVLWVFRUHGDWDVHWVDVZHOODVSURJUDPVSHFLÀFGDWDLQDSSOLFDWLRQSURJUDPLQWHUIDFH$3,IRUPDWV,Q86$,'DOVREHJDQSXEOLFO\

VKDULQJGDWDÀOHVDQGLWVRSHQGDWDSODQWKURXJKLWVQHZ2SHQ*RYHUQPHQWZHEVLWHDVSDUWRIWKH86*RYHUQPHQW·VRSHQGDWDLQLWLDWLYH

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

15

1058 The Promise of Evidence-Based Policymaking
EVIDENCE / EVALUATION CRITERIA

5. Data: Did the agency collect, analyze, share, and 

use high-quality administrative and survey data - 

consistent with strong privacy protections - to improve 

(or help other entities improve) federal, state, and local 

programs in FY16?

86$,' (cont.):

• 7KH86$,'*HR&HQWHUXVHVGDWDDQGDQDO\WLFVWRLPSURYHWKHHIIHFWLYHQHVVRI86$,'·VGHYHORSPHQWSURJUDPVE\JHRJUDSKLFDOO\DVVHVVLQJZKHUH

UHVRXUFHVZLOOPD[LPL]HLPSDFW7KH*HR&HQWHUWHDPZRUNVGLUHFWO\ZLWKÀHOGPLVVLRQVDQG:DVKLQJWRQEDVHGEXUHDXVWRLQWHJUDWHJHRJUDSKLF

DQDO\VLVLQWRWKHVWUDWHJLFSODQQLQJGHVLJQPRQLWRULQJDQGHYDOXDWLRQRI86$,'·VGHYHORSPHQWSURJUDPV7RGDWHWKH*HR&HQWHUKDVOHYHUDJHG

PLOOLRQZRUWKRIKLJKUHVROXWLRQVDWHOOLWHLPDJHU\IRUGHYHORSPHQWSURMHFWVDWQRFRVWWRWKH$JHQF\

• 86$,'·V(FRQRPLF$QDO\VLVDQG'DWD6HUYLFHV($'6XQLWKDVDSXEOLFZHEVLWHWRVKDUHGDWDDQGDOVRSURYLGHVGDWDDQDO\VLVWRROV7KHXQLWDOVR

ZRUNVWRSURYLGHDQDO\VLVXSRQUHTXHVW,QSDUWLFXODUWKH,QWHUQDWLRQDO'DWDDQG(FRQRPLF$QDO\VLVSDUWRI($'6SURYLGHV86$,'VWDIISDUWQHUVDQG

the public with analytical products and a platform for querying data.

• 86$,'XVHVGDWDWRLQIRUPSROLF\IRUPXODWLRQVWUDWHJLFSODQQLQJSURMHFWGHVLJQSURMHFWPDQDJHPHQWDQGDGDSWDWLRQSURJUDPPRQLWRULQJDQG

evaluation, and learning what works. The 3URJUDP&\FOHLV86$,'·VSDUWLFXODUIUDPLQJDQGWHUPLQRORJ\WRGHVFULEHWKLVVHWRISURFHVVHVDQGWKHXVH

of data and evidence to inform decisions is a key part of the process.

• 86$,'·V0RQLWRULQJ&RXQWU\3URJUHVV0&3V\VWHPLVDQHPSLULFDODQDO\WLFDOV\VWHPZKLFKWUDFNVDQGDQDO\]HVFRXQWU\SURJUHVVDORQJÀYH

GLPHQVLRQVHFRQRPLFUHIRUPVJRYHUQLQJMXVWO\DQGGHPRFUDWLFDOO\PDFURHFRQRPLFSHUIRUPDQFHLQYHVWLQJLQSHRSOHDQGSHDFH

DQGVHFXULW\,WLVXVHGWRIDFLOLWDWHFRXQWU\VWUDWHJLFSODQQLQJLQFOXGLQJFRXQWU\JUDGXDWLRQIURP86*IRUHLJQDVVLVWDQFHSURJUDPV

• 86$,'KDVDOVREHJXQSXEOLVKLQJIXQGLQJGDWDDORQJVLGHSURJUDPUHVXOWVRQWKH'ROODUVWR5HVXOWVSDJHRIWKH86$,'ZHEVLWH'ROODUVWR5HVXOWV

SURYLGHVLQIRUPDWLRQRQ86$,'·VLPSDFWDURXQGWKHZRUOGE\OLQNLQJDQQXDOVSHQGLQJLQSXWVWRUHVXOWVRXWSXWVDQGRXWFRPHVLQVRPHRIWKHPRUH

WKDQGHYHORSLQJFRXQWULHVZKHUHZHZRUN7KHUHDUHSODQVWRH[SDQG'ROODUVWR5HVXOWVLQWKHIXWXUH'XHWRWKHQDWXUHRIIRUHLJQDVVLVWDQFH

SURJUDPVLWLVGLIÀFXOWWRGLUHFWO\OLQN)LVFDO<HDUGLVEXUVHPHQWVWR)LVFDO<HDUUHVXOWV7KHUHLVRIWHQDWLPHODJEHWZHHQZKHQDGROODULVGLVEXUVHG

DQGZKHQDUHVXOWLVDFKLHYHGIURPWKDWLQYHVWPHQW)RUH[DPSOHLI86$,'EXLOGVDVFKRROPRVWRIWKHVSHQGLQJWDNHVSODFHLQWKHÀUVWVHYHUDO\HDUV

RIWKHSURMHFWDVFRQVWUXFWLRQEHJLQV+RZHYHUUHVXOWVPD\QRWEHDFKLHYHGXQWLO\HDUVODWHUZKHQWKHVFKRRORSHQVDQGFODVVHVEHJLQ5HVXOWVVKRZQ

RQWKHZHEVLWHJLYHDVQDSVKRWRIWKHW\SHRIUHVXOWVDFKLHYHGE\86$,'

• 7RKHOSLQIRUPWKH86*RYHUQPHQW·VDLGWUDQVSDUHQF\DJHQGD86$,'FRQGXFWHGWKUHHDLGWUDQVSDUHQF\FRXQWU\SLORWVWXGLHVLQZambia (May 2014), 

*KDQD-XQH, and Bangladesh (September 2014). The country pilots assessed the demand for and relevance of information that the U.S. 

*RYHUQPHQWLVPDNLQJDYDLODEOHDVZHOODVWKHFDSDFLW\RIGLIIHUHQWJURXSVWRXVHLW7KHÀQDOUHSRUWVXPPDUL]HVÀQGLQJVIURPWKHWKUHHSLORWVDQG

SURYLGHVUHFRPPHQGDWLRQVWRKHOSLPSURYHWKHWUDQVPLVVLRQRIIRUHLJQDVVLVWDQFHGDWDWRHQVXUHWKDWWKHWUDQVSDUHQF\HIIRUWVRIWKH86*RYHUQPHQW

create development impact.

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

16

The Promise of Evidence-Based Policymaking 1059
EVIDENCE / EVALUATION CRITERIA

5. Data: Did the agency collect, analyze, share, and 

use high-quality administrative and survey data - 

consistent with strong privacy protections - to improve 

(or help other entities improve) federal, state, and local 

programs in FY16?

86(':

• ('KDVVHYHUDOUHVRXUFHVWRVXSSRUWWKHKLJKTXDOLW\FROOHFWLRQDQDO\VLVDQGXVHRIKLJKTXDOLW\GDWDLQZD\VWKDWSURWHFWSULYDF\,(6·National 

&HQWHUIRU(GXFDWLRQ6WDWLVWLFV1&(6VHUYHVDVWKHSULPDU\IHGHUDOHQWLW\IRUFROOHFWLQJDQGDQDO\]LQJGDWDUHODWHGWRHGXFDWLRQ$OPRVWDOORI

('·V.VWDWLVWLFDODQGSURJUDPPDWLFGDWDFROOHFWLRQVDUHQRZDGPLQLVWHUHGE\1&(6YLD(')DFWV1&(6DOVRFROOHFWVGDWDWKURXJKQDWLRQDODQG

international surveys and assessments$GPLQLVWUDWLYHLQVWLWXWLRQDOGDWDDQGVWDWLVWLFDOVDPSOHVXUYH\GDWDIRUSRVWVHFRQGDU\HGXFDWLRQLVFROOHFWHG

WKURXJK1&(6LQFROODERUDWLRQZLWKWKH)HGHUDO6WXGHQW$LG2IÀFH)6$1&(6GDWDDUHPDGHSXEOLFO\DYDLODEOHonline and can be located in the 

('Data Inventory. Some data are available through public access while others only through restricted data licenses('·V2IÀFHIRU&LYLO5LJKWV

conducts the &LYLO5LJKWV'DWD&ROOHFWLRQ&5'&RQNH\HGXFDWLRQDQGFLYLOULJKWVLVVXHVLQRXUQDWLRQ·VSXEOLFVFKRROV$GGLWLRQDOO\WKH'DWD6WUDWHJ\

Team helps to coordinate data activities across the Department and the 'LVFORVXUH5HYLHZ%RDUG, the )DPLO\3ROLF\&RPSOLDQFH2IÀFH)3&2WKH

(')DFWV*RYHUQLQJ%RDUGDQGWKH3ULYDF\7HFKQLFDO$VVLVWDQFH&HQWHU all help to ensure the quality and privacy of education data.

• ('KDVPDGHFRQFHUWHGHIIRUWVWRLPSURYHWKHDYDLODELOLW\DQGXVHRILWVGDWDLQ)<:LWKWKHUHOHDVHRIWKHQHZCollege Scorecard, the Department 

now provides newly combined data in a tool that helps students choose a school that is well-suited to meet their needs, priced affordably, and 

FRQVLVWHQWZLWKWKHLUHGXFDWLRQDODQGFDUHHUJRDOV$GGLWLRQDOO\WKH&ROOHJH6FRUHFDUGSURPRWHVWKHXVHRIRSHQGDWDE\SURYLGLQJWKHXQGHUO\LQJGDWD

LQIRUPDWVWKDWUHVHDUFKHUVDQGGHYHORSHUVFDQXVH7KLVHIIRUWLVDPRGHOIRUIXWXUHUHOHDVHVRIHGXFDWLRQGDWDDQGOHGWR('·VQHZHIIRUW,QIRUP('

WRLPSURYH'HSDUWPHQWFDSDFLW\WRUHOHDVHGDWDLQLQQRYDWLYHDQGHIIHFWLYHZD\VWRLPSURYHSXEOLFXVHRIGDWD,QIRUP('ZDVSDUWRIWKHFY17 budget

request (see p. 78). 

• ('KDVVHYHUDOGDWDVKDULQJDJUHHPHQWVZLWKRWKHUDJHQFLHV)RUH[DPSOH('DQGWKH86'HSDUWPHQWRI7UHDVXU\PDWFK)HGHUDO6WXGHQW$LG

GDWDZLWKDGPLQLVWUDWLYH,QWHUQDO5HYLHZ6HUYLFHWD[UHFRUGVWRFDOFXODWHHDUQLQJVLQIRUPDWLRQE\SRVWVHFRQGDU\LQVWLWXWLRQIRUWKHCollege Scorecard

FRQVXPHUWRRO7KLVDJUHHPHQWDOORZV('WRDQQXDOO\XSGDWHDQGSXEOLVKGDWDRQPHDQHDUQLQJVPHGLDQHDUQLQJVDQGIUDFWLRQQRWZRUNLQJDPRQJ

DOOVWXGHQWVZKRUHFHLYHG7LWOH,9DLGLHIHGHUDOJUDQWVDQGORDQV('DQGWKH86'HSDUWPHQWRI/DERUDUHHQJDJHGLQDMRLQWIHGHUDOVWDWH

workgroup that is developing help for data sharing at the state level through the new State Wage Interchange System (SWIS) for the Workforce 

,QQRYDWLRQDQG2SSRUWXQLW\$FW:,2$)RUFDOFXODWLQJWKH*DLQIXO(PSOR\PHQW*(GHEWWRHDUQLQJVPHWULFWKH'HSDUWPHQWRI(GXFDWLRQREWDLQV

IURPWKH6RFLDO6HFXULW\$GPLQLVWUDWLRQ66$DQQXDOHDUQLQJVRIVWXGHQWVZKRFRPSOHWHGD*(SURJUDP('VXEPLWVWKH6RFLDO6HFXULW\QXPEHUV

RIVWXGHQWVZKRUHFHLYHG7LWOH,9DLGLHIHGHUDOJUDQWVDQGORDQVWR66$LQRUGHUWRFDOFXODWHWKHKLJKHVWRIPHDQDQGPHGLDQHDUQLQJVIRUHDFK

program. 

• $GGLWLRQDOO\('DGPLQLVWHUVWKH6WDWHZLGH/RQJLWXGLQDO'DWD6\VWHP6/'6SURJUDPPLOOLRQLQ)<ZKLFKSURYLGHVJUDQWVWRVWDWHVWR

develop their education-related data infrastructure and use these data for education improvement..

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

17

1060 The Promise of Evidence-Based Policymaking
EVIDENCE / EVALUATION CRITERIA

5. Data: Did the agency collect, analyze, share, and 

use high-quality administrative and survey data - 

consistent with strong privacy protections - to improve 

(or help other entities improve) federal, state, and local 

programs in FY16?

USHUD:

• 7KH+8'86(5ZHESRUWDOFRQWLQXHVWRSURYLGHUHVHDUFKHUVSUDFWLWLRQHUVDQGWKHSXEOLFZLWK3'	5GDWDVHWVLQFOXGLQJWKH$PHULFDQ+RXVLQJ

6XUYH\+8'PHGLDQIDPLO\LQFRPHOLPLWVDQG3LFWXUHRI6XEVLGL]HG+RXVHKROGVWDEXODWLRQVDWPXOWLSOHJHRJUDSKLFOHYHOVDVZHOODVPLFURGDWD

from research initiatives on topics such as housing discrimination, the HUD-insured multifamily housing stock, and the public housing population. To 

help users identify which data are useful to them, reference guides identify datasets and characterize their relevance and usefulness for research in 

designated categories.

• +8'·V2IÀFHRI3ROLF\'HYHORSPHQWDQG5HVHDUFK3'	5KDVDXWKRULW\WRHQWHULQWRFRRSHUDWLYHDJUHHPHQWVZLWKUHVHDUFKRUJDQL]DWLRQVLQFOXGLQJ

both funded 5HVHDUFK3DUWQHUVKLSV and unfunded 'DWD/LFHQVH$JUHHPHQWVWRVXSSRUWLQQRYDWLYHUHVHDUFKSURMHFWVWKDWOHYHUDJH+8'·VGDWDDVVHWV

DQGLQIRUP+8'·VSROLFLHVDQGSURJUDPV$GHGLFDWHGVXEMHFWPDWWHUH[SHUWLVDYDLODEOHWRDQVZHUTXHVWLRQVIRUWKRVHVHHNLQJDGDWDOLFHQVH

• +8'·V3'	5DQGWKH1DWLRQDO&HQWHUIRU+HDOWK6WDWLVWLFVDWWKH&HQWHUVIRU'LVHDVH&RQWUROKDYHVXFFHVVIXOO\OLQNHG+8'DGPLQLVWUDWLYHGDWDIRU

assisted renters with respondents to two national health surveys and made the linked data available to researchers to begin building a picture of 

tenant health issues.

• +8'LVLQYROYHGLQDZLGHDUUD\RIGDWDVKDULQJDJUHHPHQWVLQFOXGLQJJHRFRGLQJVHUYLFHVSURYLGHGE\+8'·V*HRFRGLQJ6HUYLFH&HQWHUDUHFHQW

agreement with the U.S. Department of Health and Human Services to pilot an administrative data linkage with Medicare and Medicaid utilization 

records; national compilation of local point-in-time counts of homeless individuals and administrative data from homeless service providers using 

+RPHOHVV0DQDJHPHQW,QIRUPDWLRQ6\VWHPVFROOHFWLRQRIWHQDQWGDWDIRU/RZ,QFRPH+RXVLQJ7D[&UHGLWSURSHUWLHVIURPVWDWHKRXVLQJÀQDQFH

DJHQFLHVDQRQJRLQJDJUHHPHQWZLWK$FWLRQDEOH,QWHOOLJHQFHIRU6RFLDO3ROLF\WRGHYHORSLQWHJUDWHGGDWDV\VWHPVIRUSROLF\DQDO\VLVDQGSURJUDP

reform, including local education data; and a multiagency federal agreement under development about protocols for information security in datasharing.

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

18

The Promise of Evidence-Based Policymaking 1061
EVIDENCE / EVALUATION CRITERIA

5. Data: Did the agency collect, analyze, share, and 

use high-quality administrative and survey data - 

consistent with strong privacy protections - to improve 

(or help other entities improve) federal, state, and local 

programs in FY16?

86'2/:

• '2/·V%XUHDXRI/DERU6WDWLVWLFV%/6DSSUR[LPDWHO\PLOOLRQLQ)<VHUYHVDVWKHSULQFLSDO)HGHUDODJHQF\UHVSRQVLEOHIRUPHDVXULQJODERU

PDUNHWDFWLYLW\ZRUNLQJFRQGLWLRQVDQGSULFHFKDQJHVLQWKHHFRQRP\%/6KDV&RRSHUDWLYH$JUHHPHQWV with 50 States and 4 Territories for 

ODERUPDUNHWDQGHFRQRPLFGDWDVKDULQJ ́OHWWHUVRIDJUHHPHQWμRQGDWDXVDJHZLWKDFDGHPLFVWRFRQGXFWVWDWLVWLFDOUHVHDUFKDQGdata sharing 

agreementsZLWKWKH%XUHDXRI(FRQRPLF$QDO\VLVDQGWKH&HQVXV%XUHDX

• '2/·V(PSOR\PHQWDQG7UDLQLQJ$GPLQLVWUDWLRQ(7$KDVagreements with 52 States and Territories for data sharing and exchange of wage data for 

performance accountability purposes. 

• '2/·V&(2(PSOR\PHQW7UDLQLQJ$GPLQLVWUDWLRQ(7$DQGWKH9HWHUDQV(PSOR\PHQWDQG7UDLQLQJ6HUYLFH9(76KDYHZRUNHGZLWKWKH86

Department of Health and Human Services (HHS) to develop a secure mechanism for obtaining and analyzing earnings data from the Directory of 

1HZ+LUHV,QWKLVSDVW\HDU'2/KDVHQWHUHGLQWRLQWHUDJHQF\GDWDVKDULQJDJUHHPHQWVZLWK++6DQGREWDLQHGGDWDWRVXSSRUWMREWUDLQLQJDQG

HPSOR\PHQWSURJUDPHYDOXDWLRQV5HHPSOR\PHQW$VVLVWDQFH'HPRQVWUDWLRQ(YDOXDWLRQZLWKXQHPSOR\PHQWLQVXUDQFHFODLPDQWV<RXQJ3DUHQWV

'HPRQVWUDWLRQ(YDOXDWLRQ(QKDQFHG7UDQVLWLRQDO-REV3URJUDP(YDOXDWLRQ<RXWKEXLOG(YDOXDWLRQ:RUNIRUFH,QYHVWPHQW$FW(YDOXDWLRQ*UHHQ

-REV+HDOWK&DUH'HPRQVWUDWLRQ(YDOXDWLRQ5HHQWU\IRU([2IIHQGHUV(YDOXDWLRQ7UDQVLWLRQ$VVLVWDQFH3URJUDPIRUVHSDUDWLQJDFWLYLW\GXW\PLOLWDU\

SHUVRQVDQGWKH-RE7UDLQLQJ6FRUHFDUG)HDVLELOLW\6WXG\

• '2/·VZRUNHUSURWHFWLRQDJHQFLHVKDYHRSHQGDWDSURYLVLRQVRQHQIRUFHPHQWDFWLYLW\IRUÀUPVIURP'2/·VÀYHODERUHQIRUFHPHQWDJHQFLHVRQOLQHDQG

accessible through the (QIRUFHPHQW'DWD%DVH0LQH6DIHW\DQG+HDOWK$GPLQLVWUDWLRQ:DJHDQG+RXU'LYLVLRQ2FFXSDWLRQDO6DIHW\DQG+HDOWK

$GPLQLVWUDWLRQDQGWKH(PSOR\HH%HQHÀWV6HFXULW\$GPLQLVWUDWLRQ

• 7KHSULYDF\SURYLVLRQVIRU%/6DQG'2/·V(PSOR\PHQWDQG7UDLQLQJ$GPLQLVWUDWLRQ(7$DUHSXEOLFO\DYDLODEOHonline. 

• ,Q)<'2/H[SDQGHGHIIRUWVWRLPSURYHWKHTXDOLW\RIDQGDFFHVVWRGDWDIRUHYDOXDWLRQDQGSHUIRUPDQFHDQDO\VLVWKURXJKWKH'DWD$QDO\WLFV

8QLWLQ'2/·V&(2RIÀFHDQGWKURXJKQHZSLORWVEHJLQQLQJLQ%/6WRDFFHVVDQGH[FKDQJHVWDWHODERUPDUNHWDQGHDUQLQJVGDWDIRUVWDWLVWLFDODQG

HYDOXDWLRQSXUSRVHV7KH'DWD$QDO\WLFVXQLWKDVDOVRGHYHORSHGWKH'DWD([FKDQJHDQG$QDO\VLV3ODWIRUP'($3ZLWKKLJKSURFHVVLQJFDSDFLW\DQG

SULYDF\SURYLVLRQVWRVKDUHOLQNDQGDQDO\]HSURJUDPDQGVXUYH\GDWDDFURVV'2/SURJUDPVDQGDJHQFLHVDQGZLWKRWKHUDJHQFLHV,QWHUQDOXVHRI

'($3LVDYDLODEOHQRZDQGSXEOLFDFFHVVZLOOEHDYDLODEOHLQWKHIXWXUH

• 7KH:RUNIRUFH,QQRYDWLRQ2SSRUWXQLW\$FW:,2$FDOOVIRUDOLJQHGLQGLFDWRUVRISHUIRUPDQFHIRU:,2$DXWKRUL]HGSURJUDPV'2/·V(PSOR\PHQWDQG

7UDLQLQJ$GPLQLVWUDWLRQKDVZRUNHGZLWKLQ'2/DQGZLWKWKH86'HSDUWPHQWRI(GXFDWLRQWRSXUVXHWKHGHHSHVW:,2$DOLJQPHQWSRVVLEOHLQFOXGLQJ

LQGLFDWRUVGHÀQLWLRQVGDWDHOHPHQWVDQGVSHFLÀFDWLRQVWRLPSURYHWKHTXDOLW\DQGDQDO\WLFYDOXHRIWKHGDWD'2/FKRVHWRLQFOXGHVHYHUDODGGLWLRQDO

'2/SURJUDPVLQWKLVSURFHVVZKLFKZLOOUHVXOWLQXQSUHFHGHQWHGDOLJQPHQWRIGDWDDQGGHÀQLWLRQVIRUIHGHUDOSURJUDPV'2/DQG(GXFDWLRQ

'2/DQG('KDYHLVVXHGWKHproposed ruleIRUSXEOLFFRPPHQWDQGZLOOÀQDOL]HLWLQODWHVSULQJDQGKDVDOVRLVVXHGWKHUHODWHGInformation 

&ROOHFWLRQ5HTXHVWVIRUSXEOLFFRPPHQWDQGH[SHFWWRÀQDOL]HWKRVH,QIRUPDWLRQ&ROOHFWLRQUHTXLUHVSULRUWRWKDWGDWH

• (7$FRQWLQXHVIXQGLQJDQGWHFKQLFDODVVLVWDQFHWRVWDWHVXQGHUWKHWorkforce Data Quality Initiative to link earnings and workforce data and 

HGXFDWLRQGDWDORQJLWXGLQDOO\(7$DQG'2/·V9HWHUDQ·V(PSOR\PHQWDQG7UDLQLQJ6HUYLFHKDYHDOVRPRGLÀHGVWDWHZRUNIRUFHSURJUDPUHSRUWLQJ

system requirements to include data items for a larger set of grant programs, which will improve access to administrative data for evaluation and 

SHUIRUPDQFHPDQDJHPHQWSXUSRVHV$QH[DPSOHRIWKHH[SDQGHGGDWDUHSRUWLQJUHTXLUHPHQWVLVWKH+RPHOHVV9HWHUDQV5HLQWHJUDWLRQ3URJUDP

FY16 grants.

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

19

1062 The Promise of Evidence-Based Policymaking
EVIDENCE / EVALUATION CRITERIA

6. Common Evidence Standards/What Works 

Designations: Did the agency use a common 

evidence framework, guidelines, or standards to 

inform its research and funding decisions and did it 

disseminate and promote the use of evidence-based 

interventions through a user-friendly tool in FY16?

$&):

• $&)KDVHVWDEOLVKHGDcommon evidence framework adapted for the human services context from the framework for education research developed 

E\WKH86'HSDUWPHQWRI(GXFDWLRQDQGWKH1DWLRQDO6FLHQFH)RXQGDWLRQ7KH$&)IUDPHZRUNZKLFKLQFOXGHVWKHVL[W\SHVRIVWXGLHVGHOLQHDWHG

LQWKH('16)IUDPHZRUNDLPVWRLQIRUP$&)·VLQYHVWPHQWVLQUHVHDUFKDQGHYDOXDWLRQDQGFODULI\IRUSRWHQWLDOJUDQWHHVDQGRWKHUVWKH

expectations for different types of studies.

• $&)PDLQWDLQVDQRQOLQHclearinghouse of evidence reviews of human services interventions. These reviews rate the quality of evaluation studies 

using objective standards vetted by technical experts and applied by trained, independent reviewers, and similar to those used by other agencies such 

DVWKH86'HSDUWPHQWRI(GXFDWLRQ·V:KDW:RUNV&OHDULQJKRXVHDQGWKH86'HSDUWPHQWRI/DERU·V&/($57KHFOHDULQJKRXVHLQFOXGHVUHVXOWV

RIWKHUHYLHZVLQDVHDUFKDEOHIRUPDWDVZHOODVFRPSUHKHQVLYHGHWDLOVDERXWWKHUHYLHZVWDQGDUGVDQGSURFHVV5HYLHZVWRGDWHKDYHFRYHUHG

teen pregnancy prevention; home visiting; relationship education and responsible fatherhood; and employment and trainingDQGLQFOXGHERWK$&)

sponsored and other studies.

CNCS:

• &1&6·V2IÀFHRI5HVHDUFKDQG(YDOXDWLRQ5	(2IÀFHLVDFWLYHO\LQYROYHGZLWKRWKHUIHGHUDODJHQFLHVLQWKHLQWHUDJHQF\&RPPRQ(YLGHQFH

)UDPHZRUNZRUNLQJJURXSLQRUGHUWRHQVXUHFRQVLVWHQF\LQGHÀQLWLRQVDQGXVHRIHYLGHQFHVWDQGDUGVLQJUDQWPDNLQJ&1&6XVHVWKHCross-agency 

)HGHUDO(YLGHQFH)UDPHZRUN for evaluation planning and dissemination. 

• &1&6DOVRDGDSWHGWKHHYLGHQFHIUDPHZRUNXVHGE\LWV6RFLDO,QQRYDWLRQ)XQGDQGWKH,QYHVWLQJLQ,QQRYDWLRQ)XQGDW('DQGLQFOXGHGLWDVSDUW

RIWKH$PHUL&RUSV6WDWHDQG1DWLRQDOSURJUDP·VFY16 grant competition7KHHYLGHQFHIUDPHZRUNXVHGLQWKH)<$PHUL&RUSVFRPSHWLWLRQZDV

revised from FY15 to make it more consistent with what is used in other federal agencies.

• ,Q0DUFK&1&6UHOHDVHG3KDVH,RIWKH&1&6(YLGHQFH([FKDQJH, a virtual repository of reports intended to help CNCS grantees and other 

LQWHUHVWHGVWDNHKROGHUVÀQGLQIRUPDWLRQDERXWHYLGHQFHDQGUHVHDUFKEDVHGQDWLRQDOVHUYLFHDQGVRFLDOLQQRYDWLRQSURJUDPV3KDVHLQFOXGHVD

database of single study reports with some additional descriptive information about the study, as well as a systematic review of the national service 

HYLGHQFHEDVH3KDVHLQ)<DGGHGVWXGLHVDVJUDQWHHVFRPSOHWHGWKHLULQGHSHQGHQWHYDOXDWLRQVDQGVXEPLWWHGUHSRUWVWR&1&6

MCC:

• MCC uses common, rigorous, evidence-based selection criteria to ensure objectivity in country selection for grant awards. To be eligible for selection, 

FRXQWULHVPXVWÀUVWSDVVWKH0&&scorecard2DFROOHFWLRQRILQGHSHQGHQWWKLUGSDUW\GHYHORSHGLQGLFDWRUVWKDWREMHFWLYHO\PHDVXUHDFRXQWU\·V

policy performance in the areas of economic freedom, investing in its people, and ruling justly. The criteria for passing the scorecard are applied 

XQLYHUVDOO\WRDOOFDQGLGDWHFRXQWULHV0&&·V%RDUGRI'LUHFWRUVWKHQFRQVLGHUVNH\IDFWRUVIRUVHOHFWLQJFRXQWULHVDFRXQWU\·VSHUIRUPDQFHRQ

WKHVFRUHFDUGWKHRSSRUWXQLW\WRUHGXFHSRYHUW\DQGJHQHUDWHHFRQRPLFJURZWKDQGDYDLODELOLW\RIIXQGV$QLQGHSWKGHVFULSWLRQRIWKHFRXQWU\

selection procedure can be found in the annual Selection Criteria and Methodology report.

• 0&&·VPRGHOLVEDVHGRQDVHWRIcore principles essential for development to take place and for development assistance to be effective – good 

governance, country ownership, focus on results, and transparency. In pursuing these, MCC has created a 3ULQFLSOHVLQWR3UDFWLFH series which 

GHVFULEHVKRZWRPDNHWKHVHSULQFLSOHVRSHUDWLRQDO)LQDOO\DOORI0&&·VHYDOXDWLRQVDUHWKHQSXEOLVKHGRQWKH0&&(YDOXDWLRQ&DWDORJ$VVRFLDWHG

GDWDXSRQZKLFKHYDOXDWLRQVDUHEDVHGDUHSXEOLVKHGZKHQFRQÀGHQWLDOLW\FRQFHUQVDUHDGHTXDWHO\DGGUHVVHG

• MCC is also developing an enhanced consolidated results framework that will assist it in telling the full picture of the impact of its programs and enrich 

programmatic learning. Currently in draft form, the framework will help MCC consolidate impacts across projects, compacts and sectors to assess an 

overall impact at an organizational level..

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

20

The Promise of Evidence-Based Policymaking 1063
EVIDENCE / EVALUATION CRITERIA

6. Common Evidence Standards/What Works 

Designations: Did the agency use a common 

evidence framework, guidelines, or standards to 

inform its research and funding decisions and did it 

disseminate and promote the use of evidence-based 

interventions through a user-friendly tool in FY16?

86$,':

• 86$,'KDVDVFLHQWLÀFresearch policyWKDWVHWVRXWTXDOLW\VWDQGDUGVIRUUHVHDUFK86$,'·V3URJUDP&\FOHJXLGDQFHLQFOXGHVVSHFLÀFHYLGHQFH

standards for strategic planning, project design, monitoring, and evaluation)RUH[DPSOH86$,'KDVguidance that requires evidence and data to 

DVVHVVWKHGHYHORSPHQWFRQWH[WFKDOOHQJHVDQGRSSRUWXQLWLHVLQDOORI86$,'·VFRXQWU\VWUDWHJLHV6LPLODUO\DOO86$,'SURMHFWVPXVWLQFOXGHD

detailed analytical phaseLQWKH3URMHFW$SSUDLVDO'RFXPHQW

• 86$,'GRHVPRVWRILWV$JHQF\ZLGHHQJDJHPHQWDURXQGHYLGHQFHDQGIUDPHZRUNVIRU ́ZKDWZRUNVμWKURXJKLWVERDUGPHPEHUVKLSDQGIXQGLQJ

(along with other donors) of the ,QWHUQDWLRQDO,QLWLDWLYHIRU,PSDFW(YDOXDWLRQVLHZKLFKIXQGVLPSDFWHYDOXDWLRQVDQGV\VWHPDWLFUHYLHZVWKDW

JHQHUDWHHYLGHQFHRQZKDWZRUNVLQGHYHORSPHQWSURJUDPVDQGZK\5DWKHUWKDQFUHDWLQJDVHSDUDWH ́ZKDWZRUNVμFOHDULQJKRXVH86$,'KDV

FKRVHQWRZRUNZLWKLHDQGRWKHUGHYHORSPHQWSDUWQHUVWRVXSSRUWLH·Vdatabase of impact evaluations relevant to development topics (includes 

over 2,500 entries to date), knowledge gap maps and systematic reviewsWKDWSXOOWKHPRVWULJRURXVHYLGHQFHDQGGDWDIURPDFURVVGRQRUVLHDOVR

houses a collection of policy briefsWKDWH[DPLQHÀQGLQJVIURPLWVdatabase of impact evaluations on overarching policy questions to help policymakers 

and development practitioners improve development impact through better evidence. 

• 86$,'WHFKQLFDOEXUHDXVSURYLGHJXLGDQFHEDVHGRQHYLGHQFHRI ́ZKDWZRUNVμE\VHFWRUWKDWDSSOLHVWRDOOUHOHYDQW$JHQF\SURJUDPV86$,'·V

%XUHDXIRU'HPRFUDF\&RQÁLFWDQG+XPDQLWDULDQ$VVLVWDQFH'&+$IRUH[DPSOHLQFOXGHVWKH&HQWHURI([FHOOHQFHRQ'HPRFUDF\5LJKWV

DQG*RYHUQDQFHZKLFKSXEOLVKHVHYLGHQFHEDVHGVWDQGDUGVIRUZKDWZRUNVLQWKLVÀHOG7KH'5*&HQWHUFRQYHQHVOHDGLQJVFKRODUVIURPD

UDQJHRIÀHOGVWRZRUNZLWK86$,'WRVWXG\DQDO\]HDQGDVVHVVWKHHIIHFWLYHQHVVRILWVLQLWLDWLYHVDQGSURJUDPVLQ'5*XVLQJWKLVGDWDWRVKDSH

SURJUDPPLQJ,QDGGLWLRQ86$,'HVWDEOLVKHGWKH(YDOXDWLQJ'HPRFUDF\DQG*RYHUQDQFH(IIHFWLYHQHVV('*(,QLWLDWLYHZLWKWKHREMHFWLYHWRVXSSO\

and apply sophisticated tools to measure the impact of democracy, human rights, and governance work, and infuse evidence-based programmatic 

GHFLVLRQPDNLQJWKURXJKRXWWKH'5*SRUWIROLR,QDQRWKHUH[DPSOH86$,'·V*OREDO+HDOWK%XUHDXKDVDstrategic framework that presents details in 

$QQH[RQVSHFLÀFHYLGHQFHEDVHGVWUDWHJLHVWDUJHWVDQGDSSURDFKHVIRUDFKLHYLQJJRDOVZLWKLQHDFKWHFKQLFDODUHDXQGHUWKHKHDOWKSULRULWLHV

• 6HYHUDO86$,'%XUHDXVDOVRV\QWKHVL]HDOOWKHHYDOXDWLRQVUHOHYDQWWRDVSHFLÀFVHFWRUWRVXPPDUL]HNH\ÀQGLQJVDQGLGHQWLI\JDSVLQNQRZOHGJH

that then inform sector learning agendas. For example, in March, the Bureau for Food Security (BFS) published a synthesis report summarizing 

ÀQGLQJVIURPHYDOXDWLRQVRI)HHGWKH)XWXUHSURMHFWVWKDWIRFXVHGRQWKHVL[WKHPHVRXWOLQHGLQWKH%)6/HDUQLQJ$JHQGD$FURVVWKHWKHPHV

the synthesis illuminated trends and patterns summarized in the points found below the graphic. These trends can be shared with relevant staff and 

VWDNHKROGHUVHQJDJHGLQGHVLJQLQJQHZSURMHFWVRUXSGDWLQJVHFWRUVWUDWHJLHVDQGSROLFLHV7KHV\QWKHVLVDOVRLGHQWLÀHGJDSVZKHUHPRUHHYDOXDWLRQ

research is needed, helping to inform the design of future evaluations that can contribute to the body of knowledge on food security to improve the 

GHVLJQDQGPDQDJHPHQWRILQWHUYHQWLRQVLQWKHDJULFXOWXUHDQGQXWULWLRQVHFWRUVE\VSHFLÀFDOO\DGGUHVVLQJ/HDUQLQJ$JHQGDTXHVWLRQV

86(':

• ('·VHYLGHQFHVWDQGDUGVIRULWVJUDQWSURJUDPVDVRXWOLQHGLQWKH(GXFDWLRQ'HSDUWPHQW*HQHUDO$GPLQLVWUDWLYH5HJXODWLRQV('*$5), build on 

('·VWhat Works ClearinghouseTM::&HYLGHQFHVWDQGDUGV('RIWHQLQFOXGHVWKHVHHYLGHQFHVWDQGDUGVLQLWVGLVFUHWLRQDU\JUDQWFRPSHWLWLRQVWR

direct funds to applicants proposing to implement projects that have evidence of effectiveness and/or to build new evidence through evaluation (see 

4XHVWLRQEHORZIRUPRUHGHWDLO$GGLWLRQDOO\,(6DQGWKH1DWLRQDO6FLHQFH)RXQGDWLRQLVVXHGDMRLQWUHSRUWWKDWGHVFULEHVVL[W\SHVRIUHVHDUFK

VWXGLHVWKDWFDQJHQHUDWHHYLGHQFHDERXWKRZWRLQFUHDVHVWXGHQWOHDUQLQJLQ7KHVHSULQFLSOHVDUHEDVHGLQSDUWRQWKHUHVHDUFKJRDOVWUXFWXUH

DQGH[SHFWDWLRQVRI,(6·V1DWLRQDO&HQWHUIRU(GXFDWLRQ5HVHDUFK1&(5DQG1DWLRQDO&HQWHUIRU6SHFLDO(GXFDWLRQ5HVHDUFK1&6(51&(5DQG

1&6(5FRPPXQLFDWHWKHVHH[SHFWDWLRQVWKURXJKWKHLU5HTXHVWVIRU$SSOLFDWLRQV and webinarsWKDWDUHDUFKLYHGRQWKH,(6ZHEVLWHDQGDYDLODEOHWR

all applicants.

• ('·VWhat Works ClearinghouseTM ::&LGHQWLÀHVVWXGLHVWKDWSURYLGHFUHGLEOHDQGUHOLDEOHHYLGHQFHRIWKHHIIHFWLYHQHVVRIDJLYHQSUDFWLFH

SURJUDPRUSROLF\UHIHUUHGWRDV ́LQWHUYHQWLRQVμDQGGLVVHPLQDWHVVXPPDU\LQIRUPDWLRQDQGUHSRUWVRQWKH::&ZHEVLWH7KH::&KDVUHYLHZHG

PRUHWKDQVWXGLHVWKDWDUHDYDLODEOHLQDsearchable database.

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

21

1064 The Promise of Evidence-Based Policymaking
EVIDENCE / EVALUATION CRITERIA

6. Common Evidence Standards/What Works 

Designations: Did the agency use a common 

evidence framework, guidelines, or standards to 

inform its research and funding decisions and did it 

disseminate and promote the use of evidence-based 

interventions through a user-friendly tool in FY16?

USHUD:

• +8'·V3ROLF\'HYHORSPHQWDQG5HVHDUFK3'	5RIÀFHSURYLGHVHYLGHQFHRI ́ZKDWZRUNVμSULPDULO\WKURXJK+8'86(5, a portal and web store 

IRUSURJUDPHYDOXDWLRQVFDVHVWXGLHVDQGSROLF\DQDO\VLVDQGUHVHDUFKWKH5HJXODWRU\%DUULHUV&OHDULQJKRXVHDQGWKURXJKLQLWLDWLYHVVXFKDV

,QQRYDWLRQRIWKH'D\6XVWDLQDEOH&RQVWUXFWLRQ0HWKRGVLQ,QGLDQ&RXQWU\DQGWKH&RQVXPHU·V*XLGHWR(QHUJ\(IÀFLHQWDQG+HDOWK\+RPHV7KLV

content is designed to provide current policy information, elevate effective practices, and synthesize data and other evidence in accessible formats. 

Through these resources, researchers and practitioners can see the full breadth of work on a given topic (e.g., rigorous established evidence, case 

VWXGLHVRIZKDW·VZRUNHGLQWKHÀHOGDQGQHZLQQRYDWLRQVFXUUHQWO\EHLQJH[SORUHGWRLQIRUPWKHLUZRUN

86'2/:

• '2/XVHVWKH&URVVDJHQF\)HGHUDO(YLGHQFH)UDPHZRUN for evaluation planning and dissemination. 

• '2/·V&OHDULQJKRXVHIRU/DERU(YDOXDWLRQDQG5HVHDUFK&/($5 is an internet-based evidence clearinghouse of evaluation reports that reviews 

GHVLJQVPHWKRGRORJLHVDQGÀQGLQJVDFFRUGLQJWRVSHFLÀFVWDQGDUGVGHYHORSHGE\WHFKQLFDOZRUNJURXSV(DFKVWXG\LVVFRUHGDQGJLYHQD ́FDXVDO

HYLGHQFHUDWLQJμDFFRUGLQJWRWKHVFRULQJUXEULFLQWKHVWDQGDUGV&/($5LVDXVHUIULHQGO\VHDUFKDEOHZHEVLWHWKDWLQFOXGHVDFDGHPLFTXDOLW\UHYLHZV

for each study included in the system, appropriate for peer academic researchers, potential evaluation contractors submitting technical proposals, 

SURJUDPSUDFWLWLRQHUVVHHNLQJLQIRUPDWLRQRQ ́ZKDWZRUNVμSROLF\PDNHUVDQGWKHJHQHUDOSXEOLF

• '2/XVHVWKH&/($5HYLGHQFHJXLGHOLQHVDQGVWDQGDUGVZKHQGLVFUHWLRQDU\SURJUDPJUDQWVDZDUGHGXVLQJHYLGHQFHLQIRUPHGRUHYLGHQFHEDVHG

criteria. The published guidelines and standards are thus used in grants for evidence-based programs demonstrations and in reviewing evaluations in 

WKHVWUXFWXUHGHYLGHQFHUHYLHZVFRQGXFWHGLQ&/($55HTXHVWVIRUSURSRVDOVDOVRLQGLFDWHWKH&/($5VWDQGDUGVDSSO\WRDOO&(2HYDOXDWLRQV$OVR

'2/KDVD ́'HSDUWPHQW(YDOXDWLRQ3ROLF\6WDWHPHQW” that formalizes the principles that govern all program evaluations in the Department, including 

PHWKRGRORJLFDOULJRULQGHSHQGHQFHWUDQVSDUHQF\HWKLFVDQGUHOHYDQFH,QDGGLWLRQ&(2SXEOLFO\FRPPXQLFDWHVWKHVWDQGDUGVDQGPHWKRGV

H[SHFWHGLQDOO'2/HYDOXDWLRQVDQGWKHVWDQGDUGVDUHLQFRUSRUDWHGLQWRIRUPDOSURFXUHPHQWVWDWHPHQWVRIZRUNZLWKVFRULQJIRUDZDUGVEDVHGRQ

the standards.

• $GGLWLRQDOO\'2/FROODERUDWHVZLWKRWKHUDJHQFLHV++6(',(616)&1&6RQUHÀQLQJFURVVDJHQF\HYLGHQFHJXLGHOLQHVDQGGHYHORSLQJ

technological procedures to link and share reviews across clearinghouses. The ,QWHUDJHQF\(YLGHQFH)UDPHZRUN conveys the categories of 

HYDOXDWLRQVWKHTXDOLW\UHYLHZRIHYDOXDWLRQPHWKRGRORJLHVDQGUHVXOWVDQGWKHXVHRIHYDOXDWLRQÀQLQJV7KHIUDPHZRUNLVDFFHSWHG'HSDUWPHQW

wide.

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

22

The Promise of Evidence-Based Policymaking 1065
EVIDENCE / EVALUATION CRITERIA

7. Innovation: Did the agency have staff, policies, 

and processes in place that encouraged innovation to 

improve the impact of its programs in FY16?

$&):

• $&)·V%HKDYLRUDO,QQRYDWLRQVWR$GYDQFH6HOI6XIÀFLHQF\%,$6SURMHFWLVWKHÀUVWPDMRUHIIRUWWRDSSO\DEHKDYLRUDOHFRQRPLFVOHQVWRSURJUDPV

that serve poor families in the U.S. Since its inception in 2010, the project has conducted 15 rapid-cycle randomized tests of behavioral innovations in 

seven states with nearly 100,000 sample members. 

• $&)·VBehavioral Interventions for Child Support Services (BICS) demonstration project is applying behavioral insights to child support contexts, 

developing promising behavioral interventions, and building a culture of regular, rapid-cycle evaluation and critical inquiry within the child support 

community. 

• 7KH$GPLQLVWUDWLRQ·V)<budget requestSSURSRVHVWRUHSXUSRVHWKH7HPSRUDU\$VVLVWDQFH&RQWLQJHQF\)XQGIRUDWDUJHWHGVHWRI

approaches to reducing poverty and promoting family economic security. These include demonstration projects to improve parental employment 

outcomes concurrently with child and family wellbeing outcomes; subsidized employment programs; and program improvement initiatives, such 

as monitoring and oversight, technical assistance, and research and evaluation. The proposed demonstration programs would set aside funds for 

evaluation.

• $&)KDVDFWLYHO\SDUWLFLSDWHGLQWKH++6,'($/DEDQHQWLW\ZLWKLQ++6ODXQFKHGLQWRLQYHVWLQLQWHUQDOLQQRYDWLRQOHYHUDJHH[WHUQDO

innovation, and build collaborative communities to tackle cross-cutting issues of strategic importance. Current projectsLQFOXGHWKH$&)$GPLQLVWUDWLRQ

IRU1DWLYH$PHULFDQV·$SSOLFDWLRQ7RRONLWDQG'DWD4XHVW0DNLQJ$&)1DWLYH'DWD9LVLEOHDQG8VHIXOWKH$&)2IÀFHRI)DPLO\$VVLVWDQFH·V

8QGHUVWDQGLQJ7HPSRUDU\$VVLVWDQFHIRU1HHG\)DPLOLHV7KURXJK'DWD9L]XDOL]DWLRQDQGWKH$&)2IÀFHRI+HDG6WDUW·V3DUWQHUVKLS$OLJQPHQW

,QIRUPDWLRQ5HVSRQVH6\VWHP

• $&)LVSDUWLFLSDWLQJLQWKH3HUIRUPDQFH3DUWQHUVKLS3LORWVIRU'LVFRQQHFWHG<RXWKLQLWLDWLYHE\SURYLGLQJÁH[LELOLW\IRUJUDQWHHVWRMRLQSDUWQHUVKLSV

DWWKHVWDWHOHYHO,QDGGLWLRQ$&)VWDIIKDYHVHUYHGDVWHFKQLFDODQGHYDOXDWLRQUHYLHZHUVIRUVHOHFWLQJWKH5RXQGSLORWVLWHVSDUWLFLSDWHGLQWKH

ÁH[LELOLW\UHYLHZSURFHVVDQGFRQWULEXWHGHYDOXDWLRQH[SHUWLVHWRSODQQLQJIRUORFDODQGQDWLRQDOHYDOXDWLRQVRISLORWVLWHV

CNCS:

• CNCS remains a partner in the 3HUIRUPDQFH3DUWQHUVKLS3LORW3SURJUDPDQGKDVFRQWULEXWHGWRWKHQDWLRQDOHYDOXDWLRQRIWKLVLQLWLDWLYH5	(

HVWLPDWHVWKDWRQHSLORWVLWHLVOHYHUDJLQJ&1&6IXQGLQJWKURXJK3

• CNCS awarded 10 grants that launched in FY16 as part of a new grant making initiative called 2SHUDWLRQ$PHUL&RUSV. This initiative was designed 

WRHQFRXUDJHWULEDODQGORFDOOHDGHUVWRLGHQWLI\DKLJKSULRULW\ORFDOFKDOOHQJHWKDW$PHUL&RUSV6WDWHDQG1DWLRQDO$PHUL&RUSV1&&&DQG

RU$PHUL&RUSV9,67$PHPEHUVFDQKROLVWLFDOO\DGGUHVVLQDUHODWLYHO\VKRUWSHULRGRIWLPHQRPRUHWKDQ\HDUV7KLVJUDQWPDNLQJLQLWLDWLYHLV

LQQRYDWLYHIRU&1&6DVLWLVWKHÀUVWJUDQWSURJUDPWKDWUHTXLUHVWKHEOHQGLQJRIUHVRXUFHVIURPGLIIHUHQW$PHUL&RUSVSURJUDPV2ZKLFKXVXDOO\

operate separately – to create a new transformative service solution. In addition to requiring a blended service model, the grant program streamlined 

WKHDSSOLFDWLRQSURFHVVDQGIDFLOLWDWHGDQLQWHUQDOH[DPLQDWLRQRIZD\VWKHDJHQF\·VEXVLQHVVSURFHVVHVFDQEHLPSURYHG&1&6·V5	(2IÀFHLV

FRQGXFWLQJDWZR\HDUSURFHVVHYDOXDWLRQRI2SHUDWLRQ$PHUL&RUSVZKLFKLVGHVLJQHGWRSURYLGHPRUHIRUPDOÀQGLQJVRQWKHH[WHQWWRZKLFKWKH

goals of the initiative were achieved.

FEDERAL INVEST IN WHAT WORKS INDEX (2016)



1066 The Promise of Evidence-Based Policymaking
EVIDENCE / EVALUATION CRITERIA

7. Innovation: Did the agency have staff, policies, 

and processes in place that encouraged innovation to 

improve the impact of its programs in FY16?

MCC:

• ,Q6HSWHPEHU0&&·V0RQLWRULQJDQG(YDOXDWLRQGLYLVLRQODXQFKHGWKHDJHQF\·VÀUVWOpen Data ChallengeDFDOOWRDFWLRQWR0DVWHUVDQG3K'

VWXGHQWVZRUNLQJLQHFRQRPLFVSXEOLFSROLF\LQWHUQDWLRQDOGHYHORSPHQWRURWKHUUHODWHGÀHOGVZKRZHUHLQWHUHVWHGLQH[SORULQJKRZWRXVHpublicly 

DYDLODEOH0&&ÀQDQFHGSULPDU\GDWDIRUSROLF\UHOHYDQWDQDO\VLV7KH&KDOOHQJHZDVLQWHQGHGWRIDFLOLWDWHEURDGHUXVHRI0&&·V86WD[SD\HUIXQGHG

GDWD'XHWRWKHVXFFHVVRIWKHÀUVWOpen Data Challenge, a second Open Data Challenge was launched in February 2016 in order to encourage 

LQQRYDWLYHLGHDVDQGPD[LPL]HWKHXVHRIGDWDWKDW0&&ÀQDQFHVIRULWVLQGHSHQGHQWHYDOXDWLRQV

• MCC is launching a gender data competitionLQ&{WHG·,YRLUHLQSDUWQHUVKLSZLWKWKH'DWD[LQLWLDWLYHRIWKH81)RXQGDWLRQDQGWKH:RUOG:LGH:HE

Foundation. The competition and larger partnership will spur interest in, creative use of, and new learning from data related to women and girls.

• ,Q0&&ODXQFKHGDQLQWHUQDO ́6ROXWLRQV/DEμWKDWZDVGHVLJQHGWRHQFRXUDJHLQQRYDWLRQE\HQJDJLQJVWDIIWRFRPHXSZLWKFUHDWLYHVROXWLRQVWR

some of the biggest challenges MCC faces.

• 0&&LVFRQGXFWLQJDQ ́,QQRYDWLRQ*UDQW3URJUDP” in Zambia in order to encourage local innovation in pro-poor service delivery in the water sector 

through grants to community-based organizations, civil society and/or private sector entities. 

• 0&&UHJXODUO\HQJDJHVLQLPSOHPHQWLQJSLORWSURMHFWVDVSDUWRILWVRYHUDOO&RPSDFWSURJUDPV$IHZH[DPSOHVLQFOXGHLQ0RURFFRDQLQQRYDWLYH

pay for results3)5PHFKDQLVPWRUHSOLFDWHRUH[SDQGSURYHQSURJUDPVWKDWSURYLGHLQWHJUDWHGVXSSRUWLQFOXGLQJVKRUWWHUPRQHWRVL[PRQWKVMRE

UHDGLQHVVVNLOOVWUDLQLQJWHFKQLFDOWUDLQLQJMREPDWFKLQJIROORZXSWRHQVXUHORQJHYLW\DQGRWKHUVHUYLFHVDQGD ́call-for-ideas” in Benin in 2015 

that extended an invitation to interested companies and organizations from around the world to submit information regarding potential projects that 

ZRXOGH[SDQGDFFHVVWRUHQHZDEOHRIIJULGHOHFWULFDOSRZHULQ%HQLQDQGDUHJXODWRU\VWUHQJWKHQLQJSURMHFWLQ6LHUUD/HRQHWKDWLQFOXGHVIXQGLQJ

for a UHVXOWVEDVHGÀQDQFLQJV\VWHPGHVLJQHGWRVWUHQJWKHQWKHUHJXODWRU·VUROHLQFHQWLYL]HSHUIRUPDQFHE\WKHXWLOLWLHVDQGHQKDQFHDFFRXQWDELOLW\

86$,':

• 86$,'HVWDEOLVKHGWKH86*OREDO'HYHORSPHQW/DEWKH/DELQWRLQFUHDVHWKHDSSOLFDWLRQRIWHFKQRORJ\LQQRYDWLRQDQGSDUWQHUVKLSVWR

H[WHQGWKH$JHQF\·VGHYHORSPHQWLPSDFWLQKHOSLQJWRHQGH[WUHPHSRYHUW\7KH/DEGRHVWKLVE\ZRUNLQJFORVHO\ZLWKFROOHDJXHVDFURVVWKH$JHQF\

and by bringing together a diverse set of partners to discover, test, and scale breakthrough innovations to solve development challenges faster and 

FKHDSHUDQGPRUHVXVWDLQDEO\7KH/DELVWKHKRPHIRUWKH0RQLWRULQJ(YDOXDWLRQ5HVHDUFKDQG/HDUQLQJ,QQRYDWLRQVSURJUDP0(5/,1) to source, 

co-design, implement and test solutions that innovate on traditional approaches to monitoring, evaluation, research and learning.

• 86$,'KDVDOVRODXQFKHGVL[grand challenges to engage the public in the search for solutions to development problems. 

• 7KH'HYHORSPHQW,QQRYDWLRQ9HQWXUHV',9DZDUGVJUDQWÀQDQFLQJWRZLQQHUVLQWKUHHGLVWLQFWVWDJHVRIÀQDQFLQJ)XQGLQJUDQJHVIURPXQGHU

$100,000 to $15 million, and is based on where a project is in its development and to what extent it has previously gathered evidence of success. The 

DIV modelLVGHVLJQHGWRÀQGEUHDNWKURXJKVROXWLRQVPLQLPL]HULVNDQGPD[LPL]HLPSDFWWKURXJKVWDJHÀQDQFLQJULJRURXVO\WHVWLPSDFWVDQGFRVW

effectiveness, and scale proven solutions through the public or private sectors.

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

24

The Promise of Evidence-Based Policymaking 1067
EVIDENCE / EVALUATION CRITERIA

7. Innovation: Did the agency have staff, policies, 

and processes in place that encouraged innovation to 

improve the impact of its programs in FY16?

86(':

• ('·V,QYHVWLQJLQ,QQRYDWLRQLLVWKH'HSDUWPHQW·VVLJQDWXUHLQQRYDWLRQSURJUDPIRU.2SXEOLFHGXFDWLRQ:KLOHWKHODUJHULJUDQWVDUHIRFXVHG

RQYDOLGDWLQJDQGVFDOLQJHYLGHQFHEDVHGSUDFWLFHVWKHVPDOOHULJUDQWVDUHGHVLJQHGWRHQFRXUDJHLQQRYDWLYHDSSURDFKHVWRSHUVLVWHQWFKDOOHQJHV

7KHVH ́'HYHORSPHQWμJUDQWVDUHWKHPRVWSUHYDOHQWW\SHRILJUDQWFRPSULVLQJRXWRIWKHLJUDQWVWRGDWHDQGRIWKHQHZLJUDQWV

made in FY15. In order to spur similar types of innovation in higher education, the Department made its second cohort of grantees under its First in 

the World),7:SURJUDPLQ)<7KH'HSDUWPHQWPDGH),7:JUDQWVLQ)<WKHYDVWPDMRULW\RIZKLFKRIZHUHLQWKH ́'HYHORSPHQWμ

category. 

• ('LVSDUWLFLSDWLQJLQWKH3HUIRUPDQFH3DUWQHUVKLS3LORWVIRU'LVFRQQHFWHG<RXWK initiative. These pilots give state, local, and tribal governments an 

opportunity to test innovative new strategies to improve such outcomes for low-income disconnected youth ages 14 to 24, including youth who are in 

foster care, homeless, young parents, involved in the justice system, unemployed, or who have dropped out or are at risk of dropping out of school.

• The White House Social and Behavioral Sciences TeamKDVFRQGXFWHGVHYHUDOEHKDYLRUDOVFLHQFHVWXGLHVUHODWHGWR('·VZRUNLQFOXGLQJORRNLQJDW

WKHLPSDFWRIWH[WPHVVDJHUHPLQGHUVIRUVWXGHQWVDERXWNH\WDVNVUHODWHGWRFROOHJHPDWULFXODWLRQVXFKDVFRPSOHWLQJÀQDQFLDODLGSDSHUZRUNDQG

about notices to student borrowers about income-driven repayment plans. 

• ('LVFXUUHQWO\LPSOHPHQWLQJWKH([SHULPHQWDO6LWHV,QLWLDWLYHWRWHVWWKHHIIHFWLYHQHVVRIVWDWXWRU\DQGUHJXODWRU\ÁH[LELOLW\IRUSDUWLFLSDWLQJLQVWLWXWLRQV

disbursing Title IV student aid. 

• ('KDVKLUHGDIXOOWLPH3D\IRU6XFFHVVIHOORZLQ)<('KDVHQWHUHGLQWRDQDJUHHPHQWZLWKWKH8QLYHUVLW\RI8WDK·V3ROLF\,QQRYDWLRQ/DEWR

VXSSRUWDIXOOWLPH3D\IRU6XFFHVV)HOORZDW(':LWKDGGLWLRQDOH[SHUWLVHSURYLGHGE\WKLVIHOORZ('LVGHHSHQLQJLWVFDSDFLW\DQGGHYHORSLQJZD\V

WRXVH3D\IRU6XFFHVVWRH[SDQGHIIHFWLYHHGXFDWLRQDOSURJUDPVDQGSURPRWHLQQRYDWLRQ

• 7KH,(65HVHDUFK*UDQWV3URJUDP supports the development and iterative testing of new, innovative approaches to improving education outcomes. 

,(6PDNHVUHVHDUFKJUDQWVZLWKDJRDOVWUXFWXUH ́*RDO'HYHORSPHQWDQG,QQRYDWLRQμVXSSRUWVWKHGHYHORSPHQWRIQHZHGXFDWLRQFXUULFXOD

instructional approaches; professional development; technology; and practices, programs, and policies that are implemented at the student-, 

classroom-, school-, district-, state-, or federal-level to improve student education outcomes.

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

25

1068 The Promise of Evidence-Based Policymaking
EVIDENCE / EVALUATION CRITERIA

7. Innovation: Did the agency have staff, policies, 

and processes in place that encouraged innovation to 

improve the impact of its programs in FY16?

USHUD:

• +8'·V3ROLF\'HYHORSPHQWDQG5HVHDUFK3'	5RIÀFHLVFRQGXFWLQJDQXPEHURIHYDOXDWHGUDQGRPDVVLJQPHQWSURJUDPGHPRQVWUDWLRQVWRWHVW

QHZSURJUDPPRGHOVZKLFKFDQEHIRXQGLQ3'	5·Vbiennial report: the Family Options studyRIKRPHOHVVQHVVLQWHUYHQWLRQV)DPLO\6HOI6XIÀFLHQF\

'HPRQVWUDWLRQ3UH3XUFKDVH+RPHRZQHUVKLS&RXQVHOLQJ'HPRQVWUDWLRQ6XSSRUWDQG6HUYLFHVDW+RPH6$6+'HPRQVWUDWLRQIRUHOGHUO\

KRXVHKROGV6XSSRUWLYH6HUYLFHV'HPRQVWUDWLRQIRUKHDOWKVHUYLFHVLQHOGHUO\KRXVLQJ5HQW5HIRUP'HPRQVWUDWLRQDQGWKH6PDOO$UHD)DLU0DUNHW

5HQW'HPRQVWUDWLRQ7KHODWWHUGHPRQVWUDWLRQVDUHLQHDUO\RUPLGGOHVWDJHVLQWHULPUHVXOWVDQGORQJWHUPIROORZXSUHVXOWVJHQHUDOO\ZLOOEHUHSRUWHG

WKURXJK+8'86(5GXULQJWKHQH[W\HDUV

• 3'	5DOVRLVFROODERUDWLQJZLWKWKH:KLWH+RXVHSocial and Behavioral Sciences TeamDQG86'HSDUWPHQWRI(GXFDWLRQWROLQNWHQDQWGDWDZLWK

UHFRUGVRIVWXGHQWVDQGLQGLYLGXDOVVXEPLWWLQJ)UHH$SSOLFDWLRQVIRU)HGHUDO6WXGHQW$LGKHOSLQJLQFUHDVHDFFHVVRI+8'WHQDQWVWRKLJKHUHGXFDWLRQ

through low-cost, behaviorally informed experiments about effective outreach methods. While detailed information about these experiments is not 

DYDLODEOHDWSUHVHQWVRPHFDQEHIRXQGLQ+8'·V$QQXDO5HSRUWVHHSDQGZLOOEHLQFOXGHGLQ6%67·VDQQXDOUHSRUWLQ-XO\

• 3'	5KRXVHVWKH2IÀFHRI,QWHUQDWLRQDODQG3KLODQWKURSLF,QQRYDWLRQDQGDGPLQLVWHUVÀYHW\SHVRI6HFUHWDU\·V$ZDUGV to encourage excellence: 

3XEOLF3KLODQWKURSLF3DUWQHUVKLSV2SSRUWXQLW\DQG(PSRZHUPHQW+HDOWK\+RPHV+LVWRULF3UHVHUYDWLRQDQG+RXVLQJDQG&RPPXQLW\'HVLJQ7KH

FRPSHWLWLRQVDUHMXGJHGE\MXULHVRISURIHVVLRQDOVDQGEULQJYLVLELOLW\WRWKHQDWLRQ·VPRVWFRPSHOOLQJVROXWLRQVIRUDGGUHVVLQJKRXVLQJDQGFRPPXQLW\

development challenges. 

• 3'	5VSRQVRUVDQ,QQRYDWLRQLQ$IIRUGDEOH+RXVLQJ&RPSHWLWLRQWRHQJDJHPXOWLGLVFLSOLQDU\WHDPVRIJUDGXDWHVWXGHQWVLQDGGUHVVLQJDVSHFLÀF

housing problem developed by an actual public housing agency. The competitionLQFUHDVHVWKHQDWLRQ·VIXWXUHKXPDQFDSDFLW\WRDGGUHVVWKH

DIIRUGDEOHKRXVLQJFULVLVE\H[SRVLQJIXWXUHGHVLJQHUVDGPLQLVWUDWRUVDQGSROLF\PDNHUVWRUHDOZRUOGFKDOOHQJHVRIDVSHFLÀFOHJDODQGFRPPXQLW\

context, with their proposals to be evaluated by an expert jury. 

• ,Q)<+8'·V1DWLRQDO'LVDVWHU5HVLOLHQFH&RPSHWLWLRQ is providing funding for resilient housing and infrastructure projects to states and 

communities that suffered major disasters. Collaborative teams were assisted in extensively researching and developing their proposals by nine 

5HVLOLHQFH$FDGHPLHVGHYHORSHGE\WKH5RFNHIHOOHU)RXQGDWLRQLQSDUWQHUVKLSZLWK+8'7KHLQGHSWKMXULHGSURFHVVLVHQVXULQJWKDWWKHELOOLRQ

of resources available for these communities in FY16 will result in more resilient housing and infrastructure and bridge the gap between social and 

physical vulnerabilities.

86'2/:

• '2/LVSDUWLFLSDWLQJLQWKH3HUIRUPDQFH3DUWQHUVKLS3LORWV3 for innovative service delivery for disconnected youth which includes not only waivers 

DQGEOHQGLQJDQGEUDLGLQJRIIHGHUDOIXQGVEXWJLYHVERQXVSRLQWVLQDSSOLFDWLRQUHYLHZVIRUSURSRVLQJ ́KLJKWLHUμHYDOXDWLRQV'2/LVWKHOHDGDJHQF\

IRUWKHHYDOXDWLRQRI3'2/·V&(2DQG(7$SUHSDUHGDQHYDOXDWLRQWHFKQLFDODVVLVWDQFHwebinarIRU3JUDQWHHVLQDQGZLOOEHUHSHDWHG

IRUWKHQH[WURXQGRIJUDQWHHVLQ%HJLQQLQJLQ)<WKHQDWLRQDO3HYDOXDWLRQFRQWUDFWRULVDOVRSURYLGLQJHYDOXDWLRQ7$WRJUDQWHHVIRU

methodological design issues and data and management information systems. 

• '2/KDVLQLWLDWHGVL[EHKDYLRUDOLQVLJKWVWHVWVWKUHHLQXQHPSOR\PHQWLQVXUDQFHWZRLQ26+$DQGRQHLQ(%6$IRUSHQVLRQFRQWULEXWLRQVDQGWZR

EHKDYLRUDOLQVLJKWVWHVWLQJGLIIHUHQWPHVVDJLQJWRHQFRXUDJHYROXQWDU\FRPSOLDQFHHPEHGGHGLQWRDODUJHUH[SHULPHQWDOHYDOXDWLRQVLQ26+$DQG

8QHPSOR\PHQW,QVXUDQFH7KHEHKDYLRUDOWHVWVDUHEHLQJFRQGXFWHGLQ)<,QLWLDOÀQGLQJVZLOOEHUHOHDVHGLQ$SULODQGZLOOEHSRVWHGRQWKH

&(2website.

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

26

The Promise of Evidence-Based Policymaking 1069
EVIDENCE / EVALUATION CRITERIA

8. Use of Evidence in 5 Largest Competitive 

Grant Programs: Did the agency use evidence of 

effectiveness when allocating funds from its 5 largest 

competitive grant programs in FY16?

$&):

• ,Q)<WKHODUJHVWFRPSHWLWLYHJUDQWSURJUDPVDUH+HDG6WDUW8QDFFRPSDQLHG&KLOGUHQ6HUYLFHV(DUO\

+HDG6WDUW&KLOG&DUH3DUWQHUVKLSV7UDQVLWLRQDODQG0HGLFDO6HUYLFHVDQG3UHVFKRRO'HYHORSPHQW*UDQWV

($250,000,000).

• $&)·VtemplateVHHSLQ$WWDFKPHQW&IRUJUDQWDQQRXQFHPHQWVLQFOXGHVWZRRSWLRQVUHTXLULQJJUDQWHHVWRHLWKHUFROOHFWSHUIRUPDQFH

PDQDJHPHQWGDWDWKDWFRQWULEXWHVWRFRQWLQXRXVTXDOLW\LPSURYHPHQWDQGLVWLHGWRWKHSURMHFW·VORJLFPRGHORUFRQGXFWDULJRURXVHYDOXDWLRQIRU

which applicants must propose an appropriate design specifying research questions, measurement and analysis.

• ,Q)<$&)HVWDEOLVKHGWKH+HDG6WDUW'HVLJQDWLRQ5HQHZDO6\VWHP requiring Head Start ($9.2 billion in FY16) grantees to compete for grants 

PRYLQJIRUZDUGLIWKH\IDLOHGWRPHHWFULWHULDUHODWHGWRVHUYLFHTXDOLW\OLFHQVLQJDQGRSHUDWLRQVDQGÀVFDODQGLQWHUQDOFRQWURO

• $&)·V3HUVRQDO5HVSRQVLELOLW\(GXFDWLRQ3URJUDP ($75 million in FY16) includes three individual discretionary grant programs that support evidencebased competitive grants that teach youth about abstinence and contraception to prevent pregnancy and sexually transmitted infections. 

• 7RUHFHLYHIXQGVWKURXJK$&)·V&RPPXQLW\%DVHG&KLOG$EXVH3UHYHQWLRQ&%&$3SURJUDPVWDWHVPXVW ́GHPRQVWUDWHDQHPSKDVLVRQSURPRWLQJ

WKHLQFUHDVHGXVHDQGKLJKTXDOLW\LPSOHPHQWDWLRQRIHYLGHQFHEDVHGDQGHYLGHQFHLQIRUPHGSURJUDPVDQGSUDFWLFHVμ&%&$3GHÀQHVHYLGHQFH

EDVHGDQGHYLGHQFHLQIRUPHGSURJUDPVDQGSUDFWLFHVDORQJDFRQWLQXXPZLWKIRXUFDWHJRULHV(PHUJLQJDQG(YLGHQFH,QIRUPHG3URPLVLQJ

6XSSRUWHGDQG:HOO6XSSRUWHG3URJUDPVGHWHUPLQHGWRIDOOZLWKLQVSHFLÀFSURJUDPSDUDPHWHUVZLOOEHFRQVLGHUHGWREH ́HYLGHQFHLQIRUPHGμRU

 ́HYLGHQFHEDVHGμSUDFWLFHV(%3DVRSSRVHGWRSURJUDPVWKDWKDYHQRWEHHQHYDOXDWHGXVLQJDQ\VHWFULWHULD$&)PRQLWRUVSURJUHVVRQWKH

percentage of program funds (most recently 89.4% in FY14) directed towards evidence-based and evidence-informed practices.

CNCS:

• ,Q&1&6LVRSHUDWLQJWKUHHFRPSHWLWLYHJUDQWSURJUDPVLQ)<$PHUL&RUSV6WDWHDQG1DWLRQDOSURJUDPH[FOXGLQJ6WDWHIRUPXODJUDQWIXQGV

PLOOLRQLQ)<6HQLRU&RUSV5693SURJUDPPLOOLRQLQ)<DQGWKH6RFLDO,QQRYDWLRQ)XQG6,)PLOOLRQLQ)<

• 6,)SURYLGHVFRPSHWLWLYHJUDQWVWRQRQSURÀWJUDQWPDNLQJRUJDQL]DWLRQVWRKHOSWKHPJURZSURPLVLQJHYLGHQFHEDVHGVROXWLRQVWKDWDGGUHVV

SUHVVLQJHFRQRPLFRSSRUWXQLW\KHDOWK\IXWXUHVDQG\RXWKGHYHORSPHQWLVVXHVLQORZLQFRPHFRPPXQLWLHV7KH)<2PQLEXV$SSURSULDWLRQV

$FWVKDYHDOORZHG&1&6WRLQYHVWXSWRRI6,)IXQGVHDFK\HDULQ3D\IRU6XFFHVVLQLWLDWLYHV7KHUHDUH3D\IRU6XFFHVVFRPSHWLWLRQVSODQQHG

for FY16, which will invest both the FY15 and 16 appropriations (approximately $11.6 million at minimum). 

• &1&6·V$PHUL&RUSV6WDWHDQG1DWLRQDO*UDQWV3URJUDPH[FOXGLQJ6WDWHIRUPXODJUDQWIXQGVapplication (see pp. 10-14) allocated up to 27 points 

RXWRIWRRUJDQL]DWLRQVWKDWVXEPLWDSSOLFDWLRQVVXSSRUWHGE\SHUIRUPDQFHDQGHYDOXDWLRQGDWDLQ)<6SHFLÀFDOO\XSWRSRLQWVFDQEH

assigned to applications with theories of change supported by relevant research literature, program performance data, or program evaluation data; 

DQGXSWRSRLQWVDSRLQWLQFUHDVHIURP)<FDQEHDVVLJQHGIRUDQDSSOLFDQW·VLQFRPLQJOHYHORIHYLGHQFHZLWKWKHKLJKHVWQXPEHURISRLQWV

DZDUGHGWRVWURQJOHYHOVRIHYLGHQFH7KHVHFDWHJRULHVRIHYLGHQFHDUHPRGHOHGFORVHO\RQWKHOHYHOVRIHYLGHQFHGHÀQHGLQWKH6RFLDO,QQRYDWLRQ

Fund. 

• ,Q)<6HQLRU&RUSV5693JUDQWHHVseeking fundingVHHSWKURXJKWKHDGPLQLVWUDWLYHUHQHZDOSURFHVVDUHHQFRXUDJHGWRIXOÀOOWKH1DWLRQDO

3HUIRUPDQFH0HDVXUHVUHTXLUHPHQWE\FRPPLWWLQJDFHUWDLQQXPEHURIYROXQWHHUVWRVHUYHLQDQHYLGHQFHEDVHGKHDOWKHGXFDWLRQSURJUDP$WRWDO

of $500,000 (just above 1% of program funds) is allocated to support organizations in implementing evidence-based interventions or to evaluate 

programs.

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

27

1070 The Promise of Evidence-Based Policymaking
EVIDENCE / EVALUATION CRITERIA

8. Use of Evidence in 5 Largest Competitive 

Grant Programs: Did the agency use evidence of 

effectiveness when allocating funds from its 5 largest 

competitive grant programs in FY16?

MCC:

• MCC awards all of its agency funds through two competitive grant programs: Compact and Threshold programs (whose budgets for FY16 were 

DQGPLOOLRQUHVSHFWLYHO\%RWKUHTXLUHGHPRQVWUDEOHREMHFWLYHHYLGHQFHWRVXSSRUWWKHOLNHOLKRRGRIVXFFHVVLQRUGHUWREHDZDUGHGIXQGV

For country partner selection, MCC uses twenty different indicators within the categories of economic freedom, investing in people, and ruling justly 

WRGHWHUPLQHFRXQWU\HOLJLELOLW\IRUSURJUDPDVVLVWDQFH7KHVHLQGLFDWRUVVHH0&&·V)<*XLGHWRWKH,QGLFDWRUV) are collected by independent 

third parties. When considering granting a second compact, MCC considers 1) the degree to which there is evidence of strong political will and 

PDQDJHPHQWFDSDFLW\WKHGHJUHHWRZKLFKWKHFRXQWU\KDVH[KLELWHGFRPPLWPHQWDQGFDSDFLW\WRDFKLHYHSURJUDPUHVXOWVDQGWKHGHJUHHWR

ZKLFKWKHFRXQWU\KDVLPSOHPHQWHGWKHFRPSDFWLQDFFRUGDQFHZLWK0&&·VFRUHSROLFLHVDQGVWDQGDUGV

• Following country selection, MCC conducts a constraints analysis&$WRLGHQWLI\WKHPRVWELQGLQJFRQVWUDLQWVWRSULYDWHLQYHVWPHQWDQG

entrepreneurship that hold back economic growth. The results of this analysis enable the country, in partnership with MCC, to select compact or 

threshold activities most likely to contribute to sustainable poverty-reducing growth. Due diligence, including feasibility studies where applicable, are 

conducted for each potential investment. MCC also performs &RVW%HQHÀW$QDO\VLV to assess the potential impact of each project, and estimates an 

(FRQRPLF5DWHRI5HWXUQ0&&SURMHFWVJHQHUDOO\KDYHDQ(55DERYHDWSURMHFWLQFHSWLRQDQG0&&UHFDOFXODWHV(55VDWFRPSDFWFORVHRXW

LQRUGHUWRWHVWRULJLQDODVVXPSWLRQVDQGDVVHVVWKHFRVWHIIHFWLYHQHVVRI0&&SURJUDPV,QFRQQHFWLRQZLWKWKH(550&&FRQGXFWVD%HQHÀFLDU\

$QDO\VLVZKLFKVHHNVWRGHVFULEHSUHFLVHO\ZKLFKVHJPHQWVRIVRFLHW\ZLOOUHDOL]HWKHSURMHFWEHQHÀWV,WLVPRVWFRPPRQO\XVHGWRDVVHVVWKHLPSDFW

of projects on the poor, but it has broader applicability that allows for the estimation of impact on populations of particular interest, such as women, 

WKHDJHGFKLOGUHQDQGUHJLRQDORUHWKQLFVXESRSXODWLRQV,QOLQHZLWK0&&·V0	(SROLF\0&&SURMHFWVDUHUHTXLUHGWRVXEPLWTXDUWHUO\,QGLFDWRU

Tracking Tables showing progress toward projected targets. MCC also requires independent evaluations of every project to assess progress in 

achieving outputs and outcomes throughout the lifetime of the project and beyond..

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

28

The Promise of Evidence-Based Policymaking 1071
EVIDENCE / EVALUATION CRITERIA

8. Use of Evidence in 5 Largest Competitive 

Grant Programs: Did the agency use evidence of 

effectiveness when allocating funds from its 5 largest 

competitive grant programs in FY16?

86$,':

• 86$,'LVFRPPLWWHGWRXVLQJHYLGHQFHRIHIIHFWLYHQHVVLQDOORILWVFRPSHWLWLYHFRQWUDFWVFRRSHUDWLYHDJUHHPHQWVDQGJUDQWVZKLFKFRPSULVHWKH

PDMRULW\RIWKH$JHQF\·VZRUN86$,'KDVUHEXLOWLWVplanning, monitoring, and evaluation framework to produce and use evidence through the 

introduction of a new 3URJUDP&\FOHZKLFKV\VWHPDWL]HVXVHRIHYLGHQFHDFURVVDOOGHFLVLRQPDNLQJUHJDUGLQJJUDQWVDQGDOORI86$,'·VZRUN7KH

3URJUDP&\FOHLV86$,'·VSDUWLFXODUIUDPLQJDQGWHUPLQRORJ\WRGHVFULEHDFRPPRQVHWRISURFHVVHVLQWHQGHGWRDFKLHYHPRUHHIIHFWLYHGHYHORSPHQW

LQWHUYHQWLRQVDQGPD[LPL]HLPSDFWV7KH3URJUDP&\FOHDFNQRZOHGJHVWKDWGHYHORSPHQWLVQRWVWDWLFDQGLVUDUHO\OLQHDUDQGWKHUHIRUHVWUHVVHVWKH

QHHGWRDVVHVVDQGUHDVVHVVWKURXJKUHJXODUPRQLWRULQJHYDOXDWLRQDQGOHDUQLQJ7KXVWKHGLIIHUHQWFRPSRQHQWVRIWKH3URJUDP&\FOHPXWXDOO\

UHLQIRUFHHDFKRWKHUE\KDYLQJOHDUQLQJDQGDGDSWLQJLQWHJUDWHGWKURXJKRXW7KH3URJUDP&\FOHHQFRXUDJHVSODQQLQJDQGSURMHFWPDQDJHPHQW

innovations to increase the cost-effectiveness and lasting impact of development cooperation.

• ,Q86$,'UHIRUPHGLWVSROLF\IRUDZDUGLQJQHZFRQWUDFWVWRHOHYDWHSDVWSHUIRUPDQFHWRFRPSULVHWRSHUFHQWRIWKHQRQFRVWHYDOXDWLRQ

FULWHULD)RUDVVLVWDQFH86$,'GRHVD ́ULVNDVVHVVPHQWμWRUHYLHZDQRUJDQL]DWLRQ·VDELOLW\WRPHHWWKHJRDOVDQGREMHFWLYHVRXWOLQHGE\WKH$JHQF\

This can be found in $'6VHFWLRQ&RQWUDFWRUSHUIRUPDQFHLVJXLGHGE\86$,'RSHUDWLRQDOSROLF\$'6VHFWLRQ$V

UHTXLUHGLQ)$56XESDUW86$,'PXVWHYDOXDWHFRQWUDFWRUSHUIRUPDQFHXVLQJWKH&RQWUDFWRU3HUIRUPDQFH$VVHVVPHQW5HSRUWLQJ6\VWHP

&3$56,QIRUPDWLRQLQ&3$56ZKLOHQRWDYDLODEOHWRWKHSXEOLFLVDYDLODEOHIRU&RQWUDFWLQJ2IÀFHUVDFURVVWKH*RYHUQPHQWWRXVHLQPDNLQJ

determinations of future awards.

• 86$,'KDVDOVRLQVWLWXWHGDSROLF\FDOOHGWKH$FTXLVLWLRQDQG$VVLVWDQFH5HYLHZDQG$SSURYDO'RFXPHQW$$5$'SURFHVVZKHUHDOOFRQWUDFWV

JUDQWVDQGFRRSHUDWLYHDJUHHPHQWVRYHUPLOOLRQDUHUHYLHZHGE\WKH$GPLQLVWUDWRUSULRUWREHLQJDZDUGHGDQGDOODZDUGVRYHUPLOOLRQDUH

UHYLHZHGE\WKHUHOHYDQW$VVLVWDQW$GPLQLVWUDWRUV,QFOXGHGLQWKH$$5$'UHYLHZDUHVHYHUDONH\IDFWRUVWKDWLQFOXGH3ROLF\5HOHYDQW&RPPLWPHQWWR

6XVWDLQDEOH5HVXOWV)HDVLELOLW\DQG9DOXHIRU0RQH\7KLVSROLF\HQVXUHVWKDWUHVXOWVHYLGHQFHDQGORQJWHUPVWUDWHJLHVDUHLQFRUSRUDWHGLQWRDOORI

86$,'·VPDMRUSURJUDPV,QDGGLWLRQLWHQVXUHVVHQLRUOHYHODFFRXQWDELOLW\RQ86$,'·VELJJHVWSURJUDPV7KLVSROLF\LVRXWOLQHGLQ$'686$,'

guidance for competitive grants is also available online.

• The Development Innovation Ventures program ($22.4 million in FY16) provides funding for proof of concept through rigorous evaluation of innovative 

solutions, and scale-up funding when a solution is proven to work. ',9·VDSSURDFK is unique in three ways:

1. DIV recognizes that good ideas can come from anywhere, so they welcome a wide range of potential partners to propose their concepts for 

high-impact development solutions. 

 %RUURZLQJIURPWKHH[SHULHQFHRIYHQWXUHFDSLWDO',9WDNHVDGYDQWDJHRIDVWDJHGÀQDQFLQJPRGHO7KH\SLORWSURPLVLQJQHZLGHDVZLWKVPDOO

amounts of money, and we scale only those solutions that rigorously demonstrate their impact.

 ',9HPSKDVL]HVDKLJKVWDQGDUGRIHYLGHQFHLQFOXGLQJWKHXVHRILPSDFWHYDOXDWLRQVDQGUDQGRPL]HGFRQWUROWULDOVZKHQHYHUSRVVLEOH

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

29

1072 The Promise of Evidence-Based Policymaking
EVIDENCE / EVALUATION CRITERIA

8. Use of Evidence in 5 Largest Competitive 

Grant Programs: Did the agency use evidence of 

effectiveness when allocating funds from its 5 largest 

competitive grant programs in FY16?

86(':

• ('·VÀYHODUJHVWFRPSHWLWLYHJUDQWSURJUDPVLQ)<LQFOXGH75,2PLOOLRQ*($583PLOOLRQ7HDFKHU,QFHQWLYH)XQG

PLOOLRQ&KDUWHU6FKRROV*UDQWVPLOOLRQDQG3UHVFKRRO'HYHORSPHQW*UDQWVPLOOLRQ

• 7KH(YLGHQFH3ODQQLQJ*URXS(3*DGYLVHVSURJUDPRIÀFHVRQZD\VWRLQFRUSRUDWHHYLGHQFHLQJUDQWSURJUDPVLQFOXGLQJXVHRIHYLGHQFHDVDQ

entry requirement or priority to encourage the use of practices where there is evidence of effectiveness, and/or an exit requirement or priority to build 

QHZHYLGHQFH)RUWKHSDVWVHYHUDO\HDUV('KDVUHSRUWHGSXEOLFO\RQ3HUIRUPDQFHJRY on its $JHQF\3ULRULW\*RDO$3*IRFXVHGRQGLUHFWLQJDQ

LQFUHDVLQJSHUFHQWDJHRIIXQGVDYDLODEOHIRUQHZFRPSHWLWLYHDZDUGVWRZDUGVSURMHFWVWKDWDUHVXSSRUWHGE\HYLGHQFH,Q)<('VSHQWRILWV

funding available for new discretionary awards on projects that are supported by promising, moderate, or strong evidence, based on ('*$5 evidence 

OHYHOVVXUSDVVLQJERWKWKH)<DQG)<WDUJHWVIRUWKDW$3*

• :KLOHQRWDOORI('·V)<GHFLVLRQVKDYHEHHQÀQDOL]HG\HW('KDVDQQRXQFHGWKHIROORZLQJ)<FRPSHWLWLRQVZKLFKLQFOXGHWKHXVHRIHYLGHQFH

EH\RQGDORJLFPRGHO$ODVND1DWLYHDQG1DWLYH+DZDLLDQ6HUYLQJ,QVWLWXWLRQV$VLDQ$PHULFDQDQG1DWLYH$PHULFDQ3DFLÀF,VODQGHU6HUYLQJ

,QVWLWXWLRQV3URJUDP&ROOHJH$VVLVWDQFH0LJUDQW3URJUDP(GXFDWLRQDO7HFKQRORJ\0HGLDDQG0DWHULDOVIRU,QGLYLGXDOVZLWK'LVDELOLWLHV3

6WHSSLQJXS7HFKQRORJ\,PSOHPHQWDWLRQ+LJK6FKRRO(TXLYDOHQF\3URJUDP+LVSDQLF6HUYLQJ,QVWLWXWLRQV6FLHQFH7HFKQRORJ\(QJLQHHULQJ

RU0DWKHPDWLFV1DWLRQDO3URIHVVLRQDO'HYHORSPHQW1DWLYH$PHULFDQ6HUYLQJ1RQWULEDO,QVWLWXWLRQV3URJUDP7HFKQLFDO$VVLVWDQFHDQG

'LVVHPLQDWLRQ7R,PSURYH6HUYLFHVDQG5HVXOWVIRU&KLOGUHQ:LWK'LVDELOLWLHVDQG75,27DOHQW6HDUFK

• The ,QYHVWLQJLQ,QQRYDWLRQLSURJUDPPLOOLRQLQ)<SURYLGHVFRPSHWLWLYHJUDQWVWRORFDOVFKRROGLVWULFWVDQGQRQSURÀWRUJDQL]DWLRQVWKDW

have demonstrated positive impacts to innovate, expand, and scale evidence-based activities to improve student achievement, although details for the 

)<FRPSHWLWLRQKDYHQRWEHHQDQQRXQFHG(66$DXWKRUL]HVDQ(GXFDWLRQ,QQRYDWLRQDQG5HVHDUFK(,5*UDQWVSURJUDP

• $GGLWLRQDOO\(66$UHTXLUHVWKDW('JLYHSULRULW\WRDSSOLFDQWVGHPRQVWUDWLQJVWURQJPRGHUDWHRUSURPLVLQJOHYHOVRIHYLGHQFHZLWKLQWKHIROORZLQJ

VHYHQFRPSHWLWLYHJUDQWSURJUDPV/LWHUDF\(GXFDWLRQIRU$OO5HVXOWVIRUWKH1DWLRQ6XSSRUWLQJ(IIHFWLYH(GXFDWRU'HYHORSPHQW6FKRRO/HDGHU

5HFUXLWPHQWDQG6XSSRUW6WDWHZLGH)DPLO\(QJDJHPHQW&HQWHUV3URPLVH1HLJKERUKRRGV)XOO6HUYLFH&RPPXQLW\6FKRROVDQG6XSSRUWLQJ+LJK

$ELOLW\/HDUQHUVDQG/HDUQLQJ

• (66$DXWKRUL]HVWKH6XSSRUWLQJ(IIHFWLYH(GXFDWRU'HYHORSPHQWSURJUDPWKDWDZDUGVJUDQWVWRDSSOLFDQWVZLWKDGHPRQVWUDWHGUHFRUGRILPSURYLQJ

student outcomes while giving priority to applicants demonstrating strong, moderate, or promising evidence of effectiveness (as described above). 

$QG(66$DXWKRUL]HVWKH5HSOLFDWLRQDQG([SDQVLRQRI+LJK4XDOLW\&KDUWHU6FKRROVSURJUDPWKDWDZDUGVJUDQWVWRDSSOLFDQWVEDVHGRQWKHLU

demonstrated success in improving student outcomes.

• ('·V)<EXGJHWZKLFKIRU3SURJUDPVLVEDVHGRQ(66$SULRULWL]HVIXQGLQJHYLGHQFHEDVHGDFWLYLWLHV)RUH[DPSOHWKHEXGJHWLQFOXGHV

PLOOLRQIRUWKH(,5SURJUDPDQLQFUHDVHRIPLOOLRQRYHUWKH)<HQDFWHGOHYHOIRULWVSUHGHFHVVRUWKHLSURJUDP('DOVRSURSRVHVEXLOGLQJ

QHZHYLGHQFHWRLQFUHDVHWKHHIIHFWLYHQHVVRIWKH0DJQHW6FKRROV$VVLVWDQFH3URJUDP5HTXHVWVOLNHWKHPLOOLRQLQ),7:SURJUDPPLOOLRQ

+%&806,,QQRYDWLRQIRU&RPSOHWLRQ)XQGFRPSHWLWLYHJUDQWSURJUDPDQGWKHXVHRIXSWRPLOOLRQWRGHYHORSD75,2'HPRQVWUDWLRQ,QLWLDWLYHLQ

FRQVXOWDWLRQZLWKWKH75,2FRPPXQLW\GHPRQVWUDWH('·VFRPPLWPHQWWREXLOGLQJDQGXVLQJHYLGHQFHWRLPSURYHFROOHJHDFFHVVDQGFRPSOHWLRQ

FEDERAL INVEST IN WHAT WORKS INDEX (2016)



The Promise of Evidence-Based Policymaking 1073
EVIDENCE / EVALUATION CRITERIA

8. Use of Evidence in 5 Largest Competitive 

Grant Programs: Did the agency use evidence of 

effectiveness when allocating funds from its 5 largest 

competitive grant programs in FY16?

USHUD:

• ,Q)<+8'·VPDMRUFRPSHWLWLYHJUDQWSURJUDPVDUH+RPHOHVV$VVLVWDQFH ($1.9 billion); 2) 'LVDVWHU$VVLVWDQFH1DWLRQDO'LVDVWHU5HVLOLHQFH

CompetitionPLOOLRQ&KRLFH1HLJKERUKRRGV*UDQWVSURJUDP ($125 million); 4) Service Coordinators program ($77 million); and 5) Family 

6HOI6XIÀFLHQF\3URJUDP&RRUGLQDWRUVPLOOLRQ

• 7KH1DWLRQDO'LVDVWHU5HVLOLHQFH&RPSHWLWLRQXVHGHYLGHQFHDERXWGLVDVWHUUHVLOLHQFHLQFOXGLQJEHQHÀWFRVWDQDO\VLVWRHQVXUHWKDWGLVDVWHUIXQGLQJ

LPSURYHVFRPPXQLWLHV·DELOLW\WRZLWKVWDQGDQGUHFRYHUPRUHTXLFNO\IURPIXWXUHGLVDVWHUVKD]DUGVDQGVKRFNVUDWKHUWKDQVLPSO\UHFUHDWLQJWKH

same vulnerabilities.

• 'HFLVLRQVUHJDUGLQJWKHGHVLJQIXQGLQJDQGLPSOHPHQWDWLRQRIDOO+8'FRPSHWLWLYHJUDQWSURJUDPVDUHHYLGHQFHEDVHGDVVSHFLÀHGLQIXQGLQJ

FULWHULDLQ+8'·V)<1RWLFHRI)XQGLQJ$YDLODELOLW\12)$7KH ́$FKLHYLQJ5HVXOWVDQG3URJUDP(YDOXDWLRQμIDFWRUVHHSSURYLGHVIXQGLQJ

priority for applicants that demonstrate effective use of evidence in identifying or selecting the proposed practices, strategies, or programs proposed in 

WKHDSSOLFDWLRQDQGUHTXLUHVDOOJUDQWHHVWRFRRSHUDWHLQ+8'IXQGHGUHVHDUFKDQGHYDOXDWLRQVWXGLHVVHHS$QRWKHUIDFWRU ́3DVW3HUIRUPDQFHμ

SURYLGHV ́,QHYDOXDWLQJDSSOLFDWLRQVIRUIXQGLQJ+8'ZLOOWDNHLQWRDFFRXQWDQDSSOLFDQW·VSDVWSHUIRUPDQFHLQPDQDJLQJIXQGVLQFOXGLQJEXWQRW

OLPLWHGWR«PHHWLQJSHUIRUPDQFHWDUJHWVDVHVWDEOLVKHGLQ/RJLF0RGHOVRURWKHUSHUIRUPDQFHHYDOXDWLRQWRROVDSSURYHGDVSDUWRIWKHJUDQW

DJUHHPHQW«μVHHS7KH ́$FKLHYLQJ5HVXOWVDQG3URJUDP(YDOXDWLRQμIDFWRUDQG ́3DVW3HUIRUPDQFHμIDFWRUDUHWZRRIÀYHIDFWRUVFRQVLGHUHG

that total 100 points. The maximum achievable score, with priority points and bonus points, is 106.

86'2/:

• ,Q)<WKHÀYHODUJHVWFRPSHWLWLYHJUDQWSURJUDPVDZDUGHGZHUH$PHULFDQ$SSUHQWLFHVKLS,QLWLDWLYHPLOOLRQ)DFH)RUZDUG*UDQWV

3URJUDPPLOOLRQ'LVDELOLW\(PSOR\PHQW,QLWLDWLYHPLOOLRQ+RPHOHVV9HWHUDQV5HLQWHJUDWLRQ3URJUDPPLOOLRQDQGWKH

:RUNIRUFH,QQRYDWLRQ)XQG3D\IRU6XFFHVVPLOOLRQLQ)<$OOKDYHQDWLRQDOHYDOXDWLRQVGHVLJQHGE\&(2DQGWKHUHOHYDQWDJHQFLHV

and two also require grantees to use a portion of their fund for high-quality evaluations on which incentive and priority points were received in the 

application funding competitive selection process.

• '2/LQFOXGHVULJRURXVHYDOXDWLRQUHTXLUHPHQWVLQDOOFRPSHWLWLYHJUDQWSURJUDPVLQYROYLQJHLWKHUIXOOSDUWLFLSDWLRQLQDQDWLRQDOHYDOXDWLRQDVD

condition of grant receipt; 2) an independent third-party local or grantee evaluation with priority incentives for rigorous designs (e.g., tiered funding, 

VFRULQJSULRULWLHVERQXVVFRULQJIRUHYLGHQFHEDVHGLQWHUYHQWLRQVRUPXOWLVLWHULJRURXVWHVWVRUIXOOSDUWLFLSDWLRQLQDQDWLRQDOHYDOXDWLRQDVZHOO

as rigorous grantee (or local) evaluations. The $10 million /LQNLQJWR(PSOR\PHQW$VVLVWDQFH3UH5HOHDVH*UDQW program to improve employment for 

formerly incarcerated individuals serves as an example of the requirement to participate in a national evaluation as a condition of the grant.

• The 7UDGH$GMXVWPHQW$VVLVWDQFH&RPPXQLW\&ROOHJHDQG&DUHHU7UDLQLQJ*UDQW3URJUDP7$$&&&7SURJUDPELOOLRQLQ)<DYDLODEOH

through FY 2017; including $410 million in FY 2016) provides grants to community colleges and other higher education institutions to develop and 

H[SDQGHYLGHQFHEDVHGHGXFDWLRQDQGWUDLQLQJIRUGLVORFDWHGZRUNHUVFKDQJLQJFDUHHUV8SWRRIHDFKJUDQWFDQEHVSHQWRQHYDOXDWLRQ'2/

has awarded $11 million for technical assistance and a national evaluation of the program. 

• The Workforce Innovation FundJUDQWVPLOOLRQWRWDOLQFOXGLQJPLOOLRQDZDUGHGLQ)<DQG3D\IRU6XFFHVVPLOOLRQWRWDODUH

DZDUGHGWRULJRURXVO\WHVWLQQRYDWLRQWUDLQLQJDQGHPSOR\PHQWVWUDWHJLHVZLWKULJRURXVHYDOXDWLRQVLQFRUSRUDWHGLQWRWKHSURJUDPPLQJ3)6LVDVRFLDO

investment pilot with payment based on rigorous randomized control trial impacts.

FEDERAL INVEST IN WHAT WORKS INDEX (2016)



1074 The Promise of Evidence-Based Policymaking
EVIDENCE / EVALUATION CRITERIA

9. Use of Evidence in 5 Largest Non-Competitive 

Grant Programs: Did the agency use evidence of 

effectiveness when allocating funds from its 5 largest 

non-competitive grant programs in FY16?

(Note: Meeting this criteria requires both Agency and 

Congressional action.)

$&):

• ,Q)<$&)·VODUJHVWQRQFRPSHWLWLYHJUDQWSURJUDPVDUH7HPSRUDU\$VVLVWDQFHIRU1HHG\)DPLOLHV&KLOG&DUH

DQG'HYHORSPHQW)XQG%ORFN*UDQWDQG(QWLWOHPHQWWR6WDWHVFRPELQHG)RVWHU&DUH&KLOG6XSSRUW

(QIRUFHPHQW3D\PHQWVWR6WDWHVDQG/RZ,QFRPH+RPH(QHUJ\$VVLVWDQFH

• $&)·VFoster CareSURJUDPELOOLRQLQ)<KDVDSSURYHGRYHUMXULVGLFWLRQVWRGHYHORSDQGLPSOHPHQWFKLOGZHOIDUHZDLYHUGHPRQVWUDWLRQ

SURMHFWVWRLPSURYHRXWFRPHVIRUFKLOGUHQLQIRVWHUFDUHRUDWULVNIRUHQWU\RUUHHQWU\LQWRIRVWHUFDUH7KURXJKWKHVHGHPRQVWUDWLRQV$&)ZDLYHV

SURYLVLRQVRIODZWRDOORZÁH[LEOHXVHRIIXQGLQJQRUPDOO\OLPLWHGWRIRVWHUFDUHIRURWKHUFKLOGZHOIDUHVHUYLFHV0DQ\SDUWLFLSDWLQJMXULVGLFWLRQVDUH

implementing evidence-based or evidence-informed interventions and all demonstration projects are required to have a rigorous evaluation conducted 

E\DWKLUGSDUW\HYDOXDWRU$OWKRXJK$&)GRHVQRWFXUUHQWO\KDYHVWDWXWRU\DXWKRULW\WRJUDQWQHZZDLYHUVFXUUHQWSURMHFWVDUHH[SHFWHGWRFRQWLQXH

WKURXJK6HSWHPEHU*HQHUDOLQIRUPDWLRQ on this program, including a fact sheet and summary of relevant legislation/policy, is available at the 

online &KLOGUHQ·V%XUHDX portal.

CNCS:

• &1&6RSHUDWHVRQHIRUPXODJUDQWSURJUDPWKH$PHUL&RUSV6WDWHIRUPXODJUDQWVSURJUDPPLOOLRQLQ)<&1&6DOVRRSHUDWHVIRXUGLUHFW

JUDQWSURJUDPVLQ)<1DWLRQDO&LYLOLDQ&RPPXQLW\&RUSV1&&&PLOOLRQLQ)<9,67$PLOOLRQLQ)<6HQLRU&RUSV)RVWHU

*UDQGSDUHQWVPLOOLRQLQ)<DQG6HQLRU&RUSV6HQLRU&RPSDQLRQ3URJUDPPLOOLRQLQ)<

• ,Q)<IRUWKHÀUVWWLPHWKH6HQLRU&RUSV)RVWHU*UDQGSDUHQWVDQG6HQLRU&RPSDQLRQSURJUDPVHPEHGGHGHYLGHQFHLQWRWKHLUJUDQWUHQHZDO

processes by offering supplemental funding to grantees interested in deploying volunteers to serve in evidence-based programs (see pp. 2-4) and 

SURYLGLQJHYDOXDWLRQGDWDRQLPSOHPHQWDWLRQÀGHOLW\LQFOXGLQJRXWFRPHV$WRWDORILVDOORFDWHGIRUWKH)RVWHU*UDQGSDUHQWVSURJUDPLQ

)<RURISURJUDPIXQGV$WRWDORILVDOORFDWHGIRUWKH6HQLRU&RPSDQLRQSURJUDPLQ)<RURISURJUDPIXQGV

• 9,67$LVFXUUHQWO\GHYHORSLQJDWKHRU\RIFKDQJHWKDWZLOOPDNHH[SOLFLWWKHOLQNEHWZHHQWKHZRUNWKDWWKHYROXQWHHUVSHUIRUPWKHGHVLJQRID

VSRQVRU·VSURMHFWWRDGGUHVVFRPPXQLW\QHHGVDQGWKHHYLGHQFHWRVXSSRUWWKLVDFWLYLW\7KLVHIIRUWZLOOLPSDFWVHYHUDOPDQDJHPHQWDVSHFWVLQFOXGLQJ

project approval, volunteer assignment descriptions, member activity, data collection, and the role of evidence in the design and implementation of 

projects.

MCC:

• MCC does not administer non-competitive grant programs.

86$,':

• 86$,'GRHVQRWDGPLQLVWHUQRQFRPSHWLWLYHJUDQWSURJUDPV

• 86$,'GRHVFRQWULEXWHIXQGLQJWRPXOWLODWHUDOLQVWLWXWLRQVNQRZQDV3XEOLF,QWHUQDWLRQDO2UJDQL]DWLRQV3,2VZKLFKDUHOLVWHGhere, and include the 

:RUOG%DQN81DQGPXOWLGRQRUIXQGVVXFKDVWKH*OREDO)XQG$3XEOLF,QWHUQDWLRQDO2UJDQL]DWLRQ3,2LVDQLQWHUQDWLRQDORUJDQL]DWLRQFRPSRVHG

SULQFLSDOO\RIFRXQWULHV,QWKHVHVSHFLÀFFDVHV86$,'IXQGVDUHSDUWRIRYHUDOO86*RYHUQPHQWIXQGLQJIRUWKHVHSDUWQHULQVWLWXWLRQV7KHVH

funds become subject to the monitoring and evaluation requirements of the organization that receives them. For example, the *OREDO)XQG has a 

SHUIRUPDQFHEDVHGIXQGLQJV\VWHPZKLFKEDVHVIXQGLQJGHFLVLRQVRQDWUDQVSDUHQWDVVHVVPHQWRIUHVXOWVDJDLQVWWLPHERXQGWDUJHWV86$,'·V$'6

FKDSWHUSURYLGHVPRUHLQIRUPDWLRQRQKRZ3,2VDUHGHÀQHGDQGLQFOXGHVJXLGDQFHUHODWHGWRGXHGLOLJHQFHUHTXLUHGSULRUWRDZDUGLQJJUDQWVWR

3,2V

FEDERAL INVEST IN WHAT WORKS INDEX (2016)



The Promise of Evidence-Based Policymaking 1075
EVIDENCE / EVALUATION CRITERIA

9. Use of Evidence in 5 Largest Non-Competitive 

Grant Programs: Did the agency use evidence of 

effectiveness when allocating funds from its 5 largest 

non-competitive grant programs in FY16?

(Note: Meeting this criteria requires both Agency and 

Congressional action.)

USHUD:

• +8'·VbudgetFRQWDLQVODUJHIRUPXODJUDQWSURJUDPVIRUSXEOLFKRXVLQJDXWKRULWLHV3+$VWKH3XEOLF+RXVLQJ2SHUDWLQJ)XQGELOOLRQLQ

)<WKH3XEOLF+RXVLQJ&DSLWDO*UDQWVELOOLRQLQ)<DQG+RXVLQJ&KRLFH9RXFKHU+&9$GPLQLVWUDWLYH)HHVELOOLRQLQ)<

$QRWKHUIRUPXODJUDQWSURJUDPVVHUYHFLWLHVRUWULEHV&RPPXQLW\'HYHORSPHQW%ORFN*UDQW(QWLWOHPHQW1RQ(QWLWOHPHQWELOOLRQLQ)<

+20(,QYHVWPHQW3DUWQHUVKLSVELOOLRQLQ)<DQG1DWLYH$PHULFDQ+RXVLQJ%ORFN*UDQWVELOOLRQLQ)<

• $OWKRXJKWKHIXQGLQJIRUPXODVDUHSUHVFULEHGLQVWDWXWHHYDOXDWLRQEDVHGHYLGHQFHLVFHQWUDOWRHDFKSURJUDP+8'LVXVLQJHYLGHQFHIURPD

$GPLQLVWUDWLYH)HHVWXG\RIWKHFRVWVWKDWKLJKSHUIRUPLQJ3+$VLQFXULQDGPLQLVWHULQJD+&9SURJUDPWRSURSRVHDQHZ)<DSSURDFKIRUIXQGLQJ

$GPLQLVWUDWLYH)HHVZKLOHVWUHQJWKHQLQJ3+$LQFHQWLYHVWRLPSURYH+&9RXWFRPHVE\SURYLGLQJWHQDQWPRELOLW\FRXQVHOLQJ

• HUD is also conducting a 5HQW5HIRUP demonstration and a Moving To Work07:GHPRQVWUDWLRQWRWHVWHIÀFLHQFLHVRIFKDQJLQJUHQWUXOHV

• HUD also is conducting an extensive assessmentRI1DWLYH$PHULFDQ$ODVND1DWLYHDQG1DWLYH+DZDLLDQKRXVLQJQHHGVWRVWUHQJWKHQWKHHYLGHQFH

base for the formula programs.

86'2/:

• ,Q)<WKHODUJHVWQRQFRPSHWLWLYHJUDQWSURJUDPVDW'2/DUHLQWKH(PSOR\PHQWDQG7UDLQLQJ$GPLQLVWUDWLRQDOORIZKLFKDOORFDWHIXQGLQJE\

VWDWXWHDQGDOOLQFOXGHSHUIRUPDQFHPHWULFVHJXQHPSOR\PHQWLQVXUDQFHSD\PHQWLQWHJULW\:,2$FRPPRQPHDVXUHVWUDFNHGTXDUWHUO\WKH

8QHPSOR\PHQW,QVXUDQFH6WDWHJUDQWVELOOLRQLQ)<WKH(PSOR\PHQW6HFXULW\SURJUDPVWDWHJUDQWVPLOOLRQLQ)<DQG

WKUHHDXWKRUL]HGSURJUDPVXQGHUWKH:RUNIRUFH,QQRYDWLRQDQG2SSRUWXQLW\$FW:,2$7KH:,2$DXWKRUL]HGJUDQWVDUH<RXWK:RUNIRUFH

,QYHVWPHQWSURJUDPPLOOLRQLQ)<$GXOW(PSOR\PHQWDQG7UDLQLQJSURJUDPPLOOLRQLQ)<DQG'LVORFDWHG:RUNHUV

(PSOR\PHQWDQG7UDLQLQJSURJUDPELOOLRQLQ)<

• :,2$LQFOXGHVHYLGHQFHDQGSHUIRUPDQFHSURYLVLRQVEHJLQQLQJLQ3URJUDP<HDUZKLFKLQFUHDVHWKHDPRXQWRI:,2$IXQGVVWDWHVFDQ

VHWDVLGHDQGGLVWULEXWHGLUHFWO\IURPWRDQGDXWKRUL]HWKHPWRLQYHVWWKHVHIXQGVLQ3D\IRU3HUIRUPDQFHLQLWLDWLYHVDXWKRUL]HVWDWHV

WRLQYHVWWKHLURZQZRUNIRUFHGHYHORSPHQWIXQGVDVZHOODVQRQIHGHUDOUHVRXUFHVLQ3D\IRU3HUIRUPDQFHLQLWLDWLYHVDXWKRUL]HORFDOZRUNIRUFH

LQYHVWPHQWERDUGVWRLQYHVWXSWRRIWKHLU:,2$IXQGVLQ3D\IRU3HUIRUPDQFHLQLWLDWLYHVDQGDXWKRUL]H6WDWHVDQGORFDOZRUNIRUFHLQYHVWPHQW

ERDUGVWRDZDUG3D\IRU3HUIRUPDQFHFRQWUDFWVWRLQWHUPHGLDULHVFRPPXQLW\EDVHGRUJDQL]DWLRQVDQGFRPPXQLW\FROOHJHV

FEDERAL INVEST IN WHAT WORKS INDEX (2016)



1076 The Promise of Evidence-Based Policymaking
EVIDENCE / EVALUATION CRITERIA

10. Repurpose for Results: In FY16, did the agency 

shift funds away from any practice, policy, or program 

which consistently failed to achieve desired outcomes?

(Note: Meeting this criteria requires both Agency and 

Congressional action.)

$&):

• ,Q)<$&)HVWDEOLVKHGWKH+HDG6WDUW'HVLJQDWLRQ5HQHZDO6\VWHP requiring Head Start ($9.2 billion in FY16) grantees to compete for grants 

PRYLQJIRUZDUGLIWKH\IDLOHGWRPHHWFULWHULDUHODWHGWRVHUYLFHTXDOLW\OLFHQVLQJDQGRSHUDWLRQVDQGÀVFDODQGLQWHUQDOFRQWUROV7KH+HDG6WDUW

5HDXWKRUL]DWLRQ$FWPDGHDOO+HDG6WDUWJUDQWVUHQHZDEOHÀYH\HDUJUDQWV$WWKHHQGRIHDFKÀYH\HDUWHUPJUDQWHHVWKDWDUHUXQQLQJKLJKTXDOLW\

SURJUDPVZLOOKDYHWKHLUJUDQWVUHQHZHG%XWJUDQWHHVWKDWIDOOVKRUWRIVWDQGDUGVDUHQRZUHTXLUHGWRFRPSHWHWRUHQHZJUDQWV*UDQWHHVZKRVH

UDWLQJVRQDQ\RIWKHWKUHHGRPDLQVRIWKH&ODVVURRP$VVHVVPHQW6FRULQJ6\VWHPDQDVVHVVPHQWRIDGXOWFKLOGLQWHUDFWLRQVOLQNHGWRLPSURYHG

outcomes, fall below a certain threshold, or in the lowest 10 percent of grantees, must also compete. 

• $&)·V)< budget requestSSURSRVHVWRHOLPLQDWHIXQGLQJIRU$EVWLQHQFH(GXFDWLRQJUDQWVEHFDXVHWKHSURJUDPLVQRWIRFXVHGRQIXQGLQJ

evidence-based models.

CNCS:

• ,Q)<)<0LOH+LJK8QLWHG:D\DJUDQWHHRIWKH6RFLDO,QQRYDWLRQ)XQG6,)HQGHGIXQGLQJUHODWLRQVKLSVZLWKRILWVVXEJUDQWHHVZKR

were not able to conduct rigorous evaluations of their activities. In FY15, United Way for Southeastern Michigan, also a SIF grantee, ended its 

funding relationship with one of its sub-grantees for the same reason. These actions are consistent with the 6,)1DWLRQDO$VVHVVPHQWÀQGLQJVZKLFK

recognize the role SIF has played in fostering evidence-based grant making among its grantees.

MCC:

• MCC has established a 3ROLF\RQ6XVSHQVLRQDQG7HUPLQDWLRQ that describes the process and procedures for suspension and termination of MCC 

assistance in cases in which partner countries are not living up to their commitments. MCC has suspended or terminated a compact partnership, in 

SDUWRULQIXOOVHYHQWLPHVRXWRIFRPSDFWVDSSURYHGWRGDWHDQGKDVVXVSHQGHGSDUWQHUFRXQWU\HOLJLELOLW\WRGHYHORSDFRPSDFWDQDGGLWLRQDOIRXU

times (most recently with the suspension of Tanzania in March 2016). In 2012 MCC suspended 0DODZL·V&RPSDFW due to a pattern of actions by the 

*RYHUQPHQWRI0DODZLWKDWZDVLQFRQVLVWHQWZLWKWKHGHPRFUDWLFJRYHUQDQFHHYLGHQFHFULWHULDWKDW0&&XVHVIRUVHOHFWLRQ+RZHYHUWKH*RYHUQPHQW

of Malawi took a number of decisive steps to improve the human rights environment and to ensure that laws and institutions support democratic rights 

DQGSURFHVVHV7KHVHVWHSVDQGWKHUHVXPSWLRQRIVRXQGHFRQRPLFSROLF\OHGWRWKHUHLQVWDWHPHQWRI0DODZL·V&RPSDFWLQ

• 0&&DOVRFRQVLVWHQWO\PRQLWRUVWKHSURJUHVVRI&RPSDFWSURJUDPVDQGPDNHVFKDQJHVDVQHFHVVDU\)RUH[DPSOHDQDFWLYLW\LQWKH3KLOLSSLQHVWKH

(OHFWURQLF7D[,QIRUPDWLRQ6\VWHPH7,6DQDFWLYLW\XQGHUWKH5HYHQXH$GPLQLVWUDWLRQ5HIRUP3URMHFWZDVreduced in scope in FY15 due to time 

DQGFRPSOHWLRQULVNV7KLVSURDFWLYHDSSURDFKDOORZHG0&&WRMXGLFLRXVO\UHDOORFDWHIXQGVWRÀQDQFHDGGLWLRQDOVXESURMHFWVXQGHUWKH.DODKL&,'66

&RPPXQLW\'ULYHQ'HYHORSPHQW3URMHFW.&DQGIXUWKHUPD[LPL]HWKHSURMHFW·VEHQHÀWV

FEDERAL INVEST IN WHAT WORKS INDEX (2016)



The Promise of Evidence-Based Policymaking 1077
EVIDENCE / EVALUATION CRITERIA

10. Repurpose for Results: In FY16, did the agency 

shift funds away from any practice, policy, or program 

which consistently failed to achieve desired outcomes?

(Note: Meeting this criteria requires both Agency and 

Congressional action.)

86$,':

• 86$,'XVHVULJRURXVHYDOXDWLRQVWRPD[LPL]HLWVLQYHVWPHQWV$recent independent studyIRXQGWKDWSHUFHQWRI86$,'HYDOXDWLRQVKDYHEHHQ

XVHGWRPRGLI\DQGRUGHVLJQ86$,'SURMHFWV%HORZDUHDIHZH[DPSOHVZKHUH86$,'KDVVKLIWHGIXQGVDQGRUSURJUDPPLQJGHFLVLRQVEDVHGRQ

performance:

• Mozambique: Many donors working in the education sector in Mozambique were using traditional reading programs to improve early grade literacy. 

86$,'UHFHQWO\GHVLJQHGDQLPSDFWHYDOXDWLRQWRWHVWZKHWKHUUHDGLQJLQWHUYHQWLRQVDORQHRUUHDGLQJLQWHUYHQWLRQVSDLUHGZLWKVFKRROPDQDJHPHQW

support led to improved reading outcomes. Findings from a mid-term impact evaluation found that pairing reading instruction interventions with school 

PDQDJHPHQWVXSSRUWLPSURYHGUHDGLQJRXWFRPHVPRUHWKDQUHDGLQJLQVWUXFWLRQDORQHDQGZDVPRUHFRVWHIIHFWLYH%DVHGRQWKHVHÀQGLQJV86$,'

0R]DPELTXHFKDQJHGWKHZD\LWZRUNHGLQ0R]DPELTXHDQGWKHÀQGLQJVSURPSWHGWKH*RYHUQPHQWRI0R]DPELTXHWRUHTXHVWWKDWWKLVDSSURDFKEH

scaled from 120 schools to 1,060 new schools. More information can be found in the UHFHQWO\SXEOLVKHGUHSRUWRQ86$,'HYDOXDWLRQSUDFWLFH.

• Armenia$PLGWHUPevaluationRI86$,'$UPHQLD·VÁDJVKLSKHDOWKSURJUDPUHYHDOHGDQXPEHURIVLJQLÀFDQWGHVLJQDQGLPSOHPHQWDWLRQÁDZV

ZKLFKSURPSWHGWKH0LVVLRQWRWHUPLQDWHWKHSURJUDPHDUO\DQGVDYHG86*UHVRXUFHV6LQFHWKHQ86$,'$UPHQLDKDVUHGHVLJQHGLWVKHDOWKSRUWIROLR

to focus on areas where it can make difference and leave a positive legacy, as it phases out from the sector.

• Latin America and Caribbean Bureau86$,'·V/DWLQ$PHULFDDQG&DULEEHDQ/$&%XUHDXQRORQJHUIXQGVH[SHQVLYHRXWRIFRXQWU\VFKRODUVKLS

SURJUDPVVXFKDVWKH6FKRODUVKLSIRU(GXFDWLRQDQG(FRQRPLF'HYHORSPHQW6(('DQGRWKHUSUHFXUVRUSURJUDPV$evaluationRIWKH/DWLQ

$PHULFDDQG&DULEEHDQUHJLRQ·V+LJKHU(GXFDWLRQ6FKRODUVKLSV3URJUDPORRNHGDWWKHFRVWHIIHFWLYHQHVVRISURYLGLQJVWXGHQWVZLWKVFKRODUVKLSVWR

VWXG\DW86LQVWLWXWLRQVDQGGHWHUPLQHGWKDW86$,'FRXOGSURYLGHLPSURYHGWUDLQLQJRSSRUWXQLWLHVIRUPDQ\PRUHSRRU\RXWKE\IRFXVLQJUHVRXUFHV

RQLPSURYLQJWKHTXDOLW\RI/$&UHJLRQDORULQFRXQWU\WUDLQLQJLQVWLWXWLRQV7KLVÀQGLQJLQIRUPHGDUHGHVLJQRIWKHSURJUDPDQGWKHLVVXLQJRIDQHZ

5HTXHVWIRU$SSOLFDWLRQV5)$

• Indonesia,QD86$,',QGRQHVLDFKDQJHGWKHJHRJUDSKLFWDUJHWLQJRIDIRUHVWU\SURJUDPEDVHGRQD86$,'FRPPLVVLRQHGHYDOXDWLRQWKDW

found that the program was spread out among too many geographic locations and could be more effective by focusing on fewer locations. This 

example can be found in the recently published independent VWXG\RQHYDOXDWLRQXVHDW86$,'. 

• ,QD86$,'FRPPLVVLRQHGHYDOXDWLRQRID86$,'$IJKDQLVWDQURDGUHKDELOLWDWLRQSURJUDPIRXQGWKDWFRRSHUDWLYHDJUHHPHQWVDQGJUDQWV

DUHQRWDVHIIHFWLYHLPSOHPHQWLQJPHFKDQLVPVDVFRQWUDFWVLQWHUPVRIWKHOHYHOVRILPSOHPHQWLQJSDUWQHUDFFRXQWDELOLW\WR86$,'SDUWLFXODUO\LQ

UHJDUGVWRLQIUDVWUXFWXUHDFWLYLWLHV,QSDUWDVDUHVXOWRIWKLVHYDOXDWLRQ·VÀQGLQJVLQ86$,'UHOHDVHGDQHZRSHUDWLQJSROLF\HQWLWOHG ́86$,'

,PSOHPHQWDWLRQRI&RQVWUXFWLRQ$FWLYLWLHV,” that mandates the use of contracts rather than grant or cooperative agreement mechanisms for projects 

that involve construction.

86(':

• 6LQFH('KDVZRUNHGZLWK&RQJUHVVWRHOLPLQDWHSURJUDPVVDYLQJPRUHWKDQELOOLRQLQFOXGLQJSURJUDPVOLNH(YHQ6WDUWVHHSS$

WR$PLOOLRQLQ)<DQG0HQWRULQJ*UDQWVVHHS*PLOOLRQLQ)<ZKLFKWKH'HSDUWPHQWUHFRPPHQGHGHOLPLQDWLQJRXWRI

concern based on evidence. 

• ('DOVRWULHVWRVKLIWSURJUDPIXQGVWRVXSSRUWPRUHHIIHFWLYHSUDFWLFHVE\SULRULWL]LQJWKHXVHRIHQWU\HYLGHQFH)RU('·VJUDQWFRPSHWLWLRQVZKHUH

WKHUHLVHYDOXDWLYHGDWDDERXWFXUUHQWRUSDVWJUDQWHHVRUZKHUHQHZHYLGHQFHKDVHPHUJHGLQGHSHQGHQWRIJUDQWHHDFWLYLWLHV('W\SLFDOO\UHYLHZV

such data to shape the grant competition design of future projects. For example, an impact evaluation of the Teacher Incentive Fund (TIF) will inform 

('·V)<FRPSHWLWLRQGHVLJQIRU7,)LQFOXGLQJIRFXVLQJDSSOLFDQWV·DWWHQWLRQRQSUDFWLFHVPRUHOLNHO\WREHHIIHFWLYH

• $GGLWLRQDOO\('XVHVHYLGHQFHLQFRPSHWLWLYHSURJUDPVWRHQFRXUDJHWKHÀHOGWRVKLIWDZD\IURPOHVVHIIHFWLYHSUDFWLFHVDQGWRZDUGPRUHHIIHFWLYH

SUDFWLFHV)RUH[DPSOH(66$·V(GXFDWLRQ,QQRYDWLRQDQG5HVHDUFK(,5SURJUDPWKHVXFFHVVRUWRLVXSSRUWVWKHFUHDWLRQGHYHORSPHQW

LPSOHPHQWDWLRQUHSOLFDWLRQDQGVFDOLQJXSRIHYLGHQFHEDVHGÀHOGLQLWLDWHGLQQRYDWLRQVGHVLJQHGWRLPSURYHVWXGHQWDFKLHYHPHQWDQGDWWDLQPHQWIRU

high-need students.

FEDERAL INVEST IN WHAT WORKS INDEX (2016)



1078 The Promise of Evidence-Based Policymaking
EVIDENCE / EVALUATION CRITERIA

10. Repurpose for Results: In FY16, did the agency 

shift funds away from any practice, policy, or program 

which consistently failed to achieve desired outcomes?

(Note: Meeting this criteria requires both Agency and 

Congressional action.)

USHUD:

• +8'·V)<budget request includes a new formula for funding +RXVLQJ&KRLFH9RXFKHU$GPLQLVWUDWLYH)HHV that shifts funding away from 

inappropriately compensated public housing agencies and increases overall funding according to evidence about actual costs of maintaining a highperforming voucher program. (See here for more info.) 

• 7KH$GPLQLVWUDWLRQ·V)<UHTXHVWUHFRPPHQGVVKLIWLQJVXSSRUWIURPKRPHOHVVLQWHUYHQWLRQVVKRZQWRKDYHOLPLWHGHIIHFWLYHQHVVWRZDUGKRXVLQJ

vouchers that were proven effective in the Family Options study.

86'2/:

• '2/·VHYLGHQFHEDVHGVWUDWHJ\LVIRFXVHGRQSURJUDPSHUIRUPDQFHLPSURYHPHQWDQGH[SDQVLRQRIVWUDWHJLHVDQGSURJUDPVRQZKLFKWKHUHLV

evidence of positive impact from rigorous evaluations. The department takes all action possible to improve performance before considering funding 

UHGXFWLRQVRUSURJUDPWHUPLQDWLRQ+RZHYHU'2/GRHVXVHSURJUDPSHUIRUPDQFHPHDVXUHVWRPDNHGHFLVLRQVDERXWIXWXUHIXQGLQJ)RUH[DPSOH

WKHUHLVFXUUHQWO\DSURSRVDOWRFORVHD-RE&RUSV&HQWHUEHFDXVHRILWVFKURQLFORZSHUIRUPDQFH&ORVXUHRIWKLVFHQWHUZLOODOORZ'2/WRVKLIWOLPLWHG

program dollars to centers that will better serve students by providing the training and credentials they need to achieve positive employment and 

educational outcomes. In a )HGHUDO5HJLVWHUQRWLFHSXEOLVKHGLQ0DUFK'2/UHTXHVWHGSXEOLFFRPPHQWVRQWKLVSURSRVDO$GGLWLRQDOO\DOO

discretionary grant performance is closely monitored and has been used to take corrective action and make decisions about continued funding.

FEDERAL INVEST IN WHAT WORKS INDEX (2016)



The Promise of Evidence-Based Policymaking 1079
About the Results for America Federal Invest in What Works Index

5HVXOWVIRU$PHULFD·V)HGHUDO,QYHVWLQ:KDW:RUNV,QGH[KLJKOLJKWVWKHH[WHQWWRZKLFKWKH$GPLQLVWUDWLRQIRU&KLOGUHQDQG)DPLOLHVZLWKLQ++6&RUSRUDWLRQIRU1DWLRQDODQG&RPPXQLW\6HUYLFH

0LOOHQQLXP&KDOOHQJH&RUSRUDWLRQ86$JHQF\IRU,QWHUQDWLRQDO'HYHORSPHQW86'HSDUWPHQWRI(GXFDWLRQ86'HSDUWPHQWRI+RXVLQJDQG8UEDQ'HYHORSPHQWDQG86'HSDUWPHQWRI/DERUDUHFXUUHQWO\

building the infrastructure necessary to be able to use data, evidence and evaluation in budget, policy, and management decisions. It is important to note that: 

 5HVXOWVIRU$PHULFDGHYHORSHGWKHFULWHULDDQGVFRULQJVWUXFWXUHLQWKHDWWDFKHGLQGH[LQFORVHFRQVXOWDWLRQZLWKPRUHWKDQFXUUHQWDQGIRUPHU)HGHUDOJRYHUQPHQWRIÀFLDOVDQGNH\VWDNHKROGHUVIURPDOO

across the country.

 7KHSXUSRVHRIWKHDWWDFKHGLQGH[LVWRHGXFDWHPHPEHUVRIWKHJHQHUDOSXEOLFDVZHOODVSXEOLFSULYDWHDQGQRQSURÀWVHFWRUOHDGHUVRQKRZIHGHUDOGHSDUWPHQWVDQGDJHQFLHVDUHFXUUHQWO\XVLQJGDWD

evidence and evaluation to invest taxpayer dollars in what works.

 5HVXOWVIRU$PHULFDJDYHWKHIHGHUDOGHSDUWPHQWVDQGDJHQFLHVLQFOXGHGLQWKHDWWDFKHGLQGH[PXOWLSOHRSSRUWXQLWLHVWRUHYLHZDQGFRPPHQWRQWKHFRQWHQWDQGSUHVHQWDWLRQRIWKHLQIRUPDWLRQLQFOXGHG

LQLW:HJUHDWO\DSSUHFLDWHWKHLUZLOOLQJQHVVWRKHOSXVGHYHORSWKLVGRFXPHQWDQGWKHLUFRQWLQXHGFRPPLWPHQWWRPDNLQJWKHIHGHUDOJRYHUQPHQWDVHIIHFWLYHDQGHIÀFLHQWDVSRVVLEOH6LQFHZHUHFRJQL]H

WKDWLWLVYHU\GLIÀFXOWWRGLVWLOOFRPSOH[SUDFWLFHVSROLFLHVDQGSURJUDPVLQWRDVLQJOHFURVVDJHQF\VFRUHFDUGZHH[HUFLVHGRXUEHVWMXGJPHQWDQGUHOLHGRQWKHGHHSH[SHUWLVHRIOHDGHUVERWKZLWKLQDQG

outside of the federal government during the development of the attached index. 

 5HVXOWVIRU$PHULFDUHOHDVHGIRXUSUHYLRXVYHUVLRQVRIWKLV,QYHVWLQ:KDW:RUNV,QGH[LQ-XQH, 6HSWHPEHU, May 2014 and March 2015. 

Scoring

7KHDWWDFKHGLQGH[DVVHVVHVVHYHQIHGHUDOGHSDUWPHQWVDQGDJHQFLHVDJDLQVWGDWDHYLGHQFHDQGHYDOXDWLRQFULWHULD(DFKFULWHULDZDVHTXDOO\ZHLJKWHGDQGVFRUHGRQDVFDOHRIUHVXOWLQJLQDWRWDO

SRVVLEOHVFRUHRISRLQWV)HGHUDOGHSDUWPHQWVDQGDJHQFLHVZHUHJLYHQSRLQWVLIWKH\KDYHGHPRQVWUDWHGDQLQWHQWWRPHHWWKHVWDWHGFULWHULDSRLQWVLIWKH\KDYHGHPRQVWUDWHGVRPHLQLWLDOLQWHUQDO

progress toward meeting the criteria; 6-7 points if they have made some initial public progress toward meeting the criteria; 8-9 points if they have made some meaningful public progress toward meeting the criteria; 

and 10 points if they have fully and successfully met the criteria.These scores are based on the information and links provided by these seven departments and agencies.

About Results for America

5HVXOWVIRU$PHULFDLVLPSURYLQJRXWFRPHVIRU\RXQJSHRSOHWKHLUIDPLOLHVDQGFRPPXQLWLHVE\VKLIWLQJSXEOLFUHVRXUFHVWRZDUGHYLGHQFHEDVHGUHVXOWVGULYHQVROXWLRQV5)$LVZRUNLQJWRDFKLHYHWKLVJRDOE\

EXLOGLQJDVWURQJELSDUWLVDQ ́,QYHVWLQ:KDW:RUNVμFRDOLWLRQGHYHORSLQJDQGDGYDQFLQJWKHQH[WJHQHUDWLRQRIHYLGHQFHEDVHGUHVXOWVGULYHQSUDFWLFHVSROLFLHVDQGSURJUDPVDQGVXSSRUWLQJOHDGHUVDWDOOOHYHOV

of government to invest in what works.

,Q1RYHPEHU5HVXOWVIRU$PHULFDSXEOLVKHGMoneyball for Government, the national bestselling book that brings together a group of bipartisan leaders and makes the case for government at all levels to 

LQIRUPSXEOLFSROLF\DQGIXQGLQJGHFLVLRQVE\XVLQJWKHEHVWSRVVLEOHGDWDHYLGHQFHDQGHYDOXDWLRQDERXWZKDWZRUNV,Q-DQXDU\5)$SXEOLVKHGWKHVHFRQGHGLWLRQRI0RQH\EDOOIRU*RYHUQPHQWIHDWXULQJD

QHZELSDUWLVDQFKDSWHUDERXWWKHLPSRUWDQFHRIXVLQJGDWDDQGHYLGHQFHWRGULYH86IRUHLJQDVVLVWDQFH)RUPRUHLQIRUPDWLRQDERXW5HVXOWVIRU$PHULFDYLVLWwww.results4america.org. 

FEDERAL INVEST IN WHAT WORKS INDEX (2016)

1080 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 1081
1082 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 1083
1084 The Promise of Evidence-Based Policymaking
An Automated Evidence-Based-Policy Clearinghouse for Researchers, Practitioners, Federal 

Agencies, and Policymakers: A Proposal to the New Evidence-Based Policymaking Commission 

We are aware of your new Evidence-based Policymaking Commission, recently created by Congress 

and signed into law by President Obama. The bipartisan members who conceived of the need for this 

Commission are to be congratulated—a recognition of the need to infuse scientific evidence into the 

decisions of policy-makers is the first step to effectively designing policies that improve our lives while 

not wasting tax-payer money on unproven strategies. 

Our understanding is that Commissioners have been charged with three general tasks: (1) to improve 

the federal data infrastructure while respecting privacy and security concerns; (2) to incorporate 

outcomes measurement, cost-benefit data, evaluation, randomized controlled trials (RCTs), and 

rigorous impact analysis into federal program design; and (3) to consider the value and nature of a 

clearinghouse that would facilitate access to data by various constituencies and enable the research 

community to judge what works and what does not. 

The Commission will focus on ways to incentivize the rigorous evaluation of programs and policies that 

aim to reduce the problems associated with detrimental prevailing conditions and promote more 

healthful and productive outcomes. Until now, many programs we invest in do not possess stringent 

indicators of their effectiveness and, thus, there is no justification for their continuation. 

There is a wealth of data already collected by the federal government and other agencies and 

organizations reflective of a broad range of phenomena, from physical health to juvenile and criminal 

justice to climate change. Existing data reserves are currently not well organized and thus an 

infrastructure is needed to increase the utilization of these data. 

To facilitate the process of organizing and fully utilizing the data, we recommend a means to directly 

and expeditiously improve policy decisions. Our proposal is highly compatible with the law by 

incorporating federal agency and other data, as well as methodological components that will be readily 

accessible and understandable to those who stand to benefit. And we believe there will be widespread 

support from Congress, the White House and a number of organizations which have an interest in 

evidence-based policy-making. 

We propose that the federal government (and expert contractors) develop an automated 

clearinghouse—perhaps called the “National Evidence-Based Toolkit for Intuitive Navigation” (NETIN)—

that will provide comprehensive information regarding evidence-based programs and policies (EBPs) to 

users; e.g., researchers (who can populate the database), policy-makers (who need to know what to 

legislate and fund), and community organizations, practitioners and government agencies (that need to 

identify best practices). The data populating this toolkit will provide parameters needed to readily map 

available EBPs to existing needs, whether that be to identify best violence prevention practices for any 

given community or to determine which policies to fund to reduce poverty. Also needed is flexibility to 

include innovative and/or promising programs that have yet to be subjected to rigorous evaluation but 

are in the database denoted by their stage of development and need for further study (as per the #2 

mandate above). 

Parameters will be intuitively searchable and fields will be delineated by relevant characteristics; e.g., 

outcome of interest (e.g., diabetes, violence, contaminated water); setting (e.g., school, family, 

community, national); target population (e.g., special needs children, parents, community 

stakeholders, minorities); implementation protocols and frameworks (costs, timeline expectations to 

achieve impact, strategies to shift resources from existing to promising or evidence-supported 

approaches); pertinent literature and resources on assessing and utilizing research; cost-benefit 

analyses; and other information deemed helpful. The goal is to provide a comprehensive, one-stop 

resource that is more user-friendly and searchable on dimensions that are not currently available, 

providing an efficient and valid method to guide evidence-based policy-makers and others who might 

benefit from the resource. 

)@@

0DUPCFS
)FBSJOH

/BUJPOBM1SFWFOUJPO4DJFODF$PBMJUJPOUP*NQSPWF-JWFT

%JBOF'JTICFJO

The Promise of Evidence-Based Policymaking 1085
The system would be both iterative and interactive; e.g., a search for a category of programs may 

elicit a notation about the need for extra diligence or a particular protocol for implementation. Or 

reference materials may be recommended if using certain interventions. At all stages of navigation, 

weblinks would lead to relevant information. 

Finally, the Clearinghouse would provide a searchable methodology section for researchers who want 

to fill in gaps in the Clearinghouse database. There would be guidance on design, methods, statistical 

techniques, evaluation protocols, and strategies for translation. 

We realize this will be a very large and complex undertaking that will take years to complete and will 

require continual updating. There will also be a need to establish criteria and thresholds for 

designating programs and policies as evidence-based, not only relative to the statistical findings from 

RCTs, but the population significance of those results (e.g., how broadly are effects achieved?). 

Fortunately, there are a number of existing registries that evaluate programs; they can be utilized and 

integrated as best seen fit. The Commission and their advisors will also want to make decisions about 

what policy areas to cover (from human behavior and health to security, the economy, and the 

environment). These objectives for a clearinghouse can be accomplished with sufficient funding and 

commitment, as well as by calling upon the expertise of evidence-based policy-making organizations, 

academics, researchers, current registry experts, federal government database keepers, 

implementation scientists, methodologists, computer scientists, and statisticians. And critical to this 

effort, to ensure its usability and utility, input must be sought from all potential users (e.g., 

community groups, policy-makers, agencies, foundations) working in concert with experts. 

This proposal is reflective of what policy-makers, practitioners, stakeholders and others need to make 

informed, adequately justified, and effective decisions when identifying programs and policies that will 

serve communities and the nation. We have outlined a general roadmap for the creation of a 

clearinghouse—the Commission’s 3rd consideration—with details to be fleshed out after thorough 

discussion and consultation. Our hope is that the Commission will include such a plan that will bring to 

fruition their charge to design a data infrastructure and incorporate results from existing and newly 

conducted studies. There is potential to greatly improve the operations of government, the services 

provided to citizens, and their financial impact. 

Diana H. Fishbein PhD is CoͲDirector of the National Prevention Science Coalition to Improve Lives. She is

the C. Eugene Bennett Chair in Prevention Research at the Edna Bennett Pierce Prevention Research

Center at The Pennsylvania State University in State College, Pa.

Neil Wollman PhD is CoͲDirector of the National Prevention Science Coalition to Improve Lives and Senior

Fellow at Bentley ServiceͲLearning Center, Bentley University in Waltham, Mass. Nwollman@bentley.edu

260Ͳ568Ͳ0116

1086 The Promise of Evidence-Based Policymaking
Page 1 of 7

Statement of

George F. Grob 

Chair, Evaluation Policy Task Force 

American Evaluation Association 

For the Commission on Evidence-Based Policymaking 

The Value and Methods of Evaluating

Government Programs Throughout Their Life Cycle 

Rayburn House Office Building 

Washington, D.C. 

October 21, 2016 

)@@

0DUPCFS
)FBSJOH

"NFSJDBO&WBMVBUJPO"TTPDJBUJPO

1SFTFOUFS(FPSHF(SPC

The Promise of Evidence-Based Policymaking 1087
Page 2 of 7

American Evaluation Statement 

for the 

Commission on Evidence-Based Policymaking 

Thank you for the opportunity to discuss the important topic of evaluation of federal programs. I 

am a professional independent evaluator, formerly Director of Evaluation at the Office of 

Inspector General, Department of Health and Human Services and the Federal Housing Finance 

Agency. I am writing as Chair of the Evaluation Policy Task Force of the American Evaluation 

Association (AEA), the professional organization devoted to the application and exploration of 

evaluation in all its forms since 1986. 

AEA has approximately 7,000 members across all 50 states, as well as 80 other countries. 

Members have gathered together from many interdisciplinary fields (such as public policy and 

administration, political science, economics, statistics, psychology, sociology, education, public 

health, demography, ethnology, etc.) to create a community of learning and practice over the past 

three decades. Members in academe have worked to develop, refine, and teach evaluation 

methods, while members in practice have served the evaluation needs of many organizations 

including agencies across the federal government. Members serve in many federal evaluation, 

policy, and inspector general offices, and at the Office of Management and Budget. 

AEA has developed professional standards for the quality of studies and ethics for the 

multidisciplinary members of the field. Of particular interest today is AEA’s paper: An 

Evaluation Roadmap for a More Effective Government. This document describes many types 

of evaluation that can address management requirements, as well as principles and practices for 

ensuring evaluation quality and usefulness, including methods, human resources, budgets, 

independence, transparency, and professional ethics in a government setting. For your 

convenience, I have attached a copy for your reference. 

I focus here on three main topics: 1) the importance of evidence and the availability of data for 

government decision makers; 2) evaluation methods; and 3) evaluation in government settings. 

Evidence and Data for Decision Makers 

Government decision makers, including both the Congress and Executive Branch agencies, need 

appropriate evidence to make informed decisions to assess and improve the relevance, efficiency, 

and effectiveness of government programs, policies, and activities (hereafter “programs”). AEA 

applauds the work of the Commission to help Congress embed evaluation into program design 

and to ensure that quality data are available for evaluation.

Federal program design should include an appropriate evaluation framework to guide data 

collection and use over the life of a program. This includes data needed for rigorous impact 

evaluations as appropriate. Measures of a program's key processes and outcomes should be 

established while the program is being conceived and developed. In fact, taking time during the 

process of conceptualizing a new program to specifically define expected outcomes is most 

useful in establishing relevant metrics. Preliminary metrics should be put into place by the time 

program implementation begins, thus allowing key data to be collected to monitor program 

1088 The Promise of Evidence-Based Policymaking
Page 3 of 7

implementation, determine progress, and set the stage for methodologically rigorous studies. It 

is vital for some study methods that data be collected prior to the program intervention. 

We support the Commission in its efforts to consider whether and how a clearinghouse for 

program and survey data should be established. We encourage efforts to ensure that verifiable, 

reliable, and timely data are available to permit the objective evaluation of programs, including 

an assessment of assumptions and limitations in such evaluations. Agencies should use 

evolving best practices for data security, and ensure that publicly available data are aggregated or 

otherwise stripped of all information that could be used to identify particular individuals or 

businesses. 

The proposed clearinghouse could also serve as a repository for the evidence contained in 

evaluation reports, providing an archive capacity for the collection, dissemination, and 

preservation of knowledge and lessons learned from evaluation studies. This would provide an 

enhanced base for guiding future program design and management, which often requires a 

critical mass of knowledge to properly comprehend and address the complexity of program 

processes and influences. It would also be a great benefit for future meta-analyses of evaluation 

findings. 

While recognizing the high value and strategic importance of large-scale archives and datasets, 

the availability of these existing data should not reduce the capacity to gather targeted data as 

needed to address important program evaluation questions. 

Evaluation Methods 

The Commission is charged with making “recommendations on how best to incorporate 

outcomes measurement, institutionalize randomized controlled trials, and rigorous impact 

analysis into program design.” We believe that the key to such evaluation activities is for 

federal entities, (including Congress, as well as Executive Branch agencies and the White House 

Office of Management and Budget (OMB)), to identify the important evaluation questions that 

they need answered to effectively direct the future of Federal programs. Such questions could be 

effectively embedded in authorizing legislation or in congressional committee reports associated 

with legislative authorizations or renewals. Executive Branch agencies can do so in their budget 

documents and implementation plans. Evaluators, in consultation with other experts, can then 

identify which scientific methods are best suited to answer those questions. Specifically defining 

program activities and expected outcomes has proven very useful in choosing relevant evaluation 

questions about program operations and impact. 

Such questions, and associated evaluations, are needed throughout the life cycle of programs, 

from their initial authorization through all phases of their implementation. For example, during 

early stages in the life of programs, key questions might center on the fidelity of their 

implementation with statutory requirements and on early implementation problems and 

successes. As the program matures, decision makers might want to establish metrics to track 

such features as enrollment of intended beneficiaries or establishment of required administrative 

systems and other infrastructures. Gradually, interest may shift to outputs, in terms of benefits 

provided and beneficiaries served. Ultimately, decision makers and citizens will want 

information about the impact of programs on people’s lives, the economy, public health, safety, 

The Promise of Evidence-Based Policymaking 1089
Page 4 of 7

or other factors or goals associated with the purpose of the programs. Throughout the life of the 

programs, government officials and taxpayers will want to know if funds are being misspent or 

wasted. 

While interest will ultimately focus on program impact and value received for investments made, 

citizens and decision makers do not want to wait until a program has run its course and then 

determine whether it has been working. Along the way, they will want to know if 

implementation problems can be corrected and whether the programs can be improved. 

 

All of these questions are important. But the methods for answering them can be complex. 

Evaluation professionals have a broad range of methods—based on research— from which to 

draw on to answer both impact and operational process questions. Rather than legislating, 

requiring, or overemphasizing any single specific method for impact analysis in federal 

guidance, AEA recommends that federal policy require that careful consideration be given to a 

range of evaluation methods that may be appropriate or feasible in any given circumstance. 

 

Over the years, the evaluation field has developed an extensive array of analytic approaches and 

methods that can be applied and adapted to a wide variety of programs and circumstances, 

depending on the program’s characteristics and implementation stage, the way the results will be 

used, and the kinds of decisions that need to be made. In designing evaluation studies it is 

important to recognize that every method has pros and cons and strengths and weaknesses that 

must be addressed in matching them to answer the specific questions, circumstances, and 

intended uses of results. There are real-life factors which can render designs infeasible, 

impractical, or inappropriate. To ensure adequate deployment, every study design must examine 

and address feasibility constraints, including resources (funding and time limits), conditions in 

the field, ethical considerations, stakeholder concerns, etc. All evaluation methods should be 

context-sensitive and have cultural relevance. 

Agencies should not only focus on tools for evaluation inquiry, but foster evaluation thinking as 

well. High-stakes program decisions should be based on a preponderance of evidence developed 

using sound methods. Some programs may need a high level of credibility and precision in the 

portfolio of evidence upon which leaders base a decision. This may require multiple studies and 

methods as well as a combination of process and impact evaluations to assess and understand the 

effectiveness of an approach within the portfolio of evidence. A range of analytic methods may 

be needed, and often several methods—including quantitative and qualitative approaches—

should be used simultaneously. Multiple methods can offset the shortcomings of any one 

method with the strengths of another. 

In fact, some decisions about how to improve the reach and impact of a given program may not 

require a high level of precision or a large portfolio of evidence. Some evaluation approaches 

are particularly helpful in a program’s early developmental stages, whereas others are more 

suited to ongoing and regularly implemented programs or to ex-post analysis of temporary 

programs upon their completion. The broader policy and decision-making context also can 

influence which approach is most appropriate. Sometimes information is needed quickly, 

requiring studies that can use existing data or rapid data-collection methods; at other times, more 

sophisticated long-term studies are required to understand fully the dynamics of program 

1090 The Promise of Evidence-Based Policymaking
Page 5 of 7

administration and beneficiary behaviors. Moreover, different approaches can complement one 

another. 

The opportunity to capitalize on early successes, identify implementation impediments, or make 

mid-course corrections is critical. So it is essential to conduct ongoing formative evaluation 

throughout the program’s life cycle. For example, evaluation can address questions that arise 

during implementation of the program, such as the validity of assumptions that underlie program 

design, or challenges to implementation in the field. Early in the program’s history, relatively 

simple information may be needed quickly (e.g., regarding obstacles to participation in the 

program). Evaluators should match the methodology to the questions at each stage of program 

development and to information needs, which may call for a range of methods over time, 

including targeted data collection that may not always include outcomes measurement. 

Today we see considerable interest in impact analyses, including randomized controlled trials. 

No doubt these are valuable tools and have their place. But we wish to emphasize that they 

represent only some of the methods that can and should be applied, depending on the questions 

that need to be answered. They are not intrinsically better than other methods, except in those 

circumstances where they are most appropriate and feasible. An overarching focus on these 

methods to the exclusion of others will deprive decision makers of valuable insights about ways 

to improve program effectiveness and efficiency, and, when appropriate, whether to increase or 

diminish program funding. 

Most federal evaluations need to go beyond estimating aggregate impacts to also addressing 

"what works for whom, and under what circumstances." If the data from evaluation studies are 

to be of most use in guiding evidence-based decision making, they need to be able to support 

conclusions about how program impacts vary across subgroups of those affected by the programs 

and also conclusions about the contexts in which the specific program activities are most 

effective. 

Numerous examples are available of evaluations that have enhanced the effectiveness and impact 

of programs but that were conducted early in the program’s life. One that comes to mind is the 

evaluation conducted by the Institute of Medicine during the first five year of the PEPFAR 

program. This was done at the request of Congress, embedded in the original authorizing statute. 

It provided feedback on implementation issues that was available to decision makers at the time 

of the program’s first reauthorization. It is fair to speculate that this early feedback contributed to 

the impact of that program from that time forward, and in many ways was as or more impactful 

than studies performed in later years. 

Other studies can affect programs and their impacts when performed several years into their 

implementation. One example is a series of evaluations and audits that identified serious 

problems of service quality, cost, and fraud within Medicare’s home health program. Based on 

those studies, the Congress reformed the structure of the program, leading to savings of some 

forty plus billions of dollars and the abandonment of participation in the program by many high 

risk providers. These impacts were verified by independent reviews conducted by the 

Government Accountability Office. It is especially noteworthy that it was not a single impact 

study that led to these reforms, but rather by a body of work spanning several years. 

The Promise of Evidence-Based Policymaking 1091
Page 6 of 7

Many other evaluation studies that lead to improved program impact and efficiency are 

documented on the websites of various Federal Offices of Inspectors General. They illustrate the 

value of using a body of work and mixed methods to assess both implementation and 

effectiveness of federal programs. 

Evaluation Capacity in Government Settings 

AEA believes that all federal entities should have the ability and should be encouraged (and in 

some cases, required) to evaluate programs. However, each agency or department should 

develop structures and plans for their evaluation functions that are best suited to their missions, 

organizational structures, stakeholders, environments, timing of and need for evidence in 

decision making, and available resources. 

 

Because evaluation should serve as an essential core function in good governance, agencies 

should be required to apply the findings and conclusions of evaluations to program design, 

management, reform, expansion, or termination—ensuring that policy formulation will be more 

open, consultative, and evidence-informed. Agencies should, to the extent practical, conduct 

impact evaluations on pilot programs before attempting to expand or replicate them. 

A framework for the planning and conduct of evaluations should also include: 

a. A public evaluation policy statement 

b. A sound procedure for establishing annual and multi-year evaluation agendas and 

timetables 

c. Consultation with appropriate congressional committees, OMB, and other external 

program stakeholders on their information needs 

d. A dissemination plan, preferably with public access 

e. Resources needed to support evaluation, and 

f. Plans regarding how the findings and conclusions of evaluations shall be considered 

in subsequent program design, program management, and decisions regarding 

program reform, expansion, or termination. 

The organizational structure of evaluation efforts is also important. Thus, it is vital to ensure an 

appropriate mixture of independence and collaboration between the evaluators and program 

offices with regard to evaluation design, conduct, and reporting. Consultation is needed to 

ensure relevance, but independence is needed to ensure impartiality. Depending on the unique 

organizational structure of each agency, an independent central evaluation office could be 

responsible for: developing and promoting program evaluation expertise throughout an agency; 

planning, conducting or procuring evaluation studies; and ensuring appropriate follow-up of 

evaluation findings and recommendations. 

Adequate staffing of evaluation units and support for professional development is also necessary 

if the Commission’s work is to achieve the kind of benefits foreseen by Congress. To ensure that 

decision makers use the evidence produced in evaluation studies, agencies should invest in 

training those staff responsible for program design, implementation, and management regarding 

the proper conduct of evaluation and the use of findings in program decision making. 

1092 The Promise of Evidence-Based Policymaking
Page 7 of 7

Summary 

1. Government decision makers must have appropriate credible evidence to make informed 

decisions regarding the structure and operations of federal agencies and policies, and to 

maximize their effectiveness and efficiency. 

2. For key decision points within programs, federal entities (including Congress, executive 

branch agencies, and OMB), should identify important evaluation questions. Evaluators, 

in consultation with program officials, should select methods best suited to answer those 

questions. 

3. All federal entities should have the authority and resources to conduct evaluations, and 

should be encouraged (and in some cases, required) to evaluate various programs. 

However, each agency or department should develop structures and plans for those 

evaluation functions best suited to their mission, organizational structure, stakeholders, 

environment, and timing of and need for evidence in decision making. 

4. Recognizing the importance of assessing program effectiveness, the opportunity to 

capitalize on early successes or to make mid-course corrections is also critical. So it is 

essential also to conduct ongoing formative evaluation throughout the program’s life 

cycle. 

5. The proposed Evidence Clearinghouse should serve as a repository for the evidence 

contained in evaluation reports and as an archive for the collection, dissemination, and 

preservation of knowledge and lessons learned from evaluation studies. 

The Promise of Evidence-Based Policymaking 1093
Rachel Fishman, Senior Policy Analyst at New America 

Submission for the First Public Hearing of the Commission on Evidence-Based Policymaking (CEP)

Abstract: The US Department of Education puts out $130 Billion a year on federal financial aid to help 

students go to college, and billions more are spent by other federal agencies on higher education through 

tax credits, the GI Bill, and more. Despite having a tremendous amount of administrative data, policymakers, 

students, and families know shockingly little about how particular schools and programs are serving 

students due to a law banning the connecting of these data sets. In an era when college has never been more 

important nor more necessary, we believe this issue is one the Commission should address directly. 

Oral and Written statement: Thank you for the opportunity to speak today about the better use of existing

higher education data to support improved decision making by families and policymakers. My name is Rachel 

Fishman and I am a Senior Policy Analyst at New America in the Education Policy Program which uses original 

research and policy analysis to help solve the nation’s critical education problems. 

It’s hard to open a newspaper or turn on the television these days without finding another report of the 

questionable value of college degrees. As anxiety over student debt and college costs reaches new heights, the 

public is growing increasingly uncertain about the value of a college education. The answer to the question “Is 

college worth it” is an unequivocal “yes.” On average. But the real question is: In which program, at which 

college, at which price and for which students is it worth it? 

Students, families, and taxpayers are spending unprecedented amounts on higher education, but remain largely 

in the dark about how to spend these precious dollars. And while colleges and universities spend hundreds of 

thousands of hours collecting and reporting data, they don’t know how their students are faring compared with 

similar students at similar schools. Institutions of all types are subsidized with hundreds of billions of dollars a 

year in federal financial aid (not to mention billions more in tax credits, GI Benefits, Department of Labor funds, 

and more), but taxpayers don’t know if these dollars are being wasted at diploma mills or poor-performing 

institutions. Policymakers have no sense of whether their reforms and investments are helping or hurting the 

families that most need the boost higher education can provide. At a time when higher education has never 

been as important or as expensive, it’s unimaginable that we can’t answer these critical questions. 

Why can’t we answer them? Because the federal government either doesn’t have—or can’t use—the right 

data. That’s true, not because it is technically impossible, but because it is illegal. In 2008, Congress passed a 

law that banned the creation of a federal student unit record system to enable existing data systems to speak 

with one another and answer critical questions. 

)@@

0DUPCFS
)FBSJOH

/FX"NFSJDB

1SFTFOUFS3BDIFM'JTINBO

1094 The Promise of Evidence-Based Policymaking
2 

The current hodgepodge of data systems cannot answer basic question like: 

● How do part-time and older learners fair in the current system? 

● What happens to students who transfer from particular colleges? 

● How many—and which—students complete at particular colleges? 

● Do students who get some of the more than $30 Billion spent annually on Pell Grants 

graduate? 

● Are graduates able to find jobs that allow them to pay down their debts? 

A system that uses already-collected administrative data would allow us to answer these questions. 

Creating a Student Unit Record would not require the collection of additional student data, but would allow the 

connecting of existing data already held by a variety of federal and state agencies. Protecting these data at all 

points of the lifecycle is crucial, and it is worth considering housing such a system in the Department of 

Education’s National Center for Education Statistics, which is classified as a statistical agency and therefore 

subject to stringent privacy and security requirements under the Privacy Act of 1974, the Education Sciences 

Reform Act of 2002 (ESRA), and the E-Government Act of 2002. We can also look to state level systems for best 

practices that could be implemented at the federal level. 

We believe using existing administrative data to better understand the outcomes of students at our nation’s 

colleges is exactly the type of critical policy issue the Commission was designed to address. We know we have 

just scratched the surface here today and we will provide much more detail about the existing administrative 

data sources as well as privacy and policy considerations in separate written comments. Thank you for your time 

and attention and I look forward to answering any questions. 

1899 L Street, NW, Suite 400 • Washington, DC 20036 • Phone: 202-986-2700 • Fax: 202-986-3696 • www.newamerica.org

The Promise of Evidence-Based Policymaking 1095
1 

Commission on Evidence-Based Policymaking—Public Hearing 

Friday, October 21, 2016 

Comments submitted by: 

 Amanda Janice Roberson, Research Analyst 

Institute for Higher Education Policy 

Abstract: The current postsecondary data infrastructure is fragmented and incapable of answering a 

number of important questions about how our students fare in the higher education system. Key 

stakeholders, including policymakers, institutions, researchers, and the students themselves, need better 

information about college access, progression, completion, and post-college outcomes. Given the federal 

government’s substantial investment in postsecondary education, it is imperative that existing data – at 

the institutional, state, and federal levels – are leveraged to answer these critical questions. By fostering 

these data linkages and removing existing legal barriers, the Commission can create a system where data 

drive efforts to increase postsecondary success and close equity gaps. 

Oral and Written Statement: Chairman Abraham, Co-Chair Haskins, and commissioners: thank you for the 

opportunity to address the Commission on Evidence-Based Policymaking on the importance of a cohesive 

postsecondary data infrastructure and its impact on evidence-based policymaking. 

My name is Amanda Janice Roberson and I am a research analyst with the Institute for Higher Education 

Policy. IHEP is a nonpartisan, nonprofit organization committed to promoting access to and success in 

higher education for all students, with a focus on students who have been underserved by our 

postsecondary system. Based here in Washington, D.C., we believe that all people, regardless of 

background or circumstance, have the opportunity to reach their full potential by participating and 

succeeding in higher education. 

In support of this goal, IHEP leads the Postsecondary Data Collaborative (PostsecData), a partnership 

between more than 35 organizations committed to the responsible use of high-quality postsecondary 

data to improve student outcomes. PostsecData partners represent a broad range of constituents, 

including groups that represent students, postsecondary institutions, the workforce community, and state 

and federal policy influencers and researchers. 

Since 2014, IHEP has spearheaded research on which data should be collected, how metrics should be 

defined, and through which mechanisms our currently disconnected, duplicative, and incomplete data 

systems can work together to create a cohesive postsecondary data ecosystem. IHEP supports the mission 

of this Commission to analyze and make recommendations for streamlining federal data and data 

systems. We suggest the following actions to improve the landscape of postsecondary data for use by 

policymakers, students and families, institutions, and researchers. 

x Promote best practices in privacy and security for interconnected data systems.

Recommendations by the Commission for data linkages should address the importance of privacy,

security, and confidentiality. As institutional practices and changing laws at the state level have

led to confusion around when it is permissible to share or link data, policies and procedures from

the Commission should be transparent, consult with data security experts to implement fieldrecognized best practices, and ensure that all publicly reported, aggregate data are stripped of

personally identifiable information.

)@@

0DUPCFS
)FBSJOH

*OTUJUVUFGPS)JHIFS&EVDBUJPO1PMJDZ

1SFTFOUFS"NBOEB+BOJDF3PCFSTPO

1096 The Promise of Evidence-Based Policymaking
2 

x Leverage existing data to decrease burden, streamline reporting, and answer critical questions.

Data from sources like the U.S. Department of Education (which houses the National Student Loan 

Data System [NSLDS] and Integrated Postsecondary Education Data System [IPEDS]), Social 

Security Administration (SSA), the Department of Defense (DoD) and Department of Veterans 

Affairs (VA), among others, should be linked and leveraged to create a more complete picture of 

the higher education landscape. These sources provide valuable data on important subgroups of 

students who are often overlooked, including Pell grant recipients, student loan borrowers, and 

student veterans. If linked, these data would produce valuable information about enrollment and 

completion rates, and post-college employment and earnings. The Commission should consider 

ways to increase capacity and funding available to streamline processes and link data, as these 

are the primary challenges for state and local level data linkages. 

x Expand access to wage and labor market information for postsecondary outcomes. In an era of 

scarce resources, the value of a postsecondary degree has never been greater, and post-college 

outcomes are increasingly important to policymakers and students. Now, data and metrics on 

employment and earnings are limited to voluntary initiatives, like College Measures, state 

dashboards, and the College Scorecard, revamped in September 2015. The Commission should 

explore datasets, like the Census Bureau’s Longitudinal Employer Household Dynamics (LEHD) 

program or the National Directory of New Hires (NDNH), which both utilize state Unemployment 

Insurance (UI) wage records, or the Social Security Administration and Internal Revenue Service 

tax records, to understand the return on personal investment of students and families and federal 

investment in higher education. 

x Align definitions and metrics across federal laws. Establishing common definitions for data 

metrics across federal laws like the Higher Education Act, the Workforce Innovation and 

Opportunity Act, and the Perkins Career and Technical Education Act could reduce administrative 

burden and create comparable outcomes across federal programs. Common and consistent 

metric definitions in the postsecondary ecosystem would make it much easier to link data 

between local, state, and federal sources and allow for accurate comparisons. 

x Recommend that Congress overturn the ban on a federal student-level data system. The 

statutory ban on a federal student unit record system stifles the ability of policymakers to answer 

questions about our postsecondary system, limits the information available to consumers, and 

imposes unnecessary burden onto institutions. The Commission should recommend to overturn 

the ban and direct the U.S. Department of Education to engage with the higher education 

community to design and implement a student-level data system. This system would create a 

nationwide, inclusive data set that shows how students move through higher education and their 

post-college outcomes. This system would allow for disaggregation by key student characteristics, 

like Pell Grant receipt, race/ethnicity, and others, and illuminate evidence for future policymaking 

around closing equity gaps and the federal investment in higher education and postsecondary 

programming. Given the sensitive nature of record level data, the Commission should also 

recommend rigorous data privacy and security policies to govern this system. 

Thank you for your time and the opportunity to provide a statement to the Commission. I look forward to 

answering any questions. 

The Promise of Evidence-Based Policymaking 1097
DC  CA  IL  NY  TX

younginvincibles.org

Facebook: /together.invincible | Twitter: @younginvincible 

Abstract 

Policymakers are making decisions about higher education without crucial performance 

measurements. The Commission on Evidence-Based Policymaking represents a unique 

opportunity to address this problem, and collect and use the information students prioritize the 

most: how different colleges serve today’s diverse student bodies, which majors and programs 

lead to specific occupations and industries, and whether students are repaying their student 

loans.

Statement 

Thank you for the opportunity to speak before you today. My name is Tom Allison and I am the 

Deputy Director of Policy and Research for the Young Invincibles, a national research and 

advocacy organization working to expand the economic opportunities for young adults. 

We know a lot about college and universities: how much schools charge for tuition and fees, 

how many students they enroll, and what types of programs or majors they offer.

We also know a lot about jobs and workforce trends: how many people are unemployed, how 

much money different types of workers make, which industries are growing and shrinking, and 

what skills employers are looking for in their workers.

The problem arises however, when attempting to draw connections between what we know 

about colleges and universities, and what we know about jobs and the workforce. Preventing 

us from connecting that link, is the Student Unit Record Ban, a single paragraph in the 2008 

reauthorization of the Higher Education Opportunity Act, prohibiting the Department of 

Education from collecting and using student-level data. This is frustrating for today’s students, 

who carry challenges and aspirations unique from previous generations, and the majority of 

which pursue higher education to improve their economic opportunities. Choosing where to go 

to school, what to study, and how to pay for it comes with the highest stakes of any decision in 

their life. Students and families need and deserve better insight to inform these decisions. 

Moreover, without outcomes information on which schools and programs lead to jobs and 

ultimately financial security for their graduates, policymakers are left in the dark, unable to 

intelligently align funding with policy priorities. Colleges are also blind to students’ trajectories 

after they leave campus and cannot adjust academic programs or systems to ensure students 

can land good jobs or pursue further education.

Over the course of two years, Young Invincibles conducted workshops, listening sessions, and 

roundtable discussions with current and aspiring college students across the country to better 

understand their priorities and values in attending and paying for college. We synthesized their 

voices in the Student Agenda for Data Reform and organizations representing over one million 

students currently support it. We will submit it in our written comments to the Commission, but 

in brief the agenda calls for overturning the student unit record ban, collecting more information 

about innovative platforms and alternatives to traditional higher education, and to protect the 

privacy and security of sensitive student information.

)@@

0DUPCFS
)FBSJOH

:PVOH*OWJODJCMFT

1SFTFOUFS5PN"MMJTPO

1098 The Promise of Evidence-Based Policymaking
DC  CA  IL  NY  TX

younginvincibles.org

Facebook: /together.invincible | Twitter: @younginvincible 

I’d like to dedicate the remainder of my time to read comments from a former student leader 

and recent graduate from the University of Nebraska, named Thien, whose story illustrates the 

need to improve our postsecondary infrastructure: 

As a 17-year-old, I did not have nearly enough knowledge of federal loan programs, 

extra college fees, trends in increasing tuition costs, or credit transferability to make the 

best possible decision when considering the investment I was making in paying for 

school. Some online tools can be helpful in estimating front-end costs, but they do little 

to educate on what life after graduation, or dropping out, would bring. It only takes a few 

clicks for a student to receive thousands of dollars in loans, but some can end up 

repaying them for decades afterward. Colleges need to be more transparent when 

advertising their costs by also informing prospective students on the costs that go along 

with repayment. 

We need more information on which schools best serve first generation and minority 

students like myself to feel comfortable and assured we’ll find a college committed to 

our success. A college campus can be a very unfamiliar environment when you don’t 

have family members to help navigate the strange new setting. Our institutions of higher 

education need to paint a more accurate picture of their minority communities, and the 

rate of success of those communities experience after graduation, including how 

prepared they are for the workforce. It’s a great resource for some of us, who are not 

used to asking for help and may let ourselves fail out of college before mentioning 

anything to anyone, but it’s frankly not enough.

I can’t speak for every low-income, first generation, minority college student in America, 

but I know these words resonate with a lot of my peers. While we know we need to take 

the reins of our own success, we need to be empowered to do so, and it is clear that 

there is a lot of information that needs to be made available before students can make a 

decision that will impact the rest of their lives and those close to them. What we need 

right now is better data, more of it, and to have it in a transparent and easily digestible 

form. 

You can read the rest of Thien’s story and others on our website. Thank you for your time.

The Promise of Evidence-Based Policymaking 1099
Critical Issues for the Commission on Evidence-Based Policymaking 

Statement from Workforce Data Quality Campaign 

The Commission on Evidence-Based Policymaking’s examination of federal administrative and survey 

data provides an exceptional opportunity to address the management and use of data for measuring 

postsecondary education and workforce outcomes. The Commission’s findings and impact could lead 

to more inclusive, aligned and market-relevant data systems to help educators, students, employers, 

workers and policymakers all make more informed decisions.

Workforce Data Quality Campaign (WDQC) is a non-profit initiative that promotes inclusive, aligned, 

and market-relevant education and workforce data. We engage hundreds of national experts, state 

officials, and workforce development advocates, encouraging the use of data to ensure that all of our 

nation’s education and training programs are preparing students and workers to succeed in a 

changing economy. Given our mission, we are excited about the promise of the Commission’s work, 

and are pleased to have the opportunity to share our recommendations. 

Data collected and held by the government could help to answer a range of important postsecondary 

education and workforce questions, such as: 

x Which skilled positions are employers having a difficult time filling, and what institutions

might they look to for recruitment?

x Are recent college graduates finding jobs and earning good wages?

x How much do students borrow, and can they repay these loans?

x What types of education and training pathways are helping people succeed in careers?

x Which program models are most effective at helping target populations (e.g. ex-offenders,

veterans, low-income individuals) gain skills and find stable employment?

x How can workers know which short-term credentials would likely raise their earning

potential?

x What job search strategies are most effective, and for whom?

In some instances, surveys have been able to answer those questions over a limited time frame, but 

with great effort and expense. A growing number of state longitudinal data systems are linking 

administrative records to answer questions. However, geographical coverage is limited, so they 

cannot answer questions about students who leave the state, or compare outcomes across states. 

The federal government already collects data through numerous administrative sources, in addition 

to conducting regular surveys. With improved coordination, these data could be systematically 

shared and linked to answer those and other critical questions for generations to come. 

)@@

0DUPCFS
)FBSJOH

8PSLGPSDF%BUB2VBMJUZ$BNQBJHO

1SFTFOUFS3BDIFM;JOO

1100 The Promise of Evidence-Based Policymaking
Issues for Action 

Strides have been made in recent years, but much information remains separated between agencies 

because of technological, cultural, and legal barriers. WDQC encourages the Commission to 

recommend the following actions in its final report: 

1.) Expand access to wage information 

The Commission should examine how the federal government can build on existing data collections 

and facilitate the linking of employment and earnings data across higher education and training 

programs. 

Students and workers want to know which education and training programs will help improve 

their chances of having successful careers. Researchers need access to more detailed and 

comparable data on programs to analyze which pathways are working for students and workers. 

Agencies at all levels of government want to know the short- and long-term employment 

outcomes of those they have served. 

Potential relevant data sources include the National Directory of New Hires and the Census 

Bureau, which contain Unemployment Insurance wage records submitted by states. The Internal 

Revenue Service and the Social Security Administration have individual tax records. In limited 

instances, agencies have found ways to use these data to show employment outcomes for 

programs, but the federal government needs to create efficient, strategic processes for managing 

employment data. The Commission should consider how a federal clearinghouse could 

streamline employment data collections and rationalize processes for access, while protecting 

privacy and enhancing security. 

2.) Improve information on postsecondary progress and outcomes

The Commission should examine ways ņ such as establishing a federal student record system ņ to 

measure postsecondary student progress and more effectively and efficiently answer important 

consumer and policy research questions. 

Stakeholders do not have access to comparable information at the program level, and in many 

cases, only students receiving financial aid; attending first-time, full-time; or those pursuing two 

or four year degrees are counted. These limitations exclude non-degree credentials that are 

growing in number and importance, as well as the transfer, part-time, and adult students who 

now outnumber ‘‘traditional’’ postsecondary students. The Department of Education’s College 

Scorecard and planned changes in the Integrated Postsecondary Education Data System (IPEDS) 

reflect progress toward providing and linking data for analysis and consumer-friendly interfaces, 

but the information remains scattered and incomplete. Overturning legal prohibitions on federal 

collection of data on individuals involved in postsecondary education and training programs, and 

implementing a national student record system, would allow for building a more complete 

picture with lower administrative burden. 

The Promise of Evidence-Based Policymaking 1101
3.) Provide more accessible labor market information 

The Commission should include in its examination how labor market information (LMI) might be 

better integrated to provide more comprehensive and clear information. 

Having access to LMI (e.g. occupational projections) may strengthen a worker’s ability to make 

decisions about employment and training, and help to improve the alignment of education and 

training programs with labor market demand. The Bureau of Labor Statistics, Census, and other 

statistical agencies could more effectively collaborate and incorporate additional information 

from federal programs to enhance data about employment, worker characteristics, and the job 

market. If the Commission examines LMI, it should coordinate with the newly established 

Workforce Information Advisory Council (WIAC), which reports to the Secretary of Labor. 

4.) Harmonize definitions and metrics across federal laws

The Commission should explore how the federal government could implement similar definitions and 

metrics to streamline reporting and improve opportunities for data linkages between programs. 

State agencies and service providers often face the burden of having to report on program results 

using different definitions and measures, which increases staff time and cost. Using common 

definitions and metrics from the Workforce Innovation and Opportunity Act (WIOA) for other 

programs as appropriate, such as those operating under the Perkins Career and Technical 

Education Act, would reduce administrative burden, make outcomes more comparable, and 

facilitate coordination across human capital programs. 

5.) Clarify privacy and security protections

The Commission should account for best practices in privacy and security as it conducts its review.

Institutional practices and changing laws at various levels of government have often created 

confusion around what is possible and led to blockages in sharing and linking data, even when 

doing so is legal. Policies and procedures recommended by the Commission should be 

transparent, utilize evolving best practices for data security, and ensure that publicly available 

data are aggregated or otherwise stripped of all information that could be used to identify 

particular individuals or employers. As noted by presenters in an earlier Commission meeting on 

privacy and security issues, the Commission should respect varying viewpoints on privacy rights. 

In order to strike an appropriate balance between privacy concerns and optimal use of data to 

improve publically-funded programs, the Commission should ensure that federal policy accounts 

for emerging technologies that can help protect sensitive information. 

As the Commission conducts its examination, we encourage the elevation and promotion of highquality data sources that can be used to inform human capital development policy. We hope the 

Commission will focus on maximizing the use of data to enhance decision-making and continually 

improve education and training services that allow all Americans to contribute to a 21st Century 

economy. 

1102 The Promise of Evidence-Based Policymaking
Closing the gaps in opportunity and 

achievement, pre-k through college. 

1250 H STREET N.W., SUITE 700 WASHINGTON, D.C. 20005 T 202/293-1217 F 202/293-2605 

WWW.EDTRUST.ORG 

 	–	



#"#!"$

%##$*))+#**/2

$/
%$)1(*%(%(+*%$%"/

+*%$(+)*


,""*/%4'+"*/1(%+)**)/)*#))))$*"*%"&$*

&+"+$()*$%-)*+$*)(($*&(*+"($)**+*%$)1$*/$'+*/

&)1$**($$*,0$#&(%,#$*$)+))3,$**($%(#*%$%$*

%""&(*&*%$$%+*%#)%"")*+$*)")%"&)$)+(**)*+$*)$#!

*)*&%)*)%$(/)%$)%(*#)",)1-**""%$)%%""()***("

%,($#$*$$+""/$,)*)$)*+$*3",***#%)*$*$*,

-/*%*(%#&"*$#%(%$)**)*(%+5#%6+$*(%()/)*#1

$-)+&&%(*%*$%(**%"#$**+(($*)*+$*+$*(%($$$

%(**%.&$$#&(%,*+(($**%""*%$)1$"+$1%(.#&"1*%$

&(*4*#)*+$*)$*($)()1$#!$"")**+)*($)&($*$*%""*%$)3




$!/%+%(*%&&%(*+$*/*%)&!3
/$#)$/
%$)$	#*(*%(%

(+*%$%"/*+*%$(+)*3(+)*)%##***%,$$

+*%$"%&&%(*+$*)%("")*+$*)1+*)&""/"%-4$%#)*+$*)$)*+$*)

%%"%(3#*%,$'+*/$(+*%$/$%+($&%"/#!()$

*&+"*%%"#&+))%+$*"%()*+$*%+*%#)$)+&&%(*$

#&(%,#$**#&+))%##***%)(,$"%-4$%#)*+$*)$)*+$*)%

%"%(3

$*%($"(+*%$*56-)&))$8=;:1*33)#

)+)*$*"&(%())$%""))3%""4%$(*),"#%()*+$*)(%#

""%$%#$("(%+&)3*)&**)&(%())1"%-4$%#)*+$*)*%/$(%""

$&%)*)%$(/+*%$*(*)"%-(*$4$%#)*+$*)$*#48=<7)3

	$,(/*%(/%&%)*)%$(/+*%$1"%-4$%#)*+$*)$)*+$*)%%"%(

("))"!"/*$%*()*%($*()***/-$*$$1$(#%("!"/

*%$+&-**$$%(3

%()(*%$%*-)('+($4891-!$-$%*""/**)%%")-(

$%*+*$""(%+&)%)*+$*)-""3+*-$%*!$%- +)*%-)$$**

inequities were, and we didn’t know which schools were making progress and which 

weren’t.

)@@

0DUPCFS
)FBSJOH

5IF&EVDBUJPO5SVTU

1SFTFOUFS5JGGBOZ+POFT

The Promise of Evidence-Based Policymaking 1103
THE EDUCATION TRUST 

1250 H STREET N.W., SUITE 700 WASHINGTON, D.C. 20005 T 202/293-1217 F 202/293-2605 

WW.EDTRUST.ORG 

*1+$%(*+$*"/1)-(-)*""($(+*%$—)&""/$((*%"%-4

$%#)*+$*)3,)%#"#*()(%$1%(.#&"1%,(""""(+*%$

rates, but we don’t currently know which institutions are serving these students well an

which aren’t. Pell graduation rate data will be incorporated into IPEDS in the coming years, 

+**%**)*,$%*$$"+$$$+"	*%""*%$)3	")%

doesn’t include data on part4time students or students who don’t start$*""%(

)*+$*)-%*($)($(%#$%*(%""3

If we have learned anything from past experience, it is this: that students who aren’t 

measured don’t count. If you want these students to count, and I know you do, you need to 

#!*)#)**%#$$**(***/%+,#$4893

",***#%)*$*$*,-/*%,%#&"*$#%(%$)**

)*(%+5#%6+$*(%()/)*#3+(($*$%$(")*+$*+$*

(%()/)*##!)*#&%))"%(("&%"/#!()*%*%#&($),&*+(

%%-)*+$*)(#%,$*(%+&%)*)%$(/+*%$$**$$()$

(**)3%##))%$)%+"(%##$*%,(*+($%*+$*(%($$

*$%$("*)%	*1)%**-$,$*%$-1$"+),

*)**%)%-%-&%&"(#%,$*(%+,(*/%+*%$&*-/)3

(*$)*+$*+$*(%()/)*#$$/",($.)*$()%+()(%#*

33&(*#$*%+*%$5-%+))**%$"*+$*%$*/)*#$

	$*(*%)*)%$(/+*%$*/)*#61%"+(*/#$)*(*%$1*

&(*#$*%$)$&(*#$*%*($)()1#%$%*()1*%(*#%(

%#&"*&*+(%*(+*%$"$)&3))%+()&(%,,"+"*

%$#&%(*$*)+(%+&)%)*+$*)-%(%*$%,("%%!1$"+$""($*

(&$*)1)*+$*"%$%((%-()1$)*+$*,*($)3	"$!1*)*-%+"

&(%+,"+"$%(#*%$%+*$(%""#$*$%#&"*%$(*)1$&%)*4%""

#&"%/#$*$($$)3

")%)+&&%(*$%(**%.&$$#&(%,*+(($**%""*%$)1$"+$1%(

.#&"1*%$&(*4*#)*+$*)$*($)()1$#!$"")**+)*($)&($*$

*%""*%$)3

$!/%+%(,$*)($$*!$$#&%(*$*$(*")*&*%-(

,$$*'+"*/$,""*/%(+*%$* —)&""/1)*&**

$)+()**(***$+)*%#&%-()*+$*)1#")1*&+"1,%*)1

$#&+)))-#*%$()(+*%$'+*/$)*+$*)+))3

	"%%!%(-(*%$)-($$/'+)*%$)3

1104 The Promise of Evidence-Based Policymaking
October 14, 2016 

Dear Members of the Commission on Evidence-Based Policymaking: 

The Pew Charitable Trusts promotes transparency and accountability in government through the use of 

rational, reliable decision-making based on facts and evidence. We bring forth research that shows 

which policies, practices, and programs are effective. We have used this evidence-based approach to 

support successful home visiting programs for new mothers, evaluations of state-based tax incentives, 

and public safety programs to reduce recidivism. Our experience shows that helping policymakers enact 

evidence-based policies—those that improve states’ fiscal health and enjoy broad bipartisan appeal—

shifts policymakers’ thinking about how to invest taxpayer dollars. As lawmakers see the benefits of 

evidence-based policymaking in one key policy area, they are more inclined to explore reforms in 

others. 

One of our most successful evidence-based initiatives is the Pew-MacArthur Results First project, a joint 

effort of The Pew Charitable Trusts and the John D. and Catherine T. MacArthur Foundation. A 

growing number of states and counties are partnering with the project to make evidence-informed 

decisions in eight policy areas. At present, we work with 22 states and seven counties to incorporate 

rigorous research into their policy and budget processes and use evidence to identify and invest in 

programs that achieve successful outcomes and positive returns on investment. 

We applaud the federal Evidence-Based Policymaking Commission’s work and share your commitment 

to bring data and evidence to the forefront of federal decision making. We stand ready to be a resource 

as you consider how federal policies and practices could support state efforts to use data and research in 

the policymaking process—and offer our experiences at the state and county level that can inform 

federal level policies and practices. 

The Results First approach includes: 

x Creating an inventory of currently funded programs;

x Assessing which programs are most likely to work, based on the best available research;

x Utilizing the customized Results First cost-benefit model to compare programs based on their

expected return on investment; and

x Using the results to inform budget and policy choices.

New Mexico has used the Results First approach to compare the expected outcomes of adult criminal 

justice, child welfare, early education, and behavioral health programs. Using their Results First 

analysis, state leaders directed more than $100 million to evidence-based programs. In addition, the state 

is building a culture of evidence by incorporating evidence into their policymaking processes. For 

example, the Corrections Department adopted a policy that mandates that 70 percent of funds are 

directed to evidence-based programs. The department also adopted contracting standards that require 

vendors to document their use of evidence-based practices and monitor outcomes for programs that are 

developed in New Mexico to ensure that they meet the state's goals. 

)@@

0DUPCFS
)FBSJOH

1FX.BDBSUIVS3FTVMUT'JSTU*OJUJBUJWF

1SFTFOUFS4BSB%VCF

The Promise of Evidence-Based Policymaking 1105
Mississippi passed legislation in 2014 establishing evidence standards for evaluating the state's 

corrections, health, education, and transportation programs. Using the Results First model, the state 

determined that a shock incarceration program—a paramilitary, boot-camp intervention—currently 

required by statute has been proven ineffective by national research. The legislature subsequently moved 

to eliminate the program in 2017, and is now developing an evidence-based alternative. In addition, the 

state, through its budget instructions, now requires executive agencies to justify funding for any new 

program by identifying evidence supporting the program's effectiveness. Mississippi policymakers 

expect to use this information to bolster the state's reinvigorated performance-based budget system. 

New York State has used the Results First framework to target more than $50 million in state general 

funds over three years toward effective evidence-based alternatives to incarceration programs. 

Recipients of these funds are required to show that the programs are being implemented according to 

their original design—demonstrating fidelity—and that they are achieving expected outcomes. The state 

also leveraged its Results First analysis to compete for and win a $12 million “Pay for Success” grant 

from the U.S. Department of Labor. 

Iowa’s Results First analysis confirmed that the state’s existing community-based domestic violence 

treatment program was not effective in reducing recidivism among domestic abusers. In fact, the model 

showed that the state was losing $3 for every $1 invested in the program. To improve outcomes for both 

victims and taxpayers, the department partnered with the University of Iowa to pilot an alternative 

program known as Achieving Change Through Value-Based Behavior, or ACTV. Preliminary results of 

the pilot demonstrated positive effects in reducing recidivism and the department subsequently began 

shifting funds away from the ineffective program and toward ACTV. 

Colorado has completed program inventories and cost-benefit analyses in the adult criminal justice, 

juvenile justice, and child welfare policy areas, and is using their results to re-allocate funds in the FY 

16-17 state budget. For example, the state will repurpose $1.9 million in FY 2016- 17 and $2.4 million 

in subsequent years for a new community corrections pilot project for at-risk offenders, centering the 

offender’s treatment on cognitive behavioral therapy (an evidence-based program). The state has also 

dedicated $7.2 million (in FY 2016-17, with investments of $9.5 million each year after) to 

Communities That Care, a prevention system designed to reduce levels of adolescent delinquency and 

substance use through the selection and use of effective evidence-based preventive interventions tailored 

to a community's specific profile of risk and protection. 

These are just a few of many examples of states using evidence to inform their budget and policy 

choices. We will submit additional examples and information in response to your request for comments 

via the Federal Register. 

As you develop your recommendations, please consider the effect of federal policies on these state and 

local efforts, and feel free to contact us and our partners with any questions about the lessons learned at 

the state level and how they could be applied at the federal level. 

Thank you, 

Sara Dube 

Director, Pew-MacArthur Results First Initiative 

The Pew Charitable Trusts 

1106 The Promise of Evidence-Based Policymaking
October 14, 2016

Dear Chair Katharine G. Abraham, Co-chair Ron Haskins, and members of the Commission:

On behalf of The National Campaign to Prevent Teen and Unplanned Pregnancy, thank you for 

the opportunity to submit a statement for the record for the Commission on Evidence-Based 

Policymaking (CEP) meeting to be held on October 21, 2016. The National Campaign, a 

research-based, nonpartisan, nonprofit organization was founded in 1996. We work to improve 

the lives and future prospects of children and families by ensuring that all children are born into 

families committed to and ready for the demanding task of raising the next generation by 

reducing unplanned pregnancy among teens and young adults. The National Campaign works

towards three ultimate outcomes: 

● Reduce the rate of teen pregnancy by 50% by 2026.

● Reduce the rate of unplanned pregnancy among women age 18-29 by 25% by 2026.

● Reduce the disparities in teen pregnancy and unplanned pregnancy rates among

racial/ethnic and socioeconomic groups by 50% by 2026.

Ensuring that young people have access to high quality, evidence-based teen pregnancy 

prevention education is one critical element in helping more young people delay pregnancy and 

parenting. 

Given our long-standing commitment to research, evidence and evaluation, we applaud the 

establishment of the Commission and appreciate the important issues it is tackling. In this 

statement, we offer feedback about several of the duties the Commission is tasked with, along 

with information about two tiered evidence-based programs—the Personal Responsibility 

Education Program (PREP) and the Teen Pregnancy Prevention (TPP) Program—which we hope 

will be helpful as the Commission goes about its important work.

The Commission poses several important questions with respect to data infrastructure and 

access, including a request for examples of best practices related to linking local, state and 

federal data. Not surprisingly, this type of endeavor raises many technological, ethical, and legal 

challenges, particularly as they relate to the balance between data access and privacy. One 

example that may be helpful to consider is the Longitudinal-Employer Household Dynamics 

(LEHD) program. We highlight this program for its ability to successfully navigate challenges 

associated with partnership formation, privacy protection, and data access while producing data 

that have greatly impacted policy.

Similarly, the Commission poses several questions related to the potential benefits and 

challenges of developing a clearinghouse for administrative and survey data. While The 

Campaign strongly supports greater access to administrative and survey data, and a 

)@@

0DUPCFS
)FBSJOH

/BUJPOBM$BNQBJHOUP1SFWFOU5FFOBOE

6OQMBOOFE1SFHOBODZ

1SFTFOUFS,FMMFFO,BZF

The Promise of Evidence-Based Policymaking 1107
clearinghouse would be beneficial in theory, we believe such an effort would likely fall short of 

its goals and would be difficult to maintain. It is particularly difficult to imagine a single 

clearinghouse that gathered data and evidence across all policy domains in a way that adequately 

captured the complexities of these data and the programs they reflect. Rather, we believe those 

resources would be better committed to helping agencies maintain and enhance the data access 

they already have in place. In our experience, as these agencies try to meet growing data 

collection costs with fixed or even diminishing budgets, the availability of policy relevant data 

has been shrinking in critical ways. Key questions have been cut from surveys and online access 

to data has been curtailed. This is particularly true as it pertains to tabulating results for states or 

localities. For example, one can no longer use the online vital statistics data to look at key policy 

questions like variation in Medicaid or WIC participation at the state level. It is also the case 

that some particularly rich data, such as the Medicaid Max files, are not available as de-identified 

files, thus making them difficult to obtain and underutilized. There are likely similar limitations 

in other policy domains as well. We believe that with relatively modest investments and vocal 

champions, data access could be greatly expanded. 

We also would like to comment on the Commission’s interest in how data and findings from 

evaluations can best be used to improve policies and programs. We offer two examples of tiered 

evidence grant making from the US Department of Health and Human Services (HHS) that use 

evaluation results to continually improve those programs. 

The TPP Program and PREP, like the Maternal, Infant and Early Childhood Home Visiting 

program, have been recognized as pioneering examples of tiered evidence-based policymaking,i

and represent an important contribution to building a body of evidence of what works. They 

include high quality implementation, evaluation, innovation, and learning from results. The 

majority of funding from the TPP Program and PREP goes toward replicating program models 

that have been demonstrated to change behavior using well recognized high standards of 

evidence. A smaller portion of funding is reserved for research and demonstration projects to 

develop, replicate, refine, and test additional models and innovative strategies. This ensures that 

the menu of effective approaches to reducing teen pregnancy will continue to grow and be 

refined. 

TPP Program and PREP grantees can choose from a list of effective models that have been 

identified through HHS’ ongoing systematic review of the teen pregnancy prevention research 

literature. Since 2009, HHS has sponsored this review of the literature to help identify models 

with evidence of effectiveness in reducing teen pregnancy, sexually transmitted infections 

(STIs), and associated sexual risk behaviors. The review, conducted by Mathematica Policy

Research, looked at hundreds of evaluations and initially identified 28 models that met Tier 1 

criteria. That is, they must have been evaluated using a randomized controlled trial or quasiexperimental design, demonstrate changes in behavior (not just knowledge or behavioral intent), 

and results must be published in a peer-reviewed journal. The evidence review is updated 

periodically to capture the latest evaluation studies, and now includes 44 models.ii The wide 

range of models on the HHS list of evidence-based programs gives grantees the flexibility to 

choose an effective approach that reflects their needs, population, and values, recognizing that 

what people in New York City may choose for high school age teens might be different from 

what people in Mississippi choose for middle school youth. 

1108 The Promise of Evidence-Based Policymaking
The TPP Program is a discretionary program administered by the Office of Adolescent Health 

(OAH) that was originally funded in FY 2010 at $110 million. It supported an initial cohort of 

102 grants for a five-year period. Funded at $101 million for FY 2016, the TPP Program 

currently supports 84 competitive grants to a broad range of organizations and agencies serving 

youth in 39 states and the Marshall Islands. The grantees focus intensely on communities with 

the highest teen birth rates and the most at-risk youth. These five-year grants were awarded in 

FY 2015 and are contingent on continued appropriations. As noted above, approximately 75% 

of the grant funds are used to replicate program models that have already been shown through 

careful evaluation to change teen behavior (Tier 1), and approximately 25% of the funds support 

research and demonstration projects to develop, replicate, refine, and test additional models and 

innovative strategies to prevent teen pregnancy (Tier 2). 

PREP, established in FY 2010, continues to be funded at $75 million in mandatory funding 

annually through FY 2017. Administered by the Administration on Children and Families 

(ACF), PREP supports states, communities, and tribes to educate adolescents on both abstinence 

and contraception to prevent pregnancy and STIs, and on other adulthood preparation topics such 

as healthy relationships, communication with parents, and financial literacy. PREP focuses on 

youth at greatest risk of teen pregnancy and geographic areas with high teen birth rates. For 

example, 34% of grantees targeted youth in foster care and 74% target youth in high need areas.iii 

Most of the PREP funding ($58 million) supports grants to states, territories, and tribes and 

emphasizes the use of evidence-based programs. Indeed, more than 95% of youth served by the 

state grants received one of the evidence-based programs from the HHS list referenced above.iv 

An additional $10 million supports competitive grants to public and private entities to develop, 

replicate, refine, and evaluate innovative strategies to reduce teen pregnancy and repeat 

pregnancies among youth up to age 21. These grants are subject to rigorous evaluation and 

reflect a “Tier 2” approach that supports innovation, fills gaps in existing programs for 

underserved populations, and expands knowledge about what works. 

Both programs have invested heavily in the highest standards of evaluation and learning, as well 

as in innovation. OAH funded 41 rigorous evaluations during the first round of TPP Program 

grants that ran from 2010-2014.v

 The recently released findings—90% of which were from 

randomized control trials—indicate that four of the Tier 1 programs were found effective in 

changing behavior in additional settings and new populations. Among the Tier 2 grantees, 8 

new, innovative models were found to be effective. Overall, these evaluations help build a body 

of evidence about where, when, and with whom specific models are most effective, and have 

expanded the menu of effective program models from which communities can choose. The 

results, along with implementation lessons, also help guide the second round of TPP Program 

grantees, and the many communities that look to the HHS list of evidence-based programs for 

guidance on what approaches will work best for them. Many of these findings and valuable 

implementation lessons were recently published in a special supplement of the American Journal 

of Public Health.

vi PREP grantees have also been subjected to rigorous evaluations through 

several different federally sponsored studies, and several studies have already been added to the 

HHS evidence review. 

The commitment to evidence-based investments and innovation in the area of teen pregnancy 

prevention has been pivotal in changing the landscape. Before these two programs began, there 

The Promise of Evidence-Based Policymaking 1109
were no federal investments dedicated to evidence-based teen pregnancy prevention programs; 

research in this area had primarily come from private investments, with few resources available 

to replicate or further evaluate the existing models. 

The National Campaign also offers PREP and the TPP program as two examples where 

evaluation—specifically randomized control trials and quasi-experimental designs—

have been successfully incorporated into the program designs. These are two of the few 

government programs that use evidence and evaluation criteria throughout the grant life cycle.vii 

In fact, only about $1 out of every $100 spent on federal programs is backed by any evidence 

that the money is being spent wisely. 

We believe rigorous evaluations have been successfully implemented for a few reasons. 

Importantly, the legislation for both programs specifies that some portion of funds should be 

used for evaluation. Program requirements also signify that evaluations are a priority. For 

instance, PREP grantees must participate in a federally-led evaluation, if chosen, and the “Tier 

2” innovation grantees are required to conduct their own rigorous evaluations, unless selected to 

be part of a federally-led evaluation. All TPP Program grantees are required to conduct some 

program evaluation, with a subset selected for rigorous impact evaluation.viii In addition, there 

are several federally-led evaluation studies that include large, multi-state, rigorous evaluations 

conducted under contract to OAH.ix Besides rigorous evaluations, mandatory reporting of 

performance measures is another way that OAH and ACF ensure grant projects are making 

sufficient progress toward their stated missions and that there is continuous quality improvement. 

Of course, providing support for grantees is another vital component to ensuring evaluations are 

successful. From review of initial evaluation designs to preparation of the final evaluation 

reports, TPP Program and PREP grantees received ongoing evaluation training and technical 

assistance support to ensure rigorous methods and reporting.x, xi In addition, it is essential to 

have a commitment to evaluation and learning from program leadership and adequate federal 

staff capacity to carry out that commitment. Leadership at OAH and at ACF demonstrated such 

commitment, built staff capacity, and worked closely with evaluation experts at the ACF Office 

of Planning, Research and Evaluation (OPRE) and the Office of the Assistant Secretary for 

Planning and Evaluation (ASPE). 

In closing, thank you for considering our input for the Commission for Evidence-Based 

Policymaking. If you have any questions or need additional information, please contact me at 

202-478-8512 or kkaye@thenc.org. 

Sincerely, 

Kelleen Kaye 

Vice President, Research and 

Evaluation 

i

 Results for America. (2015). Invest in What Works Fact Sheet: Federal Evidence-Based Innovation Programs.

Washington, DC: Author. Retrieved October 12, 2016 from http://results4america.org/policy-hub/invest-works-

 

1110 The Promise of Evidence-Based Policymaking
 

fact-sheet-federal-evidence-based-innovation-programs/. Also, Haskins, R. (2014). Show Me The Evidence. 

Washington, DC: Brookings Institution Press. 

ii Lugo-Gil, J., Lee, A., Vohra, D., Adamek, K., Lacoe, J., & Goesling, B. (2016). Updated findings from the HHS Teen 

Pregnancy Prevention Evidence Review: July 2014 through August 2015. Washington, DC: Mathematica Policy 

Research. Retrieved October 12, 2016 from 

http://tppevidencereview.aspe.hhs.gov/pdfs/Summary_of_findings_2015.pdf. iii Administration for Children and Families/Family and Youth Services Bureau. (2015). Adolescent Pregnancy 

Prevention Program Fact Sheet. Washington, DC. Author. Retrieved October 12, 2016 from 

www.acf.hhs.gov/programs/fysb/resource/app-fact-sheet. iv Administration for Children and Families/Family and Youth Services Bureau. (2015). Personal Responsibility 

Education Program: How States Planned and Implemented Adolescent Pregnancy Prevention Programs: State PREP 

Performance Measures of Structure, Cost, and Support for Implementation. Washington, DC: Author. Retrieved 

October 10, 2016 from www.acf.hhs.gov/sites/default/files/fysb/prep_pm_brief_20151216.pdf. v

 U.S. Department of Health and Human Services, Office of Adolescent Health. (2016). Results from the first round 

of TPP grantees. Retrieved October 12, 2016 from www.hhs.gov/ash/oah/oah-initiatives/tpp_program/cohorts-fy2010-2014.html. vi Morabia, A. (Ed.). (2016). Building the Evidence to Prevent Adolescent Pregnancy: Office of Adolescent Health 

Impact Studies (2010-2015) [Special issue]. American Journal of Public Health, 106(S1). Retrieved October 12, 2016 

from http://ajph.aphapublications.org/toc/ajph/106/S1. vii A recent GAO report includes TPP in its review of five tiered evidence grant programs, noting evidence is used 

throughout, including for assessing the evidence base and identifying evidence-based approaches, implementing 

evidence-based approaches with fidelity, conducting rigorous independent evaluations, and disseminating 

evaluation results. 

viii U.S. Department of Health and Human Services/Office of Adolescent Health. (2016). TPP Evaluation and 

Performance Measurement. Retrieved October 10, 2016 from http://www.hhs.gov/ash/oah/oahinitiatives/evaluation/grantee-led-evaluation/grantees-2010-2014.html. ix http://www.hhs.gov/ash/oah/oah-initiatives/evaluation/federal-led-evaluation/index.html x http://www.hhs.gov/ash/oah/oah-initiatives/evaluation/ta.html. xi Goerlich Zief, S., Knab, J., & Cole, RP. (2016) A Framework for Evaluation Technical Assistance (2016). American 

Journal of Public Health, 106(S1): S22–S24. Retrieved October 10, 2016 from

http://ajph.aphapublications.org/doi/full/10.2105/AJPH.2016.303365. 

The Promise of Evidence-Based Policymaking 1111
From Data to Action: 

Achieving Results People Care About Most

PRESENTATION TO THE

COMMISSION ON EVIDENCE-BASED POLICYMAKING

QUENTIN WILSON

WASHINGTON, DC

OCTOBER 21, 2016

)@@

0DUPCFS
)FBSJOH

1SFTFOUFS2VFOUJO8JMTPO	JOEFQFOEFOUSFTFBSDIFS


1112 The Promise of Evidence-Based Policymaking
Breaking Good

The effective use of good data is vital to achieving the results about 

which people care most.

The CEP environment includes a global consensus, bi-partisan support, 

high ambitions, new resources and recent successes in the use of data, 

analysis and evidence to improve public program performance.

The Commission agenda on data access and protection can help 

broaden and deepen the use of data and analysis for that purpose.

The value of this agenda can be increased and sustained to 

simultaneously take two big steps: 

◦ Providing information and other resources to help speed the 

improvement process; and 

◦ Linking programs performance to the high-level results about which 

people care most.

The Promise of Evidence-Based Policymaking 1113
Global Consensus and 

Advancement: Using Data to 

Achieve Better Results

◦ Tim Berners-Lee, Founder of the World Wide Web: The 

Next Web

◦UN Guidelines on Open Government Data

◦ White House Executive Order and 9/28/16 Open Data 

Innovation Summit

◦ Commission on Evidence-Based Policymaking

◦ Results Washington

◦ Maryland StateStat

◦ Baltimore OutcomeStat

1114 The Promise of Evidence-Based Policymaking
You’ve Got a Friend:

Emerging Research, Advocacy and 

Support Resources

Governing Institute Living Cities

Results for America USC Civic Data

Pew-McArthur Results First Initiative and Clearinghouse

Bloomberg Philanthropies What Works Cities

Harvard Data-Smart City Solutions

Hewlett Foundation Effective Philanthropy Group 

Evidence-Based Policymaking Collaborative

Postsecondary Education Data Collaborative

Bill & Melinda Gates Foundation

Institute for Higher Education Policy

The Promise of Evidence-Based Policymaking 1115
Caring about Data:

Making a Difference on Results that Matters Most

Key Result Area Data-Based Initiatives

Priority Outcomes

Education

College Completion Career & College Clubs

Child Development First Five/Parents as Teachers 

Public Safety

Violent Crime Gang intervention

Worker Safety Highway construction process

Health

Substance Abuse SAMHSA E-B Program Registry 

Prevention

1116 The Promise of Evidence-Based Policymaking
College Completion: 

Career & College Clubs

The Promise of Evidence-Based Policymaking 1117
Data Making a Difference - II

Key Result Area Data-Based Initiatives

Priority Outcomes

Economic Prosperity

Strong Neighborhoods Blight Reduction

Housing 

Employment Opportunity Pathways to Careers

Effective Government Santa Monica FD

Missouri Dept. of Revenue

Energy and the Environment

1118 The Promise of Evidence-Based Policymaking
Economic Prosperity: Bridgespan’s Billion 

Dollar Bet on Pathways to Careers

Data-based research-driven proposal

Potential investments in six result areas “emphasized the need to 

better track and manage data.”

Results measured by Return on Investment (ROI) model. (10X)

The Promise of Evidence-Based Policymaking 1119
State Initiatives:

California Performance Excellence Resources

◦California Data Collaborative

◦California Evidence-Based Clearinghouse

◦California Open Justice initiative

◦Open Data tech firms 

◦Local government leaders

1120 The Promise of Evidence-Based Policymaking
Setting the Stage for Faster, Better Results: 

What’s Needed Now

Create a repository of consensus outcomes and measures, 

and resources for improvement.

Encourage leadership and collaboration among agencies, levels 

of governments and sectors.

Build a culture of support for the use of data and outcomes 

that matter, not punishment and misguided “accountability.”

Develop more accessible, commonly accepted and usable data.

◦ “80% of data lives in forms and places our teams and 

systems can’t easily process.”- IBM Watson Team

◦ World Wide Web founder Tim Berners-Lee: Linked Data

The Promise of Evidence-Based Policymaking 1121
Next Steps: Using Data and Analysis for 

Faster, Better Results People Care About

Prioritize organizational goals, outcomes and measures from the 

inventory of options.

◦ Utilize data and information about the current, projected and 

comparative performance on these outcomes.

◦ Involve the public, elected officials, researchers, advocates and 

practitioners in the prioritization process.

Speed effective implementation with access and use of resources.

◦ Policy and program research and advocacy

◦ Promising practices

◦ Technical assistance and training

Integrate performance assessment and review with strategic 

planning.

1122 The Promise of Evidence-Based Policymaking
For More Information:

Contact:

Quentin Wilson

1230 Rosecrans Avenue

Suite 300

Manhattan Beach, CA 90266

310-800-4715

The Promise of Evidence-Based Policymaking 1123
American Principles Project

1130 Connecticut AVE, NW, Suite 425, Washington, DC 20036 

www.AmericanPrinciplesProject.org 

$+(*%##))%$*%())*"")*%(&"*)**+*%(/&(%*%$%$*

,"%&#$*2#&"#$**%$2%(#$*$$%(")*+$*+$*:(%()/)*#5+

)/)*#-%+"+(*""(*/$*()*)%*$,+"2-%+"$,**%""*%$$+)%

,(:#%(*2$-%+"+$#$*""/"*(*("*%$)&*-$*$,+"$

%,($#$*$-/**)$%#&*"-*%+(%$)**+*%$"(&+"5


"
%"#


!&"

%##))%$%$,$:)%"/#!$-)(**%&+()+"+"%"4%

#&(%,$"/))%**,$))%("&(%(#)5E+*-$)+&+()+*)+)*%

 +)*/%""*$2%$"%#(*$2$*(!$#)),#%+$*)%#($)6&()%$"*2

))(*$"/*(+$*("#%+*%$2*6)$))(/*%.#$*$()$*

*(%)5#($($&")(% *>?",)**)+*,*))+&&())*"(*)

%*&%&"$&(,(**("*%$)&*-$*&%&"$%,($#$*5+(*

%##))%$*%( *"")*%)*")(")*+$*+$*:(%()/)*#$*%$$

)+(-""$*,*/5

*%$EGH%*	(+*%$*-)"/&(%*)*,"%&#$*2#&"#$**%$2%(

#$*$$%(")*+$*+$*:(%()/)*#>%$**-%+"""%-*%,($#$**%

%""*&()%$""/$*"$%(#*%$>

?%$$,+"(:+*%$)*+$*)$"$!

+*%$**%-%(!%(*?5F

$*"/2*%+2$%()*(*#$%((&"%

*)&(%*%$)$)-""$5G



%($*%-"":+$%($0*%$)H

-*,)*$*()*$))$***%(*(

%-$&+(&%))2*("%,($#$*)+()(%#*:&(,*%$5$!%-#+#%(

$*"/%+($*%$%+"%&(*2$%-#+#%(*%,($#$*%+""&&%&"

(+$*(%-$",)2*#$*$$*("0(&%)*%(/*(!$"#%)*,(/%$,"

*&%$*%+*,(/*0$;-(**$)%%"2-*%+())*%%!2-*()

($2-*.*(+((+"((%(>%%%(?%#&"2-* %)&&"%(2

-* %)%*2-*)"(/#2-*(-)&(%#%*2-*)"(/($$)

$-&%)*%$2-*("%)*) %$-/2-*( %$*#"*(/2-*)%(*%

E

,5+"+)$27%##))%$%(,$:)%"/#!$4*&$**(*%$28

	(*5%(>(M2FDEI?2!**&499---5(*5%(9()(9(&%(*)9FDEI9DG9:%##))%$:%$:

,$:):&%"/#!$::)*&:$:*:(*:(*%$5 F

FD555)5EDDE%#!**&499$+5+9%
9FDDLEDGD=	EDE:)*+$*+$*5&5 G 

/")%$27
%)#	)%#18	 >/EG2FDEG?2!

**&)499---5$)(5%#9$-)9FDEG9DI9EG9&%"*":-$):)*:(":+$*:(%():*):%-:

#+5

H

""/"27)$$%$("$*:%(/)*#*%(!*+$*)2&%(*/)28(%$"%	(

+*%$>(EE2FDEH?2!**&499---5(%$"5%#9(*"9)$:$:%$:("9EHIFKM95

)@@

0DUPCFS
)FBSJOH

"NFSJDBO1SJODJQMFT1SPKFDU

1SFTFOUFS&NNFUU.D(SPBSUZ

1124 The Promise of Evidence-Based Policymaking
F

#"*(/(%()*")2-*(-)(()*$%(-*2-*(-$**% "2

$%$$%$ %

))$%*)(&*%$%($%&$$***)%#(5)))(&*%$%

*%*"*($)%*/**!&)*)%$*)%-$*0$);%(*(%-$%%2%%+()5
*6)")%

)(&*%$%-*-%+"$,*"/&&$-**)*")#$*%)*+$*+$*:(%(

)/)*#2""$*$#%7**(%$)+#($%(#*%$287%+$*"*/28$

7*($)&($/58

*6)-(%$-*("+$*:*(!$)/)*#1

()*2*-%+"%#&")*+$*)6&()%$""/$*"$%(#*%$>

?-*%+**(%$)$*;

%(,$*(!$%-"***(*)$%""*$)"%)5
*6)%$*$*%%""*

*(%#)*+$*-%,%"+$*("/>-%%+()&()+#)*+"$%*%*&(%(#?

&(*&*)$%,($#$*&(%(#$+$()*$)**&(*&*%$-"".&%))

*%

&(%(##$)*(*%()3*6)'+*$%*(*%%("/)+!,(/$,+"$*%*:

%""*%$)/)*#)#&"/+)$(%""$$$)**+*%$%(+*%$5""$**

)*+$***#+)*$%,()&()%$"**%&(%#%*(*(%%)$/

+(+(*)$"%/)*);%(,$-%()2 +)*(%%$$#-*%+**""$#$/*$;

))#&"/+$:#($5

%$2*&+(&%))%*&(%&%))/)*#-%+")%%&$:$***(&%)*%(/)

(*$*%.&$%,(*#*%$*("0*(/%$%""*$#&"%/#$*

*5
$*(*,+(+(*#$2"*(""/,(/*$$"$!*%+*%$5%-/

)*%&-*#&"%/#$**1/$%*)%-%$6)+*%$*))&(*&*%$$*

#"*(/1()"*1()(#$"*/1()%+)$&**($)1(*$+#(%"($

)1(-*(&+())+$1()&%"*"*,*/1
$'+($+(+(*)-$**%

!$%-2$,(/'+)*%$$ +)*/*$7**(%$)+#($%(#*%$58

$-""*)%))((*%$,(/*0$%#&(#$$*1()+#"/)%5
*%"%

&(%,$#.#+#%$)+#($%(#*%$)*%,2%*)*%("$+(($**;

%$)*$*"/+&*$.&$;#+)*%#&"$&()(,5

(&)*).&$)%$-%$6*&&$5(&)*("%,($#$*2$)*(!%$*()**%*)

,%(%,(*")*EDD/()2-"")*/-*$*)%+$()5+*("*/:)#($)

!$%-*%,($#$*-""&+)*$,"%&)()*&%))"/$2)*"-/)%)5$

*/!$%-**,$**%,($#$*))*%)+*()+(*(%,%*)$(%+)*%

&(,/$*%$,+""(*/5

(2****)#)),(&%)*%(/%

-""&(%**$)*+$+*%(0))

$*())'+*)#&"/"+)%$"5
))*$/(%2($%*	%+)

%##**%$%,($#$*,()*$%(#I

(,"*)%!$"!%)*+$*:*



I

55&(*#$*%+*%$4
$%(#*%$+(*/,-8>%,5EK2FDEI?2!

**&)499%,()*5%+)5%,9($9+:):&(*#$*:%:+*%$:$%(#*%$:)+(*/:(,-95

The Promise of Evidence-Based Policymaking 1125
G

)+(*/*(%+%+**55&(*#$*%+*%$>?5&(%"#)$%#&))%*

".%$*(%")%,(*&%&"""%-))*%)$)*,*2)-"")%+***$%"%/

$$'+*)+(*/*%&(,$*+$+*%(0))5

6))/)*#%$*$)%,(EGM#""%$%"+(*/$+#()>"("/*(%+*)%%

$$"*+$*?2"%$-*)$)*,%((%-($%(#*%$%+*)*+$*)$#")

%$*$$**%$"*+$*
%$*)5$$)%*%*
$)&*%(

$(">
?$*$("%+$*$-()*+($4

 *MK2DDD%+$*9+)()-*))*%*)$%(#*%$>%,($#$*

#&"%/)$%$*(*%()?2-(*$FD&($*,+$(%$

!(%+$!*%(,)+(*/"($5

 )+(*/#$)#)&(%**$***((%))"/$'+*5)%$


-*$))*)*27+($%+(*)*$555
*)*()-("*%$

+""))*%*&(*#$*6)$*-%(!$%+())-$*+$**/

""@*,$%(A$*&(*#$*6)%*
$%(#*%$

(58

 $%((&*-($$)(%#
***)$%(#*%$)/)*#)(

,+"$("*%)+(*/*(*)5

**("%,($#$*)%+"$%-%$)(""%%$$*)$)*,*%$*$$

*)$)+()/)*#))*)*#)+$(!"))5

,$**)/)*#)-()+(2*##$)*(*%$6)+**$%*#"/

+*%$"*)$(,/*>?#$)**%,($#$*+*%$%")>("2

)**2$"%"?$%-,$%(#%+)"-/*%)"%)

%$$,+")*+$*)-*%+**(

%$)$*5+()+$**%*($*(+"*%$)2*)%")#/)(&(,*

-*

%*(%,($#$*$)2$%$&(%*$**)2%(&%(*%$)2()(()2$"*(""/$/%$

%$*&"$*)"%$)*)"%)+($(*(0)$+*%(,"+*%$%

>(%"/$?7+*%$&(%(#58J

""*$-%$"%#(*%$%)*+$**(#%()"%)+(+$(*)

(+"*%$)1$()*%%("*%)#))5

&"%)%&"&(%"#-*(")*+$*+$*:(%()/)*#)****(*)(:%($

#($*0$))% *)%()($)*+/5
*))+#)***%"%$**$

%*()$)%*/2$,+$*%(*"-/)2+*%(0)*&%-(+"("%,($#$**%

%""*$))#$*#""%$)%*&%$*)%$$,+");-*%+**(%$)$*5)

+$#$*""/$)*("*%$)&*-$*$,+"$%,($#$*5%""*$

$%"$#)),#%+$*)%*%+*$$,+")$$*#*$*%$*

$,+"<,$**)$,(+)5)),$#%()%*)-$*%""*%()



J


**((%##($))%*%$%%""*)*(()$#))%$)()*%$")%55&*5%

+*%$>/FG2FDEE?2!**&)499(%:-5)G5#0%$-)5%#9#(*9::

%##$*)5)"5).=IFDIDELHFMGD5KKDDLGIE5).5

1126 The Promise of Evidence-Based Policymaking
H

*%(%*"-$*5+((&+"()*)%$*****0$-""(*

%,($#$*5*$$%*&&$-(%,($#$*)*)$&%)*%$%$*#*%$%,(*

$,+"5

+#**/4

##**5(%(*/2)'5

$%(""%-

#($($&")(% *

The Promise of Evidence-Based Policymaking 1127
1

Commission on Evidenced-Based Policymaking 

Submission by 

Carrie Wofford, President, Veterans Education Success & 

Mark Schneider, Vice President, American Institutes for Research 

The Need to Share and Link Federal Data on the Post-9/11 GI Bill 

We urge the Evidence Based Policymaking Commission to take steps to end the siloization of 

federal data and ensure that federal agencies share data. 

The Post-9/11 GI Bill provides an important example of the detrimental impact of siloed federal 

data. 

Why it Matters: 

Historians and economists frequently credit the original GI Bill with helping to build America’s 

Middle Class following WWII. After the 9/11 terror attacks on American soil, a new GI Bill was 

enacted to provide the current generation of veterans with their ticket to the American dream, 

helping 1.5 million veterans at a cost to taxpayers of $61 billion since August of 2009. The goal 

is to assist veterans in the transition to a successful civilian career. 

In order to best serve veterans and the federal taxpayer investment, government officials, 

higher education leaders, and policymakers need to know how the GI Bill is succeeding and 

“what works” under the GI Bill. 

Such an assessment is impossible because federal data regarding student outcomes and 

occupational outcomes for veterans remain siloed across several federal agencies. 

At this time, nobody in America knows the student veteran graduation rate, debt rate, default 

rate, or whether the Post-9/11 GI Bill is succeeding in supporting veterans’ transition to civilian 

occupational and income success. Little is known about veterans’ educational attainment, debt 

or default, because the U.S. Education Department (ED) does not know which students are 

veterans and the U.S. Department of Veterans Affairs (VA) does not track student outcomes. 

Although ED formerly tracked veterans in its database, it stopped doing so in 2009, when it 

introduced a skip-pattern in its FAFSA form, such that most veterans never seeing the question 

about military service. Unfortunately, this change at ED was launched the exact same year 

(2009) that the Post-9/11 GI Bill went into effect. A 2014 U.S. Senate Committee reported that 

for-profit colleges dominate the Post-9/11 GI Bill and generally provide poor outcomes for 

students overall, while costing taxpayers twice as much per veteran as public colleges and 

universities, but student outcomes specific to veterans was unknown. Indeed, reporters asked 

the Senate Committee what the student veteran graduation rate was, and this question was 

)@@

0DUPCFS
)FBSJOH

7FUFSBOT&EVDBUJPO4VDDFTT

"NFSJDBO*OTUJUVUFTGPS3FTBSDI

1SFTFOUFST$BSSJF8PGGPSEBOE.BSL4DIOFJEFS

1128 The Promise of Evidence-Based Policymaking
2 

impossible to answer because the data needed to answer this basic question is held in separate 

agencies.1 

In addition, occupation and income data from IRS and Census is not linked to either VA or ED 

data. 

What Data-Linking or Sharing Could Achieve: 

If data from VA and ED were shared or cross-walked and if occupational and income data from 

IRS or Census were added, the resulting combined data set would provide solid answers on the 

student veteran graduation rate, debt and default rates, job placement rate, and income. The 

shared data would: 

Help veterans make an informed college choice as they decide where to use their GI Bill by 

arming them with data about veterans’ probability of graduation and likely earnings trajectory 

from each college (and each program). VA’s GI Bill College Comparison Tool is currently the 

best source of information for veterans choosing a college, but it does not provide veteranspecific data. If federal data were shared, VA’s GI Bill College Comparison Tool could be as 

robust as ED’s College Scorecard. The College Scorecard was possible only because federal 

agencies shared data. The College Scorecard provides students with important data-points 

about student graduation (specifically, the graduation rate within 150% of expected time to 

completion for first-time, full-time students) and salary after attending (specifically, the median 

earnings of former students who received federal financial aid 10 years after entering the 

school). It should be noted that much of ED’s data is limited to first-time, full-time students, 

which is an outdated limitation and one the Commission should urge ED to change. 

Help Congress and policymakers improve regulation by providing data on Post-9/11 GI Bill 

students and their debt and occupational rates. Currently, Congress and policymakers have no 

data on the student outcomes, nor on the occupational and income outcomes, of educational 

paths under the Post-9/11 GI Bill. Understanding a return on investment would assist 

 

1 Private efforts have tried to determine the student veteran graduation rate, but it is impossible 

without VA and ED actually sharing data. The largest private effort (by Student Veterans of America, 

known as the 2014 “Million Records Project” and its 2016 update “NVEST”) undertook to match VA data 

on Post-9/11 GI Bill use against data from the National Student Clearinghouse to try to determine the 

student veteran graduation rate for a subset of GI Bill users. But the Clearinghouse data is limited to 

degree-granting schools (and covers most, but not all veterans at degree-granting schools), so it 

provides only a limited answer. Most notably, Clearinghouse does not track students at certificate, non -

degree programs, nor vocational/technical programs (both of which are covered under the GI Bill). VA 

estimates that nearly half of the GI Bill is spent at non-degree schools, meaning that the Clearinghouse 

data and the “Million Records Project” are missing half of GI Bill students. Specifically, VA reports that 

among Post-9/11 GI Bill students starting their education in 2015: some 54,000 Post-9/11 GI Bill 

students were starting non-degree college programs, and 30,000 started vocational and technical 

programs, while 87,275 started undergraduate degree programs and 19,222 started graduate degree 

programs. 

The Promise of Evidence-Based Policymaking 1129
3 

policymakers greatly, but requires federal data matches. Data-linkage would enable 

exploration of the effectiveness of the Post-9/11 GI Bill in ensuring a successful civilian career 

for the current generation of veterans. Adding in data from the U.S. Defense Department’s test 

scores on service members’ abilities and skills could serve as “controls” in determining the 

impact of the GI Bill. 

Executive Order 13607 (April 27, 2012) required VA and ED to share data to determine veteran 

student outcome measures, but, four and a half years later, the agencies still have not 

completed an MOU to do so.

Detailed Questions That Could Be Answered if Federal Data Were Combined: 

1. Participation in the Post-9/11 GI Bill: 

a. Nearly half of the Post-9/11 GI Bill goes unclaimed. What are the demographics 

of veterans who skip the GI Bill? How are they faring? What are their 

occupations and incomes, and how do those compare to their occupations and 

incomes prior to military service? Are they reliant on public assistance? Did they 

skip the GI Bill because they already had a college degree, or because they had a 

strong career before military service? By historic contrast, only 20% of eligible 

veterans skipped the original GI Bill following WWII, and such eligible nonparticipants were often older (over the age of 35).2 In terms of occupation prior 

to military service, most veterans who utilized the original GI Bill after WWII had 

“little or no pre-war experience in jobs requiring extensive skill or training,” 

while those who skipped it (eligible non-participants) had been “working in jobs 

of a fairly high level” before the war.3 Are these trends true today under the 

Post-9/11 GI Bill? Veterans organizations report anecdotal evidence that many 

veterans skipping today’s GI Bill do need higher education but feel intimidated 

by the college search process and fear their academic skills are not up to par. 

 

2 The President’s Commission on Veterans’ Pensions, Veterans Benefits in the United States (April 1954), 

Omar Bradley, Chairman (known popularly as the “Bradley Commission”), available at

http://www.cnas.org/sites/default/files/Bradley_Commission_Report1956.pdf (page 251, 259). The 

Bradley Commission found that 83-86% of veterans in the two youngest groups (under 20 or between 

20-24 years of age) used the GI Bill. 

3 Bradley Commission, available at

http://www.cnas.org/sites/default/files/Bradley_Commission_Report1956.pdf (page 261). The Bradley 

Commission also reported that that many of the younger veterans “had held no regular job before 

entering service. In general, those who had held jobs were in relatively unskilled occupations.” (page 

258). In contrast, only 30% of pre-war managers and proprietors used the original GI Bill, as did only 

39% of pre-war full-time employees. (page 261) 

1130 The Promise of Evidence-Based Policymaking
4 

b. Participation under the Post-9/11 GI Bill is increasing each year. VA data shows 

that nearly 1 million eligible individuals participated in 2011, a 15% increase over 

FY 2010 and a 71% increase over FY 2009. Do the data show better student 

outcomes (such as persistence and completion) and civilian employment success 

among more recent GI Bill students as compared to 2009 and 2010? Are student 

loan debt and default levels rising? Has the return on taxpayer investment 

changed over time? 

c. Non-Veteran vs. Veteran Participants. Veterans can choose to give some or all 

of their Post-9/11 GI Bill to their spouse or dependents, and 18% of GI Bill 

students are spouses or dependents. Do non-veteran GI Bill students enjoy 

better outcomes than veterans? Do they have better persistence and 

completion rates in college, perhaps indicating that veteran students need more 

support on campus? How do non-veteran Post-9/11 GI Bill students’ loan debt 

and default rates compare to their veteran counterparts? Do non-veteran 

participants have higher incomes and better correlation between their 

occupation and field of degree? 

d. Outcomes by Demographics. Are there differences by age, race, gender, 

ethnicity, or residential region in outcomes for Post-9/11 GI Bill users? For 

example, some VA analyses suggest that women veterans are more likely to use 

the Post-9/11 GI Bill than men. Is their persistence better? Are their outcomes 

better? 

2. GI Bill Effectiveness. How effective is the Post-9/11 GI Bill in ensuring a successful 

civilian career for the current generation of veterans? 

a. Income & Public Assistance. Do veterans have higher incomes and less 

dependence on public assistance programs after using the Post-9/11 GI Bill than 

before they used it? By point of comparison, the original GI Bill, following WWII, 

reportedly reduced reliance on unemployment assistance from 20% of veterans.4 

What is known about veterans who rely on public assistance programs? How 

many have a college degree, whether through the Post-9/11 GI Bill or otherwise? 

What was their field of study in college? What is their occupation? 

b. Degree and Occupation. Do veterans have different occupations after using the 

Post-9/11 GI Bill than before they used it? Among Post-9/11 GI Bill users, which 

occupations, degrees, and fields of study result in the highest income and least 

reliance on public assistance? Do the degrees and occupations correlate, or is 

occupational success (at least in some occupations) independent of degree 

 

4 Bradley Commission, available at

http://www.cnas.org/sites/default/files/Bradley_Commission_Report1956.pdf (pate 251). 

The Promise of Evidence-Based Policymaking 1131
5 

obtained? What about licensed occupations? Does the GI Bill help veterans 

obtain work in licensed occupations? (Approximately 20% of a sample of 300 

degree programs approved by VA for Post-9/11 GI Bill in licensed occupations 

are improperly accredited and fail to leave the graduates eligible to work.5 What 

percent of veterans working in licensed occupations used the Post-9/11 GI Bill in 

that field of study? Are public colleges more likely to have the right 

accreditation for graduates to be eligible to work in licensed fields?) Which 

degrees and institution types produce graduates employed in which fields, with 

which licenses? 

c. Debt and Default: What are the student loan debt and default rates for both 

veteran and non-veteran beneficiaries using the Post-9/11 GI Bill? Are debt and 

default rates higher for veterans than non-veteran students using the Post-9/11 

GI Bill? Are there correlations in student loan debt and default by degree and 

field of study obtained? By college type (online vs. brick and mortar) and 

institutional sector? By occupation and income? 

d. Montgomery GI Bill vs. Post-9/11 GI Bill: Does the Post-9/11 GI Bill deliver 

better return on investment, including student and occupational outcomes, than 

its immediate predecessor, the Montgomery GI Bill? 

3. Which College Experience Yields the Best Return on Investment under the Post-9/11 

GI Bill?

a. Brick and Mortar vs. Online. Among Post-9/11 GI Bill users, what type of college 

is most effective? Many experts assume brick and mortar colleges are more 

effective than online education, but online education is very popular with 

military students. What are the educational outcomes and civilian employment 

success for online student veterans? 

b. Institutional Sector. Among Post-9/11 GI Bill users, which sector of college is 

associated with the best student outcomes and civilian employment success: 

public, non-profit, or for-profit? What is the return on investment from each 

sector? 

 

5 See Veterans Education Success, “The GI Bill Pays for Degrees That Do Not Lead to a Job,” (Sept. 2015) 

available at

http://static1.squarespace.com/static/556718b2e4b02e470eb1b186/t/56ba65f8356fb040f04fb56a/145

5056377419/GI+Bill+Dollars+do+not+pay+for+accredited+programs.pdf 

1132 The Promise of Evidence-Based Policymaking
6 

i. Several government6 and private reports7 have concluded that for-profit 

colleges do not serve students well. Do for-profit colleges have lower 

persistence and completion rates among Post-9/11 GI Bill students than 

other sectors? Do for-profit colleges produce higher student loan debt 

levels and default rates? What are the civilian employment results for 

Post-9/11 GI Bill graduates? Also, what is the return on investment? 

ii. Experts often claim public community colleges produce better results for 

veterans, and significant public and private funds are invested in 

community college programs for veterans. What are the student 

outcomes and civilian employment success of veterans at community 

colleges compared to other sectors? What is the return on investment 

for veterans at community colleges? 

c. Specific Colleges: Because much of the Post-9/11 GI Bill expenditures are 

concentrated in a handful of colleges, it is possible to derive robust samples at 

those colleges, enabling an assessment of how well the colleges are serving 

veterans and the taxpayer investment. 

d. Type of Degree: Are there differences in the return on investment among GI Bill 

students attending college degree programs (e.g., Bachelor of Science in 

Engineering) vs. non-degree, certificate programs (e.g., Certificate in Radiology 

Technology) vs. vocational/technical programs (e.g., truck driving) – taking into 

account the cost of the program and the resulting civilian success? (Defense 

Department scores of service members’ skills and abilities could serve as a 

control when measuring post-education occupational success.) Do demographic 

or household income correlate with type of degree chosen? 

e. Field of Study: Is it possible to determine program-specific outcomes, such as 

the return on investment of a B.S. in Engineering vs. a B.S. in Nursing, factoring in 

demographic and pre-education differences? 

We hope the Evidence Based Policymaking Commission can take strong steps to end the 

siloization of federal data and ensure that federal agencies share data to benefit the public as 

well as policymakers. 

 

6 See, e.g., U.S. Senate Committee 2012 report, “For-Profit Education: The Failure to Safeguard the 

Federal Investment and Ensure Student Success,” available at

http://www.help.senate.gov/imo/media/for_profit_report/PartI.pdf

7 See, e.g., National Bureau of Economic Research, “Evaluating Student Outcomes at For-Profit Colleges,” 

(June 2012), available at http://www.nber.org/papers/w18201 

The Promise of Evidence-Based Policymaking 1133
PUBLIC SUBMISSION 

As of: 10/14/16 4:20 PM 

Received: October 14, 2016 

Status: Pending_Post

Tracking No. 1k0-8sgj-iuzd

Comments Due: November 14, 2016

Submission Type: Web 

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking 

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking 

Document: USBC-2016-0003-DRAFT-0010

Comment on FR Doc # 2016-22002 

Submitter Information 

Name: Christel Swasey 

Address:

Pleasant Grove - Utah, 84062 

Email: 212christel@gmail.com 

General Comment 

Dear Commission on Evidence-Based Policymaking, 

I love the American concept of voter-based, Constitution-based, elected representative-based, 

policymaking. It's why I live in America. In contrast to voter-based policymaking there is 

evidence-based policymaking, which I don't love because it implies that one entity's "evidence" 

trumps individuals' consent to new policy changes. 

Former Secretary of Agriculture Ezra Taft Benson said something about education that also 

applies to educational data and policymaking: 

"The best way to prevent a political faction or any small group of people from capturing control 

of the nation's educational system is to keep it decentralized into small local units... This may not 

be as efficient as one giant super educational system (although bigness is not necessarily 

efficient, either) but it is far more safe. There are other factors, too, in favor of local and 

independent school systems. First, they are more responsive to the needs and wishes of the 

parents and the community. The door to the school superintendent's office is usually open to any 

parent [or teacher]... But the average citizen would be hard pressed to obtain more than a form 

1134 The Promise of Evidence-Based Policymaking
letter reply from the national Commissioner of Education in Washington, D.C." 

Local control, and consent of the governed, are two foundational principles in our great nation. 

Because the CEP is not an elected body, it does not hold authority to collect, or to recommend 

collection, of student-level evidence, or of any evidence, without written consent; and, for the 

same reasons, neither does the Department of Education. 

Because the fifty, federally-designed, evidence-collecting, State Longitudinal Database Systems 

never received any consent from the governed in any state to collect data on individuals (as the 

systems were put into place not by authority, but by grant money) it follows that the idea of 

having CEP study the possible removal of barriers to federal access of those databases, is an 

egregious overstep that even exceeds the overstep of the State Longitudinal Database Systems. 

Because federal FERPA regulations altered the original protective intent of FERPA, and 

removed the mandate that governments must get parental (or adult student) consent for any use 

of student level data, it seems that the idea of having CEP study and possible influence removal 

of additional "barriers" to federal use of data, is another egregious overstep. 

As a licensed teacher in the State of Utah; as co-founder of Utahns Against Common Core 

(UACC); as a mother of children who currently attend public, private and home schools; as 

acting president of the Utah Chapter of United States Parents Involved in Education (USPIE); as 

a patriot who believes in "consent of the governed" and in the principles of the U.S. Constitution; 

and, as a current tenth grade English teacher, I feel that my letter represents the will of many who 

stand opposed to the study of the removal of protective barriers on student-level data, which the 

CEP's website has outlined it will do. 

I urge this commission to use its power to strengthen local control of data, meaning parental and 

teacher stewardship over student data, instead of aiming to broaden the numbers of people with 

access to personally identifiable student information to include government agencies and/or 

educational sales/research corporations such as Pearson, Microsoft, or the American Institutes 

for Research. 

To remove barriers to federal access of student-level data only makes sense to a socialist who 

agrees with the Marc Tucker/Hillary Clinton 1998 vision of a cradle-to-grave nanny state with 

"large scale data management systems" that dismiss privacy as a relic in subservience to modern 

government. It does not make sense to those who cherish local control. 

It is clear that there is a strong debate about local control and about consent of the governed, 

concerning data and concerning education in general. NCEE Chair Mark Tucker articulated one 

side of the debate when he said: "the United States will have to largely abandon the beloved 

emblem of American education: local control. If the goal is to greatly increase the capacity and 

authority of the state education agencies, much of the new authority will have to come at the 

expense of local control." 

The Promise of Evidence-Based Policymaking 1135
Does that statement match the philosophical stand of this commission? I hope not. Local control 

means individual control of one's own life. How would an individual control his or her own 

destiny if "large scale data management systems" in a cradle-to-grave system, like the one that 

Tucker and Clinton envisioned, override the right to personal privacy and local control? It is not 

possible.

I urge this commission to use any influence that it has to promote safekeeping of unit-record data 

at the parental and teacher level, where that authority rightly belongs. 

Sincerely,

Christel Swasey 

1136 The Promise of Evidence-Based Policymaking
United States Parents Involved in Education (USPIE) is a nonprofit, nonpartisan, nationwide 

coalition of state leaders focused on restoring local control of education by eradicating federal 

intrusion. State leaders from around the country fed up with being ignored on education policy 

have joined forces to abolish the US Education Department and put an end to all federal 

education mandates.

USPIE endeavors to inform Americans of the trillions of dollars wasted on federal education in 

the last 35 years with nothing to show for it but stagnant, and declining test scores. It is the goal 

of USPIE to return American's education to its proper local roots and restore parental authority 

over their children's education.

USPIE’s STOP FED ED campaign is led by parents, taxpayers and educators committed to 

ending the U.S. Department of Education. The fight against Common Core has exposed the 

failures of those trying to force a federally-based one size fits all curriculum on states and local 

school districts. 

So we ask, “why even have a federal department?” Because it’s not about children. It’s about 

control. Control through federal dollars. And it’s big business. It’s about pushing an agenda. And 

it’s about ending something that had worked for years and replacing it with something no one 

even understands. 

For half a century now this experiment with federal control of local public schools has gone on 

and it’s failed. Let’s stop treating our children like rats in some social engineering laboratory and 

start treating children like children again. The first step is ending the Department of Education 

and that’s what STOP FED ED is all about. 

)@@

0DUPCFS
)FBSJOH

6OJUFE4UBUFT1BSFOUT*OWPMWFEJO&EVDBUJPO

1SFTFOUFS&SJO,OPXMFT

/PUF&SJO,OPXMFTXBTVOBCMFUPCFQSFTFOU
CVUTVCNJUUFEXSJUUFO

SFNBSLT

The Promise of Evidence-Based Policymaking 1137
- 1 - 

Commission on Evidence-Based Policymaking (CEP) 

First Public Hearing: Friday, October 21, 2016 

Comments Submitted by 

Christine M Keller, PhD 

Association of Public & Land-grant Universities (APLU) 

Vice President, Research and Policy Analysis 

ckeller@aplu.org

Abstract: The most significant barrier for evidence-based policymaking and decisions within higher education is 

the ban within the Higher Education Opportunity Act of 2008 that prohibits the Department of Education from 

collecting student-level data for postsecondary students. The lack of national student-level data prevents the 

Department from calculating comprehensive progress and completion outcomes for students as they move 

across different higher education institutions, especially as they cross state boundaries. It also prevents the 

linking of postsecondary data with federal data from other agencies that would allow better evidence of 

outcomes after college (e.g., earnings, employment) as well as outcomes for students in key federally-funded 

programs (e.g., Pell grants, veterans benefits). The result is that students and families are left in the dark as they 

make the critical decision of which college or university is the right fit; policymakers struggle to appropriately 

hold accountable institutions receiving taxpayer dollars; and institutions lack the information they need to assess 

their performance and improve. 

Oral and Written Statement 

Co-chairs Abraham and Haskins and distinguished members of the Commission, thank you for the opportunity to 

submit comments for your consideration during this hearing. My name is Christine Keller and I am the Vice 

President of Research and Policy Analysis at the Association of Public and Land-grant Universities (APLU). APLU 

is a research, policy, and advocacy organization dedicated to strengthening and advancing the work of public 

universities in North America. 

A top priority for APLU is to foster the widespread use of data and evidence to support decision-making - on 

university campuses and as the basis for sound and effective federal policy. APLU believes the most significant 

barrier for evidence-based policymaking and decisions within higher education is the ban that prohibits the 

Department of Education from collecting individual-level data for postsecondary students (Higher Education 

Opportunity Act of 2008). Lifting the ban in order to create a limited federal postsecondary student-level data 

system would produce more accurate details of student enrollment patterns, progression, completion, and postcollegiate outcomes. Such a system would assist with national priorities such as providing students and their 

families with more complete and accurate information when selecting a college. And better ensure that 

policymakers can appropriately allocate public resources and evaluate program effectiveness. 

A student-level postsecondary data system would address one of the most significant shortfalls of the current 

institution-level data collections within the Department of Education – the inability to accurately report the 

)@@

0DUPCFS
)FBSJOH

"TTPDJBUJPOPG1VCMJDBOE-BOE(SBOU6OJWFSTJUJFT

1SFTFOUFS$ISJTUJOF,FMMFS

1138 The Promise of Evidence-Based Policymaking
- 2 - 

progress and completion of all students across multiple institutions and state boundaries. The current federal 

graduation rate only reports the completion of full-time students who start and finish at their first institution. 

Students who start their studies part-time or transfer institutions are not reported in the federal rate. Data from 

the National Student Clearinghouse demonstrate that these students comprise an increasingly large proportion 

of today’s students. Sixty-four (64) percent of bachelor’s degree recipients from public universities attend more 

than one institution before graduation and more than 60 percent of community college students attend parttime. Yet these students are missing from the federal graduation rate. 

The Department of Education, to its credit, has made multiple attempts to address these shortcoming within the 

constraints of an institutional level collection. However, the information provided remains inadequate for 

consumers and policymakers, adds reporting burden for institutions, and, the latest attempt was judged too 

unreliable by the Department of Education to release the data publicly after the first year of collection. A 

student-level data collection would simplify the creation of progress and outcomes measures as well as increase 

reliability and consistency of the metrics across institutions. 

A second significant shortfall of the current postsecondary data is the inability to create linkages between 

postsecondary education data and other federal data systems. Linking with other federal data systems would 

harness the data already collected through other agencies to provide key information such as employment and 

earnings after college for all students. Progress and completion rates could be reliably and accurately 

calculated for student participating in federal programs such as Pell grants or GI Benefits. Combining information 

across federal agencies would streamline data collection, minimize duplicate reporting by institutions, and 

reduce the chance of errors in the resulting metrics. 

Any student-level data system must include a robust set of protections and protocols to safeguard student data 

from unauthorized use or disclosures and to secure its collection and storage. Policies and procedures to 

protect data must be transparent and utilize evolving best practices for data security to address real and 

legitimate concerns about privacy and security, but privacy and security should not be used as an excuse for 

blocking transparency and access to more complete data. 

Following are two specific examples from APLU members that illustrate why lifting the ban to create a system 

with more comprehensive and accurate data is a top priority for our association. 

Example 1: Student Achievement Measure 

As I mentioned earlier, the current federal graduation rate only includes first-time, full-time students who start 

and finish at their first institution. To help fill the data gaps in the federal system, the higher education 

community created the Student Achievement Measure (www.studentachievementmeasure.org) or SAM. SAM 

is a voluntary initiative that allows institutions to report the progress and completion of full-time, part-time, and 

transfer students. Over 600 colleges and universities from all 50 states and the District of Columbia are 

participating in SAM and reporting the outcomes of 600,000 more students than the federal government’s 

measure. 

The Promise of Evidence-Based Policymaking 1139
- 3 - 

One of the SAM participants is the University of North Texas. The federal graduation rate for UNT shows that 

just under half of first-time, full-time student who started in Fall 2009 graduated within 6 years. With only the 

information provided by the federal graduation rate, it appears that the other half of the students who started 

at the University of North Texas failed. However, by using the SAM methodology, UNT is able to show that 

another 13% of students graduated from another institution and another 14% are still enrolled pursuing their 

degree, for a total of 76% students who have graduated or are still enrolled. 

SAM also reports the outcomes for the nearly 3,000 students who started at the University of North Texas as 

transfer students in Fall 2009 and are missing from the federal rate – 80% have graduated or are still enrolled 

after 6 years. All totaled, SAM includes another 3,000 of the University of North Texas’s students who are not 

included in federal graduation rate. 

SAM provides a powerful model of the type of information that would be available if the ban on collection of 

student-level data were lifted. However, SAM is not a substitute for a federal student-level data system. SAM is 

voluntary and does not include all postsecondary institutions. Nor is SAM the official data included in the U.S. 

Department of Education’s College Scorecard, the U.S. Department of Veterans Affairs’ college comparison tool, 

or other consumer information tools. A more complete federal solution is still needed. 

Example #2: University of Texas System 

A second example comes from the University of Texas System. The UT System has demonstrated the usefulness 

of student-level data as part of their consumer information tool – seekUT (http://utsystem.edu/seekut/). By 

combining student-level data and the state workforce data, the UT System is able to present information such as 

the average cumulative student loan debt and median earnings at one, five, and ten years post-graduation for 

students graduating from specific programs at the UT institutions. 

However, as useful and powerful as having earnings data at the state level has been, there were key limitations 

that hindered the UT System’s ability to answer critical questions. Without a federal postsecondary data 

solution they were limited to data for those graduates that remained in Texas after graduation and unable to 

account for the earnings of graduates that move out of state. UT System administrators recognized that they 

needed national data across all states to evaluate and improve academic programs. And provide students with 

more comprehensive employment and earnings information to show a realistic picture of earnings after 

graduation. This information would help students make more informed decisions about their choice of majors 

and appropriate amounts of debt. 

In fact, the UT System felt that national employment and earnings data would be of such high value that they 

recently finalized an agreement with the US Census Bureau to provide national post-college outcomes for UT 

graduates through a pilot research project. The Census-UT System collaboration is an important demonstration 

of how higher education and federal agencies can break down silos and work together. Imagine how valuable 

would it be if all colleges and universities and state systems could have access to similar information to support 

institutional evaluation and improvement and student decision-making – without each entity negotiating a 

separate agreement? A national student-level data system could help all institutions more readily reach that 

goal. 

1140 The Promise of Evidence-Based Policymaking
- 4 - 

In sum, lifting the ban on the collection of student-level postsecondary data would provide access to more 

comprehensive, meaningful data and allow for the better alignment and integration with other federal data 

systems. The results would strengthen the federal government’s ability to provide essential information on 

higher education – for student and families to make more informed decisions about where to attend college; for 

policymakers to determine allocations of public resources and evaluate program effectiveness; and for college 

leaders to develop institutional policies and practices that support successful outcomes for all students. 

As the Commission continues to develop a strategy for increasing the availability, alignment, and use of high 

quality data to inform policy and decision-making, we encourage your consideration of the acute need for more 

accurate and complete postsecondary data for all users.

APLU is a research, policy, and advocacy organization dedicated to strengthening and advancing the work of public 

universities in the U.S., Canada, and Mexico. With a membership of 236 public research universities, land-grant institutions, 

state university systems, and affiliated organizations, APLU's agenda is built on the three pillars of increasing degree 

completion and academic success, advancing scientific research, and expanding engagement. Annually, its 194 U.S. 

member campuses enroll 3.9 million undergraduates and 1.2 million graduate students, award 1 million degrees, employ 1 

million faculty and staff, and conduct $40.2 billion in university-based research. 

The Promise of Evidence-Based Policymaking 1141
Ǧǣ

ǯ



ʹͳǡʹͲͳ͸



ǯǤ

ͳ ͅ͵ͻ

ǤͳͻǡͲͲͲǡǡ

ǡǡ

ǡǡǡǤ



ǦǤ



ǦǤ

Ǧǯ

ǦǤ

ǡ

ǤǡǣȋȌ

ǢȋȌ

ǢȋȌ

ǦǢȋȌ

ǦǢȋȌ

Ǥ

Ǧ

ǡǤ

ǡ

Ǧǡ

ǤʹͲͳͷǡ

ǡ

ǡ

ǡǡǡǦ

Ǥ

Ǥ

Ǥ



ǥ

ǡǡǤ

)@@"NFSJDBO4UBUJTUJDBM"TTPDJBUJPO-FUUFS

0DUPCFS
)FBSJOH

1SFTFOUFS$MZEF5VDLFS

1142 The Promise of Evidence-Based Policymaking
ǯǡǡǡ

Ǥ

ǡ



Ǥǡǡ




ȋȌǤǦ



Ǥ



Ǥǡ

ǦǤ

Ǥ	ǡǡ

ȋȌǡ

ǯ

Ǥ

ǡǡ

ǯ

ǡǡǡ

Ǥ	ǡ

Ǥ



ǡ

ǦǤǡǯ



ȋȌǤ

Ǥ



Ǥ

ǡ

ǦǤ

Ǧ

ǡǤ

Ǥ
ǡ

ǯǤǡ

ǡ

Ǥǡ

Ǥ



Ǥ

Ǧ

Ǥ

Ȅǡǡ

Ǥǡ

Ǥǡ

ǦǤ

The Promise of Evidence-Based Policymaking 1143


ǡ

ǣȀȀʹǤǤȀȀ̴
ǤǤ

Ǥ

ǡǦ

ǤǤ

ʹͲͳͷǣ

• ǣȀȀʹǤǤȀȀ	Ǥ

• ǣȀȀǤǤȀȀǦȀʹͲͳͷȀͲͳȀʹͻȀǦǦ

ǦǦǦǦǦǦǦǦǦǦǦǦ

Ǧ



• ǣȀȀǤǤȀȀǦǦǦȀ

• ǣȀȀǤǤȀ	ȀȀ	

Ǥ

• ǣȀȀǤǤȀȀʹͲͳͳȀͳͳȀͲͳȀǦȀ



• ǣȀȀǤǤȀȀʹ͵͸͹ͲȀǦǦǦǦǦ

ǦǦǦǦǦǦ

1144 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 07, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8swo-q69r

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0036

Comment on FR Doc # 2016-22002

Submitter Information

Name: Elaine Simons

General Comment

This is a gross misuse of our government. It is sick and disgusting that information that

is not the federal governments to posses and never should be stored by the federal

government is increasing by leaps and bounds. This is leading into a socialist and corrupt

government wanting to dictate and control its citizen's lives by data control. No ones

personal data should be stored by the government, but especially those of children and

private citizens. Also, I do believe if you study federal regulations, information can NOT

be stored on a United States Citizen for any length of time, unless that person is involved

in an ongoing investigation. In more ways than one, this is highly illegal and

unconstitutional. Reading this document is just a sick show of how far our government

has fallen into corruption and does not value freedom of citizens. As you can see I am

against data collection of this form and the storing of it by the federal government. Those

organizations who are pushing this and the government employees who are invested

should be held accountable for their corruption.

The Promise of Evidence-Based Policymaking 1145
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 07, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8swp-pyw3

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0037

Comment on FR Doc # 2016-22002

Submitter Information

Name: Glenn Freeman

General Comment

We do not need another bunch of government tax paid bureaucrats to track us. When you

get track of all those rag heads and other possible terrorists then you may have enough

smarts to try this, However, I would advocate for the blood of some patriots to oil the

wheels of freedom. Damn anybody who would scheme to take my freedoms - what few

are left.

1146 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 08, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sx3-u6vz

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0038

Comment on FR Doc # 2016-22002

Submitter Information

Name: DONNA GARNER

General Comment

POSTED BY DONNA GARNER - 11.8.16 -- I am deeply opposed to a student unitrecord system because the federal government should not be tracking and building

dossiers on American citizens. (1) The dangers of such a federal tracking system being

hacked by outsiders is very probable. (2) The federal government does not do an

adequate job of protecting personally identifiable data from being shared with third

parties. These very real concerns far outdistance any positive good that could come from

such a student unit-record system. 

The Promise of Evidence-Based Policymaking 1147
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 08, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sx4-twys

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0039

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

Please don't tell me that compiling information on children, with that knowledge base

lasting through their adult years, is being done for the good of the children. Very little

these days is done for children's benefits. It's all about adult needs and wants. How can

education be used by adults to make more money, build more power bases, and thus

climb on children's backs in the process. This smacks of science fiction stories about

government control over people's lives through benign efforts to "guide" them. This has

to stop. Right now. Right here.

1148 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 08, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sx5-ihmb

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0040

Comment on FR Doc # 2016-22002

Submitter Information

Name: Lisa Jones

General Comment

I OPPOSE a student unit-record system.

The federal government should not be tracking and building dossiers on American

citizens.

The likelihood of such a federal tracking system being hacked by outsiders is probable. 

The federal government does not do an adequate job of protecting personally identifiable

data from being shared with third parties. 

These very real concerns far outweigh any positive aspect that could come from

collecting this data. 

Parents (constituents) WON'T STAND FOR IT - and THEY will find out!!

The Promise of Evidence-Based Policymaking 1149
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 08, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sx7-zwvs

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0041

Comment on FR Doc # 2016-22002

Submitter Information

Name: Helen Morris

General Comment

See attached file(s)

Attachments

CEP_PREP_LetterACAP

1150 The Promise of Evidence-Based Policymaking
Commission on Evidence-Based Policymaking Comments 

Docket ID: USBC-2016-0003 

Federal Register Number: 2016-22002 

Alexandria Campaign on Adolescent Pregnancy 

421 King Street, Suite 400 

Alexandria, VA 22314 

November 8, 2016 

Dear Chair Katharine G. Abraham, Co-chair Ron Haskins, and members of the Commission: 

The Alexandria Campaign on Adolescent Pregnancy (ACAP) welcomes the opportunity to provide comments to 

the Commission on Evidence-Based Policymaking (CEP). We offer input for questions 16 and 18 based on our 

experience as a grantee of a high-quality evidence-based program, the Personal Responsibility Education 

Program (PREP). This program has been recognized as a pioneering example of tiered evidence-based 

policymaking, and it represents an important contribution to building a body of evidence of what works. This 

includes high quality implementation, evaluation, innovation, and learning from results. 

ACAP is a coalition made up of city agencies, organizations, and community members, including parents and 

youth, dedicated to reducing adolescent pregnancy through culturally and age appropriate education, advocacy, 

technical assistance, direct service prevention programs, and public awareness. Collaboratively, we work with 

our young people, their families, and the community to secure brighter futures. Our goal is to reduce the 

adolescent pregnancy rate in the City of Alexandria from 23.1 per 1000 females age 10-19 to 19.8 per 1000 

females age 10-19 by 2018. 

As a two-time PREP grantee, we incorporate evidence into all aspects of our project. The program model we 

implement was chosen because it has been rigorously evaluated and demonstrated to change behavior. 

We also have used evidence to guide and improve our project throughout the grant period. Through data and 

feedback from student focus groups and surveys, we were able to identify additional student needs and 

implement mindfulness and financial literacy lessons into the adult preparation component of our program. As a 

result, students have reported stronger money management skills, and a better ability to calm their mind and 

body and handle stress. 

Further, due to the success of the Evidence-Based Programs we use (Be Proud! Be Responsible! and Becoming 

a Responsible Teen), the data shows our students feel they have improved their self-efficacy regarding sexual 

behavior, including being able to say no to peer pressure, use a condom correctly, talk to their partner about 

using condoms, and use a condom every time they have sexual intercourse. 

In closing, we consider PREP to be a prime example of high-quality evidence-based policymaking, and we urge 

continuation and expansion of PREP in the coming budgets. It is one of the few government programs that use 

evidence and evaluation criteria throughout the grant life cycle. Thank you for considering our input for the 

CEP. If you have any questions or need additional information, please contact me at 703-851-3660 or 

zozilla1@gmail.com. 

Sincerely, 

Helen Morris 

Chair 

Alexandria Campaign on Adolescent 

Pregnancy 

The Promise of Evidence-Based Policymaking 1151
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 08, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sx9-zstx

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0042

Comment on FR Doc # 2016-22002

Submitter Information

Name: Regulatory Studies Center Peacock, Miller, & Prez

Address:

Washingtom, DC, 20052

Email: sofiemiller@gwu.edu

General Comment

The George Washington University Regulatory Studies Center (the Center) improves

regulatory policy through research, education, and outreach. As part of its mission, the

Center approaches regulatory problems from the perspective of the public interest and

occasionally responds to government requests for input. The Center provides these

comments regarding question numbers 1, 10, and 15-19 presented in the Supplementary

Information section of the Commission on Evidence-Based Policymaking's (the

Commission's) Federal Register notice issued on September 14, 2016. These comments

are organized in six sections. The first section is an introduction. Each subsequent

section corresponds to one or more of the Commission's questions. An additional section

at the end proposes specific findings, conclusions and recommendations for legislation

or administrative action that the Commission may want to include in its final report.

(See the attached file to continue reading)

Attachments

1152 The Promise of Evidence-Based Policymaking
GWRegStudiesCenter_Evidence-Based-Rulemaking

The Promise of Evidence-Based Policymaking 1153
The George Washington University Regulatory Studies Center ی 1

Public Interest Comment1

 to the 

Commission on Evidence-Based Policymaking 

Docket ID USBC-2016-0003; Docket No. 160907825–6825–01 

“Commission on Evidence-Based Policymaking Comments”

November 8, 2016 

Marcus C. Peacock2

Sofie E. Miller3 

Daniel R. Perez4

The George Washington University Regulatory Studies Center (the Center) improves regulatory 

policy through research, education, and outreach. As part of its mission, the Center approaches 

regulatory problems from the perspective of the public interest and occasionally responds to 

government requests for input. The Center provides these comments regarding question numbers 

1, 10, and 15-19 presented in the Supplementary Information section of the Commission on 

Evidence-Based Policymaking’s (the Commission’s) Federal Register notice issued on 

September 14, 2016. These comments are organized in six sections. The first section is an 

introduction. Each subsequent section corresponds to one or more of the Commission’s 

questions. An additional section at the end proposes specific findings, conclusions and 

recommendations for legislation or administrative action that the Commission may want to 

include in its final report. 

 

1

 This comment reflects the views of the authors, and does not represent an official position of the GW Regulatory 

Studies Center or the George Washington University. The Center’s policy on research integrity is available at 

http://regulatorystudies.columbian.gwu.edu/policy-research-integrity. The Center is located at 805 21st St. NW, 

Suite 612, Washington, DC 20052. 2

 Marcus C. Peacock is a Distinguished Research Professor at the George Washington University Regulatory 

Studies Center at 805 21st St. NW, Washington, DC. He can be reached at M_Peacock@gwu.edu. 3

 Sofie E. Miller is a Senior Policy Analyst at the George Washington University Regulatory Studies Center at 805 

21st St. NW, Suite 609, Washington, DC. She can be reached at SofieMiller@gwu.edu or at (202) 994-2975. 4

 Daniel R. Pérez is a Policy Analyst at the George Washington University Regulatory Studies Center at 805 21st

St. NW, Suite 609, Washington, DC. He can be reached at DanielPerez@gwu.edu or at (202) 994-2988. 

1154 The Promise of Evidence-Based Policymaking
The George Washington University Regulatory Studies Center ی 2

Introduction: Evidence-Based Regulation (EBR) 

Regulation may have a larger impact on society than any other single federal policymaking 

process. Regulations protect public health, promote economic growth, and help preserve our 

environment. Various estimates of regulation’s impact on society vary from over $260 billion to 

over $2 trillion.5 By comparison, the total of all federal funding for research and development, 

for instance, is less than $160 billion a year.6

The Regulatory Process Differs from Other Policymaking 

As the Commission examines strategies to better build evidence-based programs and policies 

throughout government, it is vital to understand the regulatory policymaking process already 

incorporates significant requirements regarding the collection, use and accessibility of data that 

differ from other policymaking processes. For instance, the Administrative Procedure Act of 

19467

 (the APA) requires regulatory agencies to both disclose, as well as request from the public, 

data or other information pertinent to a rulemaking.8

 Likewise, the APA compels agencies to 

justify most regulatory decisions based on the data, analyses, and other information collected and 

made part of a publicly available record. If, for instance, a decision appears “arbitrary and 

capricious” compared to the evidence in the public record the resulting regulation may be 

vacated.9

 

The APA is not the only important mandate affecting the collection, dissemination, and analysis 

of data during regulatory policymaking. Other requirements unique to regulations include, but 

are not limited to:10

x The Regulatory Flexibility Act of 1980 which requires agencies collect and assess data 

regarding the effect of major proposed regulations on small businesses; 

x The Unfunded Mandates Reform Act of 1995 which established a requirement to collect 

and analyze data regarding certain regulatory burdens on state and local governments;

 

5

 Maeve P. Carey, “Methods of Estimating the Total Cost of Federal Regulations,” Congressional Research 

Service, 21 January 2016, p. 2. See https://www.fas.org/sgp/crs/misc/R44348.pdf 6

 This estimate was produced by the American Association for the Advancement of Science in 2016. See 

http://www.aaas.org/sites/default/files/Function%3B.jpg 7

 Pub.L. 79–404, 60 Stat. 237. 8

 See, for instance, the requirements to disclose information at 5 U.S. Code § 552(a) and to request information at 5 

U.S. Code § 553(c). 9

 5 U.S. Code § 706(2)(A). 

10 This list adapted from Susan E. Dudley and Jerry Brito, Regulation: A Primer, Second Edition, Washington DC: 

The Mercatus Center and The George Washington Studies Center (2012), pp. 45-47. 

The Promise of Evidence-Based Policymaking 1155
The George Washington University Regulatory Studies Center ی 3

x The Small Business Regulatory Enforcement Fairness Act of 1996 requiring ex ante

evaluations of the impact of certain regulations on small businesses;

x The Congressional Review Act of 1996 requiring the submission of certain regulatory 

data and documentation to Congress;

x The Truth in Regulating Act of 2000 allowing Congress to request the Government 

Accountability Office evaluate certain proposed and final rules;

x Executive Orders 12866, 13563 and 13579, as well as OMB Circular A-4 regarding 

analyses that must be performed before certain rulemakings can be proposed or finalized; 

and

x These Executive Orders and Executive Order 13610 also encourage agencies to perform 

ex post reviews of the effectiveness of regulations.

In addition, there are other laws affecting data collection and use which, while not unique to the 

regulatory process, originated due to concerns regarding regulations. Such laws include the 

Paperwork Reduction Act of 1980 (affecting the government collection of information) and the 

Information Quality Act of 2000 (which established minimum requirements for the utility, 

integrity, and objectivity of information used by government). 

The unique data constraints placed on regulatory policymaking makes evidence-based regulation 

a distinct subset of evidence-based policymaking. It means that in some situations a 

recommendation that may benefit most methods of policymaking may be undesirable, or even 

illegal, in the rulemaking process. For instance, the Commission could recommend agencies seek 

out particular types of data and experts in order to help determine where federal grants may have 

the greatest impact. Regulatory agencies who follow formal, or adjudicatory rulemaking 

procedures, however, may be subject to charges of inappropriate ex parte communication if they 

undertook the same action.11 Even for informal, notice-and-comment rulemaking, final actions 

are often subject to litigation, which places additional constraints on the evidence in the record. 

The Commission may well need to make recommendations that are tailored to regulatory 

agencies or, at least, identify which recommendations do, or do not, apply to regulatory 

policymaking. 

 

11 Unlike designing a grant program, the prohibition of ex parte contact during certain rulemakings recognizes that 

making regulations can have the character of an adjudication with a decision ‘on the record’ by an impartial 

decision-maker. Because such contacts may not be monitored, they create a risk that the decision-maker’s 

neutrality may be compromised. For more information see Edward Rubin, “It’s Time to Make the Administrative 

Procedure Act Administrative,” Cornell L. Rev. 89:95 (2003). See 

http://scholarship.law.cornell.edu/cgi/viewcontent.cgi?article=2940&context=clr 

1156 The Promise of Evidence-Based Policymaking
The George Washington University Regulatory Studies Center ی 4

In order to assist the Commission in making recommendations specific to regulatory 

policymaking, the following comments focus solely on the regulatory process. The Center is 

available to assist the Commission in determining whether other recommendations it wishes to 

consider may or may not improve regulatory policymaking. 

A Framework for Evidence-Based Regulation 

Regulators should be able to demonstrate they are benefitting peoples’ lives by creating policies 

that address a “compelling public need,” as directed by Executive Order 12866.12 Increasing the 

use of evidence in making regulations will make agencies smarter, improve regulatory decisions, 

and, ultimately, result in better outcomes for society. Recognizing this, we offer the following 

integrated framework describing a system that produces evidence-based regulation (EBR) (see 

box below). An EBR process plans for, collects, and uses evidence throughout the life of a 

regulation to predict, evaluate and improve outcomes. 

The framework is structured around the three main phases of regulating: design, decisionmaking, and retrospective review. It creates a feedback loop (through retrospective review) 

during implementation of the rule so that data are not only used in developing the regulation but 

also in periodically reassessing its value and modifying the rule as appropriate. Importantly, this 

framework incorporates important and current requirements of the federal rulemaking process 

pertinent to the collection and use of data. 

While it is not necessary for the Commission to endorse the EBR Framework, the Framework 

provides a coherent integrated system for answering a number of the Commission’s specific 

questions. 

 

12 E.O. 12866, §1: Statement of Regulatory Philosophy and Principles. 

The Promise of Evidence-Based Policymaking 1157
The George Washington University Regulatory Studies Center ی 5

 

Evidence-Based Regulation Framework

I. Regulatory Design 

A. Identify the problem (state the “compelling public need”). 

B. Evaluate whether modifications to existing rules can address the problem. 

C. Identify and assess available alternatives to direct regulation. 

D. If regulating, determine that the preferred alternative addresses the problem.

E. Set clear performance goals and metrics for outputs and outcomes. 

F. Exploit opportunities for experimentation. 

G. Plan and budget for retrospective review. 

II. Regulatory Decision-making 

A. Assess the expected benefits, costs, and other impacts. 

B. Clearly separate scientific evidence from policy judgments. 

C. Make relevant data, models and assumptions available to the public. 

III. Retrospective Review 

A. Reassess planned retrospective review and modify if necessary. 

B. Gather necessary data on regulatory outputs and outcomes. 

C. Implement retrospective review plan. 

D. Compare measured outcomes to original performance goals. 

E. Reassess the rule using new information and the factors in the regulatory design. 

1158 The Promise of Evidence-Based Policymaking
The George Washington University Regulatory Studies Center ی 6

Questions 1 & 15 Regarding Challenges, Barriers, and Solutions 

Question 1. Are there successful frameworks, policies, practices, and methods to overcome 

challenges related to evidence-building from state, local, and/ or international governments 

the Commission should consider when developing findings and recommendations regarding 

Federal evidence-based policymaking? If so, please describe. 

Question 15. What barriers currently exist for using survey and administrative data to 

support program management and/or evaluation activities? 

Questions 1 and 15 are combined since two of the barriers we identify in response to question 15 

can be overcome by policies, practices and other methods that we also identify for question 1. 

Specifically, agency noncompliance with internal administrative requirements and inadequate 

funding of program evaluation are two barriers the regulatory system currently faces in 

collecting and using data to improve regulations and offers potential solutions to each one. 

The Challenge of Noncompliance with Internal Directives 

A barrier to evidence-based regulation is a lack of faithful compliance with internal 

administrative requirements. For instance, since 1981 presidents have required regulators who 

were considering a new regulation to identify and disclose the problem they intended to solve by 

regulating and assess different regulatory alternatives to solving that problem (these are items 

I.A. and I.C. under “Regulatory Design” in the EBR Framework shown above). In addition, each 

president since Jimmy Carter has required regulators to assess and disclose both the expected 

benefits and the expected costs of the regulatory alternatives (the estimation of both benefits and 

costs is shown in item II.A. in the EBR Framework). 

Identifying the problem to be solved is a prerequisite for designing a regulation that provides net 

social benefits13 and for evaluating the effectiveness of a rulemaking once it is in place. Absent a 

clearly identified market failure, regulation and other forms of government intervention can 

disrupt competition, and lead to misallocation of resources.14 Thus, targeting a fundamental 

problem rather than relying on anecdotes to support regulation is important, not only for 

regulatory design but for knowing what data to collect. Likewise, laying out policy alternatives15

 

13 According to E.O. 12866, “Each agency shall identify the problem that it intends to address (including, where 

applicable, the failures of private markets or public institutions that warrant new agency action) as well as assess 

the significance of that problem.” (Principles of Regulation, Sec.1(b)(1)) 14 Office of Management and Budget, “Circular A-4: Regulatory Analysis.” September 17, 2003. 

https://www.whitehouse.gov/omb/circulars_a004_a-4 15 E.O. 12866 states, “Each agency shall identify and assess available alternatives to direct regulation, including 

providing economic incentives to encourage the desired behavior, such as user fees or marketable permits, or 

The Promise of Evidence-Based Policymaking 1159
The George Washington University Regulatory Studies Center ی 7

and using data to assess expected benefits and costs16 is a fundamental method of informing 

decision-makers. Nonetheless, in 2014 the Government Accountability Office estimated that less 

than a fourth of new significant rules disclosed these four basic presidential requirements.17 

A more recent example of agency noncompliance with internal administrative requirements 

entails the retrospective review of regulations (items I.G. and III. in the EBR Framework). Every 

president since Jimmy Carter has required the ex post evaluation of regulations (retrospective 

review). Most regulatory decisions rely on predictive models and assumptions, but rarely are 

those hypotheses evaluated based on real world evidence.18 A requirement to evaluate whether 

predicted effects of regulations were realized would provide a powerful incentive to improve ex 

ante regulatory impact analyses, as well as improve regulations that are already in place.19 

With this in mind, in 2011 and 2012 President Barack Obama signed three Executive Orders 

attempting to get agencies to more aggressively adopt the retrospective review of regulations: 

Executive Order 13563 “Improving Regulation and Regulatory Review,”

20 which reinforced the 

requirements of Executive Order 12866; Executive Order 13579,21 which expanded the 

requirements to independent regulatory agencies; and Executive Order 13610, which emphasized 

that “further steps should be taken...to promote public participation in retrospective review.”

22,23

 

providing information upon which choices can be made by the public.” (Sec.1(b)(3)) 16 E.O. 12866 states, “Each agency shall assess both the costs and the benefits of the intended regulation and, 

recognizing that some costs and benefits are difficult to quantify, propose or adopt a regulation only upon a 

reasoned determination that the benefits of the intended regulation justify its costs.” (Sec.1(b)(6)) 17 Government Accountability Office, “Federal Rulemaking: Agencies Included Key Elements of Cost Benefit 

Analysis, but Explanations of Regulations’ Significance Could Be More Transparent,” September 2014, p. 18. 

See http://www.gao.gov/assets/670/665745.pdf 18 See, for instance, a discussion of the assumptions and models used to establish the ozone national ambient air 

quality standard at Susan E. Dudley, Regulatory Science and Policy: A Case Study of the National Ambient Air 

Quality Standards, The George Wash. Univ. Reg. Studies Ctr. (September 2015), pp. 11-12. 

http://regulatorystudies.columbian.gwu.edu/regulatory-science-and-policy-case-study-national-ambient-airquality-standards 19 Susan E. Dudley, A Retrospective Review of Retrospective Review, The George Wash. University Regulatory 

Studies Center (May 2013), p.2. See http://regulatorystudies.columbian.gwu.edu/files/downloads/20130507-aretrospective-review-of-retrospective-review.pdf 20 Executive Order 13563 was followed by implementation guidance. See Memorandum from OIRA Administrator 

Cass Sunstein to the Heads of Executive Departments and Agencies, “Retrospective Analysis of Existing 

Significant Regulations,” 25 April 2011 at 

https://www.whitehouse.gov/sites/default/files/omb/memoranda/2011/m11-19.pdf 21 Executive Order 13579, “Regulation and Independent Regulatory Agencies.” July 14, 2011. 76 FR 41587. 22 It should be noted that, in addition to these Executive Orders, some laws require the retrospective review of 

certain regulations. For instance, section 812 of the Clean Air Act Amendments of 1990 requires the 

Environmental Protection Agency to periodically assess the benefits and costs of regulations promulgated under 

the Act. 

1160 The Promise of Evidence-Based Policymaking
The George Washington University Regulatory Studies Center ی 8

However, an independent review of high-impact rules issued in 2014 found that the key 

requirements in these directives were seldom followed.24 For example, the identification of 

measurable metrics that could be subsequently used to evaluate the impacts of rules were only 

identified in one-third of the regulations, and even fewer for rules issued by independent 

agencies. To be clear, this is not a recent problem. As a general matter, such levels of 

noncompliance with presidential Executive Orders and other internal Executive Branch guidance 

in modern times are not unusual.25

Solving Noncompliance through Independent Review, Codification, and 

Competition 

In examining how to improve the performance of people working in government bureaucracies, 

management expert William Medina has laid out three ways to change behavior:26

x compel them (forced change);

x persuade them (through education); and/or

x change their incentives.

A recent review of a lack of faithful compliance with government-wide reforms in U.S. federal 

agencies over a period of fifty years found three possible ways to improve behavior: create 

independent organizations to help execute the rules; codify administrative requirements into law; 

and create competition.27 The first two methods force change while the third attempts to change 

incentives. 

 23 Executive Order 13610, “Identifying and Reducing Regulatory Burdens.” May 10, 2012. 77 FR 28469. 24 Sofie E. Miller, “Learning from Experience: Retrospective Review of Regulations in 2014,” Working Paper, The 

George Washington University Regulatory Studies Center, November 2015. As a general matter, other 

researchers have also concluded that there is generally a lack of compliance with retrospective review 

requirements. See Reeve T. Bull, “Building a Framework for Governance: Retrospective Review and 

Rulemaking Petitions,” Admin. L. Rev., 67:265 (2015). 25 See, for instance, the lack of compliance with eight government-wide reforms since 1965 discussed in Marcus C. 

Peacock, “Improving the Accountability of Federal Regulatory Agencies Part II: Assessing Eight Governmentwide Accountability Reforms,” 28 June 2016. This can be accessed at 

https://regulatorystudies.columbian.gwu.edu/sites/regulatorystudies.columbian.gwu.edu/files/downloads/RegInsi

ght_Peacock-Reforms-Improving-Accountability_pt2.pdf 26 William A. Medina, Changing Bureaucracies: Understanding the Organization Before Selecting the Approach, 

Marcel Dekker, Inc: New York, NY (1982), pp. 118-119. 27 See Marcus Peacock, “Improving the Accountability of Federal Regulatory Agencies Part III: What Reforms 

Work Best,” Regulatory Insight, The George Washington University Regulatory Studies Center, 12 September 

2016, p. 22. See http://bit.ly/2fvzFLg 

The Promise of Evidence-Based Policymaking 1161
The George Washington University Regulatory Studies Center ی 9

Strengthening Independent Review 

There are many examples of governments tackling the problem of internal noncompliance by 

creating independent organizations to either monitor compliance (such as the Inspectors General) 

or to faithfully execute the requirements themselves. A specific example of the latter strategy is 

in the European Union (EU).28 Concerns regarding a lack of compliance with internal guidelines 

requiring the self-evaluation of the effectiveness of policies29 resulted in the EU creating a 

separate ex post evaluation body. This new organization is completely independent from the 

member nations and reports directly to the European Parliament.30 

Independent review does not necessarily entail creating a new entity. The Commission may wish 

to consider, for instance, enlisting the U.S. court system to improve compliance. Judicial review 

has been largely successful in achieving compliance with the public notice and evidentiary 

requirements codified in the APA (discussed above). Agencies know their regulations can be 

nullified unless they can convince a court that the standards of transparency and assessment set 

out in the APA have been met. Expanding the existing judicial review of regulations to include 

one or more elements of the EBR Framework, such as determining whether a final rule includes 

an adequate plan for retrospective review, would undoubtedly improve compliance with those 

elements. Relying on the courts would also avoid the cost of creating a new entity within the 

federal government. 

Codification of Accepted Practices 

Another approach to motivating agencies to comply with internal administrative requirements is 

to codify such requirements in law. For instance, the last section below includes a 

recommendation that elements of the EBR Framework that have been adopted by consecutive 

presidents over a long period of time be more firmly institutionalized by putting them in law. 

This would be an incremental step in improving compliance as it would increase their 

permanence and subject compliance to greater oversight, particularly by Congress. 

 

28 Céline Kauffmann, “The OECD Perspective on Good Regulatory Practices and International Regulatory 

Cooperation,” Commentary, The George Washington University Regulatory Studies Center, 19 December 2014. 

Available at https://regulatorystudies.columbian.gwu.edu/oecd-perspective-good-regulatory-practices-andinternational-regulatory-cooperation 29 See, in particular, Official Journal of the European Union, Court of Auditors, “Special Report No. 1/2006 on the 

contribution of the European Social Fund in combating early school leaving, together with the Commission’s 

replies,” 2006/C 99/01. This audit found that agencies allocating funding for the purpose of keeping students in 

school generally did not utilize readily available performance data. 30 This is the Ex Post Impact Assessment Unit in the European Parliamentary Research Service. See European 

Parliament, “Evaluation and ex-post impact assessment at EU level,” September 2016 available at 

http://www.europarl.europa.eu/RegData/etudes/BRIE/2016/581415/EPRS_BRI(2016)581415_EN.pdf 

1162 The Promise of Evidence-Based Policymaking
The George Washington University Regulatory Studies Center ی 10

For example, Senators Heidi Heitkamp (D-N.D.) and James Lankford (R-Okla.) have proposed 

the Smarter Regulations Act31 which would require agencies to include in major rules a 

framework for reassessing the rule, including the timeframe for reassessment,32 the metrics that 

should be used to gauge efficacy,33 and a plan to gather relevant data to compile these metrics.34

The framework established in this proposed legislation was approved by a Senate committee by 

voice vote in October 2015.35 The bill is consistent with the EBR Framework and our 

recommendation below. 

Changing Incentives by Creating Competition 

It would be a mistake to assume that creating an independent organization or codifying best 

practices would completely solve the problem of unfaithful execution. For instance, the Office of 

Information and Regulatory Affairs (OIRA) was created in the U.S. Office of Management and 

Budget (OMB), in part, to better enforce administrative benefit-cost analysis requirements on 

regulatory agencies. Yet compliance with these standards remains far from perfect.36 

In addition to relying on independent organizations and codification to help defeat unfaithful 

execution, it may be effective to change the incentives of federal agencies by making them 

compete with each other or other entities. Competition is long been recognized as an extremely 

powerful motivator of federal agencies.37 While it may not seem obvious, federal agencies 

already compete with each other. For instance, they are in a constant and robust competition to 

maintain or increase their budgets. As proof of competition’s effects, this long running 

competition for funding has resulted in a panoply of clever budget strategies.38 

One way to create a healthy competition among federal agencies is to use comparison data. 

While their effects may vary, comparison data has been shown to be a strong motivator in state 

governments39 particularly if the data are accessible and trustworthy. Indeed, federal agencies 

 

31 Smarter Regs. Act of 2015, S. 1817, 114th Cong. (2015). 32 S. 1817, § 2(f)(1)(D). 

33 S. 1817, § 2(f)(1)(B). 

34 S. 1817, § 2(f)(1)(C). 

35 For more information on S. 1817 see https://www.congress.gov/114/crpt/srpt282/CRPT-114srpt282.pdf 

36 Patrick McLaughlin, Jerry Ellig, John Morrall, “Continuity, Change, and Priorities: The Quality and Use of 

Regulatory Analysis Across U.S. Administrations,” Regulation and Governance, June 1, 2013. See 

http://onlinelibrary.wiley.com/wol1/doi/10.1111/j.1748-5991.2012.01149.x/full 37 See, for instance, William A. Niskanen, “Competition among Government Bureaus,” in Carol H. Weiss and 

Allen H. Barton, editors, Making Bureaucracies Work, Sage Publications: Beverly Hills, CA (1980), pp. 167-

174. 38 Aaron Wildavsky, The Politics of the Budgetary Process, Fourth Edition, Little, Brown and Company: Boston, 

MA (1984), pp. 64 – 84. 39 See E. Blaine Liner, Harry P. Hatry, Elisa Vinson, Ryan Allen, Pat Dusenbury, Scott Bryant, Ron Snell, “Making 

The Promise of Evidence-Based Policymaking 1163
The George Washington University Regulatory Studies Center ی 11

themselves are increasingly using comparison data to change the incentives of the entities they 

regulate including everything from colleges40 to nursing homes41 to chemical manufacturers.42

One idea would be to look for federal programs that have very similar goals but achieve them in 

different ways, such as through grants, regulations, tax credits, and/or loan guarantees.43 A third 

party, the Government Accountability Office, for instance, could then collect data regarding the 

efficiency of each program and rank the various programs on this criterion. This may mean, for 

instance, estimating how many homeless families are provided long-term shelter for each dollar 

spent, or how many unemployed persons get and retain a job for each dollar spent. 

One might initially expect large differences in the results agencies achieve. For instance, in 2003 

a back of the envelope comparison of flood mitigation programs showed that for the same 

federal expenditure a Department of Agriculture grant program appeared to produce 40 percent 

more floodplain protection benefits than a Federal Emergency Management Agency grant 

program.44 The periodic publication of such data from a reliable source could result in agencies 

having strong incentives to collect, analyze, and act on evidence so as to improve their program 

and achieve a better ranking.45 Evidence-based policymaking could become the method by 

which agencies in compete in a “race to the top.” 

The Problem of Inadequate Funding 

Another barrier to evidence-based regulation is funding for ex ante and ex post analysis and 

evaluation. Like the barrier of noncompliance, this problem is not unique to EBR but can block 

the collection and evaluation of data regardless of program. It may be that some of the 

substantial resources currently spent on ex ante regulatory review could be more prudently 

shifted to conducting a retrospective review of federal rules.46 Such a reallocation could in turn 

 

Results-based Government Work,” The Urban Institute, April 2001, p. 18. Available at 

http://www.urban.org/sites/default/files/alfresco/publication-pdfs/310069-Making-Results-BasedGovernmentWork.PDF 40 The U.S. Department of Education’s “College Scorecard” at https://collegescorecard.ed.gov/ 41 The Medicare program’s “Nursing Home Compare” ratings at 

https://www.medicare.gov/NursingHomeCompare/About/Ratings.html 42 Jason Scorse, 2003, “Do Pollution Rankings Affect Facility Emissions Reductions?: Evidence From The Toxic 

Release Inventory,” dissertation retrieved from 

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.511.173&rep=rep1&type=pdf 43 Budget subfunctions may be a method for narrowing these programs down. See https://www.whitehouse.gov/taxreceipt/functions 44 U.S. Office of Management and Budget, Addendum to the Fiscal Year 2006 Budget. See 

https://www.whitehouse.gov/omb/memoranda_m02_06_addendum 45 In some respects, the “Best Places to Work in the Federal Government” rankings released by the Partnership for 

Public Service provide a model for such a system of comparison. See http://bestplacestowork.org/BPTW/ 46 See Susan E. Dudley, A Retrospective Review of Retrospective Review, The George Wash. University 

1164 The Promise of Evidence-Based Policymaking
The George Washington University Regulatory Studies Center ی 12

strengthen ex ante analyses by providing direct information on the causal outcomes one would 

expect as the result of regulatory policy.47

Three Possible Solutions to the Problem of Inadequate Funding 

One means of accomplishing this goal without significantly altering the federal budget is for 

Congress and OMB to more readily allow the reallocation of resources from current ex ante

regulatory impact analyses to gathering the data and evaluation tools necessary to subsequently 

test ex ante predictions. This may simply require the appropriation of less “one-year money” and 

more “multi-year money” to allow agencies greater flexibility in when they use their budget 

authority.48 Right now the vast majority of funding for analyses is spent upfront and very little is 

used after rules are promulgated. It seems extremely unlikely this is an optimal balance. 

Another possible solution is to allow, or require, a small percentage of funds be set aside for 

program evaluation or for policies based on program evaluation. This is not unprecedented. In 

1978 Congress allowed the U.S. Department of Agriculture (USDA) to set aside up to 0.5 

percent of the program funds allocated for its Special Supplemental Nutrition Program for 

Women, Infants, and Children (WIC) to evaluate the program’s performance, including 

experimenting with different pilot projects.49 More recently, the Senate Appropriations Bill for 

FY 2014 allowed five percent of mental health block grants to states be used for “evidence-based 

programs that address the needs of individuals with early serious mental illness, including 

psychotic disorders.”

50

Constrained budgets tend to result in agencies “curtailing the funds needed for evaluation studies 

and performance monitoring systems.”

51 However, there is considerable evidence that the use of 

evaluation not only leads to improved regulatory outcomes, but also provides additional benefits 

for nonregulatory agencies—particularly those operating in an environment of stagnant or 

decreasing budgets. For example, Newcomer et al. detail several instances where the results of 

 

Regulatory Studies Center (May 2013), p.2 at 

http://regulatorystudies.columbian.gwu.edu/files/downloads/20130507-a-retrospective-review-of-retrospectivereview.pdf . We also recommend the Commission examine recommendations from the Administrative 

Conference of the United States regarding the use of resources in conducting retrospective reviews, which can be 

found at https://www.acus.gov/sites/default/files/documents/Recommendation%25202014-

5%2520%2528Retrospective%2520Review%2529_1.pdf 47 Susan E. Dudley, “Quantifying Regulatory Efficacy,” background paper for OECD Workshop on Socioeconomic 

Impact Assessment of Chemicals Management. Forthcoming. 48 “One-year money” is budget authority that expires at the end of the fiscal year in which it was appropriated. 49 Newcomer, Kathryn E., Harry P. Hatry editor, and Joseph S. Wholey editor. 2015. Handbook of Practical 

Program Evaluation. Hoboken, NJ: Jossey-Bass p. 807 50 S. 1284, Report No. 113–71, p. 114. 51 Newcomer et al., p. 807. 

The Promise of Evidence-Based Policymaking 1165
The George Washington University Regulatory Studies Center ی 13

evaluation data on program performance caused agencies to shift funding and effort away from 

less successful programs towards better-performing initiatives. The data made available to 

Congress regarding success in achieving outcomes allowed agencies to maintain or even expand 

their programs during periods of significant cuts in federal domestic spending during the 1980s. 

These programs included: the Department of Labor’s Job Corps program and the aforementioned 

WIC program at USDA.52 

Finally, it is important to note that the cost of both ex ante and ex post analyses and evaluation 

need not be high. An important principle is that the cost of conducting a regulatory analysis 

should reflect the potential value of such analysis and, if necessary, can be quite inexpensive.53

Joseph Wholey proposes that evaluators use “a sequential purchase of information” approach 

such that “resources are invested in further evaluation only when the likely usefulness of the new 

information outweighs the costs of acquiring it.”

54 EBR would benefit from such flexible 

standards regarding what constitutes useful analysis and evaluation. 

Questions 10 & 16: Access and Use of Evidence 

Question 10. How should the Commission define “qualified researchers and institutions?”

To what extent should administrative and survey data held by government agencies be made 

available to “qualified researchers and institutions?”

Question 16. How can data, statistics, results of research, and findings from evaluation, be 

best used to improve policies and programs? 

This response addresses questions 10 and 16 regarding access and use of evidence. In regulatory 

processes agencies are compelled, with narrow exceptions, to make data, analysis and other 

evidence used by decision-makers available to the public. As noted above, agencies must place 

information they use in decision-making in a public record and be able to justify their decisions 

based on the evidence in that record. 

We support the bedrock regulatory principle of openness and this is reflected in item II.C. in the 

EBR Framework under Regulatory Decision-making. Thus, in answer to question 10, with 

regards to information that will be used to make regulatory decisions, as much information as 

 

52 Ibid, p. 829. 53 Christopher Carrigan and Stuart Shapiro, “What’s wrong with the back of the envelope? A call for simple (and 

timely) benefit–cost analysis,” Regulation and Governance, 26 April 2016. 54 Newcomer et al., p. 89. This approach is one of several suggestions contained within Wholey’s framework of 

Evaluability Assessment which proposes several techniques for evaluators to leverage low cost information 

significant program improvement. 

1166 The Promise of Evidence-Based Policymaking
The George Washington University Regulatory Studies Center ی 14

possible should be made available and it should be made available to everyone. The public has a 

right to know what evidence policy officials consider in making decisions that affect them.

We offer two answers to question 16, regarding the “best use” of “data, statistics, result of 

research, and findings from evaluation.” The first relates to the need for transparency in 

regulatory decision-making. The second answer regards how evidence may be best organized to 

promote its best use. 

Data and Findings Must be Separated from Policy 

Regulatory agencies are generally compelled to request information and other data from the 

public. However, the opportunity for public comment should include access to the various data, 

statistics, findings and other information the agency is using to make a regulatory decision. The 

“best use” of information will likely occur only after it is scrubbed by public review. 

The EBR Framework addresses important guidance on how data and other evidence should be 

used and communicated. In particular, in regulatory decision-making the presentation of 

evidence must be separated from policy decisions so that the public understands what is a fact 

(what is) and what is a policy judgment (what ought to be). This has important implications for 

public access to the data, models and assumptions used to make regulatory decisions, particularly 

when it comes to scientific information. 

The boundary between objective science and policymaking is inherently fuzzy.55 Creating clarity 

regarding where this boundary is and the role of scientists at this boundary is important.56 In our 

democracy, the public must be able to hold regulatory policymakers, typically the president and 

his or her appointees, accountable for their decisions. It is for this reason the regulatory process 

already mandates requirements for policymakers to reveal and explain how they reached a 

regulatory decision based on publicly available evidence. This process assumes the public is able 

to separate the evidence the decision-maker considered from the judgments they made. 

Evidence-based policy expert Ray Pawson explains: 

Evidence does not deliver decisions; its function is to deliver decision support. 

When evidence is called into play in policy formation, it is never a case of simply 

‘following the evidence’ but rather one of ‘interpreting the evidence’ and then 

 

55 See, for instance, the discussion in Susan E. Dudley and George M. Gray, “Improving the Use of Science to 

Inform Environmental Regulation,” in Jason S. Johnson (ed.), Institutions and Incentives in Regulatory Science,

New York, NY: Lexington Books (2012), pp. 165-198. 56 See Ann Campbell Keller, Science in Environmental Policy: The Politics of Objective Advice, MIT Press: 

Cambridge, MA (2009). 

The Promise of Evidence-Based Policymaking 1167
The George Washington University Regulatory Studies Center ی 15

‘adapting the evidence’ to local circumstances. No method of synthesis can tell 

the policy maker what to do.57 

Given both the fuzzy boundary between evidence and policy and the need to keep scientific and 

policy judgments as separate as possible for reasons of accountability, the solution is for 

regulatory agencies to be as open as possible regarding the decisions they make. Recounting his 

experience as the Administrator of the EPA from 1977 to 1980, Doug Costle has explained:

People tend to think science is hard and numerical and precise. It’s not, 

particularly in the environmental area. But there is one way, and only one way, to 

deal with that, and that is just to be absolutely open and honest about the gray 

areas. Anyway you cut it, we’re making judgments, social policy judgment 

calls...58

An example of conflating evidence and policy is application of the precautionary principle. In 

short, the precautionary principle advocates for the use of preemptive regulation in the face of 

scientific uncertainty regarding possible threats to the health of humans or ecosystems.59 The 

application of the precautionary principle is not a purely scientific decision. Indeed, it confuses 

scientific uncertainty with scientific ignorance and is squarely inconsistent with an approach 

built on a foundation of evidence. As Ray Pawson has pointed out: 

The precautionary principle betokens a move from evidence to advocacy. It 

forecloses debate and stifles the search for further evidence. By definition the zero 

emission, zero concentration, zero tolerance standards are not empirically 

derived—they concede that the evidence is not yet in.60

 

57 Ray Pawson, Science of Evaluation: A Realist Manifesto, Thousand Oaks, Calif.: SAGE (2003), p. 190. 58 As quoted in Ronald Brand, Thomas Kelly, A. Stanley Meiburg, Robert Wayland, Susan Wayland, David 

Ziegele, True Green: Executive Effectiveness in the U.S. Environmental Protection Agency, Gerald A. Emison 

and John C. Morris, eds., Lexington Books: Lanham, MD (2012), p. 77. In his seminal work on the fuzzy 

boundary between scientific evidence and policy (“Science and Trans Science,” Minerva,10(2): 209-222, 1972), 

Alvin Weinberg put it another way, “Though the scientist cannot provide definite answers to trans-scientific 

questions any more than can the lawyer, the politician or a member of the lay public, he does have one crucially 

important role: to make clear where science ends and trans-science begins.” 59 Martuzzi, M. and Tickner, J. (eds) (2004) The precautionary principle: Protecting public health, the environment 

and the future of our children. Copenhagen, Denmark: World Health Organization, Europe. 60 Pawson, p. 174. 

1168 The Promise of Evidence-Based Policymaking
The George Washington University Regulatory Studies Center ی 16

Muddled Fact and Policy Causes Problems 

Despite the necessity of separating what is from a decision regarding what ought to be, scientific 

evidence and policy decisions have become increasingly muddled.61 This results in a host of 

significant problems including degrading the perceived integrity of evidence-based 

policymaking. As the Bipartisan Policy Center notes: 

Policy makers often claim that particular regulatory decisions have been driven 

by, or even required by science; their critics, in turn, have attacked the quality or 

the interpretation of that science. Such conflict has left the U.S. with a system that 

is plagued by charges that science is being “politicized” and that regulation lacks 

a solid scientific basis. As a result, needed regulation may be stymied, dubious 

regulations may be adopted, issues can drag on without conclusion and policy 

debate is degraded. Moreover, the morale of scientists is weakened, and public 

faith in both government and science is undermined.62 

The Bipartisan Policy Center concludes that “a tendency to frame regulatory issues as debates 

solely about science, regardless of the actual subject in dispute, is at the root of the stalemate and 

acrimony all too present in the regulatory system today.”

63 

Clear Separation and Broad Access Solves This Problem 

The EBR Framework calls for the separation of these elements during regulatory decisionmaking (see item II.B.). If not clearly separated, the increased use of evidence may ironically 

harm, rather than improve, the integrity of the regulatory process. As the Bipartisan Policy 

Center concluded, “the Administration needs to devise regulatory processes that, in as many 

situations as possible, could help clarify for both officials and the general public which aspects of 

disputes are truly about scientific results and which concern policy.”

64 “This transparency would 

both help force values debates into the open and could limit spurious claims about, and attacks 

on science.”

65 

 

61 The scientific community increasingly wrestles with fact more and more scientists are being encouraged to 

become engaged with the public policy process. See, for instance, Deborah Runkle, Mark S. Frankel ed., 

“Advocacy in Science: Summary of a Workshop convened by the American Association for the Advancement of 

Science,” 1 May 2012, pp. 2-3. 62 Bipartisan Policy Center, Improving the Use of Science in Regulatory Policy, Washington (DC): Bipartisan 

Policy Center; 2009;10. Available at: 

http://www.bipartisanpolicy.org/sites/default/files/BPC%20Science%20Report%20fnl.pdf 63 Ibid., p. 11. 64 Ibid., p. 4. 65 Ibid., p. 5. 

The Promise of Evidence-Based Policymaking 1169
The George Washington University Regulatory Studies Center ی 17

Given the need to make it clear what the data show vs. what policymakers decide, the public 

should have as broad an access to data, statistics, results of research, and findings from 

evaluation as possible so that people have the ability to make their own judgments regarding the 

interpretation of data. President Obama’s March 2009 Scientific Integrity Memo supports this 

goal, stating that “[t]o the extent permitted by law, there should be transparency in the 

preparation, identification, and use of scientific and technological information in 

policymaking.”

66 

Access to the “results of research” should include risk assessments, models, and the assumptions 

that were used to synthesize data for the purpose of making regulatory decisions. The National 

Research Council has concluded that there should be “unrestricted access” to public-use data that 

pose no confidentiality problems.67 This category should also include any models and other 

analytic tools used to assess data that, by their nature, do not pose concerns about the breach of 

individual, household or other confidential personal information. If such a tool was used to 

materially inform a regulatory decision, the public should have access to that tool. As is being 

shown in the case of opening up competing proprietary climate change models, scrutiny from 

others will very likely improve the models’ credibility and accuracy and result in the data’s “best 

use.”

68 

Access to Evidence Organized by ‘Program Theory’ Could Benefit Regulators 

The “best use” of evidence can also be improved by how evidence is organized. Regulatory 

evaluations are often categorized under their substantive program area (e.g., environment, health, 

or education). As a practical matter this can limit the amount of data that is consulted during 

regulatory design and decision-making, such as during the consideration of alternatives (see item 

I.C. in the EBR Framework). Categorizing evaluation data under the additional criteria of similar 

program theory domains (e.g., incentives, target setting, or behavior change) could greatly 

improve rulemaking. Consulting the widest possible range of evaluation data for similar program 

theory domains allows regulators to survey a broader knowledge base and help discover more 

constraints or barriers that might, for instance, limit the expected benefits or reduce the expected 

costs of regulations. 

Theoretically, the efforts to make evaluation data available across agencies, such as in a 

clearinghouse, will help create a wider distribution of evidence going forward. However, 

 

66 Memorandum for the Heads of Executive Departments and Agencies from Barack Obama, “Scientific Integrity,”

March 9, 2009 at 74 FR 10671. See https://www.gpo.gov/fdsys/pkg/FR-2009-03-11/pdf/E9-5443.pdf 67 National Research Council Panel on Data Access for Research Purposes, Expanding access to research data: 

reconciling risks and opportunities, Washington, DC: National Academies Press (2005), p. 3. 68 Paul Voosen, “Climate scientists open up their black boxes to scrutiny,” Science, 354:6311 (28 October 2016), 

pp. 401-402. 

1170 The Promise of Evidence-Based Policymaking
The George Washington University Regulatory Studies Center ی 18

grouping evidence by program theory can tie together seemingly different interventions and help 

regulators identify unintended consequences or important contexts to consider during their early 

design of potential regulatory approaches.69 For example, Ray Pawson’s organizing principles of 

evaluation science suggest that such a level of abstraction “provides the means of establishing a 

common language to draw out the similarities between different interventions...to link their 

evaluations” and increase learning.70

An example of this is evaluations from state/local “ban the box” legislation, which prevents 

employers from asking prospective applicants about their criminal record with the intention of 

decreasing discrimination against those with a criminal record. Evaluations of these programs 

indicate that they have the unintended/perverse effect of increasing discrimination against 

minorities, particularly African Americans.71 Rather than thinking of this, conceptually, as a 

“lesson learned” for officials at the Department of Labor, there is a broader finding that could be 

applicable to other federal agencies: namely, the unintended consequence of trying to incentivize 

certain behavior by limiting data. Additionally, this framework helps shift evaluation thinking 

from simply inquiring whether a program “works” to the more nuanced “what works for whom 

in what contexts.”

72

Questions 17 & 18: Address Retrospective Review in Regulatory design 

Question 17. To what extent can or should program and policy evaluation be addressed in 

program designs? 

Question 18. How can or should program evaluation be incorporated into program designs? 

What specific examples demonstrate where evaluation has been successfully incorporated 

into program designs? 

Ex post regulatory evaluation (retrospective review) is a vital and integral element of the EBR 

Framework (see items I.G. and III). Retrospective review advances knowledge over the mere 

hope that regulations are delivering the benefits society expects. However, it must be 

incorporated into regulatory design in order to facilitate this evaluation. Similar to other federal 

 

69 For examples of this see R. Merton, On Theoretical Sociology: Five essays old and new. New York: Free Press 

(1967) 70 Pawson, p. 190. 

71 Amanda Agan and Sonja Starr, “Ban the Box, Criminal Records, and Statistical Discrimination: A Field 

Experiment,” U of Michigan Law & Econ Research Paper No. 16-012, 2016. See also: Jennifer Doleac, “‘Ban 

the Box’ does more harm than good,” May 2016, Brookings Institute. Available at: 

https://www.brookings.edu/opinions/ban-the-box-does-more-harm-than-good/ 72 Pawson, p. xiii. 

The Promise of Evidence-Based Policymaking 1171
The George Washington University Regulatory Studies Center ی 19

programs, waiting until after a regulation is implemented to plan ex post measurement can 

greatly hamper retrospective review.73 

Both OMB74 and the Administrative Conference of the United States (ACUS) have 

recommended that agencies design their rules prospectively for retrospective analysis. For 

instance, in his report to ACUS, Joseph Aldy concludes: 

Well-designed regulations should enable retrospective analysis to identify the 

impacts caused by the implementation of the regulation. For a given select, 

economically significant rule, agencies should present in the rule’s preamble a 

framework for reassessing the regulation at a later date. Agencies should describe 

the methods that they intend to employ to evaluate the efficacy of and impacts 

caused by the regulation, using data-driven experimental or quasi-experimental 

designs where appropriate.75 

These recommendations echo a larger body of research. For instance, in a study for the World 

Bank, Paul Gertler et al. conclude that the appropriate methods for conducting program 

evaluation, or retrospective review, should be identified “at the outset of a program, through the 

design of prospective impact evaluations that are built into the project’s implementation.”

76 This 

allows evaluators to fit their evaluation methods to the program being reviewed, and to plan for 

review itself through the design and implementation of the program (or regulation). 

For these reasons we have prominently included retrospective review as a necessary element of 

regulatory design in the EBR Framework, and we recommend this design requirement be 

codified in law to emphasize its importance. 

It should be noted that the strong connection between regulatory design and retrospective review 

also strengthens the need to complete other elements of the regulatory process in the design 

stage. For instance, in addition to planning for retrospective review, the EBR Framework 

requires regulators to:77 

 

73 Other reasons to plan evaluations in advance include compliance with the Paperwork Reduction Act which 

requires the prior approval of the U.S. Office of Management and Budget before collecting information from 10 

or more members of the public. See 5 C.F.R. Part 1320.8(b)(3)(iii) (2015). 74 U.S. Office of Management and Budget. 2015 Draft Report to Congress on the Benefits and Costs of Federal 

Regulations and Unfunded Mandates on State, Local, and Tribal Entities, p. 7. 75 Joseph E. Aldy. “Learning from Experience: An Assessment of the Retrospective Reviews of Agency Rules and 

the Evidence for Improving the Design and Implementation of Regulatory Policy,” a report for the 

Administrative Conference of the United States. 2014, p. 6. 76 Paul J. Gertler, Sebastian Martinez, Patrick Premand, Laura B. Rawlings, and Christel M. J. Vermeersch, Impact 

Evaluation in Practice, The World Bank, 2011, pp. xiii–xiv. 77 These components are adapted from Miller, p. 10. 

1172 The Promise of Evidence-Based Policymaking
The George Washington University Regulatory Studies Center ی 20

x Identify the problem they are trying to solve.

x Evaluate whether modifications to existing rules can address the problem.

x Identify and assess available alternatives to direct regulation.

x If regulating, determine that the rule addresses the problem.

x Set clear performance goals and metrics for outputs and outcomes.

x Exploit opportunities for experimentation.

All six of these design components directly relate to retrospective review. One purpose for 

incorporating these components into rules at the outset is to plan for review well before much of 

the crucial information necessary for an effective evaluation has been generated. Otherwise 

agencies may not have identified the goal(s) of the regulation much less how to collect data on 

the regulation’s impacts. This information is crucial for assessing how well a rule has met its 

intended target and the extent to which there may be other, unintended, consequences. 

Independent regulatory agencies especially should make greater efforts to outline what they 

intend for their rules to accomplish.78 This transparency allows the public to know what to expect 

from new regulations and what observers should strive to measure to assess the success of a rule. 

Although few regulations have been designed to facilitate ex post review, the recent driverless 

cars policy guidance is an example of what may be possible. In September 2016 the National 

Highway Traffic Safety Administration (NHTSA) released its Federal Automated Vehicles 

Policy79 establishing how the agency will address driverless car technology through its current 

regulatory structure and identified new regulatory tools that could be used in the future. Given 

the state of change in automated vehicle technology, NHTSA plans to update this policy in an 

iterative process so as to respond to new data and technologies as they emerge. For instance, the 

agency has already noted it will consider the option of implementing a sunset on federal motor 

vehicle safety standards so that the agency can reconsider whether the standards are still effective 

as driverless car technology continues to develop. This iterative approach combined with a 

commitment to collect and synthesize evidence as it comes in appears to reflect the right 

approach to regulating a new and promising technology.80

 

78 Independent agencies are less likely than executive branch agencies to write rules that identify the problem they 

are intended to solve, provide metrics for assessing whether a problem has been solved, and link the proposed 

rule to intended outcomes. See Sofie E. Miller, “Learning from Experience: Retrospective Review of Regulations 

in 2014,” Working Paper, The George Washington University Regulatory Studies Center, November 2015, p. 18. 79 National Highway Traffic Safety Administration, the U.S. Department of Transportation. Federal Automated 

Vehicles Policy: Accelerating the Next Revolution In Roadway Safety. September 2016. See 

https://www.transportation.gov/sites/dot.gov/files/docs/AV%20policy%20guidance%20PDF.pdf 80 Although NHTSA’s approach to review and iteration is well-aligned with the principles of the EBR Framework, 

the agency does discourage state-level competition which would be aligned with the principles explored in the 

section immediately below: Keep Evaluation Options Flexible.

The Promise of Evidence-Based Policymaking 1173
The George Washington University Regulatory Studies Center ی 21

Question 19: Keep Evaluation Options Flexible 

Question 19. To what extent should evaluations specifically with either experimental 

(sometimes referred to as “randomized control trials”) or quasi-experimental designs be 

institutionalized in programs? What specific examples demonstrate where such 

institutionalization has been successful and what best practices exist for doing so? 

The EBR Framework does not specify what types of experimental designs should be used in 

analyzing or evaluating regulations. Rather, the rigor of the analysis should match the regulatory 

situation and the value such analysis may offer decision-makers.81

Randomized controlled trials are well-regarded tools used by program evaluators to understand 

the effect of different treatments on outcomes.82 However, where randomized trials are not 

feasible, pilot studies or approaches that allow for variation in regulatory treatments can serve as 

“quasi-experiments” (QEs) that provide valuable information for evaluating outcomes and their 

causal links.83 According to Coglianese: 

Variation in observational studies can arise in one of two ways: either over time 

or across jurisdictions. When regulations vary over time within a single 

jurisdiction, researchers can compare outcomes longitudinally, that is, before and 

after the adoption of the regulation. When the variation exists across jurisdictions, 

researchers can compare outcomes cross-sectionally, that is, comparing outcomes 

in jurisdictions with the regulation being evaluated with those in jurisdictions 

without that regulation.84 

Designing regulations from the outset in ways that identify and exploit variations in compliance 

could be a valuable way to understand the relationship between regulatory actions and outcomes. 

A pilot study or “an experiment in which certain regulations would be imposed on some factories 

and not on others offers the real prospect of determining whether those regulations are useful.”

85 

 

81 See supra note 45. 82 See Angela Ambroz and Marc Shotland, Randomized Controlled Trial (RCT), Better Evaluation: Sharing 

information to improve evaluation at http://betterevaluation.org/plan/approach/rct 83 Sofie E. Miller & Susan E. Dudley. “Regulatory Accretion: Causes and Possible Remedies.” Administrative Law 

Review Accord Vol. 67 Issue 2. 84 Coglianese, Cary (2012), “Evaluating The Impact of Regulation and Regulatory Policy,” in Measuring 

Regulatory Performance, OECD, available at https://www. oecd.org/gov/regulatorypolicy/1_coglianese%20web.pdf 85 John O. McGinnis, Accelerating Democracy: Transforming Governance through Technology, Princeton, NJ: 

Princeton University Press (2013), p. 112. 

1174 The Promise of Evidence-Based Policymaking
The George Washington University Regulatory Studies Center ی 22

In the U.S. federalist system, the states provide a particularly valuable opportunity for 

experimentation. For example, Oates suggests that “the introduction in the 1970s and 1980s of a 

variety of emissions trading systems at the state level demonstrated the feasibility of such 

systems and some of their very appealing properties—as well as certain pitfalls.” He suggests 

that this state-level experimentation with innovative solutions to emissions problems led to the 

successful introduction of the national system of tradable sulfur allowances under the 1990 Clean 

Air Act Amendments.86 Such quasi-experimental approaches facilitate learning from experience 

in a way that implementing large-scale, irreversible regulatory programs do not.87 

The EBR Framework calls on regulators to look for and exploit opportunities for 

experimentation during regulatory design. For instance, researchers have suggested how the 

statutorily required five-year National Ambient Air Quality Standards reviews could incorporate 

QE techniques to gather and analyze epidemiology data and health outcome trends in different 

regions of the country and compare them against predictions.88 Unfortunately, the U.S. 

Environmental Protection Agency has not pursued this idea. 

The EBR Framework also requires agencies plan and budget for retrospective review as part of 

their regulatory design. This means agencies should lay out a program for empirical testing of 

assumptions and hypothesized outcomes. To incentivize more robust evaluation, they could also 

be required to test the validity of risk-reduction predictions before commencing new regulation 

that relies on models. For example, for regulations aimed at reducing health risks from 

environmental factors, QE techniques should be used to gather and analyze epidemiology data 

and health outcome trends in different regions of the country and compare them against 

predictions.89

 

86 Oates, W.E. Environmental Federalism. Resources for the Future (RFF); 2009. Available at: 

http://www.rff.org/Publications/WPC/Pages/Environmental-Federalism-Wallace-E-Oates.aspx 87 Sofie E. Miller & Susan E. Dudley. “Regulatory Accretion: Causes and Possible Remedies.” Administrative Law 

Review Accord Vol. 67 Issue 2. 88 Francesca Dominici, Michael Greenstone and Cass R. Sunstein, “Particulate Matter Matters.” Science 344:257 

(2014). 89 Sofie E. Miller & Susan E. Dudley. “Regulatory Accretion: Causes and Possible Remedies.” Administrative Law 

Review Accord Vol. 67 Issue 2. 

The Promise of Evidence-Based Policymaking 1175
The George Washington University Regulatory Studies Center ی 23

Suggested Recommendations for Findings, Conclusions and Legislation or Administrative Actions 

We suggest the Commission consider including the following findings, conclusions and recommendation for legislation or 

administrative actions in its report to the President and Congress. 

Suggested Finding Suggested Conclusion Suggested Recommended Action 

REGULATORY POLICY

Regulatory policymaking is already subject 

to significantly different information 

requirements compared to other 

policymaking processes.

The increased use of evidence will result in 

better regulatory decisions. 

Actions to improve evidence-based 

policymaking should be tailored to 

the regulatory process.

It would be beneficial to identify a 

model process for creating evidencebased regulations.

“OMB should integrate evidence more effectively in 

its...regulatory decisions by tracking and evaluating the 

results of the policies it issues.”

90

The president should consider commissioning a set of 

experts to describe an ideal evidence-based regulatory 

process and identify specific steps necessary to move to 

such a system. 

ACCOUNTABILITY 

Regulatory decision-makers need to be held 

publicly accountable for the decisions they 

make. 

The interpretive models, analyses 

and other tools used by regulators to 

make decisions should be accessible 

to the public.

The president should provide unrestricted access to all 

interpretive data tools used by regulators to make decisions. 

COMPLIANCE 

Federal regulatory agencies do not always 

faithfully comply with presidential 

executive orders and other internal 

administrative guidance.

Compliance with presidential 

directives and administrative 

guidance should be improved. 

Codification of a requirement in law 

results in greater compliance than 

administrative guidance.

Regulatory principles accepted by the last five presidents91

should be codified in law and subject to judicial review. 

Regulatory requirements in Executive Orders 13563, 13579 

and 13610 regarding retrospective review should be 

codified in law and subject to judicial review.92 

 

90 Recommendation of the Partnership for Public Service, “From Decisions to Results: Building a More Effective Government Through a Transformed Office of 

Management and Budget,” Washington DC, October 2016, p. 24. Available at https://ourpublicservice.org/publications/viewcontentdetails.php?id=1349 91 These principles will be found in Section 1 of Executive Order 12866 issued by President William Clinton on October 4, 1993. Available at 

https://www.whitehouse.gov/sites/default/files/omb/inforeg/eo12866/eo12866_10041993.pdf 92 See Recommendation 2014-5, “Retrospective Review of Agency Rules” issued by the Administrative Conference of the United States. See 

https://www.acus.gov/recommendation/retrospective-review-agency-rules 

1176 The Promise of Evidence-Based Policymaking
The George Washington University Regulatory Studies Center ی 24

Suggested Finding Suggested Conclusion Suggested Recommended Action 

COMPETITION 

Competition can change the incentives and 

behavior of government organizations in 

positive ways.

The president and Congress should 

encourage methods of having 

programs with similar goals compete 

on the basis of program efficiency 

(e.g., desirable outcomes achieved 

per dollar spent by society). 

The president and congress should commission experts to 

categorize federal programs with similar goals and identify 

metrics that could be used to compare their efficiency.93 A 

limited set of comparisons should be implemented within 

two years.

FUNDING 

Federal discretionary spending is likely to 

be flat or decreasing in the future while 

entitlement program spending will continue 

to increase. 

Lack of funding is a barrier to collecting and 

using evidence.

The cost and depth of evaluations and their 

value to decision-making can greatly vary. 

The collection and use of evidence 

will need to be funded by shifting 

discretionary funding from lower 

priorities.

The type of evaluation performed 

should reflect its potential value to 

improving federal policy. 

Congress should provide greater flexibility to reallocate 

discretionary funding from lower priority uses to the greater 

collection and use of evidence.

The president and congress should refrain from 

institutionalizing any particular type of evaluation method. 

EVIDENCE AND POLICY 

Government officials sometimes muddle a 

description of “what is” with “what ought to 

be.” 

The use of evidence needs to better 

separate scientific descriptions from 

policy judgments.

This confusion masks policy 

decisions. This degrades political 

accountability and harms the 

integrity of evidence-based 

policymaking. 

The president should “promulgate guidelines (through 

executive orders or other instruments) to ensure that when 

federal agencies are developing regulatory policies, they 

explicitly differentiate, to the extent possible, between 

questions that involve scientific judgments and questions 

that involve judgments about economics, ethics and other 

matters of policy.”

94

 

93 Such an effort could greatly benefit from the experience of the Council of State Governments’ State Comparative Performance Measurement Project. See 

http://www.csg.org/programs/policyprograms/CPM.aspx 94 See Recommendation One at Bipartisan Policy Center, Improving the Use of Science in Regulatory Policy, Washington (DC): Bipartisan Policy Center; 2009; 

p. 4. 

The Promise of Evidence-Based Policymaking 1177
The George Washington University Regulatory Studies Center ی 25

Suggested Finding Suggested Conclusion Suggested Recommended Action 

RETROSPECTIVE REVIEW 

Regulatory retrospective review is best 

planned out when a regulation is initially 

designed. 

Regulatory retrospective review relies on 

other elements of regulatory design, such as 

defining the problem to be solved and 

identifying alternatives for comparison. 

 

Regulatory design must include 

retrospective review and its 

supporting elements. 

Regulatory requirements in Executive Orders 13563, 13579 

and 13610 regarding retrospective review should be 

codified in law and subject to judicial review.95 

Regulatory principles accepted by the last five presidents 

that support retrospective review should be codified in law 

and subject to judicial review.96

CATEGORIZATION OF EVIDENCE 

Regulators can benefit from learning lessons 

from programs not in their substantive 

expertise.

The best use of evidence may 

require it be organized by program 

theory (e.g., behavioral change) 

rather than issue area (e.g. 

transportation) 

To the extent evidence of evaluations are consolidated, 

require “type of program theory” to be a characteristic that 

can be used to find evidence of federal program impacts. 

EXPERIMENTATION 

The increased collection and use of 

evidence from regulatory evaluations will 

result in better regulatory decisions. 

Randomized controlled trials to 

evaluate regulations are not always 

feasible.

Pilot studies or approaches that 

allow for variation in regulatory 

treatments (“quasi-experiments” or 

QEs) can provide valuable 

information at less cost.

The president should encourage regulators to adopt QE 

techniques where more expensive evaluations may be 

infeasible or of less value.

If necessary, Congress should amend regulatory authorities 

to allow agencies greater flexibility to design regulations to 

facilitate differences in implementation that allow quasiexperimentation. For instance, laws should allow limited 

pilot studies, or defer more to the natural experimentation 

possible at the state level. 

 

95 This repeats a recommendation shown in the “Compliance” section above. 96 This closely matches a recommendation shown in the “Compliance” section above.

1178 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 09, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sxx-xvrc

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0043

Comment on FR Doc # 2016-22002

Submitter Information

Name: Tom Gallagher

General Comment

See attached file(s)

Attachments

Docket ID 160907825-6825-01-Commission on Evidence-Based Policymaking

Comments

The Promise of Evidence-Based Policymaking 1179
State of Wyoming

Department of Workforce Services

Research & Planning

P.O. Box 2760, Casper, WY 82602

(307) 473-3807

http://doe.state.wy.us/LMI

Matthew H. Mead

Governor

John Cox

Director

Lisa M. Osvold

Deputy Director

November 9, 2016 

To: The Commission on Evidence-Based Policy Making 

Subject: Response to the “Request for Comments for the Commission on Evidence-Based Policy 

Making,” Department of Commerce [Docket Number 160907825-6825-01] published in the Federal 

Register Vol. 81, No. 178, September 14, 2016 pages 63166-63168. 

From: Tom Gallagher 

 Manager, Research & Planning 

Tom.gallagher@wyo.gov 

307-473-3801 

Research & Planning (R&P) is a statistical Labor Market Information (LMI) office located in the State 

agency which operates the Unemployment Insurance (UI) program. R&P’s state employees are agents 

of the Commissioner of the Bureau of Labor Statistics (BLS). Our BLS work, programs operated under 

contracts to the Office of Employment and Unemployment Statistics and Office of Compensation and 

Working Conditions (including the Census of Fatal Occupational Injuries), are carried out under the 

Confidential Information Protection and Statistical Efficiency Act. These program operations include 

accessing state UI administrative data, vital records, and other administrative data. 

Grant funding from the U.S. Department of Labor’s Employment and Training Administration is used to 

develop data access agreements with all of public education student records in Wyoming, occupational 

licensing agencies, and UI data sharing agreements with 11 other states. 

R&P’s mission is: To establish an empirically based comprehensive understanding of the labor market: 

its constituent elements, systems integrating its components, and subsequent outcomes. Attached is a 

selected chronological listing of our research illustrating how we act on this mission. 

There is wide state-to-state variation in the make-up of staff and the duties in the LMI office. It is 

intended that these few paragraphs and attached references provide the reader with a greater 

understanding of the comments which follow. 

The views expressed in these comments are solely those of the author. 

Attachments

1180 The Promise of Evidence-Based Policymaking
Chronology of Selected Publications based on Linked Administrative Records and Administrative 

Records Linked to Survey Research 

Research and Planning, Wyoming Department of Workforce Services 

11/09/16 

Tracking University of Wyoming Graduates Into the Wyoming Work-force, September 1995 

Quarterly wage and employment information, 1983-1993 Graduates, UI wage records 

Under the Lamppost: Report to Workforce Development Council on Wyoming Institutions of Higher Education 

Program Completers, November 1998 

Classification of Instructional Programs, Completers by degrees, 1997 Calendar year/ 97-98 Academic 

year, UI wage records 

Workforce Development and Community College Outcomes, When Does Training Pay Off?, Wyoming Labor 

Force Trends, July 2001 

Workforce Investment Act, Casper College, Employer Survey, wage records 

The Effects of a College Degree on Wages: The Different Experiences of Men and Women, October 2001 

Degree attained: Bachelor’s v. no degree, Jobs requiring college degree, wage records 

Where Are They Now? Wyoming Community College Graduates’ Labor Market Outcomes 2004, August 2004 

Students working in partner states, Retention of students, use of administrative records and survey 

research 

Cooking Up a Career: Examining the Outcomes of a High School Training Program in the Culinary Arts and 

Hospitality Management, August 2005 

Wyoming Hospitality Alliance Mentoring, Long-term career opportunities in Leisure & Hospitality 

industry, Administrative records and survey 

Wyoming Community College Graduates’ Labor Market Outcomes 2005: An Administrative Records Approach, 

April 2006 

Employment/Enrollment Outcomes, Occupational-Technical degree programs, Data-Sharing 

Agreements, Partner states, wage records 

Retention of Nurses in Wyoming: Part II, August 2008 

Longitudinal analysis of administrative records linked to survey, Nearing age of retirement, Healthcare 

workplace quality, Quantitative and qualitative analysis of nursing experience 

Post-Injury Wage Loss: A Quasi-Experimental Design. Prepared for The National Institute for Occupational Safety 

and Health (NIOSH) Mountain and Plains Education and Research Center (MAP ERC). May 2009 

Labor market, wages, employment in Wyoming, turnover, demographic characteristics, workers 

compensation claims 

Job Attainment and Wages of Wyoming Vocational Rehabilitation Participants, Wyoming Labor Force Trends, 

February 2010

VR successful closures, Wage records database, Longitudinal outcomes 

The Promise of Evidence-Based Policymaking 1181
Chronology of Selected Publications based on Linked Administrative Records and Administrative 

Records Linked to Survey Research 

Research and Planning, Wyoming Department of Workforce Services 

11/09/16 

Driven by Demographics: Examining Employee Exits in State Government, Wyoming Labor Force Trends, 

December 2010 

Linking state government occupations to turnover, wage records 

Health Care Workforce Needs in Wyoming: Advancing the Study, Occasional Paper No. 6, Fall 2011 

Commuting impacts, linking health care professional licensing to wage records 

Monitoring School District Human Resource Cost Pressures, Fall 2013 

Educational Services salaries in Wyoming and Surrounding states, Professional Teaching Standards 

Board, Department of Education, staffing linked to wage records 

Effects of decline in teen drivers, Wyoming Labor Force Trends, September 2014 

Graduate driver’s license, Job Skills Survey, UI administrative records 

Nurses Returning to School, Wyoming Nurse Reporter, Fall 2014 

survey sample, Job satisfaction, Motivation to continue education, Employer discouragement, Time 

constraints, Nurses nearing retirement age 

Higher Wages and More Work: Impact Evaluation of a State-Funded Incumbent Worker Training Program, 

Wyoming Labor Force Trends, January 2016 

new business recruitment, enhance employee wages, reduce turnover, enhance profitability 

Impact Evaluation of a Wyoming Employment Job Assistance Program, Labor Force Trends, January 2016 

Increase wages, decrease government assistance, JobAssist Program, self-sufficiency, employment 

outcomes 

Wyoming Hathaway Scholarship Program: A Workforce Outcomes Evaluation of a State Merit-Based Scholarship 

Initiative Using Administrative Records. Casper: Wyoming Department of Workforce Services, Research & 

Planning, July 2016 

Enrollment, financing, demographics, postsecondary education, employment, wage records, program 

evaluation, turnover, continuous employment 

1182 The Promise of Evidence-Based Policymaking
 1 

Comments for the Commission on Evidence-Based Policy Making, November 9, 2016, 

Research & Planning Wyoming Department of Workforce Services 

This is the response to the “Request for Comments for the Commission on Evidence-Based Policy 

Making,” Department of Commerce [Docket Number 160907825-6825-01] published in the Federal 

Register Vol. 81, No. 178, September 14, 2016, pages 63166-63168. 

The federal register notice asks for examples of “successful frameworks, policies, practices, and methods 

to overcome challenges related to evidence-building from state ...”government without asking for 

information about how federal policies work to create challenges to evidence-building. Both success 

and its opposite have to be analyzed using the case study approach to identify features that must be 

true for successes and that must not be true for failure. Having been involved as a professional 

researcher in state government for 38 years, specifically working in state-federal Labor Market 

Information (LMI) offices, I have been a participant observer in the development, implementation, and 

evaluation of labor programs. With this in mind, I will attempt to address the overarching question of 

defining challenges to the development of evidence-based policy and institutionalized counterproductive federal policies. 

In certain domains, state agencies play a key role in implementing federal policy and serve as the data 

collection and management entity for related administrative records. One such domain is the distributed 

workforce development system which is overseen by the U.S. Department of Labor (DOL), but 

implemented by states (and for certain components by sub-state entities). Consequently, unless labor 

program delivery is federalized, any attempt at evidence-based transformational federal policy will need 

to take the states into account and incorporate policy and practices that invest in them as partners with 

the federal government. 

One, at least partial success, is the Worker Profiling and Re-employment Services (WPRS) program. 

WPRS was formally introduced into the Unemployment Insurance (UI) system through federal legislation 

in 1993. DOL’s Employment and Training Administration (ETA) designed the WPRS program as a result 

of rigorous research based on experimental techniques. Under the program, state UI agencies provide 

focused re-employment services for unemployed workers covered by UI. UI claimant eligibility for reemployment services is determined in most states through an ETA guided, state developed regression 

model in which the dependent variable is the probability of running out of weekly UI benefit payments. 

The research upon which this program was designed indicated that reduced taxes to employers and an 

earlier return to employment for workers than could otherwise be expected were reasonable 

expectations under WPRS implementation. 

However, in 2007, the General Accountability Office found that: “The national data on the workerprofiling initiative is of very limited usefulness as a measure of program ..., outcomes, and effectiveness. 

Many of the data are not usable because of inconsistent or incorrect reporting, and neither Labor nor 

the states...use the data for evaluating the worker-profiling initiative. ..., even if all the outcomes data 

were reported consistently and accurately, these data cannot, by themselves, be used to measure the 

impact of the program. ..., by requiring the submittal of data that are of such limited reliability and 

value, Labor is potentially wasting both its own and the states’ resources. Finally, absent information 

The Promise of Evidence-Based Policymaking 1183
 2 

Comments for the Commission on Evidence-Based Policy Making, November 9, 2016, 

Research & Planning Wyoming Department of Workforce Services 

about the program’s current impact, Labor may find it more difficult to make decisions regarding the 

best means for returning the unemployed to work more quickly.” (GAO, 2007, p. 30). Wandner points 

out that the George W. Bush administration actively sought to eviscerate research and evaluation 

initiatives within ETA (Wandner, 2010, Ch. 6). Given the time frame for the Bush Presidency, this could 

account for the demise of data quality in the management of the WPRS program. However, there is little 

evidence that the current administration has done anything to remedy the situation. ETA’s current FiveYear Research and Evaluation Strategic Plan; Program Years 2012-2017 fails to identify the WPRS 

program at all, see https://wdr.doleta.gov/research/FullText_Documents/ETAOP_2013_21.pdf . While 

one administration actively seeks to eliminate the role of ETA’s research capacity, its successor seeks the 

expansion of the role of government. Even under the latter situation, the quality of the underlying WPRS 

data systems within UI, needed to sustain the administration of evidence-based policy, deteriorate. In 

this case, it can be suggested that the issue of data quality is to some degree a consequence of 

partisanship. However, the absence of data system improvement under the current administration 

suggests that the more reasonable explanation for neglect is bureaucratic inertia. 

It is not the function of bureaucracy to manage based on empiricism, but to carry out those activities 

which secure its existence and expand its scope. The Workforce Investment Act (WIA) of 1998 required 

the states to conduct rigorous program evaluations (including, the statute indicates, the use of control 

groups) of the federally funded training programs they administer. However, ETA cannot identify a 

single example of a state-conducted WIA training program evaluation. The Workforce Innovation and 

Opportunity Act (WIOA), signed into law in July 2014, replaced WIA and contain the same requirements 

for state administered program evaluation, and their coordination with the Secretary’s national program 

evaluation undertakings. Up to this point in the implementation of WIOA, state program evaluations 

and the Secretary’s coordination of them remain inert and symbolic. 

Despite public endorsement of making “data driven” decisions, there is little understanding of the 

program uses of evaluation and therefore little demand for evidence-based policy making and high 

quality administrative data within the state employment and training system. As a result, it is important 

to separate the symbols of program management from practice within UI state programs. The priorities, 

business rules, and processes that generate basic UI administrative data serve as inputs to UI program 

management. Understanding the application of UI business rules is essential to the appropriate use of 

these administrative data in a research context. 

In recent testimony before House Ways and Means, a spokesperson for the National Employment Law 

Project outlined one dimension of UI program integrity. “For the one-year period ending June 2015, 

10.3% of UI [claim] payments were overpaid ... one out of three...overpayments were found to be 

fraudulent.”(Conte, 2016, p. 8) The remaining claim overpayments were attributable to agency and/or 

employer tax reporting error. The integrity of UI claims is grounded in UI wage record employer tax 

reporting. Given the level of non-fraud based error in claims payments, we begin to gain an 

understanding of the need to edit wage records prior to their statistical use. 

1184 The Promise of Evidence-Based Policymaking
 3 

Comments for the Commission on Evidence-Based Policy Making, November 9, 2016, 

Research & Planning Wyoming Department of Workforce Services 

From the standpoint of administering the UI program, the integrity of UI administrative data is 

problematic. Moreover, with rare exception, state agencies managing WIOA programs do not recognize 

a vested interest in research evaluation of the uses of UI data and, therefore, substantial data quality 

problems can be expected. The research incentives associated with profiling or investment in 

evaluations (whether conceived of in the form of financing, sanction, or human resource research 

credentials) is insufficient at either the state or federal level to argue that all has been done that would 

justify legal steps broadening access to administrative data or creating concentrations of linked 

databases. The system as it now stands neglects plentiful opportunities for evidence-based policy 

making. However, it exhibits little will or ingenuity that would demonstrate that it has exhausted 

opportunities as justification for moving to the next level of data access. The current administration 

should take that step. It should demonstrate that the data in UI administrative records to which it seeks 

access is of the highest reasonable quality for state administration of the UI system before it is exploited 

as a universally available tool in guiding policy development. Data of unknown quality and bias made 

broadly available disregards threats to the comparability of interstate replication. 

State UI administrative records make up the bulk of “federal data” in the National Directory of New 

Hires (NDNH). Employer payroll records and worker wage records represent the interface between the 

workforce and industry, mediated by UI administrative records collection, management and storage 

systems, grounded in an economic classification system implemented by state LMI offices. If the user of 

these data does not understand the composition of the workforce, industry behavior, or the business 

rules for managing a system of UI administrative records, the researcher’s capacity to use wage records 

as a theoretically relevant tool is compromised. 

Each industry exhibits different trends and seasonal employment patterns. UI reporting bias is not 

distributed evenly across industry types and firm sizes. UI administrative data collection and 

management systems vary from state to state. This means that locality is a relevant variable both from 

a labor market standpoint, and from what is represented in state UI administrative records systems. 

State specific dynamics of UI administrative records limit comparability in ways that are not recognized, 

catalogued, or accounted for in the NDNH. 

State research offices that acquire UI administrative records for statistical purposes edit UI claims, 

employer payroll records, and wage records earnings for individuals. UI administrative records are 

reconciled and reporting errors corrected before these administrative records are used for statistical 

purposes. 

State agency researchers obtain copies of UI administrative records at several points after the tax filing 

date to ensure comprehensiveness. This is not the case for UI administrative records making up the 

NDNH -- for which the administration has been seeking broader researcher access (U.S. Department of 

Health and Human Services, 2017). UI staff in Wyoming sent the second quarter of 2016 wage records 

to the NDNH at the end of September 2016. This is the only copy of the wage records file NDNH will 

receive. However, for R&P’s research purposes, the last download of 2016Q2 will take place in the 

The Promise of Evidence-Based Policymaking 1185
 4 

Comments for the Commission on Evidence-Based Policy Making, November 9, 2016, 

Research & Planning Wyoming Department of Workforce Services 

fourth quarter of 2017. Given the inferior coverage and quality of the NDNH UI files, it is clear that at 

some point an investment will need to be made in data quality (along with state-to-state uniformity) if 

UI administrative databases are to reliably serve research purposes. 

Building quality into state UI administrative records requires the development of a state UI and WIOA 

research agenda and funding if the states are to invest in data quality. The current level of funding and 

state incentive to invest in data quality to prevent UI claims overpayment is insufficient to the task. On 

the other hand, if the federal government and the states have a common research cause, data quality 

stands a greater likelihood of being addressed. 

Perhaps the most important social intervention in this country is the state domain of education. Federal 

investment of millions of dollars over the past decade in state-based evidence building for purposes of 

state level interventions has produced very little in published results. The U.S. Department of Education 

(ED) began investing in state educational entities analysis of the longitudinal student records in 2005. 

State Longitudinal Data System (SLDS) competitive awards encourage quasi-experimental design in 

tracking student cohorts from pre-school through post-secondary and into the labor market using UI 

wage records. The Senate approved another $34.5 million for the SLDS effort in FY 2017 (S. Rep. No. 

114-274, 2016, p. 197). ED funds state collection and organization of education data but requires little 

in the way of grantee publication of analysis. The fact that grantees publish so little represents an 

institutional-financial relationship and set of expectations for state performance sustaining the culture 

the Commission is attempting to transform. Without inquiry into the means by which federal agencies 

foster the appearance of decision-making, informed by empirical analysis, without holding grantees 

accountable for the substance of published analysis, the Commission would be ignoring the problem it is 

charged with addressing. 

Dr. Mark Schneider, Commissioner of the National Center for Education Statistics (NCES) from 2005 to 

2008, in 2012 testimony before the House Committee on Education and the Workforce, responded to 

the following question relating to the performance of grantees under the SLDS program: 

“Chairwoman Foxx. Okay. We are going to try one more round. Let us go back again to looking a 

little bit at the state and what the states are doing now. And if you all might respond to this; 

what factors are there that make up the high-quality state – state longitudinal data systems? 

And what factors are missing from the low-quality state data systems? 

I think, Dr. Schneider, you said that some states are doing very well; others are doing it, but are 

not publishing it. Do we know why there are those problems with the states? ... 

Mr. Schneider. Yes... 

I think one of the mistakes that we [NCES] made, and remember, we are $700 million into this 

process, was that we did not have a use requirement. And there is a long history on this. There 

was no use requirement on this data. 

1186 The Promise of Evidence-Based Policymaking
 5 

Comments for the Commission on Evidence-Based Policy Making, November 9, 2016, 

Research & Planning Wyoming Department of Workforce Services 

So, what has happened is that we have made this huge investment in these data warehouses, 

which I think of – I sometimes call them data mausoleums and going back to many years ago I 

think of them – you remember there was something called the roach motel. You know roaches 

checked in, but they never checked out. So, sometimes I think these data systems as the 

equivalent. You know data checks in and we never see them again. 

So, we spend a lot of money on building these and actually very, very few concrete products 

that actually can and should help us inform consumer choice ... It is a long complicated process 

often because of the politics of data. It is not that the data do not exist. It is that the data do 

exist. And sometimes the results [of UI wage records matches] are not – do not make people all 

that happy.” (Higher Education and Workforce Training, 2012, p. 49) 

The absence of published analysis from SLDS grantees is attributed in this testimony to the fact that 

NCES grants did not require them. However, the failure to include a requirement for publication in grant 

awards does not explain why states have not voluntarily published cohort analysis. One barrier to 

publication, testimony suggests, is that the results of wage records outcomes do not always conform to 

the level of positive market return on investment that institutions of higher education publicly ascribe to 

the value of a degree. The question is: Why should expected market returns govern what SLDS 

grantees publish? 

Personal interviews with ED’s SLDS grants management and ETA staff confirm House subcommittee 

testimony regarding the paucity of state-sponsored published scientific analysis. A literature search 

conducted after the 2012 House hearing produce few substantive works. However, some states are 

beginning to publish “dash boards” or report cards containing employment and earnings outcomes for 

selected cohorts of graduates in non-experimental formats, with little narrative or explanatory analysis. 

National SLDS/Workforce Data Quality Initiative (U.S. DOL grants to link student and training participants 

to wage records-based outcomes) grantee conferences provide anecdotal illustrations from colleagues 

regarding SLDS initiatives in publishing UI wage records outcomes conditioned by political agendas. 

Wage record dashboards often lack methodological explanation of the limitations of the data, or 

contextual information about underlying market conditions which serve as alternative explanation for 

educational outcomes in the labor market. Transparency is hampered by limiting publication to 

graduates, rather than all students, and/or the publication of outcomes for all of higher education 

combined, rather than for each institution separately. Too often, tabular data is presented with 

computational enhancements as promotional material without appropriate cautions to the consumer 

about the limitations of the data. In many cases, informed consumer choice is displaced by marketing 

gimmicks and the trappings of accountability rather than the genuine article. 

SLDS grants do not require the functional separation of state statistical entities and reporting from the 

authority of state educational program management. Therefore, it is not difficult to find anecdotes 

about grantees using SLDS funds to promote educational services. However, it is very difficult to find 

The Promise of Evidence-Based Policymaking 1187
 6 

Comments for the Commission on Evidence-Based Policy Making, November 9, 2016, 

Research & Planning Wyoming Department of Workforce Services 

publications regarding objective evaluation of labor market outcomes (or even analysis restricted to 

student outcomes within the educational domain) associated with student investment in education. 

Even if SLDS grantees chose objective reporting about the labor market, they are unlikely to have access 

to wage records for the entire market or even begin to understand the underlying market dynamics 

needed to provide contextual information. The design of the SLDS effort prevents it from obtaining and 

providing the counter factual information to establish the meaning of workforce outcomes. Grant 

awards to state educational agencies whose goals are the expansion of educational programs are often 

used to secure a growing perimeter of influence through the use of dash boards regardless of the value 

of these tabulations to students, employers, and communities. The barriers to evidence-based policymaking are often barriers created by federal policies aimed at creating the appearance of data driven 

choices at the expense of substantive analysis. 

Whether public or private, many providers of training services are formally organized bureaucracies and 

as such operate under a common set of organizational imperatives. Regardless of stated motivations, all 

bureaucracies exercise efforts at boundary control by limiting what is known publicly about them, seek 

autonomy in establishing direction, and attempt to ensure sustainability by constantly expanding the 

niches they occupy. One explanation for the political control exercised by SLDS grantees over what is 

made publicly available in the form of student wage records earnings is that they are bureaucracies 

conforming to organizational imperatives. The available evidence on this point is sketchy but of 

sufficient quality, coverage, and importance to suggest that further study of this matter is needed if 

evidence-based policy making is to inform student choice about education and public policy. 

WIOA rules expand upon the strategy of providing UI wage records to SLDS grantees by explicitly 

supporting state UI agency provision of UI wage records to the providers of training services in order to 

facilitate training provider reporting on student outcomes as dash boards (Workforce Innovation and 

Opportunity Act, Final Rule, 2015, p. 971). Without functional separation at the state reporting level, 

DOL and ED have begun actively endorsing an institutional conflict of interest at the heart of federally 

funded workforce development programs. 

Formally organized bureaucracies attempt to ensure their continued survival by gaining control of 

resources. In the case of training providers, one resource over which control is important is expanding 

enrollment. The imperative to expand as an organization may not always be consistent with the need 

for objective wage records outcomes information to inform student career decision making. Placing 

these two functions in the same organization necessarily constitutes a conflict of interest even if no 

training provider ever consciously manipulates data for the benefit of the training provider organization. 

For this reason, sound statistical practice requires insulation in the form of functional separation 

between those entities reporting statistical results from those entities providing training and 

educational services. By fostering the appearance of impropriety federal entities implementing WIOA 

institutionalize the kind of culture in workforce development about the nature of “evidence” that is 

counter-productive. 

1188 The Promise of Evidence-Based Policymaking
 7 

Comments for the Commission on Evidence-Based Policy Making, November 9, 2016, 

Research & Planning Wyoming Department of Workforce Services 

If evidence-based policy making is to become anything more than a “pop” fad among academics, the 

Commission needs to recognize the ways federal agencies implement programs without statistical 

standards, and at the same time cultivate a legitimizing appearance. Implementation requires WIOA 

provider dash boards computed in a standard manner. However, there is no federal or state quality 

control oversight and no common presentation format for students and dislocated workers to choose 

from in selecting training options. Thus, even if there were quality controls and edits for wage-records, 

the purpose of common calculation is off-set by non-comparable computations and presentation from 

provider to provider. 

Our point of arrival with the implementation of WIOA seems much like the destination of the Worker 

Profiling and Re-employment Services (WPRS) program GAO documented in 2007. In WIOA, data is 

managed largely by non-statistical entities and as such, it is as likely to be found “... of limited usefulness 

... Labor is potentially wasting both its own and the states’ resources.” In sum, we continue to see 

federal government policy multiply the non-scientific use of data for appearance value. The lack of 

access to data is not at this time the most important barrier to improved policy. The barriers to 

improved policy can be understood by way of case study. In many instances there is ample data 

available for experimental and quasi-experimental design. There is simply no connection between 

experimentation, on-going study, and the functioning of the bureaucracies that presently administer 

programs. If the incentives embedded in the bureaucracy are not understood and addressed, then the 

attempt at transformation through evidence building will merely rival the status quo and have much less 

chance of impact. Who benefits from a clearinghouse or expanded access to the NDNH when 

administrative data are of suspect quality and bias? Of what value is a clearinghouse and expanded 

access to databases, when at the same time existing programs, e.g. WPRS, and new programs, WIOA, 

are managed in ways antithetical to evidence-based policy making? It is not the successes alone that we 

need to understand but the failures. 

 

The Promise of Evidence-Based Policymaking 1189
References 

Assessing College Data: Helping to Provide Valuable Information to Students, Institutions, and Taxpayers: 

Hearing before the Subcommittee on Higher Education and Workforce Training, Committee on 

Education and the Workforce, U.S. House of Representatives. 106th Cong. (2012). Retrieved October 31, 

2016 from https://www.gpo.gov/fdsys/pkg/CHRG-112hhrg75856/pdf/CHRG-112hhrg75856.pdf

Departments of Labor, Health and Human Services, and Education, and Related Agencies Appropriation Bill, 

 2017. Committee on Appropriations. June 9, 2016. Senate Report 114-274. 

TEGL No. 26-15, Operating Guidance for the Workforce Innovation and Opportunity Act, dated June 29, 2016 

https://wdr.doleta.gov/directives/corr_doc.cfm?DOCN=9363

Unemployment Insurance: An Overview of the Challenges and Strengths of Today's System: Hearing before the 

Human Resources Subcommittee, House of Representatives Ways and Means Committee. (2016) 

(Testimony of Judith M. Conti). Retrieved October 31, 2016 from http://waysandmeans.house.gov/wpcontent/uploads/2016/09/20160907HR-Testimony-Conti.pdf

U.S. Department of Health and Human Services. Fiscal Year 2017. Administration for Children and Families: 

Justification of Estimates for Appropriations Committees. Pg. 304. Retrieved October 31, 2016 from 

https://www.acf.hhs.gov/sites/default/files/olab/final_cj_2017_print.pdf

U.S. Department of Labor Employment and Training Administration (2013, May). Five-Year Research and 

Evaluation Strategic Plan Program Years 2012-2017. Retrieved October 31, 2016 from 

https://wdr.doleta.gov/research/FullText_Documents/ETAOP_2013_21.pdf 

U.S. Government Accountability Office. (2007, June). Unemployment Insurance: More Guidance and Evaluation 

of Worker-Profiling Initiative Could Help Improve State Efforts. (Publication No. GAO-07-680). Retrieved 

October 27, 2016 from GAO Reports Main Page via GPO Access database: 

http://www.gao.gov/assets/270/262148.pdf 

Wandner, Stephen A. (2010). Solving the Reemployment Puzzle: From Research to Policy. Kalamazoo, MI. W.E. 

 Upjohn Institute for Employment Research. 

Workforce Innovation and Opportunity Act; Final Rule, 20 CFR §603, 651, 652 ... 688 (2015) 

 Retrieved October 10, 2016 from https://www.doleta.gov/wioa/Docs/wioa-regs-labor-final-rule.pdf 

1190 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 10, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sye-ghl5

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0044

Comment on FR Doc # 2016-22002

Submitter Information

Name: Florence Thompson

General Comment

STOP THE SOCIAL ENGINEERING! 

The Total Information State, as in Evidence-Based Policymaking, is designed to enslave

the entire populace, beginning with children. Encrypting the personal data does not

render it harmless. The state has no right to maintain dossiers on citizens. Personal data

should not even be collected, much less be maintained forever and used to limit freedom

of choice of individuals through "best practices" (which become the "only practices.")

This is a mechanism of social control and an egregious violation of our rights. It is also a

very unwise policy decision, as a society evolved by diverse individuals developing and

functioning freely in their own self-interests is much stronger, more vibrant and

successful than a society of individuals engineered and moulded by the state, no matter

how smart the elitists at the top think they are.

The Promise of Evidence-Based Policymaking 1191
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 10, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8syg-6uyv

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0045

Comment on FR Doc # 2016-22002

Submitter Information

Name: Gary Coe

General Comment

Communism/socialism in action. Dump this horrible idea.

1192 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 10, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8syg-qe7e

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0046

Comment on FR Doc # 2016-22002

Submitter Information

Name: Charles Swasey

General Comment

I value student privacy and student autonomy over personal and confidential data.

Therefore, I oppose and restructuring of power that would place the federal or corporate

organizations in charge of accessing the personally identifiable information of citizens of

any age, for any reason, without the express written consent of the citizen or his or her

parents.

Just say no to unit record databases.

The Promise of Evidence-Based Policymaking 1193
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 10, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8syg-zr0v

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0047

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

stop mandating collection of my children's "data". I believe it will limit my freedom of

choice because "others" will decide what best works for everyone by analyzing all that

"data". The data being collected is private and none of anyone's business. If I am not the

person in "of the people, by the people, for the people" then I am enslaved by those that

think they're above me and my children. Stay out of our heads!

1194 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 10, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8syh-n5ll

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0048

Comment on FR Doc # 2016-22002

Submitter Information

Name: Oak Norton

General Comment

How about a commission on unintended consequences? All the "evidence" in the world

doesn't always make something the right choice at all times for all people. Local control

and parental rights are constantly being trumped by people who set policy but are

removed from the local level where these policies hit home. Or should I say, hit homes?

Parents need to make informed decisions based on evidence, but also based on what they

know is best for their children. Informed consent should be the goal and not broad

policies set by those who can't foretell the consequences they didn't intend.

The Promise of Evidence-Based Policymaking 1195
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 10, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8syh-vnu1

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0049

Comment on FR Doc # 2016-22002

Submitter Information

Name: Matthew Parks

General Comment

As a parent, I believe strongly that parental consent should be pro-actively required

before any institution is authorized to access a child's data of any kind. The parents aren't

meant to serve the government, but the government is to serve the parents and it is a

*parent's* individual decision whether or not the government's services are welcome.

1196 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 10, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8syj-6t2u

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0050

Comment on FR Doc # 2016-22002

Submitter Information

Name: Phil Jensen

General Comment

We need to stop invasive government collection of personal and private data on our

children. The government cannot say as is obvious from the WikiLeaks situation. No

more collection on our children. Thank you.

The Promise of Evidence-Based Policymaking 1197
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 10, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8syl-gnyx

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0051

Comment on FR Doc # 2016-22002

Submitter Information

Name: Maryann Christensen

Address:

Murray, UT, 84107

Email: maryannwchristensen@gmail.com

General Comment

EEEEEEEK! Violating family and child privacy is NOT OK! If parents want to share

data, I have no problem with that, but it should be BY CONSENT ONLY!

1198 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 10, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8syl-pusv

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0052

Comment on FR Doc # 2016-22002

Submitter Information

Name: Linda McCay

General Comment

I value parental consent, prior to having governments or research groups access a

student's or a citizen's unit records. I am against a federal unit record database.

The Promise of Evidence-Based Policymaking 1199
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 10, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8syn-9gwk

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0053

Comment on FR Doc # 2016-22002

Submitter Information

Name: Robert Holtzclaw

Address:

Jonesboro, GA, 30236

Email: holtzclawassociates@comcast.net

General Comment

Parental consent should be required for any government intrusion.

1200 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 10, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8syn-bdg9

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0054

Comment on FR Doc # 2016-22002

Submitter Information

Name: Amy Anonymous

General Comment

No information should be kept or given out without prior parental consent.

The Promise of Evidence-Based Policymaking 1201
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 10, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8syo-wdmo

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0055

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

Of course, it is more efficient to have every bit of info on everyone from cradle to grave,

including behavioral twitches of eyebrows recorded by computer cameras, where an

individual is at any point in time. This invaluable information permits those with access

to it to control the minutiae of our lives.

The problem is that it is an un-Constitutional invasion of privacy. The Founders did not

care about the efficiency of controlling humans, only their freedom to make their own

choices, their own self-responsibility, individual privacy, property ownership without

interference from government nor big business. They were right. Free people are not

monitored moment to moment or at all. The Constitution is not an out-of-date document.

It speaks to the 21st century because The Founders wisdom and knowledge about failed

governments is even more important today than it was pre-electronic monitoring. 

My DNA and especially that of my newborn is not for manipulation ofour lives, by

anyone but the individual to whom it has been given by their God.

I know it is galling to the modern globalist elite that men with quill pens remain wiser

than they are despite all of their technology. Their only defense is to pooh-pooh them

(undermine) and prevent our children from ever knowing that this country's foundations

rest on the exceptional and far more educated minds of Washington, Adams, Madison, to

our everlasting benefit.

1202 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 10, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8syo-zm2i

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0056

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

There must be parental consent prior to having government or research groups access a

student's or a citizen's unit records.

The Promise of Evidence-Based Policymaking 1203
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 10, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8syo-le2w

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0057

Comment on FR Doc # 2016-22002

Submitter Information

Name: Tammy Erickson

General Comment

I would like too have to have parental consent prior to ANY information being collected

on my child or myself for that matter!

1204 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 10, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8syp-5o10

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0058

Comment on FR Doc # 2016-22002

Submitter Information

Name: Jesse and Sarah Dean

General Comment

We wish to register our opposition to standardized record-keeping of all children's

behavior, health or educational progress.

I understand that in the 'big picture' this information could be valuable to researchers.

However, our constitution does not exist to make life easier for those conducting

research or formulating government policies. Our constitution exists to protect the

independence and liberty of the individual from governmental overreach.

As benign as your motives might be, as parents whose first obligation is to protect our

children, we must ask ourselves, "How might this information be misused in the future?"

In the face of budget cuts, or changing social mores, could this data lead to limits on

health care treatments? Could this data be used to direct my child or grandchild into

specific career paths based on the outcome of standardized tests rather than personal

drive or merit?

Data is power. We oppose increasing government power over citizens without the

consent of the individual.

Sincerely

The Promise of Evidence-Based Policymaking 1205
Sarah and Jesse Dean

1206 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 10, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8syq-89cd

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0059

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

I am very concerned about this data collection and how it might be used. I think at the

very least there needs to be parental consent before data is collected on children.

The Promise of Evidence-Based Policymaking 1207
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 10, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8syq-wev3

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0060

Comment on FR Doc # 2016-22002

Submitter Information

Name: John Hofker

Address:

Athens, TX, 75752

Email: John@BestTexasRates.com

General Comment

I am absolutely opposed to gathering information on children - through electronic means

or any other method.

Once information is gathered, it can be used by anyone for any purpose.

We have seen this over and over and over throughout history.

Once you have the information, you can control their future.

This is exactly how this information will be used - control.

Education control, career control, future control.

It always ends the same way: Hitler, Stalin, Mao - all said, "We just want to get some

information. We're just trying to help the children." It never ends well.

1208 The Promise of Evidence-Based Policymaking
I am absolutely opposed to gathering information on children: electronic or otherwise.

John Hofker

The Promise of Evidence-Based Policymaking 1209
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 10, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8syr-yfjc

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0061

Comment on FR Doc # 2016-22002

Submitter Information

Name: Marianne Dwyer

General Comment

Parents have the responsibility to ensure their children's safety. Parental consent should

be required before research groups gain access to student records.

1210 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 10, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sys-5nx2

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0062

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

I believe that student privacy is very important, and that consent is important, and that a

move to a database of individual unit records is unacceptable in our free country.

The Promise of Evidence-Based Policymaking 1211
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 11, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8syv-8k4a

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0063

Comment on FR Doc # 2016-22002

Submitter Information

Name: Ginny Meacham

General Comment

I prefer a child's records be stored in one secure file in their respective school district,

and that parental consent be obtained prior to that data being viewed by anyone. We

need to allow parents the control over information about their children.

1212 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 11, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8syx-96x0

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0064

Comment on FR Doc # 2016-22002

Submitter Information

Name: Sarah Anonymous

General Comment

I value parental consent, PRIOR to having governments or research groups access a

student's or a citizen's unit records. Every citizen has the right to privacy. In the case of

minors, that right to privacy extends to the stewardship of the parent. Without such a step

in place, it leaves a minor unprotected and open to data mining and potential data and

privacy misuse for a secondary or third party's gain.

The Promise of Evidence-Based Policymaking 1213
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 11, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8syx-vgyg

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0065

Comment on FR Doc # 2016-22002

Submitter Information

Name: Ed Felt

General Comment

This increases the possibility of another party, illegally, or through means of changing

current law, to access data that students or minor student's parents, do not give

permission to said party access.

1214 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 11, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sz1-cygp

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0066

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

To Whom it Concerns,

Data mining, or any sort of data collection having to do with a minor child, with out the

consent of a parent, is an invasion of privacy. Period. 

Operating under a shroud of secrecy isn't necessary, when you're dealing with legitimate

purposes. State your case, and then get parental consent before any data collection. 

Regards.

The Promise of Evidence-Based Policymaking 1215
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 11, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sz4-iowj

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0067

Comment on FR Doc # 2016-22002

Submitter Information

Name: Susan Wilcox

General Comment

Census for genealogical purposes is a good thing and should be thorough. But using that

information for any other purpose should be banned.

1216 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 11, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sz4-rgfx

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0068

Comment on FR Doc # 2016-22002

Submitter Information

Name: Diane Sekula

Address:

Nashua, NH, 03064

Email: dianerosesekula@gmail.com

General Comment

Teachers give tests, and look at, "data" in the form of knowing their students and looking

at their work. Data is helpful, standardized tests are helpful. However, this colossal

takeover of data-informed education does not work for children and is often not much

more than a thinly disguised effort to cash in on government money (amongst other

things). As a teacher, the reality of collecting information on children that is often not

accurate and can be dangerously subjective and then sharing it with others in a way that

although promised as being safe, has no guarantees is shameful in a country such as the

United States. ( Do children always try when they are sick of sitting still and sick of

being tested? If their parent is sick? If they are hungry? Are the questions good

questions? Do they assess what they are supposed to? Are they appropriate for English

Learners and students with an IEP?) Furthermore, the potential implications of labeling

and mislabeling children through inaccurate and insecure data include civil rights

violations. I saw it happen multiple times myself as a teacher in an, "Innovation Lab

Network" state (NH). I am beyond disgusted. Somebody needs to listen to teachers and

parents. We are already experiencing major problems (lawsuits etc.) because of this. You

simply do not collect data on children (who turn into adults) and attach it to them

The Promise of Evidence-Based Policymaking 1217
permanently, or make decisions based on data that is not controlled at the local level; no

matter what you believe the benefits are, in a free country.

1218 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 11, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sz4-70un

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0069

Comment on FR Doc # 2016-22002

Submitter Information

Name: Nicole Truhe

General Comment

[Docket Number 160907825-6825-01], Commission on Evidence-Based Policymaking

Comments

Thank you for the opportunity to provide a response to the Department of Commerce's

September 14, 2016 Federal Register Request for Comments regarding the Commission

on Evidence-Based Policymaking. 

Data Infrastructure and Access

Question 5. America Forward Coalition members share a commitment to using data to

track progress and ensure accountability. That is why we are particularly supportive of

the focus of the Commission on developing a strategy for increasing the availability and

use of data in order to build evidence about government programs. The availability and

use of administrative data is of particular interest because, as we know from our

members' experiences, it is extremely important for engaging in meaningful evaluations

and for driving down the cost of randomized controlled trials. In addition to availability

and use of data, our Coalition members have identified other challenges associated with

data in the context of building the evidence of what works. These challenges include:

antiquated data systems, allowing specific datasets to "speak to" each other, need for

state or national database, service delivery and untrained government workforce. 

Data Use in Program Design, Management, Research, Evaluation, and Analysis

Question 15. Using the example of two administrative datasets, the National Directory of

The Promise of Evidence-Based Policymaking 1219
New Hires and Unemployment Insurance Wage Records, subject to appropriate privacy

protections, the federal government should amend laws and regulations to permit federal,

state, and local education and workforce agencies, state and local workforce boards,

nonprofit organizations, social enterprises, and post-secondary institutions certified by

the Department of Education for participation in Title IV of the Higher Education Act

(HEA) to have access to the National Directory of New Hires and to Unemployment

Insurance Wage Records. Access to such data could be used to help more rigorously

evaluate the impact of a variety of programs and interventions funded by federal

dollars.In addition to access to datasets, another barrier associated with the use of

administrative data in marrying it with state, local, or provider collected data is the lack

of established uniform definitions of metrics. By way of example, the Department of

Labor, in consultation with the Departments of Education and Health and Human

Services, should establish for all federal workforce, education and training programs a

uniform definition of job placements, recognizing the changing nature of work in our

economy, and an accurate methodology for calculating job placement rates of program

participants.

Questions 16-19.Tiered evidence models and innovation funds have been authorized

across different agencies and included in various issue areas to develop and build the

evidence base of what works in education and social services. One example, the Social

Innovation Fund (SIF), makes grants to experienced intermediary organizations that are

well positioned within communities to identify innovative, evidence-based programs

with potential for expansion in the areas of economic opportunity, healthy futures, and

youth development. In total, more than 400 nonprofit organizations are being funded by

the SIF to conduct diverse interventions and evaluate results through highly rigorous

models. One challenge facing innovations funds like SIF and tiered evidence programs

such as the Investing in Innovation program (i3) at the Department of Education is

identifying a pathway to integrating and expanding approaches determined to have

evidence of effectiveness under these funds in a manner that aligns with existing federal

efforts to address national and local challenges. In fact, despite the SIF's widespread

success and the bi-partisan support of i3, many federal agencies are unaware of the

innovations supported by these funding streams that could help improve existing

programs, yielding better outcomes for vulnerable communities. As the Commission

builds out its priorities and scope of work, our Coalition suggests that you encourage

federal agencies to adopt relevant data-driven programs/strategies with evidence of

effectiveness currently supported by the Social Innovation Fund, the Investing in

Innovation Fund and other similar programs whose evaluations, data, and experience of

engaging partner organizations are of particular value to federal agencies tasked with

providing effective services in the areas of economic opportunity, healthy futures, and

youth development.

Thank you for the opportunity to submit formal comments on the work of the

Commission on Evidence-Based Policymaking. We would be happy to provide

clarification of any of the points raised or to provide additional information. 

Nicole Truhe, Government Affairs Director, America Forward

1400 Eye Street, NW, 400 Washington, DC 20004

Nicole_truhe@newprofit.org (202) 780-4401

1220 The Promise of Evidence-Based Policymaking
Attachments

November 2016 America Forward Comments on Cmsn on Evidence-Based

Policymaking FINAL

The Promise of Evidence-Based Policymaking 1221
:%!*+#(BGAJAHICF6GICF6AB;

%##))%$%$,$6)%"/#!$%##$*)

%,#(BB1CABG

(3*($3(#1(


(3%$	)!$)1%6(

%##))%$%$,$6)%"/#!$

33$)+)+(+

EGAA",(	""%

+*"$1
CAHEG

((-%#$(#$%6(#$	)!$)2

$!/%+%(*%&&%(*+$*/*%&(%,()&%$)*%*&(*#$*%%##(4)&*#(BE1

CABG(")*('+)*%(%##$*)(($*%##))%$%$,$6)

%"/#!$3

!!&!)$%$&(*)$%(**%+$*)%"$$%,*%()-*&%"/#!()$,$

&+"&%"/$**%)*()$$%,*%$1(-()()+"*)1*"/0)(%))6)*%(&(*$()&)1$

*($)"*)"%"#&*$*%$*%$"$3*#(%(-(1-#&%$$$%,*,1*,1

$$*)%"+*%$)**("&$*%*!"%+(%+$*(/4)#%)*&())$)%"&(%"#)3%

*)*(%+$*-%(!%#%(*$HA)%"$$%,*%$%($0*%$)1*!!&!#1

-%((,$&(%())$())+)+*%$1-%(!%(,"%&#$*1("/"($$1&+"

"*1&/%()+))1$$*%$")(,$#%(*$BE1FAA%##+$*)$*%$-1*%+$*

",)%I#""%$#($)/(3+(-%(!)(%+$$*("-%("1%##+$*/6)

.&($)%*)%($0*%$)$*%)*/)(,3)(%+$$1-*$!1)(,)+).*(#"/

-""$$%*%$"/$*/$$#&%$$&%"/)%"+*%$)**-"",("#$$+"#&*+*

%+("*/*%&%$**%*$".#&")%*))%"+*%$)$*%$1$*%("&%&"-%*)

)%"+*%$),*%+1)*)+)&(*(%#%*(%($0*%$)3

*#(%(-(1-,"+$$%,*%$)$))$*"#$)#%(#&(%,$%+($*%$4)

$/$*,$))$())$)%"""$)3,-$$%,*%$)/"$-

&**($6$$),"%&1))))1*)*1,"+*1$*$($8*)+))+""/

,)(*(,"+9%($%$8*%)$%*93%(*)/"*%-%(!$*)%"&%"/($1

*(#+)*"(*/(($*$#")%+*1'#(*%*(/$-&&(%)$&(%,()1

"$!#"("#*%*(#$*&&(%)*,1$!"$!"##&"#%

!"$#"3$%(*+$*"/1*)*((%,($#$*&(%(#**%&*)$)+&&%(*)*)/"3%%

%*$1%+*%#)($%*"("/)&%(%+)%$*""1&&(%)(*%%$((%-"/**$

%$%*%+)%$&(,$*%$1*"*/*%#)+(()+"*)(),("/"#*1$()%+()("%!

$*%)&&(%,()%(&(%(#)1,$%*()-%+"/"(*(,"+3)()+"*1*("

%,($#$*#/+$*)#$*,&(%(#)%()-*%+*%#**$*(%%*+)%

**(*&(%"#1%)*$*.&/()""%$)%%""()$+$(#$$*&%*$*"%*&%&"

-%*&(%(#)($*$*%"&3

1222 The Promise of Evidence-Based Policymaking
,"+%%+*%#)1"."*/1#)+(#$*1$&/$%(-*-%(!)(""#&%(*$**$*)%

#%,$*%-()$,$6)&%"/#!$$,(%$#$*+(($*("&%")$&(%(#)

($#$/-/)#)"$-**)3#(%(-(",)*((!/&%")**%+"

,$*%())*))%$$*$%+(%"*%$##()(&(%%&%$*)%%-*))

%+"*($)"*$*%("#&*%($,+")$%##+$*)(%))%+(%+$*(/3

#!"#!$#$!""

4,#"$!!#('"#"####!#*!#!"$""$

"#"&!#""%!""*

#(%(-(%"*%$##())(%##*#$**%+)$**%*(!&(%())$$)+(

%+$*"*/3*)-/-(&(*+"("/)+&&%(*,%*%+)%*%##))%$%$,"%&$

)*(*/%($()$*,""*/$+)%*$%((*%+",$%+*%,($#$*

&(%(#)3,""*/$+)%#$)*(*,*)%&(*+"($*()*+)1)-!$%-

(%#%+(##()4.&($)1*).*(#"/#&%(*$*%($$$#$$+","+*%$)$

%((,$%-$*%)*%($%#0%$*(%""*(")3


$*%$*%,""*/$+)%*1%+(%"*%$##(),$*%*(""$)

))%*-**$*%$*.*%+"$*,$%-*-%(!)3)""$)$"+2

B3 ,$*(%""*$,""1# $##"("#"&(*+"("/$*&+"

)*%(#!*+"**%$*"/))%($$$"/))%***%#)+(

%+*%#)$*(#$#&*3

C3 ""$)**#""%$)%#($),(//+*(%))$)$)*%()3

""%-$)&#"#"#-"#.#!)$#&%(*$*%$)(*%$%(*

%##))%$*%.&"%(3
$%((*%'+*"/#)+(()+"*)$))))#&*1*))

$+*"0*%$#+)*("**$*(%$$*$*+(%*&(%"#)/#($)$

*)%"+*%$)$,"%&/%($0*%$)"!*%)$*#(%(-(%"*%$3

D3 #%"*/%#($)$#!*+"**%*(!%+*%#))$*"##!#

#"**$))$+)%(+"$,$$)*($*$$%(*)/

&(%,()$%,($#$*)*%#%(,$6)3""%*)#+)*%$-*$

))+($%&(,/$%$$*"*/3

E3 ,$*"%$*#%+)%$%#&"$-*(+")(*(*$,#$*%%+*%#)1%*

*"!%%!(%!#&!!!#$"##&!&##%(

&+(&%))%,$*(#$*%$3

#"!!"+	#+"!+%$#+(""

34,#!!!"$!!#('"#!$""$!%("#!#%##"$!#!!

#/!%$##%#"*

)$*.#&"%*-%#$)*(*,*)*)1*
#!#!(
&!"

(#"$!!"+)+ **%&&(%&(*&(,/&(%**%$)1*("

%,($#$*)%+"#$"-)$(+"*%$)*%&(#*("1)**1$"%"+*%$$

-%(!%($)1)**$"%"-%(!%(%()1$%$&(%*%($0*%$)1)%"$*(&())1$

&%)*6)%$(/$)**+*%$)(*/*&(*#$*%+*%$%(&(*&*%$$*"
%*

	(+*%$*8	9*%,))*%**%$"(*%(/%-	()$*%$#&"%/#$*

The Promise of Evidence-Based Policymaking 1223

$)+($%()3))*%)+*%+"+)*%"&#%((%(%+)"/,"+**

#&*%,(*/%&(%(#)$$*(,$*%$)+$/("%""()3


$*%$*%))*%*)*)1$%*(((())%*-**+)%#$)*(*,*$

#((/$*-*)**1"%"1%(&(%,(%""**)*"!%)*")+$%(#$*%$)%

#*()3/-/%.#&"1*&(*#$*%%(1$%$)+"**%$-**&(*#$*)%

+*%$$	"*$	+#$(,)1)%+")*")%(""("-%(!%(1+*%$$

*($$&(%(#)$!##"1(%$0$*$$$*+(%-%(!$

%+(%$%#/1$$+(*#*%%"%/%("+"*$ %&"#$*(*)%&(%(#&(*&$*)3

35,&#+"##"#"+!"$#"!"!+"!%$#+"#$"#!%

"!!"*

36,
&#'##!"$!!(%$#!""!!""*

37,&!"$!!%$#!!##!!""*#"

'""#!#&!%$#""$""$(!!#!!""*

38,
&#'##"$%$#""(&##!'!#! $"0'!#

"""##$#)!!"*#"'""#!#&!"$

"##$#)#""$""$&#"#!#"'"#!"*

("%,($#$*)*%&&%(*+$*/*%$%+(*+)%"%-6%)*$%#0%$*(%""

(")8)9)-"")*+)%'+)6.&(#$*")$)1-%$%*('+(($%#))$#$*1

*(/$)+($**)(,)($%*$*%*%)-%$*#1+*#$*$%$*(%"$

.&(#$*"(%+&(%(3(($+"%.#&")%("&(%(#)**$%(&%(*

&(%(#,"+*%$)$*%*(&(%(#)$)$**.&($)%+""($(%#$

(&"*(%))"(((%+&%&(%(#)$+$$)*(#)3

(,$#%")$$$%,*%$+$),$+*%(0(%))($*$)$

$"+$,(%+)))+()*%,"%&$+"*,$)%-*-%(!)$+*%$$

)%")(,)3$.#&"1*%"
$$%,*%$+$8
91#!)($*)*%.&($

$*(#(/%($0*%$)**(-""&%)*%$-*$%##+$*)*%$*/$$%,*,1,$6

)&(%(#)-*&%*$*"%(.&$)%$$*()%%$%#%&&%(*+$*/1"*/+*+()1$

/%+*,"%&#$*3$CABA1*
)#-()*%*"$>CED#""%$-*#%(*$>FCI

#""%$$&(,*$$%$6("#*$+$)3
$*%*"1#%(*$EAA$%$&(%*%($0*%$)(

$+$/*
*%%$+*,()$*(,$*%$)$,"+*()+"*)*(%+"/(%(%+)

#%")3

$""$$$$%,*%$)+$)"!
$*(,$&(%(#))+)*
$,)*$$


$$%,*%$&(%(#8D9**&(*#$*%+*%$)$*/$&*-/*%$*(*$$

.&$$&&(%)*(#$*%,,$%*,$))+$(*)+$)$#$$(

**"$)-*.)*$("%(*)*%())$*%$"$"%"""$)3
$*1)&**
4)

-)&()+))$*6&(*)$)+&&%(*%D1#$/("$)(+$-(%*

$$%,*%$))+&&%(*/*)+$$)*(#)**%+""&#&(%,.)*$&(%(#)1/"$

**(%+*%#)%(,+"$("%##+$*)3)*%##))%$+")%+**)&(%(*)$)%&%

-%(!1%+(%"*%$)+)*)**/%+$%+(("$)*%%&*(",$**6(,$

&(%(#)5)*(*)-*,$%*,$))+(($*"/)+&&%(*/*%"
$$%,*%$+$1

*
$,)*$$
$$%,*%$+$$%*()#"(&(%(#)-%),"+*%$)1*1$.&($%

1224 The Promise of Evidence-Based Policymaking
$$&(*$(%($0*%$)(%&(*+"(,"+*%("$)*)!-*&(%,$*,

)(,)$*()%%$%#%&&%(*+$*/1"*/+*+()1$/%+*,"%&#$*3

#(%(-()*$)(/*%-%(!-**)#&%(*$*%##))%$*%$)+(***%$&*)%

,$$%+*%#)($+)*%,"%&&%")$+$$*(,$*%$)3",*)

)%+"%#%##%$&(***("1)**$"%"","*%())*+(($*$+*+(

""$)-)$*%$3

$!/%+%(*%&&%(*+$*/*%)+#*%(#"%##$*)%$*-%(!%*%##))%$%$,$6

)%"/#!$3-%+"&&/*%&(%,"(*%$%$/%*&%$*)()%(*%&(%,

$/*%$"$%(#*%$/%+('+)*3")%$%*)***%%$**%"(+1%,($#$*

()(*%(%#(%(-(*%"7*(+<$-&(%*3%(/%+-%+""!*%)+))%+(

(%##$*%$)+(*(3

$("/1

%"(+

%,($#$*()(*%(

#(%(-(

BEAA/*(*1*EAA

)$*%$1CAAAE

%"7*(+<$-&(%*3%(

8CAC9HIA6EEAB

The Promise of Evidence-Based Policymaking 1225
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 11, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sz4-30f1

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0070

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anna Gray

General Comment

I feel that our government does not need to be involved in tracking personal data on our

children and even adults. Parents have an innate and God-given responsibility to guard

and guide their children in matters pretaining to their personal lives. Government's role

should be to protect families, not intimidate in any way. Families are the lifeblood of a

healthy society, and children should be valued, respected, and protected.

1226 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 11, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sz5-7is8

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0071

Comment on FR Doc # 2016-22002

Submitter Information

Name: Janelle Ruesch

Address:

Salt Lake City, UT, 841030

Email: Janelle.ruesch@gmail.com

General Comment

It is unconstitutional for the Federal Government to be collecting data on individuals

without probable cause, for any reason. The fact that Congress is discussing and

considering these types of data collection shows how far away from the intent of the

Constitution the three branches of government have wondered. Please do not open this

door. Leave it to parents guide and protect their children.

The Promise of Evidence-Based Policymaking 1227
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 11, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sz5-q854

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0072

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

I value parental consent before government or research groups have access to a student's

unit records.

1228 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 11, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sz6-krox

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0073

Comment on FR Doc # 2016-22002

Submitter Information

Name: Mellany Lamb

General Comment

Nazi Germany had the same system in place to track people. Especially those whom they

did not find desirable for their diabolical Aryan race. Do not implement this system that

could be also used for evil. And how many times does technology err? Many and it is not

without fault and can and will impede a child's future, if not ruin, forever.

The Promise of Evidence-Based Policymaking 1229
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 11, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sz7-qh2h

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0074

Comment on FR Doc # 2016-22002

Submitter Information

Name: Dennis McDonald

Address:

Alexandria, VA, 22302

Email: ddmcd@outlook.com

General Comment

See attached file(s)

Attachments

how_to_combat_data_trutherism

the_case_for_data_literacy_and_federal_data

1230 The Promise of Evidence-Based Policymaking
0 

 

How To Combat 

“Data Trutherism” 

Dennis D. McDonald, Ph.D. 

The Promise of Evidence-Based Policymaking 1231
1 

How To Combat “Data Trutherism” 

By Dennis D. McDonald, Ph.D.1

Oct. 21, 2016 

Data trutherism

The recent Washington Post editorial When the facts don’t 

matter, how can democracy survive? is a disturbing piece. 

In it author Catherine Rampell documents recent research 

showing that many Americans doubt the veracity of data 

provided by the U.S. government. 

Rampell discusses this lack of trust in the context of politics

and a tendency on the part of many Americans to view 

conspiracies around every corner. Many people don't believe the numbers the government agencies use day in and 

day out to guide their legislatively mandated policies and 

programs. Rampell concludes: 

"This is how a democracy crumbles: not with a bang, but with data trutherism."

While it is probably well known that modern media allow people to surround themselves with 

information content and messaging that reinforces their beliefs and prejudices, it is rather 

shocking (at least to me) to see this "filter bubble" extended to basic facts and figures. 

But perhaps it's not surprising. People like to pick and choose their media. We probably shouldn't be surprised that people pick and choose their numeric facts as well. If you're surrounded by 

people who are unemployed, for example, is it surprising that you might have some doubts 

about, say, the improvement of national unemployment numbers since Obama took office? 

 

 

1 Copyright (c) 2016 by Dennis D. McDonald, Ph.D. Dennis (ddmcd@outlook.com) is an independent consultant located in Alexandria Virginia. Interests include project, program, 

and data management; market assessment, digital strategy, and program planning; 

change and content management; social media; and, technology adoption. Clients have 

included HHS CMS, U.S. Dept. of Veterans Affairs, National Academy of Engineering, the 

World Bank, Catalyst Rx, the National Library of Medicine, and the U.S. Environmental 

Protection Agency. Dennis’ web site is located at www.ddmcd.com. 

1232 The Promise of Evidence-Based Policymaking
2 

The growing disconnect

One problem is, the greater the disconnect between government generated statistics and the 

people’s belief in these numbers, the greater the potential disconnect between how government services are managed and the people that are served. Many federal agencies’ delivery of 

services is triggered by various metrics. If the connection between the two is not understood, 

that means how the government operates is not understood. Such ignorance is bad for a democracy where institutions rely on "the consent of the governed" and the taxes the governed 

pay. 

I wouldn't be the first to recognize a difference between ignorance about what numbers 

"mean" and whether some sort of conspiracy exists to “spin” the numbers. Unfortunately, ignorance and willingness to believe in an imaginary conspiracy theories do reinforce each other, 

especially when leaders arise who fan the flames of bigotry and intolerance. 

Today the belief might be that unemployment statistics are being "cooked" to make the current 

Administration look good. Tomorrow, reliance on the U.S. Census to apportion legislative representation as specified in the Constitution might be questioned. The day after that? Massive refusal to pay taxes because so much money is being spent on helping people “who don't live in 

my neighborhood”? 

Mistrust in basic government statistics can be a bad thing. This mistrust can't be understood 

and appreciated without knowing what other factors influence the mistrust. These other factors may include a lack of basic numeric literacy, a sense of resentment about one's status, willingness to respond positively to the appeals of a would-be demagogue, and the tiresome politicians' strategy of "running against Washington" (until they get to Washington, of course). 

What's the solution?

I'd like to think that at least part of the solution for “data trutherism" is education (of the electorate) and more transparency (on the part of the government). 

Solution #1: Education

Regarding education, that's only part of the solution given that one of the reasons we have 

reached the point of widespread data trutherism is the failure of our educational system to adequately prepare a large segment of the population to objectively understand and evaluate 

basic demographic and population statistics. This results in the dissemination of amazing errors 

like the AP’s report on Clinton Foundation donations and State Department access. There a 

sample percentage metric was erroneously projected to an entire population. Such errors can 

be widely disseminated in no time at all. Even when sponsors admit the errors, the damage is 

done. Erroneous data have already been disseminated, digested, and redistributed. 

The Promise of Evidence-Based Policymaking 1233
3 

In the face of incompetence such as AP's, one defense is an educated populace that learns to 

ask basic questions like, "Where did those numbers come from?" This is not a perfect solution, 

as reading about the LA Times’ “tracking poll” and how its weighting factors can impact its results will attest. I understood that explanation since once upon a time I designed and manage 

surveys that involved segmented samples. Someone with less quantitative experience might 

find such explanations difficult to follow. 

Solution #2: Government Transparency

This brings us to the need for greater government transparency so that more people understand the relationship between government services and how they are delivered. A step in this 

direction is the movement to provide more “open data” by many government agencies. Many 

open data programs now distribute data files along with analytical tools to help users interpret 

the constantly increasing amounts of data. That's a good thing.

Problem is, making data "open" and accessible is only the tip of the usage “iceberg,” even when 

analysis and visualization tools are also provided. People have to make sense of the data and 

interpret it in ways that make it meaningful to them. For most people that means that the data 

has to be delivered in an understandable fashion. If they don’t understand the basics of data 

analysis, making data “open” won’t necessarily help them. 

My belief is that, in the face of so much “data illiteracy," government agencies must do a better 

job of making the data it provides about its operations and its services more understandable. 

This means going beyond making data files and analysis tools available to providing consultative 

and educational services that help citizens understand data and what the data mean to them 

personally. 

One challenge is that such “outreach” programs are expensive. I would argue that doing so 

should be part of standard descriptions of government agency responsibilities and not something done separately. 

Dangers

There are dangers to involving the government in interpreting its own data. One danger is that 

government agencies will themselves “spin” the numbers to make themselves look good. How 

to combat this possibility? The first approach of course is the oversight role played by the legislative branch of government. While this can obviously be politicized, a transparently performed 

oversight role can help bring context and meaning to more people even when hearings and investigations are highly partisan. 

The second approach gets us back to education: the more people know and understand about 

how government operates, the harder it is to “spin” the numbers, no matter who is doing the 

"spinning." 

1234 The Promise of Evidence-Based Policymaking
0 

 

Just The Facts, 

Ma’am: The Case For 

Data Literacy -- and 

Federal Data

Dennis D. McDonald, Ph.D. 

The Promise of Evidence-Based Policymaking 1235
1 

Just The Facts, Ma’am: The Case For Data Literacy -- 

and Federal Data 

By Dennis D. McDonald, Ph.D.1

Nov. 10, 2016 

Fear and uncertainty are powerful motivators. So it is for 

those who have over the past 8 years devoted untold 

hours to the promotion of government transparency and 

“open data” programs. 

Given the recent election outcome, what will happen to 

the programs that were set up to make previously “hidden” Government data sets available to the public? What 

will happen to the standard-setting and data stewardship 

programs that prepare data for public consumption? What 

will happen to innovative data research and analysis programs designed to improve medical care funded by the 

Federal government programs? 

I don't know the answers to these questions. There’s always a lot of uncertainty when Administrations change. 

I do suspect that Federally sponsored statistical, data access, and data analysis programs that don't 

have direct ties to legislation and program mission statements will be among the first to experience 

scrutiny, starting with a review of how much these programs cost the taxpayer. 

Here are my biases: 

1. The public has a right to know how tax money is being spent and to what effect. 

2. Many "open data" efforts have not made a convincing link with the impacts their sponsoring 

programs or missions are supposed to be supporting. 

Here are some comments and suggestions based on my own consulting and research: 

 

1 Copyright (c) 2016 by Dennis D. McDonald, Ph.D. Dennis (ddmcd@outlook.com) is an independent consultant located in Alexandria Virginia. Interests include project, program, and data 

management; market assessment, digital strategy, and program planning; change and content 

management; social media; and, technology adoption. Clients have included HHS CMS, U.S. 

Dept. of Veterans Affairs, National Academy of Engineering, the World Bank, Catalyst Rx, the 

National Library of Medicine, and the U.S. Environmental Protection Agency. Dennis’ web site is 

located at www.ddmcd.com. 

1236 The Promise of Evidence-Based Policymaking
2 

1. Data access programs should be designed from the ground up with usage metrics and impact 

measures in mind. If that means that money needs to be shifted from data preparation to performance and measurement, so be it. Better to have a few clearly meaningful data sets available with reliable usage and impact data than many more files that are being made accessible 

because they're easy to publish. 

2. Government-sponsored data access programs need top-down support for provision of realtime and human engagement between government staff and the users and intermediaries 

who really interact with the data. Don't just toss the data out there, be prepared to explain 

what the data mean. Data access should not be treated as a PR function but as a service function that is directly integrated with how the government does its job, not something that is 

"added on" as a separately managed operation. 

3. Given the current anti-government atmosphere, the private sector must speak up about the 

value of government collected statistics on unemployment, productivity, prices, and educational performance. We saw attacks in the recent national campaign on the trustworthiness of 

government statistics. "Big business" – as well as nonprofits and the research community -- 

must demand government accountability and the gathering and publishing of authoritative 

performance data that reflect the actual state of our economy as well as the performance of 

the programs we’re funding. Otherwise we're flying blind. 

4. Rising professionals with an aptitude for and understanding of data science and statistics must 

take some responsibility for explaining what data mean to the public, just as many younger 

scientists feel some responsibility for making sure the public understands what research is 

telling them. Having spent almost a decade as a “number cruncher” myself I personally understand the joys and fascination of data analysis, modeling, and interpretation. As more and 

more data are gathered and analyzed, though, we need to make sure the public understands 

and appreciate the basics of data and data analysis. Everyone needs to know what questions 

to ask when numbers start getting thrown around. 

I'm not suggesting that everyone become a “data scientist.” I am suggesting that basic data literacy is 

becoming just as important as learning to read and write. 

After all, if we think, for example, that we can bring those manufacturing jobs back to the Midwest 

and the Rust Belt, how will we know in four years if we’re being successful? And how do we convince 

people that the numbers are reliable? 

The Promise of Evidence-Based Policymaking 1237
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 11, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sz9-7mrd

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0075

Comment on FR Doc # 2016-22002

Submitter Information

Name: Tara Mancini

General Comment

See attached file(s)

Attachments

The National Campaign Comments for CEP_Final

1238 The Promise of Evidence-Based Policymaking
Commission on Evidence-Based Policymaking Comments 

Docket ID: USBC-2016-0003 

Federal Register Number: 2016-22002 

November 11, 2016 

Dear Chair Katharine G. Abraham, Co-chair Ron Haskins, and members of the Commission: 

On behalf of The National Campaign to Prevent Teen and Unplanned Pregnancy, thank you for 

the opportunity to provide comments for the Commission on Evidence-Based Policymaking. 

The National Campaign is a research-based, nonpartisan, non-profit organization founded in 

1996. We work to improve the lives and future prospects of children and families by ensuring 

that all children are born into families committed to and ready for the demanding task of raising 

the next generation by reducing unplanned pregnancy among teens and young adults. The 

National Campaign works towards three ultimate outcomes: 

● Reduce the rate of teen pregnancy by 50% by 2026. 

● Reduce the rate of unplanned pregnancy among women age 18-29 by 25% by 2026. 

● Reduce the disparities in teen pregnancy and unplanned pregnancy rates among 

racial/ethnic and socioeconomic groups by 50% by 2026. 

Ensuring that young people have access to high quality, evidence-based teen pregnancy 

prevention education is one critical element in helping more young people delay pregnancy and 

parenting. 

Given our long-standing commitment to research, evidence, and evaluation, we applaud the 

establishment of the Commission and appreciate the important issues it is tackling. We offer 

feedback on five of the core questions that the Commission will consider as it goes about its 

important work. 

5. What challenges currently exist in linking state and local data to federal data? Are there 

successful instances where these challenges have been addressed? 

Not surprisingly, this type of endeavor raises many technological, ethical, and legal challenges, 

particularly as they relate to the balance between data access and privacy. One example that may 

be helpful to consider is the Longitudinal-Employer Household Dynamics (LEHD) program. 

We highlight this program for its ability to successfully navigate challenges associated with 

partnership formation, privacy protection, and data access while producing data that have greatly 

impacted policy. 

The Promise of Evidence-Based Policymaking 1239
6. Should a single or multiple clearinghouse(s) for administrative and survey data be established 

to improve evidence-based policymaking? What benefits or limitations are likely to be 

encountered in either approach? 

While the Campaign strongly supports greater access to administrative and survey data, and a 

clearinghouse would be beneficial in theory, we believe such an effort would likely fall short of 

its goals and would be difficult to maintain. It is particularly difficult to imagine a single 

clearinghouse that gathered data across all policy domains in a way that adequately captured the 

complexities of these data and the programs they reflect. Rather, we believe those resources 

would be better committed to helping agencies maintain and enhance the data access they 

already have in place. In our experience, as these agencies try to meet growing data collection 

costs with fixed or even diminishing budgets, the availability of policy relevant data has been 

shrinking in critical ways. Key questions have been cut from surveys and online access to data 

has been curtailed. This is particularly true as it pertains to tabulating results for states or 

localities. For example, one can no longer use the online vital statistics data to look at key policy 

questions like state-level variation in Medicaid or WIC participation associated with pregnancy. 

It is also the case that some particularly rich data, such as the Medicaid Max files, are not 

available as de-identified files, thus making them difficult to obtain and utilize. This is in 

contrast to the Medicare data files, which are available on a de-identified basis and thus much 

more informative and widely used. There are likely similar limitations to data access in other 

policy domains as well. We believe that with relatively modest investments and vocal 

champions, data access could be greatly expanded. 

15. What barriers currently exist for using survey and administrative data to support program 

management and/or evaluation activities? 

We have found that for analytic results to truly inform programs and resonate with stakeholders, 

they must be reflective of the populations in need and the localities being served. Being able to 

disaggregate results by race/ethnicity, by socioeconomic status, and by state or even sub-state 

geographies is critical; yet data available through online access is increasingly limited in this 

regard, and, while data available through restricted access is often more complete in this regard, 

it is typically out-of-date by the time the lengthy application process is completed. 

We are most familiar with vital statistics (and specifically the natality data), though we expect 

similar limitations could be found in access to other data systems as well. We consult the 

natality data continuously in our work and have especially appreciated the ability to analyze 

outcomes for various subpopulations of interest. Even so, we recognize that the availability of 

these data are subject to various pressures, including fiscal constraints and privacy concerns, and 

we worry that access is becoming more restricted—in particular, access to data that enables us to 

tabulate a variety of state-level indicators—something our state partners have found greatly 

informative. 

One of the earliest examples of this restriction accompanied the public release of the 2005 

natality data files—the raw data that researchers can analyze using statistical programs such as 

STATA. Beginning in 2005, these data no longer included any geographic identifiers, including 

state of residence. No longer able to tabulate state-level indicators using the raw data files, we 

1240 The Promise of Evidence-Based Policymaking
turned to NCHS’s online customized table generator, “Beyond 20/20,” and were able to continue 

tabulating most of our state-level indicators of interest. 

Unfortunately, this online tool was recently discontinued. In its place, users are now directed to 

use “CDC Wonder”—another online table generator that is also greatly appreciated but much 

more limited. In particular, this tool does not enable the user to break out natality statistics for 

younger vs. older teens, does not provide state-level teen birth rates by Hispanic origin, and does 

not feature other critically informative elements such as interpregnancy intervals, receipt of WIC, 

or whether the birth was funded by Medicaid. These elements are available in the raw data files 

but, again, those files no longer contain state identifiers. One can special-request the full data 

with geographic identifiers, but this is a fairly lengthy process. 

While we understand the need for strict procedures in releasing the full natality files, we believe 

there could be some middle ground—a publicly downloadable file that is more limited and 

contains at least some geographic identifiers but that suppresses values as needed to protect 

confidentiality. Alternatively, restoring the “Beyond 20/20” tool or modestly expanding the 

“CDC Wonder” tool would be of great value to researchers in the field of maternal and infant 

health. 

16. How can data, statistics, results of research, and findings from evaluation be best used to 

improve policies and programs? 

We offer two examples of tiered evidence grant making from the US Department of Health and 

Human Services (HHS) that use evaluation results to continually improve those programs. 

The Teen Pregnancy Prevention (TPP) Program and PREP, like the Maternal, Infant and Early 

Childhood Home Visiting program, have been recognized as pioneering examples of tiered 

evidence-based policymaking,i

 and represent an important contribution to building a body of 

evidence of what works. They include high quality implementation, evaluation, innovation, and 

learning from results. The majority of funding from the TPP Program and PREP goes toward 

replicating program models that have been demonstrated to change behavior using well 

recognized high standards of evidence. A smaller portion of funding is reserved for research and 

demonstration projects to develop, replicate, refine, and test additional models and innovative 

strategies. This ensures that the menu of effective approaches to reducing teen pregnancy will 

continue to grow and be refined. 

TPP Program and PREP grantees can choose from a list of effective models that have been 

identified through HHS’ ongoing systematic review of the teen pregnancy prevention research 

literature. Since 2009, HHS has sponsored this review of the literature to help identify models 

with evidence of effectiveness in reducing teen pregnancy, sexually transmitted infections 

(STIs), and associated sexual risk behaviors. The review, conducted by Mathematica Policy

Research, looked at hundreds of evaluations and initially identified 28 models that met Tier 1 

criteria. That is, they must have been evaluated using a randomized controlled trial or quasiexperimental design, demonstrate changes in behavior (not just knowledge or behavioral intent), 

and results must be published in a peer-reviewed journal. The evidence review is updated 

periodically to capture the latest evaluation studies, and now includes 44 models.ii The wide 

The Promise of Evidence-Based Policymaking 1241
range of models on the HHS list of evidence-based programs gives grantees the flexibility to 

choose an effective approach that reflects their needs, population, and values, recognizing that 

what people in New York City may choose for high school age teens might be different from 

what people in Mississippi choose for middle school youth. 

The TPP Program is a discretionary program administered by the Office of Adolescent Health 

(OAH) that was originally funded in FY 2010 at $110 million. It supported an initial cohort of 

102 grants for a five-year period. Funded at $101 million for FY 2016, the TPP Program 

currently supports 84 competitive grants to a broad range of organizations and agencies serving 

youth in 39 states and the Marshall Islands. The grantees focus intensely on communities with 

the highest teen birth rates and the most at-risk youth. These five-year grants were awarded in 

FY 2015 and are contingent on continued appropriations. As noted above, approximately 75% 

of the grant funds are used to replicate program models that have already been shown through 

careful evaluation to change teen behavior (Tier 1), and approximately 25% of the funds support 

research and demonstration projects to develop, replicate, refine, and test additional models and 

innovative strategies to prevent teen pregnancy (Tier 2). 

PREP, established in FY 2010, continues to be funded at $75 million in mandatory funding 

annually through FY 2017. Administered by the Administration on Children and Families 

(ACF), PREP supports states, communities, and tribes to educate adolescents on both abstinence 

and contraception to prevent pregnancy and STIs, and on other adulthood preparation topics such 

as healthy relationships, communication with parents, and financial literacy. PREP focuses on 

youth at greatest risk of teen pregnancy and geographic areas with high teen birth rates. For 

example, 34% of grantees targeted youth in foster care and 74% target youth in high need areas.iii 

Most of the PREP funding ($58 million) supports grants to states, territories, and tribes and 

emphasizes the use of evidence-based programs. Indeed, more than 95% of youth served by the 

state grants received one of the evidence-based programs from the HHS list referenced above.iv 

An additional $10 million supports competitive grants to public and private entities to develop, 

replicate, refine, and evaluate innovative strategies to reduce teen pregnancy and repeat 

pregnancies among youth up to age 21. These grants are subject to rigorous evaluation and 

reflect a “Tier 2” approach that supports innovation, fills gaps in existing programs for 

underserved populations, and expands knowledge about what works. 

Both programs have invested heavily in the highest standards of evaluation and learning, as well 

as in innovation. OAH funded 41 rigorous evaluations during the first round of TPP Program 

grants that ran from 2010-2014.v

 The recently released findings—90% of which were from 

randomized control trials—indicate that four of the Tier 1 programs were found effective in 

changing behavior in additional settings and new populations. Among the Tier 2 grantees, 8 

new, innovative models were found to be effective. Overall, these evaluations help build a body 

of evidence about where, when, and with whom specific models are most effective, and have 

expanded the menu of effective program models from which communities can choose. The 

results, along with implementation lessons, also help guide the second round of TPP Program 

grantees, and the many communities that look to the HHS list of evidence-based programs for 

guidance on what approaches will work best for them. Many of these findings and valuable 

implementation lessons were recently published in a special supplement of the American Journal 

of Public Health.

vi PREP grantees have also been subjected to rigorous evaluations through 

1242 The Promise of Evidence-Based Policymaking
several different federally sponsored studies, and several studies have already been added to the 

HHS evidence review. 

The commitment to evidence-based investments and innovation in the area of teen pregnancy 

prevention has been pivotal in changing the landscape. Before these two programs began, there 

were no federal investments dedicated to evidence-based teen pregnancy prevention programs; 

research in this area had primarily come from private investments, with few resources available 

to replicate or further evaluate the existing models. 

19. To what extent should evaluations specifically with either experimental (sometimes referred 

to as “randomized control trials”) or quasi-experimental designs be institutionalized in 

programs? What specific examples demonstrate where such institutionalization has been 

successful and what best practices exist for doing so? 

The National Campaign also offers PREP and the TPP program as two examples where 

evaluation—specifically randomized control trials and quasi-experimental designs—have been 

successfully incorporated into the program designs. These are two of the few government 

programs that use evidence and evaluation criteria throughout the grant life cycle.vii In fact, only 

about $1 out of every $100 spent on federal programs is backed by any evidence that the money 

is being spent wisely. 

We believe rigorous evaluations have been successfully implemented for a few reasons. 

Importantly, the legislation for both programs specifies that some portion of funds should be 

used for evaluation. Program requirements also signify that evaluations are a priority. For 

instance, PREP grantees must participate in a federally-led evaluation, if chosen, and the “Tier 

2” innovation grantees are required to conduct their own rigorous evaluations, unless selected to 

be part of a federally-led evaluation. All TPP Program grantees are required to conduct some 

program evaluation, with a subset selected for rigorous impact evaluation.viii In addition, there 

are several federally-led evaluation studies that include large, multi-state, rigorous evaluations 

conducted under contract to OAH.ix Besides rigorous evaluations, mandatory reporting of 

performance measures is another way that OAH and ACF ensure grant projects are making 

sufficient progress toward their stated missions and that there is continuous quality improvement. 

Of course, providing support for grantees is another vital component to ensuring evaluations are 

successful. From review of initial evaluation designs to preparation of the final evaluation 

reports, TPP Program and PREP grantees received ongoing evaluation training and technical 

assistance support to ensure rigorous methods and reporting.x, xi In addition, it is essential to 

have a commitment to evaluation and learning from program leadership and adequate federal 

staff capacity to carry out that commitment. Leadership at OAH and at ACF demonstrated such 

commitment, built staff capacity, and worked closely with evaluation experts at the ACF Office 

of Planning, Research and Evaluation (OPRE) and the Office of the Assistant Secretary for 

Planning and Evaluation (ASPE). 

In closing, thank you for considering our input for the Commission for Evidence-Based 

Policymaking. If you have any questions or need additional information, please contact me at 

202-478-8512 or kkaye@thenc.org. 

The Promise of Evidence-Based Policymaking 1243
Sincerely, 

Kelleen Kaye 

VP, Research and Evaluation 

i

 Results for America. (2015). Invest in What Works Fact Sheet: Federal Evidence-Based Innovation Programs.

Washington, DC: Author. Retrieved October 12, 2016 from http://results4america.org/policy-hub/invest-works-factsheet-federal-evidence-based-innovation-programs/. Also, Haskins, R. (2014). Show Me The Evidence. Washington, 

DC: Brookings Institution Press. 

ii Lugo-Gil, J., Lee, A., Vohra, D., Adamek, K., Lacoe, J., & Goesling, B. (2016). Updated findings from the HHS 

Teen Pregnancy Prevention Evidence Review: July 2014 through August 2015. Washington, DC: Mathematica 

Policy Research. Retrieved October 12, 2016 from 

http://tppevidencereview.aspe.hhs.gov/pdfs/Summary_of_findings_2015.pdf. iii Administration for Children and Families/Family and Youth Services Bureau. (2015). Adolescent Pregnancy 

Prevention Program Fact Sheet. Washington, DC. Author. Retrieved October 12, 2016 from 

www.acf.hhs.gov/programs/fysb/resource/app-fact-sheet. iv Administration for Children and Families/Family and Youth Services Bureau. (2015). Personal Responsibility 

Education Program: How States Planned and Implemented Adolescent Pregnancy Prevention Programs: State 

PREP Performance Measures of Structure, Cost, and Support for Implementation. Washington, DC: Author. 

Retrieved October 10, 2016 from www.acf.hhs.gov/sites/default/files/fysb/prep_pm_brief_20151216.pdf. v

 U.S. Department of Health and Human Services, Office of Adolescent Health. (2016). Results from the first round 

of TPP grantees. Retrieved October 12, 2016 from www.hhs.gov/ash/oah/oah-initiatives/tpp_program/cohorts-fy2010-2014.html. vi Morabia, A. (Ed.). (2016). Building the Evidence to Prevent Adolescent Pregnancy: Office of Adolescent Health 

Impact Studies (2010-2015) [Special issue]. American Journal of Public Health, 106(S1). Retrieved October 12, 

2016 from http://ajph.aphapublications.org/toc/ajph/106/S1. vii A recent GAO report includes TPP in its review of five tiered evidence grant programs, noting evidence is used 

throughout, including for assessing the evidence base and identifying evidence-based approaches, implementing 

evidence-based approaches with fidelity, conducting rigorous independent evaluations, and disseminating evaluation 

results. 

viii U.S. Department of Health and Human Services/Office of Adolescent Health. (2016). TPP Evaluation and 

Performance Measurement. Retrieved October 10, 2016 from http://www.hhs.gov/ash/oah/oahinitiatives/evaluation/grantee-led-evaluation/grantees-2010-2014.html. ix http://www.hhs.gov/ash/oah/oah-initiatives/evaluation/federal-led-evaluation/index.html x http://www.hhs.gov/ash/oah/oah-initiatives/evaluation/ta.html. xi Goerlich Zief, S., Knab, J., & Cole, RP. (2016) A Framework for Evaluation Technical Assistance (2016). 

American Journal of Public Health, 106(S1): S22–S24. Retrieved October 10, 2016 from

http://ajph.aphapublications.org/doi/full/10.2105/AJPH.2016.303365. 

 

1244 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 11, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sz9-a6kg

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0076

Comment on FR Doc # 2016-22002

Submitter Information

Name: Richard Bayless

General Comment

Please do not support the data collection of our students (children). They are resilient,

growing and maturing into a designed destiny that should not be directed or monopolized

by, but not limited to, third party interests. They are not a commodity for revenue or the

benefit of the state.

The Promise of Evidence-Based Policymaking 1245
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 11, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8szb-vovo

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0077

Comment on FR Doc # 2016-22002

Submitter Information

Name: M. Jones

General Comment

No government has the right nor should have the ability to collect and/or retain personal

information for any induvidual unless that person has committed and been convicted of

serious crimes.

1246 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 11, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8szb-8tz0

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0078

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

I strongly oppose any type of data collection on my children and family. There is

absolutely no legitimate reason that in a free society the government ever needs to

collect personal data of any sort on it's citizens. This would be serious government

overreach and will not be tolerated. My family's personal information is private!

The Promise of Evidence-Based Policymaking 1247
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 11, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8sze-jo2i

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0079

Comment on FR Doc # 2016-22002

Submitter Information

Name: Melissa Draper

Address:

Smithfield, UT, 84335

Email: melissadraper3161@msn.com

General Comment

1. I am concerned that this commission is looking for ways to overcome the safeguards

that have been put into place to protect the data of individuals and especially student

data.

5. The challenges that exist in integrating local and state data with federal data are

privacy protections for individual citizens and should not be overcome.

11. Personally identifiable information should not be collected on students for review by

any "qualified researchers and institutions" because personally identifiable information

should not be collected on students and held in any way by the federal government. This

is not necessary to educate children.

19. Evaluations with experimental or quasi-experimental designs should NEVER be

institutionalized in relation to education. Our children are not guinea pigs on whom to try

out the latest theory and/or whim of so-called experts.

I am writing as a concerned citizen, parent, and grandparent. We need less federal

interference in our local schools, not more data mining. Data gathering on our children,

as well as on any citizens, is an invasion of privacy that is entirely a move away from

1248 The Promise of Evidence-Based Policymaking
freedom for our country. Thank you for your consideration.

Sincerely,

Melissa Draper

The Promise of Evidence-Based Policymaking 1249
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 12, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8szh-tc5x

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0080

Comment on FR Doc # 2016-22002

Submitter Information

Name: Paulette Bird

General Comment

Parents should have the right to decide how much, if any, of their child's information is

used/accessed/collected.

1250 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 12, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8szh-3ga3

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0081

Comment on FR Doc # 2016-22002

Submitter Information

Name: Lisa Halliday

General Comment

I echo the testimony of Emmett McGroarty: We urge the Commission to resist calls to

repeal the statutory prohibition on the development, implementation, or maintenance of a

federal student unit-record system. Such a system would curtail liberty interests of the

individual, would invite the collection and use of ever-more data, and would

fundamentally alter the relationship between the individual and government in a way that

is incompatible with our constitutional republic.

This kind of data collection is not necessary in education. If the commission is trying to

practice evidence based policy, I see zero evidence of the need for a federal student unitrecord system. 

Sincerely,

Lisa Halliday 

Mother of 2 and homemaker

The Promise of Evidence-Based Policymaking 1251
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 12, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8szi-nrhx

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0082

Comment on FR Doc # 2016-22002

Submitter Information

Name: Jennifer McCarthy

General Comment

I oppose any sort of national student database. Not only does the government not have

the right to our children's information, but it is absolutely appalling to me that

corporations want access to more and more of our children's data. We must not allow

tracking of citizens from school throughout their careers. Our children's privacy rights

must be respected ... whether from corporations or from the government. Please do not

allow this to continue.

1252 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 12, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8szm-o101

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0083

Comment on FR Doc # 2016-22002

Submitter Information

Name: Pamela Smith

General Comment

I value parental consent, and have been an advocate for parental rights for many years.

Collecting data from citizens by force or without consent goes against 4th Amendment

rights of privacy. Our country was founded on the principle that the government serves

the people, and does not control them. Do not allow this to continue!

Pamela Smith

"Principles of Freedom Forum"

The Promise of Evidence-Based Policymaking 1253
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 12, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t00-vhvq

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0084

Comment on FR Doc # 2016-22002

Submitter Information

Name: Charlotte Greenbarg

Address:

Lutz, FL, 33559

Email: cgreenbarg@outlook.com

General Comment

I am against the efforts to establish a federal "unit record" database that would create an

individual data dossier of college students linked to their employment, with the strong

likelihood that it would be connected to the K-12 longitudinal system.

1254 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 12, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t00-yrm7

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0085

Comment on FR Doc # 2016-22002

Submitter Information

Name: Van Harvey

Address:

Saint Charles, MO, 63304

Email: vanharvey2@msn.com

General Comment

Neither our schools nor our students are suitable subject materials for social science

experiments in government sponsored research. Our students are not suitable 'objects of

research', the 'data' you seek to collect is not yours to gather, nor is data harvested from

their time in school in anyway suitable for government use in plotting out 'successful'

new strategies for public policy.

Our public schools exist solely to assist in developing informed students, capable of the

judgment necessary to becoming self governing individuals, living in society with others.

What lives they may go on to lead, or how useful they might be to furthering various

business interests, or any benefits which crunching their data might seem to yield

through such unwarranted research, is none of the School's business, and none of the

government's business.

Your business is providing a means for their education - nothing more.

The Promise of Evidence-Based Policymaking 1255
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 12, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t01-6t66

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0086

Comment on FR Doc # 2016-22002

Submitter Information

Name: KAren Bracken

Address:

Ocoee, TN, 37361

Email: karen.bracken@reagan.com

General Comment

There is no need for the federal government to collect so much data on the American

people. This is what Communists do to their people. STOP COLLECTING DATA. You

already know more about us than you have any right to know.

1256 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 12, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t02-kkg9

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0087

Comment on FR Doc # 2016-22002

Submitter Information

Name: Mary Neubecker

General Comment

I oppose a federal "unit record" data base system creating a dossier on college students

linked to their employment and would probably be linked to a K-12 longitudinal data

system. Oppose S 1195. This whole data system is a direct reflection the same system

that is used in Communist China. It tracks children throughout all their schooling

including college right into the workforce which was predetermined for them as to which

jobs they would work in for the rest of their lives.

This is not freedom to learn, to choose a job or career; this is communism.

The Promise of Evidence-Based Policymaking 1257
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 13, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t05-2q20

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0088

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

I believe data collection on any American citizen, in any form, without consent or court

order is invasion of privacy! Especially when it concerns minors.

1258 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 13, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t0c-v1u9

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0089

Comment on FR Doc # 2016-22002

Submitter Information

Name: Kathryn McCoy

General Comment

I do not support s. 1195 and neither should Senator Rubio, I voted for you sir and my

hope that you were also against this mindless tracking of students.

Please vote no on s.1195

Thank you,

The Promise of Evidence-Based Policymaking 1259
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 13, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t0e-mmnk

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0090

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

I am a parent of 3 children ages 5-12. I vehemently oppose a national database of

children's educational and social and emotional data. This will further line the pockets of

the for-profit ed tech and educational industry and will do actual harm to our children.

The amount of money being spent for data, and technology, and software programs is

astronomical, and is a tragedy when you consider how far that same money would go

towards truly in investing in children through smaller class sizes, social workers,

counselors, behavioral specialists, arts, music, school gardens. 

My children have always taken whatever standardized tests are thrown their way. I've

seen how those end of year and after every unit tests twist and damage what teachers

actually do in the classroom. The computer based tests are absolutely shocking in how

poor quality, confusing and developmentally inappropriate they are. 

Enough. Let teachers teach and let's support teachers in the classroom in public schools.

1260 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 13, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t0f-jxqn

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0091

Comment on FR Doc # 2016-22002

Submitter Information

Name: Jay Simpson

General Comment

A nationwide tracking database of students, whatever the reason, is neither true to the

U.S. constitution nor common sense. Please oppose at all costs, and do not give way to

corporate interests.

The Promise of Evidence-Based Policymaking 1261
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 13, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t0f-je4u

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0092

Comment on FR Doc # 2016-22002

Submitter Information

Name: James Poulsen

General Comment

Our schools a have been taken over by Democratic ideologues who have squelched all

other opinions except what they wish this began in 1965 so three generations have not

been taught how to think only what to think, In addition the unions have allowed all

teachers and administrators to be shoddy employees raping the sick day programs and

lying about their workload.

So many teachers are lazy and more tied to the unions than to the students. Students are

used only for discussion with little learning being done and the results are shown in the

many, many high school and college students protesting on issues they know nothing

about having never been taught the subjects only brainwashed by the teachers, the unions

and George Soros network of anti-American one-world globalists who have been ousted

with Trump. 

Two books cover these subjects with facts:

The Closing Of The American Mind" By Allen Bloom and "One-Party Classroom" By

David Horowitz and Jacob Laksin, exposing the brainwashing for 50 years in America's

colleges.

In addition you may look up the hundreds of George Soros' funded protests and antiAmerican globalist policies and the scams he funds and the media never exposes.

http://www.ruthfullyyours.com/2010/11/16/read-this-mind-boggling-list-of-sorosfunded-organizations/

1262 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 13, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t0f-mf97

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0093

Comment on FR Doc # 2016-22002

Submitter Information

Name: Julie Anonymous

General Comment

This should NOT be approved!! The government is OVER STEPPING into private lives

of citizens on the premise of "just gathering data". You are spying on school kids and

their families and you know it. Our government should be in place to protect the citizens,

not to slowly erode their freedoms. The more data you have on each citizen just strips

more freedoms away. This is merely data mining for more control of "we the people",

who you should serve properly, not probe constantly.

The Promise of Evidence-Based Policymaking 1263
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 13, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t0g-60b4

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0094

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

I'm opposed to the blanket tracking of all students K-12 in a national database, let alone

that it later potentially be linked so that employers can query on a per individual basis.

Enough of the broad tracking (invasion of privacy), of individuals (let alone our

children), on the off chance that at some point in the future it may of use. Any data

collected (let alone kept), should be PII free

1264 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 13, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t0g-l2iy

Comments Due: December 14, 2016

Submission Type: API

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0095

Comment on FR Doc # 2016-22002

Submitter Information

Name: Andrew Reamer

Address:

Washington, DC, 20052

Email: areamer@gwu.edu

General Comment

See attached file(s). Submitted on behalf of the American Economic Association's

Committee on Economic Statistics.

Attachments

AEAStat CEP Comments 11-11-16

The Promise of Evidence-Based Policymaking 1265
1

AEAStat Comments to Commission on Evidence-based Policymaking 

The American Economic Association’s Economic Statistics Committee (AEAStat) is pleased to 

respond to the questions posed by the Commission on Evidence-based Policymaking in 

the Federal Register of September 14, 2016 [FR Doc. 2016–22002]. AEAStat’s responses are 

organized under three topics: 

x the synchronization of data among economic statistical agencies (responding to 

questions 7 and 9) 

x qualified researchers and institutions (responding to questions 10 and 12) 

x microdata privacy and confidentiality (responding to question 11) 

Data Synchronization 

7. What data should be included in a potential U.S. government data clearinghouse(s)? 

What are the current legal or administrative barriers to including such data in a 

clearinghouse or linking the data? 

9. What specific administrative or legal barriers currently exist for accessing survey and 

administrative data? 

AEAStat has long been concerned about fundamental weaknesses in federal economic statistics 

due to the inability of the Census Bureau to share with the Bureau of Labor Statistics (BLS) and 

the Bureau of Economic Analysis (BEA) microdata that includes information obtained from 

Internal Revenue Service (IRS) records. While Title 26, Section 6103(j)(1)(A) authorizes the 

Census Bureau full access to IRS returns for statistical purposes, BLS and BEA do not have 

similar authority. As a result, the nation’s three primary economic statistical agencies are 

severely limited in their capacity to collaborate in creating an integrated, rich, accurate picture 

of the state and dynamics of the U.S. economy. 

More specifically, agencies’ inability to synchronize their data results in: 

x The absence of a single complete, reliable, detailed set of aggregate data on the 

organization of the U.S. economy by sector 

x Principal Federal Economic Indicators (PFEIs) of lower quality and utility 

x Unnecessarily high business respondent burden—as each statistical agency must 

independently contact each respondent to obtain (often the same) information 

1266 The Promise of Evidence-Based Policymaking
2

x Insufficient understanding of microeconomic behaviors and outcomes —as economic 

researchers are restricted in the ability to link business microdata and household 

microdata records. Recent evidence has highlighted that understanding economic 

outcomes for workers depends critically on knowing the characteristics of their 

employers. 

x Diminished operational value of any federal data clearinghouse—as linking certain 

datasets would be legally impermissible 

Through significantly diminishing the quality and raising the costs of federal economic statistics, 

the barriers to data synchronization create major impediments to evidence-based 

policymaking. 

AEAStat believes that a successful federal data clearinghouse will require permitting qualified 

researchers and institutions access to microdata that permits integrated data sourced from 

federal statistical agencies as well as key holders of administrative data at the federal and state 

level. To create such integrated data will require elimination of the legal barriers discussed 

above. To that end, in April 2015 the AEA Executive Committee approved the attached 

resolution supporting congressional legislation that enables data synchronization between BLS, 

BEA and Census. 

Qualified Researchers and Institutions 

10. How should the Commission define ‘‘qualified researchers and institutions?’’ To what 

extent should administrative and survey data held by government agencies be made 

available to ‘‘qualified researchers and institutions?’’ 

12. If a clearinghouse were created, what types of restrictions should be placed on the 

uses of data in the clearinghouse by ‘‘qualified researchers and institutions?’’ 

AEAStat recommends that the definition of "qualified researchers and institutions" includes 

researchers in Principal Federal Statistical Agencies, federal policy and program research 

organizations, and academic and non-profit research institutions that agree to and meet terms 

similar to those now required for access to microdata in the federal statistical system. In 

particular, AEAStat recommends that qualified researchers and institutions obtaining microdata 

through the federal data clearinghouse should: 

x not use these data for any regulatory or enforcement purpose, and 

x adhere to confidentiality and privacy guidelines consistent with the Confidential 

Information Protection and Statistical Efficiency Act (CIPSEA) and Title 13 of the U.S. 

Code. 

Privacy and Confidentiality 

11. How might integration of administrative and survey data in a clearinghouse affect 

the risk of unintentional or unauthorized access or release of personally-identifiable 

information, confidential business information, or other identifiable records? How can 

The Promise of Evidence-Based Policymaking 1267
3

identifiable information be best protected to ensure the privacy and confidentiality of 

individual or business data in a clearinghouse? 

AEAStat recommends that any federal data clearinghouse follow best practices for disclosure 

review as used, or in development, by the federal statistical system and by the research 

community. 

AEAStat thanks the Commission for its efforts and invitation for input and looks forward to the 

Commission’s responses to the comments from us and other evidence-based policymaking 

stakeholders. 

Robert Moffitt, Chair 

November 11, 2016 

1268 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 13, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t0h-f5x9

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0096

Comment on FR Doc # 2016-22002

Submitter Information

Name: Lois Kaneshiki

Address:

DUNCANSVILLE, PA, 16635-7712

Email: lois@nb.net

General Comment

I am against creating a federal database that tracks individuals and their records. This is

not the proper function of the federal government, nor is it necessary or productive, in

my view. It is an invasion of privacy, and I don't agree that the federal government

should be keeping such records on individuals.

The Promise of Evidence-Based Policymaking 1269
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 13, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t0j-uwkt

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0097

Comment on FR Doc # 2016-22002

Submitter Information

Name: Debi Demien

General Comment

We do not want a federal Unit record database of college kids. This is an invasion of

privacy and another step toward Big Brother monitoring all of our lives. Sen. Rubio

should understand how dangerous this is!

1270 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 13, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t0k-zl29

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0098

Comment on FR Doc # 2016-22002

Submitter Information

Name: Toni Weaver

General Comment

A federal unit-tracking system which compiles personally identifiable information (PII)

without consent and with or without knowledge goes against the very concept of

individual freedom upon which this country was founded. 

Such a practice reduces each person to the status of a government possession whose

main purpose is to provide an endless stream of data to be analyzed and studied. 

When the government has the power to demand such information, the sovereign person

is no longer sovereign, but a wholly owned asset of the state. 

Just look at the de-humanizing language that is used: people are now referred to as

"units", simply a measure of "human capital". 

How far we have descended from the recognition that each person is a unique creation; a

combination of the physical (quantifiable), the intellectual (interpretive), and the spiritual

(unknowable).

The goal of any federal tracking system is to guide, manage, and enforce predictable

outcomes, which runs counter to the dynamism that has allowed our country to flourish.

Planned societies are indeed orderly, but they are also necessarily rigid and tyrannical

because the individual is sacrificed on the altar of the "common good".

When outcomes are left to "chance", there is a natural transcendent order that emerges,

because outcomes are individual-driven (potential; interest; creativity; ambition), not

government-driven (evaluation score; attributes; geography; industry demand,

The Promise of Evidence-Based Policymaking 1271
malleability).

I should mention also that whenever information is housed in any data base - even a

filing cabinet - it is subject to theft and abuse. 

Such a tracking system is guaranteed to be targeted and breeched, causing untold harm to

those whose information is contained within. It's just a fact. No data base is hack-proof. 

Kill this idea, and kill it now. It does not belong in America.

1272 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 13, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t0k-rjdh

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0099

Comment on FR Doc # 2016-22002

Submitter Information

Name: Matthew Snipp

Address:

Stanford, CA, 94305

Email: snipp@stanford.edu

General Comment

November 13, 2016

Dear Commissioners:

I write to bring your attention to the American Opportunity Study (AOS). When

completed, the AOS will support evidence-based decision-making for social issues and

interventions by strengthening the infrastructure necessary for conducting quality

research to better understand and address these issues. Rather than the study of a single

issue, the AOS is best viewed as an initiative to develop the country's capacity to better

exploit existing data by linking them in a longitudinal structure that will enable research

on social conditions and economic mobility across the lifespan of children into

adulthood. Government agencies and nonprofit organizations, especially universitybased researchers can utilize this resource capacity to better understand specific social

issues and build the evidence base for workable decisions and interventions.

The backbone of this longitudinal structure will be an intergenerational panel created

The Promise of Evidence-Based Policymaking 1273
using existing data from the decennial censuses linked together at the person level.

Subsequent linking of this longitudinal backbone with data from federal surveys and

administrative data (from federal, state, and local sources) will create both the breadth

and depth of information needed for social program evaluation. It will allow researchers

to track adult outcomes for individuals who, either prenatally or postnatally, were

exposed to favorable or unfavorable events or polices. 

The United States has seen major changes in recent decades in family structures, gender

roles, immigration patterns, occupational and industrial patterns, and labor markets. All

of these factorsand othersaffect people's long-term health, social status, educational

attainment, and economic opportunity. The (then) Chairman of the Council of Economic

Advisors, Alan Krueger, openly worried that the negative cross-national correlation

between income inequality and social mobility implies just such a reduction in

opportunities to get ahead in the United States (Krueger 2012). Governments and

nonprofits attempt to establish/modify programs to address social issues and help

provide economic opportunities and healthy outcomes for children as they move into

adulthood. However, the country's capacity to monitor trends and make long-term

evidence-based policy decisions to effect positive change has languished. 

The AOS will create an intergenerational panelusing existing data at the person levelto

study both social and economic mobility and the effectiveness of programs and policies

that affect that mobility. The AOS approach will produce a high-quality resource for

monitoring intergenerational processes because it: (a) rests on contemporaneous rather

than retrospective reports; (b) exploits administrative and census data that, given cost

constraints, would be difficult to replicate in any new or existing survey; (c) ascertains a

wide range of explanatory and outcome variables (e.g., education, income, occupation)

and thus allows mobility and program effectiveness to be examined comprehensively;

(d) supports analyses based on the full U.S. population (including tax non-filers, the

incarcerated, and immigrants) and thus avoids the selective processes that use only

administrative or survey sources; (e) provides large sample sizes and therefore supports

data-intensive analyses such as studies within key subgroups; and (f) allows for a more

accurate accounting of total error from estimates using linked data sources.

The National Academies of Sciences, Engineering and Medicine (NASEM) sponsored a

workshop on May 9, 2016 whose goal was to better understand the potential impact of

the AOS on future research, program evaluation, and policy analysis across a broad

range of social sciences. The invited speakers represented a broad array of social science

research, including research related to social and economic mobility, health, education,

life experiences, and program evaluation. Discussions took place on how the AOS could

work either alone and in conjunction with other existing resources to strengthen research

and program evaluation capabilities. The following highlights from the workshop are

cited from the published workshop summary (National Academy of Sciences,

Engineering and Medicine, 2016), and provide evidence of the value of the AOS. 

For your information, I am attaching to this comment two additional documents. One is a

summary of the aforementioned workshop. The other is a brief albeit more detailed

1274 The Promise of Evidence-Based Policymaking
description of the AOS. My colleagues and I would be delighted to discuss the AOS with

the Commission. We hope you find this useful. 

Sincerely,

C. Matthew Snipp

Burnet C. and Mildred Finley Wohford Professor of Humanities and Sciences

In the Department of Sociology

Stanford University

Stanford, CA 94305

Attachments

Worshop Summary

AOS brochure

The Promise of Evidence-Based Policymaking 1275
AUTHORS

DETAILS

Distribution, posting, or copying of this PDF is strictly prohibited without written permission of the National Academies Press. 

(Request Permission) Unless otherwise indicated, all materials in this PDF are copyrighted by the National Academy of Sciences.

Copyright © National Academy of Sciences. All rights reserved.

THE NATIONAL ACADEMIES PRESS

Visit the National Academies Press at NAP.edu and login or register to get:

±$FFHVVWRIUHH3')GRZQORDGVRIWKRXVDQGVRIVFLHQWL¿FUHSRUWV

– 10% off the price of print titles

±(PDLORUVRFLDOPHGLDQRWL¿FDWLRQVRIQHZWLWOHVUHODWHGWR\RXULQWHUHVWV

– Special offers and discounts

^

` b

BUY THIS BOOK

FIND RELATED TITLES

This PDF is available at http://www.nap.edu/23583 SHARE û ù Ą 

8 pages | 8.5 x 11 | 

ISBN 978-0-309-44701-0 | DOI: 10.17226/23583

Using Linked Census, Survey, and Administrative Data to 

Assess Longer-Term Effects of Policy: Proceedings of a 

Workshop—in Brief 

Standing Committee on Creating the American Opportunity Study; 

Division of Behavioral and Social Sciences and Education; National 

Academies of Sciences, Engineering, and Medicine 

1276 The Promise of Evidence-Based Policymaking
Copyright © National Academy of Sciences. All rights reserved.

Using Linked Census, Survey, and Administrative Data to Assess Longer-Term Effects of Policy: Proceedings of a Workshop—in Brief

Using Linked Census, Survey, and Administrative Data to Assess 

Longer-Term Effects of Policy

Proceedings of a Workshop—in Brief 

“IF WE BUILD IT, WILL THEY COME?”

Standing committee chair Michael Hout (Department of Sociology, New York University) began the workshop by 

stating its goal: to better understand the potential impact of the AOS on future research, program evaluation, and 

policy analysis across the social sciences. The work envisioned—developing the necessary linkages, at the person 

level, between decennial censuses, important survey and program evaluation data, and relevant administrative 

data—is foundational to future social research. These data would be linked in the Census Bureau’s secure data environment to provide researchers and policy analysts a robust, longitudinal national data infrastructure for the study 

of social welfare and the evaluation of public policy. Hout suggested that the workshop theme could well be: “If 

we build it, will they come?” 

Responding to the question, Timothy Smeeding (School of Public Affairs, University of Wisconsin–Madison; chair, 

workshop steering committee) said that much policy relies on assessing longitudinal processes and long-run efProceedings of a Workshop

July 2016 IN BRIEF

THE AMERICAN OPPORTUNITY STUDY

The United States has seen major changes in recent decades in family structures, gender roles, immigration patterns, occupational and industrial patterns, and labor markets. All of these factors—and others—affect people’s 

long-term health, social status, educational attainment, and economic opportunity. At the same time, the 

country’s capacity to monitor trends and make long-term evidence-based policy to effect positive change has 

languished. 

The American Opportunity Study (AOS) is envisioned to create an intergenerational panel—using existing data 

at the person level—to study both social and economic mobility and the effectiveness of programs and policies 

that affect that mobility. It will develop the capacity to link existing data as needed for approved research purposes 

within a secure data environment. To begin work on the AOS, the National Academies of Sciences, Engineering, 

and Medicine, with support from the Carnegie Corporation of New York, established the Standing Committee 

on Creating the American Opportunity Study. 

To begin its work, the committee has explored the feasibility of capturing names of the people in the 1990 

census and convened its fi rst workshop. The committee’s goal for the workshop, held on May 9, 2016, in 

Washington, D.C., was to more fully explore the value and potential uses of the AOS throughout a broad range 

of social science research. The committee also wanted to explore researchers’ data needs and how those might 

converge with the vision for the AOS.

The Promise of Evidence-Based Policymaking 1277
Copyright © National Academy of Sciences. All rights reserved.

Using Linked Census, Survey, and Administrative Data to Assess Longer-Term Effects of Policy: Proceedings of a Workshop—in Brief

2

fects of programs, treatments, and life experiences. 

“We need to move beyond the one-off studies to systematically allow limited, orderly, and secure access to 

data, which we already have,” he said. Smeeding was 

followed by Amy O’Hara (Center for Administrative 

Records Research & Applications, U.S. Census Bureau; 

member of the workshop organizing committee), who 

talked about the infrastructure within the Census Bureau that can enable the kind of data linkages envisioned. Much of the linking methodology is in place, 

and the 2000 and 2010 censuses have been linked 

to each other and to some survey and administrative 

data. A big gap is the 1990 census. “We have built an 

infrastructure and have pulled in tax data, Social Security data, housing data, and data from other sources,

 . . . and we are trying to create the most robust linkages. We are now interested in opening that data infrastructure up in a controlled fashion . . . to enable more 

studies,” O’Hara said. 

Smeeding reemphasized that the goal of the workshop 

is to provide “proof of usefulness,” moving beyond the 

core concept of social mobility to using the AOS for 

evidence-based policy making. Explaining the organization of the workshop, he said that the invited speakers represent a broad array of social science research 

that the workshop steering committee grouped into 

four categories: social and economic mobility; health, 

education and the life course; program evaluation 

and life experiences; and using the AOS cooperatively 

with other sources of data. The speakers were asked 

to discuss how the AOS, when fully actualized, could 

contribute to their own research and, more broadly, to 

their fi eld of research. 

THE AOS IN THE STUDY OF SOCIAL AND 

ECONOMIC MOBILITY

The study of social mobility speaks to issues of class formation, equal opportunity, and lifetime inequality. The 

speakers in this session addressed this area of social research with emphases on neighborhood effects and family structure.

David Grusky (Department of Sociology, Stanford 

University; member, standing committee) spoke 

about how the AOS would improve the country’s capacity to monitor and understand social mobility. 

Grusky said there are growing worries about a decline 

in social mobility in the United States, but because 

existing data cannot provide reliable information, everyone has been left with clever attempts to “make 

do.” He listed seven specifi c advantages the AOS 

would bring to the study of mobility: (1) improved 

measurement of occupational mobility; (2) multidimensional measurement of mobility; (3) better measurement of racial and ethnic heterogeneity; (4) an 

annual measurement protocol; (5) neighborhood heterogeneity (see the next presentation, by Nathaniel 

Hendren); (6) improved representation of family effects (see Laura Tach’s presentation); and (7) better 

measurement of the intergenerational reproduction of 

poverty.

Nathaniel Hendren (Department of Economics, Harvard University) discussed how social mobility might 

be rooted in neighborhood effects. His research (joint 

with several colleagues) aggregates data from U.S. 

income tax returns and tax information returns (such 

as W-2s) to study geographical variation in intergenerational mobility. He provided an example study involving 5 million families that moved among areas in 

the country: it showed that the earlier in a child’s life 

a family moves to a better neighborhood, the better 

that child does in adulthood. These results prompted 

Hendren and his colleagues to reexamine a 1990s-era 

social policy experiment program known as the “moving to opportunity experiment,” it provided families 

living in high-poverty housing projects with vouchers 

to move to lower-poverty neighborhoods. The initial 

evaluations of this experiment showed few effects on 

economic outcomes, but the reexamination—which 

focused on younger children—showed different results 

as the children reached adulthood: a higher percentage going to college and a 30 percent increase in earnings. Hendren said that an intergenerational panel as 

proposed by the AOS would be of enormous value for 

such research. He also identifi ed some potential challenges, including legal constraints, privacy concerns, 

and logistical constraints. 

Laura Tach (College of Human Ecology, Cornell University) addressed the topic of family structure and 

family demography in the context of social mobility. 

Three specifi c aspects of family demography—family 

formation, family disruptions, and family complexity—

are important to study, and such work is diffi cult to do 

with administrative and survey data. She said that an 

AOS-type data structure might help to overcome many 

of the diffi culties. 

Tach described her research using the Fragile Families 

and Child Wellbeing Study, a prospective cohort study 

of children born in the late 1990s that tracks the children over time. A 15-year follow-up is now in the fi eld. 

The study follows the child, the child’s mother, and 

the child’s father. She said that the chief limitations of 

the study are its restriction to an urban sample and 

its coverage for only the 1990s and forward. She indicated that AOS might have the potential to help with 

both limitations, as well as to help recover important 

information about unmarried fathers. She concluded: 

“A panel like the AOS, with its intergenerational linkages, is probably the only data source I can think of that 

1278 The Promise of Evidence-Based Policymaking
Copyright © National Academy of Sciences. All rights reserved.

Using Linked Census, Survey, and Administrative Data to Assess Longer-Term Effects of Policy: Proceedings of a Workshop—in Brief

3

would allow us to be able to look at how growing family 

instability and complexity affects the processes of intergenerational mobility and the reproduction of inequality.”

THE AOS IN THE STUDY OF HEALTH, EDUCATION, AND THE LIFE COURSE

Speakers in this session addressed the early life effects of 

environmental factors, health conditions, poverty-related 

stressors, and education on people’s long-term health 

and opportunities.

Janet Currie (Woodrow Wilson School, Princeton 

University) spoke on early life experiences as a root 

of inequality. Currie’s main research focuses on two 

questions: Which environmental factors affect early 

life health? How does early life health affect later life 

outcomes? In pursuing these questions, Currie merges 

birth and death records from vital statistics with hospital discharge and emergency room visit records. She 

discussed the main advantages and disadvantages of 

each source, along with examples. Linking these data 

sources to census data would add considerable value, 

she said: “It is extremely helpful just to know where 

people were born.” 

For an example, she cited forthcoming research showing that children who were born in counties that were 

subject to the U.S. Clean Air Act have higher employment and earnings and lower levels of disability as 

adults than children born in counties not subject to 

the act.1 In discussing barriers, Currie pointed out that 

most health data are “owned” by states, which vary 

in their openness to share data and in their technical 

capabilities to construct usable data structures. She 

said that it might be possible to build on existing state 

collaborations with the Agency for Healthcare Research 

and Quality through the Healthcare Cost and Utilization Project.

Ken Smith (Huntsman Cancer Institute, University of 

Utah), who works extensively with state data, focused 

his presentation on work with the Utah Population Database (UPDB). He said he thinks the UPDB includes 

the kind of breadth for Utah that the AOS is hoping 

to achieve nationwide. Smith described the Early Life 

Conditions, Survival, and Health Project, which is looking at how early life conditions affect later life health. 

One goal of this project is to try to identify at-risk populations and—by improving their understanding of 

health disparities that may arise early in life—to help 

guide interventions and surveillance strategies. Smith 

enumerated some of the administrative sources of 

data that have been brought together in the UPDB and 

1Adam Isen, Maya Rossin-Slater, and Reed Walker. (In 

press). Every breath you take—every dollar you’ll make: The 

long-term consequences of the Clean Air Act of 1970. Journal of Political Economy. 

noted that there is also a large ongoing record-linkage 

operation. “We are able to link not only parent to child, 

but we are able to link tenth cousin to tenth cousin. 

Any kind of any family connection that you can imagine—polygamy, consanguinity, all of the complexities 

that you would expect to see in any population—we 

are able to capture,” he emphasized. The goal is to 

establish family histories of health and medical conditions, some of which could have effects on social mobility, but also on individual health outcomes. Smith 

concluded by noting that the AOS as envisioned could 

help close some gaps in the UPDB, such as following 

people who migrate to different states, and working 

with administrative data from cancer registries in other 

states. 

C. Cybele Raver (Vice Provost for Research and Faculty 

Affairs, New York University) described her research 

on the role of poverty-related stressors (such as scarcity, threat, and turbulence) for children’s diffi culty in 

school. The work is based on a longitudinal follow-up 

of a randomized trial in 2003-2004 in very low-income 

neighborhoods in Chicago. In the study, teachers intervened in specifi ed ways to help children regulate 

their emotions and their cognitive functions. Raver reported: “We saw signifi cant benefi t . . . and those interventions worked for young children. What was really 

challenging is that those benefi ts were quickly eroded 

as children transitioned into both low-quality schools 

and continued to live in very dangerous, unsupported, 

environmentally less stable, and less resourced environments.” She said that she sees environmental insult 

and environmental repair as fundamentally intergenerational, and thus could benefi t from the type of intergenerational data structure in the proposed AOS, with 

stronger administrative, census-level, and populationlevel data. Raver concluded: “The extent to which [research on] long-range educational outcomes is a possibility from these kind of data is phenomenal . . . It 

would be great to be able to estimate two-generational 

effects.”

Susan Dynarski (School of Public Policy, University of 

Michigan) spoke on the linkage between education 

and economic mobility. She described several examples of her research on the effect of education policies 

on educational attainment, achievement, and adult 

well-being. Her work documents levels, trends, and inequality in educational outcomes by parental income 

and by race and gender. Dynarski uses data from both 

traditional surveys and administrative sources obtained 

through research partnerships with state education 

agencies. In cooperation with state agencies in Massachusetts and Michigan, she is building a longitudinal data system that covers students starting in 2003. 

Dynarski stated that grants from the Institute of Education Sciences is encouraging these types of partnerThe Promise of Evidence-Based Policymaking 1279
Copyright © National Academy of Sciences. All rights reserved.

Using Linked Census, Survey, and Administrative Data to Assess Longer-Term Effects of Policy: Proceedings of a Workshop—in Brief

4

ships between academics and the state agencies that 

hold educational data, creating usable data systems. 

She added: “These state data systems are where educational research is currently being done—they are 

huge, they are comprehensive, they are cheap.” It is 

pretty important to tie these partnerships and data systems into the AOS vision for linking existing data and 

data systems, she said. Better data are needed to help 

understand the effects of educational interventions on 

adult well-being—the long-term effects. Important to 

this understanding is knowing parental background in 

a more detailed way to understand variability and the 

effects of policies. 

Dynarski identifi ed several barriers in this work. On 

postsecondary intergenerational mobility, a “big constraint is that there are no comprehensive national data 

on attendance, attainment, or degrees.” She said that 

variability among state agencies in interpreting federal 

laws, such as the Family Educational Rights and Privacy 

Act (FERPA), is also a barrier. Lastly, Dynarski said there 

is an organized constituency that opposes the linkage 

of student data to anything else, presumably for reasons of privacy: she warns that this opposition will be a 

barrier to including more educational data in the AOS 

infrastructure. 

THE AOS IN PROGRAM EVALUATION AND LIFE 

EXPERIENCES

Speakers in this session discussed the issues involved in 

evidence-based program evaluation and the importance 

and diffi culty in studying the impact of two different lifecourse connections: military service and criminal involvement.

Gordon Berlin (President, MDRC) talked about the 

work of his organization, a nonprofi t, nonpartisan education and social policy research organization. MDRC 

performs evaluation studies of the effects of services for 

low-income families, typically using randomized controlled trials with mostly short-term (5 years or less) follow-up. The studies span the life cycle and many fi elds, 

including families and children, education (all levels), 

and the economic outcomes for low-wage workers and 

the hard-to-employ. To conduct these studies, MDRC 

uses primary data collected from study participants, 

state and federal administrative data, and data from 

program management information systems—which 

are usually linked by MDRC. 

Berlin talked about the limitations of the current approach to program evaluation. State administrative 

data are hard to access and diffi cult to use. There is considerable variability across agencies in terms of regulations (which may limit access), consent requirements, 

interpretations of federal policies such as the Health 

Insurance Portability and Accountability Act (HIPAA), 

and data security requirements. State agencies have 

limited resources to assist. Similarly, federal data also 

have limitations for program evaluation purposes. Often the data are designed for limited types of research; 

some are “de-identifi ed,” which makes them hard to 

link to other data; and there is often a substantial lag 

in their availability. Berlin said one of the potential benefi ts of the proposed intergenerational data in the AOS 

would be to greatly reduce the effort of collecting data 

so that MDRC could possibly examine longer-term impacts. For MDRC’s work, the intergenerational panel 

would need identifi ed data that could be linked to a 

study sample, have a short lag for timely assessment of 

outcomes, and are consistent with the study’s consent 

requirements. Other potential uses of the AOS data 

would be to help in the study design, to use in predictive modeling, and to help generalize results from the 

study sample to a larger population.

John Laub (Department of Criminology and Criminal 

Justice, University of Maryland) and William Sabol 

(Westat) jointly discussed the potential of the AOS for 

research and program evaluation on crime, punishment, and human development. To illustrate the power of using robust, longitudinal data in criminology, 

Laub described the Glueck Project, which followed 

1,000 juvenile males (500 delinquents and 500 nondelinquents) from 1940 to about 1963. In a 50-year 

update of the Glueck men’s lives,2 researchers found 

enormous variability in criminal activity among the 

men over the full life course. They concluded that trajectories of crime are infl uenced but not determined by 

prior childhood differences: desistance (from criminal 

activity) is a process and occurs even for those at highest risk for continued offending. Interventions may be 

potential turning points in the life course. 

Sabol talked about the effects of growing up in an era 

of mass incarceration. Questions continue about the 

effects of this phenomenon on labor market outcomes, 

physical and mental health outcomes, family formation 

and dissolution, family outcomes, victimization risk, 

and persistent criminal involvement. Sabol noted a key 

question in the research: “What happens to life-course 

changes of people who have been through the system 

in the 1970s versus the 1990s? I think the AOS clearly 

provides opportunities for addressing that question.” 

Sabol said the major challenge for him is “how to bring 

into this mix of information on earnings, employment, 

and so forth some measure of criminal history.” 

Alair MacLean (Department of Sociology, Washington 

State University) talked about her research exploring 

the life-course trajectories of veterans who served in 

2John H. Laub and Robert J. Sampson. (2003). Shared Beginnings, Divergent Lives: Delinquent Boys to Age 70. Cambridge, MA: Harvard University Press.

1280 The Promise of Evidence-Based Policymaking
Copyright © National Academy of Sciences. All rights reserved.

Using Linked Census, Survey, and Administrative Data to Assess Longer-Term Effects of Policy: Proceedings of a Workshop—in Brief

5

the U.S. armed forces, focusing on the effects of military service and combat exposure on work and health. 

She looks at what happens to veterans immediately 

after they get out of the military, as their postservice 

careers unfold, and when they retire. MacLean examines factors, such as earnings, income, unemployment, 

health, and disability across different cohorts (such as 

World War II, Vietnam). “They come home to very different societies, so I look at how that might differ,” 

MacLean said. Addressing the data that she uses for 

research, she mentioned the Wisconsin Longitudinal 

Study, the Health and Retirement Study, the National 

Longitudinal Study of Youth, and the Educational Longitudinal Study. MacLean identifi ed three limitations 

to these data: lack of information about family background, small sample sizes (rates of military service 

in the general population are quite low), and limited 

coverage of eras. Regarding the potential usefulness of 

the AOS, MacLean said: “I was very excited to read the 

proposal because the idea of being able to have these 

very large population-based samples that link people 

to their families would be extremely helpful for military 

research.” She believes that it would be important to 

link Department of Defense data to the AOS because it 

would enable studies of the long-term effectiveness of 

veteran’s benefi ts (such as health care and the GI bill), 

which are currently a large part of the federal budget. 

THE AOS USED COOPERATIVELY WITH OTHER 

SOURCES OF DATA AND INFORMATION

In this session, speakers discussed examples of existing 

work being done to provide or link data for longitudinal 

research in social science and how these efforts might 

benefi t from and contribute to the AOS. 

Katherine Harris (Carolina Population Center, University of North Carolina at Chapel Hill) spoke about 

the National Longitudinal Study of Adolescent to Adult 

Health (Add Health). Add Health started with a nationally representative cohort of adolescents in grades 7 

through 12 in 1995 and now follows them into adulthood. The study’s fi fth wave is about to be fi elded. 

Harris said the study has an integrative design to understand social, environmental, behavioral, biological, 

and genetic linkages to health across the life course. 

Add Health is primarily a health study with a focus on 

the role of social context. The researchers have linked 

to school records and military records; they also asked 

respondents to consent to linking to their Social Security Administration (SSA) records, and about 50 percent 

consented. 

One example of how the AOS could enhance the research work being done through Add Health would 

be to confi rm the linkages between the sampled children and their parents and to obtain more information 

about the parents. A second example is that it could 

be used to link to important administrative data that 

would provide annual income and unemployment 

data on the parents over time. This linkage would enable research on the effects of early life socioeconomic 

status (SES) that is now missing. Linkages to tax data 

would provide residential information on an annual 

basis that would be incredibly valuable for the study 

of environmental threats to health. With such data, the 

researchers could then model residential selection processes among parents and also among children after 

they leave home. Harris talked about the peer networks 

that the children have in adolescence. Linkage with the 

AOS would provide an opportunity to track these peer 

networks as the adolescents disperse later in life. All of 

this research would advance the goal of understanding 

how adolescent experience and exposures affect adult 

health. 

David Johnson (Survey Research Center, University of 

Michigan; member, standing committee) discussed his 

work as codirector of the Panel Study of Income Dynamics (PSID). The PSID began in 1968 with 4,800 families, 

with oversampling of low-SES families; it was designed 

to evaluate causes and consequences of poverty. It 

has now been running for almost 50 years. Researchers follow the original families, their children, and the 

new families that have been formed among them. The 

PSID added immigrant families in 1997. Johnson said 

that the best feature of the PSID is that it is multigenerational, containing fi ve generations of families: “We 

have the base for looking at mobility.” The PSID survey 

data are linked to administrative data, such as Medicare 

claims data, college data from the Integrated Postsecondary Education Data System, and data on assisted 

housing from the Department of Housing and Urban 

Development. The researchers now plan to link to data 

from the real estate Website Zillow and the SSA. 

Johnson briefl y described two examples of how the 

PSID and AOS could be used in concert with each other 

to evaluate long-term policy. He said that the biggest 

complaints researchers have with the PSID is that the 

sample is too small, and it is no longer representative of 

the U.S. population. The AOS would allow the PSID to 

effi ciently supplement its data and obtain more recent 

information about respondents who dropped out of the 

survey and historical information about the new people 

in the PSID sample, such as those who married into the 

PSID families. He also suggested that the PSID could assist the AOS. The AOS intends to link 1990 census data 

to data from subsequent censuses at the person level. 

Johnson suggested that since PSID participants are 

closely tracked, it could be used to fi nd the PSID participants in the 1990, 2000, and 2010 censuses, and 

The Promise of Evidence-Based Policymaking 1281
Copyright © National Academy of Sciences. All rights reserved.

Using Linked Census, Survey, and Administrative Data to Assess Longer-Term Effects of Policy: Proceedings of a Workshop—in Brief

6

evaluate whether the AOS procedures matched those 

individuals correctly across censuses. 

Martha Bailey (Department of Economics, University 

of Michigan) discussed the Longitudinal Intergenerational Family Electronic Micro-Database (LIFE-M). Just 

coming out of a pilot phase, LIFE-M is an administrative database that is going to use vital records (birth, 

marriage, death) as a basis for a long-term family network created with linkages of individuals from birth 

to death and across generations. The assembled data 

will further longitudinal analyses of large samples and 

subsamples of individuals and families over recent U.S. 

history. There are plans to link these vital records to 

census and other data. This database is going to span 

the late 19th century and most of the 20th century for 

a subset of states. Bailey said they would like to harness 

a variety of data relating to birth-family characteristics 

and to obtain economic and demographic outcomes 

from census data. The researchers are also interested 

in geographic information, specifi cally at the time of 

vital events. In discussing limitations, Bailey said that 

the coverage of vital records by state and period varies, and not every state has its records available online. 

Furthermore, if parents had children and then moved 

to a different state, it will be diffi cult to continue to link 

these individuals. In addition, the study will be limited 

by the periods that are covered. Lastly, she noted, clerical review turns out to be really important for linking, 

and such review is very expensive. The AOS has the 

potential to overcome or mitigate some of these limitations, Bailey said.

Jennifer Noyes (Institute for Research on Poverty, University of Wisconsin–Madison) discussed the collaboration between the Institute for Research on Poverty 

(IRP) and Wisconsin state agencies to inform policy and 

practice. Figure 1 graphically displays components of 

that collaboration. Through this collaboration, IRP has 

created a single record for each individual by matching and merging from the primary data sources, using 

identifying variables: these matches are complex and 

time-consuming to program. Noyes said the database 

is very fl exible for different uses, such as measuring 

program participation, understanding the relationships between programs, and measuring the effect of 

policies. For example, one study addressed the reasons 

that program participants become “disconnected” 

from programs. Noyes described three key lessons 

from this work: (1) collaboration with data custodians 

has been and will continue to be essential; (2) involvement by an outside partner who does not “own” any 

of the programs or the associated data can provide 

objectivity to the entire process; and (3) infrastructure 

requires sustained commitment by all parties and signifi cant funding. In thinking about linkage with the 

AOS, Noyes said that it could help fi ll in the gaps in the 

IRP’s information about income, education, and occu-

 

Figure 1 Logic of collaboration between the Institute for Research on Poverty and Wisconsin state agencies to support policy 

development and academic research.

1282 The Promise of Evidence-Based Policymaking
Copyright © National Academy of Sciences. All rights reserved.

Using Linked Census, Survey, and Administrative Data to Assess Longer-Term Effects of Policy: Proceedings of a Workshop—in Brief

7

pation. It could help track people outside of Wisconsin 

and could help the IRP improve its matches. 

A RELATED ACTIVITY IN CONGRESS

Bruce Meyer (School of Public Policy, University of Chicago) briefl y described the Evidence-Based Policymaking Commission, recently created by Congress. Meyer, 

a commission member, said the commission, which is 

to report to Congress in 15 months, will play a critical 

role in facilitating policy evaluation and research. The 

commission will have 15 members, 3 each appointed 

by the President and by the leaders of both houses of 

Congress. The commission’s mandate is fi rst to conduct 

a comprehensive study of the inventory, infrastructure, 

and data security of administrative and survey data 

and then to make recommendations on an optimal arrangement for using these data together. The commission is to make recommendations on the incorporation 

of outcome measures, randomized controlled trials, 

and impact analysis in program design. It might recommend the creation of a data clearinghouse. Meyer 

noted some opportunities emerging from the commission’s work: it could help show the value of administrative data; push agencies to share data while giving 

them cover; build a consensus for evidence-based legislation; and reassure the public that data are secure. 

“If there is a statement by a congressionally appointed 

commission that sharing administrative data is good 

for the country, good for policy making, and good for 

research, then I think it does help,” Meyer said.

STATISTICAL CHALLENGES

Stephen Fienberg (Department of Statistics, Carnegie 

Mellon University; member, standing committee) cautioned that enhancements in data matching and estimation methodology will be needed to fulfi ll the promise 

of the AOS. “Everyone has been talking about matching as if it is a simple thing to do and you can do exact 

matching. It just ain’t so,” he said. The most common 

approach to data linkage with multiple fi les today, and 

the one that is used at the Census Bureau, is to match 

each fi le pair-wise with the Social Security Numident fi le 

(containing names and information from applications 

for Social Security cards). Fienberg recommends an approach he calls “multiple record linkage” in which all 

fi les are analyzed together in a systematic way to make 

effi cient use of all information across all fi les and avoid 

the transitivity problems that can arise from sequential 

steps. He said that, once linked, users will have to consider how to incorporate uncertainty about matching 

into the statistical estimates based on those matches. 

Since there is error in matching no matter how well it is 

done statistically, both bias and uncertainty need to be 

measured. “We need to be able to ‘propagate uncertainty’ from the record linkage as an added component 

of uncertainty into the statistical estimation based on 

linked fi les.” Fienberg noted that research into matching and estimation methodology is part of the agenda 

before the AOS standing committee. 

NEXT STEPS

Three members of the workshop steering committee answered the questions: What did we hear? What are the 

big questions? What else do we need to do? 

Hout spoke fi rst: “The big lesson from today, and I was 

really heartened, is that we are not inventing the wheel 

here. There is a lot of experience out there that we can 

start to draw on and collect and pool together. The 

American Opportunity Study is the next big step.” Hout 

noted that the need to build relationships with statelevel agencies was repeated by several presenters. 

“There is a quid pro quo in dealing with administrative 

data. We have to offer back to those who have the data 

the products and insight they could not otherwise get. 

We are not just bringing together data or assembling 

data but enhancing and passing back better, more usable information to those who have provided data to 

us.” 

Smeeding said he has concluded that this meeting provided extensive proof of usefulness. “We heard many 

times today that ‘longitudinal’ is important, and the 

AOS can and will lead to big improvement in current 

longitudinal surveys and vice versa. A second point, he 

said, is that there is a lot of work going on using state 

data on health and education. The AOS will not get 

comparable data from all the states, but these statebased studies are all asking the same questions: How 

do kids do as adults as a result of their SES and experiences in early life? Smeeding said that there is considerable value added that the AOS can give back to the 

states on this subject once the state data are aligned to 

longer-term national data on education, health, social, 

and economic outcomes that will be available in the 

AOS. 

Bhashkar Mazumder (Federal Reserve Bank of Chicago) offered several points from the day’s speakers. 

There is a strong need for both administrative and survey data, he noted. Even the presenters whose primary 

work was with administrative data talked about how 

results could be enhanced by linking them to survey 

data. He commented that several people touched on 

the issue of data access and whether the Federal Statistical Research Data Centers might be a good vehicle 

to provide that access. He noted that there was a lot 

of discussion about the importance of linking various 

survey and other data to vital statistics from state agencies. He said he agrees with Currie’s comment that the 

AOS work should start with low-hanging fruit—the 

The Promise of Evidence-Based Policymaking 1283
Copyright © National Academy of Sciences. All rights reserved.

Using Linked Census, Survey, and Administrative Data to Assess Longer-Term Effects of Policy: Proceedings of a Workshop—in Brief

8

Copyright 2016 by the National Academy of Sciences. All rights reserved.

DISCLAIMER: This Proceedings of a Workshop—in Brief was prepared by Carol House, on the staff of the Committee 

on National Statistics, as a factual summary of what occurred at the meeting. The statements made are those of the 

author or individual meeting participants and do not necessarily represent the views of all meeting participants; the 

planning committee; or the National Academies of Sciences, Engineering, and Medicine.

REVIEWERS: To ensure that it meets institutional standards for quality and objectivity, this Proceedings of a Workshop—in Brief was reviewed by Thomas A. Kecskemethy, Offi ce of Executive Director, American Academy of Political 

and Social Sciences, Philadelphia, PA; Jared S. Murray, Department of Statistics, Carnegie Mellon University; and C. 

Matthew Snipp, Department of Sociology, Center for Comparative Studies in Race and Ethnicity, and Stanford Secure Data Center, Stanford University. Patricia Morison, National Academies of Sciences, Engineering and Medicine, 

served as review coordinator.

SPONSOR: This workshop was supported by the Carnegie Corporation of New York.

For additional information regarding the meeting, visit 

www.sites.nationalacademies.org/dbasse/cnstat/dbasse_172506.

Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2016. Using Linked Census, Survey, 

and Administrative Data to Assess the Longer-Term Effects of Policy: Proceedings of a Workshop—in Brief. Washington, 

DC: The National Academies Press. doi: 10.17226/23583.

Division of Behavioral and Social Sciences and Education

most populous states or states that make their vital statistics reasonably easy to access. 

Mazumder also summarized several issues that might 

be important for the new congressional commission to 

address. One is the in-house expertise, or lack thereof, 

of federal and state agencies to develop memoranda of 

understanding to make effective state-federal partnerships. A second issue for consideration is the apparent 

variability in interpretation of federal regulations, such 

as FERPA and HIPAA. Some federal help will be needed 

on this issue. Mazumder also noted the timeliness issue: 4-5 years may be fi ne for a research project, but 

program evaluation requires a more rapid turnaround. 

“We need to remember the need for timeliness in the 

procedures to access data for these purposes,” he concluded. 

1284 The Promise of Evidence-Based Policymaking






34

*&"!#'(-'"

%'!

!'%!%'"!#!


		'"

&'(,&( #"%'!'

'"#&&"(#'"!.

&"!"!" 

 "',.&*&'

')!&&"

#%"% &!#"&.

&(&('"!!

'.* #'

&( "',/&&

"(!'"!%&%00

)"#!'

!&&%,!&.'

'#%&"!).

'*!!!

!&(&&. #"%'!'

&(%),'&(&'

%!" (!',

(%),.!##

 !&'%')''"

"% '&#!/&'*!*'!

'!&(&&(% !&'%')%"%'

!)%"! !'.!#%'!%&#*''+&'!

!)"#!!&(&(%(!

!%&'%('(%/)%&%"(#"!,&'&!

%&%%&*'""'!&&'"'

"%##%")#%"'&*'!%''&'

&%'!'%!&(##"%'")!0

&&"!"!" #",%&%/

%!"%#"%'"!"*"%&

&#"!&"%'!'#&"'&%&%/

What is the American Opportunity Study? 


American Opportunity Study 

Contact us for more information 

%1&01%" 

%6&01%"""!"! 	

%1%0""+ ,*-)&0%"! 

%1/"1%"+"5101"-0 

 

Workshop: May 9, 2016 

Potential for Research Using 

Linked Census, Survey, and 

Administrative Data to 

Assess Longer Term Effects 

on Policy 

http://sites.nationalacademies.

org/DBASSE/CNSTAT/

DBASSE_172151 





!	
#


!
 " 


!
"
 

The Promise of Evidence-Based Policymaking 1285
&'(," "',

 ''%&(&'

&#&'"&&(&"&&

"% '"!.$(

"##"%'(!',.!' 

!$(',00'+'!'

'"*%!"%!

!'"%!' &

)%!'

!&!"('" &/

"!)!'"!

"% (&'''

%!&'"%'"%

)! %+'% 

!$(',(&',

)''

"##"%'(!'&'"'

%*,

)!''

"('" &%''!'

!"%'%'%'!

'!'"%'/

"%%'$(&'"!'"&&








	

	


1
"',

Why is the American Opportunity Study Needed? 

!,&&&!"* !,

"%!''"*%

%&&!(&''&

$(&'"!!'(&

&'!'"!*'%

'!'''&&

!)!(#'"'&

&''" ' !''"

$("##"%'(!',/

!'''&&

&! &&)!&!

%!'&! ,

&'%('(%&.!%%"&.

%0'!!

!'"!"%!.

 %'"!.

('"!''! !'&.

"(#'"!!

!(&'%&'%('(%.

"!" !$(',.

% !(&'!

"% %'&/"

'&'"%&%

%)!''"!&!

&".('"!.!

"!" "##"%'(!',

! "',/''

& ' .'"(!'%,5&

#','" "!'"%

'%!&! "',&

!(&. !'

(''")('!*

"!%!&'' "',

 ,!!"%'"

)"#)!0&

#","! "',/

&'%0&&(%),"

&" "',!'

!'''&*&

!7?=9/!

'!.'"(!'%,2&

#','" "!'"%

'%!&! "',&

 ''#%"%&&.

 !'(''"

)('%"*!

"!%!&'' "',

 ,!!"%'"

)"#)!0&

#","! "',/



• $(','

• "&'&)!&.!&!%&#"!!'(%!%('"!

• 0#%"#'!

• !!(%!"(!'"!"%'(&!!,&&" !&'%')

'&"(%&

• &(%%&"(%

• !*%&"(%"%)!0&#",)('"!

Expected dividends -- 

!&!%&#"!!'

'"!"%'(&!!

(%!%('"!

,&&" !&'%')

“The AOS is 

proposed—not as a 

new, large-scale, costly 

survey—but instead as 

a way to capitalize on 

existing data.” 

$"	 	



With passage of the 

Evidence-Based 

Policymaking 

Commission Act of 

2016, this work will 

become evern more 

important and relevant 

for future research and 

evidence-based 

evaluation of social 

policies. 

1286 The Promise of Evidence-Based Policymaking
	

 $"


			%"4&))"%&$%.2)&16/"0,2/ "#,/*,+&1,/&+$&+1"/$"+"/1&,+)-/, "00"0

" 20"&1/"010,+ ,+1"*-,/+",20/1%"/1%+/"1/,0-" 1&3"/"-,/10
$&3"+ ,01 ,+01/&+104,2)!

"!&##& 2)11,/"-)& 1"&++6+"4,/"5&01&+$02/3"6
 )),40*,&)&161,""5*&+"! ,*-/"%"+0&3")6
!

)),40#,/+)60"00"!,+1%"#2))-,-2)1&,+&+ )2!&+$15+,+#&)"/01%"&+ / "/1"!+!

&**&$/+10
+!"&0)/$""+,2$%1,"5*&+"*,&)&164&1%&+("602$/,2-0+!1,)),4#,/!1

&+1"+0&3"+)60"0

	


	




%"4&))"5-),&1!11%1%3"

)/"!6""+ ,))" 1"!#,/,1%"/-2/-,0"0+!!!3)2"1,1%,0"!1600"*)&+$1%")1"+1-+")

2+!"/)6&+$1%"*&3"+1%"0/ %&1" 12/"+602##& &"+1)6)/$"02/3"6"$

%")1%4&1%&+!&3&!2)&!"+1&#&"/0 +)0,")&+("!1,&1*(&+$&1-,00&)"1,0" 2/"+"43/&)"0#,/

1%"020"1,#&+!&3&!2)04%,--"/&+,1%1%"+!1%"02/3"6


		0+"4*"/& +,**2+&162/3"60/"#&")!"!+!+"4!*&+&01/1&3"!1" ,*"

3&))"1%")1"01&+ ,*", 2-1&,+)+!,1%"//")"3+1!1 +"--"+!"!1,1%"&+!&3&!2)

/" ,/!0)/"!6&+1%"0*-)"+!+"4"+1/+10&+1,1%"+&1"!11"0"&1%"/3&&/1%,/&**&$/1&,+4&))

 ,/!&+$)6"!!"!1,1%"&+#/01/2 12/"%&070")#/"#/"0%&+$8#"12/",#1%",3"/ ,*"01%"

-/,)"*04&1%2+/"-/"0"+11&3"+"00+!11/&1&,+1%1/"&+"3&1)6-/"0"+1&+02/3"60"!-+")!1


	

					
		
%"-,1"+1&)20",#

!*&+&01/1&3"!10,2/ "0#,//"0"/ %+!20"&+-/,!2 &+$"!"/)011&01& 0-/,*&0"01,",+",#1%"

*',/0, &)0 &"+ "!"3"),-*"+10,#,2/1&*"21!,-1&,+%0""+0),4" 20"!1)&+($""##,/10

/"+,4,+",####&/0+ ,+1/011%"&+#/01/2 12/"4,2)!"+,+$,&+$/"0,2/ "1%14,2)!0201&+

$"+"/1&,+0,#0, &)0 &"+ "/"0"/ %+!"3&!"+ "0"!-,)& 6 +)60"0





%"4&))+,1/&0"+6+"4,/ ,*-)& 1"!!1-/&3 6 ,+ "/+0&+-/1

" 20"+,-"/*+"+1)601,/"!))"+ ,*-00&+$!10"4&))" /"1"!+01"!!14&))")&+("!,+

!"*+!0!&/" 1)6+""!"!#,/--/,3"!/"0"/ %-/,'" 10%")&+("!!14&1%,21-"/0,+)&!"+1&#& 1&,+

4&))"*!"3&))",+)61, /"#2))63"11"!/"0"/ %"/0+!-/,'" 101%/,2$%"+02001%20

-/,3&!&+$6"1+,1%"/)6"/,#-/,1" 1&,+








	

		%"+)&+("!1,,1%"/02/3"6+!!*&+&01/1&3"

!11%"4&))" ,*"3)2)"0,2/ ",#),+$&12!&+)!1#,/0, &)-/,$/*"3)21&,+14&)))),4

/"0"/ %"/01,1/ (!2)1,21 ,*"0#,/&+!&3&!2)04%,"&1%"/-/"+1))6,/-,01+1))64"/""5-,0"!1,

#3,/)",/2+#3,/)""3"+10,/-,)& &"002 %0*&)&1/60"/3& ",/ /&*&+)'201& "-"+)1&"0%&04,2)!

$&3"1%" ,2+1/6+"4+!2+-/" "!"+1"! - &161,"3)21"1%"),+$/2+"##" 10,#3/&,20#,/*0,#

-/,$/*-/1& &-1&,+("6-,)& 6"5-"/&*"+10+!3/&,20#,/*0,#&+01&121&,+)-/1& &-1&,+)1%,2$%&190

4&!")6$/""!1%1-,)& 6!" &0&,+00%,2)!1("&+1, ,2+11%"),+$/2+"##" 10,#-/,-,0"!-/,$/*0+!

&+1"/3"+1&,+01%"%&$% ,01,##&")!&+$),+$1"/*-+")0%0,)&$"!201,*("-,)& 6!" &0&,+0)/$")6,+

1%"0&0,#&**"!&1"+!-,1"+1&))61/+0&1,/6"##" 10%"4,2)!-/,3&!"!1&+#/01/2 12/"1,

)),41%"1,!"3"),-+!"3)21"-,)& 6,+1%"0&0,##2))"/00"00*"+1,#"##" 10

The Promise of Evidence-Based Policymaking 1287
#%"'&'*"!'!('")"#.!0'(!.!&&&& '""",''*"*'7??6

!&(&! &'"&!!."#',%.! '*' "%%!'!&(&&/

#%"'&'*+#"%!! !'&'"+&'!%"%!! '!'!"",''

&(%%!',(&*'!'!&(&(%(!!%&'%('(%/'*&"+#"% #%")

 '"&"&' '!''"'&''&'%%"%&&"'&' '&%" !'&'&/

*&#%'!%&#&*'"%!-'"!&'"(!'"#%'"!#&"&!!!.%!.!

!!! &%" '786.666%&" %" '' (#'7??6!&(&/

What are the next steps? 

#%"'&#%'!%*''!&(&(%(!'!"%!)%&',!!!'"(&'"!'

7??6!!!&(&'"'8666!8676!&(&&.*)%,!!'%"(!&(&

(%('!"",/""*!'7??6!.'#%"'*"(&&$(!',"!!!'7?>6.

7?=6.7?<6.!7?;6!&(&&/


&!3867;07<4"'&+ !!'&',"'#%"&&.!)"#!' '""",

'""*'#%"''" ")!'"!"#%'"!&"!#&/

• ! &""(&"  %&*%!"''%"!,#'(%"!'7??6!&(&.('

'&! &%!"%#%&"!)!/&! &+&'& &"!786.666%&

" %" /
%"'&'& #9?8%&!%'<=;.;:?&!! &/'&"

%'
	,!,!<?"'& #%&/'%('"!'!&:<.:9?&'

! &!:<.99<%&'! &/

• 
%"'&'&!*"%!*'&("!'%'"%&"!"#'%'%%"!'"!34

 '""",'"'"%'&!!! &/*"%*'%! &&'"

#%"% *'!&(%!&(&',/&*"%&!#%"%&&.('%,%&('&%)%,

!"(%!/

• 
%"'&'&!'!'%!')##%"'" '!7??6!&(&%"%&''&&&

%!'"!%&('&"%!',!! &/'*%&%!!&&&&!'&

##%"")%'!+',%"'#%"'/

&#%'!%!*'!"!"!"%'''!&(&(%('")"#!*#'&"%

&'%',%(&! !&'%')'%" %.&''.!" %#%")%&/&!'')&

(	

3"%	"!'(!!%&'%('(%
"#('"!
%"'4/!&(&(%(&!'

8666!8676!&(&&/(%%!'"(&"	

&'"!)%'," !&'%')&.&(&'"&

*'&"#%"% #%'#'"!'.*''%" "!"!&(%),&!!&(&&/,)&"!

*"%!*'%&%%&'"!7?:6!&(&'.*&#(,).'"'8666!8676

!&(&&/"!'!(&'""(&"!!"'7?;607??6!&(&&.*%%'"%&'(&"

"!' #"%%,&" "',!"%)"#!#% !!'!%&'%('(%"% "!'"%!'%!!

 "',/7?;607??6!&(&&%,(&',#%")!" .('"!.!"(#'"!

!"% '"!"!#%!'&""!' #"%%,*"%%&/"%!"(''("*"%%&(%%!',!'"%

"%.*!'"%'"'7?;607??6!&(&&/

$" 	



What has been accomplished? 

1288 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 13, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t0k-2imr

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0100

Comment on FR Doc # 2016-22002

Submitter Information

Name: Stephanie Chamberlain

General Comment

Please respect the privacy of students and individuals. I am strongly opposed to the

federal collection of student data. This places children at risk.

The Promise of Evidence-Based Policymaking 1289
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 13, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t0k-e6ih

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0101

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anne Lucas

General Comment

It is horrifying to think that individual privacy could be sacrificed for social engineering,

commercial and/or political reasons. A federal unit record database is yet one more step

toward a dystopian society. This kind of intrusion, like political correctness, destroys the

very strengths of our country. I did not grow up in this kind of environment and i don't

want my grandchildren subjected to such dehumanizing scrutiny.

1290 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 13, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t0o-sur8

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0102

Comment on FR Doc # 2016-22002

Submitter Information

Name: Lora Hubbel

General Comment

Stop with the Data Collection...we are supposed to be "secure in our papers". There is

NO Evidence in your "Eveidence Based Policy making"...its is pure control of the

population. The more you know about a child the more you can "control" that child.

Leave the children alone so they can form their OWN destiny!. Stop taking away our

DESTINY!!!

The Promise of Evidence-Based Policymaking 1291
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 13, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t0p-cykx

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0103

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

No way is this a good thing.

1292 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 13, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t0p-t0qb

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0104

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

The federal government doesn't have a stellar reocord of successful public education. We

don't need their policy regulations, therefore we don't need this legislation to track

students from kindergarten through their work life.

The Promise of Evidence-Based Policymaking 1293
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 13, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t0q-aqxd

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0105

Comment on FR Doc # 2016-22002

Submitter Information

Name: Susan Stephens

Address:

Jasper, AL, 35504

Email: smhstephens@gmail.com

General Comment

As a grandmother of 6 who has kept abreast of education issues/policies/trends ever

since I completed my degree in this field of study, I am totally opposed to the federal

gathering of data and tracking our children (at any age from kindergarten through college

& beyond, etc.) under the guise of educational improvement/ better teaching practices/

targeted instruction & help for students/ or even facilitating our classroom teachers'

interactions with their student.

While the proposal is targeted for college (and career/employment) at this time, we all

know such data collection will extend to earlier and earlier years in the lives of our

youngsters.

Such gathering of data of a free people has no place in our nation, especially by our own

government, and in fact, is a violation of our Constitution. Space and time precludes

giving all the reasons such data mining & collection & usage is wrong on all levels. Data

gathered never remains private; in fact, keeping it private would defeat the supposed

purpose. Too often data ends up in the wrong hands for the wrong purposes. Third party

1294 The Promise of Evidence-Based Policymaking
sharing of data (and even the government's own record of that data) could impact a

person throughout their life.....and not always for their good.

The "right to privacy" was deemed a reason by the U.S Supreme Court as a justification

for legalizing abortion in 1973 Roe. vs. Wade. So why would that "right to privacy" be

thrown to the winds regarding our children and their future via massive data collection

via our public schools and testing??

My husband and I adamantly state our total opposition to any such proposal or

implementation of establishing a massive data dossier on any individuals in our country.

The Promise of Evidence-Based Policymaking 1295
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 13, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t0r-sfx4

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0106

Comment on FR Doc # 2016-22002

Submitter Information

Name: Marilyn Oakes

General Comment

Must we put every child in this nation into a "national" data base? What is the point and

who is pushing this idea? We need to keep this kind of reporting and record keeping in

the hands of the states, not the federal government. Thank you for listening.

1296 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 13, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t0r-p8z5

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0107

Comment on FR Doc # 2016-22002

Submitter Information

Name: Cathy New Yorkers United

General Comment

Say no to government tracking of student data and teacher data. Protect the privacy of

our children.

The Promise of Evidence-Based Policymaking 1297
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 13, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t0r-betv

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0108

Comment on FR Doc # 2016-22002

Submitter Information

Name: Loie TenBrink

General Comment

I oppose S 1145. Stop common core.

1298 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t0t-960a

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0109

Comment on FR Doc # 2016-22002

Submitter Information

Name: Erin Greene

Address:

KINGWOOD, TX, 77339

Email: erin.rustgreene@yahoo.com

General Comment

I oppose any type of national student database. Taking information on my child is not

necessary for education and is unnecessary to teach her the necessary skills required to

succeed in life. Stop the collection of data on our school children and return to teaching

our students to read, write, and do math well and live the American dream! Until the

government can show undeniable evidence that this is necessary, I am opposed to data

collection.

The Promise of Evidence-Based Policymaking 1299
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t0t-1csl

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0110

Comment on FR Doc # 2016-22002

Submitter Information

Name: Lou Teff

Address:

Tujunga, CA, 91042

Email: twslou@msn.com

General Comment

We need to come off the nazi-like dossier mentality that government knows all about

everyone. (The U.S. Consititution is meant to protect the people from the government,

not the other way around!)

1300 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t0t-rt9k

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0111

Comment on FR Doc # 2016-22002

Submitter Information

Name: Lindsey Mills

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all students, commonly

referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008, the reasons against creating it

have only become more compelling.

In the past few years, much highly personal data held by federal agencies has been

hacked, including the release of the records of the Office of Personnel Management

involving more than 22 million individuals, not only federal employees and contractors

but also their families and friends.

The US Department of Education has been found to have especially weak security

The Promise of Evidence-Based Policymaking 1301
standards in its collection and storage of student data, and received a grade of D for its

security protections.

In addition, preK-12 student data currently collected by state departments of education

that would potentially be shared with the federal government include upwards of 700

highly sensitive personal data elements, including students' immigrant status, disabilities,

disciplinary records, and homelessness data.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from PreK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban in

the report you provide to Congress.

1302 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t0u-iu99

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0112

Comment on FR Doc # 2016-22002

Submitter Information

Name: Joshua Mitchell

Address:

Washington, DC, 20037

Email: mitchelj@nber.org

General Comment

See attached file(s)

Attachments

CEP comments_v2

The Promise of Evidence-Based Policymaking 1303
Commission on Evidence-Based Policymaking Comments 

Docket ID USBC-2016-0003 

 

To the Commission on Evidence-Based Policymaking: 

 In support of the Commission’s mission to “develop a strategy for increasing the availability and 

use of data in order to build evidence about government programs, while protecting privacy and 

confidentiality,” I would like to submit the following comment in response to question #3: “Based on 

identified best practices and existing examples, how should existing government data infrastructure be 

modified to best facilitate use of and access to administrative and survey data?” Please note that the 

views expressed are entirely my own and do not necessarily reflect the views of my employer. 

Based on my extensive experience working with government survey data, government administrative 

data, and linked survey-administrative data, I have developed an understanding of where some of the 

major gaps in the Federal Statistical System are, and how they could be addressed. Under the current 

system, research analysts who wish to access administrative data to analyze government programs face 

a number of important obstacles. The set of data elements relevant for completing a successful project 

are rarely housed at a single department or agency. For example, an impact analysis that seeks to 

understand the effect of a community college program on future earnings would require both education 

data and labor market data, which are typically maintained by separate agencies and require data 

sharing agreements to link the two types of data together, and coordination between the legal staffs at 

the two agencies. Broader evaluations that wished to compare outcomes across states would 

potentially multiply these challenges by fifty (one for each state). Even after obtaining data, researchers 

face additional hurdles when comparing across states and programs which each have their own 

proprietary databases which may not be directly comparable to each other or easy to link together. The 

burdens on both researchers and state/local agencies can quickly become insurmountable. 

My comments focus on improving the measurement of and access to critical economic outcomes such 

as income, in-kind benefits, wealth, consumption, and savings with an emphasis of improving the quality 

and scope of existing federal tax data. I emphasize improving and expanding the use of federal tax data 

because this would create uniform data quality at one agency and would require fewer steps for 

researchers to obtain access. While some of my suggestions may at first sound like minutiae, I argue 

that these are in fact incredibly important issues for accurately assessing economic trends, performing 

comprehensive distributional analyses, and conducting program evaluation. I believe that even modest 

changes to existing tax forms would provide large improvements in data quality that are needed for 

statistical purposes, research, and program evaluation. In addition, an expansion of information return 

reporting would help close the gaps in existing data, which would substantially improve the scope of 

federal data and reduce the need for analysts to burden state and local government agencies. All of this 

1304 The Promise of Evidence-Based Policymaking
could be achieved while maintaining the strict standards for privacy and confidentiality that exist for tax 

data today. 

Below I list my suggestions for improving and expanding information return reporting to make tax data 

more useful for program evaluation. 

1. Earnings and Total Compensation (IRS Form W-2) 

Current IRS administrative records on earnings (W-2s) only contain earnings amounts that are subject to 

income and/or payroll taxes. Compensation that is not taxable typically does not appear on any tax 

form, frustrating attempts to accurately measure total compensation. The largest of such exclusions 

from the tax base include employer and employee premiums for health insurance. Elective deferrals to 

retirement plans are also not income taxable although they are payroll taxable. There are therefore two 

potential problems—a lack of any information on certain types of compensation, and an inability to 

decompose total compensation into its various subcomponents which policy analysts directly compare 

about. There has been some recent progress. The W-2 has recently included under Box 12 the 

employer premiums for health insurance, for example. However, employee premiums are not available, 

nor are other non-taxable benefits such as commuting expenses. As a result, a large share of 

compensation remains invisible in tax data. Yet this compensation is important to accurately measuring 

worker well-being, considering policy alternatives around how different elements of compensation is 

taxed, and creating proper aggregates for the national accounts. 

A "Box 0" could be added to the W-2 that has gross pay before any deductions are taken. This more 

closely aligns with desired economic concepts and survey questions on earnings and would allow 

program evaluation analysts to better measure full employee compensation which they cannot currently 

do with the W-2. In addition, individual components of gross compensation that are not already 

available on the W-2 could be added so that total compensation could be properly decomposed. For 

example, Box 12 could be augmented to include employer contributions (matches) to retirement 

accounts as there is no micro data currently available on this (only employee contributions are 

available). I would also recommend indicating whether an employee is vested under an employersponsored defined benefit plan. Currently, there is no information about defined-benefit plan 

participation in tax data. Although a decreasing share of workers have such plans, they are still 

important for public sector workers and for properly measuring economy-wide retirement plan 

participation. 

2. Wealth 

The Promise of Evidence-Based Policymaking 1305
Large amounts of financial wealth do not show up in any tax data (once exception is Form 5498 IRA 

balances). This is despite the importance of wealth as an indicator of well-being and as a measure of 

retirement preparation adequacy. Yet taxable income derived from stocks of wealth do show up on 

current information returns. A very small addition to forms 1099-DIV and 1099-INT would be to require 

beginning and end of year account balances on those forms in addition to the income elements already 

reported. This information is already maintained by administrators of those accounts so it would 

impose few additional burdens on the information return tax preparer. Not only would this information 

be incredibly useful for understanding the distribution of wealth and examining the impact of 

government programs on wealth accumulation, but it would also be helpful for corroborating the 

income information that is currently available on those forms. 

There also is no existing information on individual account balances from 401(K) and other employer 

defined contribution plans—the primary intended savings vehicle for workers and a subject of much 

policy discussion. An information return issued for each account with its recorded beginning and ending 

year account balance, contributions, and withdrawals, would be extraordinarily useful for understanding 

savings decisions and considering policy interventions surrounding these plans. 

3. Expansion of information return 1099s for research on program participation and poverty alleviation 

Currently IRS data do no include information on transfer income that is excluded from taxation. For 

example, Supplemental Security Income is unavailable in tax data yet is an important part of the safety 

net. As a result, analysts who want to examine trends or program impacts on poverty and who would 

like to incorporate realistic features of the tax code as well as non-taxable transfer programs are unable 

to do so effectively with current tax administrative data. One alternative is to use income and program 

participation data collected from government surveys. Yet, survey data are known to suffer from 

income and program participation underreporting. Further, one of the promises of improved 

administrative data is the ability to follow individuals in a consistent way over time without sample 

attrition that is common in panel surveys. Expanding tax data to include safety net programs would put 

all the necessary information analysts would need for program evaluation in one place thus alleviating 

burdens of contacting many federal, state, and local agencies, signing interagency agreements, and 

dealing with data inconsistencies across states and programs. 

One possible way to accomplish this would be for information returns to be filed by the appropriate 

departments/agencies for the following programs: 

1. Veterans pensions 

2. Supplemental Security Income 

3. Public housing and section 8 vouchers 

1306 The Promise of Evidence-Based Policymaking
4. TANF 

5. SNAP 

6. WIC 

7. LIHEAP 

8. CCDF 

etc. 

A simple rule would be that any transfer program receiving federal funding whether federal or 

state/local administered would have that agency complete a basic electronic form indicating who 

received the benefit, the months the benefit were received and the monthly amounts (when relevant) 

for that tax year. A copy of that form would be sent to IRS although those benefits would remain 

nontaxable. This would create uniform reporting of benefits and allow researchers to conduct 

nationwide program evaluations without signing separate data-sharing agreements with 50 states (and 

sometimes multiple agencies within a state). Because the reporting process could be automated, it 

would ultimately alleviate burdens on federal, state, and local agencies who must currently respond to 

individual researcher inquiries, draft legal agreements, and manage specific data transfer requests. 

4. Planning future changes to tax data 

Besides modifications to existing forms, it is important to think about maintaining the scope and quality 

of tax data available as policy changes occur. As the tax treatment of different types of income is 

subject to change, information on that income should remain available as it is essential for statistical 

and program evaluation purposes. It also allows a direct analysis of the behavioral response to the 

particular change in the tax code if additional income becomes excluded from taxation. whereas it 

would be impossible to do this exercise if data on that type of income simply disappears from tax data 

when the tax law changes 

. 

Other economic trends may require future data adjustments as well. The rise of independent 

contractors and other types of contingent workers may mean that a larger number of people who 

receive earnings will not show up in the most readily available tax data—the W-2. Proposals to create 

new types of labor arrangements such as “independent workers” need to include discussion on how 

those arrangements would be adequately reflected in tax data. 

The Promise of Evidence-Based Policymaking 1307
I suggest that a panel of external advisors/stakeholders be created to issue non-binding 

recommendations to IRS to design and modify future tax reporting requirements. The panel would 

consist of those from other federal agencies and other experts in microsimulation and program 

evaluation who would benefit from improved tax data. For example, the Bureau of Economic Analysis 

requires adequate measurement of income aggregates (including non-taxable income) to construct the 

national accounts and would have relevant input about what concepts would ideally be reflected in the 

design of future tax forms. Obviously, the priority of the IRS must remain on tax administration but there 

are often small adjustments to forms that would be equally useful to the IRS but would make the data 

significantly more useful to other federal agencies and researchers. 

I believe the above changes and reforms going forward would dramatically increase the utility and 

improve access to administrative data for statistical and program evaluation purposes while maintaining 

strong privacy protections. I am happy to discuss these issues further if desired. 

Thank you for your time and attention. 

Joshua W. Mitchell 

Senior Economist 

1308 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t0y-aul3

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0113

Comment on FR Doc # 2016-22002

Submitter Information

Name: Alan Tomko

General Comment

I believe the benefits of this effort are far, far outweighed by the potential drawbacks,

and I do not support any invasion of my privacy, nor my children's.

The Promise of Evidence-Based Policymaking 1309
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t0y-yk9x

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0114

Comment on FR Doc # 2016-22002

Submitter Information

Name: Lisa Jones

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all students, commonly

referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008, the reasons against creating it

have only become more compelling.

In the past few years, much highly personal data held by federal agencies has been

hacked, including the release of the records of the Office of Personnel Management

involving more than 22 million individuals, not only federal employees and contractors

but also their families and friends.

The US Department of Education has been found to have especially weak security

1310 The Promise of Evidence-Based Policymaking
standards in its collection and storage of student data, and received a grade of D for its

security protections.

In addition, preK-12 student data currently collected by state departments of education

that would potentially be shared with the federal government include upwards of 700

highly sensitive personal data elements, including students' immigrant status, disabilities,

disciplinary records, and homelessness data.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from PreK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban in

the report you provide to Congress.

Yours,

Lisa A. Jones

High School Math Teacher

Missouri

The Promise of Evidence-Based Policymaking 1311
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t11-caze

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0115

Comment on FR Doc # 2016-22002

Submitter Information

Name: Kimberly Elliott

Address:

Washington, DC, 20006

Email: kelliott@rwjf.org

General Comment

Comments from the Robert Wood Johnson Foundation to the Commission on EvidenceBased Policymaking. Please see attached file.

Attachments

RWJF_CEP_CommentLetter_11142016_FINAL

1312 The Promise of Evidence-Based Policymaking
Office of the President and CEO

November 14, 2016

Ms. Shelley Martinez

Executive Director

Commission on Evidence-Based Policymaking 

Docket Number: 160907825-6825-01

Commission on Evidence-Based Policymaking Comments

Dear Ms. Martinez:

The Robert Wood Johnson Foundation (RWJF) is pleased to have the opportunity to provide 

comment on the Commission on Evidence-Based Policymaking (CEP).

Building a Culture of Health 

RWJF is the nation’s largest philanthropy devoted solely to health. We believe that good health 

and health care are essential to the well-being and stability of our society, as well as to the 

vitality of families and communities. Evidence-based policies, programs, and laws are the key to 

spreading what works, making them a cornerstone of a Culture of Health, where everyone has a

fair opportunity to be as healthy as they can be.

We commend the sponsors of the Evidence-Based Policymaking Commission Act (P.L. 114-

140), U.S. House Speaker Paul Ryan (R-WI) and Senator Patty Murray (D-WA), for their efforts 

to ensure that evidence and measurement guide our decisions at every level of government. We 

support CEP’s mission to develop a strategy for increasing the availability and use of data in 

order to build evidence on government programs and policies, while protecting privacy and 

confidentiality. For nearly 45 years, RWJF’s research and evaluation efforts have provided

insight and a strong appreciation for the power of evaluation, data access, and data use in 

creating change. As such, we commend you in this new endeavor and we appreciate the 

opportunity to comment as the CEP works to understand how data, research, and evaluation are 

used to build evidence and how best to strengthen the government’s evidence-building efforts

going forward.

The Promise of Evidence-Based Policymaking 1313
RWJF’s comments will focus primarily around evidence and data systems. We would like to 

emphasize the following areas of our comment letter:

1) To promote the best evidence, we need to be open to expanding the universe of 

acceptable methodologies that measure impact and we must understand context. 

2) Achievements made thus far to improve data access underscore the need to ensure that 

these gains continue and that we use the power of technology to make even greater

advancements.

3) While data access and evidence-based decision making moves forward, we need to be 

mindful that data quality, access, and use varies greatly. There are great technical 

assistance needs among data systems at all levels of government and service delivery.

Evidence

RWJF’s Culture of Health Action Framework seeks to help transform a broad range of sectors 

and a variety of individuals all involved in building a Culture of Health into four interconnected 

action areas and one outcome area:

1. Making Health a Shared Value

2. Fostering Cross-Sector Collaboration to Improve Well-Being

3. Creating Healthier, More Equitable Communities

4. Strengthening Integration of Health Services and Systems

Taken together, our goal is for these action areas to lead to one outcome area: improved 

population health, well-being, and equity. The framework identifies the drivers of the change 

needed and a system of measures to monitor our progress toward our goal of improving 

population health, well-being, and equity.

Given our experience developing and vetting the Culture of Health Action Framework and our 

strong tradition of building and using evidence, RWJF offers two recommendations for building 

the evidence-base for policy making: 1) expand the methodologies; and 2) consider the context 

and capacity to build evidence at the time of policy and program implementation. 

Expand Methodologies: The word ‘evidence’ suggests rigorous, clinical analysis of an 

implemented program or policy intervention with a randomized control trial (RCT) 

methodology. RCT methodology is one way to attribute causality. However, well-known 

limitations of RCT can often undermine their use. Moreover, for many entities, the RCT process 

is both costly and time-consuming. Therefore, RWJF recommends expanding the universe of 

acceptable methodologies that are capable of attributing causality, for example: quasiexperimental data, natural experiments, and difference-and-difference models. In addition, at the 

time of implementation, programs should pay careful attention to implementation and consider 

models that could generate the random variation needed to evaluate, and build evidence for, a 

program in the future. 

For example, our national research program, Policies for Action, accepts various methodologies 

that benefit from natural or statistical variation to attribute causality. One recently funded study 

1314 The Promise of Evidence-Based Policymaking
will link data from a national food retailer with data from the U.S. Department of Agriculture’s 

the Supplemental Nutrition Assistance Program (SNAP) to understand the causal role of SNAP 

provision on the choice of food purchases and nutritional content.

In addition, we often leverage powerful work at the communities and states level. This work 

provides a natural laboratory of study for policies, programs, and practices. Study of state and 

community level policy change can identify the impact of policies that show promise. For 

example, through our tobacco policy research, we were able to promote the use of natural 

experiments to identify the role of tobacco taxes, smoke-free laws, and tobacco 21 laws in 

reducing teen and adult smoking. We continue to use these natural experiments methodology 

today to study laws governing e-cigarettes and smoke-free environments.

As another example, we partnered with others to fund a study of Oregon’s Medicaid System. 

The lottery system implementation design was essential to providing the random variation 

needed to study the causal impact of the Oregon Medicaid Experiment. This research 

demonstrated Medicaid’s impacts on outcomes of significant policy interest in United States. In 

addition to understanding the role of Medicaid in improving health related outcomes we were 

also able to identify Medicaid’s role in improving the financial and mental well-being of 

families. 

Consider the Context for Evidence Building: We believe context is extremely important to 

building evidence. Even the highest quality evidence may not be used effectively to inform 

decisions if it is deemed irrelevant, infeasible, or difficult to interpret. It is essential to expand the 

universe of acceptable evidence creating methodologies to include methodologies designed to 

rigorously understand the context for using evidence. This context includes the end-user’s 

values, needs, and capacity to access and analyze the evidence.

As we developed our Culture of Health Action Framework we engaged a diverse group of 

stakeholders. As a result, we have to change our thinking. If we are going to use evidence in the 

policymaking process, multiple stakeholders must value the goals and outcomes of the policies. 

For example, our efforts to promote the use of evidence-based policy to achieve our goal will be 

enhanced if keeping everyone as healthy as possible is a core value held by all stakeholders.

Also, the research and review process driving the Culture of Health Action Framework

illustrated that the demand for evidence-based programs and policies is greater than the supply. 

We have only scratched the surface on what it will take to build the evidence and we have a long 

journey ahead. We encourage the CEP to focus on investment needed to improve the research 

infrastructure. This research infrastructure is necessary to rigorously build evidence and should 

include an emphasis on collaboration and partnership. Partnerships between researchers, service 

providers, and large institutions–both governmental and corporate–can improve the capacity to 

develop evidence. 

Understanding the policy and program implementation, including the context and capacity for 

building evidence is essential to improve and replicate evidence-based policies. In addition, at 

the time of program implementation, data systems need to be developed and monitored to 

understand the needs, values, and capacity to assess, analyze, and innovate. For example, our 

The Promise of Evidence-Based Policymaking 1315
work on Complete Streets Policies documents how communities may work with their own urban 

planners, Departments of Transportation and engineers to address the community’s needs, 

develop policy changes, and create a data collection and monitoring plan to ensure the 

implemented changes in policy achieve outcomes that the community desired and valued.

Data Systems

RWJF’s Culture of Health Action Framework details a measurement strategy that is reliant on 

the data systems we have at our disposal. This includes data that we collected over the past 

nearly 45 years of work as well as data from outside sources. We followed five criteria to select 

these measures: availability of national data, application across time, representation of the broad 

determinants and upstream drivers, utility and understanding for multiple stakeholders, and 

ability to address of equity. This measurement selection process taught us many lessons. Below 

we will highlight the most salient lessons for your work.

Data Access and Data Linking: RWJF commends the efforts of individuals like Todd Park, who 

served as the Obama Administration’s Chief Technology Officer. Efforts of these individuals 

have leveraged current and developing technologies to create open data platforms. These data 

platforms should be a model for the future. For this model to maintain and ensure future success, 

we must continuously push technological growth. 

To create an open data platform, ideally the United States would have a centralized and open 

data warehouse that would be easily accessible by researchers. This data warehouse would have 

multiple data sets that could serve as a data backbone. These backbone data sets would be linked 

to other administrative data sets, surveys or other data sets. 

Data systems to serve as back bones of the data infrastructure exist. For example the Internal 

Revenue Service (IRS) and the American Community Survey (ACS) hold extremely rich 

longitudinal data. These data can be linked to other sources of data, across multiple federal and 

state agencies. In addition, we encourage the use of, and linkages to, data collected by mobile 

devices via GPS systems. The resulting data system would be powerful natural resource for the 

United States. Rigorous analysis of these data systems and the research produced could provide 

insight and, potentially, real time information on how to achieve improvements in the drivers of 

population health, well-being, and equity.

For a project with the National Academy of Medicine RWJF convened a panel of 17 experts to 

discuss the reasons why a national data system was needed to study human development over the 

life-course (such a study is important for the future of the United States and is not being 

considered or funded under current efforts). The resulting perspective papers describe how such 

a study could be designed in a cost effective manner by employing a diverse set of existing data 

sources, collected both administratively and passively from mobile devices, social media, and 

crowd sourcing platforms. These data can be linked through GPS and common identifiers. 

Without such a data system, the United States is at a comparative disadvantage relative to other 

countries that can follow children from birth to adulthood and ensure outcomes are achieved for 

these cohorts.

1316 The Promise of Evidence-Based Policymaking
As mentioned above, the 17 experts who authored the National Academy Perspective Papers 

encouraged maximizing the use of existing technologies. We encourage the Commission to 

consider using the full benefits of existing technologies not only in data collection, but also, to 

reduce the transaction costs of data use and data access. This will improve the effectiveness of 

evidence through improving the use of the optimal data for a given task. As researchers avoid 

transaction costs, they will use the most accessible data in lieu of the most appropriate data. It is 

essential to keep the transaction costs of the data systems as low as possible. As transaction costs 

fall, data access and linkages will be facilitated and the quality of the research, monitoring and 

effective policy making will improve. 

For example, through our Policies for Action research program, we have witnessed that 

governments using technology optimally create accessible data systems. These accessible 

systems closely monitor and act on challenges as they arise. Using data from both the State of 

New York and New York City, researchers at New York University are able to study how 

changes in the location of health services are impacted by access to transportation. They are able 

to link transit modalities with patterns of health care service use and are able to study the ways in 

which public housing rehabilitation projects impact health. Through the use of these newly 

linked and powerful data, researchers are making ground breaking change by developing a 

measure of housing-sensitive health conditions that will enable physicians to understand the 

impact that housing has on health and allows physicians to understand what symptoms may 

result from low housing quality. 

Investing in Data Quality: It has been our experience that public or private service organizations 

working within communities often produce data systems, or administrative data, and these data 

are important and essential tools that could be used to monitor improvements in health equity. 

From this experience, we caution the implicit assumption that all administrative data and data 

systems are useful to researchers. Data held by these organizations may not be robust enough for 

use in research and may require a tremendous amount of technical assistance before they can 

inform researchers. 

Many of our grantees working tirelessly to advance health equity are nonprofit service providers 

or on the ground community-based organizations. Focusing on the need for administrative data 

places these small nonprofits at odds with the evidence based programming model. There is a 

balance; these trusted partners are working at a scale and level of community engagement that 

allow us to monitor efforts with the appropriate level of analysis. Linking these disaggregated 

datasets with large administrative data sets create a very powerful tool for us to monitor: context,

needs, and values. 

We recommend providing technical assistance to nonprofit service providers to promote building 

usable data systems and integrating data systems to ensure we are meeting the data needs 

necessary to build evidence on programs that serve the most vulnerable populations. This 

assistance is necessary because the quality of data held within a system varies greatly across 

administrative data systems. As an example, we have invested in a public-private partnership that 

is fostering cross sector collaboration. This partnership aims to overcome the silos and create 

long-lasting systems change by integrating services across the child welfare agency, the local 

housing authority, housing and homeless service providers, and other partners in five 

The Promise of Evidence-Based Policymaking 1317
communities within the United States. This program is being evaluated using a RCT 

methodology. The evaluator’s initial plan for the RCT evaluation was to include the use of 

administrative data systems across the participating agencies. This would have reduced the cost 

of the RCT evaluation. Unfortunately, many of the administrative data systems from

participating agencies were not useful. The initial plan was implemented after additional 

resources were deployed to ensure the administrative data could be used during the evaluation 

and in the future. 

We encourage the CEP to consider ways of providing resources, to community-based programs 

and other small nonprofits to help them develop their data systems in a way that will allow for 

linkages to the larger data systems. For example, to facilitate building the data infrastructure of 

community-based programs, research funding opportunities at the federal level could: 

 Encourage collaborations (not just financial relationships) that can provide these community 

based organizations with ongoing technical assistance to build and maintain their data 

systems for use in research. 

 Incent linking community data with other data sets and systems.

 Consider what and where we should build evidence and, then, ensure the data systems in 

place create and maintain that evidence over time.

Conclusion

Our nearly 45 years of experience informed these comments. RWJF appreciates the opportunity 

to provide input and commends you in this new endeavor as the CEP works to understand how 

data, research, and evaluation are used to build evidence and how best to strengthen the 

government’s evidence-building efforts going forward. We stand ready to provide more details 

and information at any time. 

Sincerely,

Risa Lavizzo-Mourey

President and CEO

1318 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t12-5687

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0116

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

I'm opposed to tracking of students of any age in a national data base. The thought that

anyone is, additionally, considering linking such a database to anything accessible by

future employers or anyone else is APPALLING. Our country, the land of the free --

where privacy rights of the individual are expressly enumerated -- should not be

planning anything of the sort. As a citizen, I'm outraged that my own government -- with

the help of certain individuals and corporate interests -- keeps putting out propaganda to

deceive citizens into accepting this sort of invasive manuevering. As a parent, I'm doubly

infuriated. Please knock it off and stop telling me that bad is good!

Any information you may wish to collect should be PII free. Period.

The Promise of Evidence-Based Policymaking 1319
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t12-i0gy

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0117

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all students, commonly

referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008, the reasons against creating it

have only become more compelling.

In the past few years, much highly personal data held by federal agencies has been

hacked, including the release of the records of the Office of Personnel Management

involving more than 22 million individuals, not only federal employees and contractors

but also their families and friends.

The US Department of Education has been found to have especially weak security

1320 The Promise of Evidence-Based Policymaking
standards in its collection and storage of student data, and received a grade of D for its

security protections.

In addition, preK-12 student data currently collected by state departments of education

that would potentially be shared with the federal government include upwards of 700

highly sensitive personal data elements, including students' immigrant status, disabilities,

disciplinary records, and homelessness data.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from PreK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban in

the report you provide to Congress.

The Promise of Evidence-Based Policymaking 1321
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t13-9j25

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0118

Comment on FR Doc # 2016-22002

Submitter Information

Name: Catherine Imperatore

Address:

Alexandria, VA, 22314

Email: cimperatore@acteonline.org

General Comment

The Association for Career and Technical Education (ACTE) appreciates this

opportunity to provide feedback on the work of the Commission on Evidence-based

Policymaking through the Commission's recent solicitation for public input (Federal

Register Docket ID USBC-2016-0003). Please see the attached for more information.

Attachments

CEBP-ACTE-Response-Nov2016

1322 The Promise of Evidence-Based Policymaking
 

November 14, 2016 

Chairperson Katharine Abraham, University of Maryland 

Co-Chair Ron Haskins, Brookings Institution 

Commission on Evidence-Based Policymaking 

Docket ID USBC-2016-0003 

Dear Chairperson Abraham, Co-Chair Haskins and commissioners, 

Thank you for this opportunity to provide feedback on the work of the Commission on Evidencebased Policymaking through the Commission’s recent solicitation for public input (Federal 

Register Docket ID USBC-2016-0003). Through its examination of federal administrative and 

survey data, the Commission has the opportunity to facilitate an improved education and 

workforce data system that assists educators and industry partners in improving education and 

training and helps students and workers of all ages make informed decisions about their futures. 

The Association for Career and Technical Education is the nation’s largest not-for-profit 

association committed to the advancement of education that prepares youth and adults for 

successful careers. ACTE represents the community of CTE professionals, including educators, 

administrators, researchers, guidance counselors and others at all levels of education. ACTE is 

committed to excellence in providing advocacy, public awareness and access to resources, 

professional development and leadership opportunities. 

ACTE submits the following description of challenges and recommendations for action, which 

are most relevant to the Commission’s questions 3, 5, 6, 7 and 9. In addition to the following 

comments, I direct your attention to the responses submitted by the Workforce Data Quality 

Campaign and the Postsecondary Data Collaborative. ACTE endorses these responses, which 

address some of the topics raised here in greater depth. 

The Challenge 

Key challenges in the current education and workforce system are a lack of access to data on 

student attainment of non-degree credentials, including postsecondary certificates, licenses and 

industry-recognized certifications; and a lack of access to employment and earnings data. 

 

The Promise of Evidence-Based Policymaking 1323
Addressing these gaps will enable students and workers to make better-informed plans about 

education and career pathways; help educators and engaged business partners to improve 

programs; and support better evaluation and informed policymaking. 

Individuals are increasingly pursuing credentials beyond those awarded directly by 

postsecondary institutions, including certifications awarded by third-party industry providers as 

well as licenses awarded by state agencies. States such as Florida and Kansas are beginning to 

validate and incentivize student attainment of these awards. However, institutions and state 

education agencies currently have little access to data on industry-recognized certifications and 

licenses. There are also gaps in the collection of data on postsecondary certificates awarded by 

colleges. 

In addition, while required for performance measures in both the Carl D. Perkins Career and 

Technical Education Act and the Workforce Innovation and Opportunity Act, access to 

employment and earnings data is limited. Datasets are fragmented, with key information, such as 

occupation, missing in many cases. States have tried to follow students and workers across state 

lines, but participation in linked datasets such as the Wage Record Interchange System 

(WRIS/WRIS2) and the Federal Employment Data Exchange System is voluntary, and many 

states lack the capacity and resources to pursue this work. 

Recommendations 

1. Ensure that data requirements and definitions align across federal programs through common 

measures, as appropriate. 

2. Require data sharing within and among states and local areas to better track post-program 

outcomes, including further education and employment. 

3. Provide guidance on the Family Educational Rights and Privacy Act (FERPA) to clarify 

allowable data sharing with workforce and economic development agencies and third-party 

industry certification bodies for evaluation purposes. 

4. Include information in data systems on certificates, licenses and industry-recognized 

certifications earned by students. 

5. Ensure postsecondary data systems include all students, including returning and part-time 

students, and their short- and long-term outcomes. 

6. Expand the use of labor market information in program evaluation and ensure such 

information is accessible to students at the program level. 

7. Eliminate the ban on a federal student unit record data system to ease data collection and 

reporting burdens. Although the federal government is banned from retaining individual-level 

data, much of this information is already collected and held by a range of agencies and 

institutions. Bringing this data together in one system, with appropriate protections for privacy 

1324 The Promise of Evidence-Based Policymaking
and security, would reduce burden on states and institutions, provide students and workers with 

a more comprehensive base of knowledge on which to make education and career decisions, and 

help policymakers more effectively evaluate programs and create sound policy. 

ACTE appreciates the opportunity to comment and looks forward to the Commission’s 

recommendations to facilitate evidence-based policymaking through a more robust education 

and workforce data system. If you have any questions or need additional information, please 

contact Catherine Imperatore, ACTE’s research manager, at cimperatore@acteonline.org. 

Sincerely, 

LeAnn Wilson 

Executive Director 

ACTE 

The Promise of Evidence-Based Policymaking 1325
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t13-i58m

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0119

Comment on FR Doc # 2016-22002

Submitter Information

Name: James Mikolowsky

Address:

Washington, DC, 20005

Email: jmikolowsky@dataqualitycampaign.org

General Comment

Respectfully submitted on behalf of the Data Quality Campaign

Attachments

Data_Quality_Campaign CEP Letter_Final

1326 The Promise of Evidence-Based Policymaking
1 

Chairperson Katharine Abraham, University of Maryland 

Co-Chair Ron Haskins, Brookings Institution 

Commission on Evidence-Based Policymaking 

Docket ID USBC-2016-0003 

Dear Chairperson Abraham, Co-Chair Haskins, and Commissioners, 

The Data Quality Campaign (DQC) is pleased to respond to your request for comments as you work to 

develop recommendations that will help federal policymakers use evidence to inform policy 

decisionmaking while protecting privacy. 

The Commission on Evidence-Based Policymaking (Commission) provides a momentous opportunity for 

the federal government to create a culture of data use and trust. Your recommendations will lay the 

groundwork for a new era of data-driven federal policymaking that moves away from a focus on 

compliance and uses data to empower people. While the Commission has been charged with making 

recommendations to the federal government, you must also recognize the ways in which your work can 

provide value to states, districts, and citizens. 

Action by the federal government can provide value to state policymakers by serving as a template for 

state policy decisions, incentivizing certain state policy actions, or providing direct support to state 

policy efforts. Though states have led the way in developing education data systems and policies, the 

infusion of federal funding in the effort to develop State Longitudinal Data Systems (SLDSs) was a 

watershed moment which established a foundation for many subsequent state-level policy 

accomplishments. Moving forward, the federal government should better link existing federal data 

systems to provide meaningful answers to the most pressing education questions being asked by 

families and state policymakers. 

This document provides responses to a selection of the Commission’s questions where DQC thought 

that examples of state policy could inform the Commission’s deliberation. As always, it’s important for 

the Commission to remember that people won’t use data that they don’t trust and safeguarding privacy 

is paramount to building that trust. 

1. Are there successful frameworks, policies, practices, and methods to overcome challenges related 

to evidence-building from state, local, and/or international governments the Commission should 

consider when developing findings and recommendations regarding Federal evidence-based 

policymaking? If so, please describe. 

When developing its recommendations for federal policymakers, the Commission should consider DQC’s 

Four Policy Priorities to Make Data Work for Students framework. By using this framework to inform the 

development of its recommendations, the Commission can help federal policymakers create a culture 

that values and uses data for continuous improvement and that empowers families, educators, and 

education leaders with the high-quality information they need and deserve to make great decisions for 

students. 

MEASURE WHAT MATTERS: Be clear about what students must achieve and have the data to ensure that 

all students are on track to succeed. 

x Be clear what questions you want to answer across federal agencies and collect the data you 

need—and no more. 

The Promise of Evidence-Based Policymaking 1327
2 

x Support the development and use of purposeful data linkages across systems, sectors, and 

states. 

MAKE DATA USE POSSIBLE: Provide teachers and leaders the flexibility, training, and support they need 

to answer their questions and take action. 

x Lead by example and use data to inform decisionmaking and improve policy alignment within 

and across federal, state, and district levels. 

x Build capacity within state and local agencies to support the effective use of data. 

x Ensure that all educators have the skills and supports to use data in service of learning 

BE TRANSPARENT AND EARN TRUST: Ensure that every community understands how its schools and 

students are doing, why data is valuable, and how it is protected and used. 

x Provide useful information to the public about how the education sector is performing. 

x Communicate clearly about how federal data is safeguarded and used to support student 

learning and incentivize states and districts to do the same. 

GUARANTEE ACCESS AND PROTECT PRIVACY: Provide teachers and parents timely information on their 

students and make sure it is kept safe. 

x Ensure that those closest to students have role-based access to the student data they need to 

support individual learning. 

x Ensure that student privacy is safeguarded at every level as data is used in service of learning. 

Data Quality Campaign staff would be pleased to provide additional details or action steps on any of 

these nine higher-level recommendations. 

12. If a clearinghouse were created, what types of restrictions should be placed on the uses of data in 

the clearinghouse by “qualified researchers and institutions?” 

As the Commission seeks to determine how to best facilitate secure researcher access to data needed to 

answer critical policy and practice questions, it may be helpful to look at the data governance and access 

practices of leading states. Both Maryland and Washington have developed strong data governance 

structures that allow the state to make consistent and purposeful decisions about researcher data 

access and to securely link data systems in ways that support research. 

By allowing the “qualified researchers” at Johns Hopkins University School of Education’s Center for 

Technology in Education (CTE) to have limited access to its data systems, the Maryland State 

Department of Education (MSDE) was able to answer critical questions about the success of its early 

intervention services program. 

MSDE’s partnership with CTE was created to answer three specific questions: 

x What difference do early intervention services (EIS) make for children from birth to 3 years of 

age? 

x What are the effects of EIS on future student performance? 

x Which interventions work better than others? 

1328 The Promise of Evidence-Based Policymaking
3 

As MSDE and CTE began the work of answering their initial questions, they found that they were 

stymied by silos of data. MSDE had electronic data systems, but these systems did not talk to each 

other. And it was difficult to accurately find information about children in these data silos without a 

unique student identification system. With the support of a federal grant, MSDE and CTE created a 

longitudinal data system and a unique student identification system. By using data to answer their 

critical questions, the state learned that children with disabilities who received earlier, more rigorous 

services were more ready for school than children with disabilities who did not. By third grade, 68 

percent of early intervention students exited special education. 

Another example of a qualified institution using education data for the public good is Washington’s 

Education Research & Data Center – a state-legislated and state-funded effort to use the state’s early 

education, K–12, postsecondary, and workforce data to answer the state’s own questions about its 

students, schools, and pathways and to inform policy decisions. The Center is responsible for defining 

the purpose and appropriate uses of Washington’s state longitudinal data system and is accountable to 

the governor and legislature. The Center’s activities include protecting and maintaining data as well as 

determining data ownership and access. In addition, the Center is responsible for making sure the data 

system’s activities are transparent. 

14. What incentives may best facilitate interagency sharing of information to improve programmatic 

effectiveness and enhance data accuracy and comprehensiveness? 

The Commission should recognize the crucially important role that federal policy can play in helping 

state and federal agencies shift away from a compliance culture and facilitate appropriate sharing of 

data. While states’ direct investments in data infrastructure have been critically important to using 

education data in service of student learning, it is ultimately policy that affects change in ways that are 

rapid, lasting, and consequential. 

Throughout the (ongoing) process of creating an evidence-based culture in education, federal 

investments and policy incentives have been vital drivers of change. Many states had made some 

progress in building education data systems by 2009, but the infusion of federal dollars from the SLDS 

Grant Program was critical to securing state policymakers’ interest and helped them move their systems 

from an emerging tool to a robust source of information. States responded to this “seed funding” by 

increasing their own investments to ensure long-term sustainability. While just a handful of states were 

funding their systems in 2009 when the bulk of the federal grant funds were distributed, 41 states were 

funding their data systems by 2014. 

When it comes to federal policy incentives, the Commission should consider how federal programs have 

spurred the use of data to understand program effectiveness. For example, even though the Reading 

First program was not explicitly related to data, its requirement that states and districts use 

“scientifically-based” reading instruction incentivized the use of information to improve student 

outcomes. In addition, NCLB and ESSA have laid the framework for data collection and moved the 

education sector forward by requiring data be used for accountability. However, it was the Race to the 

Top competitive grant program that marked the first time that federal policy called specifically for using 

data for continuous improvement (e.g., delivering student growth data to teachers) rather than for 

building systems and using data for accountability (e.g., measuring annual yearly progress toward 

academic proficiency) and reporting (e.g., high school graduation rate). This innovation was critical 

The Promise of Evidence-Based Policymaking 1329
4 

because it provided an opportunity for state policymakers to think strategically about empowering 

stakeholders with data and shifted the conversation from building data systems to data use. 

Spurred by new data capacity and a focus on continuous improvement, states began to roll out 

innovative uses of data and policies to support them: 

x Colorado garnered widespread praise with its student growth model as state leaders introduced 

it to the nation with a data visualization tool thereby helping non-statisticians understand the 

value of a complex data model at a glance. 

x Arkansas provided one of the first direct student benefits from a SLDS by using its state data 

system to determine student eligibility for the Arkansas Challenge Scholarship. 

x Kentucky linked its K-12 and postsecondary data to provide better information about how their 

students fared in postsecondary institutions. 

16. How can data, statistics, results of research, and findings from evaluation, be best used to improve 

policies and programs? 

Again, the Commission can look to leading states for examples of how securely connected data systems 

can be used to inform education policies and programs. From using cross-sector data to better 

understand and support underserved populations to creating early warning systems that keep more 

students on track to graduate to creating data reports and dashboards that inform families, educators, 

and policymakers, states are making data work for students in innovative ways. The federal government 

can also incentivize this data-driven policy improvement work through the use of pilot programs. For 

example, in 2005 the Department of Education launched a pilot program for states to develop new 

models of measuring student growth. Because of the federal government’s leadership, many states now 

use the growth data based on these models as a factor in assessing the quality of educators and schools. 

The following state-level examples of data-driven policy improvement could inform the Commission’s 

recommendations: 

x In California, a study linking child welfare and education data uncovered an “invisible 

achievement gap” between children in foster care and other students. The research findings 

spurred widespread state attention and reforms targeted to this population. 

x Chicago Public Schools has used data to keep high school freshmen on track with great results 

(the on-track to graduation rate went from 57 percent in 2007 to 84 percent in 2013). The 

district could not have gotten these results without research to identify academic pathways and 

effective interventions. 

x Arkansas uses its state data system to inform students, parents, and educators about whether a 

student is on track to receive the Arkansas Challenge Scholarship. 

x Georgia combined local data with state-level resources and made using education data in 

meaningful ways easy. Now all Georgia teachers can easily view each of his or her students’ 

progress over time in different subjects and create personalized learning activities that build on 

strengths and fill gaps. 

x Maine provides funding for technology that allows parents to access their child’s longitudinal 

data through a locally managed portal. These data help parents understand whether their child 

1330 The Promise of Evidence-Based Policymaking
5 

is on track for success in college and career and how well their school compares to others in the 

area. 

x Hawaii reports on a number of college and career readiness indicators, such as postsecondary 

enrollment, credit accumulation, and persistence; this valuable information on where students 

go after graduation, how well they are prepared for college and beyond, and how states can 

better support students in the transition from high school to college can only be generated by 

connecting data from the state’s K-12 and higher education data systems. 

x Colorado’s SchoolView tool provides publicly available education data and analyses, including a 

growth model and related reports on academic growth across the state. The state and districts 

use this information to inform statewide policy decisions and classroom practices. 

18. How can or should program evaluation be incorporated into program designs? What specific 

examples demonstrate where evaluation has been successfully incorporated in program designs? 

The federal government has taken steps to prioritize program evaluation in education data policy, 

including through the SLDS grant program which identifies evaluation as one of six data use priorities 

that states should address. The Commission can look to the SLDS grant program language and to state 

program evaluation designs (such as Oregon below) for examples of how evaluation can be embedded 

into any program to meaningfully support continuous improvement. 

The Oregon Department of Education’s (ODE’s) Direct Access to Achievement (DATA) Project is an 

example of a program that successfully integrates evaluation mechanisms into its design. The DATA 

Project aims to improve student achievement by training teachers about strategies for accessing, 

analyzing, and using data to target instruction to the needs of individual students. 

Although the main participants are K–12 districts, valuable feedback comes from university 

representatives who meet with the project director quarterly to receive updates on project activities, 

and then identify issues and concerns related to teacher and administrator training. The project's 

professional development model consists of five phases: field input, in-service training, job-embedded 

professional development, pre-service training, and evaluation.

 After just two years, results indicated that teachers have made tremendous progress in adopting 

classroom-level data-driven decision making, connecting data to their teaching approaches. 

Data Quality Campaign staff welcome the opportunity to further discuss these recommendations, and 

we are optimistic about the opportunities for the Commission to help ensure that data works for 

students and helps meet our nation’s diverse education and workforce goals. 

The Promise of Evidence-Based Policymaking 1331
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t14-q8n4

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0120

Comment on FR Doc # 2016-22002

Submitter Information

Name: DONNA GARNER

General Comment

I am firmly against the efforts of the Commission on Evidence-Based Policymaking

(CEP) which is to create a longitudinal higher education/workforce database. This

database is presently banned by federal law and must remain so. 

It is paramount that student privacy and parental consent be preserved because children

belong to their parents - not to the federal government. 

A higher education/workforce database would undoubtedly be used to psychologically

profile students by using highly personal, identifiable, and sensitive socioemotional data.

Inevitably the data would be shared with third party entities and would also be

susceptible to hackers. 

This federal database would be used to drive students into vocations designed by the

federal government. Parents together with their children are the only ones who should

determine any future vocational plans for their families. 

To be perfectly honest, the federal government has done a deplorable job any time it

involves itself in education. The Common Core Standards Initiative is case in point. The

CCSI is being used to indoctrinate students into the social justice agenda rather than

strengthening students' academic achievement. Because the U. S. Department of

Education itself is unconstitutional, then any programs designed by it are

unconstitutional.

1332 The Promise of Evidence-Based Policymaking
Donna Garner

Wgarner1@hot.rr.com

The Promise of Evidence-Based Policymaking 1333
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t14-iwfq

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0121

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anne Gassel

Address:

Ellisville, MO, 63011

Email: agassel@att.net

General Comment

The October 21, 2016 hearing of Commission for Evidence-Based Policymaking

produced many witnesses who spoke in favor of data collection and the benefit they

would receive from it. There was only one witness who expressed deep concern about

the potential invasion of privacy and illegality of such efforts. It is clear from the

Supplemental Information, that the CEP will be looking for ways to remove existing

prohibitions, like the PPRA, which specifically states that the federal government shall

not collect information from student and parent surveys, to facilitate the collection of

data. Rest assured that the witness, Emmet McGroarty, was speaking for thousands of

parents across this country like myself who have not warmed to the idea of lifting the

restrictions currently in place which keep the government from collecting personal

information on our children.

While it is beginning to feel like a tired phrase, it still bears repeating with regards to the

work of the CEP. The 10th Amendment of the U.S. Constitution makes clear that any

powers not specifically enumerated therein remain with the states. Education is not one

of the constitutionally enumerated federal powers and therefore any educational program

1334 The Promise of Evidence-Based Policymaking
or policy generated at the federal level is unconstitutional, regardless of whether there is

a little or a massive amount of data to support such policies. Efforts by the federal

government to collect data to support federal education policy are therefore

unconstitutional and unnecessary.

In addition, the creation of a federal unit record of individual students violates the 4th

amendment of the Constitution which guarantees the right of due process. A person must

be charged with a crime and a warrant from the court issued before the wide scale

collection of personal data about that person can be begun. Any system that collects

individual student data without the informed consent of the adult responsible for that

child, and a process for review and correction of such collected data violates the

Constitutionally guaranteed right to privacy. 

As to the work of the CEP itself, it is focused on figuring out "how to increase the

availability and use of government data in support of evidence-building activities related

to government programs and policies, while protecting the privacy and confidentiality of

such data." This amounts to two diametrically opposed goals; the broad sharing of

information and the protection of such information. While I am very much opposed to

the collection of student data at the federal level (or through identically created

individual state longitudinal data systems) I recognize that the Commission will be

responding to their mandate to find ways to collect such data and therefore strongly urge

the Commission to err on the side of the second goal, protecting the data and, when

making its recommendations, resist the temptation to collect some of the very personal

data proposed by the Data Quality Campaign which, examples of recent data breaches

have shown, cannot realistically be protected.

As members of the ASA you know that 100% sampling is not necessary in order to

produce statistically relevant results. Therefore an individual student record for every

single child is unnecessary. Further, you are well aware that attempting to draw causative

conclusions regarding data not collected in a controlled experiment is at best inaccurate

and borders on unethical. George Borjas a respected labor economist of the Harvard

Kennedy School, in his recent book "We Wanted Workers," which looked at national

immigration policy from a statistical perspective, wrote, "Social scientists in general, and

economists in particular have done a very good job of convincing many people that the

mathematical models we build and the empirical findings we generate can be the

foundation for a 'scientific' determination of social policy... Nevertheless, I happen to

believe that the idea that mathematical modeling and data analysis can somehow lead to

a scientific determination of social policy is sheer nonsense. Social policy could not be

scientifically determined even if there were universal agreement on the underlying facts.

Ideology and values matter as well." (p. 198) To give Congress policy recommendations

with the apparent backing of scientific statistical analysis of data gives them a false sense

of conviction that they are doing the right thing with the policy.

The only proper use of educational data by the federal government would be in the form

of a research clearinghouse of state policies with the associated aggregate data to

demonstrate such policies' effectiveness in producing the desired outcome. Any large

The Promise of Evidence-Based Policymaking 1335
scale collection of raw individual data is nothing more than a fishing expedition to find

data to support a predetermined policy. As a parent, I strongly object to this practice and

urge the Commission to reject a federal unit record system. 

1336 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t14-2025

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0122

Comment on FR Doc # 2016-22002

Submitter Information

Name: Gordon Berlin

Address:

New York, NY, 10016

General Comment

See attached document.

Attachments

CEP_comments_MDRC_111416

The Promise of Evidence-Based Policymaking 1337
Comments to the Commission on Evidence-Based Policymaking

from Gordon Berlin, President of MDRC

Docket ID USBC-2016-0003

November 14, 2016

Thank you for the opportunity to offer some thoughts on the opportunities and challenges 

facing effective evidence-building for you to consider in your important work with the 

Commission on Evidence-Based Policymaking. 

My name is Gordon Berlin, and I am President of MDRC, a nonprofit, nonpartisan education 

and social policy research organization that is dedicated to learning what works to improve 

policies and programs that affect the poor. Founded in 1974, MDRC evaluates existing 

programs and develops new solutions to some of the nation’s most pressing social problems, 

using rigorous random assignment research designs or near equivalents to assess their 

impact. 

The federal government spends billions of dollars on policies and programs designed to 

improve the human condition; ameliorate poverty; increase employment, earnings, and 

income; invest in education to build human capital; and ensure America’s competitive 

position in a technologically advancing world. But to make a real difference, to ensure a 

return on investment for both taxpayers and the beneficiaries of these programs, we have to 

do things that actually work. 

Over the last decade and a half, during a period defined in the public consciousness by 

political partisanship, the legislative and executive branches have quietly forged a bipartisan 

consensus around the need to build evidence of effectiveness that would ensure high rates of 

return on investment for the nation’s social programs. The establishment by Congress of the 

new Commission on Evidence-Based Policymaking is only the most recent example of this 

consensus. 

My comments focus on the following issues: putting rigorous evidence at the center of 

policymaking, improving access to administrative data (while acknowledging its limitations), 

protecting confidentiality, bolstering the federal research agencies, addressing process and 

procurement issues, and maintaining the independence of third-party evaluators.

Putting Rigorous Evidence at the Center of Policymaking

The Commission has a great opportunity to offer recommendations to solidify the gains made 

in promoting evidence-based policymaking over the last two decades, particularly in 

bolstering the evaluation functions of the federal government and in making the use of 

rigorous evidence in policymaking more prevalent. 

1338 The Promise of Evidence-Based Policymaking
2

At the broadest level, I think the Commission should use its mandate to recommend that the 

federal government:

Validate the role of independent evaluation of programs and policies in the federal 

government: Evaluation findings that are credible, relevant, accurate, and timely are critical 

for policymakers and practitioners to make informed decisions about how to spend the 

resources of government. This is an issue of some urgency in a time of severe budget 

constraints and fiscal austerity. As is true for the federal statistical agencies, certain principles 

should underpin federally supported evaluation: relevance to policy issues, credibility with 

subjects and consumers of evaluations, and independence from political and other undue 

external influence. By upholding these principles, evaluations are well-positioned to provide 

the information that policymakers and the public require. 

Create a culture of continuous improvement: Rather than being focused on up-or-down 

judgments about programs or policies, government must develop incentives for using 

research evidence to make programs more effective over time — just like a business 

committed to becoming a dynamic learning organization. A good example is the Maternal, 

Infant, and Early Childhood Home Visiting (MIECHV) program, the largest evidence-based 

program of the era, which provided $1.5 billion in funding for home visiting programs over 

five years. There are several elements of MIECHV worth emphasizing: 

• Prior evidence was used to influence how federal funds could be spent, making it 

more likely that the funds would make a difference for families.

• The legislation recognized that there were areas where home visiting was not as 

effective as desired and offered states funds to test innovative approaches. 

• Funds were set aside for research to make sure that learning continued under 

MIECHV and could influence future realizations of home visiting.

Build on the tiered evidence strategies embodied in the Investing in Innovation Fund 

($650 million) at the U.S. Department of Education, the Workforce Innovation Fund ($125 

million) at the U.S. Department of Labor, the Teen Pregnancy Prevention Program ($110 

million) at the U.S. Department of Health and Human Services (DHHS), and the Social 

Innovation Fund ($50 million) at the Corporation for National and Community Service. 

These funds set clear guidelines about standards of evidence and provided incentives for both 

innovative new programs and, perhaps more importantly, for testing the scaling of models 

with evidence of effectiveness — truly the next frontier in the evidence-based policymaking 

agenda.

Embed evidence within existing funding streams: As the MIECHV example illustrates, 

when we have evidence of what works, we should build incentives into current funding 

streams to make sure that dollars follow the evidence. And while the innovation funds have 

been a source of new ideas from the field, incorporating resources within major program 

funding streams would allow federal agencies to develop evaluation agendas that would 

focus on continuous improvement of existing programs. A one-percent set-aside within these 

funding streams would both be an adequate investment and be cost-neutral.

The Promise of Evidence-Based Policymaking 1339
3

Improving Access to Administrative Data (But Not Relying on Admin Data Exclusively)

In evaluating the effectiveness of social programs, researchers need ready access to 

administrative data, and the Commission’s focus to date on improving federal, state, and 

local data systems is laudable. Research firms that are funded by federal agencies to evaluate 

programs often rely on data collected by states from employers on employment and earnings, 

including, for example, data that the states already report to the federal government for 

certain child support enforcement and other purposes. These data are housed in accessible 

form at the federal level within the National Directory of New Hires (NDNH) database. 

However, research contractors face severe restrictions in accessing this essential database for 

assessing whether federally supported programs actually work. Instead, they are forced to get 

the very same data directly from the states, at great cost to the federal government and at 

considerable burden in duplicative reporting for the states. If the NDNH database were made 

more widely available to evaluators (with appropriate privacy safeguards), it would enable 

Congress and the federal agencies to assess the impact that social programs have on jobs and 

earnings at much less cost and burden to the federal government and the states.

Similar opportunities exist for building robust data systems from the wealth of data about 

individuals’ experiences with health care, public assistance, criminal justice, child welfare, 

school, and college systems. Integrated data systems would save time and money and reduce 

paperwork burdens in the conduct of evaluations, while providing comprehensive 

information about program effectiveness over time. The public sector has the data to build 

comparable integrated systems to track progress longitudinally and to enhance program 

performance, yet federal and states agencies (and their contractors) cannot regularly access 

and share data for evaluation purposes. We can do better. Agencies and their contractors need 

clear authority to access and link administrative data for evaluation purposes when it is 

housed at the federal level and similarly clear authority when it is housed at the state level. 

Government efficiency hangs in the balance.

At the same time, access to administrative data is not a panacea. And the importance of 

survey data should not be dismissed. There are at least four factors that affect the fit of 

administrative records for a research study.

• Access: What kinds of sign-offs, permissions, and review processes/approvals are 

necessary to get access? 

• Standardization: Records that have already been compiled (e.g., in a state or 

national system) have common types and formats of information across lots of 

individuals, sites, etc. There is a single source, or a limited number of sources, to go 

to in order to collect data. However, the more decentralized the records, the more 

challenging the work becomes. For example, a project that spans multiple school 

districts may require transcript data, discipline data, attendance data, and enrollment 

data, which all vary across districts and which all need to be standardized by the 

researcher. This can be particularly laborious. 

• Completeness: The two biggest ways that completeness can affect evaluations are in 

terms of (1) how many individuals, schools, etc., are represented (i.e., how many 

respondents/units have missing data, and is that “missingness” associated with 

particular respondent characteristics?) and (2) whether the variables necessary to 

1340 The Promise of Evidence-Based Policymaking
4

answer relevant questions are all represented in the database. Frankly, administrative 

records often do not include the data required to answer important questions. For 

example, in MDRC’s welfare-to-work studies, the original 13 experiments were 

based entirely on administrative records with four key measures: employment, 

earnings, welfare receipt, and welfare amount. Did people leave welfare, did they do 

so for work, did they lose their jobs and return to welfare, did the program make 

participants better or worse off in terms of income, and did the program save 

government budgets money or cost more? What we could not learn is about effects 

on children. But as the project’s early results found increases in employment and 

earnings and reductions in welfare that exceeded the cost of running the programs, 

many states moved to reduce the age of child exemption. Women with children under 

age 6, then age 3, then age 1 would be required to work in return for welfare. Now

the central question became: would very young children be harmed if parents were 

required to find work? We needed a survey to answer this. 

• Data Lags: Federal data sets like the National Directory of New Hires or tax records 

can have very long lags before the data are assembled, cleaned, and available for use

in a study. This has significant implications for timeliness and limits the use of 

administrative data in quick-turnaround studies with multiple follow-up periods. The 

process to access National Directory of New Hires data, for example, is incredibly 

onerous and usually takes between one and two years, and only two years of 

historical data are retained in the database (although a researcher can request that data 

be retained for research purposes). This situation may improve as technology 

becomes more ubiquitous and efficient, but, given the pressures for quicker, cheaper 

evaluations, it still poses a major problem.

Protecting Confidentiality 

At all levels and branches of government, there is a tug of war between those who are 

focused on improving program effectiveness and those who are concerned with protecting 

privacy. Staff responsible for managing data are rightly charged with keeping it secure and 

protecting privacy but too seldom with developing protocols for sharing it securely with 

other agencies and evaluators. Although the stakes are high and the opportunities significant, 

the program office that houses the data often has little or no interaction with the same 

agency’s evaluation office. If these two objectives — measuring program effectiveness and

safeguarding privacy — remain mutually exclusive, continued paralysis is the inevitable 

result. 

The Family Education Rights and Privacy Act (FERPA) illustrates the challenge. Congress is 

considering amending FERPA because of concerns over threats to the privacy of student data, 

and meanwhile state legislatures have stepped in. Last year 47 state legislatures introduced 

more than 180 bills to address student data protection issues, a reaction originally prompted 

by public outcry over educational technology vendors and their use of children’s information 

for advertising and commercial gain. Unfortunately, education researchers from academia 

and other nonprofit institutions have gotten swept up in the furor. Under current federal law, 

education agencies can share data with researchers only for research projects designed to 

benefit students and improve instruction — and only under extremely strict privacy 

conditions. But some are suggesting that Congress should significantly scale back even that 

authority. Indeed, many states are interpreting FERPA to preclude the sharing of any 

The Promise of Evidence-Based Policymaking 1341
5

individually identifiable data with researchers, even though that data would only be reported 

in aggregate form for policy purposes. Without access to student data, little education 

research could be conducted at all. The bottom line is that it’s essential to continue to protect 

the security and privacy of student data, but we must be careful to not unintentionally end the 

analysis of student data for its original purpose: assessing and improving education. 

I am confident that privacy and confidentiality can be protected while still allowing access 

for research. Congress could start safely by specifying required levels of encryption and 

protection using the highest standards established by the National Institute of Standards and 

Technology (NIST). The NIST standards are appropriate for research data that must be kept 

confidential to protect the privacy and well-being of study participants and the integrity of 

the findings, but for which disclosure of confidential data would not jeopardize national 

security. 

Bolstering the Federal Research Agencies

Inconsistencies in federal authority to conduct independent research and evaluation pose 

additional hurdles for efficient evidence-building. To guide policy, research must be 

independent, objective, and reliable. However, the authorizing legislation establishing agency 

research departments does not always set forth these requirements — for example, the law 

governing the Institute of Education Sciences does, while the one over the Office of 

Planning, Research and Evaluation at the U.S. Department of Health and Human Services

does not. When authority is clear, agencies and their contractors have less difficulty accessing 

data, recruiting sites, establishing data-sharing agreements, and getting local buy-in. For 

example, in our experience studying home visiting programs, we found states and localities 

willing and ready partners in a random assignment research design when legislation 

instructed the federal agency to make program funding contingent upon participation in the 

evaluation. In a complementary home visiting study that was not explicitly described by 

Congress, however, site recruitment proved difficult. The authorizing legislation establishing 

agency research departments should allow them the federal authority to conduct independent 

research and evaluation.

Further, the research arms of federal agencies should be charged with building bodies of 

evidence about what works to address broad and persistent problems. Agencies should create 

a portfolio of research about a problem’s underlying causes and testing a range of possible 

solutions, always answering three questions: What difference did the program make, how did 

it do so, and why? To support that work, Congress could authorize federal agencies to set 

aside at least one percent of existing program funds for evaluation, a solution that is budget

neutral. And because context matters, agencies should also be encouraged to pay attention to 

the systems in which programs operate — something federal agencies too seldom do. It is not 

enough to learn what works; introducing what works into broader systems and maintaining 

quality at scale are the next frontiers in evidence-based policymaking. 

Addressing Process and Procurement Issues and Maintaining the Independence of 

Third-Party Evaluators

Finally, a few thoughts from the particular perspective of a federal research contractor. 

Procurement and process obstacles to cost-effective evaluations should be addressed. 

1342 The Promise of Evidence-Based Policymaking
6

The Paperwork Reduction Act is overly burdensome. While its goals are laudable, the 

Paperwork Reduction Act’s requirements for clearance by the Office of Management and 

Budget (OMB) and for filing two public notices for every survey involving more than 10 

people add time and money to fielding studies — as much as eight months or more between 

internal agency review and OMB clearance. Should the exact same rules that apply to 

nationwide rules and regulations that affect all citizens also apply to research and evaluations 

that affect a few hundred or even a few thousand people who volunteer to be part of the 

studies? This doesn’t seem to make sense at the same time that agencies are under 

tremendous pressure to speed up the evaluation process and enhance the timeliness and 

relevance of the work.

The independence of researchers must be maintained. Most federally funded evaluations are 

conducted by third parties, including nonprofit and for-profit firms and academics, an 

arrangement that is supposed to guarantee the independence and integrity of the research 

findings and conclusions and protect them from political agendas. However, this 

independence — and the concomitant right of the researchers to publish (whether the 

government agency decides to publish or not) — is not always made clear in statutes, 

regulations, or contracts.

Federal funding contracts should strike an appropriate balance between the need for 

accountability on the part of government and the need for independence on the part of the 

evaluator. Funding contracts for social program evaluation include one of two data rights 

provisions: general and special. The special provision is not appropriate for research and 

evaluation contracts designed to build policy-relevant evidence; the general provision is

appropriate for social program evaluations. The special rights clause is intended for 

production of data for internal use of government; the general clause provides a balanced 

distribution of rights between the government and the contractor and allows for wider 

distribution of the results of the research. The special rights clause was originally designed 

for activities in which the government has full ownership and control, as in a defense 

department procurement; the general provision was designed to govern products and 

knowledge that have broad applications and are likely to benefit and inform a broad range of 

stakeholders.

The general rights clause still protects government’s interest in quality and accountability. To 

that end, it contains “alternate provisions” that are mandated in contracts for basic and 

applied research with universities or colleges and are permitted in other contracts upon 

agency determination that the alternate provision is appropriate. Alternate IV directs agencies 

to loosen the restrictions that apply to contractors under the general rights in data clause 

when those contractors are colleges or universities performing applied or basic research or 

when the agency determines that similar treatment is warranted for other contractors. 

Agencies can also add other clauses in Section H of the contract to ensure contractor 

accountability. We think that agencies should use the general rights clause, Alternate IV, for 

program evaluation by research contractors and use Section H for oversight and use of data.

The special rights clause leaves release solely in the hands of the government, a power that 

can and has led to work of a broad public interest never seeing the light of day. 

Dissemination is key. Good evaluation research is only valuable if decisionmakers know and 

understand it. Most agencies adequately fund the writing of research reports, but they rarely 

provide resources to disseminate research findings broadly and effectively, including creating 

The Promise of Evidence-Based Policymaking 1343
7

products that are accessible to wide policymaker and practitioner audiences and investing in 

dissemination techniques that leverage social media and other communications tools. 

Grantees have a natural incentive to disseminate. The special rights clause can eliminate that 

incentive.

Procurement rules can be counterproductive. Rules designed to improve government 

procurement of goods and services can create havoc in evaluation research — for instance, 

an 18-month follow-up survey may span fiscal years and the pace at which the survey will be 

fielded is difficult to predict, but the funding of that survey can’t cross fiscal years. Research, 

demonstration, and evaluation projects are not readily severable. Given the need for long 

follow-up periods, studies may spread over five or more years. Under current procurement 

rules, multiyear federal evaluation contracts must be broken down into short renewable 

phases called “contract line items,” or “CLINs,” wherein contract requirements and dollar 

amounts must be specified separately. This has dramatically increased costs to administer and 

manage research, especially where there is a mismatch between activities required for a longterm research project and funding available under a contract line item. Under these 

restrictions, researchers have trouble, for example, entering into meaningful partnerships 

with program sites and participants when funding for an entire research effort is not 

guaranteed.

The Commission has the opportunity to secure the role of evidence in the making of social 

policy. But to do so it will need to tackle not just questions of principle, purpose, and policy 

but also the nuts and bolts of accessing, protecting, analyzing, disseminating, and using 

evidence to advance the public good. The effective functioning of this process will determine

whether data does in fact become evidence. 

Sincerely yours,

Gordon L. Berlin

President

MDRC

16 E. 34th Street, 19th Floor

New York, NY 10016

gordon.berlin@mdrc.org

(212) 532-3200

1344 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t15-ey66

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0123

Comment on FR Doc # 2016-22002

Submitter Information

Name: Talib Madyun

General Comment

Y.O.U.R. Community Center welcomes the opportunity to provide comments to the

Commission on Evidence-Based Policymaking (CEP). We offer input for questions 16

and 18 based on our experience as a grantee of a high-quality evidence-based program,

the Office of Adolescent Health's Teen Pregnancy Prevention (TPP) Program. This

program has been recognized as a pioneering example of tiered evidence-based

policymaking, and represents an important contribution to building a body of evidence of

what works. This includes high quality implementation, evaluation, innovation, and

learning from results. 

Y.O.U.R. Community Center was incorporated in 1997 and is recognized as a 501(c)(3)

tax-exempt organization. The mission of Y.O.U.R. Community Center is to promote

academic, cultural, social and emotional development; improve computer literacy; help

build self-esteem; and strengthen leadership and confidence in and out of the classroom.

Y.O.U.R. has served over 3,000 youth throughout Washington, DC. Our programs and

services have provided positive youth development and made a meaningful and positive

difference in the lives of youth through: education, emotional support, and social

development. While participating in Y.O.U.R. programs, none of the youth have been

lost to violence, been incarcerated, or become teenage parents. On satisfaction surveys,

90% parents and youth report a change in attitude after completing our program.

The Promise of Evidence-Based Policymaking 1345
As a TPP Tier 1 sub-grantee we incorporate evidence into all aspects of our project. The

program model we implemented was chosen from a list of those that had already been

rigorously evaluated and demonstrated to change behavior. 

We also used evidence to guide and improve our project throughout the grant period. 

1. Over the past five years as a sub-grantee, our program has completed an evaluation

using our pre/post test results to ensure that our practices and strategies are effective and

if not effective, we have made adjustments in the curriculum. We have used a few

different curriculums to try to work with the population of youth that we do. We have

used Why Am I Tempted (WAIT), Game Plan and Hip Hop 2 Prevention (H2P). All

have been successful, as we had positive results with the students we have worked with

youth in the District of Columbia Public and Public Charter Schools. 

2. This past year's results of pre and post survey revealed that there has been an increase

in the number of youth who had engaged in sexual activity before going through our

program, who have chosen to stop having sex after the program. This year's results

showed that 75% of the youth reported that they would stop after the program concluded.

This is up from 42% two years ago. We attribute these results to several things including

adjusting and incorporating strategies used to reach our youth where they are and using

facilitators who are closer to their age and look like them. Without compromising the

integrity of the curricula that we use and ensuring that all objectives are met, we also

made additions and adjustments and added also supplemental materials based on the age

and demographics of the varying groups. We have learned that one size does not fit all.

3. In closing, we consider The TPP Program to be a prime example of high-quality

evidence-based policymaking. It is one of the few government programs that uses

evidence and evaluation criteria throughout the grant life cycle. Thank you for

considering our input for the CEP. If you have any questions or need additional

information, please contact me at (202) 291-3034 or talib@yourcommunitycenter.org.

Attachments

TPP Acomplishments

1346 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 1347
1348 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t16-3q70

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0124

Comment on FR Doc # 2016-22002

Submitter Information

Name: Lisa Williford

Address:

Huntsville, TX, 77340

Email: lisawilliford@sbcglobal.net

General Comment

Please do not remove the human component of deciding a child's future by adopting a

national data base collecting and holding student data. While the idea of identifying a

child's strengths and weaknesses and using that information to help them make decisions

about their future is a good idea, the United Nation's idea of data collection and

globalizing our nation's children, making them human capital, is unamerican and

Hitleresque . As a teacher in the public school system in Texas, I have witnessed

education go from it's original intent to a huge money making enterprise. The budgets

are huge, and the accountability for spending that money is nonexistent. Research is

beginning to speak loudly regarding the effects of screen time on the developing brain.

Recent scientific research in the area of brain development and reading is being ignored.

The present movement in education is another human experiment with no true researchbased methods involved. Children are suffering and their futures are being co-opted for

cold hard cash. Just because technology presents almost unlimited possibilities doesn't

mean we should just keep pushing it's limits to see what will happen. Give the term

"research-based" a true meaning, and hold someone accountable for it.

The Promise of Evidence-Based Policymaking 1349
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t16-jdmf

Comments Due: December 14, 2016

Submission Type: API

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0125

Comment on FR Doc # 2016-22002

Submitter Information

Name: Steve Voytek

Address:

Silver Spring, MD, 20910

Email: svoytek@careertech.org

General Comment

See attached file(s)

Attachments

CEP Comment

1350 The Promise of Evidence-Based Policymaking
Advance CTE: State Leaders Connecting Learning to Work 

8484 Georgia Avenue Suite 320 

Silver Spring, MD 20910 

www.careertech.org 

November 14, 2016 

Chairperson Katharine Abraham, University of Maryland 

Co-Chair Ron Haskins, Brookings Institution 

Commission on Evidence-Based Policymaking 

Docket ID USBC-2016-0003 

Dear Chairperson Abraham, Co-Chair Haskins, and commissioners, 

On behalf of Advance CTE, the national association representing the state and territory leaders of our 

nation’s Career Technical Education (CTE) system, I write in response to the Commission on EvidenceBased Policymaking’s (CEP) recent solicitation for public input regarding increasing the availability and 

use of government data, while protecting privacy and confidentiality (Federal Register Docket ID USBC2016-0003). Our members oversee secondary, postsecondary, and adult CTE across the country and 

have long championed the use of data to ensure CTE programs are adequately preparing all learners for 

lifelong career success. 

We are extremely encouraged by CEP’s ongoing examination of federal administrative and survey data 

as it provides an important opportunity to reassess the current federal data infrastructure which, in our 

view, remains too diffuse to provide meaningful and actionable information to key stakeholders. For a 

more exhaustive response to the overarching and specific questions posed by this solicitation, we direct 

your attention to the comments submitted by the Workforce Data Quality Campaign and the 

Postsecondary Data Collaborative, both of which we fully endorse and support. 

In addition to those comments, which outline in greater detail a host of related recommendations that 

would support a more cohesive federal data system, we would like to respond in greater detail to the 

seventh question posed in this solicitation: “What data should be included in a potential U.S. 

government data clearinghouse(s)? What are the current legal or administrative barriers to including 

such data in a clearinghouse or linking the data?” 

At present, both the Higher Education Act (P.L. 110-315) and the Workforce Innovation and Opportunity 

Act (P.L. 113-128) contain statutory bans on the creation of a federal student-level data system (Secs. 

113 and 501 respectively). These bans prevent policymakers from answering some of the most basic 

questions about the effectiveness of our postsecondary education and training systems, creates an 

opaque marketplace where consumers of postsecondary education and training have little ability to 

The Promise of Evidence-Based Policymaking 1351
Advance CTE: State Leaders Connecting Learning to Work 

8484 Georgia Avenue Suite 320 

Silver Spring, MD 20910 

www.careertech.org 

discern value among providers, and creates an unnecessary administrative burden borne by 

postsecondary institutions. 

The CEP should therefore incorporate a recommendation for Congress to repeal these bans in its 

forthcoming work over the next year. In order to fully leverage such an opportunity, the CEP should also 

recommend that federal agencies, such as the U.S. Departments of Education and Labor, engage with 

postsecondary stakeholders to design and implement a student-level data system for all postsecondary 

students. Such a system would create a national data set that would more accurately establish how 

students navigate through the postsecondary education landscape and into the labor market. Right now 

more than half of states have processes in place to track this sort of student progress, but that 

information is lost when students move out of state for further education or employment making these 

efforts scattershot at best. 

Moreover, a national student-level data system should allow for the disaggregation of student data by 

key characteristics such as race/ ethnicity, socioeconomic status, and others to ensure that this 

information can be used effectively by policymakers at all levels of government to make more informed 

decisions about current and future investments in the nation’s postsecondary education and training 

infrastructure. Finally, and equally as important, the CEP should articulate a series of strenuous data 

privacy and security policies to govern such a system, as part of this overarching recommendation. 

Thank you for the opportunity to provide input into your ongoing efforts to improve upon our national 

data infrastructure and the utility of the information that it produces. We look forward to engaging 

further with the CEP and this work. Should you have additional questions or comments please contact 

our Government Relations Manager, Steve Voytek, at 240-398-5406 or svoytek@careertech.org. 

Sincerely, 

Kimberly Green 

Advance CTE Executive Director 

 

1352 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t16-sx6k

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0126

Comment on FR Doc # 2016-22002

Submitter Information

Name: David Muhlhausen

Address:

Washington, DC, 20002

Email: david.muhlhausen@heritage.org

General Comment

See attached file(s)

Attachments

Muhlhausen CEP Comment

The Promise of Evidence-Based Policymaking 1353
Large-Scale, Multisite Experimental Evaluations Produce the Most 

Credible Evidence of Effectiveness of Federal Programs

Comments before 

Commission on Evidence-Based Policymaking

Docket Number 160907825-6825-01

November 14, 2016

David B. Muhlhausen

Research Fellow

The Heritage Foundation

www.heritage.org

1354 The Promise of Evidence-Based Policymaking
2

My name is David B. Muhlhausen. I am a Research Fellow at The Heritage Foundation. Based 

largely on my book, Do Federal Social Programs Work?,

1

my commentary specifically addresses 

question 16: “How can data, statistics, results of research, and findings from evaluation, be best used to 

improve policies and programs?”

The best way to improve the performance of federal social programs is to find out whether they 

are, in fact, effective. To accomplish this task, the Commission for Evidence-Based Policymaking, 

hereinafter referred to as the Commission, should recommend:

• Evaluations of federal social programs need to be experimental, meaning the use 

random assignment to allocate membership in intervention and control groups. 

• Evaluations should be large-scale experiments that assess effectiveness in multiple 

locations throughout the nation.

• The term “evidence-based” should mean that experimental evaluations of a 

program model have found consistent statistically significant effects that 

meaningfully ameliorate a targeted social problem in at least three different 

settings.

The federal government’s total debt is $19.8 trillion.2 Given scarce federal resources, 

federal policymakers need to fund programs that work and defund programs that do not work. 

Americans, who fund these programs with hard-earned tax dollars, deserve better than 

Congress’s current habit of funding programs that may not produce their intended results. 

To plug this information gap, the evidenced-based policy movement seeks to inform 

policymakers through scientifically rigorous evaluations of the effectiveness of government 

programs.3 In other words, the movement provides tools to identify what works and what 

does not work. 

However, there is disagreement over what can be counted as evidence.4 For example, 

should high-quality quasi-experiments be given the same level of scientific credibility as 

experimental evaluations? Not only should quasi-experiments not be considered as rigorous as 

experiments, but the federal government should place a much lower priority on performing 

quasi-experiments. Experimental evaluations are the most credible and accurate method by 

which to assess effectiveness. Quasi-experiments are not suitable substitutes. 

For federal social programs, the best way to gain this knowledge is to conduct largescale, multisite experimental evaluations that attempt to isolate the direct effects of social 

programs apart from other factors that affect the outcomes of interest. Thus, the Commission’s 

focus should be on identifying ways to increase the number of large-scale multisite 

experimental impact evaluations of federally funded programs. This is why the Commission’s 

inaugural meeting on July 22, 2016 that highlighted the quasi-experimental research is cause 

for concern.5 Experimental designs are superior to quasi-experimental designs. 

In experimental impact evaluations, the treatment or intervention is randomly assigned 

to one group, while the other group does not receive the treatment or intervention. The group 

randomly receiving the services is called the intervention or treatment group, while the group 

The Promise of Evidence-Based Policymaking 1355
3

not receiving the services is the control group. The findings for the control group serve as the 

counterfactual condition. 

Experimental evaluations have high internal validity, while large-scale multisite 

evaluations have high external validity. 

Threats to Internal and External Validity 

The ability to make causal observations comes down to handling threats to internal and 

external validity. Validity refers to the truth or falsity of statements about cause and effect. 

Internal validity is the “basic minimum without which any experiment is uninterpretable.”6

Did in fact the social program being evaluated make a difference in this specific instance? To 

establish internal validity, some kind of control condition is required in order to determine 

what would have happened to the people in the intervention group had they not received the 

intervention. Any threat to internal validity is a hazard to the ability of the evaluation to make 

causal inferences. 

Internal validity is concerned with making causal inferences.7 Did, in fact, social 

program X cause outcome Y? There are several factors that threaten internal validity. A threat 

to internal validity is an objection that an evaluation design allows the causal link between the 

intervention and outcome to remain uncertain. The design is weak in some way. The design 

does not allow the person assessing effectiveness to have confidence in the results. If 

uncontrolled, these factors confound real effects.8 

External validity deals with questions of generalizability. To what populations, settings, 

and times can the effect of a social program be generalized? For example, can the results of an 

evaluation of a delinquency prevention program in Baltimore, Maryland, be generalized to a 

similar program operating in Little Rock, Arkansas? Threats to external validity cast doubts on 

the extent to which the results of the social program as conducted would be duplicated with 

the same program at a different time, or place, or with different participants. Evaluators of 

social programs should strive to select evaluation designs that are strong in both external and 

internal validity. 

External validity is concerned with questions of generalizability. To what populations, 

settings, and times can the particular effect of a social program be generalized? Threats to 

external validity casts doubts on the extent to which the results of a social program as 

conducted would be duplicated with the same program at a different time, or place, or with 

different participants.9 In regards to social programs, there are four patterns of generalization. 

The narrow-to-broad generalization occurs when someone infers a causal relationship from 

the particular persons, setting, treatments, and outcomes of the original social program to a 

larger population.10 This generalization can occur, for example, when a particular social 

program operating in Baltimore, Maryland, is assumed to have the same effect if implemented 

as a nation-wide program. Just because the social program worked in Baltimore does not mean 

that it will work everywhere. Some social programs will work under at least some conditions 

and not under others. Another example is the assumption that all types of drug treatment 

programs are effective based on the results of a particular drug treatment program. 

1356 The Promise of Evidence-Based Policymaking
4

Experimental evaluations are, by far, the best design for dealing with threats to internal 

validity. What about external validity? Experimental evaluations that are large scale—meaning 

large sample sizes—and are conducted in multiple sites have the most validity for making 

generalizations. 

The Advantages of Experimental Evaluations 

The impact of social programs cannot be estimated with 100 percent certainty. All such 

impact evaluations face formidable control problems that make valid estimates difficult. As a 

general rule, the more rigorous the research methodology is, the more reliable the evaluation’s 

findings are. 

Determining the impact of social programs requires comparing the conditions of those 

who received assistance with the conditions of an equivalent group that did not experience the 

intervention. However, evaluations differ by the quality of methodology used to separate the 

net impact of programs from other factors that may explain differences in outcomes between 

comparison and intervention groups. 

Experimental evaluations are the “gold standard” of evaluation designs. Randomized 

experiments attempt to demonstrate causality by (1) holding all possible causes of the 

outcome constant, (2) deliberately altering only the possible cause of interest, and (3) 

observing whether the outcome differs between the intervention and control groups. In 

reality, we can never be 100 percent sure that all the potential causes (confounding factors) 

were held constant. 

When conducting an impact evaluation of a social program, identifying and controlling 

for all the possible factors that influence the outcomes of interest is impossible. We simply do 

not have enough knowledge to accomplish this task. Even if we had the capability to identify all 

possible causal factors, collecting complete and reliable data on all these factors would likely 

still be beyond our abilities. For example, it is impossible to isolate a person participating in a 

social program from his family in order to “remove” the influences of family. This is where the 

benefits of random assignment become clear. 

Because we do not know enough about all possible causal factors to identify and hold 

them constant, randomly assigning test subjects to intervention and control groups allows us 

to have a high degree of confidence that these unidentified factors will not confound our 

estimate of the intervention’s impact. 

Random assignment helps to ensure that the control group is equivalent to the 

intervention group in composition, predispositions, and experiences.11 First, randomization 

means that the intervention and control groups will have an identical composition. The groups 

are comprised of the same types of individuals in terms of their program-related and outcomerelated characteristics. Second, the intervention and control groups will have identical 

predispositions. Members of both groups “are equally disposed toward the program and 

equally likely, without intervention, to attain any given outcome status.”12 Third, the 

intervention and control groups will have identical experiences with regards to time-related 

processes, such as maturation, and history. 

The Promise of Evidence-Based Policymaking 1357
5

Because the intervention and control groups differ from one another by chance only, 

factors outside of the intervention that are related to the outcomes are assumed to be equally 

present in each group. Subjects whose characteristics may make them more responsive to 

treatment are just as likely to be in either the intervention or control groups. The intervention 

and control groups should have the same portion of subjects favorably predisposed to benefit 

from the intervention. Thus, outcome differences should be attributable to the intervention. In 

sum, randomization eliminates any systematic association between intervention status and the 

observed and unobserved participant characteristics, thus largely eliminating the selection 

bias that potentially contaminates other evaluation designs.13

In quasi-experimental designs, failure to remove the influence of differences that affect 

program outcomes leaves open the possibility that the underlying differences between the 

groups, not the program, caused the net impact. While quasi-experimental designs often use 

sophisticated techniques, experimental evaluations are still considered better at producing 

reliable estimates of program effects. For example, evidence in criminal justice policy indicates 

that quasi-experimental evaluations tend to find results contrary to the findings of 

experimental evaluations.14

After conducting a meta-analysis of 308 criminal justice program evaluations, Professor 

David Weisburd of George Mason University and his colleagues found that weaker evaluation 

designs are more likely to find favorable intervention effects and less likely to find harmful 

intervention effects.15 They caution that quasi-experimental designs, no matter how well 

designed, may be incapable of controlling for the unobserved factors that make individuals 

more likely to respond favorably to the intervention. 

Given that experimental evaluations produce the most reliable results, the Commission 

should promote the use of experimental evaluations to assess the effectiveness of federal 

programs. Given the underlying statute that created the Commission, it has a responsibility to 

focus on ways to increase the use of experimental evaluations to assess the performance of 

federal programs. Quasi-experimental designs, no matter how well designed, may be incapable 

of controlling for nonprogram factors that influence how participants respond to the 

intervention. 

Due to the importance of criminal justice policy, Weisburd argues that researchers have 

a moral imperative to conduct randomized experiments16 because of their “obligation to 

provide valid answers to questions about the effectiveness of treatments, practices, and 

programs.”17 This moral imperative also applies to the federal government, which spends 

hundreds of billions of dollars on social programs every year. Yet too few social programs 

funded by the federal government have their effectiveness assessed by experimental 

evaluations. 

Addressing Threats to Internal Validity 

The remaining task regarding the merits of random assignment is to demonstrate that 

this method is the most capable of handling threats to internal validity. Experimental 

evaluations do so by distributing internal validity threats randomly of intervention 

conditions.18 Intervention group members will have the same average characteristics as 

1358 The Promise of Evidence-Based Policymaking
6

control group members. Random assignment, however, does not prevent alternative factors 

from influencing outcomes. Random assignment simply ensures that these factors or events 

are no more likely to happen to intervention group members than to control group members. 

The only systematic difference between the groups will be the intervention. For example, 

consider an experimental evaluation of a social program intended to help disadvantaged high 

school students attend college. Family breakup, such as divorce, may have a causal impact on 

whether students attend college. Random assignment ensures that family breakup is no more 

likely to happen to intervention group members than to control group members. Quasiexperimental designs cannot make such guarantees. 

Random assignment ensures that confounding variables are unlikely to be correlated 

with intervention conditions.19 The random chance of being selected is unrelated to the preexisting conditions of the pool of individuals being allocated to the intervention or control 

groups. Consider a coin toss as the process for allocating individuals to the groups. The results 

of the coin toss are unrelated to the individuals’ race, ethnicity, age, income, or anything else. 

Therefore, we can expect that the preintervention correlation between intervention 

assignment and potential confounding factors should not be significantly different from zero.20

For statistical purposes, this zero correlation is extremely helpful. 

Consider a linear regression model of a job-training evaluation: 

Yi αȽΪȾXXi ΪȾTTi Ϊɂi 

ǡȽthe constant or intercept, 

ȾX ǡǡȾT is 

the regression coefficient that measures the impact of program participation, T is a dummy 

variable coded as 0 when the individual is in the control group and 1 when the individual is in 

ǡɂǡ

and the subscript i ǤȾT is 

significantly different than zero, then job training had a significant effect on income. That is, if 

ȾT is positive and statistically significant, then participation in the job-training program was 

associated with increased income. The opposite would bȾT was negative and 

ǤȾT is statistically indistinguishable from zero, then job training had 

no impact on income. 

However, for this model to provide us with valid impact estimates, T must not be 

ɂ. The statistical procedure used in the linear regression model chooses values 

ȾT, ɂǤ

zero correlation assumption whether or not the correlation really was zero in the study. 

ɂ

will be zero. However, in quasi-experimental evaluations, many confounding factors are likely 

to be correlated with assignment to the intervention. The statistical procedure still chooses the 

ȾǡȾǤ

ǡȾT will an incorrect or biased measure of 

the effect of job training. 

The Promise of Evidence-Based Policymaking 1359
7

However, estimating this equation when assignment to the job-training program is not 

based on random assignment poses a serious threat to yielding unbiased impact estimates. 

When members of the intervention group are purposely selected by job-training 

administrators or they enter into the job-training program through self-selection, selection 

Ǥǡɂ

factors that are also correlated with T. The evaluator cannot measure and account for the 

Ǥǡɂǡ

of program effect. 

Consider an example of a quasi-experimental evaluation of a smoking cessation 

program. If the intervention group is solely comprised of smokers seeking assistance in 

quitting their habit and the comparison group consists of smokers unwilling to seek help in 

ǡɂǤǫ—the 

higher motivation to quit—is unlikely to be observed and accurately measured by the 

evaluator. Controlling for the race, age, and education levels of participants is easier. 

Controlling for motivation of the individual participants is vastly more difficult. The 

unobserved selection process ɂǡ

participation, T. The effect of selection bias in this case means that the quasi-experimental 

evaluation will almost certainly overstate the effectiveness of the smoking cessation program. 

What about High-Quality Quasi-Experimental Evaluations? 

Some will argue that high-quality quasi-experimental designs, such as regression 

discontinuity designs and propensity score matching, are suitable alternatives to experimental 

evaluations. However, these techniques are still less reliable for demonstrating causality, 

compared to rigorous experimental designs. 

Regression Discontinuity Designs. In experimental evaluations, the selection procedure 

is random assignment. In quasi-experimental evaluations, as previously discussed, selection is 

a nonrandom process. This problem renders most quasi-experimental evaluations susceptible 

to selection bias. Instead of trying to figure out what variables are related to selection, in 

regression-discontinuity designs, the evaluator has the selection variable. Many consider 

regression-discontinuity designs to be the second best evaluation design, behind experimental 

evaluations, because they are believed to yield the least biased impact estimates of quasiexperimental designs.21

Regression-discontinuity designs work by applying a case-by-case allocation rule for 

parsing subjects into the intervention and comparison groups according to their “scores” on 

the selection variable. For this method, the selection procedure is some quantitative 

assignment variable, A, such that one group is made of subjects with scores below some cut 

point on A and the other group comprises those subjects whose scores on A are above that cut 

point. The selection variable is formally called the quantitative assessment variable (QAV). It is 

also called the cutting-point or cut-off variable because it applies a cutting point to some 

continuum of need, merit, or other selection variable. Using values along the continuum of the 

selection variable, subjects with scores over a certain point—cutting point—are assigned into 

one group, and those with scores under the cutting point are assigned to the other group.22

1360 The Promise of Evidence-Based Policymaking
8

While regression-discontinuity designs are considered second only to designs using 

random assignment, this method has limited applications because of the need for the QAV. Few 

social programs strictly follow a QAV score for the allocation of program services. In addition 

to limited applications, regression-discontinuity designs have two noteworthy weaknesses. 

First, compared to randomized experiments, regression-discontinuity designs require more 

assumptions on the proper way to specify the regression model.23 Second, randomized 

experiments are more powerful in their ability to detect impacts than regression-discontinuity 

designs.24 Regression-discontinuity designs require 2.7 times more study participants to have 

the same statistical power in detecting statistically significant outcomes compared to 

experimental designs.25

Propensity Score Matching. During the 1970s, the National Supported Work (NSW) 

Demonstration evaluated, through random assignment, voluntary training and assisted work 

programs targeting long-term participants in Aid to Families with Dependent Children (AFDC) 

at 12 sites across the nation. In 1986, Robert J. LaLonde compared the experimental 

evaluation’s results to quasi-experimental methods.26 Specifically, LaLonde used data on the 

NSW intervention group and constructed comparison groups using data from the Panel Study 

of Income Dynamics (PSID) and Current Population Survey (CPS) to serve as the 

counterfactual conditions, instead of the data from the NSW demonstration control group. The 

comparison groups were carefully constructed from the PSID and CPS to be as similar in 

earnings histories as possible to the NSW intervention group. According to LaLonde, the 

results of quasi-experiment “often differ significantly from experimental results.”27 Further, 

“this evidence suggests that policymakers should be aware that the available nonexperimental 

evaluations of employment and training programs may contain large and unknown biases 

resulting from specification errors.”28

The sophistication of quasi-experimental methods has substantially grown since 

LaLonde’s 1986 study. Using propensity score matching, Rejeev H. Dehejia and Sadek Wahba 

produced results that were close to the results of the NSW Demonstration.29 Propensity score 

matching carefully matches intervention groups to nonexperimental comparison groups on 

their likelihood of participating in an intervention based on preintervention characteristics. 

The matching process yields a single propensity score for each member of the intervention and 

comparison groups. In theory, this propensity score summarizes the preintervention 

characteristics and, thus, can be used to control for the preintervention differences between 

the experimental intervention group and the nonexperimental comparison group. Their 

comparison groups were derived from the PSID and CPS. According to Dehejia and Wahba, 

their method obtained “estimates of the treatment impact that are much closer to the 

experimental treatment effect than LaLonde's nonexperimental estimates.”30

However, Jeffrey A. Smith and Petra E. Todd found that that the propensity score 

matching by Dehejia and Wahba worked well only for a precise subsample of the NSW 

Demonstration data.31 For example, Dehejia and Wahba’s propensity score modeling excluded 

almost 40 percent of LaLonde’s observations due to the inclusion of an additional variable in 

their propensity score model.32 Despite excluding such a large portion of the data, Dehejia and 

Wahba’s estimates were close to the original NSW Demonstration results. 

The Promise of Evidence-Based Policymaking 1361
9

However, Larry L. Orr and his coauthors conclude that these propensity score matching 

studies “are less encouraging than they might originally seem.”33 A 2005 study by Jeffrey A. 

Smith and Petra E. Todd found that the results from Dehejia and Wahba’s study were 

extremely sensitive to sample selection and the specification of matching variables used to 

create the propensity scores.34 While Dehejia and Wahba were able to produce similar results 

using a particular subsample, Smith and Todd were unable to produce similar results when 

they applied the same modeling methods to other reasonable subsamples.35 Smith and Todd 

conclude that their “findings make it clear that propensity score matching does not represent a 

‘magic bullet’ that solves the selection problem in every context.”36 Based on the analysis by 

Smith and Todd, Larry L. Orr and his coauthors concluded that while “it was possible to find a 

nonexperimental approach that yielded estimates similar to the (known) experimental results, 

equally plausible approaches—in fact, only slight variations in the nonexperimental 

methods—yielded results different (sometimes very much so) from the experimental 

results.”37

Daniel Friedlander and Philip K. Robins attempted to replicate the results of welfare-towork experimental evaluations done in the 1980s.38 While they found that the results using instate comparison groups produced some improvement compared to the results using out-ofstate comparison groups, both techniques were problematic because they yielded 

inaccuracies.39 Using propensity score methods, another analysis by Wang-Sheng Lee 

attempted to replicate the experimental evaluation of an Indiana welfare-to-work program.40

Even when the comparison group had similar labor market characteristics, the quasiexperimental methods yielded biased impact estimates that were quite large. 

In the education policy field, propensity score matching has not been successful at 

replicating the results of experimental evaluations. Roberto Agodini and Mark Dynarski 

attempted to use propensity score matching to replicate the results of an experimental 

evaluation of middle and high school dropout prevention programs.41 They found “no 

consistent evidence that propensity-score methods replicate experimental impacts of the 

dropout prevention programs ... In fact, we find that evaluating these programs using 

propensity-score methods might have led to misleading inferences about the effectiveness of 

the programs.”42 Another propensity score matching study by Elizabeth Ty Wilde and 

Robinson Hollister attempted to replicate the experimental evaluation results of the 

Tennessee’s Student Teacher Achievement Ratio Project (Project STAR)—a class size 

reduction program.43 Based on their rigorous assessment, “propensity score estimators do not 

perform very well when judged by standards of how close they are to the ‘true’ impacts 

estimated from experimental estimators based on a random assignment design.”44 Further, 

“We hope that this study raises a flag of caution for decision-makers: Do not rush to adopt a 

propensity score matching estimator thinking it will be an adequate substitute for one derived 

from a true experimental design.”45

Steven Glazerman and his coauthors reviewed 12 studies of nonexperimental statistical 

techniques, including propensity score matching, used to replicate the results of experimental 

evaluations of job-training and welfare-to-work programs.46 While they found that some of the 

statistical techniques reduced the bias of quasi-experimental methods to some degree, none of 

the individual techniques reliably replicated the experimental results.47

1362 The Promise of Evidence-Based Policymaking
10

According to Howard L. Bloom and his coauthors, the basic problem is that 

[A]ll propensity-score balancing or matching methods have the limitation that 

they can balance only measured characteristics. If all relevant characteristics are 

measured and included in the estimated propensity score, then balancing the 

program and comparison groups with respect to this score eliminates selection 

bias. But if some important characteristics are not measured—perhaps because 

they cannot be—selection bias might remain. Thus, the quality of program 

impact estimates obtained from propensity-score balancing methods depends 

on how well the comparison group matches the program group before matching 

and on the nature and quality of the data available to measure (and thus further 

balance) sample members’ characteristics. As for any impact estimation 

procedure, the result is only as good as the research design that produced it.48

Propensity score matching in the subject area of voter mobilization has similarly failed 

to accurately replicate the results of an experimental evaluation.49

As far as I am aware, no one has first conducted a propensity score evaluation of a 

social program and then confirmed the quasi-experimental results by performing an 

experimental evaluation. All of these propensity score comparison studies carefully attempted 

to replicate experimental results with known benchmarks in mind. These studies are an 

academic exercise in replicating true experimental results with quasi-experimental techniques. 

No one has done the opposite by demonstrating ex ante that propensity score matching results 

can be replicated using experimental methods. Without the true experimental results as the 

target, evaluators using propensity score matching have no guide in doing their statistical 

analysis. As succinctly put by Phoebe H. Cottingham, and Douglas J. Besharov, “The propensity 

score approach has to rely on modeling using observable baseline data, so one cannot know 

for sure whether the unobservables are introducing substantial bias into the findings”50

In sum, propensity score matching is not a valid substitute for experimental 

evaluations. Propensity score matching’s main failing is that the method is unable to remove 

the effects of selection bias caused by unobserved variables. No matter the level of statistical 

sophistication, the method cannot account for unobserved factors like the ability of random 

assignment. 

Standards for Assessing the Effectiveness of Federal Social Programs 

To properly and accurately assess the effectiveness of federal social programs, the 

Commission should recommend that experimental evaluations should be large in scale and 

based on multiple sites to avoid problems of external validity. Given the multitude of 

confounding factors that may influence the performance of social programs, the larger the size 

of the evaluation, the more likely the social program will be assessed under all the conditions 

that it operates under. 

When Congress creates social programs, especially state and local grant programs, the 

funded activities are implemented in multiple cities or towns. Federal social programs are 

intended to be spread out across the nation. For this reason, the Commission should 

The Promise of Evidence-Based Policymaking 1363
11

recommend that national, multisite experimental evaluations of these federal programs should 

be a priority. 

While individual social programs funded by the federal government may undergo 

experimental evaluations, these small-scale, single-site evaluations do not inform 

policymakers of the general effectiveness of national social programs. Small-scale evaluations 

assess only the impact on a small fraction of people served by federal social programs. The 

success of a single program that serves a particular jurisdiction or population does not 

necessarily mean that the same program will achieve similar success in other jurisdictions or 

among different populations. In addition to using multiple sites, the sample sizes if these 

studies should be large. Commenting on the need for large samples when doing experiments, 

Charles Murray of the American Enterprise Institute correctly observes, 

The main problem is the small size of the samples. Treatment and control groups 

work best when the numbers are large enough that idiosyncrasies in the 

randomization process even out. When you’re dealing with small samples, even 

small disparities in the treatment and control groups can have large effects on 

the results. There are reasons to worry that such disparities existed in both 

programs.51

Simply, small-scale evaluations are poor substitutes for large-scale evaluations. Thus, federal 

social programs should be evaluated in multiple sites so that social programs can be tested in 

the various conditions they operate under and the numerous types of populations they serve. 

Consider the following analogy: If Congress wanted to know the characteristics of the 

population served by Head Start, conducting a survey of a single Head Start program operating 

in Houston, Texas, would not tell us much about the national population served by Head Start. 

To find out the characteristics of the population served by Head Start, a national 

representative sample of Head Start programs would need to be used. The same reasoning 

holds for evaluating effectiveness. If we want to find out the effectiveness of Head Start as a 

national program, then we cannot rely on examining the effects of a single Head Start center. 

The obvious scientific approach would be to undertake a multisite evaluation of Head Start 

that reflects the various conditions that the program operates under nationally. 

For this reason, the Head Start Impact Study is an excellent example for how to assess 

the effectiveness of a federal social program. The Head Start Impact Study (HSIS) began in 

2002, and the immediate-term, short-term, and long-term results released in 2005, 2010, and 

2012, respectively, are disappointing.52 Not only did the HSIS have high internal validity due 

the study’s experimental design, but it also had extremely high external validity because Head 

Start sites were randomly selected. Random selection of sties means that the results tell 

policymakers how the average Head Start site works. A weaker design that selected highly 

performing or sites volunteering to participate in the evaluation will be less likely to tell 

policymakers how the average programs works. 

Multisite experimental evaluations that examine the performance of particular 

programs in numerous and diverse settings can potentially produce results that are more 

persuasive to policymakers than results from a single locality.53

1364 The Promise of Evidence-Based Policymaking
12

The case of police departments performing mandatory arrests in domestic violence 

incidents is a poignant example of why caution should be exercised when generalizing findings 

from a single evaluation. During the 1980s, criminologists Lawrence W. Sherman and Richard 

A. Berk analyzed the impact of mandatory arrests for domestic violence incidents on future 

domestic violence incidents in Minneapolis, Minnesota.54 Compared to less severe police 

responses, the Minneapolis experiment found that mandatory arrests lead to significantly 

lower rates of domestic violence. Sherman and Berk urged caution, but police departments 

across the nation adopted the mandatory arrest policy based on the results of one evaluation 

conducted in one city. 

However, what worked in Minneapolis did not always work in other locations. 

Experiments conducted by Sherman and others in Omaha, Nebraska; Milwaukee, Wisconsin; 

Charlotte, North Carolina; Colorado Springs, Colorado; and Dade County, Florida, found mixed 

results.55 Experiments in Omaha, Milwaukee, and Charlotte found that mandatory arrests lead 

to long-term increases in domestic violence. Apparently, knowing that they would 

automatically be arrested prompted repeat offenders to become more abusive. It seems that 

the following sick logic occurred: If I’m going to automatically spend the night in jail, I might as 

well beat my wife extra good. In a subsequent analysis of the disparate findings, Sherman 

postulated that arrested individuals who lacked a stake in conformity within their 

communities were significantly more likely to engage in domestic violence after arrest, while 

married and employed arrested individuals were significantly less likely to commit further 

domestic violence infractions.56

Contradictory results from evaluations of similar social programs implemented in 

different settings are a product not only of implementation fidelity, but also of the enormous 

complexity of the social context in which these programs are implemented. Jim Manzi, a senior 

fellow at the Manhattan Institute, uses the conflicting results of experimental evaluations to 

explain the influence of “causal density” on the social sciences.57 “Causal density,” a term 

coined by Manzi, is “the number and complexity of potential causes of the outcomes of 

interest.”58 Manzi postulates that as causal density rises, social scientists will find greater 

difficulty in identifying all of the factors that cause the outcome of interest. 

The confounding influence of causal density likely contributed to contradictory effects 

of mandatory arrest policies by location. To address causal density, experimental impact 

evaluations of federal social programs should be conducted using multiple sites. In fact, the 

total sum of the multiple sites should be a nationally representative of the populations served 

by the social program being evaluated. 

A Rigorous Definition of What Counts as “Evidence-Based” 

The term “evidence-based” should mean that experimental evaluations of a program 

model have found consistent statistically significant effects that meaningfully ameliorate a 

targeted social problem in at least three different settings.59 Once a program model has been 

found to produce meaningful results in multiple settings, the likelihood of its successful 

replication elsewhere should increase greatly.60

The Promise of Evidence-Based Policymaking 1365
13

Can Government Replicate Success? In practice, policymakers frequently assume that 

when something has been found effective in one setting, the same results will be repeated 

elsewhere. However, the history of social programs is replete with examples of programs 

effective in one location that simply failed to work elsewhere.

The federal government has a poor record of replicating effective social programs.61 An 

excellent example of a federal attempt to replicate an effective local program is the Center for 

Employment Training (CET) replication.62 Of 13 youth job-training programs evaluated, the 

JOBSTART demonstration found only one program to have a positive impact on earnings: the 

CET in San Jose, California.63 Based on the results for the CET, the U.S. Department of Labor

replicated and evaluated the impact of CET in 12 other sites using random assignment.64 The 

CET model had little to no effect on short-term and long-term employment and earnings 

outcomes at these other locations. According to the evaluation’s authors, “[E]ven in sites that 

best implemented the model, CET had no overall employment and earnings effects for youth in 

the program, even though it increased participants’ hours of training and receipt of 

credentials.”65

Another example is the Head Start CARES Demonstration. In search of evidence that 

Head Start can be an effective program, the U.S. Department of Health and Human Services 

(HHS) initiated the Head Start CARES demonstration project. The demonstration project tested 

three “evidence-based” social-emotional interventions to determine whether these 

interventions help disadvantaged children to develop appropriate social-emotional behaviors. 

The answer was no. Experimental evaluations released in 2014 and 2015 by HHS found that 

enhanced Head Start CARES demonstration programs had little to no effect, compared with 

regular Head Start services, on the social-emotional and academic skills of participating 

children.66

A more recent example is the Obama Administration’s funding of Teen Pregnancy 

Prevention (TPP) grants. The Department of Health and Human Services (HHS) “invests in the 

implementation of evidence-based TPP programs, and provides funding to develop and 

evaluate new and innovative approaches to prevent teen pregnancy.”67 In June 2016, Ron 

Haskins, a research fellow at the Brookings Institution and co-chair of the Evidence-Based 

Policymaking Commission, testified before Congress that HHS requires “high-quality evidence 

showing that the programs produced significant impacts on important measures of teen sexual 

activity or teen pregnancy for the TPP program.”68

According to HHS, Tier 1 grants are awarded to grantees replicating programs that 

“have been shown, in at least one program evaluation, to have a positive impact on preventing 

teen pregnancies, sexually transmitted infections, or sexual risk behaviors.”69 The belief is that 

these grants will be effective because they are replicating programs labeled “evidence-based.” 

Is this assumption correct? 

Each of the Tier 1 grantees is supposed to evaluate the impact of the evidence-based 

model they are replicating. So far from 2015 to 2016, 21 experimental evaluations of nine 

“evidence-based” models have been published by HHS or in the American Journal of Public 

Health.

70 Overwhelmingly, these evaluations demonstrated that replicating “evidence-based” 

1366 The Promise of Evidence-Based Policymaking
14

models failed to affect outcomes assessing program effects on the sexual behaviors of 

participants. Clearly, replicating an evidenced-based program model does not guarantee 

similar results. 

The reason for this failure may because of the inconsistent evidence used to label the 

program models as evidence-based. For example, HHS used contradictory evidence of the 

effectiveness of Becoming A Responsible Teen (BART) program to label this model as 

“evidence-based.” Of the three randomized experiments that were classified with a “high 

ranking” for scientific rigor, two of the studies found the model to be ineffective.71 How can the 

body of research on BART that leans strongly toward the program being ineffective be used to 

promote it as an “evidence-based” model? 

Just because an evidence-based program appears to have worked in one location, does 

not mean that the program can be effectively implemented on a larger scale or in a different 

location. Proponents of evidence-based policymaking should not automatically assume that 

pumping taxpayer dollars toward programs attempting to replicate previously successful 

findings will yield the same results. 

The faulty reasoning that drives such failed expansions of social programs is known as 

the “single-instance fallacy.”72 This fallacy occurs when a person believes that a small-scale 

social program that works in one instance will yield the same results when replicated 

elsewhere. Compounding the effects of this fallacy, one often does not truly know why a 

certain program worked in the first place. In particular, the dedication and entrepreneurial 

enthusiasm of a program’s founder is difficult to quantify or duplicate. HHS’s definition that 

defines a program model as “evidence-based” based on a single evaluation is faulty. 

Model Legislation for Multisite Experimental Evaluations

The following model legislation can be used as a starting point for the Commission to 

make recommendations for how Congress can mandate multisite experimental evaluations of 

federal social programs. First, when authorizing a new program or reauthorizing an existing 

program, Congress should specifically mandate multisite experimental evaluation of the 

program. Second, the experimental evaluations should be large-scale, nationally 

representative, multisite studies. Third, Congress should specify the types of outcome 

measures to be used to assess effectiveness. Fourth, Congress should institute procedures that 

encourage government agencies to carry out congressionally mandated evaluations, despite 

any entrenched biases against experimental evaluations. Fifth, Congress should require that 

congressionally mandated evaluations be submitted to the relevant congressional committees 

in a timely manner after completion. 

SEC. <Insert number>. EVALUATIONS. 

(a) PROGRAMS AND ACTIVITIES CARRIED OUT UNDER THIS TITLE.—For the 

purpose of improving the management and effectiveness of programs and activities 

carried out under this title, the Secretary shall provide for the continuing impact 

evaluation of the programs and activities, including those programs and activities 

carried out under section <Insert number>. Such impact evaluations shall address— 

(1) Outcomes measures of the effectiveness of such programs and

activities in relation to their cost, including the extent to which the programs and 

The Promise of Evidence-Based Policymaking 1367
15

activities— 

(A) Improve the <Insert outcome measures> of participants in 

comparison to comparably situated individuals who did not participate in 

such programs and activities; 

(B) Increase the <Insert outcome measures> over the level that 

would have existed in the absence of such programs and activities; and 

(C) Increase the <Insert outcome measures> of participants in 

comparison to comparably situated individuals who did not participate in 

such programs and activities; 

(2) The effectiveness of the performance measures relating to such 

programs and activities; 

(3) The effectiveness of the structure and mechanisms for delivery of 

services through such programs and activities; 

(4) The impact of such programs and activities on the community and 

participants involved; 

(5) The impact of such programs and activities on related programs and 

activities; 

(6) The extent to which such programs and activities meet the needs of 

various demographic groups; and 

(7) Such other factors as may be appropriate. 

(b) OTHER PROGRAMS AND ACTIVITIES.—The Secretary may conduct impact 

evaluations of other federally funded programs related to <Insert policy area (e.g., 

employment, early childhood education)> and activities under other provisions of law. 

(c) TECHNIQUES.—Impact evaluations conducted under this section shall use 

appropriate methodology and research designs, including the use of intervention and 

control groups chosen by scientific random assignment methodologies. In addition, 

scientific random assignment methodologies shall be used to select the program sites 

that will undergo the evaluation. For each impact evaluation, the Secretary shall fulfill 

all the notification and reporting requirements under subsections (e), (f), and (g). The 

Secretary shall conduct as least 1 multisite control group evaluation under this section 

by the end of fiscal year <Insert year>. 

(d) MANDATORY PARTICPATION.—None of the funds authorized or 

appropriated by Federal law, and none of the funds in any trust fund to which funds are 

authorized or appropriated by Federal law, shall be allocated to sites that refuse to take 

part in the multi-site evaluation after chosen by scientific random assignment 

methodologies for participation in the evaluation. 

(e) NOTIFICATION OF IMPACT EVALUATION PROGRESS.— 

(1) REPORTS TO CONGRESS.—Not later than 1 year after the date of the 

enactment of the <Insert name of Act>, and annually thereafter, the Secretary 

shall transmit to the <Insert two or more House committees> of the House of 

Representatives and the <Insert two or more Senate committees> of the Senate a 

report on the progress the Secretary is making in evaluating the programs and 

activities carried out under this section. 

(2) AVAILABILITY TO GENERAL PUBLIC.—Not later than 1 year after the 

date of the enactment of the <Insert name of Act>, and annually thereafter not 

later than 30 days after the transmission of an annual report under paragraph 

1368 The Promise of Evidence-Based Policymaking
16

(1), the Secretary shall make available the reports to the general public on the 

Internet website of the Department of <Insert name>. 

(f) REPORTS.—The entity carrying out an impact evaluation described in 

subsection (a) or (b) shall prepare and submit to the Secretary a draft report and a final 

report containing the results of the evaluation. 

(g) REPORTS TO CONGRESS.—Not later than 30 days after the completion of 

such a report described in subsection (e), the Secretary shall transmit the draft report 

to the <Insert House committees from subsection (e)> of the House of Representatives 

and the <Insert House committees from subsection (e)> of the Senate. Not later than 30 

days after the completion of such a final report, the Secretary shall transmit the final 

report to such committees of the Congress. All reports must be made available to the 

general public on the Department’s internet web site within 30 days of being 

transmitted to such committees of Congress. 

 (h) DEFINITIONS.—In this section: 

(1) IMPACT EVALUATION—The term “impact evaluation” means an 

evaluative study that evaluates, in accordance with subsection (a), the 

outcomes of programs and activities carried out under this title, including the 

impact on social conditions such programs and activities are intended to 

improve. 

(2) SCIENTIFIC RANDOM ASSIGNMENT METHODOLOGIES—The term 

“scientific random assignment methodologies” means research designs 

conducted in program settings in which intervention and control groups 

are— 

(A) formed by random assignment; and 

(B) compared on the basis of outcome measures for the 

purpose of determining the impact of programs and activities carried 

out under this title participants. 

(3) CONTROL GROUP.—The term “control group” means a group of 

individuals— 

(A) who did not participate in the programs and activities 

carried out under this title; and 

(B) whose outcome measures are compared to the outcome 

measures of individuals in an intervention group. 

(4) INTERVENTION GROUP—The term “intervention group” means a 

group of individuals— 

(A) who participated in the programs and activities carried out

under this title; and 

(B) whose outcome measures are compared to the outcome 

measures of individuals in a control group. 

Conclusion

The best way to improve the performance of federal social programs is to find out whether they 

are, in fact, effective. First, the Commission should recommend that evaluations of federal social 

programs need to be experimental, meaning the use random assignment to allocate membership in 

The Promise of Evidence-Based Policymaking 1369
17

intervention and control groups. Second, these evaluations should be large-scale experiments that 

assess effectiveness in multiple locations throughout the nation. Last, the commission should 

recommend that program models labeled as “evidence-based” be based upon experimental 

evaluations that have found consistent statistically significant effects that meaningfully 

ameliorate a targeted social problem in at least three different settings. The adoption of these 

recommendations will greatly assist the federal government promote the development and 

use of rigorous evidence of program effectiveness. 

 

1 David B. Muhlhausen, Do Federal Social Programs Work? (Santa Barbara, CA: Praeger, 2013). 2

U.S. Department of the Treasury, “The Debt to the Penny and Who Holds It,” http://treasurydirect.gov/NP/debt/current

(accessed November 4, 2016).

3

David B. Muhlhausen, “Evidence-Based Policymaking: A Primer,” Heritage Foundation Backgrounder No. 3063, 

October 15, 2015, http://www.heritage.org/research/reports/2015/10/evidence-based-policymaking-a-primer . 4

Stewart I. Donaldson, “In Search of the Blueprint for an Evidence-Based Global Society,” in What Counts as Credible 

Evidence in Applied Research and Evaluation Practice? eds. Stewart I. Donaldson, Christina A. Christie, and Melvin M. 

Mark (Thousand Oaks, CA: SAGE Publications, 2009), pp. 2–18.

5 Commission on Evidence-Based Policymaking, “First Meeting - July 22, 2016,” http://cep.gov/events/2016-07-22.html

(Accessed November 7, 2016). 

6 Donald Campbell and Julian C. Stanley, Experimental and Quasi-Experimental Designs for Research (Boston: 

Houghton Mifflin Company, 1963), p. 5.

7 William R. Shadish, Thomas D. Cook, and Donalt T. Campbell, Experimental and Quasi-Experimental Designs for 

Generalized Causal Inference (Boston, MA: Houghton Mifflin Company, 2002), p. 54. 8 For a more detailed discussion of internal validity, see Muhlhausen, Do Federal Social Programs Work? 9 Shadish et al., Experimental and Quasi-Experimental Designs for Generalized Causal Inference, p. 83. 10 Ibid. 11Peter H. Rossi, Mark W. Lipsey, and Howard E. Freeman, Evaluation: A Systematic Approach (Thousand Oaks, CA: 

SAGE Publications, Inc., 2004), p. 239.

12 Ibid. 13Gary Burtless, “Randomized Field Trials for Policy Evaluations: Why Not in Education?” in Evidence Matters: 

Randomized Trials in Education Research, eds. Frederick Mosteller and Robert Boruch (Washington, DC: Brookings 

Institution, 2002), pp. 179–197.

14 David Weisburd, Cynthia M. Lum, and Anthony Petrosino, “Does Research Design Affect Study Outcomes in 

Criminal Justice?” Annals of the American Academy of Political and Social Sciences, No. 578 (November 2001), pp. 50–

70.

15 Ibid. 16David Weisburd, “Ethical Practice and Evaluation of Interventions in Crime and Justice,” Evaluation Review Vol. 27, 

No. 23 (June 2003), pp. 336–354.

17Ibid., p. 350. 18 Shadish, Cook, and Campbell, Experimental and Quasi-Experimental Designs for Generalized Causal Inference, p. 

249.

19 Ibid., p. 251. 20 Ibid. 21 Rossi, Lipsey, and Freeman, Evaluation, p. 288. 22 For detailed discussion of regression discontinuity designs, see Muhlhausen, Do Federal Social Programs Work? 23 Mohr, Impact Analysis for Program Evaluation, p. 135; Rossi, Lipsey, and Freeman, Evaluation, p. 147. 24 Rossi, Lipsey, and Freeman, Evaluation, 147. 25 Ibid. 26 Robert J. LaLonde, “Evaluating the Econometric Evaluations of Training Programs with Experimental Data,” 

American Economic Review, Vol.76, No. 4 (September 1986), pp. 604–620. 27 Ibid., p. 617. 28 Ibid. 29 Rajeev H. Dehejia and Sadek Wahba, “Causal Effects in Nonexperimental Studies: Reevaluating the Evaluation of 

Training Programs,” Journal of the American Statistical Association, Vol. 94, No. 448 (December 1999), pp.1053–1062. 30 Ibid., p. 1053.

1370 The Promise of Evidence-Based Policymaking
18

 31Jeffrey A. Smith and Petra E. Todd, “Reconciling Conflicting Evidence on the Performance of Propensity-Score 

Matching Methods,” American Economic Review, Vol. 91, No. 2 (May 2001), pp. 112–118. 32 Ibid., p. 113. 33 Larry L. Orr, Stephen H. Bell, and Jacob A. Klerman, “Designing Reliable Impact Evaluations,” in The Workforce 

Investment Act: Implementation Experiences and Evaluation Findings, eds. Douglas J. Besharov and Phoebe H. 

Cottingham (Kalamazoo, MI: W. E. Upjohn Institute for Employment Research, 2011), p. 437.

34 Jeffrey A. Smith and Petra E. Todd, “Does Matching Overcome LaLonde’s Critique of Nonexperimental Estimators?” 

Journal of Econometrics, Vol. 125 (2005), pp. 305–353. 35 Ibid. 36 Ibid., p. 307. 37. Orr, Bell, and Klerman, “Designing Reliable Impact Evaluations,” p. 437.

38 Daniel Friedlander and Philip K. Robins, “Evaluating Program Evaluations: New Evidence on Commonly Used 

Nonexperimental Methods,” American Economic Review, Vol. 85, No. 4 (September 1995), pp. 923–937. 39 Ibid., p. 935. 40 Wang-Sheng Lee, “Evaluating the Effects of a Mandatory Government Program Using Matched Groups within a 

Similar Geographic Location,” Melbourne Institute of Applied Economic and Social Research, University of Melbourne, 

November 20, 2006.

41 Roberto Agodini and Mark Dynarski, “Are Experiments the Only Option? A Look at Dropout Prevention Programs,” 

Review of Economics and Statistics, Vol. 86, No. 1 (February 2004), pp. 180–194. 42 Ibid., p. 192. 43 Elizabeth Ty Wilde and Robinson Hollister, “How Close Is Close Enough? Evaluating Propensity Score Matching 

Using Data from a Class Size Reduction Experiment,” in Social Experimentation, Program Evaluation and Public 

Policy, ed. Maureen A. Pirog (Association for Public Policy Analysis and Management, 2008), pp. 65–90. 44 Ibid., p. 86. 45 Ibid. 46 Steven Glazerman, Dan M. Levy, and David Myers, “Nonexperimental versus Experimental Estimates of Earnings 

Impacts,” Annals of the American Academy of Political and Social Science, Vol. 589 (September 2003), pp. 63–93. 47 Ibid. 48 Howard S. Bloom, Charles Michalopoulos, and Carolyn J. Hill, “Using Experiments to Assess Nonexperimental 

Comparison-Group Methods for Measuring Effects,” in Learning More from Social Experiments: Evolving Analytic 

Approaches, ed. Howard S. Bloom (New York: Russell Sage Foundation, 2005), p. 178. 49 Kevin Arceneaux, Alan S. Gerber, and Donald Green, “Comparing Experimental and Matching Methods

Using a Large-Scale Voter Mobilization Experiment,” Political Analysis, Vol. 14 (2006), pp. 37–62. 50 Phoebe H. Cottingham and Douglas J. Besharov, “Introduction” in The Workforce Investment Act: Implementation 

Experiences and Evaluation Findings, eds. Douglas J. Besharov and Phoebe H. Cottingham (Kalamazoo, MI: W. E. 

Upjohn Institute for Employment Research, 2011), p. 40.

51 Charles Murray, “The Shaky Science Behind Obama’s Universal Pre-K,” Bloomberg.com, February 20, 2013, 

http://www.bloomberg.com/news/2013-02-21/the-shaky-science-behind-obama-s-universal-pre-k.html (accessed

November 7, 2016).

52U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, 

Research, and Evaluation, Head Start Impact Study: First Year Findings, June 2005, and U.S. Department of Health and 

Human Services, Administration for Children and Families, Office of Planning, Research, and Evaluation, Head Start 

Impact Study: Final Report, January 2010; and Mike Puma, Stephen Bell, Ronna Cook, Camilla Heid, Pam Broene, 

Frank Jenkins, Andrew Mashburn, and Jason Downer, “Third Grade Follow-Up to the Head Start Impact Study: Final 

Report,” U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, 

Research and Evaluation, OPRE Report 2012-45, October 2012. 53Erica B. Baum, “When the Witch Doctors Agree: The Family Support Act and Social Science Research,” Journal of 

Policy Analysis and Management 10, no. 4 (Autumn 1991): 603–615; Judith M. Gueron, “The Politics of Random 

Assignment: Implementing Studies and Affecting Policy,” in Evidence Matters: Randomized Trials in Education 

Research, eds. Frederick Mosteller and Robert Boruch (Washington, DC: Brookings Institution, 2002), 15–49. 54Lawrence W. Sherman and Richard A. Berk, “The Specific Deterrent Effects of Arrest for Domestic Assault,” 

American Sociological Review 49, no. 2 (April 1984): 261–272. 55Lawrence W. Sherman, Domestic Violence: Experiments and Dilemmas (New York: Free Press, 1992); Lawrence W. 

Sherman, Douglas A. Smith, Janell D. Schmidt, and Dennis Rogan, “Crime, Punishment, and Stake in Conformity: 

Legal and Informal Control of Domestic Violence,” American Sociological Review 57 (October 1992): 680–690; 

Lawrence W. Sherman, Janell D. Schmidt, Dennis Rogan, Douglas A. Smith, Patrick R. Gartin, Ellen G. Cohn, Dean J. 

The Promise of Evidence-Based Policymaking 1371
19

 Collins, and Anthony R. Bacih, “The Variable Effects of Arrest on Criminal Careers: The Milwaukee Domestic 

Violence Experiment,” Journal of Criminal Law & Criminology 83, no. 1 (1992): 137–169. 56 Sherman, Domestic Violence. 57Jim Manzi, “What Social Science Does—and Doesn’t—Know,” City Journal 20, no. 3 (Summer 2010): 14–23, 

http://www.city-journal.org/2010/20_3_social-science.html (March 14, 2011).

58Ibid. 59David B. Muhlhausen, “Evidence-Based Policymaking: A Primer,” Heritage Foundation Backgrounder No. 3063, 

October 15, 2015, http://www.heritage.org/research/reports/2015/10/evidence-based-policymaking-a-primer. 60Muhlhausen, Do Federal Social Programs Work?; Stuart M. Butler and David B. Muhlhausen, “Can Government 

Replicate Success?” National Affairs (Spring 2014), pp. 25–39, http://www.nationalaffairs.com/publications/detail/cangovernment-replicate-success (accessed November 9, 2016); and Muhlhausen, “Evidence-Based Policymaking: A 

Primer.” 

61Muhlhausen, Do Federal Social Programs Work? 62Cynthia Miller et al., “The Challenge of Replicating Success in a Changing World: Final Report on the Center for 

Employment Training Replication Cites,” Manpower Demonstration Research Corporation, September 2005, 

http://www.mdrc.org/publication/challenge-repeating-success-changing-world (accessed September 2, 2015).

63George Cave et al., “JOBSTART: Final Report on a Program for School Dropouts,” Manpower Demonstration 

Research Corporation, October 1993, http://www.mdrc.org/project/jobstart (accessed September 2, 2015). 64Miller et al., “The Challenge of Replicating Success in a Changing World.” 65Ibid., p. xi.

66 Pamela Morris, Shira K. Mattera, Nina Castells, Michael Bangser, Karen Bierman, and Cybele Raver, Impact Findings 

from the Head Start CARES Demonstration: National Evaluation of Three Approaches to Improving Preschoolers’

Social and Emotional Competence, U.S. Department of Health and Human Services, Administration for Children and 

Families, Office of Planning, Research and Evaluation OPRE Report No. 2014-44, August 2014, 

http://www.acf.hhs.gov/programs/opre/resource/impact-findings-from-the-head-start-cares-demonstration-nationalevaluation-of-three-approaches-to-improving-preschoolers-social (accessed November 8, 2016) and JoAnn Hsueh, Amy 

E. Lowenstein, Pamela Morris, and Michael Bangser Impacts of Social-Emotional Curricula on Three-Year-Olds: 

Exploratory Findings from the Head Start CARES Demonstration, U.S. Department of Health and Human Services, 

Administration for Children and Families, Office of Planning, Research, and Evaluation, OPRE Report 2014-78, 

December 2014, http://www.acf.hhs.gov/programs/opre/resource/exploratory-impacts-of-three-social-emotionalcurricula-on-three-year-olds-in-the-head-start-cares-demonstration (accessed November 8, 2016). For a summary of the 

demonstration, see David B. Muhlhausen, “The Head Start CARES Demonstration: Another Failed Federal Early 

Childhood Education Program,” The Heritage Foundation Backgrounder No. #3040, August 6, 2015, 

http://www.heritage.org/research/reports/2015/08/the-head-start-cares-demonstration-another-failed-federal-earlychildhood-education-program. 67U.S. Department of Health and Human Services, Office of Adolescent Health, “Teen Pregnancy Prevention,” 

http://www.hhs.gov/ash/oah/oah-initiatives/tpp_program/about/ (accessed July 22, 2016).

68Ron Haskins, “Renewing Communities and Providing Opportunities Through Innovative Solutions to Poverty,” 

testimony before the Committee on Homeland Security and Governmental Affairs, U.S Senate, June 22, 2016, 

http://www.brookings.edu/research/testimony/2016/06/22-renewing-communities-and-providing-opportunities-throughinnovative-solutions-to-poverty-haskins (accessed July 22, 2016).

69U.S. Department of Health and Human Services, Office of Adolescent Health, “Evidence-Based TPP Programs,” 

http://www.hhs.gov/ash/oah/oah-initiatives/tpp_program/db/ (accessed July 22, 2016). 70U.S. Department of Health and Human Services, Office of Adolescent Health, “Grantees FY 2010–2014,” 

http://www.hhs.gov/ash/oah/oah-initiatives/evaluation/grantee-led-evaluation/grantees-2010-2014.html (accessed 

September 26, 2016) and American Journal of Public Health, Vol. 106, No. S1 (October 2016). 71 The two studies that found BART to be ineffective are Angela R. Robertson, Janet St. Lawrence, David T. Morse, 

Connie Baird-Thomas, Hui Liew, and Kathleen Gresham, “The Healthy Teen Girls Project: Comparison of Health 

Education and STD Risk Reduction Intervention for Incarcerated Adolescents Females,” Health Education & Behavior

Vol. 38, No. 3 (2011), pp. 241-250 and Janet S. St. Lawrence, R.A. Crosby, L. Belcher, and N. Yazdani, “Sexual Risk 

Reduction and Anger Management Interventions for Incarcerated Male Adolescents: A Randomized Controlled Trial of 

Two Interventions,” Journal of Sex Education and Therapy, Vol. 24, No. 1-2 (1999)), pp. 9-17. The study that found at 

least one beneficial effect is Janet S. St. Lawrence, Ted L. Brasfield, Aaron Shirley, Kennis W. Jefferson, Edna Alleyne, 

and Robert O’Bannon, “Cognitive-Behavioral Intervention to Reduce African American Adolescents’ Risks for HIV 

Infection,” Journal of Consulting and Clinical Psychology Vol. 63 No. 2 (1995), pp 221-237. 72Butler and Muhlhausen, “Can Government Replicate Success?” 

1372 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t16-vmjq

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0127

Comment on FR Doc # 2016-22002

Submitter Information

Name: Judy Freudenthal

Address:

Nashville, TN, 37203

Email: jfreudenthal@oasiscenter.org

General Comment

See attached file(s)

TO: Commission on Evidence-Based Policymaking 

Comments Docket ID: USBC-2016-0003 

Federal Register Number: 2016-22002

FROM: Oasis Center

1704 Charlotte Ave, Suite 200

Nashville, TN 37203 November 14, 2016

Dear Chair Katharine G. Abraham, Co-Chair Ron Haskins, and members of the

Commission:

Oasis Center welcomes the opportunity to provide comments to the Commission on

Evidence-Based Policymaking (CEP). We offer input for questions 16 and 18 based on

our experience as a grantee of the Administration for Children and Families, Personal

Responsibility Education Program (PREP) supporting implementation of a high quality,

The Promise of Evidence-Based Policymaking 1373
evidenced based program. PREP has been recognized as a pioneering example of tiered

evidence-based policymaking, and represents an important contribution to building a

body of evidence of what works. This includes high quality implementation, evaluation,

innovation, and learning from results. 

At Oasis Center we work to create a community where all young people are safe, valued,

connected with caring adults, and prepared for productive and fulfilling lives. Therefore,

our work involves providing emotional and physical supports now, as well as leading

social change that will improve the odds for greater success for more youth in the future.

Key goals include: to institutionalize positive youth development practices to improve

the quality of care for foster youth in residential care in TN; develop and implement a

citywide plan to ensure youth have safe, stable housing solutions; and to replace

systemic exclusionary practices in Metro schools with restorative interventions that

promote equity and success for all students. 

As a PREP sub-grantee we incorporated evidence into all aspects of our project. The

program model we implemented, Wyman's Teen Outreach Program (TOP), was chosen

from a list of those that had already been rigorously evaluated and demonstrated to

change behavior. We also used evidence to guide and improve our project throughout the

grant period. 

1. Weekly review of fidelity data helps us understand the strengths and challenges of

each of our 23 implementation sites and guides our provision of additional training,

coaching, and technical assistance to the organizations implementing TOP. Measuring

the rate at which youth at each site achieve the prescribed program dosage over time also

helps us understand our sites' performance. Feeding both these kinds of data back to

leadership at implementing organizations helps guide their understanding of their

performance and identify needs for improvement.

2. Data from Wyman Pre- and Post-Surveys appears to show positive trends in reducing

teen pregnancy and reducing problematic or risky behavior among foster youth. With

regard to teen pregnancy, youth who have completed both Pre- and Post- Surveys report

a decline in the rate at which they have become or caused a pregnancy since entering

TOP or intend to do so within 4 months of exiting TOP. Similarly, youth who have

completed both Pre- and Post-Surveys report a decline in school suspensions, failing

courses, getting any failing grades, and cutting classes without permission while

participating in TOP and project a decline in these areas in the next school year. 

3. From youth responses on the Wyman Post-Surveys, it is obvious youth feel very

positive about their experiences in TOP. This data set supports anecdotal reports from

staff and foster youth about the value of TOP. TOP Facilitators have successfully created

positive, caring environments for the youth and appear to be implementing TOP

according to a positive youth development perspective, as intended. Youth seem to be

learning valuable life skills from their participation in TOP. These positive results have

motivated the State of TN to add state-run juvenile justice facilities to the list of TOP

implementation sites in TN.

1374 The Promise of Evidence-Based Policymaking
In closing, we consider PREP to be a prime example of high-quality evidence-based

policymaking. It is our understanding that PREP is one of the few government programs

that use evidence and evaluation criteria throughout the grant life cycle. Thank you for

considering our input for the CEP. If you have any questions or need additional

information, please contact me at 615/327-4455 or jfreudenthal@oasiscenter.org. 

Sincerely,

Judy Freudenthal, Ed.D

Vice-President of Youth Engagement, Action & Program Evaluation

Oasis Center

Attachments

Evid-Based Commission_Oasis Ctr 11 2016

The Promise of Evidence-Based Policymaking 1375
TO: Commission on Evidence-Based Policymaking 

Comments Docket ID: USBC-2016-0003 

Federal Register Number: 2016-22002 

FROM: Oasis Center 

 1704 Charlotte Ave, Suite 200 

 Nashville, TN 37203 November 14, 2016 

Dear Chair Katharine G. Abraham, Co-Chair Ron Haskins, and members of the Commission: 

Oasis Center welcomes the opportunity to provide comments to the Commission on Evidence-Based 

Policymaking (CEP). We offer input for questions 16 and 18 based on our experience as a grantee of the 

Administration for Children and Families, Personal Responsibility Education Program (PREP) supporting 

implementation of a high quality, evidenced based program. PREP has been recognized as a pioneering 

example of tiered evidence-based policymaking, and represents an important contribution to building a 

body of evidence of what works. This includes high quality implementation, evaluation, innovation, and 

learning from results. 

At Oasis Center we work to create a community where all young people are safe, valued, connected 

with caring adults, and prepared for productive and fulfilling lives. Therefore, our work involves 

providing emotional and physical supports now, as well as leading social change that will improve the 

odds for greater success for more youth in the future. Key goals include: to institutionalize positive 

youth development practices to improve the quality of care for foster youth in residential care in TN; 

develop and implement a citywide plan to ensure youth have safe, stable housing solutions; and to 

replace systemic exclusionary practices in Metro schools with restorative interventions that promote 

equity and success for all students. 

As a PREP sub-grantee we incorporated evidence into all aspects of our project. The program model we 

implemented, Wyman’s Teen Outreach Program (TOP), was chosen from a list of those that had already 

been rigorously evaluated and demonstrated to change behavior. We also used evidence to guide and 

improve our project throughout the grant period. 

1. Weekly review of fidelity data helps us understand the strengths and challenges of each of our 

23 implementation sites and guides our provision of additional training, coaching, and technical 

assistance to the organizations implementing TOP. Measuring the rate at which youth at each 

site achieve the prescribed program dosage over time also helps us understand our sites’ 

performance. Feeding both these kinds of data back to leadership at implementing 

organizations helps guide their understanding of their performance and identify needs for 

improvement. 

2. Data from Wyman Pre- and Post-Surveys appears to show positive trends in reducing teen 

pregnancy and reducing problematic or risky behavior among foster youth. With regard to teen 

pregnancy, youth who have completed both Pre- and Post- Surveys report a decline in the rate 

at which they have become or caused a pregnancy since entering TOP or intend to do so within 

4 months of exiting TOP. Similarly, youth who have completed both Pre- and Post-Surveys 

report a decline in school suspensions, failing courses, getting any failing grades, and cutting 

1376 The Promise of Evidence-Based Policymaking
classes without permission while participating in TOP and project a decline in these areas in the 

next school year. 

3. From youth responses on the Wyman Post-Surveys, it is obvious youth feel very positive about 

their experiences in TOP. This data set supports anecdotal reports from staff and foster youth 

about the value of TOP. TOP Facilitators have successfully created positive, caring environments 

for the youth and appear to be implementing TOP according to a positive youth development 

perspective, as intended. Youth seem to be learning valuable life skills from their participation in 

TOP. These positive results have motivated the State of TN to add state-run juvenile justice 

facilities to the list of TOP implementation sites in TN. 

In closing, we consider PREP to be a prime example of high-quality evidence-based policymaking. It is 

our understanding that PREP is one of the few government programs that use evidence and evaluation 

criteria throughout the grant life cycle. Thank you for considering our input for the CEP. If you have any 

questions or need additional information, please contact me at 615/327-4455 

or jfreudenthal@oasiscenter.org. 

Sincerely, 

Judy Freudenthal, Ed.D 

Vice-President of Youth Engagement, Action & Program Evaluation 

Oasis Center 

The Promise of Evidence-Based Policymaking 1377
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t16-okpn

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0128

Comment on FR Doc # 2016-22002

Submitter Information

Name: Inspector General Tighe ED-OIG

General Comment

See attached letter, which references a January 2013 Forum on Data Analytics for

Oversight and Law Enforcement and a related GAO report, GAO-13-680SP.

Attachments

comments.Commission on Evidence-Based Policy Making.11.16

1378 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 1379
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t17-vqpn

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0129

Comment on FR Doc # 2016-22002

Submitter Information

Name: Keith Ewald

Address:

Columbus, OH, 43082

Email: keith.ewald@jfs.ohio.gov

General Comment

See attached file(s)

Attachments

Federal Register - commisison on evidence-based policymaking -CP Edits

1380 The Promise of Evidence-Based Policymaking
Docket ID: 160907825-6825-01 

Commission on Evidenced-Based Policymaking Comments 

Thank you for the opportunity to respond to this Federal Register Notice. For your 

consideration, please see the response to your questions below: 

Overarching Questions 

1. Are there successful frameworks, policies, practices, and methods to overcome challenges related to 

evidence-building from state, local, and/ or international governments the Commission should consider 

when developing findings and recommendations regarding Federal evidence-based policymaking? If so, 

please describe. 

In relatively recent years there have been a number of experiments to explore the methods and 

governance for the use of administrative data for policy analysis. Administrative Data Research 

and Evaluation (ADARE) is an early research university partnership model. More recently the 

Statewide Longitudinal Data System (SLDS) and Workforce Data Quality Initiative (WDQI) grants 

have encouraged state agency partnership models. Also, the NASWA LMI committee could 

provide examples of cross state partnerships as well. These should all be reviewed for ideas and 

best practices. 

The BLS/State cooperative programs is a long established and interesting model for blending 

state and federal goals. It is a data management model that allows state applications of data 

and input into policy. 

The U.S. Census Bureau’s Longitudinal Employer-Household Dynamics (LEHD) program serves as 

model for cross state data integration and MOUs, with the caveat that it is very limited in access 

for state research interests. 

The most critical issue is providing a governance structure that addresses the interests, needs 

and concerns of the data providers/owners and setting policy and boundaries in respect to data 

access and uses. Data flow, utility and application should not be one way. 

2. Based on identified best practices and existing examples, what factors should be considered in 

reasonably ensuring the security and privacy of administrative and survey data? 

There are well established IT practices for protecting data. However, I believe the most critical 

aspects relate to governance. A few basic principles are: 

• Data should be cross linked and anonymized before researcher access. 

• No linking of researcher survey data that may allow for re-identification. 

• An established Internal Review Board for all research. 

The Promise of Evidence-Based Policymaking 1381
• All researchers must go through training on human subject research such as offered by 

CITI. 

• Governance approval of research results before publication/release. 

• There should be explicit requirements for data suppression before release or 

publication. 

• Research requests must address the objectives and questions the research is to address 

and be within the priorities established by the governance process. 

• The research priorities should represent the interests of the data providers and program 

partners and possibly some representation of their associations or academics. In short, 

there must be some boundaries not a free for all. 

• Research should not be a right but a privilege. Requests for access must meet 

requirements or evaluation criteria. Possibly structured competitively and approving 

only the best of the best. 

• Consider creating a synthetic database to meet certain research needs or objectives. 

• Establish periodic and independent audit of security and practices of data centers. 

Data Infrastructure and Access 

3. Based on identified best practices and existing examples, how should existing government data 

infrastructure be modified to best facilitate use of and access to administrative and survey data? 

This question hits upon the biggest challenge and largest of tasks but is also the one essential to 

success. Administrative data is organized for case management and to meet federal or state 

reporting requirements. As such it seldom can be used in raw form to meet research objectives. 

Data must be reorganized, transposed and otherwise structured in a linked-longitudinal fashion, 

reviewed for quality and validity, documented and cast into a metadata file for researchers to 

reasonably assess research feasibility and options. This must be maintained as administrative 

programs go through changes in data structure, content and definitions through time. 

4. What data-sharing infrastructure should be used to facilitate data merging, linking, and access for 

research, evaluation, and analysis purposes? 

Here I think it is important to avoid additional burden and costs on administrative programs as 

much as possible. Mechanisms need to be setup to pretty much transfer data as it exists in the 

programs to the clearing house for incorporation into a linked-longitudinal database. Several 

standard data formats should be acceptable to streamline the process for administrative 

programs. 

5. What challenges currently exist in linking state and local data to federal data? Are there successful 

instances where these challenges have been addressed? 

Historically administrative programs across levels of government have been developed 

independently or siloed in structure. The corresponding legal frameworks were not written with 

data alignment in mind. Furthermore, different administrative entities have developed varying 

practices and policies relative to their circumstances and experiences. This mishmash of legal 

1382 The Promise of Evidence-Based Policymaking
interpretation and varying policy and practice across government is the biggest barrier to 

overcome. The BLS Federal/State Cooperative programs is one example where consistent 

practices have been established for a number of related data series. Although not specific to 

this issue, it does offer a model of state/federal partnership in the management of data. The 

other prime example would be the Census LEHD program. 

 

6. Should a single or multiple clearinghouse(s) for administrative and survey data be established to 

improve evidence-based policymaking? What benefits or limitations are likely to be encountered in 

either approach? 

I generally frown on disseminating this role or a federated system because I see that as 

duplication of effort, legal and administrative costs and risks. It’s also more likely to blur 

authority and accountability. Given the breadth of the effort, I would recommend a middle 

approach where a single linked-longitudinal archiving center exists within each state and they in 

turn have a cooperative relationship with a federal center. 

7. What data should be included in a potential U.S. government data clearinghouse(s)? What are the 

current legal or administrative barriers to including such data in a clearinghouse or linking the data? 

I think this should be addressed by some kind of assessment and consensus on what are the 

most pressing research priorities of the states and federal government. In short begin with a 

manageable objective, demonstrate success and expand the players based on experience. My 

take would be to focus on programs that are geared toward workforce outcomes. 

8. What factors or strategies should the Commission consider for how a clearinghouse(s) could be selffunded? What successful examples exist for self-financing related to similar purposes? 

Given a focus on policy analysis (which I take this to be) there ought to be some level of 

government commitment, including financial support. Government and their citizenship should 

be the main beneficiaries. Additionally, the government’s role and responsibility for 

administrative data ought to be maintained through some level of direct funding. Beyond this, I 

see no problem with charging for access and pursuing relationships with grant funders. 

9. What specific administrative or legal barriers currently exist for accessing survey and administrative 

data? 

See question 5. The mishmash of legal interpretation and varying policy and practice across 

government is the biggest barrier to overcome. With programs and their legal foundations 

developed independently and policy and practice variations across programs and states, there is 

little consensus. 

 

10. How should the Commission define ‘‘qualified researchers and institutions?’’ To what extent should 

administrative and survey data held by government agencies be made available to ‘‘qualified 

researchers and institutions?’’ 

The Promise of Evidence-Based Policymaking 1383
 Repeating from question 2: A few basic principles are: 

• Data should be cross linked and anonymized before researcher access. 

• No linking of researcher survey data that may allow for re-identification. 

• An established Internal Review Board for all research. 

• All researchers must go through training on human subject research such as offered by 

CITI. 

• Governance approval of research results before publication/release. 

• There should be explicit requirements for data suppression before release or 

publication. 

• Research requests must address the objectives and questions the research is to address 

and be within the priorities established by the governance process. 

• The research priorities should represent the interests of the data providers and program 

partners and possibly some representation of their associations or academics. In short, 

there must be some boundaries not a free for all. 

• Research should not be a right but a privilege. Requests for access must meet 

requirements or evaluation criteria. Possibly structured competitively and approving 

only the best of the best. 

• Consider creating a synthetic database to meet certain research needs or objectives. 

• Establish periodic and independent audit of security and practices of data centers. 

11. How might integration of administrative and survey data in a clearinghouse affect the risk of 

unintentional or unauthorized access or release of personally-identifiable information, confidential 

business information, or other identifiable records? How can identifiable information be best protected 

to ensure the privacy and confidentiality of individual or business data in a clearinghouse? 

The present reality is that there is immense pressure from all corners for access to 

administrative data to engage in evidenced based research and policy analysis. Currently we see 

more and more examples of programs and administrative entities feeling obligated to comply 

with such requests. The problem is that the bulk of these are ad hoc and one-up events with 

little standardization of approach or consistent practices. In particular, across entities and 

governments we lack consistent standards, overall governance and monitoring systems. 

Establishing an overall governance process, data management and access standards and 

monitoring systems along with best IT practices and elements discussed in question 2, through 

the clearing house concept, increases security and accountability. 

12. If a clearinghouse were created, what types of restrictions should be placed on the uses of data in 

the clearinghouse by ‘‘qualified researchers and institutions?’’ 

See question 2. 

1384 The Promise of Evidence-Based Policymaking
13. What technological solutions from government or the private sector are relevant for facilitating data 

sharing and management? 

There are many well established IT platforms and security practices. Asking this question 

implies to me that in the visionary quest for “big data” it is assumed that it is an IT problem with 

a simple IT solution. It is not. The primary issue is partnership alignment, legal procedures, 

governance structure, data management, linked-longitudinal archiving, data quality and 

validation processes, and metadata construction. These are organizational and management 

tasks. 

14. What incentives may best facilitate interagency sharing of information to improve programmatic 

effectiveness and enhance data accuracy and comprehensiveness? 

 A few suggestion: 

• A streamlined system that has minimal impact and cost to the administrative programs 

providing the data. 

• An overall governance structure and management processes that agencies can buy into. 

• Assure that administrative agencies have a say in research priority setting and a benefit 

from participating. It should not be a one way street to just serve external entities or 

researcher needs. 

• Generally administrative entities are only funded to maintain case management systems 

and grantor reporting requirements. They need support in expanding business analyst 

functions to benefit from participating in evidence based policy making. 

Data Use in Program Design, Management, Research, Evaluation and Analysis 

15. What barriers currently exist for using survey and administrative data to support program 

management and/or evaluation activities? 

See questions 3, 4 and 5. 

 

16. How can data, statistics, results of research, and findings from evaluation, be best used to improve 

policies and programs? 

In general, what’s measured gets better is a sound operating principle. I think the best way to 

answer this question is to engage administrative entities as full partners, providing input into 

their research priorities. In principle, this effort should ultimately service the agencies providing 

the data and their customers. There is a government obligation and public interest to be served. 

17. To what extent can or should program and policy evaluation be addressed in program designs? 

All programs ought to, and I think generally do, try to evaluate their programs and how to 

improve. This effort merely provides additional insight and tools to do so. Additionally, it is 

The Promise of Evidence-Based Policymaking 1385
becoming very common practice for private, non-profit and public funders to require an 

evaluating component within their grants. 

18. How can or should program evaluation be incorporated into program designs? What specific 

examples demonstrate where evaluation has been successfully incorporated in program designs? 

I do not see this as anything new and generally see such an objective as a historical trend though 

a challenging one. We are just at a point where we can, if we choose, more readily apply data to 

the task, offer new tools for evaluation and be more transparent for informed choice by those 

we serve. 

19. To what extent should evaluations specifically with either experimental (sometimes referred to as 

‘‘randomized control trials’’) or quasi-experimental designs be institutionalized in programs? What 

specific examples demonstrate where such institutionalization has been successful and what best 

practices exist for doing so? 

Experimental designs are preferable where possible. However, they are more challenging with 

human subject research, particularly when applied to public services. There is risk of being 

challenged for disparate treatment. Quasi-experimental design are probably the norm moving 

forward. The SLDS and WDQI grants should be reviewed for examples. 

1386 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t17-v37j

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0130

Comment on FR Doc # 2016-22002

Submitter Information

Name: Jose Luis Santos

Address:

Washington, DC, 20005

Email: tjones@edturst.org

General Comment

Attached, please find comments from the Education Trust

Attachments

Written Statement for the Commission on Evidence Based Policymaking

The Promise of Evidence-Based Policymaking 1387
Closing the gaps in opportunity and 

achievement, pre-k through college. 

1250 H STREET N.W., SUITE 700 WASHINGTON, D.C. 20005 T 202/293-1217 F 202/293-2605 

WWW.EDTRUST.ORG 

ŚĂŝƌƉĞƌƐŽŶ<ĂƚŚĂƌŝŶĞďƌĂŚĂŵ͕hŶŝǀĞƌƐŝƚLJŽĨDĂƌLJůĂŶĚ

ŽͲŚĂŝƌZŽŶ,ĂƐŬŝŶƐ͕ƌŽŽŬŝŶŐƐ/ŶƐƚŝƚƵƚŝŽŶ

ŽŵŵŝƐƐŝŽŶŽŶǀŝĚĞŶĐĞͲĂƐĞĚWŽůŝĐLJŵĂŬŝŶŐ

ŽĐŬĞƚ/h^ͲϮϬϭςͲϬϬϬϯ

Dear Chairperson Abraham, Co-Chair Haskins, and Commissioners: 

On behalf of The Education Trust — an organization dedicated to closing long-standing gaps in 

opportunity and achievement separating low-income students and students of color from their 

peers — thank you for the opportunity to provide comments in response to the Commission on 

Evidence-Based Policymaking’s request for comments. 

The information below addresses question one in the request for comments in the Federal 

Register: “Are there successful frameworks, policies, practices, and methods to overcome 

challenges related to evidence-building from state, local, and/or international governments the 

Commission should consider when developing findings and recommendations regarding Federal 

evidence-based policymaking? If so, please describe.” 

Since the original Higher Education Act (HEA) was passed in 1965, the United States has made 

substantial progress in college access. College-going rates have climbed for students from all 

economic and racial groups. Yet, despite this progress, low-income students today enroll in 

postsecondary education at rates lower than high-income students did in the mid-1970s. In every 

category of postsecondary education, low-income students and students of color are less likely 

than others to earn the degrees that they want and need, and far more likely to end up with debt 

and no degree. 

Past experience is clear: Tackling big problems in education requires good information. For 

example, before disaggregation of data was required in K-12, we knew anecdotally that schools 

were not educating all groups of students well. But we did not know just how significant the 

inequities were, and we didn’t know which schools were making progress and which weren’t. 

That, unfortunately, is where we still are in higher education — especially in regard to lowincome students. We have some limited research on, for example, overall Pell graduation rates, 

but we don’t know which institutions are serving these students well and which aren’t: Pell data 

aren’t included in annual Integrated Postsecondary Education Data System (IPEDS) data 

collections. IPEDS also doesn’t include data on part-time students or students who don’t start in 

the fall or students who transfer in from another college. 

1388 The Promise of Evidence-Based Policymaking
THE EDUCATION TRUST 

1250 H STREET N.W., SUITE 700 WASHINGTON, D.C. 20005 T 202/293-1217 F 202/293-2605 

WW.EDTRUST.ORG 

Yet if we have learned anything from past experience, it is this: Students who aren’t measured 

don’t count. If we want these students to count, we must make the same shift we have made in 

K-12 to demanding better data. 

The availability of high-quality, robust data systems is essential to helping the public understand 

how students are faring at particular institutions, identifying gaps by income and race, and better 

identifying and incentivizing improvement and success. Having better information on the college 

participation and outcomes of all students also helps ensure that the billions of dollars that the 

federal government annually invests in higher education is being used in the most effective ways. 

There are a range of ways to do this — from an expanded IPEDS collection to a fuller unit 

record system. Both are viable. What is important is not how the data are collected, but that they 

be collected and publicly reported. Therefore, we propose two options to ensure we are counting 

all students, rather than just some. 

Option One 

The first and most comprehensive approach is to lift the student unit record ban, implementing 

a stripped down version of the data system that most states have already put into place. While a 

federal version of such a system doesn’t have to have all of the information collected in statelevel systems, such an approach would allow for the basic tracking functions necessary to 

monitor student progress from one institution to the next, and also link to employment and wage 

data. 

This approach is the least burdensome for colleges and universities, relieving them of many of 

the reporting requirements in the current HEA. And it will produce the most comprehensive data 

for policymakers and students, as well as inform institutional improvement. The ban on a federal 

student unit record system makes it impossible for federal policymakers to get a comprehensive 

picture of how students are moving through postsecondary education and attaining degrees and 

certificates. 

The commission should recommend the overturn of the HEA ban and the ban on a federal 

database of Workforce Innovation and Opportunity Act (WIOA) data, so that we can have 

a nationwide, inclusive data set to show how people are moving through a variety of 

education pathways. 

Creating a student unit record system can begin by leveraging existing resources from the U.S. 

Department of Education (which houses the National Student Loan Data System (NSLDS) and 

IPEDS), the Social Security Administration (SSA), the Department of Defense (DoD), and the 

Department of Veterans Affairs (VA), among others, to create a more complete picture of the 

higher education landscape. These sources provide valuable data on important groups of students 

The Promise of Evidence-Based Policymaking 1389
THE EDUCATION TRUST 

1250 H STREET N.W., SUITE 700 WASHINGTON, D.C. 20005 T 202/293-1217 F 202/293-2605 

WW.EDTRUST.ORG 

who are often overlooked, including Pell Grant recipients, student loan borrowers, and student 

veterans. If linked, these data would produce valuable information about enrollment and 

completion rates, and post-college employment and earnings. 

Option Two 

At the same time, because we are not naïve about the concerns that exist around a unit record 

system, we also actively support other possibilities for improving data. A second option is to 

expand and improve the current data collections through IPEDS, including by adding data on 

part-time students and transfers, and by making Pell status transparent in the collections. 

This approach creates additional burdens for institutions, but our own experience with a 

voluntary data collection effort among the 25 public university systems that participated in our 

Access to Success Initiative suggests that the additional burden is not insurmountable, especially 

given how much more useful the resulting data are. 

The data collections for two-year colleges should also be revised to disaggregate outcomes that 

are now combined. Transfers from one community college to another, for example, are currently 

treated no differently than transfers from a community college to a university. 

We support efforts to improve IPEDS, including supporting the Department of Education’s 

decisions to create Pell subgroup cohorts within the Outcome Measures. These and other 

improvements to IPEDS can greatly improve the quality of the data and provide invaluable 

information about the success of the nation’s most vulnerable students. 

We are hopeful that the activities of the Commission will help us take an important and critical 

step toward advancing the quality and availability of higher education data — specifically, a step 

that ensures better data that can be used to empower students, families, the public, advocates, and 

campuses as we aim to increase postsecondary success for all students, especially our most 

vulnerable. 

Cordially, 

José Luis Santos 

Vice President of Higher Education Policy and Practice 

1390 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t17-n0ha

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0131

Comment on FR Doc # 2016-22002

Submitter Information

Name: Stephen Wandner

Address:

Bethesda, MD, 20817

Email: stephen.wandner@gmail.com

General Comment

See attached file(s)

Attachments

Commission for Evidence Based Policymaking Comments_20161114

The Promise of Evidence-Based Policymaking 1391
1 

Commission for Evidence-Based Policymaking: Comments 

Docket Number 160907825-6825-01 

Docket ID USBC-2016-0003 

Stephen A. Wandner, PhD 

President, Wandner Associates Inc 

301-785-6670 

stephen.wandner@gmail.com

Visiting Fellow, The Urban Institute 

Visiting Scholar, W.E. Upjohn Institute for Employment Research 

The comments and opinions below are mine alone and do not represent those of either the 

Urban Institute or the W.E. Upjohn Institute for Employment Research. They are based on my 

years directing unemployment insurance and employment and training research at the U.S. 

Department of Labor (USDOL) and my experience as an economic consultant since 2010. My 

comments deal largely with the Overarching Questions 1 and 2. 

Comments 

My comments cover the following areas: 1) scope of evidenced-based policymaking, 2) 

improving state data and research systems, 3) interstate data, and 4) regional and national data 

repositories. 

Scope of Evidenced-Based Policymaking 

Rigorous demonstration projects or evaluations of existing programs are critical at the 

beginning of the process of evidence-based policymaking. But rigorous evaluations are not 

enough. Such evaluations may or may not result in policy changes that use the lessons learned 

from the evaluation. Even if the lessons learned from evaluations are implemented, the 

process of implementing new and more rigorous evidenced-based processes should be 

continual if success is to be accomplished over the long run. 

Evidence-based policymaking should be thought of as a continuous process. In fact, it is a 

process that should never end. In my book, Solving the Reemployment Puzzle: From Research to 

Policy (2010), I illustrate this process of re-evaluation for two legislative initiatives – Worker 

Profiling and Reemployment Services (WPRS) and Self-Employment Assistance (SEA) – in 

chapters 2, 3, 4 and 8. 

In each case the adoption and operation of a proven approach had multiple steps. There was 

progression: 

• from demonstration design and implementation 

1392 The Promise of Evidence-Based Policymaking
2 

• to demonstration evaluation 

• to policy development 

• to legislative development and enactment 

• to program implementation with fidelity to the demonstration design 

• to program evaluation. 

In both cases, the design, implementation, and evaluations were rigorous, using random 

control trial (RCT) methods. But equally important was the fidelity of transferring the tested 

design to policy development, legislative enactment, and program implementation. Finally, an 

evaluation examined how the operational program results compared to the design and 

outcomes obtained in the experiment. 

Even after implementing a successful model and evaluating it, the process does not end. 

Keeping fidelity of the tested and evaluated model takes work by state and local practitioners 

as well as oversight by the federal government. In the case of WPRS, successful 

implementation depended on updating a Worker Profiling regression model that was 

developed by each state during the implementation period. Many states, however, failed to 

update their models, and the U.S. Department of Labor eliminated the staff technical assistance 

to states to update their models. 

 Successful implementation also depended on the commitment of USDOL staff who developed 

careful plans regarding how to operationalize the program across three USDOL components: 

Unemployment Insurance, the Employment Service, and the Workforce Investment Act 

programs. 

Finally, successful implementation depended on Congress appropriating sufficient funds to 

implement a comprehensive program of job search assistance (reemployment services). 

Unfortunately, resources to provide reemployment services were limited, and substantial 

federal funding was only briefly available through the American Recovery and Reinvestment Act 

of 2009. 

Improving State Data and Research Systems 

Federal Funding of State Longitudinal Data Systems 

The USDOL Workforce Data Quality Initiative (WDQI) and Department of Education’s State 

Longitudinal Data System (SLDS) funding have been very helpful in creating and improving state 

longitudinal data system. However, some of the money has been wasted on states that have 

not been able to build and use such systems. Especially on the workforce side, funding these 

systems should be continued, but funding should be targeted to states with the technical 

capacity and political will to use them, as well as to those states that have already successfully 

built, maintained and utilized the data. 

The Promise of Evidence-Based Policymaking 1393
3 

Most states that have built effective systems have done so working with major state research 

universities. Building useful data systems will likely be achieved by states with state research 

universities. However, many states don’t have such universities or the staff that is willing and 

able to support such research efforts. As a result, I recommend that the federal government 

competitively fund regional Workforce Data Research Centers that can serve multiple states. 

These regional centers are likely to be located in states that already have major state research 

universities. (See my recommendation below about regional repositories.) 

State Models1

Administrative Data Research and Evaluation (ADARE) 

A number of states established longitudinal data systems on their own. They started these 

systems in a number of different ways, but all these systems depend on having state workforce 

agencies and researchers getting together to make use of the data for performance 

measurement, research and evaluation purposes. The efforts that each of the ADARE states 

made to successfully develop their longitudinal administrative data systems is documented in 

Stevens (2004). In the 1990s, the USDOL began funding studies in nine states (California, 

Florida, Georgia, Illinois, Maryland, Missouri, Ohio, Texas, and Washington) of interest to both 

USDOL and the states. The federal ADARE effort was an attempt to take advantage of state 

data and research capacity and to harness it for research and evaluation efforts of mutual 

interest. The output of the ADARE effort was a wide variety of studies, including a quasiexperimental evaluation of the Workforce Investment Act. Other studies brought together data 

from other agencies. One studied Temporary Assistance for Needy Families (TANF) in six states 

(King and Mueser 2005). The U.S. Department of Agriculture funded a six-state study of the 

relationship between the Supplemental Nutrition Assistance Program (SNAP) and 

Unemployment Insurance during the Great Recession, revealing how the SNAP and UI programs 

have worked separately and interacted. For this study, six ADARE states merged SNAP data 

with their UI wage record and benefit data (O’Leary et al. forthcoming). 

Ohio Longitudinal Data Archive (OLDA) 

Ohio used its WDQI funds received in 2010 to develop and establish the Ohio Longitudinal Data 

Archive (OLDA). This data archive is located at the Center for Human Resource Research at 

Ohio State University. The data in the archive includes data from four state agencies dealing 

with: workforce and social services, K-12 education, higher education, and vocational 

rehabilitation. (Additional Ohio state agencies are expected to join OLDA.) The data archive is 

subject to the same security and confidentiality provisions that the Center for Human Resource 

 

1

 USDOL has funded a “State Research Capacity” study because, under the new Workforce Innovation and 

Opportunity Act of 2014, states are responsible for increased research and evaluation. That study is ongoing. It 

consists of a survey of all states regarding their research capacity. It also includes two case studies on Ohio and 

Washington state because of their exemplary research efforts. 

1394 The Promise of Evidence-Based Policymaking
4 

Research uses for the National Longitudinal Survey that it operates for the Bureau of Labor 

Statistics. 

Federal funding for OLDA was exhausted in early 2016. The four participating agencies are now 

each contributing $200,000 to fund the archive that has a total cost of $800,000 per year. For 

their contribution, each agency has its data cleaned and entered into OLDA, and they can 

request staff from the Center for Human Resource Research to conduct a moderate level of 

research and analysis for them at no additional cost. More extensive research must be funded 

by the participating state agencies. 

OLDA is available for use by the participating state agencies, Ohio State University, other Ohio 

and out-of-state universities, and private researchers. It has been used for 154 projects in the 

past three years. OLDA encourages use by all bona fide researchers and research institutions. 

Research has to meet criteria set by OLDA and the Ohio state agencies. Researchers must 

adhere to strict confidentially provisions and agree to destroy the merged, de-identified data 

which they receive at the end of their projects. 

Two examples of approved university and private research projects by OLDA are as follows: 

Alex Mas (Princeton), Pauline Leung (Cornell), and Zhuan Pei (Cornell) were approved to 

conduct a project about education and re-training experienced by unemployed workers and 

their employment and earnings outcomes (“Further Education During Unemployment”). A 

second project by Scott Davis (IMPAQ International), Louis Jacobson and myself (consultants to 

IMPAQ) was approved for a “Scorecard II” project that compares training performance 

measures using Ohio state data and the National Directory of New Hires (NDNH), with 

particular emphasis on the difference between the data sets and on the effect of adding 

interstate employment and earnings outcomes from NDNH. The latter project is funded by 

USDOL. 

Washington Education Research and Data Center (ERDC) 

Washington has a different state model for longitudinal administrative data systems and 

research and evaluation than is used in Ohio. It maintains its own data system and both 

conducts its own research and contracts for outside research. 

Washington state has a long history of developing and using longitudinal administrative data. 

In the late 1970s, it developed a longitudinal administrative data system using mostly 

unemployment insurance data, during the time when the U.S. Department of Labor funded a 

Continuous Wage and Benefit History. They maintained and expanded that system with state 

funds after USDOL abandoned the federal program. Then they used federal funds from State 

Longitudinal Data Systems (SLDS) and WDQI to build a greatly enhanced education and 

workforce system using SLDS and WDQI funds. State agencies and other researchers have wide 

access to data from the Washington state Education Research and Data Center. 

Interstate Wage Data 

The Promise of Evidence-Based Policymaking 1395
5 

Federal agencies, state agencies, university researchers and private researchers all need access 

to nationwide UI wage records, since they are the main source of data for determining 

employment and earnings outcomes. Employment and earnings are key outcomes for many 

programs, and yet there is no way of following individuals as they move around the country. 

The NDNH is not the answer to providing UI wage records for research and evaluation. It is 

limited in its scope, most importantly because it contains only two years of UI wage record 

data. More importantly, the NDNH data is held by a program agency (i.e., the Department of 

Health and Human Services (HHS)) and is not readily available for research and evaluation 

purposes. One research project that I have been working on to develop training and education 

scorecards using state and NDNH data has faltered because of many delays in obtaining 

approval of data matching requests and then having the data matched by HHS. After two years 

of effort, the data matching has still not occurred. 

National or Regional Repositories 

A national repository makes sense for UI wage record data, but not for a much more complex 

longitudinal administrative data system that includes data from many programs, such as Ohio’s 

OLDA. One way of achieving a national UI wage record repository would be to amend federal 

UI law to make the exchange and storage of UI wage records a federal responsibility. In 1935, 

when the UI program was created as part of the Social Security Act, it may have made sense to 

leave interstate wage record exchange and data use for performance and research purposes to 

the states. It certainly does not make sense now, given voluntary state participation in the 

system and lack of support for using the data for research and evaluation purposes. At present, 

the system is overseen by the state workforce agencies through a committee of the National 

Association of Workforce Agencies. 

The interstate UI wage record system should be made mandatory, transferred to USDOL, and 

USDOL should be responsible for data exchange for UI benefit claims and performance 

measurement purposes. USDOL also should establish a national UI wage record repository for 

research, public policy and program management purposes. 

By contrast, longitudinal administrative data systems that include data from multiple state 

agencies are far more complex, and they cannot be part of a national repository. For these 

multi-program longitudinal administrative data systems, a series of regional data and research 

centers should be established. For these systems to be effective, state cooperation is needed, 

and gaining that cooperation would require that that each state get feedback in the form of 

research and analysis, making use of their state data by the repository organizations. States 

should regularly transfer high quality data to a regional center, work with the regional center to 

ensure that erroneous data is corrected, and have the incentive of access to the data and 

receipt of analysis of that data that can be used for individual state program improvement and 

program assessment and evaluation. That process cannot be carried out through a national 

data center, because a single national repository organization could likely not work effectively 

1396 The Promise of Evidence-Based Policymaking
6 

with 50 states. It would require establishing approximately 6 to 8 regional data and research 

centers that each can work with 6 to 8 states to improve their data and conduct research and 

analysis for each state as a quid pro quo for providing their state data. 

Since the OLDA, the Ohio data system, costs $800,000 per year, a regional repository for a wide 

range of state programs could cost as much as $40 million yearly, although there likely would 

be economies of scale. 

Approaches to Evidence-Based Policy: Geographic Alternatives 

Evidence-based policy making should be encouraged at the local, state, regional and national 

levels. Let us look at the workforce system as an example. Local models could be developed 

by creating a small number of model local workforce offices that could test alternative program 

and policy options or smaller nudges to improve the workforce system. As we saw above, some 

states already are doing good work in implementing evidence-based policymaking, such as Ohio 

and Washington, and these state efforts should be encouraged. Regional efforts could be 

accomplished by states coming together, either voluntarily on their own or with federal 

assistance, to create multi-state data centers. Finally, a national system should be developed 

by the federal government, although practically I believe it would have to be limited to creating 

a single UI wage record data center and not a complex multi-program data system. 

References 

King, Christopher T., and Peter R. Mueser. 2005. Welfare and Work: Experiences in Six Cities. 

Kalamazoo, MI: W. E. Upjohn Institute for Employment Research. 

Office of Management and Budget. “Commission on Evidence Based Policymaking.” 

https://www.whitehouse.gov/omb/management/commission_evidence 

O’Leary, Christopher, David Stevens, Stephen Wandner, and Michael Wiseman, eds. 

Forthcoming. Two Strands of the Safety Net: Unemployment Insurance, Supplemental Nutrition 

Assistance, and the Great Recession. Kalamazoo, MI: W.E. Upjohn Institute for Employment 

Research. 

Stevens, David. 2014. “Responsible Use of Administrative Data Records for Performance 

Accountability: Features of Successful Partnerships.” Baltimore, MD: University of Baltimore 

(April) pp. 28-30. http://www.jacob-france-institute.org/documents/dol_guide_rev_draft_1-5-

03.pdf

Wandner, Stephen A. 2010. Solving the Reemployment Puzzle: From Research to Policy. 

Kalamazoo, MI: W.E. Upjohn Institute for Employment Research. 

The Promise of Evidence-Based Policymaking 1397
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t17-28un

Comments Due: December 14, 2016

Submission Type: API

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0132

Comment on FR Doc # 2016-22002

Submitter Information

Name: NASWA Anonymous

Address:

Washington, DC, 20001

Email: ychocolaad@naswa.org

General Comment

The attached document is submitted on behalf of the National Association of State

Workforce Agencies (NASWA).

NASWA is a national, nonpartisan organization representing workforce agencies in 50

states, DC and Guam as they administer the publicly-funded state workforce system,

including the Workforce Innovation and Opportunity Act (WIOA) programs that fund

job training and career services, the Unemployment Insurance program, and labor market

and workforce information. 

Thank you for this opportunity to provide input on the issues the Commission laid out in

FR Doc. 2016-22002. 

1398 The Promise of Evidence-Based Policymaking
Attachments

CEBP FINAL

The Promise of Evidence-Based Policymaking 1399
444 N. Capitol St. NW, Suite 142, Washington, DC 20001 | (202) 434-8020 

To Members of the Commission on Evidence-Based Policymaking: 

On behalf of the National Association of State Workforce Agencies (NASWA), thank you for this 

opportunity to provide input on the issues the Commission laid out in FR Doc. 2016-22002 that will 

inform the future deliberations of the Commission. 

NASWA is a national, nonpartisan organization representing workforce agencies in 50 states, DC and 

Guam as they administer the publicly-funded state workforce system, including the Workforce 

Innovation and Opportunity Act (WIOA) programs that fund job training and career services, the 

Unemployment Insurance program, and labor market and workforce information. 

State workforce agencies have a longstanding interest in and history of using data to operate 

performance accountability systems, conduct research and evaluations, and otherwise use workforce 

information to improve workforce development policies, programs and strategies. State workforce 

programs are among the few federally-funded grant programs with a history of using administrative 

data sets to implement performance accountability systems, and state workforce agencies have an 

over-25-year history of involvement in rigorous research and evaluations focused on reemployment 

services, job search assistance, and training.i

The new WIOA law places greater emphasis on integrating state administrative data across a range of 

workforce, education and human services programs, as well as implementing common performance 

metrics and research and evaluations for many of the programs. Given the large real-term cuts in 

funding for workforce development programs and UI administration in recent years (see Appendix A), 

these are both more formidable yet even more important challenges. 

NASWA members and staff are available to share expertise and provide up-to-date information on 

challenges and proven practices related to evidence-building. Since WIOA passed, NASWA has 

hosted numerous national and committee meetings, often including our education and human services 

partners, in order to inform policymakers on WIOA implementation issues and practices. To develop 

the attached comments, we drew on the insights of NASWA members at these meetings and at a 

specially-called November teleconference that focused on the Commission’s questions. Over 40 

members of four NASWA Committees (Employment & Training, Labor Market Information, 

Unemployment Insurance, and Technology) participated in the November teleconference. 

Sincerely, 

Scott B. Sanders 

Executive Director 

1400 The Promise of Evidence-Based Policymaking
COMMENTS FOR THE COMMISSION ON EVIDENCE-BASED POLICYMAKING 

COMMISSION QUESTION #1: Are there successful frameworks, policies, practices, and 

methods to overcome challenges related to evidence-building from state, local, and/or 

international governments the Commission should consider when developing findings and 

recommendations regarding Federal evidence-based policymaking? If so, please describe. 

NASWA members report that resource limitations related to data access, funding, IT, and/or 

staff capacity impede or have even stalled research and evaluation activities in a large number 

of state workforce agencies. In fact, in commenting on proposed WIOA regulations, the 

National Governors Association joined NASWA in suggesting that states will need dedicated 

funding and federal support to meet the evidence-building requirements of WIOA. There are 

some states, however, such as Ohio and Washington, that have made great advances in 

developing an evidence-building infrastructure, and they provide good examples of 

frameworks, policies, practices and methods. They have not only developed longitudinal 

administrative data sets, but also the infrastructure to convert and analyze the data to inform 

policy and customer choices. Appendix B provides information on the Ohio case. 

Other examples of successful approaches to developing data infrastructure include the 

numerous other state longitudinal administrative data sets funded, in part, under the 

Workforce Data Quality Initiative (WDQI) and State Longitudinal Data Systems (SLDS) grants. A 

good cross-state effort is the Western Interstate Commission for Higher Education’s (WICHE) 

multi-state longitudinal data exchange (MLDE). This latter effort is important for showcasing 

the value of tracking individuals across state lines to understand the impact of our education, 

training and other investments. Many states are engaged in cross-state efforts, including 

Wyoming, which has agreements with twelve states and has used cross-state data to measure 

community college outcomes and understand migration of young people.ii

NASWA would like to make the Commission aware that, under a recent USDOL WIOA technical 

assistance grant, NASWA is documenting successful approaches to evidence building among 

state workforce agencies, as well as the challenges many state workforce agencies are facing. 

NASWA will share the final report with the Commission when it is released by USDOL (most 

likely this winter). The USDOL-NASWA project is: 

The Promise of Evidence-Based Policymaking 1401
• Capturing information, through a national scan, on the current capacity of 

state workforce agencies to conduct research and evaluationsiii; 

• Developing a reference document of research studies and evaluations 

conducted over calendar years 2011-2015; and 

• Developing two case examples, based on in-depth, structured interviews in 

Ohio and Washington, which will illuminate factors and practices that have 

enabled Ohio and Washington to produce a broad range of recent workforce 

research and evaluation activity. 

COMMISSION Question #2: Based on identified best practices and existing examples, 

what factors should be considered in reasonably ensuring the security and privacy of 

administrative and survey data? 

Some examples of the many factors states have considered are: (1) a formal approval 

process and development of legal contracts for data users; (2) restricted data use 

(there must be a specific project plan); (3) role differentiation to ensure duplication is 

built into the security framework; (4) a timeout feature; (5) two-factor authentication; 

(6) data aggregation rules to ensure confidentiality; (7) background checks for outside 

data users; (8) IRB (institutional review board) training and approval for outside users; 

(9) periodic security and practices audits to ensure outside review; and (10) database 

development issues (using a synthetic database to meet certain research needs and/or 

a more federated infrastructure for sharing data). 

Challenges have included co-mingling of the data, which requires addressing questions 

of ownership, liability, and additional security measures. Some states, such as Ohio, 

have addressed these challenges at the state level, when co-mingling data across 

multiple state agencies (six agencies, in the case of Ohio). Often, entities should 

maintain ownership of their data.

COMMISSION QUESTION #3: Based on identified best practices and existing examples, how 

should existing government data infrastructure be modified to best facilitate use of, and 

access to, administrative and survey data? 

The existing data infrastructure should be modified by: 

• Streamlining federal data initiatives, such as SLDS and WDQI, which may mean 

combining some initiatives that have related goals. Existing federal funding methods for 

investment in new insight-generating infrastructure such as longitudinal data systems 

1402 The Promise of Evidence-Based Policymaking
are unpredictable, uncoordinated, and tend to promote overlapping and uncoordinated 

projects. If the 21st century is about lifelong learning with people moving back and forth 

between education and employment, or participating in employment and education 

simultaneously, we need to develop pre-K to career longitudinal systems, not separate 

education and workforce program systems. 

• Providing more predictable, coordinated and scalable funding support for the 

development of state longitudinal administrative data systems, and the development of 

electronic tools that deliver information in customer-helpful formats. 

• Supporting government off-the-shelf (GOTS) projects and products after initial 

investments are exhausted. For GOTS projects to be successful, continued support and 

sustaining funds are needed. 

• Supporting the newly-established, federally-funded WIOA Information Technology 

Support Center. The Center will be run by NASWA to develop IT solutions and training 

that support state workforce agencies and the workforce system broadly in 

implementing the WIOA vision of cross-agency integrated data and technology solutions 

that support integrated service delivery, case management, performance measurement, 

and research and evaluation. 

• Investing in and continuously improving labor market information so that high-quality 

data relevant for the current economy is available to guide policymakers and help 

customers make career, education and training decisions. Episodic, competitive grants 

cannot provide the solid foundation needed for this key piece of data infrastructure. For 

example, the last survey regarding the contingent workforce was conducted by the 

Bureau of Labor Statistics in February 2005 and estimates of the size of this population 

today vary widely as a result. 

• Ensuring state agencies have access to the funding, skilled staff and tools needed to 

analyze labor market and workforce data and to conduct research and evaluations, so 

data can be translated efficiently into useful information for policy makers and 

customers. State workforce agencies should not be wholly or largely dependent on 

arrangements by which private entities take the data and develop information, tools 

and products they sell back to government agencies, especially where it is more efficient 

to produce information in-house and/or to create tools and solutions that can be shared 

across a number of states. 

The Promise of Evidence-Based Policymaking 1403
• Improving state workforce agency access to national UI wage record data that can be 

used for research and evaluation. Many state workforce agencies do not have the staff 

capacity and resources necessary to broker and maintain bilateral or multi-state 

agreements to share UI wage record data for research and evaluation purposes, and 

developing and maintaining such state-to-state agreements is an extremely inefficient 

solution. Also, some state workforce programs cannot receive their own state’s UI wage 

record data for research and evaluate purposes, due to legal or other factors (e.g., state 

laws, risk aversion). 

• Providing state UI agencies dedicated funding to support state wage record data 

exchange and related matters.

• Considering the impact on employers to any changes made in how UI wage records are 

collected and what elements are included.iv Some states have adopted enhancements. 

This includes understanding the environment within which employers maintain and 

report employment-related information, and balancing the additional cost against the 

value of additional employer data to find the least-burdensome solution. 

• Supporting a technological solutions repository similar to the recently launched 

code.gov website where open-source and federally-funded projects can be tracked and 

shared across state programs. 

• Supporting promising state or state-federal pilot projects that increase data access or 

improve data analysis and research.

COMMISSION QUESTION #4: What data sharing infrastructure should be used to facilitate 

data merging, linking, and access for research, evaluation and analysis purposes? 

Any Commission recommendations should not place additional burdens on state administrative 

programs but take data as it is available from the programs. States should not have to modify 

the data systems of their administrative programs in order to share data for research, 

evaluation and analysis purposes. 

COMMISSION QUESTION #5: What challenges currently exist linking state and local data to 

federal data? Are there successful instances where these challenges have been addressed? 

WIOA requires common performance measures across several workforce development and 

partner programs. These measures are of sufficient value that applying them and their 

1404 The Promise of Evidence-Based Policymaking
reporting concepts more universally would improve service delivery, as well as data sharing and 

program evaluations. The WIOA measures are now required under the SNAP employment and 

training program. However, the Food and Nutrition Service is implementing slightly different 

measure concepts (e.g., definitions of participant and program exit) which makes integration of 

services, and data comparability and data sharing, more difficult. Further, while TANF is a 

mandatory WIOA partner program (barring a governor opting out), the new WIOA measures 

and accountability constructs do not apply to TANF, which again makes integration of services 

and data comparability and sharing more difficult. 

Administrative data is usually organized for case management and to meet federal or state 

reporting requirements. As such, it seldom can be used in raw form to meet research 

objectives. Even if common definitions are achieved, additional challenges include reorganizing 

and transposing administrative data, and otherwise structuring it in a linked longitudinal 

fashion so it is documented and cast into a metadata file. This file must be separately 

maintained because administrative programs go through changes in data structure, content 

and definitions. 

Another challenge is that efforts to create state longitudinal administrative data sets are often 

limited by the application of the Family Educational Rights and Privacy Act (FERPA). Student 

education records are important to developing a complete pre-K to career longitudinal data. 

Also, state workforce agencies need access to these student data to meet WIOA performance 

reporting requirements to track credential attainment, youth outcomes, and “measurable skills 

gains.” 

Finally, also creating challenges is government spending on varied and sometimes duplicative IT 

solutions across federal, state and local workforce, education, and human services agencies. 

COMMISSION QUESTION #6: Should a single or multiple clearinghouse(s) for administrative 

and survey data be established to improve evidence-based policymaking? 

In general, state workforce agencies are not supportive of a single Federal clearinghouse, and 

support a more federated system with communications pathways at the national level to 

facilitate data sharing. 

 

The Promise of Evidence-Based Policymaking 1405
COMMISSION QUESTION #7: What data should be included in a potential U.S. government 

data clearinghouse(s)? What are the current legal or administrative barriers to including such 

data in a clearinghouse or linking the data? 

Whether there is one or more than one clearinghouse, state workforce agencies need to be 

able to access state and national data to operate a performance accountability system and 

conduct program evaluations and research. At the present time, state workforce agencies are 

creating and sending data to federal and state partners, but not receiving enough information 

in return. 

Major decisions about data infrastructure, including which data should run through one or 

more national clearinghouses, should be made in partnership with the state agencies that 

control such data, so that the needs and interests of both the federal and state governments 

are taken into account. 

In most cases, states should not be coerced to provide data through national or regional 

clearinghouses by way of conditional grants or other means; rather, the governance structure, 

technology framework, quality of and ease of access to data, funding environment, and TA 

environment should provide the necessary incentives for governmental partners to participate. 

(See also Question #14 below.) 

In order to ensure data linkages, laws for sharing and protecting data must be 

reconciled. There are variances in federal and state laws at the source level (e.g., 

FERPA, UI wage records) that impede data linkages. Some states have access to certain 

datasets (NDNH, DMV records), while others do not, and restrictions on data use can 

vary.

COMMISSION QUESTION #8: What factors or strategies should the Commission consider for 

how a clearinghouse(s) should be self-funded? What successful examples exist for selffinancing related to similar purposes? 

The data and human infrastructure needed to build and maintain an evidence base for federal 

and state programs are public goods. NASWA questions the premise that data clearinghouses 

should be self-funded, except outside researchers and institutions should cover the marginal 

costs of their data acquisitions. 

1406 The Promise of Evidence-Based Policymaking
Government entities have a role and responsibility to maintain and be good stewards of 

program administrative data, and should receive adequate funding and support for the integrity 

of the system. 

COMMISSION QUESTION #9: What specific administrative or legal barriers currently exist for 

accessing survey and administrative data? 

Programs have been developed independently and in a siloed fashion, as have the legal 

requirements around them, so there are numerous legal barriers. Also creating barriers are the 

many levels of government involved in program administration and the varied policies and 

practices associated with these programs. 

COMMISSION QUESTION #10: How should the Commission define “qualified researchers and 

institutions?” To what extent should administrative and survey data held by government 

agencies be made available to “qualified researchers and institutions?” 

States should be considered partners to any clearinghouses, not qualified researchers and 

institutions. As governmental entities, state workforce agencies should have access to 

governmental data needed for federally required performance reports and to conduct research 

and evaluations. 

Governmental entities, especially WIOA partner agencies at the federal and state level, should 

have priority access to data infrastructure. 

The Commission could learn from state workforce agencies, such as Ohio’s, that provide data, 

selectively, to outside researchers and institutions. Given resource constraints, such uses could 

be limited to specific project requests that support the research agendas of federal and state 

agencies, as a first priority. 

QUESTION #11: How might integration of administrative and survey data in a clearinghouse 

affect the risk of unintentional or unauthorized access or release of personally identifiable 

information, confidential business information, or other identifiable records? How can 

identifiable information be best protected to insure the privacy and confidentiality of 

individual or business data in a clearinghouse? 

Facing high demand for government data with constrained resources, many entities are 

currently approaching privacy and confidentiality in an ad hoc way that involves more risk than 

having an organized approach with clear processes, safeguards and rules. 

The Promise of Evidence-Based Policymaking 1407
QUESTION #13: What technological solutions from government or the private sector are 

relevant for facilitating data sharing and management? 

See the answer to Question #1. 

COMMISSION QUESTION #14: What incentives may best facilitate interagency sharing of 

information to improve programmatic effectiveness and enhance data accuracy and 

comprehensiveness? 

The best incentives to facilitate interagency sharing of information are to: 

• Create efficient and streamlined data collection, confidentiality, and data access 

processes, with excellent privacy safeguards.

• Ensure that state agencies that collect, safeguard, clean, and share data are able to 

benefit from the data soon after it becomes available. 

• Provide state workforce agencies adequate funding, training, and technical assistance 

as they fulfill their responsibilities for collecting, safeguarding and sharing information.

• Provide state workforce agencies (and their state and federal partners) the critical 

funding and other support needed to translate data into information that is useful to 

customers – including policymakers, program managers, job seekers and employers. 

Otherwise, only the private sector will have the capacity to build an evidence base and 

develop customer tools and information, which is problematic in cases where it is 

more efficient to develop in-house and/or cross-state solutions. 

• Ensure that evidence is used to inform policy. For example, despite 25 years of 

evidence by federal and state partners that job search assistance and UI claimant 

reemployment services are high return-on-investment strategies, funding for the 

major programs supporting these strategies has declined significantly over the last 20 

years (see Appendix A). 

COMMISSION QUESTION #15: What barriers currently exist for using survey and 

administrative data to support program management and/or evaluation activities? 

To ensure the support of entities or individuals providing data, the business case needs to be 

made that research and evaluation activities help inform service delivery and the development 

of effective program strategies. The Commission can help develop the business case. 

1408 The Promise of Evidence-Based Policymaking
A large barrier is the inability of many states to hire and retain required expertise in research 

and evaluation. State workforce agencies have no dedicated funding for this activity. Such 

expertise is not covered by USDOL’s Workforce Information Grant funding. States can use 

statewide set-aside funds under certain WIOA grants, but these and other major sources of 

funding for workforce programs have experienced large real-term cuts over the last twenty 

years (see Appendix A).

Barriers also exist with respect to limitations on the use by states of certain data sources. For 

example, the IRS allows states to use 1099 data to evaluate possible worker misclassification 

from an unemployment tax perspective. However, such data cannot be used to evaluate 

misreported earnings from an unemployment integrity perspective. Given the flexibility that is 

being asked of states to share items like UI wage records for a broader scope of evaluation 

(outside of just UI program administration), the same expansion should be asked of federal 

agencies. 

The timeliness of data is also a constraint in using data to inform policy development, program 

management, and research. 

COMMISSION QUESTION #16: How can data, statistics, results of research, and findings from 

evaluation, be best used to improve policies and programs?

Federal, state, local, and even private sector money available for public services have not kept 

up with the expanding need. That means that we can keep doing the same things but do them 

for fewer people, or we can keep serving the same or more people but simply do less for them, 

or we can analyze our data and learn how to be more effective. Ideally this would not simply 

mean looking at data to see which services work and which don’t. Instead, we should be able to 

use data to better understand which services work best and for whom. It is likely that most 

services have some effectiveness but that effectiveness varies by population served and other 

factors, which calls for a customized, evidenced-based approach. 

COMMISSION QUESTION #17: To what extent can or should program and policy evaluation be 

addressed in program designs?

Government should evaluate program and service effectiveness and use findings to improve 

effectiveness and efficiency. That obligation is both to taxpayers in general and to those our 

services are intended to benefit. 

The Promise of Evidence-Based Policymaking 1409
COMMISSION QUESTION #18: How can or should program evaluation be incorporated into 

program designs? What specific examples demonstrate where evaluation has been 

successfully incorporated in program designs? 

Unemployment Insurance program “worker profiling models,” if timely and regularly updated, 

are an example of an approach that aids state evaluation of UI programs. These predictive 

models incorporate changes in the economy that impact who is likely to exhaust benefits. They 

enable state workforce agencies to directly engage those claimants most likely to have long 

unemployment durations in effective reemployment strategies early in their unemployment 

insurance spells. 

The limiting factor in the application of these worker profiling models has been funding to 

support reemployment services for those profiled, despite 25 years of rigorous evaluation and 

evidence that reemployment services help speed reemployment and reduce UI duration.v

COMMISSION QUESTION #19: To what extent should evaluations specifically with either 

experimental (sometimes referred to as “randomized control trials”) or quasi-experimental 

designs be institutionalized in programs? What specific examples demonstrate where such 

institutionalization has been successful and what best practices exist for doing so? 

The Commission should explicitly recognize there is a place for both experimental and quasiexperimental design in evaluation work. While experimental design may be the “gold standard” 

of evaluation, it is extremely costly and also not the most appropriate research design in many 

cases. Also, quasi-experimental design is more easily accepted by those who deliver services. 

State workforce agencies have used the results of quasi-experimental research to inform policy 

and practice, and should have the flexibility to use research designs appropriate to their needs. 

 

1410 The Promise of Evidence-Based Policymaking
Appendix A 

Funding for the Major Workforce Development Programs 

Under the U.S. Department of Labor 

(adjusted for inflation) 

FY 2001 

($2009)

FY 2016 ($2009) % Reduction 

Dislocated worker grant 1,712 1195 30 

Disadvantaged adult grant 1,134 749 34 

Youth grant 984 813 17 

Employment services 909 603 34 

The Promise of Evidence-Based Policymaking 1411
Appendix B 

1412 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 1413
1414 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 1415
1416 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 1417
1418 The Promise of Evidence-Based Policymaking
 

The Promise of Evidence-Based Policymaking 1419
i

 See, for example: Reemployment Unemployment Insurance Claimants: A Good Government Investment, Richard A. Hobbie and 

Yvette Chocolaad, in Transforming U.S. Workforce Development Policies for the 21st Century, Carl Van Horn, Tammy Edwards, 

and Todd Greene, Editors, Federal Reserve Bank of Atlanta, 2015; and Solving the Reemployment Puzzle: From Research to 

Policy, Stephen A. Wandner, Urban Institute, 2010. 

ii See: Crossing Boundaries: Regional Data Sharing to Study Worker Mobility, by Michelle Massie, WDQC, December 2014. 

iii Initial findings from the scan (41 state responses): 

o Two-thirds of state workforce agencies report that staff capacity (a concept that encompasses staffing 

levels, staff skills, and staff experience) for research and evaluation is nonexistent, inadequate or fair. 

o Looking across 15 specific research skills areas (such as using statistical methods, employing technical 

writing, analyzing large data bases, and conducting experiments), forty percent to 78 percent of state 

workforce agencies report they would like some or need more technical assistance or staff capacity. 

o Over half the state workforce agencies report funding is inadequate, and 56 percent of state workforce 

agencies with knowledge of funding trends report that funding is lower or much lower than in the past. 

iv Some states have made enhancements to their wage records. See: Enhancing Unemployment Insurance Wage Records: 

Potential Benefits, Barriers and Opportunities, Prepared for the Workforce Information Council by the Administrative Wage 

Record Enhancement Study Group, September 2015. 

v

 See footnote i.

 

1420 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t17-3fcg

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0133

Comment on FR Doc # 2016-22002

Submitter Information

Name: Christine Calpin

Address:

Washington, DC, 20036

Email: ccalpin@casey.org

General Comment

See attached file(s)

Attachments

Casey Family Programs Comments_Commission on EvidenceBased

Policymaking_FINAL.11.8.2016

The Promise of Evidence-Based Policymaking 1421
1422 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 1423
1424 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 1425
1426 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 1427
1428 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 1429
1430 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 1431
1432 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 1433
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t17-38vb

Comments Due: December 14, 2016

Submission Type: API

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0134

Comment on FR Doc # 2016-22002

Submitter Information

Name: Victoria Jones

Address:

Alexandria, VA, 22314

Email: vjones@nhsa.org

General Comment

See attached file(s)

Attachments

11.14.16 COMMENTS_Commission on Evidence-Based Policymaking_v2

1434 The Promise of Evidence-Based Policymaking
1651 Prince Street, Alexandria, VA 22314

Tel: (703) 739-0875 - Fax: (703) 739-0878 - www.nhsa.org

The National Head Start Association, an independent membership organization, advocates on behalf of the entire 

Head Start community and provides training and resources to Head programs nationwide.

November 14, 2016 

U.S. Department of Commerce 

Commission on Evidence-Based Policymaking 

 

RE: RESPONSE OF THE NATIONAL HEAD START ASSOCIATION TO THE 

REQUEST FOR COMMENTS REGARDING THE COMMISSION ON EVIDENCEBASED POLICY MAKING 

Thank you for providing us with the opportunity to submit our comments regarding the 

Commission on Evidence-Based Policymaking (Federal Register Volume 81, Number 178 

(Wednesday, September 14, 2016) Pages 63166-63168). 

The National Head Start Association is a non-partisan, not-for-profit organization that believes 

that every child, regardless of circumstances at birth, has the ability to succeed in life if given the 

opportunity that Head Start offers to children and their families. NHSA is that national voice for 

more than a million children in Head Start and Early Head Start programs in the United States. 

Below we offer responses to some of the questions posed in your request for comments. We 

appreciate the opportunity to contribute to the efforts of the Commission and your consideration 

for ensuring that all developed policies are sensitive to the needs of our nation’s children. 

 

(Re: Question 5) Current challenges in linking federal Head Start data with other local and 

state data systems include data software platforms that are not interoperable, data 

reporting requirements that are incongruent, and difficulty securing data sharing 

agreements. Currently local Head Start programs use a number of different software platforms 

to store and manage data, not all with the capability of connecting with state data software 

platforms. An example of a state that managed to address this is Georgia’s Department of Early 

Care and Learning which secured a license that allows for a direct upload from ChildPlus (the 

software vendor used by the majority of Head Start programs). However, securing a license can 

be costly and only works if there is common vendor. A potential solution would be for the state 

or federal government to create an open source software platform, such as Blockchain. 

Another major challenge results from incompatible reporting requirements. Head Start programs 

are required to collect substantial amounts of data around children and families and report them 

via the Public Information Report, which is sent to the Office of Head Start. However, often the 

The Promise of Evidence-Based Policymaking 1435
U.S. Department of Commerce 

Commission on Evidence-Based Policymaking 

Page 2 

1651 Prince Street, Alexandria, VA 22314

Tel: (703) 739-0875 - Fax: (703) 739-0878 - www.nhsa.org

The National Head Start Association, an independent membership organization, advocates on behalf of the entire 

Head Start community and provides training and resources to Head programs nationwide.

types of data and the timing of the collection are out of sync with state and local reporting 

requirements, which can lead to additional administrative burden on programs. 

In addition, there is also the substantial challenge of negotiating individual data sharing 

agreements. Having data-sharing agreements in place ensures that data will be stored and 

managed centrally and used appropriately. However, without having a centralized figure to 

create a template, states are often reluctant to enter into the process. North Carolina and Georgia 

have overcome this challenge by meaningfully involving all stakeholders early on in the process. 

These states now have nearly 100 percent participation of Head Start programs linking their 

federal program data with a state data system. Again, the use of Blockchain technology, which 

allows for more trust through enabling collaboration of decentralized data systems, should be 

given due consideration. 

(Re: Question 15) Current barriers for using survey and administrative data include 

establishing unique identifiers, standardizing data definitions, and balancing privacy and 

access. Unique identifiers are one of the largest roadblocks in using data in early childhood 

education. The resistance to and difficulty in assigning these identifiers often prevents the 

matching of data required for program management and evaluation. Even when data can be 

matched across organizations, there are often discrepancies in definitions that make it difficult to 

use. And finally, while regulations protecting information on children often make it difficult, if 

not impossible, to share data, it is also important to maintain students’ privacy. We encourage the 

commission to establish recommendations that reflect both the specific privacy and consent 

needs of young children, while also encouraging purposeful data sharing to improve services. 

(Re: Question 16) Data, statistics, research results, and evaluation findings can best improve 

programs through improvement science. Improvement science is a process wherein rapidcycle data collection and analysis is used to evaluate the implementation and impact of any 

programming and inform small adjustments to improve quality. This process allows for 

programs to clearly see successes and failures in implementation, adjust accordingly for the next 

iteration, and continue improving to get better service and outcomes. 

(Re: Question 19) Both experimental and quasi-experimental designs have valid and 

important roles in evidence-based policymaking. We have recently seen an over-emphasis on 

randomized control trials in the field of education. While RCTs are a valuable evaluation tool, 

they do not meet all the needs of the education field. We are particularly concerned about the 

slow turnaround of results from RCTs and whether they can be done practically and ethically 

with young children, particularly vulnerable children. While RCTs certainly have a place in 

policymaking, quasi-experimental methodologies have substantial value and should be a part of 

any foundation for evidence collection that is developed by the Commission. 

1436 The Promise of Evidence-Based Policymaking
U.S. Department of Commerce 

Commission on Evidence-Based Policymaking 

Page 3 

1651 Prince Street, Alexandria, VA 22314

Tel: (703) 739-0875 - Fax: (703) 739-0878 - www.nhsa.org

The National Head Start Association, an independent membership organization, advocates on behalf of the entire 

Head Start community and provides training and resources to Head programs nationwide.

It is our hope that you will develop recommendations that support the goals of the Head Start 

community: to improve the early environment and educational opportunities of children in 

poverty, while also protecting and supporting the nation’s most vulnerable children. 

Thank you again for the opportunity to provide comments. We look forward to seeing the 

recommendations of the Commission. 

Sincerely, 

Yasmina Vinci 

Executive Director 

The Promise of Evidence-Based Policymaking 1437
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t18-6d6v

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0135

Comment on FR Doc # 2016-22002

Submitter Information

Name: James Scanlan

Address:

Washington, DC, 20007

Email: jps@jpscanlan.com

General Comment

Attached are comments for the Commission on Evidence-Based Policymaking, pursuant

to request dated September 8, 2016, Docket ID USBC-2016-0003, Question 16.

James P. Scanlan

Attorney at Law

1529 Wisconsin Ave., NW

Washington, DC 20007

jps@jpscanlan.com

(202) 338-9224

Attachments

Comments of J Scanlan for Comm on Evidence-Based Policymaking (Nov. 14, 2016)

1438 The Promise of Evidence-Based Policymaking
COMMISSION ON EVIDENCE-BASED POLICYMAKING COMMENTS, 

DOCKET ID USBC–2016–0003, QUESTION 16 

 Submitted by: 

 James P. Scanlan 

 Attorney at Law 

 1529 Wisconsin Ave., NW 

 Washington, DC 20007 

 jps@jpscanlan.com 

 (202) 338-9224 

The Promise of Evidence-Based Policymaking 1439
i 

Table of Contents 

Introduction ......................................................................................... 1 

I. The Unsoundness of Analyses of Demographic Differences That Fail to 

Consider the Effects of the Prevalence of an Outcome on the Measure Employed ....... 8 

 A. Relative Differences in Rates of Experiencing Favorable and 

 Adverse Outcomes ........................................................................ 14 

 

 B. Absolute Differences and the Odds Ratios ......................................... 29 

 C. The Appraisal of Demographic Differences in Terms of the Proportion a 

 Group Comprises of Persons Potentially Experiencing an Outcome and the 

 Proportion it Comprises of Persons Actually Experiencing the Outcome ......... 39 

 

 D. Subgroup Effects Issues .............................................................. 41 

II. The Fundamental Unsoundness of Analyses of Discrimination Issues That 

Examine Data Solely on Persons Who Accepted Some Outcome or Situation ............. 43 

III. Recommendations ............................................................................. 45 

Table 1. Illustration of effects on relative differences in pass and fail rates 

of lowering a cutoff from a point where 80% of AG passes to a point where 

95% of AG passes, with proportions DG comprises of persons who pass and 

of persons who fail (when mean scores differ by approximately half a standard 

deviation and DG comprises 50% of test takers) ............................................... 20 

Table 2. Rates at which white and blacks fall above and below 125%, 100%, 

 and 75% of the poverty line, with measures of differences between rates (2004) ........ 22 

Table 3. Favorable outcome rates of advantaged group (AG) and disadvantaged 

group (DG) at four settings with different favorable outcome frequencies, 

with measures of difference ....................................................................... 32 

Figure 1. Ratios of (1) DG Fail Rate to AG Fail Rate, (2) AG Pass Rate to 

DG Pass Rate, at Cutoffs Defined by AG Failure Rates ...................................... 22 

Figure 2. Absolute differences between rates of AG and DG pass (or fail) 

rates at various cutoff points defined by AG fail rate ......................................... 31 

Figure 3. Ratios of (1) DG fail rate to AG fail rate, (2) AG pass rate to DG 

pass rate, (3) DG failure odds to AG failure odds ............................................. 31 

1440 The Promise of Evidence-Based Policymaking
INTRODUCTION 

 

 Set out below are comments in response to Question 16 of the September 8, 2016 

Request for Comments for the Commission on Evidence-Based Policymaking. The question 

read: “How can data, statistics, results of research, and findings from evaluation, be best used to 

improve policies and programs?”

1

 

 The principal subject of these comments, addressed in Part I, involves the facts (a) that 

virtually all analyses of demographic and other differences involving rates of experiencing 

favorable or adverse outcomes have been fundamentally unsound as a result of the failure to 

recognize the ways measures of such differences tend to be affected by the prevalence of an 

outcome and (b) that, therefore, it is impossible for such analyses to improve policies and 

programs.

2

 One focus of Part I concerns health and healthcare disparities research, a subject to 

which recently the budget of the National Institutes of Health has devoted over $2.7 billion per 

year. Among other problems with such research, a large part of which involves efforts to 

determine whether health and healthcare disparities are increasing or decreasing over time, it has 

universally failed to recognize that as health and healthcare generally improve, relative 

(percentage) differences between the rates at which advantaged and disadvantaged groups 

experience adverse outcomes (e.g., mortality, non-receipt of appropriate care) tend to increase 

while relative differences in the rates at which such groups experience the corresponding 

favorable outcomes (e.g., survival, receipt of appropriate care) tend to decrease. In fact, although 

more than a decade ago the National Center for Health Statistics recognized that relative 

differences in adverse health and healthcare outcomes and relative differences in the 

corresponding favorable health and healthcare outcomes tend to change in opposite directions as 

health and healthcare generally improve, no other federal agency and no more than a few 

nongovernmental researchers have recognized that it is even possible for the two relative 

differences to change in opposite directions as there occur general changes in health and 

healthcare outcomes.3

 

 Thus, researchers analyzing health and healthcare disparities in terms of relative 

differences in favorable outcomes commonly reach opposite conclusions about directions of 

changes from researchers examining such disparities in terms of relative differences in the 

corresponding adverse outcomes. They do so, moreover, without recognizing even the 

 

1

 To facilitate consideration of issues raised in documents such as this I include links to referenced materials in 

electronic copies of the documents. I do not include links with every mention of an item, but attempt to include 

them often enough to make online items readily available without the reader’s having to search for an earlier link. 

An electronic copy of this document is available on the Institutional Correspondence subpage of the Measuring 

Health Disparities page of jpscanlan.com. Since the online copy may be corrected or annotated, I include here a link

to the most recent version (which, if corrected or annotated, will so indicate on the first page). 

2

 The request for comments and underlying statute are focused on government data. The analyses discussed here are 

sometime based on government data and sometimes based on other data. But issues regarding soundness of 

analyses and potential to improve programs are the same regardless of the origin of the data. 

3

 I do not want to encumber this introduction with either tabular illustrations or discussions of illustrative numbers. 

Readers not readily comprehending why relative differences in favorable outcomes and the corresponding adverse 

outcomes can, or will tend to, change in opposite directions as the prevalence of an outcome changes may wish to 

refer to Tables 1 and 2 in Section I.A infra before proceeding further. 

The Promise of Evidence-Based Policymaking 1441
2 

possibility that the relative difference in the opposite outcome could yield a conclusion contrary 

to that yielded by the relative difference in the outcome they are examining. More important, 

they do so without thought to the effects of changes in the prevalence of an outcome on the 

relative difference examined. As of result of this and related failures of understanding regarding 

the ways measures tend to be affected by the prevalence of an outcome, virtually nothing written 

about whether health and healthcare disparities are increasing or decreasing over time, or 

regarding the comparative size of such disparities in one setting compared with another or 

regarding one outcome compared with another, has had a sound statistical basis. And virtually 

all of it has been in some manner misleading. 

 The same holds for analyses of demographic differences regarding matters other than 

health and healthcare outcomes. For example, no analysis of racial differences regarding things 

like meeting proficiency standards and graduating from high school has yet recognized that 

general increases in proficiency and graduation rates will tend to reduce relative racial/ethnic 

differences in rates of reaching various proficiency levels and graduating while increasing 

relative differences in rates of failing to reach the proficiency level and failing to graduate. The 

failure to understand this matter with regard to educational outcomes remains essentially 

universal notwithstanding that data from the tests underlying such patterns make abundantly 

clear that lowering a test cutoff, or generally improving test performance, will tend to reduce 

relative differences in pass rates while increasing relative differences in failure rates. 

 Another focus of Part I involves the belief underlying federal civil rights law enforcement 

policies pertaining to fairness in lending, school discipline, criminal justice, and employment that 

relaxing standards or otherwise reducing the frequency of adverse outcomes will tend to reduce 

relative racial/ethnic and other demographic differences in rates of experiencing those outcomes. 

Exactly the opposite is the case. As with lowering a test cutoff, relaxing a standard and thereby 

reducing the frequency of an adverse outcome, while tending to reduce relative differences in 

rates of experiencing the corresponding favorable outcome, tends to increase relative differences 

in rates of experiencing the adverse outcome itself. 

 In consequence of the failure to understand this matter on the part of federal civil rights 

enforcement agencies and the research community, for decades countless entities covered by 

federal civil rights law that have complied with government encouragements to reduce the 

frequency of adverse outcomes have increased the chances that the government will sue them for 

discrimination. Such entities include, in recent years, the hundreds or thousands of school 

districts (and a number of states) that have been relaxing public school discipline standards while 

mistakenly believing that doing so will tend to reduce relative racial/ethnic and other 

demographic differences in rates of adverse discipline outcomes like suspension and expulsion. 

In fact, rarely will a month pass without the prominent reportage of efforts to reduce relative 

demographic differences in discipline rates by generally relaxing discipline standards, even as 

jurisdiction after jurisdiction reports that general reductions in discipline rates have been 

accompanied by increased racial/ethnic differences in discipline rates. 

 Meanwhile, the Department of Justice has been vigorously seeking to reduce relative 

differences in criminal justice outcomes (or measures that are functions of those relative 

differences) in Ferguson, Missouri, Baltimore, Maryland, and any number of other jurisdictions, 

1442 The Promise of Evidence-Based Policymaking
3 

by modifications of practices in ways that are more likely to increase those differences than 

reduce them. And reports of a presidential commission and the National Research Council 

reflect no better an understanding of the connection between the prevalence of adverse criminal 

justice outcomes and relative differences in experiencing the outcomes than the Department of 

Justice. 

 Further, in a range of contexts, entities covered by civil rights laws are required to justify 

practices with disparate impacts on protected groups and to implement less discriminatory 

alternatives even to justified practices. The governmental bodies imposing these requirements or 

monitoring and enforcing compliance with the requirements, as well as those adjudicating 

disputes as to compliance, do so while unaware that modifications to practices will commonly 

increase relative differences in one outcome while reducing relative differences in the opposite 

outcome. 

 The same misunderstanding reflected in the above-discussed civil rights enforcement 

activities are involved in perceptions about a wide range of policies directly aimed at mitigating 

demographic differences in adverse outcomes. For example, programs providing relief against 

foreclosure on home loans are universally regarded as means of reducing the relative 

racial/ethnic differences in foreclosure rates. While such programs will tend to reduce relative 

differences in rates of avoiding foreclosure, they will tend to (in fact will almost certainly) 

increase relative differences in foreclosure rates. 

 Recommendations to the Commission pertaining to Part I include (a) that the 

Commission form a committee to examine the soundness of analyses of differences involving 

outcome rates and explore methods for improving such analyses; (2) that the Commission 

recommend to Congress that it establish a permanent body to advise Congress and Executive 

Branch agencies on the statistical soundness of government-funded and other research; (3) that 

the Commission recommend that Congress require that all requests for federal funding include a 

statement as to whether and how the contemplated research will attempt to distinguish between 

the extent to which changes in the measures employed in research (or the comparative size of 

such measures in different settings) are functions of the prevalence of an outcome and the extent 

to which the changes reflect the effects of policies or indicate actual changes in the comparative 

circumstances of advantaged and disadvantaged groups; (4) that the Commission recommend 

that Congress take measures to ensure that federal civil rights law enforcement policies, and its 

own legislative actions, are not based on the mistaken belief that reducing the frequency of 

adverse outcomes tends to reduce relative demographic differences in rates of experiencing those 

outcomes; and (5) that the Commission recommend that Congress take measures to ensure that 

all laws and regulations requiring the monitoring of demographic differences or imposing 

obligations regarding disparate impacts provide guidance on how such differences and impact 

are to be measured. 

*** 

 

 Recent substantial treatments of issues addressed in Part I, with many examples of the 

pervasive failure to understand elementary statistical phenomena, may be found in my “The 

Mismeasure of Health Disparities,” Journal of Public Health Management and Practice 

The Promise of Evidence-Based Policymaking 1443
4 

(July/Aug. 2016) (JPHMP commentary),4 “Race and Mortality Revisited,” Society (July/Aug. 

2014), “The Perverse Enforcement of Fair Lending Laws,” Mortgage Banking (May 2014), 

“Measuring Health and Healthcare Disparities,” Proceedings of the Federal Committee on 

Statistical Methodology 2013 Research Conference (March 2014) (FCSM paper), “The 

Mismeasure of Discrimination,” Faculty Workshop, University of Kansas School of Law (Sept. 

2013) (Kansas Law paper), as well as my amicus curiae brief in the U.S. Supreme Court in the 

case of Texas Department of Housing and Community Development v. The Inclusive 

Communities Project, Inc. (Nov. 2014) (TDHCA brief) and my October 8, 2015 letter to the 

American Statistical Association (ASA) (October 2015 ASA letter). The most recent extended 

collection of graphical and tabular illustrations of the pertinent statistical patterns may be found 

in my quantitative methods seminar “The Mismeasure of Health Disparities in Massachusetts 

and Less Affluent Places,” Department of Quantitative Health Sciences, University of 

Massachusetts Medical School (Nov. 2015) (UMMS seminar). A less recent extended treatment 

of the issues may be found in my guest editorial in the Spring 2006 issue of the American 

Statistical Association publication Chance, titled “Can We Actually Measure Health 

Disparities?” (2006 Chance editorial), to which I will make a number of references below.5 

 Recent, more succinct treatments of key statistical issues, with a focus on federal law 

enforcement anomalies, may be found in my “Misunderstanding of Statistics Confounds 

Analyses of Criminal Justice Issues in Baltimore and Voter ID Issues in Texas and North 

Carolina,” Federalist Society Blog (Oct. 3, 2016), “Things DoJ doesn’t know about racial 

disparities in Ferguson,” The Hill (Feb. 22, 2016), “Things government doesn’t know about 

racial disparities,” The Hill (Jan. 28, 2014), “The Paradox of Lowering Standards,” Baltimore 

Sun (Aug. 5, 2013), and “Misunderstanding of Statistics Leads to Misguided Law Enforcement 

Policies,” Amstat News (Dec. 2012). A recent, somewhat more extended treatment of the law 

enforcement anomalies, with a focus on misunderstandings reflected in President Barrack 

Obama’s July 7, 2016 speech on racial/ethnic differences in criminal justice outcomes, may be 

found in my “Things the President Doesn’t Know About Racial Disparities,” Federalist Society 

Blog (Aug. 5, 2016). A recent succinct treatment of the pattern whereby relative differences in 

adverse outcomes tends to be larger, while relative differences in the corresponding favorable 

outcomes tend to smaller, in settings where the adverse outcomes are comparatively uncommon 

(including nations like Norway and Sweden and states like Minnesota) than in settings where 

adverse outcomes are comparatively common may be found in my “It’s easy to misunderstand 

gaps and mistake good fortune for a crisis,” Minneapolis Star Tribune (Feb. 8, 2014).6

 

4

 These comments do not attempt to employ a standard or consistent citation format. They simply employ the 

formatting that is most useful or convenient in the circumstances, in some cases simply copying the citations used in 

other works. 

5

 The recent extended treatments all discuss a measure of differences between outcome that is theoretically 

unaffected by the prevalence of the outcome, while the 2006 Chance editorial does not. 

6

 The patterns discussed in the Star Tribune commentary with regard to Norway, Sweden, and Minnesota are 

addressed with regard to Massachusetts in the UMMS seminar. See the UMMS seminar abstract. See also the 

discussion of patterns of discipline disparities in Massachusetts in the letter to the Boston Lawyers’ Committee for 

Civil Rights and Economic Justice (Nov. 12, 2015), which provides a narrative explanation of the data in Tables B1 

and B2 (slides 67 and 68) of the UMMS seminar. 

1444 The Promise of Evidence-Based Policymaking
5 

 A recent, brief treatment of the uncertainties facing entities with obligations regarding the 

disparate impacts of practices as a result of the near universal failure to understand the ways 

measures tend to be affected by the prevalence of an outcome may be found in my “Is HUD’s 

Disparate Impact Rule Unconstitutionally Vague?,” American Banker (Nov. 10, 2014). A 

recent, more comprehensive treatment of the subject may be found in “Is the Disparate Impact 

Doctrine Unconstitutionally Vague?,” Federalist Society Blog (May 6, 2016). See also Part II 

the TDHCD brief. 

*** 

 As reflected in the October 2015 ASA letter, on October 8, 2015, I requested that ASA 

take certain actions to reform analyses of demographic differences and to explain to arms of the 

federal government that reducing the frequency of an adverse outcome will tend to increase, 

rather than reduce, relative racial and other demographic differences in rates of experiencing the 

outcome. The matter was referred to the organization’s Science and Public Affairs Advisory 

Committee, which subsequently sought additional information regarding situations where the 

government encouraged the relaxing of standards to reduce relative differences in adverse 

outcomes. I responded by memoranda of December 14, 2015 and January 5, 2016. In a letter to 

ASA of July 25, 2016, in addition to informing the organization of developments since the 

October 2015 letter, I urged ASA to explain to President Barrack Obama that understandings 

about effects of modifications of practices on measures of racial/ethnic differences in criminal 

justice outcomes reflected in the President’s July 7, 2016 speech on criminal justice disparities 

were incorrect. 

 In September 2016, the ASA informed me that the Science and Public Affairs Advisory 

Committee believed that I was effectively highlighting the issues raised in my communications 

to the organization and did not see an additional role ASA could play in disseminating 

information on such issues. But, whereas ASA has chosen not to take affirmative steps to 

promote the understanding of such issues, as a result of the organization’s consideration of my 

October 2015 ASA letter and other communications by its Science and Public Affairs 

Subcommittee, the organization should be in an excellent position to advise the Commission 

regarding the validity of the points made in Part I of these comments. That holds both for the 

broader points in the comments and for the explanation that the government’s belief that 

generally reducing adverse lending, school discipline, criminal justice, and employment 

outcomes will tend to reduce relative differences in rates of experiencing those outcomes is not 

merely incorrect, but is the opposite of reality.7

 

 

7

 Explanations of why reducing the prevalence of an outcome tends to increase relative differences in experiencing 

the outcome (while reducing relative differences in the corresponding outcome) in ASA publications include the 

December 2012 Amstat News column and Spring 2006 Chance guest editorial previously mention and the article 

“Divining Difference,” Chance (Fall 1994).

The Promise of Evidence-Based Policymaking 1445
6 

 By letter of March 28, 2016, I requested the same action of the Population Association of 

American (PAA) and Association of Population Centers (APC) that I had requested of the ASA. 

At the end of April, the organizations advised that they were taking none of the requested actions 

because they did not prescribe statistical methods to the government or to their members. As 

with ASA, however, the leadership of these organizations should be able to advise the 

Commission as to the validity of the points I make in Part I of these comments. 

 In seeking the views of any person of organization of presumptive expertise, however, 

the Commission should be mindful that, so far as the published record reveals, there have been 

no recognitions that relative differences in experiencing an outcome and relative differences in 

avoiding the outcome tend to change in opposite directions as the prevalence of the outcome 

changes (or even that reducing the prevalence of an adverse outcome tends to increase relative 

differences in rates of experiencing it) other than in response to my explanations of these 

patterns.8

 And, for example, the belief that reducing adverse outcomes – whether regarding 

health and healthcare, lending, school discipline, criminal justice, or employment – will tend to 

reduce relative differences in experiencing those outcomes is widespread within the scientific 

community, while the understanding that it will tend to increase those differences is essentially 

non-existent (National Center for Health Statistics excepted).9

 Thus, when first asked, an 

experienced statistician or other data analyst might well say that a statement that lowering test 

cutoffs tends to increase relative differences in failure rates or that reducing poverty tends to 

increase relative differences in poverty rates is plainly false; but on reflection, and simply 

viewing the first two tables of these comments, the same person would almost certainly say that 

the statement is plainly true. Thus, while I encourage the Commission to seek input from the 

scientific community on the issues raised in these comments, I urge it to ensure that the persons 

providing such input consider the issues in depth. 

 

8

 I emphasize the published record. Given the elementary nature of the patterns I describe and that they are obvious 

in so many types of data, I assume that countless individual with and without statistical training have independently 

recognized the patterns even if they have not had occasion to publish such recognitions. See discussion of the work 

of Andrew Ho in Section I.B regarding the recognition of the pattern by which absolute differences tend to be 

affected by the prevalence of an outcome that might be deemed implicit recognition of the pattern by which the two 

relative differences tend to be affected by the prevalence of the outcome. Further, the patterns I describe as to 

relative differences in both outcomes are implicit in the widely understood pattern by which relative differences in 

favorable outcomes tend to be affected by the prevalence of an outcome. See October 2015 ASA letter at 14. 

9 A reflection of the current understanding of this subject in the social science community with respect to school 

discipline issues may be found in an article in the Winter 2017 issue of Education Next titled “What Do We Know 

About School Discipline Reform.” Calling for research into whether modifications to practices that generally reduce 

discipline rates will in fact reduce relative racial/ethnic differences in discipline rates, the article cited one recent 

study showing that after substantial reductions in suspensions, relative racial/ethnic differences in discipline rates 

“continue” to be high. While referencing an instance where the conventional understanding was not borne out, the 

article showed no awareness whatever of the reasons to expect general reductions in discipline rates to increase 

relative racial/ethnic differences in suspension rates or of the numerous situations across the country in which such 

increases are in fact being observed. Current titles reflecting the near universal expectation that reductions in the 

prevalence of an outcome should reduce relative differences in experiencing it (or a measure that is a function 

thereof) are commonplace. See, among many others, “Despite Progress, Racial Disparities Persist,” Sentencing 

Project (Aug. 19, 2016) and “Huge racial disparities persist despite slow infant mortality drop,” USA Today (Mar.

9, 2016). 

1446 The Promise of Evidence-Based Policymaking
7 

 Finally, I note that, while most of my longer recent papers and all of my workshops and 

conference presentations since 2008 discuss a method for appraising the difference between the 

circumstances of advantaged and disadvantaged groups reflected by their outcome rates 

(sometimes characterized as the strength of the forces causing outcome rates to differ) that is 

unaffected by the prevalence of an outcome, I give little attention to such measure here. The 

purpose of these comments is to cause the Commission to address the failings of standard 

analyses of differences in outcome rates and to explore methods for effectively measuring such 

differences. The measure I have discussed is merely one of a number of measures the 

Commission may consider.10 Further, attention to my proposed measure has on occasion 

diverted attention from the crucial issue of whether it is possible to analyze demographic 

differences in ways that can inform policy without consideration of the ways the measures 

employed tend to be affected by the prevalence of an outcome.11 Thus, these comments can 

better serve their purpose with only limited attention to the measure that I have discussed at 

length elsewhere. 

*** 

 Part II of the comments addresses the fundamental unsoundness of analyses of 

discrimination issues that are based on the examination of data solely on persons who accepted 

some outcome or situation and thus fail to consider the entire universe of persons seeking the 

most desirable outcome or situation. Almost all monetary recoveries in discrimination cases that 

have approached or exceeded $100 million have involved such analyses. These include the 

recent settlements of lending discrimination cases brought by the Department of Justice against 

Countrywide Financial Corporation and Wells Fargo Bank, with total recoveries exceeding half a 

billion dollars, that are discussed in the December 2012 Amstat News column and several other 

of the above references. This subject is treated in Section F (at 32-35) of the Kansas Law paper

and Section I.C (at 27-30) of the TDHCD brief. 

The subject is treated fairly succinctly with reference to fair lending issues in “Fair 

Lending Studies Paint Incomplete Picture,” American Banker (April 24, 2013) and on the final 

page of “The Perverse Enforcement of Fair Lending Laws,” Mortgage Banking (May 2014). 

Older treatments of varying length involving employment discrimination issues may be found in 

my “Illusions of Job Segregation” Public Interest (Fall 1988), "Multimillion-Dollar Settlements 

May Cause Employers to Avoid Hiring Women and Minorities for Less Desirable Jobs to 

Improve the Statistical Picture," National Law Journal (Mar. 27, 1995), and several other items 

referenced on the Employment Discrimination page of jpscanlan.com. 

 

10 See "Race and Mortality Revisited" at 337. See also discussion of the odds ratio in note 49 infra. 

11 See (a) Mackenbach JP. Response to Scanlan. BMJ (July 14, 2016), and (b) Penman-Aguilar, Talih M, 

Moonesinghe R, Huang M. “Response to Scanlan Concerning: Measurement of Health Disparities, Health 

Inequities, and Social Determinants of Health to Support the Advancement of Health Equity. J Public Health 

Management Practice 2016;22(6), 614-615 [lttr]. While focusing on the measure I suggest as theoretically 

unaffected by the prevalence of an outcome, these responses leave entirely unaddressed whether health and 

healthcare disparities research can be of any value without considering the ways the measures employed tend to be 

affected by the prevalence of an outcome. 

The Promise of Evidence-Based Policymaking 1447
8 

 Analyses that fail to examine the entire universe at issue are also involved in the cases of 

Bank of America Corp. et al. v. City of Miami and Wells Fargo & Co. v. City of Miami, Sup. Ct. 

Nos. 15-1111 and 15-1112, which were argued before the Supreme Court on November 8, 2016 

(though the issue before Court do not involve the merits of the claims).12

 This subject is given brief attention at page 32 of the October 2015 ASA letter. But I 

would not regard the organization’s consideration of my letter as putting it in a position to have a 

sound understanding of the subject. The organization, of course, does have the expertise to 

provide informed views on the issue should the Commission seek such views.13

 Recommendations to the Commission regarding issue addressed in Part II are similar in 

to the above-mentioned recommendations regarding issues addressed in Part I. 

 Part III discuses the recommendations to the Commission regarding the issues addressed 

in Parts I and II. 

I. The Unsoundness of Analyses of Demographic Differences That Fail to Consider the 

Effects of the Prevalence of an Outcome on the Measure Employed 

 There are four standard measures by which observers commonly quantify differences 

between rates at which advantaged and disadvantaged groups experience favorable or adverse 

outcomes: (1) relative (percentage) differences between rates of experiencing the outcome; (2) 

relative differences between rates of avoiding the outcome (i.e., experiencing the opposite 

outcome); (3) absolute (percentage point) differences between the outcome rates; and (4) odds 

ratios. None of these measures provides a sound basis for quantifying the differences in the 

circumstances of advantaged and disadvantaged groups reflected by their outcome rates because, 

for reasons related to the shapes of underlying risk distributions, each measure tends to be 

systematically affected by the prevalence of an outcome.14

 

12 Claims in these cases also implicate the subjects of Part I with regard to the failures to recognize that (a) the less 

the incentive for loan officers to issue subprime loans, the larger (not smaller) will tend to be relative racial/ethnic 

differences in receipt of subprime loans; (b) concentrations of foreclosures in disadvantaged neighborhoods will 

vary inversely with the frequency of foreclosures; (c) relative differences in assignment to subprime status and 

foreclosures will tend to be larger, while relative differences in avoiding those outcomes will tend to be smaller, 

among higher-income than lower-income borrowers. See, e.g., the 2014 Mortgage Banking article. 

 

13 The letter to the Population Association of America and the Association of Populations Centers did not mention 

this subject at all. As with ASA, however, the organizations should be able to form views on the subject if requested 

by the Commission. 

14 As will be discussed in Sections A and C, observers often appraise demographic differences in terms, not of 

differences between outcome rates, but of differences between the proportion a group comprises of persons 

potentially experiencing an outcome and proportion it comprises of persons actually experiencing the outcome. It is 

for that reason that the Introduction refers to “differences involving outcome rates” rather than “differences between 

outcome rates.” These comments will give attention both to the relationship between differences between outcome 

rates and differences between the two referenced proportions and to the impossibility of soundly analyzing 

demographic differences based solely on information regarding the two proportions. But, for simplicity, issues 

regarding the two proportions are given minimal attention in introductory material. 

1448 The Promise of Evidence-Based Policymaking
9 

 By way of broad summary, as an outcome changes in prevalence, the relative difference 

in rates of experiencing it and the relative difference in rates of avoiding it tend to change in 

opposite directions. More specifically, the relative difference in the decreasing aspect of the 

outcome tends to increase while the relative difference in the corresponding increasing aspect of 

the outcome tends to decrease. As the prevalence of an outcome changes, absolute differences 

between rates and differences measured by odds ratios tend also to change as the prevalence of 

an outcome changes, though in a more complicated way than the two relative differences. 

Roughly, as an outcome goes from being rare to being common, the absolute difference tends to 

increase; as an outcome goes from being common to being very common, the absolute difference 

tends to decrease. As the frequency of an outcome changes, the absolute difference tends to 

change in the same direction as the smaller relative difference. Since persons relying on relative 

differences to appraise demographic differences typically examine the larger of the two relative 

differences, such persons tend to systematically reach opposite conclusions about directions of 

changes over time from persons relying on absolute differences. As the prevalence of an 

outcome changes, the difference measured by the odds ratio tends to change in the opposite 

direction of the absolute difference. 

 As the prevalence of an outcome changes, all measures may change in the same direction. 

In that case, one may infer that there occurred an actual change in the difference between 

circumstances of two groups reflected by the outcome rates. But whenever a relative difference 

and the absolute difference change in different directions, the other relative difference will 

necessarily have changed in the opposite direction of the first relative difference and the same 

direction as the absolute difference. 

 For a variety of reasons, including actual changes in the strength of the forces causing 

outcome rates to differ (or actual differences in the strength of those forces in different 

settings),15 one may observe many departures from these patterns. But it is impossible to 

evaluate the efficacy of policies aimed at mitigating differences in the circumstances of 

advantaged and disadvantaged groups (or to draw inference about underlying processes) without 

understanding the above patterns and attempting to distinguish between the effects of changes in 

measure that are functions of changes in the prevalence of an outcome and changes that are 

functions of other factors. It is also impossible to determine whether a particular measure 

indicates a large or small difference between the circumstances of advantaged and disadvantaged 

groups without understanding the effects of the prevalence of an outcome on the measure. 

 Further, the patterns I describe may provide useful benchmarks for identifying flawed 

interpretations of data (and potentially for divining a means of measuring demographic 

differences unaffected by the prevalence of an outcome). But with regard to the fundamental 

 

15 Other factors include sampling variation, irregularities in the underlying distributions, factors related to the 

approaching of an irreducible minimum regarding outcomes like infant mortality. They also include the possibility 

that the underlying distributions may substantially depart from the normal distribution, among other reasons, 

because the universe being examined is comprised of truncated portions of normal distributions. See "Race and 

Mortality Revisited" and the materials it references (at 337) regarding factors creating potential problems with the 

measure discussed there. The same factors affect the likelihood of observing the described patterns in particular 

situations. 

The Promise of Evidence-Based Policymaking 1449
10 

unsoundness of the standard approaches to analyzing demographic differences, the degree to 

which the measures in fact tend to be affected by the prevalence of an outcome comports with 

my descriptions is unimportant, as is the precise utility of the measure I suggest in various places 

is theoretically unaffected by the prevalence of an outcome. For as long as a measure tends to be 

affected by the prevalence of an outcome in any manner, it is not possible to employ the measure 

for a useful purpose without attempting to divine the role of the prevalence of the outcome in the 

setting examined. Indeed, the mere possibility for the patterns shown in the tables of these 

comments to exist in nature raises the same interpretative issue as the fact that such patterns will 

commonly, if not almost invariably, be found whenever there exist substantial differences in the 

prevalence of an outcome in the settings being compared.16

 Yet, with few or no exceptions, persons analyzing demographic differences have done so 

without even recognizing that the measures employed tend to change as the frequency of an 

outcome changes and hence have never attempted to divine the role of the prevalence of an 

outcome in the settings examined. Thus, almost all such analyses have been statistically unsound 

and in some manner misleading. Even when a study may have correctly identified such things as 

the change in the direction of a demographic disparity, it will have been misleading by implying 

that the measure employed is effectively quantifying either the difference in the circumstances of 

two groups reflected by their outcome rates or the size of the change in such difference. Even a 

study that merely reports the size of a disparity is commonly misleading by suggesting that the 

measure employed is effectively quantifying the disparity between the circumstances of the 

groups being compared.17 

 Section A addresses the patterns by which relative differences in a favorable outcome and 

relative differences in the corresponding adverse outcome tend to be affected by the prevalence 

of an outcome and some implications of the failure to understand those patterns, while discussing 

other measures only to the extent necessary to make particular points. It gives particular 

attention to interpretations of health and healthcare disparities based on relative differences in a 

favorable or adverse outcomes and the 2004-05 recommendation of the National Center for 

Health Statistics regarding which relative difference to rely on, as well as the recent reversal of 

that recommendation and the repudiation of a decade of research that relied on the earlier 

recommendation. The section also gives particular attention to federal law enforcement policies 

 

16 See the October 2015 ASA letter (at 27) regarding the interaction of other factors with the prevalence-related 

patterns described here, including discussion of why the patterns will be having an influence even when they are not 

specifically observed. 

17 Studies that attempt to determine the role of a particular factor in an observed disparity may not suffer from same 

problems as studies that attempt to examine changes over time or otherwise to compare the size of disparities in 

difference settings, at least so far as the primary purpose of the former studies is concerned. But the issues 

addressed here do affect the quantification of the observed disparity before and after adjustment for the factor and 

thus the quantifications of the part of the difference that is accounted for by the factor. That is so even though, say, 

in a situation where when the rates at which an advantaged group and a disadvantaged group experience an adverse 

outcome are 10% and 20% and adjustment for a factor would reduce the latter rate to 15%, one would observe the 

same 50% reduction in both relative differences and in the absolute difference. See Comment on Lynch JECH 

2006 (2006) and Second Comment on Lynch JECH 2006 (2009). But one would not observe that same reduction 

for the difference measured by the odds ratio or the measure discussed in "Race and Mortality Revisited.” 

1450 The Promise of Evidence-Based Policymaking
11 

based on the mistaken belief that reducing the frequency of an adverse outcome will tend to 

reduce relative racial/ethnic and other demographic differences in rates of experiencing it. 

 Section B discusses the patterns by which absolute differences and odds ratios tend to be 

affected by the prevalence of an outcome and some of the implications of the failure to 

understand those patterns or the relationship of those patterns to the patterns by which the two 

relative differences tend to be affected by the prevalence of an outcome. The section gives 

particular attention to the analyses of healthcare disparities and public school proficiency 

disparities in terms of absolute differences between rates without consideration of the ways 

improvements in healthcare or education will tend to increase absolute differences regarding 

uncommon outcomes and reduce absolute differences regarding common outcomes. 

 Section C discusses problems with analyses of demographic differences based on the 

proportion a group comprises of persons potentially experiencing an outcome and the proportion 

it comprises of persons actually experiencing the outcome that go beyond the problems involved 

in standard analyses of differences between outcome rates. The section explains why a 

demographic difference should never be analyzed on the basis of comparisons of the two 

proportions. 

 Section D discusses problems with standard analyses of subgroup analyses arising from 

the failure to understand the patterns described here and the failure to understand the illogic of an 

expectation that a factor will tend to have the same relative effect on an outcome in situations 

involving different baseline rates for the outcome. 

 The reader will find considerable redundancy in the discussion in Sections A though D, 

especially with regard to relative measures. To some degree the redundancy occurs because 

many points are necessarily implied in other points. For example, the statement that reducing the 

prevalence of an outcome tends to increase relative differences in the outcome, by necessarily 

implying that increasing the prevalence of an outcome tends to reduce relative differences in 

experiencing the outcome, necessarily also implies that reducing the prevalence of an outcome 

tends to reduce relative differences in the opposite outcome. The statement that as an outcome 

changes in prevalence the group with the lower baseline rate tends to experience a larger 

proportionate change in the rate than the other group necessarily means that the other group will 

tend to experience the larger proportionate change in the opposite outcome, since the other group 

has the lower baseline rate for the opposite outcome. See October 2015 ASA letter (at 10 n.14). 

The statement that relative racial differences in some adverse outcome like rejection of a loan 

application or poor self-rated health tend to be greater, while relative racial differences in the 

corresponding favorable outcomes tends be smaller, among higher-income than lower-income 

groups means the same thing as the statement that having high income tends to reduce the 

adverse outcome proportionately more for whites than blacks while increasing the favorable 

outcome proportionately more for blacks than whites. In fact, all statements about failure to 

understand patterns of relative differences are effectively statements about misperceptions 

regarding subgroup effects. And, of course, the statements made above about higher- and lowerincome groups effectively illustrate the fallacy of essentially all perceptions about racial/ethnic 

or other differences in favorable or adverse outcomes within advantaged subpopulations or the 

effects of being in an advantaged subpopulation on different racial/ethnic groups. 

The Promise of Evidence-Based Policymaking 1451
12 

 Nevertheless, I believe it useful to make various points that are already implied in other 

points in order (a) to make the points clearer to readers to whom the relationship between the 

points may not be obvious, (b) to illustrate how commonly the patterns I describe are found in 

reality, and (c) to illustrate how vast is the waste of resources devoted to study of demographic 

differences, and the misleading reportage of the results thereof, in consequence of the failure to 

understand the patterns described in this part. 

 The discussion that follows contains a number of tabular or graphical illustrations, as do 

the more extended treatments of this subject listed above. Larger collections of graphical and 

tabular illustrations, which include many examples from actual studies, may be found in the 

methods workshops given at American universities between 2012 and 2015,18 and the conference 

presentations given in Europe and North American between 2001 and 2011.19 

 Letters to institutions and organizations written since 2009 provide numerous examples 

of misinterpretations of data by entities believed to have considerable expertise in the analyses of 

demographic differences and suggest the universality of the unsoundness of such analyses even 

among organizations and institutions whose missions principally involve the analysis of such 

differences.20 Other examples may be found in online comments to medical and health policy 

 

18 In addition to the November 2015 UMMS seminar referenced supra, see “The Mismeasure of Discrimination,” 

Center for Demographic and Social Analysis, University of California, Irvine (Jan. 20, 2015); “The Mismeasure of 

Demographic Differences in Outcome Rates” Public Sociology Association of George Mason University (Oct. 18, 

2014); “Rethinking the Measurement of Demographic Differences in Outcome Rates,” Maryland Population 

Research Center of the University of Maryland (Oct. 10, 2014); “The Mismeasure of Association: The 

Unsoundness of the Rate Ratio and Other Measures That Are Affected by the Prevalence of an Outcome,” 

Minnesota Population Center and Division of Epidemiology and Community Health of the School of Public Health 

of the University of Minnesota (Sept. 5, 2014); “The Mismeasure of Group Differences in the Law and the Social 

and Medical Sciences,” Institute for Quantitative Social Science at Harvard University (Oct. 17, 2012); “The 

Mismeasure of Group Differences in the Law and the Social and Medical Sciences,” Department of Mathematics 

and Statistics of American University (Sept. 25, 2012). 

19 There were twenty-two such presentations given in eight countries (mainly between 2006 and 2011). Links to 

them are collected here. Like the methods workshops, many presentations bear much similarity to one another. 

Some, however, give particular attention to misunderstandings pertinent to a particular country or region, as in the 

case of the 2006 British Society for Populations Studies paper “The Misinterpretation of Health Inequalities in the 

United Kingdom” (presentation) and the 5th Nordic Health Promotion Research Conference (2006) presentation 

“The Misinterpretation of Health Inequalities in Nordic Countries” (abstract), which is discussed in the abovereferenced 2014 Minneapolis Star Tribune commentary. The 16th Nordic Demographic Symposium (2008) 

presentation “Measures of Health Inequalities that are Unaffected by the Prevalence of an Outcome” uses 

illustrations from studies of demographic differences in health and healthcare outcomes pertaining to Finland, where 

the conference was held. The 2011 International Conference on Health Policy Statistics presentation “Perverse 

Perceptions of the Impact of Pay for Performance on Healthcare Disparities” (abstract) focuses on misperceptions 

about the effects of pay-for-performance programs on healthcare disparities, a subject later addressed more fully in 

"Race and Mortality Revisited" and the FCSM paper and a number of the methods workshops. The 2011 Joint 

Statistical Meetings presentation “Interpreting Differential Effects in Light of Fundamental Statistical Tendencies” 

(abstract) focused on perceptions about subgroup effects. Only studies from 2008 on, commencing with the 7th 

International Conference on Health Policy Statistics presentation “Can We Actually Measure Health Disparities?” 

(abstract) ,discuss a method for measuring demographic differences unaffected by the prevalence of an outcome. 

20 Recipient of such letters include entities whose activities involves the analyses of demographic differences or the 

oversight of entities analyzing demographic differences as well as entities who are affected by analyses of 

1452 The Promise of Evidence-Based Policymaking
13 

journal since 2005.21 Further examples of misinterpretations of demographic differences and 

explanations as to certain nuances of patterns by which measures tend to be affected by the 

prevalence of an outcome may be found on the pages and subpages of jpscanlan.com devoted to 

measurement issues.22 

 Many examples may also be found in older articles addressing issues that were then 

topical or had been for some time.23 For example, the 1987 Plain Dealer article “The 

‘Feminization of Poverty’ is Misunderstood” discusses the failure to recognize that reductions in 

 

demographic differences by other entities. Recipients include Oklahoma City School District (Sept. 20, 2016), 

Antioch Unified School District (Sept. 9, 2016), American Statistical Association II (July 25, 2016), Federal Judicial 

Center (July 7, 2016), University of Oregon Institute on Violence and Destructive Behavior and University of 

Oregon Law School Center for Dispute Resolution II (July 5, 2016), University of Oregon Institute on Violence and 

Destructive Behavior and University of Oregon Law School Center for Dispute Resolution (July 3, 2016), New 

York City Center for Innovation through Data Intelligence (June 6, 2016), Consortium of Social Science 

Associations (Apr. 6, 2016), Population Association of America and Association of Population Centers (Mar. 29, 

2016), Council of Economic Advisers (Mar. 16, 2016), City of Madison, Wisconsin (Mar. 12, 2016), Stanford 

Center on Poverty and Inequality (Mar. 8, 2016), City of Boulder, Colorado (Mar. 5, 2016), Houston Independent 

School District (Jan. 5, 2016), Boston Lawyers’ Committee for Civil Rights and Economic Justice (Nov. 12, 2015), 

House Judiciary Committee (Oct. 19, 2015), American Statistical Association (Oct. 8, 2015), Chief Data Scientist of 

White House OSTP (Sept. 8, 2015), McKinney, Texas Independent School District (Aug. 31, 2015), Department of 

Health and Human Services and Department of Education (Aug. 24, 2015), Agency for Healthcare Research and 

Quality (July 1, 2015), City of Minneapolis, Minnesota (June 8, 2015), Texas Appleseed (Apr. 7, 2015), Senate 

Committee on Health, Education, Labor and Pensions (Mar. 20, 2015), United States Department of Justice and City 

of Ferguson, Missouri (Mar. 9, 2015), Vermont Senate Committee on Education (Feb. 26, 2015), Portland, Oregon 

Board of Education (Feb. 25, 2015), Wisconsin Council on Families and Children’s Race to Equity Project (Dec. 23, 

2014), Financial Markets and Community Investment Program, Government Accountability Office (Sept. 9, 2014), 

Education Law Center (Aug. 14, 2014), IDEA Data Center (Aug. 11, 2014), Institute of Medicine II (May 28, 2014), 

Annie E. Casey Foundation (May 13, 2014), Education Trust (April 30, 2014), Investigations and Oversight 

Subcommittee of House Finance Committee (Dec. 4, 2013), Mailman School of Public Health of Columbia 

University (May 24, 2013), Senate Committee on Health, Education, Labor and Pensions (Apr. 1, 2013), Federal 

Reserve Board (March 4, 2013), Harvard University et al. (Oct. 26, 2012), Harvard University (Oct. 9, 2012), 

United States Department of Justice (Apr. 23, 2012), United States Department of Education (Apr. 18, 2012), The 

Commonwealth Fund (June 1, 2010), Institute of Medicine (June 1, 2010), National Quality Forum (Oct. 22, 2009), 

Robert Wood Johnson Foundation (Apr. 8, 2009). 

 

21 There are approximately 150 such comments, mainly involving interpretations of data on health and healthcare 

disparities but in some cases (particularly BMJ comments) involving perceptions about subgroup effects. Links to 

most are available here. 

22 The principal measurement pages are Measuring Health Disparities, Scanlan’s Rule, Mortality and Survival, 

Statistical Reasoning, Immunization Disparities, Educational Disparities, Disparate Impact, Discipline Disparities, 

Lending Disparities, Employment Discrimination, Feminization of Poverty. The pages have close to 100 subpages. 

Brief summaries of the pages and most of their subpages may be found on the home page of jpscanlan.com. 

23 In addition to the 2006 Chance editorial and 1994 Chance article mentioned supra, see “Race and Mortality,” 

Society (Jan./Feb. 2000) (reprinted in Current, Feb. 2000), “Mired in Numbers,” Legal Times (Oct. 12, 1996), 

“When Statistics Lie” (Legal Times, Jan. 1 1996), “Getting it Straight When Statistics Can Lie,” Legal Times ( June 

23, 1993), "Bias Data Can Make the Good Look Bad," American Banker (Apr. 27, 1992), “Comment on 

“McLanahan, Sorensen, and Watson's 'Sex Differences in Poverty, 1950-1980’," Signs (Winter 1991), “An Issue of 

Numbers,” National Law Journal (Mar. 5, 1990), “The ‘Feminization of Poverty’ is Misunderstood,” Plain Dealer 

(Nov 11, 1987) (reprinted in Current, May 1988, and Annual Editions: Social Problems 1988/89, Dushkin 1988). 

The Promise of Evidence-Based Policymaking 1453
14 

poverty will tend to increase relative differences between the poverty rates of female-headed 

families and other units and cause poverty to become more feminized (while increases in poverty 

will tend to have the opposite effect). The 1990 National Law Journal article “An Issue of 

Numbers,” explains that the high proportion blacks comprised of persons disqualified from 

intercollegiate athletics by National Collegiate Athletic Association academic standards was a 

function of the leniency rather than the stringency of the standards. The 1993 Legal Times

article “Getting it Straight When Statistics Can Lie” discusses the failure to recognize that 

improving the performance of all employees (which had been suggested as a means of reducing 

large relative racial differences in terminations for inadequate performance) would tend to 

increase relative racial differences in such terminations or that employers that strive for fairness 

by providing procedural safeguards against arbitrary termination will tend to show larger relative 

differences in termination rates than other employers. It also discusses a court of appeals 

decision premised on the belief that a large relative difference in failure to meet a performance 

standard was a consequence of the stringency, rather the leniency, of the standard. The 1996 

Legal Times article “When Statistics Lie” discusses a putative class action based on a study that 

ranked lenders according to the size of relative racial difference in mortgage rejection rates 

without recognizing that more lenient lending criteria tended to be associated with larger relative 

differences in rejection rates. The 1996 Legal Times article “Mired in Numbers” explains why 

relaxing a three-strikes law will tend to increase the proportion blacks make up of persons 

affected by the law. The 1991 Public Interest article “The Perils of Provocative Statistics and 

the 2000 Society article “Race and Mortality” provide a wide range of examples.24 The failures 

of understanding reflected in these examples are just as pervasive today as they were when the 

articles were published. 

A. Relative Differences in Rates of Experiencing Favorable and Adverse Outcomes 

 The section discusses the pattern whereby the rarer an outcome the greater tends to be the 

relative difference between rates at which advantaged and disadvantaged groups experience the 

outcome and the smaller tends to be the relative difference between rates at which such groups 

avoid the outcome (i.e., experience the opposite outcome).25 It also discusses some of the 

implications of the failure to understand the pattern in analyses of demographic differences. 

 Discussion of patterns by which absolute differences tend to affected by the prevalence of 

an outcome will be largely deferred to the next section. But it would be useful for readers to 

understand at this point that all cases where a relative difference between rates at which two 

groups experience an outcome changes in a different direction from the absolute difference 

between those rates involve situations (a) where the relative difference between rates of 

 

24 References to medical and health policy journal articles underlying many of the statements about health outcomes 

in "Race and Mortality," and tables based on data from those articles, may be found in my unpublished paper “The 

Relationship Between Declining Mortality and Increasing Racial and Socioeconomic Disparities in Mortality”

(1992). 

 

25 A more precise description of the pattern would state, rather than “the rarer an outcome,” “the more the outcome 

is restricted toward either end of the overall distribution.” But I have characterized the pattern in the manner done in 

the text above for some time and those discussing it have not been confused by the usage. Thus, I am not at this 

time inclined to depart from the usage in the text. 

1454 The Promise of Evidence-Based Policymaking
15 

experiencing the opposite outcome has changed in the opposite direction of the relative 

differences between rates of experiencing the first outcome and (b) where the relative difference 

between rates of experiencing the increasing outcome has decreased while the relative difference 

between rates of experiencing the decreasing outcome has increased.26 The same holds for 

situations where the relative difference is larger in one setting while the absolute difference is 

larger in the other setting.27 That is, the relative difference for one outcome will be larger in the 

setting where that outcome is less common while the relative difference for the opposite outcome 

will be larger in the other setting (which is the setting where the opposite outcome is less 

common). Thus, all situations where attention has recently been given to the fact that a relative 

difference and absolute difference show different patterns as to the comparative size of some 

disparity involve situations where the relative difference in rates of experiencing an outcome is 

larger, while the relative difference in avoiding the outcome is smaller, where the outcome is less 

common. 

 Tables 1 and 2 illustrate the pattern whereby the rarer an outcome the greater tend to be 

the relative differences in experiencing it and the smaller tends to be the relative differences in 

avoiding it by means of test score and income data. The tables will show how lowering a test 

cutoff will tend to increase relative differences in pass rates while reducing relative differences in 

pass rates and how reducing poverty tends to increase relative differences in poverty rates while 

reducing relative differences in rates of avoiding poverty. But the patterns that will be shown in 

the tables may be found in virtually any data showing the proportions of two groups that fall 

 

26 I have often noted (usually in discussions of perceptions about value judgments involved in choosing to appraise 

a demographic disparity in terms or the relative difference the observer happens to be examining or the absolute 

difference) that anytime it is mentioned that a relative difference and the absolute difference have changed in 

opposite directions, the unmentioned relative difference will necessarily have changed in the opposite direction of 

the mentioned relative difference and the same direction as the absolute difference. See, e.g., "Race and Mortality 

Revisited" (at 335) and "The Mismeasure of Health Disparities" (at 415). But, at least partly because I think many 

readers would regard the point to be obvious, I have not often explained why that is so. My Comment on Boscoe 

BMJ 2015 (2016) provides a rather inefficient explanation. The point may be somewhat more efficiently explained 

in the following paragraph. 

 In order for a relative difference and the absolute difference for an outcome to change in opposite directions, the 

rates for both groups must change in the same direction, and the group with the lower baseline rate must experience 

the larger relative change while the other group experiences the larger absolute change. Any other scenario would 

cause the relative difference and the absolute difference to change in the same direction. That includes the scenario 

in which the group with the lower baseline rate experiences the larger absolute change, which necessarily means that 

it also experiences the larger relative change and that therefore the relative and absolute difference will change in the 

same direction. The fact that the group with the higher baseline rate for the outcome experiences the larger absolute 

change in the outcome means that it also experiences the larger absolute change in the opposite outcome. Since that 

group has the lower baseline rate for the opposite outcome, the fact that it experiences the larger absolute change in 

the opposite outcome means that it also experiences the larger relative change in the opposite outcome. Thus, the 

absolute difference and the relative difference for that outcome must change in the same direction. 

27 I sometimes use the phrase “differentiated other than temporally” to refer to comparisons of settings that do not 

involve changes over time (i.e., comparisons of one place or subpopulation with another or one condition with 

another). But all involve the same statistical phenomenon. And regardless of the nature of the situations being 

compared, the extent to which the described patterns are observed will principally turn on the degree to which the 

sizes of difference between the forces causing outcome rates of advantaged and disadvantaged to differ varies in the 

two situations and the degree to which the prevalence of the outcome varies in the two situations. 

The Promise of Evidence-Based Policymaking 1455
16 

below or above various points on a continuum of a quantifiable factor associated with 

experiencing some favorable outcome and the corresponding adverse outcome or simply 

showing the proportions of different groups that fall into various categories reflecting 

quantification of a factor. 

 For example, income data on which Table 2 is based show how lowering an income 

requirement to receive some favorable outcome will tend to increase relative difference in rates 

of failure to meet the requirement while reducing relative differences in rates of meeting the 

requirement.28 Credit score data show the same thing with respect to a credit score 

requirement.29 National Health and Nutrition Survey data show how generally reducing systolic 

blood pressure tends to increase relative racial differences in hypertension while reducing 

relative racial differences in rates of avoiding hypertension or that generally improving folate 

levels will tend to increase relative racial differences in low folate while reducing relative racial 

differences in rates of adequate folate.30 Even data on rates at which groups fall into categories 

of health literacy show that the lower the category, the greater the relative difference in failing to 

achieve it and the smaller the relative difference in achieving it.31 Data on rates at which groups 

meet certain proficiency levels will show the same thing.32 And life tables show that the lower 

the age (and hence the lower are the rates of failing to survive to it) the larger tends to be the 

relative difference in failing to survive to the age while the smaller tends to be the relative 

difference in surviving to the age.33

 As mortality declines relative difference in mortality tend to increase while relative 

differences in survival tend to decrease. As healthcare improves relative differences in receipt of 

appropriate care tend to decrease while relative differences in non-receipt of such care tend to 

increase. Generally decreasing any adverse outcomes like mortgage rejection, mortgage 

foreclosure, suspension or expulsion from school, dropping out of school, arrests, low birth 

weight tends to increase relative differences in rates of experiencing the outcome while reducing 

relative differences in avoiding it. 

 Similarly, relative racial and other relative differences in adverse outcomes tend to be 

larger, while relative racial differences in the corresponding favorable outcomes tend to be 

smaller, among advantaged populations/subpopulations/settings with comparatively low adverse 

 

28 See Table of the 2006 Chance editorial. 

29 See the Credit Score Illustrations subpage of the Scanlan’s Rule page of jpscanlan.com. 

30 See the NHANES Illustrations subpage of the Scanlan’s Rule page

31 See the Numeracy Illustration subpage of the Scanlan's Rule page 

32 Rate of achieving a certain level of some sort of proficiency (including anything below it) should not be confused 

with rates of falling into a certain proficiency category. Rates at which certain groups fall into any intermediate 

category of an outcome – like rates at which different groups of students receive the grade of C – can never be 

effectively analyzed. See the Intermediate Outcomes subpage of the Scanlan’s Rule page of jpscanlan.com. 

33 See the Life Tables Illustrations subpage of the Scanlan's Rule page and the 

1456 The Promise of Evidence-Based Policymaking
17 

outcome rates (e.g., the young, persons receiving better healthcare like the insured and the 

military, the well-educated, persons with higher income, generally higher socioeconomic groups, 

states like Minnesota and Massachusetts) than among populations/subpopulations/settings with 

comparatively high adverse outcome rates. 

 For example, relative racial differences in infant mortality and low birth weight tend to be 

larger, while relative racial difference in infant survival and normal birth weight tend to be 

smaller among the well-educated and other low risk groups (where such outcomes are 

comparatively uncommon) than among high risk groups.34 Relative racial differences in rates of 

mortgage rejection, mortgage foreclosure, and poor self-rated health tend to be greater, while 

relative racial differences in rates for the corresponding favorable outcomes tend to be smaller, 

among higher-income than among lower-income groups.35 Relative racial differences in adverse 

discipline outcomes tend to be larger, while relative differences in the avoidance of such 

outcomes tend to be smaller, in settings where suspensions are less common (e.g., schools 

without zero tolerance policies, suburban schools, states like Massachusetts with generally low 

suspension rates, pre-school) than in the corresponding settings where suspensions are more 

common.36 Similarly, relative racial differences in adverse discipline outcomes tend to be larger, 

while relative racial differences in avoiding those outcomes tend to be smaller, among girls than 

among boys; correspondingly, relative gender differences in adverse discipline outcomes tend to 

be larger, while relative gender differences in avoiding those outcomes tend to be smaller, among 

whites than blacks.37 

 With only minor exception, the above patterns are utterly unknown in the law and the 

social and medical sciences. In fact, particularly with regard to infant and cancer outcomes, 

researchers will often refer to survival and mortality interchangeably, often stating they are 

examining relative differences in the former while in fact examining relative differences in the 

latter. They do so without recognizing the possibility, much less the likelihood, that the two 

relative differences will provide opposite results as to whether some demographic disparity is 

increasing or decreasing over time or is larger in one setting than another or with regard to one 

condition than another. See the Mortality and Survival page of jpscanlan.com. 

 

 And a large part of those communities expect the exact opposite of the above patterns. 

That is, a large part of the community expects that general reductions in an adverse outcome 

should reduce relative demographic differences in rates of experiencing the outcome and that 

relative racial/ethnic difference in adverse outcomes will tend to be smaller within advantaged 

 

34 See "Race and Mortality” and the unpublished paper “The Relationship Between Declining Mortality and 

Increasing Racial and Socioeconomic Disparities in Mortality.” 

35 See the Disparities – High Income and Foreclosure Disparities subpage of the Lending Disparities page of 

jpscanlan.com and Figure 5 (slide 66) of the UMMS seminar. 

36 See the Suburban Disparities, DOE Equity Report, and the Preschool Disparities subpages of the Discipline 

Disparities page; letter to Boston Lawyers’ Committee for Civil Rights and Economic Justice (Nov. 12, 2015); 

Table 8 of "Race and Mortality Revisited" (at 342). 

37 See Tables 1 and 2 of the Discipline Disparities page. 

The Promise of Evidence-Based Policymaking 1457
18 

than disadvantaged subpopulations or settings. See especially “Race and Mortality Revisited,” 

Society (July/Aug. 2014) and “Race and Mortality,” Society (Jan./Feb. 2000). 

 That, of course, is the precise issue addressed in the shorter items discussed in the 

Introduction, which involve the longstanding belief of the government that reducing adverse 

lending, school discipline, criminal justice, and employment outcomes will tend to reduce 

relative racial/ethnic and other demographic differences in rates of experiencing those outcomes. 

But the belief persists because it is shared by a larger proportion of the scientific community, 

possibly close to all it. And the belief exists within the scientific community with regard to 

every type of adverse outcome. 

 There are two corollaries to the above-described pattern by which relative differences 

tend to be affected by the prevalence of an outcome.38 One corollary is best described in the 

following terms with reference to a universe comprised of two groups: reducing the frequency 

of an outcome tends cause the group more susceptible to an outcome to make up a larger 

proportion of persons experiencing the outcome, and a larger proportion of persons failing to 

experience, than it previously did. For example, reducing poverty will tend to cause more poorer 

groups to make up both a larger proportion of the poor and of the non-poor, as shown, for 

example, in Table 1 of the 2006 Chance editorial (which underlies Table 2 below). Reducing 

public school suspensions and adverse criminal justice outcomes will tend to increase the 

proportion racial minorities and other disadvantaged groups make up of persons experiencing 

those outcome (as well as increase the proportions such groups make up of persons failing to 

experience such outcomes).39

 The appraisal of demographic disparities based on differences between the proportion a 

group comprises of persons experiencing an outcome and the proportion it comprises of persons 

actually experiencing the outcome is especially common in recent discussions of school 

discipline and criminal justice outcomes. Almost universally discussions of such differences 

reflect an expectation that reducing the frequency of adverse outcomes will tend to decrease the 

proportion disadvantaged groups make up of persons experiencing those outcomes. In addition 

to the 2016 items in The Hill and the Federalist Society Blog mentioned in the Introduction, see 

the letters to Oklahoma City School District (Sept. 20, 2016), Antioch Unified School District

(Sept. 9, 2016), American Statistical Association II (July 25, 2016), City of Boulder, Colorado

 

38 I sometimes refer to these corollaries as manifestations of the pattern rather than corollaries. I have also varied the 

order in which I present them. Compare the October 2015 ASA letter (at 9-10) with the UMMS seminar (slides 72-

80). 

39 The pattern whereby reducing the frequency of an adverse outcome tends to increase the proportions 

disadvantaged groups make up of persons experiencing it and failing to experience it could just as well be termed 

the cause of the pattern of relative differences. For it is the fact that the disadvantaged group tends to comprise up a 

larger proportion of persons below and above each increasingly lower point on a continuum of a quantifiable factor 

associated with the likelihood of experiencing the favorable outcome that underlies the described pattern of relative 

differences. See Table 1 of the 2006 Chance editorial. My earliest treatments of this subject principally addressed 

perceptions about disproportionate representations of certain groups among persons experiencing an adverse 

outcome. See “The ‘Feminization of Poverty’ is Misunderstood,” Plain Dealer ( Nov 11, 1987); “An Issue of 

Numbers,” National Law Journal (Mar. 5, 1990) and “The Perils of Provocative Statistics,” Public Interest (Winter 

1991). 

1458 The Promise of Evidence-Based Policymaking
19 

(Mar. 5, 2016), Houston Independent School District (Jan. 5, 2016), McKinney, Texas 

Independent School District (Aug. 31, 2015), Department of Health and Human Services and 

Department of Education (Aug. 24, 2015), City of Minneapolis, Minnesota (June 8, 2015), Texas 

Appleseed (Apr. 7, 2015), and United States Department of Justice and City of Ferguson, 

Missouri (Mar. 9, 2015). As explained above, that expectation is the exact opposite of reality. 

The fact that, even apart from the implications of the patterns described here, it is impossible to 

soundly analyze demographic differences on the basis of a comparison of the proportion a group 

comprises of persons potentially experiencing an outcome and the proportion it comprises of 

persons actually experiencing the outcome, is discussed in Section C infra. 

 A second corollary to the described pattern of relative differences is that as an outcome 

changes in prevalence, including when a factor or intervention causes an outcome rate to change, 

the group with the lower baseline rate for the outcome will tend to experience a larger 

proportionate change in the outcome than the other group while the other group will tend to 

experience a larger proportionate change in the opposite outcome. As discussed previously, the 

fact that relative difference in mortgage rejection rates (or poor self-rated health) tend to be 

larger, while relative racial differences in mortgage approval rates (or good self-rated health) 

tend to be smaller, among higher-income groups than lower-income groups may also be regarded 

as reflecting the fact that having high income tends to causes a larger proportionate reduction in 

mortgage rejection rates (or rates of poor self-rated health) among whites than blacks while 

causing a larger proportionate increase in mortgage approval rates (or rates of good self-rated 

health) among blacks than whites. Every analysis of subgroup effects based on the comparative 

size of a relative effect (which is to say almost every analysis of subgroup effects/interaction) has 

been fundamentally unsound as a result of the failure to understand this pattern. See "Race and 

Mortality Revisited" at 339-341. The fact that, irrespective of the patterns described here, it is 

illogical to identify a subgroup effect on the basis of differing proportionate effects on different 

baseline rates is discussed in Section D infra. 

 As with the described pattern of relative differences, because other factors are also 

influencing observed patterns, the two corollaries will not always be observed. But in all 

situations where the described pattern of relative differences is observed, the corollaries will 

necessarily be observed as well. 

*** 

 The pattern whereby the rarer an outcome the greater tends to be the relative difference in 

experiencing it and smaller tends to be the relative difference in avoiding it, as well as its two 

corollaries, can be easily illustrated with normally distributed test score data. Table 1 below, 

which is a version of Table 1 of "Race and Mortality Revisited" (at 329) and which reflects the 

same hypothetical employed in the shorter articles listed in the Introduction, is based on a 

situation where the means of normal test score distributions of an advantaged group (AG) and a 

disadvantaged group (DG) differ by half a standard deviation and both distributions have the 

same standard deviation. In addition to showing the pass and fail rates of each group, the table 

shows the ratio of AG’s pass rate to DG’s pass rate and the ratio of DG’s fail rate to AG’s fail 

The Promise of Evidence-Based Policymaking 1459
20 

rate at each cutoff (the third and fourth last columns).40 Based on a situation where AG and DG 

each make up half of the test takers, the final two columns show the proportion DG makes up of 

persons who pass and persons who fail at each cutoff. 

Table 1. Illustration of effects on relative differences in pass and fail rates of lowering a 

cutoff from a point where 80% of AG passes to a point where 95% of AG passes, with 

proportions DG comprises of persons who pass and of persons who fail (when mean scores 

differ by approximately half a standard deviation and DG comprises 50% of test takers) 

Cutoff AG Pass DG Pass AG Fail DG Fail AG/DG 

Pass Ratio

DG/AG 

Fail Ratio

DG 

Prop of 

Pass 

DG 

Prop of 

Fail 

High 80% 63% 20% 37% 1.27 1.85 44% 65% 

Low 95% 87% 5% 13% 1.09 2.60 48% 72% 

 According to the specifications underlying the table, at the cutoff where 80% of AG 

passes the test, approximately 63% of DG would pass the test (with corresponding failure rates 

of 20% for AG and 37% for DG). The ratio of AG’s pass rate to DG’s pass rate would be 1.27 

while the ratio of DG’s fail rate to AG’s fail rate would be 1.85. 

 When the cutoff is lowered to the point where the pass rate for AG is 95%, the pass rate 

for DG would be approximately 87% (with corresponding failure rates of 5% for AG and 13% 

for DG). The ratio of AG’s pass rate to DG’s pass rate would thus decrease to 1.09 (from 1.27), 

while the ratio of DG’s fail rate to AG’s fail rate would increase to 2.60 (from 1.85). That is, the 

relative difference in the outcome that is reduced in frequency (test failure) increases (from 85% 

to 160%), while the relative difference in the increasing outcome (test passage) declines (from 

27% to 9%).41 

 The final two columns also illustrate the first corollary to the described pattern by which 

relative differences tend to be affected by the prevalence of an outcome. Lowering the cutoff 

and reducing the frequency of test failure caused an increase in the proportion DG makes up of 

those who pass the test (from 48% to 52%) and the proportion DG makes up of persons who fail 

the test (from 65% to 72%). Because the proportion DG makes up of persons taking the test is 

 

40 While I commonly refer to patterns of relative differences in these comments, the table actually presents rate 

ratios. The relative difference is the rate ratio minus 1 where the rate ratio is above 1 and 1 minus the rate ratio 

where the rate ratio is below one. In the former case, the larger the rate ratio, the larger the relative difference; in the 

latter case, the smaller the rate ratio, the larger the relative difference. One should be careful not to mistakenly refer 

to the rate ratio as the relative difference. But the distinction between the two terms is not pertinent to the discussion 

here of patterns by which the two relative differences tend to be affected by the prevalence of an outcome. In recent 

years I commonly present the rate ratios for both outcomes with the larger figure in the numerator, in which case, as 

to both outcomes, the larger the rate ratio, the larger the relative difference. In the 1994 Chance article and the 2006 

Chance editorial I used the disadvantaged group’s rate as the numerator in both ratios (which is the approach of the 

“four-fifths” or “80 percent” rule for identifying disparate impact under the Uniform Guideline for Employee 

Selection Procedures). Choice of numerator in the ratio, however, has no bearing on the patterns by which the two 

relative differences tend to be affected by the prevalence of the outcome. 

41 I commonly speak of the decreasing outcome first because failures of understanding typically involve that 

outcome. But, partly out of convention and partly to be consistent with certain earlier illustrations, Table l presents 

the increasing outcome first and discusses that outcome first save in the sentence to which this note is attached. 

1460 The Promise of Evidence-Based Policymaking
21 

unaffected by the cutoff, lowering the cutoff would increase all measures of differences between 

the proportion DG makes up of test takers and the proportion it makes up of persons who fail 

(while reducing all measures of difference between the proportion DG makes up of test takers 

and the proportion it makes up of persons who pass). 

 The table also illustrates the second corollary to the described pattern by which relative 

differences tend to be affected by the prevalence of an outcome. Lowering the cutoff caused a 

larger proportionate decline in failure rates for AG (the group with the lower baseline failure 

rate) while causing a larger proportionate increase in the pass rate for DG (the group with the 

lower baseline pass rate). That is, lowering the cutoff caused failure rates to decrease by 75% for 

AG but only 65% for DG, while causing pass rates to increase by 38% for DG but only 19% for 

AG. Raising the cutoff back to the original point would similarly show a larger proportionate 

increase in the failure rate for AG but a larger proportionate decrease in the pass rate for DG, 

thus causing the relative difference in failure rates to decrease and the relative difference in pass 

rates to increase. 

 Table 1 can also illustrate the pattern whereby relative differences in the adverse outcome 

tend to be larger, while relative differences in the corresponding favorable outcome tend to be 

smaller, in advantaged populations/subpopulations/settings than in disadvantaged 

populations/subpopulations/settings. The reader need simply regard the second row as reflecting 

the situation in the advantaged population/subpopulation/setting (where the adverse outcome is 

less common) and the first row as reflecting the situation in the disadvantaged 

population/subpopulation/setting (where the adverse outcome is more common). In terms of the 

point of the preceding paragraph, this means that being in the advantaged 

population/subpopulations/setting tends to reduce adverse outcome rates proportionately more 

for advantaged groups than disadvantaged groups while increasing favorable outcome rates 

proportionately more for disadvantaged groups than advantaged groups. To make the last point 

more concrete, having high income or high education tends to reduce adverse outcome rates 

proportionately more for whites than disadvantaged racial minorities, while increasing favorable 

outcome rates proportionately more for disadvantaged racial minorities than whites. 

 The pattern of relative differences in pass and fail rates shown in Table 1 exists across 

the full range of test scores. Figure 1, which employs the same specifications as Table 1, shows 

the effects on the two relative differences of lowering a cutoff from a point where almost 

everyone fails to a point where almost every passes. The relative difference in the decreasing 

outcome (test failure) consistently increases, while the relative difference in the increasing 

outcome (test passage) consistently decreases. 

 

The Promise of Evidence-Based Policymaking 1461
22 

Figure 1. Ratios of (1) DG Fail Rate to AG Fail Rate, (2) AG Pass Rate to DG Pass Rate, at 

Cutoffs Defined by AG Failure Rates 

 The two corollaries to the above pattern will similarly be observed across the full range 

of prevalence of the outcome. 

 Table 2, which is an abbreviated version of Table 2 or "Race and Mortality Revisited" 

and Table 1 of the 2006 Chance editorial, shows proportions of the white and black populations 

living on income below and above the poverty line and 75% of the poverty line, along with the 

ratios of the black to white ratios of falling below each point and the white to black ratios of 

falling above the point. The table also includes the percentage point difference between rates, 

though I will make only limited reference to it in this section. 

Table 2. Rates at which white and blacks fall above and below 125%, 100%, and 75% of 

the poverty line, with measures of differences between rates (2004) 

Percent of 

Poverty 

Line 

Percent of 

Whites

Below 

Percent of 

Blacks 

Below 

Percent of 

Whites

Above 

Percent of 

Blacks 

Above 

B/W Below 

Ratio 

W/B Above 

Ratio 

Percentage 

Point Diff 

100% 10.8% 24.7% 89.2% 75.3% 2.29 1.18 14 

75% 7.2% 17.8% 92.8% 82.2% 2.48 1.13 11 

 Movement from the first to the second row shows the implications of reducing poverty 

such as to enable everyone living on incomes above 75% of the poverty line to escape poverty. 

The black-white poverty ratio would increase from 2.29 to 2.48 (i.e., the relative difference 

would increase from 129% to 148%), while the ratio of the white rate of avoiding poverty to the 

black rates of avoiding poverty would decline from 1.18 to 1.13 (i.e., relative difference would 

decreases from 18% to 13%). And the absolute difference would decline from 14 to 11 

percentage points. 

Cutoff Defined by AG Fail Rate 

1 

2 

3 

4 

99 90 80 70 60 50 40 30 20 10 1 

Ratios 

(1) DG Fail Rate/AG 

Fail Rate 

(2) AG Pass Rate/DG 

Pass Rate 

1462 The Promise of Evidence-Based Policymaking
23 

 As shown in Table 2 of "Race and Mortality Revisited" and Table 1 of 2006 Chance 

editorial, one would observe the opposite patterns of changes in measures if poverty were to 

increase such as to pull into poverty all persons living on incomes below 125% of the poverty 

line. Such pattern may also be observed in Table 2 above if one simply envisions that after the 

initial reduction in poverty simulated by the movement from the first to the second row of the 

table, poverty rose to its prior level. 

 The table does not present the proportion blacks make up of the combined black and 

white population below and above each point. Such information may be found in Table 1 of the 

2006 Chance editorial. But, as explained with regard to test score illustration in Table 1 above, 

given the data in Table 2 above, the proportions blacks comprise of the combined black and 

white populations falling both below and above 75% of the poverty line would necessarily be 

greater than the proportions falling below and above the poverty line itself. As discussed in note 

39, the fact that such is the case may be deemed the reason for the patterns of relative differences 

shown in the table. 

 In "Race and Mortality Revisited" I discuss with regard to its Table 2 whether there could 

be justification for devoting any resources to exploring why, during a period when there occurred 

a general increase or decrease in poverty, the relative difference between black and white 

poverty rates or rates of avoiding poverty changed – say, for example, to attempt to determine 

whether a particular administration’s civil rights enforcement policies may have played some 

role – without consideration of the patterns in Table 2. The obvious answer is that there could be 

no justification. 

 The same point holds with regard to any exploration of reasons observed patterns of 

differences in the proportionate changes in the black and white poverty rates or rates of avoiding 

poverty, as might commonly be done in discussions that one group or another was 

disproportionately affected by the general change in poverty. The point also holds with regard to 

exploration of reasons for changes in the proportion blacks comprise of the poor or of the nonpoor. And it would hold as well with regard to explorations of reasons for observed pattern of 

changes in absolute differences between poverty rates. 

 With regard to the potential focus on differences in the relative (or absolute) change in 

each group’s rate in the case of a decline (or increase) in poverty or anything else, the following 

should be borne in mind. Given the manner in which demographic differences are commonly 

analyzed, in the case of the reduction in poverty simulated by the movement from the first to the 

second row of Table 2, it would not be surprising (a) for researchers exploring the cause of the 

increase in the relative difference in poverty rates to attribute the increase to the fact that the 

white poverty rate fell proportionately more than the black poverty rate or (b) for researchers 

exploring the cause of the decrease in the absolute difference to attribute the decrease to the fact 

that the black poverty rate fell by a greater absolute amount than the white rate. That in fact is 

what is done in the discussion of increasing relative socioeconomic differences, but declining 

absolute socioeconomic differences, in mortality in the BMJ article42 that is the subject of my 

Comment on Mackenbach BMJ 2016 (2016). 

 

42 Mackenbach JP, Kulhánová I, Artnik B, et al. Changes in mortality inequalities over two decades: register based 

study of European countries. BMJ 2016;353:i1732. 

The Promise of Evidence-Based Policymaking 1463
24 

 Some observers might regard such information as providing insight into causes of 

increasing or decreasing disparities. But as with discussion of the changes in the size of the 

disparities themselves, such discussion is providing nothing about whether the observed pattern 

is anything other than the consequence of the change in the prevalence of the outcome. That is, 

such discussion merely describes a mathematical relationship; it provides no information about 

whether the differences in the circumstances of the groups have actually changed or about the 

role of policies in effecting changes in the difference in the circumstances of the groups. 

 As suggested above, however, research examining patterns of changes in demographic 

differences in poverty or any other matter has never taken patterns like those in reflected in Table 

2 into account. Thus, it has not attempted (that is, attempted in a sound manner)to identify things 

that it might be actually be useful for policymakers to know. 

*** 

 In the main Introduction to these comments and the introduction to this part, I mentioned 

that the National Center for Health Statistics (NCHS) came to understand more than a decade 

ago that determinations of whether health and healthcare disparities were increasing would 

commonly turn on whether one examined relative differences in a favorable outcome or relative 

differences in the corresponding adverse outcome. This occurred as a result of my bringing to 

the attention of the NCHS statisticians in 2002 “Race and Mortality,” Society (Jan./Feb. 2000) 

and “Divining Difference,” Chance (Fall 1994). This was not, however, the agency’s first 

recognition of the pattern. As discussed at the end of "Race and Mortality” the director of the 

agency had come to recognize the pattern several years earlier, which had occurred as a result of 

my contacts to the directors of the Race and Health Initiative.43 But as of 2002 the information 

had not found its way to the NCHS statisticians who developed policy on the measurement of 

health disparities. 

 In any case, the communications directly to NCHS statisticians led their publishing four 

official or unofficial papers between 2004 and 2009 discussing that determination of directions 

of change in health and healthcare disparities would commonly turn on which relative difference 

one examined. The most important of these was a 2005 monograph titled “Methodological 

Issues in Measuring Health Disparities” (authored by NCHS personnel and other experienced 

health disparities researchers).44 Each of the documents simply concluded that in order to 

 

43 See the January 20, 1999 letter from NCHS Director Edward J. Sondik. See also my January 25, 1999 letter to 

Director Sondik explaining that the absolute difference between rates did not provide a solution to the problem of 

the contradictions in interpretations provided by the two relative differences because the absolute difference tended 

also to be affected by the prevalence of an outcome. 

44 Keppel K, Pamuk E, Lynch J, et al. Methodological issues in measuring health disparities. Vital Health Stat. 

2005;2(141):116. Other papers include: (a) Keppel KG, Pearcy JN, Klein RJ. 2004. Measuring progress in Healthy 

People 2010. Healthy People statistical notes. No. 25. Hyattsville, Md.: National Center for Health Statistics; (b) 

Keppel KG, Pearcy JN. 2005. Measuring relative disparities in terms of adverse events. J Public Health Manag 

Pract, 11(6):479–483; and (c) Keppel KG, Pearcy JN. 2009, Healthy People 2010: Measuring disparities in health. 

1464 The Promise of Evidence-Based Policymaking
25 

promote consistency, for purposes of measuring health disparities reduction goals in Healthy 

People 2010, both health and healthcare disparities would be measured in terms of relative 

differences in adverse outcomes (meaning, in the case of healthcare, relative differences in nonreceipt of care). 

 I have in many places criticized the NCHS manner of dealing with this issues (both as to 

health and healthcare) for the failure to recognize that the purpose of health and healthcare 

disparities research is to understand processes and inform policies, which cannot be achieved by 

arbitrarily choosing a measure that tends to indicate that disparities are increasing over one that 

tends to indicate that they are decreasing (or vice-versa). As indicated above, that purpose can 

only be achieved by taking into account the ways the measures employed in such research tend 

to be affected by the prevalence of an outcome. 

 Some implications of the NCHS approach are discussed in "Race and Mortality 

Revisited" and the FCSM paper, especially with regard to (a) a 2008 study where, ignoring or 

unaware of NCHS guidance, the authors measured disparities in terms of relative difference in 

receipt of immunization and found a dramatic increase in immunization rates to be associated 

with dramatically reduced racial/ethnic disparities (while NCHS, relying on relative differences 

in non-receipt of immunization, would have reached opposite conclusions) and (b) a 2009 study 

where, specifically relying on NCHS guidance, the authors measured disparities in terms of 

relative differences in non-receipt of mammography and found dramatic increases in 

mammography rates to be associated with dramatic increases in area socioeconomic disparities 

(while those relying on relative differences in receipt of mammography would have reached 

opposite conclusions). See discussion of Table 4 in "“Race and Mortality Revisited” and Tables 

3 and 7 in the FCSM paper. The two papers discuss a number of similar situations where 

analyses of changes in healthcare disparities would reach opposite results depending on whether 

researchers followed or ignored (usually meaning were unaware of) NCHS guidance, as well as a 

situation where NCHS’s views as to the largest and smallest disparities in skilled birth 

attendance in certain low and middle income countries would be the opposite of the views of the 

World Health Organization. See Table 4 of the FCSM paper. 

 As discussed in the 2016 JPHMP commentary, however, the NCHS has now reversed its 

position regarding the measurement of healthcare disparities and now measures them in terms of 

relative differences in favorable outcomes. Thus, NCHS would now agree with the authors of 

the 2008 immunization study who ignored the earlier guidance and disagree with the authors of 

the 2009 mammography study who specifically followed NCHS guidance. Reversals of 

interpretation would apply to all situations discussed in "Race and Mortality Revisited" and the 

FCSM paper regarding healthcare disparities interpretation issues as to which the position taken 

in the 2005 NCHS monograph is pertinent. 

 Another illustration of the implications of the NCHS reversal of position with regard to 

researchers who relied on the guidance may be found at pages 5-6 of the March 8, 2016 letter to 

Stanford Center on Poverty and Inequality regarding a Poverty and Inequality Report titled 

 Chance, 22(1):6-9. See also Keppel K.G., Pearcy J.N. 2006. Response to Scanlan concerning: measuring health 

disparities in terms of adverse events. J Public Health Manag Pract, 12(3):295. 

The Promise of Evidence-Based Policymaking 1465
26 

“State of the Union” that the Center issued in early 2016. The letter discusses the way that 

authors of a portion of the report relied on NCHS guidance to measure demographic differences 

in insurance coverage in terms of relative difference in uninsurance and drew inferences about 

underlying processes based on the comparative size of those differences in different settings. 

Prior to issuance of the report, NCHS had already reversed its position such that (though the data 

were unavailable to confirm the pattern) observers relying on the revised guidance would 

commonly draw very different inferences about processes from those drawn by authors of the 

Stanford Center’s report.

 The most notable consequence of the NCHS reversal, however, is that it constitutes a 

repudiation of more than a decade of yearly National Healthcare Disparities Reports issued by 

the Agency for Healthcare Research and Quality (AHRQ). Following NCHS guidance, the 

report had relied on the guidance to measure healthcare disparities in terms of relative 

differences in non-receipt of care. This issue and whether AHRQ is yet aware of the reversal or 

its implications is discussed further in Section B. 

 While a Google search indicates that the 2005 NCHS monograph has been cited 180 

times, to my knowledge, none of the papers citing it or citing other items in which NCHS has 

recommended measuring healthcare disparities in terms of relative differences in adverse 

outcomes (excluding situations where I have cited them) has indicated an awareness that reliance 

on relative differences in adverse outcomes will commonly result in opposite conclusions from 

reliance on relative differences in the corresponding favorable outcomes.45 Other federal 

agencies involved with health and healthcare disparities research have yet to show an awareness 

that it is even possible for the two relative differences to yield opposite conclusions about 

patterns of changes in health and healthcare disparities. In consequence of the retirement of the 

principal NCHS author of the 2005 monograph, it is questionable whether more than a few 

persons at NCHS are aware that NCHS statisticians ever reached the conclusion they did in the 

monograph and the three other papers. 

 Most important, neither NCHS nor any other agency conducting research into health and 

healthcare disparities, providing guidance on such research, or funding such research has 

considered the implications of the prevalence of an outcome on measures employed in such 

research. The same holds for the private sector. 

*** 

 As discussed in the August 24, 2015 letter to the Department of Health and Human 

Services (HHS) and Department of Education (DOE), the Secretary of HHS has a view as to the 

effects or reducing adverse outcomes on relative differences in rates of experiencing those 

outcomes that is both the opposite of the view NCHS reached a decade ago and the opposite of 

 

45 See pages 26-27 of the FCSM paper regarding Healthy People 2010 guidance on measurement of healthcare 

disparities indicating that healthcare disparities will be discussed in terms of favorable outcome even though they 

are being measured in terms of relative differences in adverse outcomes and the fact that few readers would infer 

from the language that the relative difference in the adverse outcome may well be yielding an opposite conclusion 

from the relative difference in the favorable outcome. See also the slide 32 of the presentation associated with the 

FCSM paper. 

1466 The Promise of Evidence-Based Policymaking
27 

reality. Specifically, the Secretary of HHS, like the Secretary of Education, believes that 

reducing the reducing public school suspensions will tend to reduce demographic differences in 

discipline rates and the proportions disadvantaged groups make up of persons experiencing those 

outcomes. 

 The reasons that such actions will tend to have the opposite effect have been explained 

above. In addition, in point of fact, recent reductions in discipline rates have consistently been 

accompanied by increased relative differences in discipline rates (and thus the proportion more 

susceptible groups make up of those disciplined) notwithstanding that teachers and 

administrators are no doubt taking various actions that would be expected to reduce all measures 

of differences between outcome rates. See the subpages to the subpages of the Discipline 

Disparities page of jpscanlan.com discussing the situation with regard to the jurisdictions 

indicated in the title of the subpage: California Disparities, Colorado Disparities, Connecticut 

Disparities, Maryland Disparities, Minnesota Disparities, Oregon Disparities. Beaverton, OR 

Disparities, Denver Disparities, Henrico County, VA Disparities, Los Angeles SWPBS, 

Minneapolis Disparities, Montgomery County, MD Disparities, Portland, OR Disparities, St. 

Paul Disparities.

46 See also the DOE Equity Report subpage regarding a Department of 

Education report showing that relative racial differences in expulsions are larger in school 

districts without zero tolerance policies than school districts with such policies; the Suburban 

Disparities subpage regarding the greater relative racial differences in suspensions in suburban 

than central city schools; "Race and Mortality Revisited" (at 342) regarding the larger relative 

differences in suspension rates in pre-school than in K-12; and the Boston Lawyers’ Committee 

letter mentioned in note 6 regarding larger relative differences in suspensions in Massachusetts 

(which has generally low suspension rates) than nationally. 

 Some of the problems faced by particular entities as a result of the mistaken belief of the 

government or others that reducing the prevalence of adverse discipline outcomes will tend to 

reduce relative differences in rates of experiencing the outcomes, and the proportions 

disadvantaged groups make up of persons experiencing the outcomes, are discussed in the 

Oakland Agreement page and the letters to school districts listed on pages 18-19. See especially 

the letters to Oklahoma City School District (Sept. 20, 2016) regarding a recent agreement with 

the Department of Justice, the Antioch Unified School District (Sept. 9, 2016) regarding a recent 

suit brought against that district by a public interest group, and McKinney, Texas Independent 

School District (Aug. 31, 2015) regarding a public interest group’s analyses of racial/ethnic 

differences in the district. 

Similar problems faced by particular entities as a result of the government’s beliefs about 

the effect of generally reducing criminal justice outcomes on measures of racial disparity may be 

found in the letters to cities listed on pages 18-19. The particular problems facing Ferguson, 

Missouri are discussed in the letter to United States Department of Justice and City of Ferguson, 

 

46 Reportage of situations where general reductions in discipline rates have been accompanied by reduced racial 

differences in discipline have generally pertained to studies that measured disparities in terms of absolute differences 

between rates. As discussed in the letter to the Antioch Unified School District, absolute differences between rates 

tend to decline when outcomes in the rate ranges commonly observed for adverse school discipline outcomes 

generally decline. 

The Promise of Evidence-Based Policymaking 1467
28 

Missouri (Mar. 9, 2015) and in the “Things DoJ doesn’t know about racial disparities in 

Ferguson,” The Hill (Feb. 22, 2016). See also my Submission re Ferguson Consent Decree (Apr. 

11, 2016) regarding specific issues in the consent decree resolving the Department of Justice’s 

suit against the city. The recent “Misunderstanding of Statistics Confounds Analyses of 

Criminal Justice Issues in Baltimore and Voter ID Issues in Texas and North Carolina,” 

Federalist Society Blog (Oct. 3, 2016) gives an indication of similar problems facing Baltimore, 

Maryland. 

*** 

 A few concluding points are in order regarding the two relative differences. Congress 

has no better an understanding of these issues than the federal law enforcement agencies or the 

federal agencies involved with health and healthcare disparities research. In fact, as discussed in 

"Race and Mortality Revisited" (at 342) and the Disabilities – Public Law 104-446 subpage of 

the Discipline Disparities page, the Individuals with Disabilities Education Act mandates that 

recipients of federal assistance with “significant discrepancies” in long-term suspensions of 

students with and without disabilities, which are commonly measured in terms of relative 

differences in suspension rates, must consider approaches to discipline of the type that generally 

reduce suspension rates. The implementation of those approaches thus will tend to increase the 

discrepancies according to the standard approach to measuring them. See the Keep Kids in 

School Act subpage of the Discipline Disparities regarding proposed legislation contemplating 

that, in accordance with beliefs of the Department of Justice, Education, and Health and Human 

Services, generally reducing discipline rates will reduce relative racial differences in discipline 

rates. 

 The Civil Rights Actually of 1991 provides that employers must justify practices having 

a disparate impact on protected groups and implement less discriminatory alternatives to 

practices that can be justified. In enacting the statute Congress had no understanding that 

relaxing standards or otherwise reducing adverse employment outcomes, and increasing the 

corresponding favorable outcomes, would tend to increase relative differences in the adverse 

outcomes while reducing relative differences in the favorable outcomes. Congress continues to 

have no understanding that it is even possible for the two relative differences to change in the 

opposite outcome as the prevalence of an outcome changes. Meanwhile it remains possible for 

courts either to measure disparate impacts in terms of relative differences in favorable outcome 

or relative differences in adverse outcomes. No court has yet recognized that it is possible for 

the two relative differences to change in opposite directions as the prevalence of an outcome 

changes. See the letter to Federal Judicial Center (July 7, 2016). See the Less Discriminatory 

Alternative - Substantive subpage of the Disparate Impact. See generally the main Disparate 

Impact and each of its subpage regarding the lack of understanding of the relationship between 

the frequency of an outcome and measures of differences between outcome rates among all 

entities having a role in the enforcement of laws against employment discrimination. 

 Federal legislations impose a variety of obligations to monitor demographic differences 

in certain outcomes. The obligation to monitor demographic differences in assignment to special 

education programs imposes what would seem to be an very large administrative burden on 

states. Yet, neither the Congress that imposes the obligations, the Department of Education that 

1468 The Promise of Evidence-Based Policymaking
29 

promulgates regulations implementing or expanding those obligations, the Government 

Accountability Office that attempts to monitor the actions of the Department of Education, nor 

the divisions of the state government that attempt to fulfill those obligations are yet aware that 

actions that generally reduce assignment to special education tend to increase relative racial 

differences in assignment rates. 

 In sum, the matters discussed above only suggest the scope of activities of government 

and private actors where failure to understand the ways relative differences in outcome rates tend 

to be affected by the prevalence of an outcome undermine programs aimed at monitoring or 

ameliorating demographic differences. 

 B. Absolute Differences and the Odds Ratios 

 Appraisals of differences in the circumstances of advantaged and disadvantaged groups 

reflected by their outcome rates in terms of absolute (percentage point) differences between rates 

and differences measured by odds ratios are unaffected by which outcome one examines. But in 

order for a measure to effectively quantify the strength of the forces causing outcome rates to 

differ, it must remain unchanged as there occurs a general change in the prevalence of an 

outcome akin to that effected by the lowering of a test cutoff. And, like the two relative 

differences, absolute differences and odds ratios also tend to be affected by the frequency of an 

outcome, though in a more complicated way than the two relative differences. 

 Roughly, as uncommon outcomes (below 50% for both groups) become more common, 

absolute differences between rates tend to increase; as common outcomes (above 50% for both 

groups) become even more common, absolute differences tend to decrease. The frequencydriven direction of change of the absolute difference is harder to predict when the outcome is 

neither common nor uncommon or moves between categories of prevalence during a period 

examined. In the situations reflected in Tables 1 and 2, the rates of experiencing the favorable 

and corresponding adverse outcomes happen to be in ranges where movements from the first 

row to the second row would reduce absolute differences, while movement from the second 

row to the first row would increase absolute differences. The school discipline and criminal 

justice outcomes where focus is generally on relative differences in adverse outcomes or the 

proportions disadvantaged groups make up of persons experiencing those outcomes – both of 

which tend to increase as the outcome generally decreases – also happen to be in ranges where 

reductions in the adverse outcome will tend to reduce absolute differences between rates. 

 The absolute difference and both relative differences may all change in the same 

direction as the frequency of an outcome changes (in which case the difference measured by 

the odds ratio will also change in the same direction as the other measures). But in the 

common situation where all measures do not change in the same direction as the frequency of 

an outcome changes, the absolute difference will tend to change in the same direction as the 

smaller relative difference. Observers who rely on relative differences to appraise the 

difference in the circumstances of two groups reflected by their differing outcome rates tend 

usually to rely on the larger of the two relative differences (as discussed above). 47 Thus, there 

 

47 The tendency to rely on the larger of the two relative differences sometimes causes researchers to rely both on 

relative differences in favorable outcomes and on relative differences in adverse outcomes in the same study. See 

the Immunization Disparities page of jpscanlan.com regarding a study that examined immunization disparities in 

The Promise of Evidence-Based Policymaking 1469
30 

exists a systematic tendency for observers relying on the absolute difference to reach opposite 

conclusions about directions of changes in the size of demographic differences, or about the 

comparative size of the differences in different settings, from observers relying on a relative 

difference. 

 Further, anytime an observer notes that the absolute difference has changed in a 

different direction from the relative difference the observer happens to be examining, the 

unmentioned relative difference will necessarily have changed in the opposite direction of the 

mentioned relative difference and the same direction as the absolute difference. Thus, 

observers maintaining that one must make a value judgment in choosing between the relative 

difference and the absolute difference with respect to the appraisal of a particular change in 

some demographic disparity over time have already made a choice (usually without thought) to 

rely on the relative difference that yields an opposite conclusion from the absolute difference 

rather than the relative difference that yields the same conclusion as the absolute difference. 

 As the frequency of an outcome changes, and all measures do not change in the same 

direction, the difference measured by the odds ratio tends to change in the opposite direction of 

the absolute difference and in the same direction as the larger relative difference. 

 These patterns are illustrated in Figures 2 and 3, which are based on the same 

specifications as Figure 1, and similarly show the implications of lowering a cutoff from a 

point where almost everyone fails to a point where almost everyone passes. Figure 2 presents 

the pattern for the absolute difference. Figure 3 presents the pattern for the ratio of the 

disadvantaged group’s odds of failing the test to the advantaged group’s odds of failing the 

test.48 But in order to illustrate the relationship with the two relative differences, a line for the 

odds ratio is simply added to the lines for the two relative differences previously shown in 

Figure 1. 

 terms of relative differences in the adverse outcome for receipt/non-receipt of any immunization and relative 

differences in the favorable outcome for receipt/non-receipt of full immunization and the McKinsey Achievement 

Gap Study subpage of the Educational Disparities page regarding a study where the authors relied on the relative 

difference in the adverse outcome for reaching/failing to reach the basic proficiency level but on the relative 

difference in the favorable outcome for reaching/failing to reach the advanced proficiency level. In both cases the 

authors relied on the larger of the two relative differences with respect to each of the subjects examined and without 

recognizing that general increases in favorable outcome would tend to increase relative differences in the adverse 

outcomes while reducing relative differences in the favorable outcomes. See also the letter to New York City Center 

for Innovation through Data Intelligence (June 6, 2016). 

 

48 There are four odds ratios depending on which outcome is examined and which group’s odds is used as the 

numerator of the ratio. Two yield one value and two yield a value that is the reciprocal of the first value. The ratio 

of DG’s failure odds to AG’s failure odds is the same as the ratio of AG’s pass odds to DG’s pass odds. Thus, the 

odds ratios underlying Figure 3 are also the ratios of AG’s odds of passing the test to DG’s odds of passing the test.

1470 The Promise of Evidence-Based Policymaking
31 

Figure 2. Absolute differences between rates of AG and DG pass (or fail) rates at various 

cutoff points defined by AG fail rate 

Figure 3. Ratios of (1) DG fail rate to AG fail rate, (2) AG pass rate to DG pass rate, (3) 

DG failure odds to AG failure odds 

 Table 3 below, which is based on the same specifications as Table 1 and Figures 1 to 3, 

presents favorable outcome rates at four prevalence levels (benchmarked on the advantaged 

group’s favorable outcome rate), along with rate ratios for the favorable and adverse outcomes as 

well as absolute differences and odds ratios. The parenthetical numbers indicate the ways one 

would rank the size of the disparity between the situation of the advantaged and disadvantaged 

groups according to each of the four measures. Those numbers show that rankings according to 

relative differences in favorable outcomes are the opposite of rankings according to relative 

differences in adverse outcomes. They also show that rankings according to absolute differences 

and odds ratios, while different from the rankings according to either relative difference, are the 

opposite of each other. 

0 

10 

20 

99 90 80 70 60 50 40 30 20 10 1 

Percentage Points 

Cutoffs Defined by AG Fail Rate 

Abs Df 

1 

2 

3 

4 

99 90 80 70 60 50 40 30 20 10 1 

Ratios 

(1) DG Fail Rate/AG 

Fail Rate 

(2) AG Pass Rate/DG 

Pass Rate 

(3) DG Fail Odds/AG 

Fail Odds 

Cutoffs Defined by AG Fail Rate 

The Promise of Evidence-Based Policymaking 1471
32 

Table 3. Favorable outcome rates of advantaged group (AG) and disadvantaged group 

(DG) at four settings with different favorable outcome frequencies, with measures of 

difference 

 AG Fav 

Rate 

DG Fav 

Rate 

AG/DG 

Fav Ratio 

DG/AG 

Adv Ratio 

 Abs Df 

 (Perc Pnts) 

Odds 

Ratio 

A 20.0% 9.0% 2.22 (1) 1.14 (4) 11.0 (4) 2.53 (1) 

B 40.0% 22.6% 1.77 (2) 1.29 (3) 17.4 (2) 2.28 (3) 

C 70.0% 51.0% 1.37 (3) 1.63 (2) 19.0 (1) 2.24 (4) 

D 80.0% 63.4% 1.26 (4) 1.83 (1) 16.6 (3) 2.31 (2) 

 A version of this table appears as Table 5 (at 335) of "Race and Mortality Revisited," 

where, as in many other places, it is used to refute the notion that choice of a measure in analyses 

of demographic differences in health or healthcare outcomes involves a value judgment. I will 

rest here on the treatment of that issue in "Race and Mortality Revisited" (at 335-336) and other 

places. 

 But I also note the following. In the case of Table 5 in "Race and Mortality Revisited" I 

cast the matter cast in terms of situations where one has to rank, according to the degree of 

discrimination or the likelihood of discrimination, four employers or one employer at four points 

in time. I cast the matter that way in order to cause the reader to recognize that there can be only 

one answer to the question of whether the forces causing the outcome rates to differ are larger in 

one situation than another and that, while the answer may be difficult to divine, value judgments 

have can have no role in the matter. 

 The same considerations apply when the purpose of examining patterns of differences in 

outcome rates is to inform policies aimed at mitigating differences in the circumstances of two 

groups. Further, anyone inclined to believe that the particular measure ought to be the focus in 

the appraisal of demographic differences irrespective of prevalence considerations ought to 

recognize that policymakers have a great interest in knowing whether policies will cause that 

measure to change more than, less then, or in a different direction from what would be likely to 

occur solely as a consequence of changes in the prevalence of an outcome. 

 But I include Table 3 here principally to illustrate certain points about the absolute 

difference in contexts where observers commonly rely on that measure. The premise of the 

table, as reflected by its specifications, is that there is no rational basis to maintain that the 

strength of the forces causing the favorable (or adverse) outcome rates of the advantaged and 

disadvantaged groups to differ varies among the rows. 

 Areas of research or commentary where observers commonly or increasingly rely on 

absolute differences between rates include appraisals of demographic differences in healthcare 

and academic proficiency. 

 In the case of healthcare, increases in rates of appropriate care will tend to increase 

absolute differences between rates of advantaged and disadvantaged groups for uncommon 

procedures/outcome, while reducing absolute differences for common procedures/outcomes. 

1472 The Promise of Evidence-Based Policymaking
33 

Such patterns are reflected by movement from Row A to Row B of Table 3 for the former 

procedures/outcomes and from Row C to Row D for the latter procedures/outcomes. When 

healthcare outcomes increase from being fairly uncommon to being very common (as in the 

case of certain vaccines and screening practices) the absolute differences will tend to increase 

for a time and then decrease. 

 Further, higher-performing hospitals (which tend to have generally higher appropriate 

care rates than lower-performing hospitals) will tend to show larger absolute differences 

between appropriate care rates of advantaged and disadvantaged groups than lower-performing 

hospitals for procedures/outcomes where rates are generally low, while showing smaller 

absolute differences for procedures/outcomes where rates are generally high. This pattern, too, 

is illustrated in Table 3 if Rows A and B are regarded as the lower- and higher-performing 

hospitals with respect to procedures/outcomes with generally low rates and Rows C and D are 

regarded as such hospitals with respect to procedures/outcomes with generally high rates. 

 Odds ratios would tend to show patterns that are the opposite of those just described for 

absolute differences. In situations where observers draw initial conclusions about such things 

as changes in the size of healthcare disparities over time on the basis of absolute differences 

between rates, and then employ logistic regression to adjust for the role of possible 

confounders, the odds ratio yielded by the adjustment will tend to show a pattern that is the 

opposite of that shown by the absolute difference.49 

 Observers relying on absolute differences to measure healthcare disparities, however, 

have yet to recognize the ways absolute differences tend to be affected by the frequency of an 

outcome or that other measures would tend to systematically yield opposite results (i.e., the 

larger relative difference and the odds ratio) or consistent results (i.e., the smaller relative 

difference). Commonly observers relying on absolute differences, like those relying on one of 

the two relative differences, show no awareness that choice of measure is of any consequence. 

 The confusion arising from the failure to understand patterns by which absolute 

differences between health and healthcare outcome rates of advantaged and disadvantaged 

groups tend to be affected by the prevalence an outcome is vast. "Race and Mortality Revisited" 

(at 337-339) discusses the way the failure to understand these patterns caused Massachusetts 

unwisely to include a health disparities element in its Medicaid pay-for-performance program 

and then to measures disparities in a way that, by generally favoring higher-performing hospital 

for reasons unrelated to actual differences in equity of care, will tend to increase healthcare 

disparities. 

 

49 Though I give only limited attention to the odds ratio, I recognize that some might deem it a measure essentially 

unaffected by the prevalence of an outcome (even though it seems principally to be employed as a convenient 

substitute for a rate ratio). I discuss the potential for the odds ratio to be an effective measure in note 22 (at 14) of 

letter to the Antioch Unified School District (Sept. 9, 2016), but leave the utility of the measure, like the utility of 

the measure I discuss in "Race and Mortality Revisited," as a matter for the Commission to consider in its efforts to 

reform the analyses of demographic differences with regard to the way such analyses affect appraisals of policies 

and programs. I merely note here that, as with other measures, researchers should be required to address whether 

and how the differences measured by the odds ratio tends to be affected by the prevalence on an outcome in the case 

of the matter under study. The same holds for any regression approach to appraising demographic differences. 

The Promise of Evidence-Based Policymaking 1473
34 

 A useful illustration of the way prevalence issues are ignored may be found in a group of 

articles appearing in the August 18, 2015 issue of the New England Journal of Medicine 

(NEJM), along with a commentary thereon and a letter to the editors by three authors of the 2005 

NCHS monograph. A study by Vaccarino et al.50 relied on relative differences in favorable 

healthcare outcomes (though relative differences in adverse outcomes for health status issues) 

with regard to outcome rates that were not changing much in overall prevalence during the 

period examined; and, as commonly happens when overall prevalence does not change much, the 

study found little to remark on with respect to changes in disparities over time. A study by Jha et 

al.51 relied on absolute differences between rate in examining disparities in rates of receiving 

certain fairly uncommon procedures that were generally increasing in overall prevalence; and, as 

commonly happens when outcome rates in the rate ranges at issue are generally increasing, the 

authors found absolute differences usually to have increased. A study by Trivedi et al.52 relied 

on absolute differences between rates in examining adequacy of care (which included both 

treatment and control of conditions) where adequacy of care rates (especially as to treatment) 

were at generally high levels and increasing; and, as commonly happens in such circumstances, 

the authors found absolute difference between rates usually to have decreased (especially as to 

treatment).53 A commentary54 discussed the various findings and their perceived implications 

and stressed the need for more health disparities research and action to reduce such disparities. 

As was common in 2005, as it is now, neither the commentary nor any of the articles mentioned 

anything about the way different measures might yield different conclusions or the way any 

measure might be affected by general changes in the prevalence of the outcome being 

examined.55 

 A number of letters were published in response to the series.56 Only Keppel et al.

57

directly addressed measurement issues. The letter, by three authors of the NCHS 2005 health 

 

50 Vaccarino V, Rathore SS, Wenger NK, et al. Sex and racial differences in the management of acute myocardial 

infarction, 1994 through 2002. N Engl J Med 2005;353:671-682. 

51 Jha AK, Fisher ES, Li Z, Orav EJ, Epstein AM. Racial trends in the use of major procedures among the elderly. N 

Engl J Med 2005;353:683-691. 

52 Trivedi AN, Zaslavsky AM, Schneider EC, Ayanian JZ. Trends in the quality of care and racial disparities in 

Medicare managed care. N Engl J Med 2005;353:692-700. 

53 See Comment on Trivedi JAMA 2006 (2007) regarding the authors’ later effort to explain different patterns as to 

treatment and control, making very reasonable points in doing so, but without consideration of the generally lower 

rates of control compared with treatment. 

54 Lurie N. Health disparities – Less talk. more action. N Engl J Med 2005;353:727-729. 

55 Two studies on healthcare disparities in a recent issue of the NEJM are by groups of authors that include coauthors of the Trivedi 2005 study. See (a)Trivedi AN, Nsa W, Hausmann LRM, et al. Quality and equity of care in 

U.S. hospitals. N Engl J Med 2014;371:2298-308; and (b) Ayanian JZ, Landon BE, Newhouse JP, Zaslavsky AM. 

Racial and ethnic disparities among enrollees in Medicare Advantage plans. N Engl J Med 2014;371:2288-97.) 

These studies also rely on absolute differences between rates to measure healthcare disparities, again without 

consideration of the effects of frequency of the outcomes on the measures employed or mention that other measures 

might yield different conclusions. 

56 Trends in racial disparities in care. N Engl J Med 2005;353:2081-2084. 

1474 The Promise of Evidence-Based Policymaking
35 

disparities measurement monograph that recommend that all disparities be measured in terms of 

relative differences in adverse outcomes, presented an elaborate table to show, inter alia, that in 

four of the cases where the Trivedi study found decreasing disparities, relative differences in 

adverse outcomes indicated increasing disparities. The letter urged greater consistency in the 

reporting of disparities. Observers relying on the revised NCHS guidance, however, would 

instead note that relative differences in the favorable healthcare outcomes either (a) were usually 

consistent with the Trivedi study or (b) were usually inconsistent with the Jha study. 58 

 The Jha study also has a role in an extreme example of the confusion among persons 

attempting to measure health and healthcare disparities. A 2004 American Journal of Public 

Health (AJPH) study59 examined changes in racial disparities in certain fairly uncommon 

procedures among Medicare beneficiaries between 1986 and 1997, a period in which the 

procedures were generally increasing. As commonly occurs in the circumstances, the results 

showed that, in the main, relative difference in receipt of procedures decreased while absolute 

differences (and relative differences in non-receipt) increased. Relying on relative differences 

between rates of receiving such procedures to measure disparities, a common approach at the 

time, the authors found that disparities generally decreased. The Jha study mentioned above had 

examined changes in racial disparities in similarly uncommon procedures among Medicare 

beneficiaries between 1992 and 2001, also a period when rates were generally increasing. The 

results were much like those of the 2004 AJPH study, i.e., usually decreasing relative differences 

in receipt of procedure and increasing absolute differences (and relative differences in nonreceipt of the procedures). In this case, however, because the authors relied on absolute 

differences between rates to measure disparities, they found disparities usually to be increasing. 

A 2008 article60 in Medical Care Research and Review then discussed the seemingly contrasting 

findings of directions of changes in the two studies. And without consideration of the role of 

choice of measure in the reportage of results or showing any awareness of the measures used in 

the two studies, the article opined that differing conclusions in the studies may have had to do 

with the absence of complete overlap of the time periods studied. In essence, the study called for 

more research into why studies yielded contrasting results when in fact the studies had yielded 

essentially the same results. See the Spurious Contradictions subpage of the Measuring Health 

Disparities (MHD) page of jpscanlan.com. 

 

57 Keppel KG, Pearcy JN, Weissman JS. Untitled. N Engl J Med 2005;353:2082-2083. 

58 I discuss the 2005 NEJM study in "Race and Mortality Revisited" (at 338), and quite a few other places. See the 

above-mentioned October 9, 2012 letter to Harvard University (at 34-35). 

59 Escarce JJ, McGuire TG. Changes in racial differences in use of medical procedures and diagnostic tests among 

elderly persons: 1986-1997. Am J Public Health 2004;94:1795-1799.

60 Lê Cook B, McGuire TG, Zuvekas SH. Measuring trends in racial/ethnic health care disparities. Med Care Res 

Rev. 2009 Feb; 66(1):23-48. 

The Promise of Evidence-Based Policymaking 1475
36 

 An extreme example of wasted resources as result of the failure to grasp even that 

different measure commonly yield different conclusions about changes over time may found in 

AHRQ-funded review, costing between $5 million and $10 million, of the effects of 

improvement in healthcare on healthcare disparities. The study examined 4,258 studies but did 

not even report the measures employed in the studies. See "Race and Mortality Revisited" (at 

333) and the AHRQ’s Vanderbilt Study subpage of Measuring Health Disparities page. 

 In the prior section, I discussed that National Healthcare Disparities Reports had relied on 

NCHS recommendations to measure disparities in terms of relative differences in non-receipt of 

care. But it has not invariably implemented that approach successfully. The letter July 1, 2015 

letter to the Agency for Healthcare Research and Quality discusses that confusion about 

measurement issues caused the 2012 National Healthcare Disparities Report to highlight as some 

of the largest reductions in healthcare disparities over a particular period situations where the 

report would also find the disparities to be substantially larger at the end of the period than at the 

beginning of the period. This occurred because the agency apparently measured changes in 

disparities according to the comparative size of the percentage point changes in each group’s rate 

without understanding that doing so could yield a different result from the comparative size of 

the relative difference in the adverse outcome at the beginning of the period and the end of the 

period.61

 As a result of contacts from AHRQ personnel, I know that AHRQ gave some attention to 

the issues raised in my letter. But I cannot tell whether the agency yet understands the matter. In 

consequence of the agency’s no long publishing the tables underlying its analyses, I cannot tell 

precisely how it measured disparities in the 2015 report. The Chartbook on Healthcare for 

Blacks (at 49) discusses disparities in prenatal care in terms that “Black mothers were 2.3 times 

as likely as White mothers to delay prenatal care to the third trimester or to not receive prenatal 

care at all (10.0% compared with 4.3%).” That seems to indicate that the report continues to 

measure healthcare disparities in terms of relative differences in adverse outcomes, 

notwithstanding the NCHS reversal of position. On the other hand, the Chartbook on Health 

Living presents a map ranking states according to the size of absolute differences between black 

and white rates of receiving early and adequate prenatal care. Given the rate ranges at issues for 

receipt/non-receipt of prenatal care, rankings according to relative differences in non-receipt will 

depart substantially from rankings according to absolute differences and general increases in 

appropriate care rates will tend to increase the former while reducing the latter. Thus, as in the 

2012 report, the agency seems still to employ measures that tend to yield opposite conclusions 

from one another without apparent awareness of the conflict. 

 

61 The following may have played a part in the agency’s confusion. Like NCHS, AHRQ measures changes in 

disparities in terms of percentage point changes in relative differences. Unlike NCHS, however, AHRQ uses the 

“%” sign to mean percentage points, while also using the sign to mean percent in the relative sense. See the 

Percentage Points subpage of the Vignettes page of jpscanlan.com. In any case, the 2011 report (at 44) described 

changes in disparities in terms both of (a) % changes (meaning percentage point changes) in the disparities (i.e., 

changes in relative differences in adverse outcomes) and (b) % differences (meaning percentage point differences) 

between the percentage point changes in the rates of the groups being compared. In the situation where the relative 

difference in the adverse outcome and the absolute difference change in opposite directions, methods (a) and (b) will 

yield opposite conclusions as to directions of changes. 

1476 The Promise of Evidence-Based Policymaking
37 

 I have found no indication in the 2015 report that AHRQ is aware that NCHS has 

reversed the recommendation to rely on relative differences in non-receipt of care rather than 

receipt of care. One reason why AHRQ would not appreciate the significance of the revision is 

that AHRQ has not previously shown an understanding of the possibility that the direction of 

changes in disparities may turn on which relative difference is examined (though my letter 

should have made that clear enough). That is also a reason why AHRQ would not recognize that 

the NCHS reversal of position constitutes a repudiation of the first decade of National Healthcare 

Disparities Reports.62

 In any case, confusion issues aside, the National Healthcare Disparities Reports have 

been universally undermined by failure to recognize the ways that the measures it employed or 

intended to employ tended to be affected by the prevalence of an outcome. The same holds for 

AHRQ funded-research. 

 The above discussion of the failures of understanding on the part of AHRQ should not be 

read to suggest that those failures are more serious than found elsewhere. The same failures 

exist in all parts of the federal health and healthcare disparities research establishment, including 

the Centers for Disease Control and Prevention (CDC), and at all institutions conducting such 

research.63 

As indicated, the NCHS’s recognition of the pattern by which the two relative differences 

tend to be affected by the prevalence of an outcome has done nothing to improve that agency’s 

own research or the guidance it provides to other agencies or private researchers. Indeed, the 

agency’s manner of dealing with such issue would tend affirmatively to lead other entities and 

researchers to believe that a crucial issue is a non-issue. As discussed in the 2016 JPHMP 

commentary, the recent guidance on which relative difference in healthcare outcomes to examine 

for purposes of Healthy People 2020 obscures the issue even more than prior guidance. 

 As discussed in the JPHMP commentary, the 2016 article64 on health disparities 

measurement CDC and NCHS scientists/statisticians to which the commentary responded may 

also be deemed an affirmative obscuring of the need to address the implications of the effects of 

the prevalence of an outcome on measures employed in health and healthcare disparities 

research. The author’s response65 to the commentary, by attempting to defend the earlier work 

 

62 My letter to AHRQ was written before I became aware of the NCHS reversal of position (which may not have yet 

then occurred). 

63 As indicated the NCHS’s recognition of the pattern by which the two relative differences tend to be affected by 

the prevalence of an outcome has done nothing improve that agency’s own research or the guidance it provides to 

other arms. Indeed, the agency’s manner of dealing with such issue would tend affirmatively to lead other entities 

and researchers that a crucial issue is a non-issue. As discussed in "The Mismeasure of Health Disparities," the 

recent guidance obscures the issue even more than the prior guidance. 

64 Penman-Aguilar A,Talih M, Huang D, et al. Measurement of health disparities, health inequities, and social 

determinants of health to support the advancement of health equity. J Public Health Manag Pract. 2016;22(1) 

(suppl):S33-S42. 

65 Penman-Aguilar, Talih M, Moonesinghe R, Huang M. “Response to Scanlan Concerning: Measurement of 

Health Disparities, Health Inequities, and Social Determinants of Health to Support the Advancement of Health 

Equity. J Public Health Management Practice 2016;22(6), 614-615 [lttr].

The Promise of Evidence-Based Policymaking 1477
38 

without addressing the patterns by which standard measures tend to be affected by the prevalence 

of an outcome or the need for researchers to attempt to sort out the effects of the prevalence of an 

outcome on the measure employed, constitutes a further obscuring of the issue. 

 

*** 

 In the case of academic proficiency, irrespective of any change in the forces causing 

outcome rates of advantaged and disadvantaged groups to differ, general improvements in 

proficiency scores will tend to increase absolute differences between rates at which such 

groups reach the advanced level (where favorable outcome rates tend to be well below 50%) as 

reflected by movement from Row A to Row B in Table 3. On the other hand, general 

improvements in scores will tend to reduce such differences between rates of achieving the 

basic level (where favorable outcome rates are often well above 50%), as illustrated by 

movement from Row C to Row D. General deterioration in test performance will tend to have 

the opposite effect. Like patterns will tend to be observed when proficiency standards are 

altered or a proficiency test is replaced with one that is easier or harder. 

 Discussions of situations where observers commonly rely on absolute differences to 

measure educational disparities, but without understanding the patterns by which such 

measures tend to be affected by the prevalence of an outcome, may be found in the Educational 

Disparities page of jpscanlan.com and its Disparities by Subject, New York Proficiency Rate 

Disparities, Education Trust High Achiever Study , Education Trust Glass Ceiling Study, 

Education Trust High Achiever Study, Education Trust Glass Ceiling Study, Annie E. Casey 

2014 Proficiency Disparities Study subpage. See also the letters to the Annie E. Casey 

Foundation (May 13, 2014) and Education Trust (April 30, 2014).66

 But see the discussion at page 3-4 of the letter to the Stanford Center on Poverty and 

Inequality (Mar. 8, 2016) regarding the work of work of Harvard Professor Andrew Ho and 

Stanford Professor Sean Reardon. Professor Ho independently recognized the patterns by which 

absolute differences between outcome rates tend to be affected by the prevalence of an outcome 

and the implications of those patterns regarding the appraisal of demographic differences in 

proficiency rates.67 While the Ho article and the illustration in its Figure 2 is focused solely on 

patterns by which absolute differences tend to be affected by the prevalence of an outcome, the 

rates that underlie the figure would also form the basis for the illustrations of patterns of relative 

differences and odds ratios in Figures 1 and 3 in these comments. Work of Professors Ho and 

Reardon regarding the measurement of proficiency disparities in ways unaffected by the 

 

66 In some cases, differences in proficiency outcomes (or comparable outcomes) are measured in terms of either the 

relative difference in the favorable outcome or the relative difference in the adverse outcome or both relative 

differences. Invariably, however, those employing such measures do so without apparent awareness of the ways the 

measures employed tend to be affected by the prevalence of the outcome. The Harvard CRP NCLB Study and 

McKinsey Achievement Gap Study subpages of the Discipline Disparities page and the letter to New York City 

Center for Innovation through Data Intelligence (June 6, 2016). 

67 Ho, Andrew D. 2008. “The problem with ’proficiency’: Limitations of statistics and policy under No Child Left 

Behind,” Educational Researcher, 37,351–360. 

1478 The Promise of Evidence-Based Policymaking
39 

prevalence of an outcome, 68 seems to be suggest approaches generally along the lines of the 

approach suggested in "Race and Mortality Revisited" and the October 2015 ASA letter. But, 

for reasons discussed in the Introduction, such matter is beyond the scope of these comments. 

 The following observations are in order regarding the general rigor in the study of 

demographic differences in terms of relative or absolute differences between outcome rates. 

 The attentions researchers have recent given to value judgments in the analysis of 

demographic differences, such as I discuss in "Race and Mortality Revisited," the 2016 JPHMP 

commentary, and the Comment on Mackenbach BMJ 2016 (2016) as principally diverting 

attention from the crucial measurement issues, remains an exception to usual practice. Usual 

practice continues to be characterized by reliance on a preferred measure without mention of any 

other measure or whether such measure might, or in fact does, yield a different conclusion from 

the one employed in a particular study, even when the other measure may be the one most 

commonly employed in the circumstances. So far as I can tell, such practices is permitted in 

essentially all medical and other scientific journals. And it allows observers to make whatever 

claim they wish about how particular policies or trends disparately affect different groups. See 

discussions regarding the suburbanization of poverty on the Feminization of Poverty page of 

jpscanlan.com, Table 30 (slide 118) of the University of Maryland Workshop referenced in note 

18 and the letter to Antioch Unified School District (Sept. 9, 2016). 

 Requirements that persons analyzing demographic differences explain the reasons for 

their choice of measure and how other measures might or do yield contrary results would not 

address the key measurement issues discussed here and might even detract from them in the way 

recent discussions of value judgments have. But the fact that it is usually deemed permissible to 

use any measure one chooses to analyze demographic differences nevertheless highlight the need 

for fundamental reform in this area. 

C. The Appraisal of Demographic Differences in Terms of the Proportion a Group 

Comprises of Persons Potentially Experiencing an Outcome and the Proportion it 

Comprises of Persons Actually Experiencing the Outcome 

 Sometimes observer measure demographic differences on the basis of comparisons 

between the (a) proportion a group makes up of persons potentially experiencing a favorable or 

adverse outcomes outcome (the pool) and (b) the proportion it makes up of persons actually 

experiencing the outcome. This has long been the case with regard to selection issues in 

employment, where the analyses of discrimination issues commonly involve a comparison of the 

proportion a group makes up of the pool with the proportion is comprises of persons 

experiencing a favorable employment outcome. See Hazelwood School District v United States, 

433 U.S. 299 (1977). The issues addressed here apply to those situations. See Kansas Law 

paper (at 23-26) and the TDHCD brief (at 23-27). 

 

68 Ho, Andrew D., and Reardon, Sean F. 2012. “Estimating Achievement Gaps From Test Scores Reported in 

Ordinal ‘Proficiency’ Categories,” Journal of Educational and Behavioral Statistics, 37(4), 489–517. 

The Promise of Evidence-Based Policymaking 1479
40 

 But I principally address this subject with regard to perceptions about differences 

between the proportion a group comprises of the pool and the proportion it comprises of persons 

experiencing an adverse outcome. Such comparisons are increasingly a focus of attention to 

disparities issues involving school discipline and criminal justice. See the Department of Health 

and Human Services and Department of Education (Aug. 24, 2015) and the letters to school 

districts and to Texas Appleseed (Apr. 7, 2015) regarding school discipline. See “Things DoJ 

doesn’t know about racial disparities in Ferguson,” The Hill (Feb. 22, 2016) and Things the 

President Doesn’t Know About Racial Disparities,” Federalist Society Blog (Aug. 5, 2016) 

regarding criminal justice. 

 Commonly such comparisons are made without indicating how one might measure the 

difference between (a) and (b). But a Department of Education-funded document titled 

“Methods for Assessing Racial/Ethnic Disproportionality in Special Education” which I discuss 

on the IDEA Data Center Disproportionality Guide subpage of the Discipline Disparities page, 

shows how to calculate both relative and absolute differences between (a) and (b) and those are 

the likely approaches of anyone who wants to quantify differences between (a) and (b).69

 The proportion a group comprises of persons experiencing an outcome is a function of 

the relative difference and therefore the appraisal of disparities based on any measure of the 

difference between (a) and (b) is unsound in the same way the relative difference is unsound 

when employed without regard to the manner in which the measure tends to be affected by the 

prevalence of an outcome. 

 But there are additional reasons why one can never soundly analyze a demographic 

difference on the basis of a comparison of (a) and (b). The more extended treatments of 

measurement issues discussed in the Introduction should make clear than any sound effort to 

appraise a demographic difference must be based on the actual rates at which two groups 

experience the outcome. That also holds for observers who might maintain that the odds ratio is 

a measure unaffected by the prevalence of an outcome, since the actual rates are necessary in 

order to calculate the odds ratio. See note 49 supra.

70

 

69 In the employment setting, there occur misguided efforts to quantify the difference between (a) and (b) in terms of 

the statistical significance of such difference given the number of selections at issue. Problems with that approach 

are different from the quantification issues addressed in these comments. See the Kansas Law paper (at 26-27). But 

such are problems are among many issues not specifically addressed here that the Commission must address in 

attempting to reform analyses of demographic differences that may inform policies and programs. 

70 One also needs the actual rates to determine the absolute difference between rates. It is doubtful that anyone 

would maintain that absolute differences does not tend to be affected by the prevalence of an outcome (save in the 

case of uniform underlying distributions). It warrants note, however, that in cases where the only information 

available is (a) and (b), observers who would otherwise rely on absolute differences may well rely on some 

measures of the difference between (a) and (b). Comparisons of (a) and (b) will commonly yield opposite 

conclusions about directions of change from those based on absolute differences in matters like school discipline 

disparities. See the letter to Antioch Unified School District (Sept. 9, 2016) regarding a suit where the discipline 

disparities measure employed in the complaint tends to yield opposite conclusions from the measure commonly 

employed by the principal expert discussed in the complaint. 

, 

1480 The Promise of Evidence-Based Policymaking
41 

 Information on (a) and (b) allow one to calculate a ratio of the rates of experiencing the 

outcome addressed in (b) (though not the rate ratio for the opposite outcome). See Kansas Law 

paper (at 24 n. 26). But such information does not enable to one to divine the actual rates at 

which the two groups experience the outcome. 

 Further, the proportion a group comprises of the pool will affect both relative and 

absolute differences between (a) and (b) in ways that have nothing to do with the strength of the 

forces causing (a) and (b) to differ. That is, suppose that the rates for an adverse outcome of an 

advantaged group (AG) and disadvantaged group (DG) are 10% and 20% respectively in two 

settings (e.g., different school districts). Putting aside how one might effectively quantify the 

difference in the circumstances of AG and DG reflected by the 10% and 20% outcome rates, 

there is no basis to say the difference is larger in one setting than the other. Yet, given those 

10% and 20% rates, the relative and absolute differences between (a) and (b) will vary in a 

variety of ways depending on the proportion the disadvantaged group comprises of the pool of 

persons potentially experiencing the outcome. 

 The patterns by which the proportion a group comprises of the pool affects relative and 

absolute differences between (a) and (b) are discussed and illustrated in the above-mentioned my 

IDEA Data Center Disproportionality Guide subpage. They are also illustrated in the slides 97 to 

108 of the University of Maryland methods workshop mentioned in note 9. 

 It is unnecessary to illustrate those patterns here, however. For the previously mentioned 

issues provide sufficient reason why demographic differences should never be analyzed on the 

basis of comparisons between the proportion a group comprises of the pool and the proportion it 

comprises of persons experiencing some favorable or adverse outcome. 

 The failure of the government or research communities to recognize these issues does, 

however, further illustrate the need the Commission to take actions of the kind outlined in Part 

III. 

D. Subgroup Effects Issues 

 Section A discusses the failure of persons examining subgroup effects or drawing 

inferences about processes on the basis of the comparison of relative effects of a factor on 

different subgroups (or in different settings) to understand the reasons to expect a factor affecting 

an outcome rate to cause a larger proportionate change in the rate for the group with the lower 

baseline rate for the outcome while causing a larger proportionate change in the opposite 

outcome rate for the other group.71 That section and the materials it references also explain that, 

so far as the published reveals, never has anyone identifying a subgroup effect or drawing an 

inference based on the comparative sizes of relative effects recognized that examination of the 

opposite outcome could (or commonly would or in fact would in the particular situation 

examined) form the basis for an opposite subgroup effect or opposite inference. See especially 

"Race and Mortality Revisited" at 339-341. 

 

71 See also the October 9, 2012 letter to Harvard University at 39-40. 

The Promise of Evidence-Based Policymaking 1481
42 

 Irrespective of such pattern, however, it should be recognized that it is illogical to believe 

that, absent a subgroup effect as such effect is typically analyzed, one would expect a factor that 

affects on outcome rate to have the same relative effect on different baseline rates for the 

outcome. For if a factor causes two groups to experience equal proportionate changes in 

different baseline rates for one outcome, it necessarily causes the groups to experience unequal 

proportionate changes in their rates for the opposite outcome. Since there is no more reason to 

expect equal proportionate changes in one outcome than there is to expect equal proportionate 

changes in the opposite outcome, it is illogical to expect equal proportionate changes in either 

outcome. 

 Aspects of this issue are discussed on following subpages to the Scanlan’s Rule page of 

jpscanlan.com. The pages are in many respect duplicative of one other. But in my view (at least 

of the time of creating it) each added an additional element of sufficient importance, or caused a 

particular aspect of the matter sufficiently be better focused, to warrant the creation of the page. 

 The Subgroups Effects subpage discusses this issue with regard to the clinical setting, 

while explaining the implications of the above-described pattern of relative effects with regard to 

employing an observed risk reduction in a clinical trial to estimate the absolute risk reduction 

(and corresponding number-needed-to-treat) in circumstances involving baseline rates different 

from that in the trial. Illustrations pertinent to this subpage may be found in my presentation at 

the 2009 Joint Statistical Meetings “Interpreting Differential Effects in Light of Fundamental 

Statistical Tendencies” (abstract) 

 The Subgroup Effects – Nonclinical subpage discusses the subject with regard to 

perceptions about subgroup effects in non-clinical settings. 

 The Illogical Premises subpage discusses the illogic of the belief that a factor will tend to 

cause the same proportionate effect on different baseline rates while addressing whether similar 

expectations for the absolute difference and odds ratio would be illogical as well. 

 The Illogical Premises II discusses that, for the same reason that it is illogical to expect a 

factor typically to show the same relative effect on different baseline rates of experiencing the 

outcome, it is illogical to regard a particular rate ratio as reflecting the same strength of an 

association for different baseline rates for an outcome. 

 The Inevitability of Interaction subpage why any time a factor in fact shows equal 

proportionate changes for rates of experiencing an outcome (thus, the absence of an interactive 

effect) it will necessarily show a different proportionate effect on the opposite outcome. 

 The Interactions by Age subpage, which is closely related to the Life Tables Illustrations

subpage, discusses the fact that almost invariably in comparisons of age groups with 

substantially different mortality rates, one will find opposite patterns of interaction depending on 

whether one examines a factor’s effect on mortality or on survival (that is, that the age group 

with the smaller proportionate effect on its mortality rate will show the larger proportionate 

effect on its survival rate). 

1482 The Promise of Evidence-Based Policymaking
43 

 Other treatments of these issues may be found in my Comment on FDA Proposed 

Subgroup Regulations (May 16, 2014) Comment on European Medicines Agency Subgroup 

Guidelines (July 31, 2014). 

 Relatively succinct treatments of the issues with reference to journal articles regarding 

subgroup effects, reporting heterogeneity, perceptions about the constancy of the rate ratio across 

different baseline rates, calculation of number-needed-to-treat, or stratified medicine may be 

found in Comment on Hingorani BMJ 2013, Comment on Barrett CMAJ 2004 (2012), Comment 

on Delpierre BMC Public Health 2012 (2012), Comment on Wang Emerging Themes in 

Epidemiology 2009 (2012), Comment on Altman BMJ 2003 (2011), Comment on Kent Trials 

2010 (2011), Comment on White BMC Med Res Meth 2005 (2011), , Comment on Schwartz 

BMJ 2007 (2011), Comment on Cook BMJ 1995 (2011), Comment on Huisman Int J Epidemiol 

2007 (2010), Comment on Chatellier BMJ 1996 (2011), Comment on Mullins Health Affairs 

2010 (2010), Comment on Berrington de Gonzalez MEJM 2010 (2010), Comment on Sun BMJ 

2010 (2010), Comment on Gabler Trials 2009 (2009), Comment on Kawachi Health Affairs 

2005 (2008), Comment on Gan NEJM 2000 (2007), Comment on Mustard JECH 2003 (2007), 

Comment on Thurston AJE 2005 (2007), Comment on Martikainen JECH 2007 (2007), 

Comment on Kaplan JECH 2006 (2006). 

 As discussed in the introduction to Part I, the patterns discussed in this section are merely 

manifestation of other patterns described in the introduction and Section A. But given the 

resources devoted to studies of subgroups effects in clinical and other contexts, and the varied 

flawed inferences based on perceptions about such effects, perceptions about subgroups effects 

warrant special attention from the Commission. 

II. The Fundamental Unsoundness of Analyses of Discrimination Issues That Examine 

Data Solely on Persons Who Accepted Some Outcome or Situation. 

 Most discrimination cases that have yielded recoveries approaching or exceeding $100 

million have been based on analyses that examined data solely on persons who accepted some 

outcome or situation. In the 1990s such cases principally involved claims that persons who were 

hired into grocery stores or other retailers were disproportionately assigned to jobs that were 

believed to be less desirable than other jobs. See, e.g. "Multimillion-Dollar Settlements May 

Cause Employers to Avoid Hiring Women and Minorities for Less Desirable Jobs to Improve the 

Statistical Picture," National Law Journal (Mar. 27, 1995) and "Unlucky Stores: Are They All 

Guilty of Discrimination?" San Francisco Daily Journal (Jan. 29, 1993). More recently, such 

cases have involve claims that lenders disproportionately assigned loans to minorities to 

subprime status, as in the cases brought by the Department of Justice against Countrywide 

Financial Corp. and Wells Fargo Bank that were settled for $335 and $175 million. See “The 

Perverse Enforcement of Fair Lending Laws,” Mortgage Banking (May 2014), and “Fair 

Lending Studies Paint Incomplete Picture,” American Banker (April 24, 2013). 

 I first discussed this issue with regard so-called job segregation or assignment 

discrimination claims in "Illusions of Job Segregation," Public Interest (Fall 1988) and "Are Bias 

Statistics Nonsense?" Legal Times (Apr. 17, 1989). The former item presents a number of 

tabular illustrations of why one cannot draw inferences about discriminatory exclusion from 

The Promise of Evidence-Based Policymaking 1483
44 

putatively better jobs without consideration of the part of the applicant pool that was not 

interested in the putatively poorer job. 

 Table IV of that article explains why one also cannot draw inference about the absence of 

discrimination of the basis of weaker qualifications of persons from a particular group among 

persons are hired, which has been asserted as showing that the group is being favored. The 

reason one cannot draw such an inference is that the group with weaker qualifications among 

applicants will commonly also have weaker qualifications among persons who are hired even 

when the group has been subject to discrimination. I have explained this point with regard to 

arguments that weaker qualifications among athletes from a particular group or collegeadmissions from a particular group suggests that such groups are being favored "The Perils of 

Provocative Statistics," Public Interest (Winter, 1991). I have also explained it with regard to 

the claim that higher default rates among minority borrowers suggests that they are being favored 

in "Both Sides Misuse Data in the Credit Discrimination Debate," American Banker (July 22, 

1998). 

The job segregation/assignment discrimination claims discussed in “Illusions of Job 

Segregation” or other articles of the 1980s or 1990s involved binary outcomes (at least in the 

sense that the putative victims of discrimination were being disproportionately assigned to a less 

desirable rather than a more desirable situation). The same holds for claims of discriminatory 

assignment of loans to subprime status in recent fair lending cases. But as discussed in the 

Kansas Law paper (Section F, at 32-35) and the TDHCD brief (Section I.C, at 27-30), the 

fundamental failing of analyses that fail to examine data on the entire universe at issue also 

applies to situations of continuous variables like salaries and loan terms. 

 That is, the validity of allegations that persons from a particular group received lower 

starting salaries or higher interest rates than persons another group cannot be appraised without 

consideration of persons who refused to accept those terms (and persons offered no terms at all). 

Further, in the employment context, whether one group’s salary progression was impeded by 

discrimination cannot be analyzed without consideration of the treatment of the persons who left 

the employer because of what such persons regarded as inadequate salary progression or for any 

other reasons. 

 But an understanding of the implications of the failure to examine data on the entire 

universe at issues is as universally absent from analyses of discrimination issues as the 

understanding of patterns by which measures tend to be affected by the prevalence of an 

outcome is from analyses of demographic differences in outcome rates.72 

 

72 The aspect of risk distributions underlying the patterns discussed in part are implicated in all efforts to analyze 

discrimination issues in the same way they are implicated in all efforts to adjust for pertinent factors in any analysis. 

See "The Perils of Provocative Statistics," Public Interest (Winter, 1991) and the Sears Case Illustration subpage. In 

my view rarely if ever are efforts to adjust for differences in characteristics fully adjust for such differences. 

Moreover, observers seem rarely to acknowledge even the obvious shortcoming of such efforts. But, while the 

failure to adjust for characteristics is commonly a serious problem in analyses of discrimination issues and many 

other issues, such failure does raise the same issues as to the fundamental unsoundness of analyses of demographic 

differences as the failure to recognize the ways measures tend to be affected by the prevalence discussed in Part I 

and the failure to examine the entire universe at issue discussed in Part II. 

1484 The Promise of Evidence-Based Policymaking
45 

III. Recommendations 

 Set out below are a number of recommendations for Commission action to address the 

issues discussed in Parts I and II and related matters. 

 First, the Commission should create a committee to explore problems in analyses 

involving demographic and other differences in outcome rates (including with regard to 

interpretations of subgroup effects in clinical and non-clinical settings) and to recommend 

actions to improve such analyses. In accomplishing its tasks, the committee should become 

familiar with all the materials referenced above and seek informed input from the scientific 

community. 

 Second, the Commission should recommend that Congress establish a permanent body 

charged with appraising the soundness of statistical analyses in matters affecting public policy, 

including law enforcement. It might be suggested that there already exist such bodies, including 

the National Academies and the Government Accountability Office. But these and like entities 

have failed even to recognize that reducing adverse outcomes, whether involving health and 

healthcare, lending, school discipline, school performance, criminal justice, or any other matter, 

tends to increase rather than reduce relative differences in rates of experiencing those outcomes. 

Whether or not a creation of a new entity is necessarily a solution, some action is necessary to 

ensure that statistical analyses involving important policy issues receive much greater scrutiny 

than they have in the past. 

 Third, the Commission should recommend to Congress that it require reports of the 

agencies involved with funding or conducting statistical analyses involving outcome rates, or 

involved with law enforcements based on such analyses, describing actions they will take to 

address the issues discussed in these comments and the materials they reference. 

 Fourth, the Commission should recommend that Congress require that requests for 

federal funding of health and healthcare disparities and other research involving differences in 

outcome rates include statements like the following: 

1. We are aware that there exist patterns by which measures commonly employed in this 

type of research tend to be affected by changes in the prevalence of the outcome 

examined irrespective of (a) actual changes in differences in the circumstances of 

advantaged and disadvantaged groups or (b) effects of policies aimed at mitigating those 

differences. 

2. We intend to attempt to distinguish between the effects of the patterns by which 

measures tend to be affected by the prevalence of an outcome and (a) and (b) in the 

following manner: 

Studies themselves should include at their beginning the following statements: 

The Promise of Evidence-Based Policymaking 1485
46 

1. This study has (or has not) attempted to distinguish between the effects of changes in 

the prevalence of an outcome on the measures employed and (a) actual changes in 

differences in the circumstances of advantaged and disadvantaged groups (b) effects of 

policies aimed at mitigating those differences. 

2. Because of 1, this study may be (should not be) used to inform policy. 

 

 The language can be adjusted to address the situation of research aimed at appraising the 

effects of factors other than policies on demographic differences. It can also be adjusted to 

address situations where, rather than examining changes in differences over time, researchers 

compare the size of some difference within settings differentiated other than temporally (and 

where the settings differ in the overall prevalence of an outcome). 

 Similar statements should be required for the continuation of funding already authorized. 

 Fifth, the Commission should recommend that Congress take all steps necessary to 

ensure that no federal law enforcement actions are based the belief that reducing the frequency of 

an adverse outcome tends to increase relative demographic differences in rates of experiencing 

the outcome or the proportion disadvantaged groups make up of persons experiencing those 

outcomes. 

 Sixth, the Commission should recommend that Congress identify all existing legislation 

(a) that reflects the belief that reducing the frequency of an adverse outcome will tend to reduce 

relative demographic differences in rates of experiencing the outcome or the proportion 

disadvantaged groups make up of persons experiencing the outcome; (b) that require the 

monitoring of demographic differences with regard to some outcome; (c) that impose liability for 

a practice that has a disparate impact; (d) that require implementation of a less discriminatory 

alternative to practices having a disparate impact. Congress should then consider options for 

eliminating any false beliefs reflected in such legislation and for either clarifying how 

differences and disparate impacts are to be measured or eliminating the requirements. 

 Seventh, the Commission should recommend that Congress require that federal agencies 

take the same actions regarding regulations that the prior paragraph suggests Congress take 

regarding legislation. 

 

1486 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t18-ser7

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0136

Comment on FR Doc # 2016-22002

Submitter Information

Name: Sara Dube

General Comment

The Pew-MacArthur Results First Initiative, a joint effort of The Pew Charitable Trusts

and the John D. and Catherine T. MacArthur Foundation, works with states and counties

to implement an innovative evidence-based policymaking approach that helps them

invest in policies and programs that are proven to work. 

We share the Evidence-Based Policymaking Commission's commitment to bringing data

and evidence to the forefront of policymaking. As you consider how to increase support

of evidence-building activities at the federal level, we hope that the lessons we have

learned working with 22 states and eight counties will be informative to you. We also

ask that you keep in mind these state and county efforts already underway and consider

how federal policies can best align with and support them. 

In our submission we seek to answer the overarching question you posed in the request

for comments: "Are there successful frameworks, policies, practices, and methods to

overcome challenges related to evidence-building from state, local, and/or international

governments the Commission should consider when developing findings and

recommendations regarding Federal evidence-based policymaking?" 

Our response is attached: Attachment 1_Results First Official Comment.pdf

The Promise of Evidence-Based Policymaking 1487
Additional materials attached are: 

Attachment 2_Evidence-Based Policymaking Guide for Effective Government.pdf

Attachment 3_Public Administration Review Article.pdf

Attachment 4_New York's Investment in Evidence-Based Policymaking.pdf

Thank you for your time and consideration. 

Attachments

Attachment 1_Results First Official Comment

Attachment 2_Evidence-Based Policymaking Guide for Effective Government

Attachment 3_Public Administration Review Article

Attachment 4_New York's Investment in Evidence-Based Policymaking

1488 The Promise of Evidence-Based Policymaking
Overarching Question Posed in Request for Comments 

Are there successful frameworks, policies, practices, and methods to overcome challenges 

related to evidence-building from state, local, and/or international governments the Commission 

should consider when developing findings and recommendations regarding Federal evidencebased policymaking? If so, please describe. 

Answer by the Pew-MacArthur Results First Initiative 

The Pew-MacArthur Results First Initiative, a joint effort of The Pew Charitable Trusts and 

the John D. and Catherine T. MacArthur Foundation, works with states and counties to 

implement an innovative evidence-based policymaking approach that helps them invest in 

policies and programs that are proven to work. 

We share the Evidence-Based Policymaking Commission’s commitment to bringing data and 

evidence to the forefront of policymaking. As you consider how to increase support of evidencebuilding activities at the federal level, we hope that the lessons we have learned working with 22 

states and eight counties will be informative to you. We also ask that you keep in mind these 

state and county efforts already underway and consider how federal policies can best align with 

and support them. 

Overview 

Through practice and research, we have identified five key components to implementing 

evidence-based policymaking in a state or county:1

x Program assessment: Systematically reviewing available evidence on the effectiveness 

of public programs. 

x Budget development: Incorporating evidence of program effectiveness into budget and 

policy decisions, giving funding priority to programs that deliver a high return on 

investment of public funds. 

x Implementation oversight: Ensuring that programs are effectively delivered and are 

faithful to their intended design. 

x Outcome monitoring: Routinely measuring and reporting outcome data to determine 

whether interventions are achieving desired results. 

x Targeted evaluation: Conducting rigorous evaluations of new and untested programs to 

ensure that they warrant continued funding. 

Within this framework, Results First specializes in building the capacity of governments to 

address the first two components – program assessment and budget development. It does so by 

training budget and research staff from both legislative and executive branches to: 

1) Create a comprehensive inventory of currently funded programs and compare them to the 

evidence base; 

2) Conduct cost-benefit analyses to compare the return on investment of programs; and 

3) Use this information to target funds to evidence-based programs and embed evidence into 

the budgeting and policymaking processes. 

The Promise of Evidence-Based Policymaking 1489
In addition, through communications and outreach support, we help states and counties build a 

climate for decision-making based on research and evidence.2

Program Assessment 

Results First partners begin their work by developing a comprehensive list of all the programs 

they fund in a particular policy area; this is called a program inventory. This includes collecting 

information such as the program name, description, cost, target population, duration, and number 

served. Creating this kind of comprehensive picture of programs across a policy area is not a 

simple task because programs are delivered by vast networks of agencies, contracted providers, 

and local non-profits; as a result, agency budgets contain only aggregated information (e.g., one 

line item for all substance abuse programs) and do not provide granular program budget 

information. Results First partners have found program inventories to be valuable as they provide 

a snapshot of currently funded programs in a specific policy area. 

Once all the information is gathered, the next step is to determine if there is any rigorous 

evidence on the effectiveness of the jurisdiction’s programs, and if so, what that evidence 

indicates. To help partners with this step, Results First developed the Results First Clearinghouse 

Database. The Clearinghouse Database is a web-based "clearinghouse of clearinghouses." It pulls 

together the program evidence ratings issued by eight national research clearinghouses, including 

some that are run by the federal government, such as the U.S. Department of Education's What 

Works Clearinghouse, the U.S. Substance Abuse and Mental Health Services Administration’s 

National Registry of Evidence-based Programs and Practices, and the U.S. Department of 

Justice's CrimeSolutions.gov.3 

The clearinghouses curate evaluation of programs by policy area to provide policymakers and 

practitioners with the information they need to make evidence-based decisions. This vital 

information is currently underutilized, largely because each clearinghouse operates 

independently, uses different nomenclature when reporting results (e.g., rating programs as 

model, promising, potentially positive, etc.), and has limited capacity to publicize its work. 

To use the information to help make better investment choices, policymakers need to be aware 

that the clearinghouses exist, choose which to consult, find programs of interest, interpret the 

different ratings, and compare and contrast the findings. The Results First Clearinghouse 

Database bridges this gap. It reconciles the different systems and vocabularies used by each 

clearinghouse, and provides the data in a clear, accessible format. It allows users to search more 

than 1000 programs, by intervention type, policy area, and evidence rating. The Clearinghouse 

Database is available to the public.4 

Results First partners incorporate the ratings (or lack thereof) into their program inventories. This 

allows policymakers and practitioners to know which programs are being offered, whether they 

have been evaluated, and how effective they are. 

For a subset of the programs in the program inventory, Results First partners can also conduct a 

cost-benefit analysis using the Results First model. This cloud-based model, which is based on 

the work of the Washington State Institute for Public Policy, uses econometric simulations and 

1490 The Promise of Evidence-Based Policymaking
meta-analyses of rigorous evaluations, as well as jurisdiction-specific population and cost data to 

predict and monetize the return on investment that individual programs can be expected to 

achieve. The model covers approximately 250 programs across eight social policy areas 

including adult criminal justice, juvenile justice, substance abuse, mental health, early education, 

and child welfare. Only programs with a sufficiently rigorous evidence base to allow for 

computation of effect sizes are included in the model. This model is proprietary to Results First 

and requires significant technical assistance to utilize correctly, which is why it is only available 

to Results First partner jurisdictions.

5

 

With Results First technical support, state and county teams then customize the cost-benefit 

model with the required jurisdiction-specific data, including program costs, program duration, 

resource costs (e.g., cost of a day in prison) and resource use (e.g., number of days in prison). 

Results First partner states and counties can then use the model to run cost-benefit analyses on 

their programs to determine expected benefits, costs, and return on investment. Moreover, the 

model shows the source of the benefits (e.g., higher earnings, less crime, etc.), who the benefits 

accrue to (taxpayers, participants, or others in society) and how they accrue over time (through a 

cash flow analysis). 

Sample Cost-Benefit Table 6

(For demonstrative purposes only, not intended to show actual results) 

PROGRAM NAME PROGRAM 

BUDGET RATINGS COSTS BENEFITS

BENEFIT 

TO COST 

RATIO

Correctional industries $125,000 Highest rated $1,485 $6,818 $4.59 

Correctional education $50,000 Highest rated $431 $21,720 $18.40

Vocational education $300,000 Second-highest rated $1,645 $19,594 $11.91 

Drug courts $250,000 Second-highest rated $4,951 $15,361 $3.10 

Adult boot camps $180,000 No evidence of 

effects — — —

Veterans courts $100,000 Not rated — — —

All others $950,000 Not rated — — —

Budget Development 

When these analyses are incorporated into states’ and counties’ regular budget processes, 

policymakers are equipped with a comprehensive view of the investment of taxpayer funds in a 

particular policy area and the likelihood that those investments will achieve intended outcomes. 

Legislators and executive branch decision-makers can use this information to consider whether 

funding should be shifted to programs that have more evidence of effectiveness, or that are 

predicted to achieve desired outcomes at a lower cost. 

However, just because the information is available does not mean that it will be used – 

increasing the supply of evidence is not the same as creating demand for evidence. This is why, 

as part of our technical assistance, Results First also helps its partners to reach out to and build 

the support of policy leaders in the legislative and executive branches, as well as other key 

stakeholders such as members of the business community. For instance, with Results First staff 

The Promise of Evidence-Based Policymaking 1491
support, our jurisdiction partners develop outreach plans for releasing their reports and briefing 

policymakers. 

With sufficient buy-in and commitment from decision-makers, the use of evidence and research 

can also be embedded in the policymaking process. For instance, legislatures can create statutory 

definitions of evidence, require agencies to use program inventories and return on investment 

analyses in their budget proposals, and mandate funding preference to evidence-based programs. 

Agencies can also incorporate the use evidence into their own processes through administrative 

policy. Enshrining evidence-based policymaking in these ways helps build consistency and 

continuity across policymakers and staff, ultimately contributing to the overall culture of using 

evidence to inform budget and policy decisions. 

For example, lawmakers in Mississippi established evidence standards in 2014 and currently 

require agencies to justify funding requests for any new program by identifying evidence 

supporting the program's effectiveness. Similarly, New Mexico’s Legislative Finance Committee 

requires rigorous evidence and return on investment information to be considered by staff 

analyzing budget requests. The state’s Corrections Department also adopted contracting 

standards that require vendors to document their use of evidence-based practices and monitor 

outcomes for programs that are developed locally to ensure that they meet the state's goals. In 

addition, New York State’s Division of Criminal Justice Services has issued several competitive 

grants that prioritize funding to programs that are predicted to be effective and generate strong 

returns. 

Conclusion 

Throughout the Results First process, our state and county partners develop the capacity to 

methodically and consistently incorporate evidence into their budget and policymaking practices. 

To date, Results First partner governments have used the approach to target $186 million to 

evidence-based programs – a number that we expect will increase over time. 

As mentioned at the outset, our work focuses on the first two components of the evidence-based 

policymaking framework. Assessing program effectiveness and directing funds to evidencebased programs are critical steps, but to truly improve outcomes, programs have to be 

implemented with fidelity to their design. Plus, outcomes must be monitored and additional 

programs have to be rigorously evaluated, to build up the evidence base. 

 

Endnotes 

1

 For more information on the framework, please see attached report “Evidence-Based Policymaking: A Guide for 

Effective Government,” also available online: http://www.pewtrusts.org/en/research-andanalysis/reports/2014/11/evidence-based-policymaking-a-guide-for-effective-government 

2

 For a concise but informative description of Results First work, please see attached case study “Bridging the Gap 

between Evidence and Policy Makers: A Case Study of The Pew-MacArthur Results First Initiative,” published in 

Public Administration Review, June 25, 2016. 

3

 Clearinghouses included in the Results First Clearinghouse Database: 

x U.S. Department of Education's What Works Clearinghouse 

1492 The Promise of Evidence-Based Policymaking
 

x Blueprints for Healthy Youth Development 

x U.S. Department of Justice's CrimeSolutions.gov 

x California Evidence-Based Clearinghouse for Child Welfare 

x Coalition for Evidence-Based Policy 

x U.S. HHS Substance Abuse and Mental Health Services Administration’s National Registry of Evidencebased Programs and Practice 

x Promising Practices Network 

x What Works in Reentry Clearinghouse 

4

 You can see the Results First Clearinghouse Database here: http://www.pewtrusts.org/en/multimedia/datavisualizations/2015/results-first-clearinghouse-database 

5

 Information on the Results First process for selecting partner jurisdictions: 

http://www.pewtrusts.org/~/media/assets/2013/results-first-in-your-state-brief.pdf?la=en 

6 For an example of a cost-benefit table produced by a state, see “New York State Results First Program Impact 

Table” in the Results First New York Case Study, attached and available online: 

http://www.pewtrusts.org/~/media/assets/2015/11/rf_nys_case_study.pdf

The Promise of Evidence-Based Policymaking 1493
Evidence-Based 

Policymaking 

A guide for effective government

A report from the Pew-MacArthur Results First Initiative Nov 2014

1494 The Promise of Evidence-Based Policymaking
Gary VanLandingham 

Torey Silloway 

Valerie Chang 

Meredith Klein 

Darcy White 

Amanda Hoey 

Elizabeth Davies

The Pew Charitable Trusts

Susan K. Urahn, executive vice president

Michael Caudell-Feagan, vice president 

John D. and Catherine T. MacArthur Foundation

Julia Stasch, interim president 

Pew-MacArthur Results First Initiative

Senior Advisers 

Katherine Barrett 

Richard Greene

External Reviewers

This report benefited from the insights and expertise of the Results First Research Advisory Panel, which provided 

critical feedback on various drafts. Members of the panel included: Beth Blauer, director of GovStat at Socrata; 

David Coburn, partner at Capital Analytics; Marc Holzer, dean and board of governors professor, School of Public 

Affairs and Administration, Rutgers University, Newark; Michele Jolin, managing partner, America Achieves; R. 

Kirk Jonas, director of research compliance and integrity, University of Richmond; John Kamensky, senior fellow, 

IBM Center for the Business of Government; Elaine C. Kamarck, senior fellow in governance studies, Brookings 

Institution; Donald F. Kettl, professor in the School of Public Policy, University of Maryland, College Park; Donald 

Moynihan, professor, Robert M. La Follette School of Public Affairs, University of Wisconsin, Madison; John 

O’Brien, former director, Texas Legislative Budget Board; and John Turcotte, director of the Program Evaluation 

Division, North Carolina General Assembly. Although these individuals have reviewed the report, neither they nor 

their organizations necessarily endorse its findings or conclusions.

Acknowledgments

We would like to thank the following Pew colleagues for their insights and guidance: Samantha Chao, Diane Lim, 

and Karen Lyons. We also thank Kristin Centrella, Jennifer V. Doctors, Jessica Hallstrom, Jennifer Peltak, and Kodi 

Seaton, as well as former Pew staff member Nicole Dueffert, for providing valuable feedback and production 

assistance on this report. 

The Promise of Evidence-Based Policymaking 1495
Contact: Gary VanLandingham, director, Pew-MacArthur Results First Initiative 

Email: gvanlandingham@pewtrusts.org Phone: 202-540-6207 

Pew-MacArthur Results First Initiative, a project of The Pew Charitable Trusts and the John D. and Catherine T. MacArthur Foundation, 

works with states to implement an innovative cost-benefit analysis approach that helps them invest in policies and programs that are 

proved to work.

Cover photo: 

1. Getty Images/Mel Curtis 1

The Pew Charitable Trusts

901 E St. NW 

Washington, DC 20004

pewtrusts.org

The Pew Charitable Trusts is driven by the power of 

knowledge to solve today’s most challenging problems. 

Pew applies a rigorous, analytical approach to improve 

public policy, inform the public, and invigorate civic life.

The John D. and Catherine T. 

MacArthur Foundation

140 S. Dearborn St. Chicago, IL 60603

macfound.org

The John D. and Catherine T. MacArthur Foundation 

supports creative people and effective institutions 

committed to building a more just, verdant, and 

peaceful world. In addition to selecting the MacArthur 

Fellows, the Foundation works to defend human rights, 

advance global conservation and security, make cities 

better places, and understand how technology is 

affecting children and society.

1496 The Promise of Evidence-Based Policymaking
Contents

1 Overview

2 Why evidence-based policymaking?

2 A new era in responsible governance

Ongoing fiscal pressures 3

Increasing availability of evidence on what works 3

Federal funding incentives 3

Growing interest from state leaders 3

4 Key components of evidence-based policymaking

Program assessment 4

Budget development 8

Implementation oversight 10

Outcome monitoring 14

Targeted evaluation 16

18 Conclusion

19 Appendix A: Methodology

20 Appendix B: Potential roles in state government

23 Endnotes

The Promise of Evidence-Based Policymaking 1497
pewtrusts.org/resultsfirst 1

Overview

Governments make budget and policy choices each year that have long-term effects on both their fiscal 

futures and the outcomes they deliver for constituents. Recognition is growing that policymakers can achieve 

substantially better results by using rigorous evidence1 to inform these decisions, enabling governments to select, 

fund, and operate public programs more strategically. Until now, however, no comprehensive road map has 

provided clear guidance on using this approach. 

To fill this gap, the Pew-MacArthur Results First Initiative has developed a framework that governments 

can follow to build and support a system of evidence-based policymaking. Based on an extensive review of 

research and in-depth interviews with government officials, practitioners, and academic experts, the framework 

identifies steps that both the executive and legislative branches can take to drive the development, funding, 

implementation, and monitoring of policies and programs. 

The framework has five key components, each with multiple steps that enable governments to make better 

choices through evidence-based policymaking: (1) program assessment, (2) budget development, (3) 

implementation oversight, (4) outcome monitoring, and (5) targeted evaluation.

1. Program assessment. Systematically review available evidence on the effectiveness of public programs.

a. Develop an inventory of funded programs.

b. Categorize programs by their evidence of effectiveness.

c. Identify programs’ potential return on investment. 

2. Budget development. Incorporate evidence of program effectiveness into budget and policy decisions,

giving funding priority to those that deliver a high return on investment of public funds.

a. Integrate program performance information into the budget development process.

b. Present information to policymakers in user-friendly formats that facilitate decision-making.

c. Include relevant studies in budget hearings and committee meetings.

d. Establish incentives for implementing evidence-based programs and practices.

e. Build performance requirements into grants and contracts.

3. Implementation oversight. Ensure that programs are effectively delivered and are faithful to their

intended design.

a. Establish quality standards to govern program implementation.

b. Build and maintain capacity for ongoing quality improvement and monitoring of fidelity to program 

design. 

c. Balance program fidelity requirements with local needs.

d. Conduct data-driven reviews to improve program performance.

4. Outcome monitoring. Routinely measure and report outcome data to determine whether programs are 

achieving desired results.

a. Develop meaningful outcome measures for programs, agencies, and the community. 

b. Conduct regular audits of systems for collecting and reporting performance data.

c. Regularly report performance data to policymakers.

1498 The Promise of Evidence-Based Policymaking
2 pewtrusts.org/resultsfirst

5. Targeted evaluation. Conduct rigorous evaluations of new and untested programs to ensure that they 

warrant continued funding.

a. Leverage available resources to conduct evaluations. 

b. Target evaluations to high-priority programs. 

c. Make better use of administrative data—information typically collected for operational and compliance 

purposes—to enhance program evaluations. 

d. Require evaluations as a condition for continued funding for new initiatives. 

e. Develop a centralized repository for program evaluations.

This report discusses how and why evidence-based policymaking is a growing national trend and reviews the 

framework in detail to provide tips and strategies that policymakers can use to instill evidence in decision-making 

at all levels of government. 

Why evidence-based policymaking?

Evidence-based policymaking uses the best available research and information on program results to guide 

decisions at all stages of the policy process and in each branch of government. It identifies what works, highlights 

gaps where evidence of program effectiveness is lacking, enables policymakers to use evidence in budget 

and policy decisions, and relies on systems to monitor implementation and measure key outcomes, using the 

information to continually improve program performance. By taking this approach, governments can:

• Reduce wasteful spending. By using evidence on program outcomes to inform budget choices, policymakers 

can identify and eliminate ineffective programs, freeing up dollars for other uses.

• Expand innovative programs. Requiring that new and untested programs undergo rigorous evaluation helps 

determine whether they work and identifies opportunities to target funding to innovative initiatives that deliver 

better outcomes to residents or reduce costs. 

• Strengthen accountability. Collecting and reporting data on program operations and outcomes makes it easier 

to hold agencies, managers, and providers accountable for results.

A new era in responsible governance

Support is growing across the country for using evidence to inform policy and budget decisions and guide 

the implementation of programs, in good times as well as bad. Although the need to improve government 

performance has long been recognized, researchers from the Results First Initiative identified several factors 

that are driving renewed attention to this issue, including ongoing fiscal pressures, the increasing availability of 

data on program effectiveness, federal funding incentives, and state legislation that support—and in some cases 

require—the use of evidence-based programs and practices. 

Previous attempts to address these challenges by linking program performance to budget allocations—for 

example, performance-based budgeting—have met with limited success because of insufficient analytical 

capacity or limited data, among other reasons.2

 Now, with better technology, easier access to data, and the ability 

to more accurately measure the performance and cost-effectiveness of government services, policymakers have 

an opportunity to put their jurisdictions on a sustained path of evidence-based decision-making. 

The Promise of Evidence-Based Policymaking 1499
pewtrusts.org/resultsfirst 3

Ongoing fiscal pressures

In recent years, many governments were forced to make major budget reductions due to revenue shortfalls that 

occurred during the Great Recession. Although some states have seen tax revenue rebound, others continue to 

confront tight budgets due to lagging revenue, increasing costs in areas such as Medicaid, and other pressures.3

Many governments at both the state and local levels also face long-term fiscal challenges, such as meeting 

retirement benefit obligations for public employees.4 This has increased demands by policymakers for better 

information on the outcomes that programs deliver for constituents and better tools to identify activities that fail 

to deliver desired results.

Increasing availability of evidence on what works 

Over the past two decades, a growing body of research has evaluated the effectiveness of public programs. 

Multiple clearinghouses are compiling this information by reviewing and categorizing hundreds of research 

studies to identify effective and promising programs across a range of policy areas.5 As a result, policymakers 

have access to more information about what works than ever before.6 States and local governments can avoid 

duplication of effort and use this evidence to inform their policy and budget decisions.

Federal funding incentives 

Increasingly, federal grant recipients, including states and localities, are required to target federal funds to 

evidence-based programs. Since 2009, for example, the U.S. departments of Education, Health and Human 

Services, and Labor have directed approximately $5.5 billion to seven initiatives that support proven programs.7

Although they represent only a small percentage of total federal spending, these grants provide incentives for 

recipients to implement proven programs.8

 These include the Investing in Innovation (i3) Fund, which prioritizes 

education programs with strong evidence of effectiveness and evaluation of innovative programs; the Maternal 

and Infant Early Childhood Home Visiting program, which requires grantees to direct 75 percent of federal 

dollars to evidence-based programs and to evaluate the impact on key outcomes; and the Workforce Innovation 

Fund, which supports projects that use data to design new approaches to improving employment and training 

outcomes.9

Growing interest from state leaders

State policymakers are using legislation as a vehicle to encourage investment in programs that have been 

proved effective. Results First researchers identified over 100 state laws across 42 states passed between 2004 

and 2014 that support the use of evidence-based programs and practices.10 These laws provide incentives for 

agencies to implement proven programs and help establish common standards with which to compare programs.

State leaders are also using cost-benefit analysis to inform their policy and spending decisions. A recent Results 

First study found that the number of states assessing the costs and benefits of programs and policy options 

increased 48 percent between 2008 and 2011, and 29 states reported using cost-benefit studies to inform policy 

or budget decisions.11 In addition, since 2011, 16 states and four California counties have partnered with the 

Results First Initiative to apply a customized, innovative cost-benefit approach to policy and budget decisionmaking.

1500 The Promise of Evidence-Based Policymaking
4 pewtrusts.org/resultsfirst

Key components of evidence-based policymaking 

Results First researchers identified five key components that support a system of evidence-based policymaking 

(see Figure 1). In developing this report, our research found that while many states have put one or more of these 

in place, none has developed a comprehensive approach across all branches of government. For each of the 

components, our framework includes specific steps that help to ensure successful implementation. Governments 

may lack capacity to implement all of the elements at once, but they can still strengthen their use of evidencebased policymaking by focusing on particular features highlighted in this report.

© 2014 The Pew Charitable Trusts

Figure 1

Steps in Evidence-Based Policymaking

Evidence-Based 

Policymaking

Targeted 

evaluation 

Rigorously evaluate 

programs that lack 

strong evidence of 

effectiveness

Program 

assessment 

Review evidence 

of effectiveness of 

public programs

Budget 

development 

Incorporate 

evidence into budget 

and policy decisions

Outcome 

monitoring 

Determine whether 

programs are 

achieving desired 

results Implementation 

oversight 

Ensure programs are 

effectively delivered 

Program assessment. Systematically review available evidence on the 

effectiveness of public programs 

Government leaders should develop an inventory of the programs they currently operate and then assess the 

available evidence of effectiveness and return on investment for each one. This provides important baseline 

information that enables government leaders to identify which programs are working and achieving high returns 

on taxpayer dollars, which need further evaluation, and which are not delivering expected outcomes (see 

Appendix B: Potential roles in state government). 

The Promise of Evidence-Based Policymaking 1501
pewtrusts.org/resultsfirst 5

Develop an inventory of funded programs 

Many state and local governments do not have a complete catalog of the programs they fund, which is a 

necessary starting point for determining which are effective and which are not. Government leaders can require 

agencies to conduct a census to identify all publicly operated and contracted programs and collect standard 

information about each, including their funding levels, services delivered, and populations served. To help 

facilitate this process, governments often find it beneficial to develop a common definition of “program” to 

provide consistency across agencies. 

In 2014, Rhode Island’s Office of Management and Budget worked with the state’s departments of Corrections 

and Children, Youth, and Families and the judiciary to develop an inventory of 58 state-funded programs intended 

to reduce recidivism in adult and juvenile justice systems. In its initial report, published in March 2014, the office 

found that 33 percent of the programs inventoried were not evidence-based, and only two had been recently 

evaluated to determine whether they were implemented according to research-based standards. As a result of 

this process, the office recommended additional evaluations to ensure fidelity to these standards.12

Categorize programs by their evidence of effectiveness

Policymakers need clear information about the effectiveness of the programs they fund. By requiring agencies 

to categorize the programs they operate according to the rigor of their evidence of effectiveness, lawmakers and 

agency leaders can ensure they have access to the information they need to make this determination. A first 

step is to develop definitions for each category, based on the strength of evidence. For example, some states use 

“evidence-based programs,” which may be defined as requiring multiple evaluations that use rigorous methods 

such as randomized controlled trials. A second is “promising programs,” which may include those that have been 

evaluated and shown effective but through a less rigorous research design. State or local governments can use 

resources from national clearinghouses or other states in developing these definitions. 

Embedding such standards of evidence in statute can increase the likelihood that they will be enforced 

consistently and endure political changes. In 2012, Washington passed legislation to increase the number of 

evidence-based children’s mental health, child welfare, and juvenile justice services.13 The law has three key 

requirements: 

1. The Washington State Institute for Public Policy and the University of Washington Evidence-Based Practice 

Institute, in consultation with the Department of Social and Health Services, will publish definitions of 

“evidence-based,” “research-based,” and “promising practices.” To be considered an evidence-based 

program, the law requires that the benefits produced outweigh its cost. In addition, the institute and the 

university will review existing national and international research to identify programs that meet the criteria 

based on these definitions.

2. The state’s Department of Social and Health Services and the Health Care Authority will complete a baseline 

assessment of evidence- and research-based practices in child welfare, juvenile rehabilitation, and children’s 

mental health services. This includes the extent to which currently funded programs meet the standards of 

evidence, the utilization of those services, and the amount of funding received by each program.

3. The Department of Social and Health Services and the Health Care Authority must report to the governor 

and Legislature on strategies, timelines, and costs for increasing the use of evidence- and research-based 

practices.

1502 The Promise of Evidence-Based Policymaking
6 pewtrusts.org/resultsfirst

In 2014, Mississippi passed similar legislation mandating that its Legislative Budget Office and Joint Committee 

on Performance Evaluation and Expenditure Review, known as PEER, categorize programs in four state agencies 

as evidence-based, research-based, promising practices, or other programs and activities with no evidence of 

effectiveness.14 The legislation includes definitions of each evidence level to guide the work of the budget office 

and PEER.

Leveraging National Research Clearinghouses

In recent years, several national research clearinghouses have been established that conduct 

systematic literature reviews to identify effective public programs across a range of policy 

areas, including adult criminal and juvenile justice, child welfare, mental health, pre-K to 

higher education, and substance abuse.*

 Although the clearinghouses use slightly different 

criteria for evaluating the strength of evidence, most have adopted a tiered structure that 

allows researchers and policymakers to easily determine the relative effectiveness of each 

program. For example, the What Works Clearinghouse, an initiative of the U.S. Department of 

Education’s Institute of Education Sciences, uses a system of recognizable symbols to convey 

this information: two plusses mean a program has positive effects, while an oval means there is 

no evidence of an effect on outcomes.†

 The What Works Clearinghouse has rated the impact of 

approximately 130 education programs on 26 educational outcomes.

Policymakers and agency leaders can use these clearinghouses to compare the programs that 

their state or locality operates to those the clearinghouses have deemed to be effective. For 

example, a state might find that only a small percentage of its adult criminal justice programs 

had nationally recognized evidence of positive outcomes, which would raise questions about 

whether the remaining programs should continue to receive funding.‡

* There are several widely recognized national research clearinghouses, including the U.S. Department of Education’s 

What Works Clearinghouse, the U.S. Department of Justice’s CrimeSolutions.gov, Blueprints for Healthy Youth 

Development, the Substance Abuse and Mental Health Services Administration’s National Registry of Evidence-Based 

Programs and Practices, the California Evidence-Based Clearinghouse for Child Welfare, What Works in Reentry, and 

the Coalition for Evidence-Based Policy. 

† What Works Clearinghouse, U.S. Department of Education Institute of Education Sciences, accessed July 29, 2014, 

http://ies.ed.gov/ncee/wwc/findwhatworks.aspx.

‡ The Pew-MacArthur Results First Initiative recently created a central database that compiles information from eight 

research clearinghouses to enable policymakers and their staffs to readily identify effective, evidence-based programs in 

multiple policy areas, including adult criminal justice, juvenile justice, mental health, substance abuse, early education, 

K-12 education, and child welfare. For more information, please see: http://www.pewtrusts.org/en/research-andanalysis/issue-briefs/2014/09/results-first-clearinghouse-database.

The Promise of Evidence-Based Policymaking 1503
pewtrusts.org/resultsfirst 7

Over the past two fiscal years, five states—Iowa, Massachusetts, 

New Mexico, New York, and Vermont—have used the Results 

First model to target $81 million in funding to more effective 

programs that the model shows will achieve higher returns.

Identify programs’ potential return on investment

In addition to knowing whether programs have been rigorously evaluated, it is also important for government 

leaders to know if investing in them would generate enough benefits to justify their costs. Governments can use 

cost-benefit and cost-effectiveness analyses to answer this question. These studies calculate the dollar value 

of the outcomes that different programs achieve and weigh them against the costs. Conducting such analyses 

requires technical expertise and extensive fiscal and outcome data and may not be practicable for all programs. 

When feasible, however, this approach enables governments to rank programs by their potential return on 

investment, providing policymakers with critical information on which alternatives can achieve the greatest 

returns for constituents. 

The Pew-MacArthur Results First Initiative is working with 16 states and four counties to implement cost-benefit 

analysis models that enable policymakers to use this approach in their budget and policy decisions. Results First 

uses a nationally recognized, peer-reviewed model and a three-step process: 

1. Employ the best national research on program outcomes to identify what works, what doesn’t, and how 

effective various alternatives are in achieving policy goals.

2. Apply jurisdiction-specific data to predict the impact each program would achieve.

3. Compare the costs of each program to its projected benefits and produce a report that ranks each alternative 

by the relative value it would generate for taxpayers. 

Over the past two fiscal years, five states—Iowa, Massachusetts, New Mexico, New York, and Vermont—have 

used the Results First model to target $81 million in funding to more effective programs that the model shows will 

achieve higher returns.15

Getty Images/Sam Edwards

1504 The Promise of Evidence-Based Policymaking
8 pewtrusts.org/resultsfirst

Budget development. Incorporate evidence of program effectiveness into budget 

and policy decisions, giving funding priority to those that deliver a high return on 

investment of public funds 

For evidence-based policymaking to be successful, governments must systematically use evidence of 

program effectiveness to inform their processes for making budget and policy decisions. This requires regular 

communication between researchers, budget staff, and policymakers as well as the development of strong 

executive and legislative champions. Analytic results must be reported to policymakers in timely and accessible 

ways. 

Integrate program performance information into the budget development process

Executive branch agencies should use performance information when developing their budgets to ensure funds 

are directed to programs that have strong evidence of effectiveness and away from those that are not delivering 

results. To accomplish this, agencies can develop output and outcome measures for all major programs and 

report those metrics in their budget requests. Agencies should develop numerical performance targets that can 

be used by policymakers to measure progress against key benchmarks and goals. For evidence-based programs, 

the targets should reflect outcomes predicted by research. 

A well-functioning performance measurement system can help governments decide where to pull back on 

funding as well as where to provide greater support. Connecticut’s Result-Based Accountability system has 

been operating for eight years and has become an important part of the state’s appropriations process. When 

outcome measures showed that the state’s $20 million annual investment in early reading programs was having 

no positive effect on reading skills, they were first denied funding and later analyzed in-depth to identify potential 

solutions. The study found that reading specialists, a central element of the initiative, lacked sufficient training 

to achieve expected results and that funding to support early reading efforts was often used for other purposes. 

Based on this, the state has turned to other approaches, such as adding reading-related graduation requirements 

for education degrees and implementing techniques based on a reading program in Norwalk that has had 

success. “Our reading scores are now creeping up instead of going down,” said Representative Diana Urban, cochair of the Connecticut General Assembly’s Select Committee on Children.16

Present information to policymakers in user-friendly formats that facilitate decision-making

To increase the likelihood that policymakers will use evidence to inform critical budget decisions, complex 

information must be presented in ways they can readily understand and act on. For any program, policymakers 

need answers to at least three important questions: 

• Is the program working?

• Do its benefits outweigh its costs?

• How does the program compare to alternative programs? 

To provide this information, agencies can produce annual rankings that compare programs targeting similar 

outcomes based on effectiveness, cost, and benefits produced. When practicable, governments can use costbenefit analyses to calculate a return on investment for each program, providing policymakers with data on how 

to best allocate resources to achieve each agency’s goals.17 At a minimum, policy staff should compare programs 

with common goals according to their documented impact on specific outcomes—for example comparing a set of 

programs that all have as their primary goal reducing child abuse and neglect. 

The Promise of Evidence-Based Policymaking 1505
pewtrusts.org/resultsfirst 9

Several states, including Washington, Iowa, and New Mexico, have developed Consumer Reports-type analyses, 

which rank programs by their benefit-to-cost ratios.18 In 2012, the Iowa Department of Corrections issued a 

report highlighting the costs and benefits of various criminal justice programs over a 10-year period.19 The 

analysis showed that among prison-based programs, cognitive behavioral therapy programs were inexpensive to 

operate and highly effective in reducing recidivism, returning $37.70 in benefits for every dollar spent. In contrast, 

correctional educational programs, although also effective, returned only $2.91 in benefits per dollar invested.20

As a result, the department is considering expanding its cognitive behavioral therapy programs and plans to 

reduce other, less effective activities proportionally.

Include relevant studies in budget hearings and committee meetings

Policymakers can use executive and legislative budget hearings and committee meetings as opportunities to 

discuss key findings from program evaluations, audits, cost-benefit analyses, and other research. Governments 

can establish procedures requiring research offices to provide relevant reports to budget and policy committees, 

which should, in turn, be encouraged to consider the findings in their deliberations. 

The New Mexico Legislative Finance Committee regularly presents program evaluations, agency performance 

report cards, and cost-benefit analyses during budget hearings and committee meetings to support its budget 

and legislative recommendations. In 2013, for example, the committee presented a report in budget hearings 

showing that reducing recidivism by 10 percent using proven programs could save the state $8.3 million in prison 

costs and approximately $40 million in avoided costs to victims.21 The findings, in addition to other analyses, 

helped inform decisions to allocate $7.7 million to effective criminal justice programs.

Establish incentives for implementing evidence-based programs and practices 

Governments can use grant competitions to encourage adoption or expansion of evidence-based programs. 

Agencies can also partner with private philanthropies or businesses to scale up promising programs—those that 

demonstrate the potential to achieve a positive return on investment. 

Wisconsin’s Treatment Alternatives and Diversion grant program provides funding to counties to implement 

data-driven alternatives to prosecution and incarceration of criminal offenders with a history of substance abuse.

A county is eligible for a grant if, among other criteria, the services provided are consistent with evidence-based 

practices. Between 2006 and 2013, these grants funded nine county diversion or drug court programs. A recent 

evaluation found that grant-funded projects averted 231,533 incarceration days for offenders, 57 percent of whom 

were not convicted of a new crime three years after being discharged from the program.22

Governments can also develop pay-for-success models and social impact bond agreements, both of which raise 

capital from private investors or philanthropic organizations to scale up programs that have the potential to 

achieve better outcomes and save the government money. Although these efforts are still in their infancy, several 

states, including Massachusetts and New York, are moving forward with plans to provide incentives for datadriven programming. 

When practicable, contracts and grants should include performance 

goals that encourage organizations to provide evidence-based 

programs and to implement those services as designed.

1506 The Promise of Evidence-Based Policymaking
10 pewtrusts.org/resultsfirst

New York raised $13.5 million through its social impact bond to support the Center for Employment 

Opportunities, which provides evidence-based employment services to ex-offenders including job training, 

transitional employment, and job placement. Bank of America Merrill Lynch (BAML) and Social Finance Inc. 

raised funding from more than 40 individual and philanthropic investors, which included several BAML clients, 

as well as foundations, among them the Laura and John Arnold Foundation and the Robin Hood Foundation. The 

Rockefeller Foundation agreed to guarantee up to 10 percent of the investors’ principal. An independent evaluator 

will determine whether the program is reaching its goals of reducing recidivism and increasing employment.23

The state will repay investors only if the outcomes outlined in the bond agreement are achieved. 

Build performance requirements into grants and contracts 

When practicable, contracts and grants should include performance goals that encourage organizations 

to provide evidence-based programs and to implement those services as designed. To realize the benefits 

of performance-based contracts, program administrators should work closely with providers and program 

developers to create measures that accurately gauge performance, while striking a balance between the need for 

accountability and the importance of continuous quality improvement and increased capacity. These contracts 

need to be carefully crafted and monitored to protect against unintended consequences, such as creating 

incentives for providers to take only those clients most likely to succeed and to reject those considered high-risk. 

In the early 2000s, the Connecticut Judicial Branch’s Support Services Division, which oversees state-run juvenile 

justice programs, developed a Center for Best Practices to review research on evidence-based interventions and 

integrate effective strategies into current programs, most of which were contracted out.24 The center determined 

that several programs were achieving poor outcomes, and the division began working with contractors to identify 

the aspects of service delivery that yielded desired outcomes and to incorporate those elements into their 

contracts. Through this process, the division developed a standard report card, which includes performance data 

and other quality assurance information, that is updated semiannually and is reported to the Legislature each 

year. Division staff members also meet quarterly with contractors to review performance data, identify areas for 

improvement, and determine technical assistance needs.25

When properly designed, performance-based contracts can help move agencies away from a fee-for-service 

model, which pays providers for the amount of services they deliver, toward a system that rewards results. For 

example, in Tennessee, under more traditional fee-for-service contracting methods, foster care providers that 

were most successful in finding permanent homes for children could suffer financially because the children no 

longer needed their services. In contrast, the state’s pay-for-success program, which was introduced in 2009, 

provides contracts that pay more to agencies that achieve permanent placements for children. Over a five-year 

period, this helped reduce the time children spent in foster care by 235,000 days and saved $20 million, which 

has been reinvested to further improve services.26

Implementation oversight. Ensure that programs are effectively delivered and are 

faithful to their intended design

The quality of program implementation can dramatically affect outcomes: Even the most effectively designed 

interventions can produce poor results when poorly run. To ensure proper implementation, governments 

should establish strong monitoring systems that assess all funded programs, including those administered by 

nongovernmental entities. This monitoring should ensure that evidence-based programs are carried out with 

fidelity to their design and incorporate the elements that are critical to their effectiveness, and it should include 

processes that improve quality by using information gathered through monitoring to make adjustments that 

improve performance. 

The Promise of Evidence-Based Policymaking 1507
pewtrusts.org/resultsfirst 11

Too often, program support and oversight is one of the first areas cut when budgets are tight, resulting in 

inadequate implementation and poor outcomes. To sustain the positive results, policymakers should include 

funding for support and monitoring in the base budgets of programs. Then, if budgets are reduced, effective 

services can still be delivered to high-need clients, which is preferable to serving more people ineffectively by 

poorly implemented programs. 

Establish quality standards to govern program implementation 

Broad-based implementation standards can promote the consistent delivery of high-quality services by providing 

baseline requirements for monitoring and oversight. These criteria should also be included in agency contracts to 

help ensure that providers understand and comply with expectations. Evidence-based programs frequently have 

detailed implementation manuals that managers can use to set quality standards. 

For example, state leaders tasked the Washington State Institute for Public Policy with developing standards to 

implement evidence-based juvenile justice programs after an evaluation found that sites where the programs 

were not implemented with fidelity had poor results.27 The standards address four key elements of quality 

assurance—program oversight, provider development and evaluation, corrective action, and ongoing outcome 

evaluation—and include protocols for hiring, staff training and assessment, and management and oversight 

of service delivery. Providers are required to undergo an initial probationary period during which they receive 

training and feedback. Thereafter they are evaluated annually. The state regularly monitors program completion 

and recidivism rates for juveniles who receive certain services. The implementation standards are credited with 

helping the state achieve greater reductions in crime and juvenile arrest rates compared with the national average 

and a decrease of more than 50 percent in youth held in state institutions.28

Build and maintain capacity for ongoing quality improvement and monitoring of fidelity to program 

design 

Governments can support effective implementation by offering—or partnering with organizations that offer—

training, technical assistance, and other services to program providers. They can also offer infrastructure support, 

including computer systems that facilitate data collection and outcome reporting. Some nationally recognized 

evidence-based programs also provide training or technical assistance services to assist implementation. 

The Evidence-based Prevention and Intervention Support Center, or EPISCenter, provides technical assistance to 

communities and service providers in Pennsylvania to support the implementation of evidence-based prevention 

and intervention programs.29 Since 2008, the center has assisted in the establishment of nearly 300 evidencebased programs in more than 120 communities throughout the state.30 The center is a collaborative partnership 

among the Pennsylvania Commission on Crime and Delinquency and Penn State University. It receives funding 

and support from the commission and from the Pennsylvania Department of Public Welfare. Experts from the 

center provide technical assistance to local staff on implementation, evaluation, and sustainability and help 

develop the infrastructure to monitor the program for fidelity to its original design. Over time, providers build 

internal capacity for these operations and many continue to report data to the EPISCenter even after their initial 

funding has ended. These efforts have been highly beneficial. 

Balance program fidelity requirements with local needs

Many evidence-based programs have identified the key service elements that are critical to achieving desired 

outcomes but they also note that some services may need to be modified for local conditions. Administrators 

monitoring programs should ensure that key elements are implemented with fidelity while allowing other features 

1508 The Promise of Evidence-Based Policymaking
12 pewtrusts.org/resultsfirst

The goal is to meet communities where they are so that this is 

sustainable. Whatever you’re building needs to be part of the 

community you’re working with. You maintain the fidelity of the 

model, but ensure that it’s tailored to the community.” 

William Baney, director of the Systems of Care Institute at Portland State University’s Center for 

Improvement of Child & Family Services

to be adapted to meet community and cultural differences. Administrators, program developers, and service 

providers should work together to ensure that program adaptations do not negatively affect outcomes.

In 2009, the Oregon Legislature passed a bill to utilize the nationally recognized “Wraparound” system of care 

for emotionally disturbed and mentally ill children, with statewide programs in place by 2015.31 A fundamental 

part of Oregon Wraparound is fidelity monitoring, overseen by the Oregon Department of Human Services. The 

National Wraparound Initiative has provided assessment tools to ensure that programs remain faithful to its 10 

basic principles. However, administrators may adapt other services to local conditions and needs, which can 

vary across the state. “The goal is to meet communities where they are so that this is sustainable. Whatever 

you’re building needs to be part of the community you’re working with. You maintain the fidelity of the model, 

but ensure that it’s tailored to the community,” says William Baney, director of the Systems of Care Institute, 

at Portland State University’s Center for Improvement of Child & Family Services, which provides training and 

systems support to Oregon Wraparound.32

Conduct data-driven reviews to improve program performance 

Regularly scheduled data-driven performance management meetings enable agency and state leaders to 

discuss performance data, develop or refine performance objectives, identify areas for improvement, promote 

innovative strategies, foster coordination, and hold managers accountable for results.33 Agencies should hold 

similar meetings with their staffs and service providers to pinpoint opportunities for improvement and address 

performance barriers. 

This approach was developed by the New York City Police Department and popularized by the city of Baltimore 

through CitiStat. The CitiStat model allowed Baltimore leaders to focus on performance goals, improve service 

delivery, and generate $350 million in savings over a seven-year period, enabling it to reinvest $54 million in new 

programming for children.34

Using a similar approach, Maryland StateStat measures statewide performance and tracks key indicators from 

biweekly agency data, which are analyzed for trends to inform strategies for improvement. Regular meetings 

are held with the governor, agency heads, and StateStat staff to clarify goals, refine approaches for achieving 

outcomes, and track performance.35 This use of data has engendered a culture of organizational learning in which 

program managers and agency leaders discuss challenges and solve problems. 

The Promise of Evidence-Based Policymaking 1509
pewtrusts.org/resultsfirst 13

Aligning Existing Services With Key Elements of Evidence-Based Programs

Governments can often improve the outcomes from programs that are not evidence-based by 

aligning their key characteristics with those that are. For example, a locally developed program 

for juvenile offenders may be able to improve its results by incorporating features of programs 

that research shows are highly effective in reducing recidivism.

The Standardized Program Evaluation Protocol, or SPEP, developed by the Peabody Research 

Institute at Vanderbilt University, provides a standardized measure to determine how closely a 

particular program conforms to the most effective practices, according to scientific research, in 

juvenile justice.*

 The tool assesses programs in four primary areas that research has identified 

as critical to effectiveness, including the primary service provided, the quantity of service, 

the quality of delivery, and the risk level of the juveniles served. The tool is currently being 

implemented in three jurisdictions—Milwaukee County, Wisconsin, and in Iowa and Delaware. 

They are part of the federal Office of Juvenile Justice and Delinquency Prevention’s Juvenile 

Justice Reform and Reinvestment Initiative, established to support improvements to current 

service delivery models.†

 The information gathered through the tool is used by states and 

localities to improve existing juvenile justice services and align them with evidence-based 

practices without having to redesign entire service systems. Arizona and North Carolina have 

also used the SPEP tool to assess the effectiveness of their juvenile justice programs, and initial 

data show that larger reductions in recidivism correlated with higher SPEP ratings.

“The SPEP tool allows states to look at programs that may not be name brand, but to determine 

whether they have the common elements that research suggests works,” says Mark Lipsey, 

Ph.D., director of the Peabody Research Institute.‡

 “From a practical standpoint, in some policy 

areas there are relatively few evidence-based programs; they can be expensive and require 

significant training to get providers up to speed. We see our approach as complementary with 

model programs which are also part of our scheme, but it allows states to look at a broader set 

of programs.”

*

 Peabody Research Institute, “Standardized Program Evaluation Protocol,” accessed July 29, 2014, 

 https://my.vanderbilt.edu/spep. The protocol was developed by Mark Lipsey, Ph.D., of the Peabody Research Institute, 

 Vanderbilt University.

† Shay Bilchik and Kristen Kracke, “How Do You Scale Evidence-Based Programs: A look at OJJDP’s Juvenile Justice 

Reform and Reinvestment Initiative,” Cost-Benefit Knowledge Bank on Criminal Justice, (Dec. 4, 2013), accessed July 29, 

2014, http://cbkb.org/2013/12/how-do-you-scale-evidence-based-programs-a-look-at-ojjdps-juvenile-justice- 

 reform-and-reinvestment-initiative.

‡

 Pew-MacArthur Results First Initiative interview with Mark Lipsey, director, Peabody Research Institute, Vanderbilt 

 University, Jan. 8, 2014.

1510 The Promise of Evidence-Based Policymaking
14 pewtrusts.org/resultsfirst

Outcome monitoring. Routinely measure and report outcome data to determine 

whether programs are achieving desired results 

Many governments have made significant investments to build and implement performance reporting systems, 

but these too often focus on outputs, such as the number of programs provided or clients served, rather than 

results, such as reduced recidivism or increased graduation rates, and are of limited use to policymakers. 

Governments should make sure that performance measurement systems collect and report essential outcome 

data for all major programs. 

Develop meaningful outcome measures for programs, agencies, and the community 

Performance monitoring systems should provide output and outcome data that meet the information needs of 

various stakeholders, including program administrators, policymakers, and constituents. For example: 

• Administrators can monitor operations by using data on program outputs, such as the number of families 

served, the percentage of families achieving program milestones, and the caseloads of field staff. 

• Agency leaders can use intermediate outcome data to assess progress toward key goals, such as reducing the 

percentage of participating mothers who deliver low-birth-weight babies. 

• Policymakers and constituents can use measures that gauge long-term trends, such as the percentage of 

children graduating from high school, to determine whether public programs are achieving their overall 

objectives. 

For example, Virginia Performs is an interactive, publicly available database that collects and reports performance 

data on a wide range of government functions at multiple levels—including program, agency, department, and 

cross-cutting strategic government priority—and for diverse audiences such as program administrators, agency 

leadership, policymakers, and the public. As part of Virginia’s strategic planning process, state agencies identify 

performance measures, which are then tracked through the Virginia Performs system.36 These data are one set of 

inputs used to generate the annual Virginia Report, a balanced accountability scorecard created by the bipartisan 

Council on Virginia’s Future, which is headed by the governor.37 Where data are available, Virginia’s performance 

is compared with the national average, the top performing state in the nation, and three similar states. The data 

allow users to consider high-level strategic goals and a wide range of performance indicators at the department, 

agency, and program levels. 

When determining what measures to track, governments can consult resources available from several national 

organizations. For example, in 2012, the Federal Department of Housing and Urban Development launched the 

Healthy Communities Transformation Initiative to provide governments with the tools to assess the “physical, 

social, and economic roots of community health.” The initiative’s first deliverable, a collection of 28 key indicators 

that governments can use to track outcomes across 10 policy domains, was created following review of existing 

models and is now being tested in select jurisdictions. Many of the indicators can be derived from publicly 

available data and customized by state, municipality, or neighborhood.38

Agencies can also visit the national clearinghouses to identify the outcomes predicted for various programs by 

rigorous research and use those findings to set performance targets for funded programs. Governments can 

require programs that lack strong evidence of effectiveness to develop theories of change or logic models that 

specify their expected results and can then use this information to establish outcome measures and performance 

targets for those programs. 

The Promise of Evidence-Based Policymaking 1511
pewtrusts.org/resultsfirst 15

Performance measures also should periodically be examined to ensure they still serve as reliable indicators of 

success. For contracted services, governments should ensure that providers collect and report common outcome 

metrics so that officials can compare performance and aggregate the overall program effects.

Conduct regular audits of systems for collecting and reporting performance data 

Effective performance measurement systems should be user-friendly and provide data that meet the needs of 

multiple stakeholders. Even the best-designed system, however, will be of little value if the reported data are 

inaccurate or misleading. Governments should provide training to agency staff and contracted providers on how 

to collect, analyze, and report performance data, and develop processes for regularly verifying that these data are 

accurate. 

Performance measurement systems can easily fall into disuse without strong leadership supporting them or 

adequate training for providers and agency personnel. In 2012, Louisiana’s auditors confronted this issue during 

a review of the state’s performance budgeting system, once considered a model program.39 The audit noted that 

many statutory processes were no longer being followed and that reported information was not being used to 

inform budget decisions. The findings emphasized the need to increase awareness of the system, improve how 

performance data were presented to policymakers, and ensure reliability. The report also noted the importance of 

Using ‘Benchmarking’ to Gauge Performance 

Many governments are also using benchmarking—comparing their program outcomes with 

those achieved in other jurisdictions—as a way to assess performance. One example of this 

is the National Core Indicators project, which over the last decade has developed common 

sets of outcome measures, including some 60 indicators measuring personal, family, and 

health and safety, that states can use to gauge the effectiveness of the services they provide 

to developmentally disabled individuals. Currently, 40 states and the District of Columbia 

participate, with the remaining 10 expected to join by 2017.

Individual states have used the data to focus attention on problem areas. For example, 

policymakers in Kentucky found that employment of its adult-with-developmental-disabilities 

population trailed the national average substantially—18.5 percent compared with 37.8 

percent.* At the same time, National Core Indicators data showed the importance of 

employment for improving quality of life, including better relationships, increased exercise, and 

greater participation in community activities. This information spurred a number of strategies 

in Kentucky to effect change: a revision of Medicaid waivers, an emphasis on employment in 

communications developed by state agencies, more staff training, and an increase in the hourly 

rate for supported employment.

* National Core Indicators Project, “NCI Adult Consumer Survey Outcomes: Kentucky Report 2011-2012 Data,” http://

www.nationalcoreindicators.org/states/KY.

1512 The Promise of Evidence-Based Policymaking
16 pewtrusts.org/resultsfirst

training, for both legislative and agency staff, on using the system. “State agencies have all of this data but do not 

necessarily have the tools or the skill set to analyze the data and use it for performance management purposes,” 

says Karen LeBlanc, performance audit manager at the Louisiana Legislative Auditor’s office.

Regularly report performance data to policymakers 

Performance data can be a valuable tool for managing, overseeing, and assessing the value of programs, but it is 

critical to provide the information to policymakers on a regular basis, in easy-to-digest formats that highlight key 

findings, and readily translate to budget and policy decision-making. Several state and local governments have 

developed report card systems that focus on agency or program performance on key outcomes. Report card data 

are often reported through public websites and may be presented to policymakers through regular hearings and 

meetings. Data dashboards, interactive business tools that display a set of performance indicators, can also be 

beneficial in tracking and focusing on high-level outcomes in real time. 

In Michigan, for example, a frequently updated performance dashboard provides past and current data on a 

variety of indicators relevant to the administration’s key policy objectives, including economic strength, health 

and education, quality of life, and public safety.40 Policymakers and the public can quickly see which programs 

are succeeding or struggling based on simple graphics such as a green “thumbs up” for progress and a red 

“thumbs down” for a lack of achievement. For example, in spring 2014, third-grade reading test scores were 

slowly continuing to trend upward. The dashboard featured this information using a graph showing proficiency 

increasing from 63.5 percent in fiscal year 2011 to 70 percent three years later. On the other hand, the dashboard 

provided a warning signal that the self-reported percentage of students being bullied rose from 22.7 percent in 

2011 to 25 percent in 2013.41

Targeted evaluation. Conduct rigorous evaluations of new and untested programs 

to ensure that they warrant continued funding 

Programs with little or no evidence of effectiveness carry a higher risk of yielding poor outcomes. Governments 

should therefore direct evaluation resources to programs that lack rigorous outcome data, receive significant 

funding, or pose other risks in order to ensure they are delivering desired results and that further support is 

warranted.

Governments should also allocate funding for evaluation to limit the risk that investments are made in programs 

that do not work or that are less effective over time. Rather than assuming that programs can find money within 

existing budgets, governments should dedicate resources for this purpose once existing evaluation capacity and 

expertise have been maximized.

Leverage available resources to conduct evaluations

Almost all states have offices that conduct program evaluations and performance audits, and these can provide 

unbiased information to help policymakers assess program effectiveness. Governments should develop an 

inventory of their resources and dedicate at least a portion of them to conducting rigorous outcome evaluations. 

For example, legislative audit and research offices can be a critical resource in conducting independent program 

evaluations, but historically much of their work has focused on assessing compliance and management issues 

rather than outcomes. Legislators can work with these offices as they set their research agendas to identify 

opportunities to dedicate a larger portion of their resources to determining whether programs are achieving 

desired results. 

The Promise of Evidence-Based Policymaking 1513
pewtrusts.org/resultsfirst 17

Target evaluations to high-priority programs 

No government has the capacity to regularly evaluate all of its funded programs, so it is important to set 

priorities. Governments can develop a list of programs to be evaluated, weighing factors such as the program’s 

purpose, existing evidence of effectiveness, spending level, potential for cost savings, and risk of poor outcomes. 

Make better use of administrative data—information typically collected for operational and compliance 

purposes—to enhance program evaluations 

Over the past decade, researchers have made significant advances in using existing data sources to conduct 

rigorous program evaluations, for example, linking education, child welfare, and juvenile and criminal justice 

records to determine child outcomes.42 Because much of this information is already collected for other 

administrative purposes, the costs are much lower than more traditional program evaluations.

For example, Hawaii’s Opportunity Probation with Enforcement program, a supervision program for offenders 

at high risk for probation violation, was evaluated in a randomized controlled trial using existing administrative 

data sources. The state’s existing probation case-management system included records on supervision activities, 

drug test results, offenses, and other probationer interactions with the criminal justice system, and the Criminal 

Justice Information System provided comprehensive criminal record data. By linking these data sources, the 

evaluation was able to determine that the program was effective in reducing recidivism. Participants were 55 

percent less likely to be rearrested and 53 percent less likely to have their probation revoked compared with highrisk offenders who did not participate in the program.43

Require evaluations as a condition for continued funding for new initiatives 

Governments frequently operate small-scale programs as a way to test innovations before fully implementing 

them. When designing these programs, governments should specify the desired results to help managers and 

evaluators focus on specific objectives, and before financial support is renewed, outcome studies should be 

required to determine whether tested programs are effective.

In New York City, the Center for Economic Opportunity requires rigorous evaluations of all pilot programs 

to determine whether they were effective in achieving one or more of three primary goals: reducing poverty, 

encouraging savings, or empowering low-income workers to advance their careers. Center staff oversee 

monitoring and evaluation activities, working in partnership with city agencies and external research 

organizations. The center uses the results to help determine whether to expand or discontinue each program.44

Develop a centralized repository for program evaluations

As noted earlier, several national research clearinghouses are reviewing studies to identify what works in public 

programming across policy areas. Governments can support these efforts by designating a central entity to house 

the studies they conduct and requiring all agencies to submit copies of outcome evaluations and performance 

audits. This agency or unit should screen the reports, identify significant outcome findings, and incorporate 

the information into a comprehensive list of local programs. Governments can also report these studies to the 

national research clearinghouses to help expand the available knowledge base and help governments across the 

country more effectively direct funding to programs that have demonstrated strong results for residents. 

1514 The Promise of Evidence-Based Policymaking
18 pewtrusts.org/resultsfirst

Conclusion

Government leaders are increasingly using rigorous evidence to identify policies and programs that work and are 

cost-effective. To date, however, policymakers had no comprehensive road map to guide them in this endeavor. 

The framework presented in this report identifies the steps that all levels and branches of government can take 

to build and support a system of evidence-based policymaking for strategically selecting, funding, operating, 

monitoring, and evaluating public programs that deliver the best returns on taxpayer investments.

Getty Images/ TC Lin

The Promise of Evidence-Based Policymaking 1515
pewtrusts.org/resultsfirst 19

Appendix A: Methodology

Developing a framework for a system of evidence-based policymaking required a two-step approach. Results First 

staff began by reviewing extensive academic research on systems that support evidence-based policymaking. 

Second, we conducted 46 interviews with academics, practitioners, and government experts to discuss their 

research and experiences with this approach and used the information to identify activities governments should 

undertake to establish and sustain a system of evidence-based policymaking. An external panel of experts in this 

area reviewed our findings and provided valuable input on the key components and the overall report.

1516 The Promise of Evidence-Based Policymaking
20 pewtrusts.org/resultsfirst

Appendix B: Potential roles in state government

Key steps Governor’s office Agency or program 

leadership Legislature 

Program assessment: Systematically review available evidence on the effectiveness of public programs.

Develop an inventory of 

funded programs

Issue an executive order 

requiring agencies to develop 

program inventories 

Lead the agency through the 

inventory process

Enact legislation requiring a 

program inventory

Categorize programs by their 

evidence of effectiveness

Create a workgroup to lead 

the development of research 

standards

Lead the agency through the 

categorization process

Enact legislation establishing 

criteria for the levels of 

research rigor

Identify programs’ potential 

return on investment 

Require agency budget 

requests to include cost-benefit 

information when practicable

Conduct or contract out costbenefit analyses

Direct legislative research 

staff to conduct cost-benefit 

analyses

Budget development: Incorporate evidence of program effectiveness into budget and policy decisions, giving 

funding priority to those that deliver a high return on investment of public funds.

Integrate program 

performance information 

into the budget development 

process

Create standard operating 

procedures and formats for 

agencies to report performance 

data in budget requests

Develop performance 

measures and benchmarks

Direct research and agency 

staff to develop performance 

measures and benchmarks

Present information to 

policymakers in user-friendly 

formats that facilitate 

decision-making

Require executive branch 

agencies to develop Consumer 

Reports-type summaries

Develop Consumer Reports-type 

summaries

Support the development and 

use of Consumer Reports-type 

summaries

Include relevant studies 

in budget hearings and 

committee meetings

Include relevant studies in 

budget hearings and committee 

meetings; direct agencies to 

support budget requests with 

evidence of outcomes

Simplify evidence-based 

requests to include clear, 

concise, and verifiable 

information about program 

results

Require agencies to regularly 

report on program outcomes 

and evaluations and to use a 

standard format for reports 

Establish incentives for 

implementing evidencebased programs and 

practices 

Set aside funding for 

competitive grants Administer grant competitions Set aside funding in budgets for grant competitions

Build performance 

requirements into grants and 

contracts

Require that performance 

measures be incorporated in 

contracts where practicable 

Work with contracted providers 

to develop common outcomes 

for reports and provide training

The Promise of Evidence-Based Policymaking 1517
pewtrusts.org/resultsfirst 21

Key steps Governor’s office Agency or program 

leadership Legislature 

Implementation oversight: Ensure that programs are effectively delivered and are faithful to their intended 

design.

Establish quality standards 

to govern program 

implementation

Direct agencies to develop 

statewide standards

Meet with contract providers to 

gather input on standards

Incorporate standards into 

statute

Build and maintain capacity 

for ongoing quality 

improvement and monitoring 

of fidelity to program design

Emphasize the importance of 

building internal capacity to 

faithfully implement evidencebased programs 

Provide training and technical 

support to contract providers 

and local governments charged 

with implementing programs

Provide resources for training 

and technical support

Balance program fidelity 

requirements with local 

needs

Hold regular meetings to 

review implementation 

practices and gather feedback 

from providers

Conduct data-driven 

reviews to improve program 

performance

Develop a structure for 

program review meetings, 

define the roles of the 

participants, and provide 

leadership and support

Analyze data and provide it for 

the meetings

Outcome monitoring: Routinely measure and report outcome data to determine whether programs are 

achieving desired results.

Develop meaningful outcome 

measures for programs, 

agencies, and the community 

Provide leadership, 

emphasizing the importance 

of measuring outcomes; create 

workgroups to guide the 

process

Establish consistent processes 

to review measures and goals

Provide input on performance 

measurement process to 

increase its usefulness to 

decision-makers

Conduct regular audits of 

systems for collecting and 

reporting performance data

Create work groups to guide 

the process

Establish systems for collection 

and validation of data; 

administer the process

Direct legislative staff to 

participate in work groups

Regularly report 

performance data to 

policymakers

Develop a standardized 

format for reporting outcome 

data (e.g., report cards, data 

dashboards)

Collect data and develop 

reports (e.g., report cards, data 

dashboards)

Request performance data at 

relevant committee hearings

1518 The Promise of Evidence-Based Policymaking
22 pewtrusts.org/resultsfirst

Key steps Governor’s office Agency or program 

leadership Legislature 

Targeted evaluation: Conduct rigorous evaluations of new and untested programs to ensure that they warrant 

continued funding.

Leverage available to 

conduct evaluations

Create a work group to study 

resources available in the 

executive branch

Provide information to the 

work group

Create a work group to study 

resources available in the 

legislative branch

Target evaluations to highpriority programs

Develop criteria for prioritizing 

programs and provide funding 

for them in program budgets

Create a prioritized list within 

each agency

Prioritize program evaluation 

for legislative, fiscal, or 

research offices

Make better use of 

administrative data—

information typically 

collected for operational 

and compliance purposes—

to enhance program 

evaluations

Facilitate data-sharing among 

agencies

Identify university or other 

partners with experience in 

using administrative data for 

evaluations

Facilitate data-sharing among 

agencies

Require evaluations as a 

condition for continued 

funding of new initiatives

Review evaluations in the 

budget development process

Administer and monitor test 

projects

Enact legislation or include 

language in appropriations 

act requiring the evaluation 

of test projects, and review 

evaluations in the budget 

review process

Develop a centralized 

repository for program 

evaluations

Work jointly with the 

legislature to identify and staff 

the central repository

Contribute data and analysis to 

repository

Work jointly with the executive 

branch to identify and staff the 

central repository

Note: The roles of each branch of government described in this table may differ by state depending on laws and budget rules.

© 2014 The Pew Charitable Trusts

The Promise of Evidence-Based Policymaking 1519
pewtrusts.org/resultsfirst 23

Endnotes

1 Although there are several classification systems that rank the strength of evidence on program outcomes, our framework uses four 

categories that consider the rigor of research methods used and the amount of evidence available: 

• Evidence-based programs and practices have been evaluated multiple times and found to be effective using rigorous methods such 

as randomized controlled trials, statistically controlled evaluations, or a single large multisite randomized or statistically controlled 

evaluation. Typically, these programs have specified a set of procedures that allow for successful replication. 

• Research-based programs or practices have been tested using rigorous methods (usually a single randomized control study or 

multiple studies that use strong comparison group designs) but do not meet the evidence-based standard. These programs typically 

have specified a set of procedures that allow for successful replication. 

• Promising programs and practices have been tested using less rigorous research designs that do not meet the research-based 

standard. These programs and practices typically have a well-constructed logic model or theory of change. 

• Non-evidence-based programs and practices lack sufficient evidence to meet the promising standard. 

Three of the four categories—evidence-based, research-based, and promising—are based on standards initially developed in 

Washington state. See University of Washington, Washington State Institute for Public Policy, “Inventory of Evidence-Based, ResearchBased, and Promising Practices for Prevention and Intervention Services for Children and Juveniles in the Child Welfare, Juvenile Justice, 

and Mental Health Systems” (Sept. 30, 2012), http://www.wsipp.wa.gov/ReportFile/1332/Wsipp_Inventory-of-Evidence-BasedResearch-Based-and-Promising-Practices_Full-Report.pdf. 

2 Yilin Hou et al., “States Performance-Based Budgeting in Boom and Bust Years: An Analytic Framework and Survey of the States,” Public 

Administration Review (May/June 2011), 71(3): 370-388. 

3 The Pew Charitable Trusts, “State Tax Revenue Grows, but a Full Recovery Eludes 26 States” (May 19, 2014), http://www.pewtrusts.org/

en/research-and-analysis/analysis/2014/05/19/state-tax-revenue-grows-but-a-full-recovery-eludes-26-states.

4 The Pew Charitable Trusts, “The Fiscal Health of State Pension Plans Funding Gap Continues to Grow,” (April, 8, 2014), http://www.

pewtrusts.org/en/research-and-analysis/analysis/2014/04/08/the-fiscal-health-of-state-pension-plans-funding-gap-continues-togrow.

5 Examples of clearinghouses include: Blueprints for Healthy Youth Development, University of Colorado Boulder Institute of Behavioral 

Science Center for the Study and Prevention of Violence, http://www.blueprintsprograms.com; Campbell Systematic Reviews, the Campbell 

Collaboration; “What Works in Social Policy?” Coalition for Evidence-Based Policy, http://evidencebasedprograms.org; Cochrane Reviews, 

the Cochrane Collaboration, http://www.cochrane.org/cochrane-reviews; Crimesolutions.gov, Office of Justice Programs, U.S. Department 

of Justice, https://www.crimesolutions.gov/Programs.aspx; Evidence-Based Home Visiting Service Delivery Models, Health Resources 

and Services Administration, U.S. Department of Health and Human Services, http://mchb.hrsa.gov/programs/homevisiting/models.html; 

SAMHSA’s National Registry of Evidence-Based Programs and Practices, Substance Abuse and Mental Health Services Administration,

U.S. Department of Health and Human Resources, http://www.nrepp.samhsa.gov; Teen Pregnancy Prevention Resource Center, Office of 

Adolescent Health, U.S. Department of Health and Human Services, http://www.hhs.gov/ash/oah/oah-initiatives/teen_pregnancy/db/tppsearchable.html; and What Works Clearinghouse, Institute of Education Sciences, U.S. Department of Education, http://ies.ed.gov/ncee/

wwc/findwhatworks.aspx.

6 Ron Haskins and Jon Baron, “Building the Connection Between Policy and Evidence: The Obama Evidence-Based Initiatives,” 

NESTA (September 2011), http://coalition4evidence.org/wp-content/uploads/2011/09/Haskins-Baron-paper-on-fed-evid-basedinitiatives-2011.pdf. 

7 This number is based on total funding for the seven federal grant programs listed in “A Guide to Evidence and Innovation,” Interagency 

Working Group on Youth Programs, accessed July 21, 2014, http://evidence-innovation.findyouthinfo.gov/investingEvidence.

8 The U.S. Office of Management and Budget, “Next Steps in the Evidence and Innovation Agenda” (letter dated July 26, 2013), http://

www.whitehouse.gov/sites/default/files/omb/memoranda/2013/m-13-17.pdf.

9 U.S. Department of Health and Human Services, Health Resources and Services Administration, “The Maternal, Infant, and Early 

Childhood Home Visiting Program,” http://mchb.hrsa.gov/programs/homevisiting/; Health Resources and Services Administration, 

“Affordable Care Act Maternal, Infant, and Early Childhood Home Visiting Program: (2011) 10, http://www.hrsa.gov/grants/manage/

homevisiting/sir02082011.pdf; and U.S. Department of of Labor, Employment and Training Administration, “The Workforce Innovation 

Fund,” last modified March 26, 2014, http://www.doleta.gov/workforce_innovation. 

10 Results First researchers used LexisNexis to identify all legislation passed between 2004 and 2014 that included the following phrases: 

“evidence-based policymaking” (or “policy making”), “evidence-based decision making,” “evidence-based program(s),” “evidence-based 

1520 The Promise of Evidence-Based Policymaking
24 pewtrusts.org/resultsfirst

practice,” “research-based decision making,” “research-based practice,” “research-based program(s).” Some laws were omitted from 

the final list because they did not focus specifically on evidence-based policymaking. Researchers also reviewed laws that did not meet 

these criteria but were identified in previous work as supporting evidence-based policymaking.

11 The Pew-MacArthur Results First Initiative, States’ Use of Cost-Benefit Analysis: Improving Results for Taxpayers (July 2013), http://www.

pewtrusts.org/~/media/legacy/uploadedfiles/pcs_assets/2013/PewResultsFirst50statereportpdf.pdf. 

12 Rhode Island Office of Management and Budget, “Results First—Adult & Juvenile Justice Program Inventory” (March 2014), http://

www.omb.ri.gov/documents/performance/results-first/Results%20First%20Program%20Inventory%20March%202014.pdf.

13 H.B. 2536, 2012 Regular Session, Washington State Legislature, http://lawfilesext.leg.wa.gov/biennium/2011-12/Pdf/Bills/House%20

Passed%20Legislature/2536-S2.PL.pdf.

14 The four agencies are the state’s departments of Corrections, Health, Education, and Transportation; H.B. 677, 2014 Regular Session, 

Mississippi Legislature, http://billstatus.ls.state.ms.us/documents/2014/html/HB/0600-0699/HB0677SG.htm. 

15 The Pew-MacArthur Results First Initiative regularly tracks budget allocations that were informed by each state’s Results First model 

analyses. Results First regularly publishes the total allocations of Results First partner states, which go through a strict verification 

process. Total allocations have increased since our most recent publication, “Achieving Success With the Pew-MacArthur Results 

First Initiative: A State Progress Report 2011-13” (February 2014), http://www.pewtrusts.org/en/research-and-analysis/issuebriefs/2014/02/05/achieving-success-with-the-pewmacarthur-results-first-initiative, from $38 million to $81 million. This figure and 

any additional 2014 allocations will be published in early 2015. 

16 Pew-MacArthur Results First Initiative interview with District 43 Representative Diana Urban, Connecticut General Assembly, April 7, 

2014. 

17 Although cost-benefit analysis is one important tool that can help policymakers determine the best way to allocate scarce resources, the 

cost-benefit ratio produced should not be the only data used in making policy decisions. For certain programs, such as those concerning 

residents’ health and safety, policymakers may opt to maintain a program whose benefits are only slightly greater than or are less than 

the costs. 

18 Iowa Department of Corrections, “Return on Investment: Evidence-Based Options to Improve Outcomes” (May 2012), http://www.

doc.state.ia.us/Research/DOC_HandoutROI_OffenderPrograms.pdf; New Mexico Legislative Finance Committee, “Evidence-Based 

Programs to Reduce Recidivism and Improve Public Safety in Adult Corrections” (July 2013), http://www.nmlegis.gov/lcs/handouts/

CCJ%20072213%20Item%201%20LFC%20Results%20First%20Brief.pdf; Washington State Institute of Public Policy, “Prison, 

Police, and Programs: Evidence-Based Options That Reduce Crime and Save Money” (November 2013), http://www.wsipp.wa.gov/

ReportFile/1396/Wsipp_Prison-Police-and-Programs-Evidence-Based-Options-that-Reduce-Crime-and-Save-Money_Full-Report.pdf. 

19 Iowa Department of Corrections, “Return on Investment. Evidence-Based Options to Improve Outcomes” (May 2012), http://www.doc.

state.ia.us/Research/DOC_HandoutROI_OffenderPrograms.pdf. 

20 Ibid., 5. 

21 The Pew-MacArthur Results First Initiative: “New Mexico’s Evidence-based Approach to Better Governance” (August 2014), http://

www.pewtrusts.org/~/media/Assets/2014/08/NM_Results_First_Brief_web.pdf.

22 Kit R. Van Stelle, Janae Goodrich, and Stephanie Kroll, “Treatment Alternatives and Diversion (TAD) Program: Participant Outcome 

Evaluation and Cost Benefit Report (2007-2013)” (July 2014), http://uwphi.pophealth.wisc.edu/about/staff/van-stelle-kit/tad-2014-

outcomes-report.pdf.

23 Center for Employment Opportunities, Social Finance US, and the State of New York, “Investing in What Works: ‘Pay for Success’ in 

New York State—Increasing Employment and Improving Public Safety Detailed Project Summary,” accessed Oct. 5, 2014, http://www.

budget.ny.gov/contract/ICPFS/PFSProjectSummary_0314.pdf.

24 Connecticut Judicial Branch, Court Support Services Division, “Connecticut’s Probation Risk Reduction Program: A Blueprint for 

Evaluation and Increased Effectiveness of Community Based Services” (spring 2002), 6, http://www.jud.state.ct.us/external/news/

SpringSanctions.pdf.

25 Pew-MacArthur Results First Initiative interview with Cynthia Theran, assistant director of programs and services, Connecticut Court 

Support Services Division (Jan. 16, 2014).

26 The Center for Foster Care and Adoption Data, “Performance-Based Contracting in Tennessee’s Foster Care System,” https://fcda.

chapinhall.org/wp-content/uploads/2012/10/2012_TN-PBC-case-hx2.pdf.

27 Washington State Institute of Public Policy, “Recommended Quality Control Standards: Washington State Research-Based Juvenile 

Offender Programs” (December 2003), http://www.wsipp.wa.gov/ReportFile/849. 

The Promise of Evidence-Based Policymaking 1521
pewtrusts.org/resultsfirst 25

28 Washington State Institute for Public Policy, “Outcome Evaluation of Washington State’s Research-Based Programs for Juvenile

Offenders” (January 2004), http://www.wsipp.wa.gov/ReportFile/852/Wsipp_Outcome-Evaluation-of-Washington-States-ResearchBased-Programs-for-Juvenile-Offenders_Full-Report.pdf.

29 The EPISCenter is a project of the Prevention Research Center at Pennsylvania State University, jointly funded by the Pennsylvania 

Commission on Crime and Delinquency (PCCD) and the Pennsylvania Department of Public Welfare as part of PCCD’s Resource Center 

for Evidence-Based Practice. 

30 Pew-MacArthur Results First Initiative interview with Brian Bumbarger, director, Pennsylvania State University EPISCenter, and assistant 

director for knowledge translation and dissemination, Prevention Research Center (Oct. 29, 2013).

31 H.B. 2144, 2009 Regular Session, Oregon State Legislature, https://olis.leg.state.or.us/LIZ/2009R1/Measures/Text/HB2144/Enrolled.

32 Pew-MacArthur Results First Initiative interview with William Baney, director, Systems of Care Institute, Portland State University, 

March 17, 2014.

33 The data-driven performance management meeting approach was initially developed by the New York City Police Department in the 

1990s. Commonly referred to as CompStat or PerformanceStat, this format has been adopted by other state and local governments, 

notably Baltimore’s CitiStat and Maryland’s StateStat. See A Guide to Data-Driven Performance Reviews, http://www.urban.org/

UploadedPDF/1001559-Data-Driven-Performance-Reviews.pdf.

34 Robert Behn, What All Mayors Would Like to Know About Baltimore’s CitiStat Performance Strategy” (2007), http://www.

businessofgovernment.org/sites/default/files/Behn_BOG.pdf.

35 “Maryland StateStat,” accessed July 21, 2014, http://www.statestat.maryland.gov.

36 “Virginia Performs,” accessed July 21, 2014, http://vaperforms.virginia.gov. 

37 Other members of the Council on Virginia’s Future include the lieutenant governor, as well as eight members of the Virginia General 

Assembly, including the speaker of the House, the president of the Senate, chairs from both House and Senate finance committees, the 

majority and minority leaders from each house, and cabinet and citizen appointments made by legislators and the governor. 

38 Healthy Housing Solutions Inc., “HUD Healthy Communities Transformation Initiative (HCTI): National Advisory Panel Review of the 

Healthy Communities Index ‘Straw Man’” (May 3, 2013), http://www.housingnm.org/files/Library/hud_healthy_communities_index_

may2013.pdf.

39 Pew-MacArthur Results First Initiative interview with Nicole Edmonson, director of performance audit services, Louisiana Legislative 

Auditor and Karen LeBlanc, performance audit manager, Louisiana Legislative Auditor, Dec. 16, 2013.

40 “Michigan Dashboard,” accessed July 21, 2014, http://www.michigan.gov/midashboard.

41 “Michigan Education Dashboard,” accessed June 27, 2014, http://www.michigan.gov/midashboard/0,4624,7-256-58084---,00.html.

42 See, for example, Actionable Intelligence for Social Policy, http://www.aisp.upenn.edu/about-us/, and Coalition for Evidence-Based Policy, 

“Demonstrating How Low-Cost Randomized Controlled Trials Can Drive Effective Social Spending: Project Overview and Update” (Dec.

13, 2013), http://coalition4evidence.org/low-cost-rct-competition. 

43 Angela Hawken and Mark Kleiman, “Managing Drug Involved Probationers With Swift and Certain Sanctions: Evaluating Hawaii’s 

HOPE” (December 2009), https://www.ncjrs.gov/pdffiles1/nij/grants/229023.pdf. 

44 NYC Center for Economic Opportunity, “Evaluation Reports,” accessed May 19, 2014, http://www.nyc.gov/html/ceo/html/data/reports.

shtml.

1522 The Promise of Evidence-Based Policymaking
pewtrusts.org/resultsfirst Philadelphia�Washington

pewtrusts.org macfound.org

The Promise of Evidence-Based Policymaking 1523
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

PUAR puar_12603 B Dispatch: Journal: PUAR CE: 

Journal Name Manuscript No. Author Received: No. of pages: 5 TS: SPV

Bridging the Gap between Evidence and Policy Makers: A Case Study of The Pew-MacArthur Results First Initiative 1

Public Administration Review, 

Vol. xxxx, Iss. xxxx, pp. xx–xx. © 2016 by 

The American Society for Public Administration. 

DOI:10.1111/puar.12603

Torey Silloway is manager of the PewMacArthur Results First Initiative and leads 

a team of researchers aimed at advancing 

the field of evidence-based policy making. 

Previously, he served as a program manager 

at the Finance Project, as an analyst with 

the strategic consulting division of the 

Annie E. Casey Foundation, and as a policy 

and budget analyst with the Maryland 

Department of Budget and Management. 

He has a master ’ s degree in public policy 

from Georgetown University 

E-mail : tsilloway@pewtrusts.org 

Gary VanLandingham directs the 

Pew-MacArthur Results First Initiative, 

which supports evidence-based policy 

making in state and local governments. He 

previously led the Florida Legislature ’ s Office 

of Program Policy Analysis and Government 

Accountability. He has held leadership 

positions in the National Conference of 

State Legislatures, the National Legislative 

Program Evaluation Society, the Southeast 

Evaluation Association, and the North 

Florida chapter of the American Society for 

Public Administration and has published 

on public management, performance 

budgeting, cost–benefit analysis, and policy 

research utilization 

E-mail : gvanlandingham@pewtrusts.org 

Evidence in Public 

Administration

 Abstract Developing ways to bridge the long-recognized gap between researchers and policy makers is increasingly 

important in this age of constrained public resources. As noted by recent scholarship, progress toward evidence-informed 

policy making requires both improving the supply of research that is reliable, timely, and relevant to the policy process 

and promoting demand and support for this information among decision makers. This article presents a case study of 

the Pew-MacArthur Results First Initiative, which is working in a growing number of states and local governments to 

build systems that bring rigorous evidence on “what works” into their budget processes and to support its use in resource 

allocation decisions. The initiative ’ s experience to date is promising, although creating lasting and dynamic evidencebased policy-making systems requires a long-term commitment by both researchers and policy makers.

The gap between the knowledge of “what works” 

generated by researchers and the policy choices 

made by public leaders has long been a concern 

and is well documented. Factors contributing to this 

divide are numerous and include differing priorities, 

values, and incentives in the two spheres; the lengthy 

process of research creation versus policy makers’ short 

decision windows; and diverging communication 

strategies of scholars and policy makers (Bansal et al. 

 2012 ; Martin 2010 ). While several strategies have 

been explored for bridging this divide, including 

“engaged scholarship,” “evidence coproduction,” and 

“knowledge brokers,” improving research utilization 

remains a critical challenge for the field (Cairney 2016 ; 

Newman, Cherney, and Head 2016 ). 

 Nonetheless, some have proclaimed that we are in 

the “Golden Age of Evidence-Based Policy” (the 

title of the Association for Public Policy Analysis 

and Management ’ s Fall 2015 Research Conference) 

as a result of growing interest in the topic and 

rapidly expanding knowledge on the effectiveness of 

human service programs (Jennings and Hall 2011 , 

 2012 ). However, at least three things must happen 

to realize the goal of evidence-based policy making. 

First, reliable and relevant evidence must be curated, 

aggregated, and made readily available to policy 

makers. Second, techniques for incorporating this 

evidence into the budget and policy analysis processes 

must be developed. Third, support for using evidence 

must be built among policy makers. These actions 

require improving both the supply and demand for 

evidence in the policy system. 

 This article presents a case study of the PewMacArthur Results First Initiative and its efforts to 

address these challenges and promote an evidencebased approach to governing in state and local 

governments. 

The Results First Approach 

 The Pew-MacArthur Results First Initiative was 

established in 2010 by the Pew Charitable Trusts, a 

nonpartisan, nonprofit organization, and the John 

D. and Catherine T. MacArthur Foundation, one 

of the nation ’ s largest charitable foundations. The 

two entities established the initiative after noting 

that states typically resorted to across-the-board 

cuts when struggling with budget gaps during the 

Great Recession, in part because policy makers 

lacked the information they needed to make more 

strategic choices about what to cut (NCSL 2011 , 

 2012 ). 

 Results First seeks to build the capacity of 

governments to systematically use rigorous evidence 

to inform their budget and policy decisions by 

creating tools that aggregate rigorous evidence 

on “what works” across multiple social policy 

areas, providing technical assistance to enable 

state and local government staff to use these tools 

in their routine budget analyses, and providing 

communications and outreach services to build 

understanding and support for evidence use among 

policy makers. Twenty-two states and four counties 

were participating in the Results First Initiative as of 

March 2016. 

Kimberley R . Isett , Brian W . Head ,and Gary VanLandingham , Editors

Gary VanLandingham 

 Torey Silloway 

Pew-MacArthur Results First Initiative 

 Bridging the Gap between Evidence and Policy Makers: A 

Case Study of The Pew- MacArthur Results First Initiative 

1524 The Promise of Evidence-Based Policymaking
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

2 Public Administration Review • xxxx | xxxx 2016

Evidence Aggregation Tools 

 As policy makers must have ready access to information before 

they can be expected to use it, Results First has developed two 

tools that aggregate curated evidence on the effectiveness of social 

programs. These tools address “what works” in the policy areas of 

adult criminal justice, juvenile justice, child welfare, mental health, 

substance abuse, pre-K–12 education, and prevention. 

 The first of these tools is a web-based “clearinghouse of 

clearinghouses” that consolidates the evidence ratings issued by eight 

national research clearinghouses. 1

 These entities—which include 

organizations such as the U.S. Department of Education ’ s What 

Works Clearinghouse, Blueprints for Healthy Youth Development, 

and the U.S. Department of Justice ’ s CrimeSolutions.gov—screen 

evaluation studies and rate programs based on the level of rigorous 

evidence that exists about their effectiveness (Nuehoff et al. 2015 ). 

While the clearinghouses can enable users to quickly identify 

what is known about the effectiveness of a wide range of social 

programs, knowledge and use of this information by states and 

local governments has been limited because the clearinghouses 

have little capacity to publicize their work and use significantly 

different nomenclature when reporting results. For example, 

different clearinghouses designate the programs with the highest 

level of evidence of effectiveness as, alternatively, “model,” “top tier,” 

“effective,” and “3.0–4.0.” 

 The Results First Clearinghouse Database addresses these challenges 

by enabling users to access the evidence ratings of the eight 

clearinghouses through a single web portal, and it reconciles the 

varying evidence ratings using a traffic light color system: programs 

with the highest level of evidence of their effectiveness are shown 

with a green symbol; those with moderate evidence are depicted 

with a yellow symbol; and those found to have negative impacts 

are depicted with a red symbol. Consolidating this information 

also enables users to search the 1,200 programs by policy area, 

intervention type, and evidence ratings. 

 Our second evidence aggregation tool is a benefit–cost model 

that enables governments to compare the long-term return on 

investment they could achieve through funding programs across 

the social policy areas. This model is based on the well-recognized 

work of the Washington State Institute for Public Policy, which 

has been developing and using the approach for more than 20 

years (VanLandingham and Drake 2012 ). 2

 The model predicts 

and monetizes the return on investment that each program would 

achieve, based on its effect size—derived from a meta-analysis of 

rigorous evaluations—applied to jurisdiction-specific population 

and cost data. The model uses Monte Carlo simulations to assess 

the investment risk of each program (the percentage of simulations 

that generate a positive net present value), and it generates cash-flow 

and break-even analyses that disaggregate costs and benefits by level 

of government (federal, state, and local), policy area (e.g., criminal 

justice, education, and child welfare), and primary beneficiary 

(taxpayer, client, and societal). 

Use in Budget Analyses 

 Results First ’ s goal is for our partner governments to incorporate 

the tools into their routine budget analysis processes. To do so, 

we train implementation teams (which typically include budget 

and research staff within the jurisdictions’ executive and legislative 

branches) in three tasks: compiling inventories of currently funded 

programs, assessing the evidence available on these programs’ 

effectiveness in generating desired outcomes, and computing 

the programs’ return on investment. We provide this training 

and technical support through a mixture of site visits, webinars, 

conference calls, and e-mail exchanges. Partner governments 

typically initially focus their work on one or two policy areas 

(such as adult and juvenile justice) and extend their analysis into 

additional areas over time. 

 The first task—developing a program inventory—is not a trivial 

exercise, as governments often have very limited information on the 

interventions that are delivered by their networks of agencies and 

contracted providers. Developing the inventory typically requires 

reviews of budget and contract documents as well as interviews of 

agency and provider staff. 

 In the second task, the implementation teams identify the level of 

rigorous evidence that exists about the effectiveness of each locally 

funded program. They do so by matching the local programs to 

those in the Clearinghouse Database to identify the evidence rating; 

if no match is found, the team determines whether other reliable 

data is available about the programs’ effectiveness such as local 

outcome evaluations and/or performance data. This information 

is added to the completed program inventory, providing a 

comprehensive perspective of how funding is allocated in a policy 

area and whether current programs are supported by rigorous 

evidence of their effectiveness. The completed inventories typically 

include descriptions of each program and its goals, the services 

provided, the target population and number of clients served, perclient and total costs, and the evidence rating of each program. 

 In the third task, the implementation teams customize the benefit–

cost analysis model with local population and cost data and use the 

tool to examine whether currently funded programs are likely to 

generate benefits that exceed their costs. This analysis is limited to 

those programs that have a sufficiently rigorous evidence base to 

allow computation of effect sizes (approximately 250 programs are 

included in the model). The teams generate reports that compare 

the programs based on the relative return on investment that they 

could achieve if implemented with fidelity in the jurisdiction. 

Table 1 depicts the type of reports generated by our participating 

jurisdictions, showing the program inventory, return on investment 

analysis results if available, and evidence ratings of those programs 

that lack sufficient evaluations to permit benefit–cost analysis. 

Table 1 Example Results from First Benefit–Cost Analysis and Program Inventory 

Report 

Program Costs Benefits 

Benefit–

Cost Ratio 

Evidence 

Rating 

Cognitive behavioral therapy $431 $10,095 $23.42 

Vocational education $1,645 $19,594 $11.91 

Correctional industries $1,485 $6,818 $4.59 

Drug courts $4,951 $15,361 $3.10 

Intensive supervision $4,305 –$1,139 –$0.26 

Boot camps n/a n/a n/a No effects 

Veterans courts n/a n/a n/a Not rated

Note: Results are for demonstrative purposes only. 

The Promise of Evidence-Based Policymaking 1525
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

Bridging the Gap between Evidence and Policy Makers: A Case Study of The Pew-MacArthur Results First Initiative 3

 These analyses, when incorporated into the governments’ routine 

budget process, provide a more comprehensive view of how taxpayer 

funds are being spent and the likelihood that these investments will 

achieve desired outcomes. These analyses also enable policy makers 

to consider new questions when contemplating budget choices, such 

as whether funding should be shifted to alternative programs that 

have more evidence of their effectiveness and/or are predicted to 

achieve higher returns on the investment of public funds. 

Build Stakeholder Understanding and Support 

 In addition to increasing the supply of evidence available to 

governments, Results First seeks to build demand for this 

information among state and local policy makers. This reflects our 

understanding, grounded in research (see Bansal et al. 2012 ), that 

simply providing information to policy makers is not enough to 

achieve impact—they must also understand how to use the evidence 

to inform policy and budget choices and have the desire to do 

so. Accordingly, we work with our partner governments to secure 

buy-in from a broad range of stakeholders and to build this support 

over time. This typically includes outreach to legislative leadership 

and caucuses, appropriations committee chairs and members, 

executive branch budget officials, agency secretaries and senior 

managers, and key external stakeholders such as the business, media, 

and advocacy communities. To promote buy-in, we ask our partner 

governments to develop a communication plan for this outreach, 

and they have complete control over the release of analysis results. 

Impact and Lessons Learned 

 The Results First initiative has attained notable successes but 

also encountered challenges and compiled lessons learned. 

Successes include ongoing growth in the number of participating 

governments, which has expanded each year. These jurisdictions 

are successfully completing the key analytical tasks of developing 

program inventories, assessing evidence levels, and implementing 

the benefit–cost model; they are also taking steps to institutionalize 

the approach within their budget processes. We have also learned 

some key lessons, including the importance of carefully assessing 

partner capacity, the need to focus capacity building on central 

policy-making units, and the necessity of long-term engagement to 

help our partner governments develop cultures of evidence-based 

policy making. 

Assessing partner capacity . An early challenge for Results First 

involved the selection of partner jurisdictions. We recruited our 

initial partners through presentations at forums such as the Council 

of State Governments and the National Conference of State 

Legislatures, and we accepted partnerships with all states that 

expressed interest. While this enabled the project to quickly recruit 

our fi rst partners, it became apparent that some jurisdictions lacked 

the analytical capacity and political support needed to be successful; 

in some jurisdictions, staff struggled to complete project tasks, while 

in others, policy makers paid little attention to analysis results. It 

became clear that Results First, like other government reform 

efforts, requires both a signifi cant commitment of staff resources 

and broad stakeholder support to be successful (Bansal et al. 2012 ; 

Miller and Oliver 2015 ). 

 To address this problem, Results First developed an extensive 

due diligence process to assess potential partners. This process 

includes a series of conference calls, site visits, and consultations 

with government relations firms within the jurisdiction, and our 

assessment focuses on gauging whether the government has the 

requisite data capacity and supportive political landscape needed 

to achieve meaningful outcomes. If these assessments are positive, 

Results First requires letters of invitation from the leadership of the 

government ’ s legislative and executive branches before entering into 

a partnership. While this due diligence process takes more time, it 

has generated enhanced commitments to the initiative and increased 

the odds of success. 

Capacity building focused on central policy-making units. We 

also learned the importance of project team location within partner 

governments. In the initial years of the project, we were open to 

proposals to house implementation teams in a wide variety of 

organizational settings, such as agency research offi ces, interagency 

coordinating bodies, executive budget offi ces, and legislative 

research units. It became apparent that while teams housed in 

agency research offi ces typically had strong technical capacity, they 

often had weak access to executive and legislative policy makers, 

which limited the use of their analytical results. Teams housed in 

interagency coordinating bodies had stronger access to policy 

makers but often lacked access to technical staff and needed data, 

which again limited success. We found that the ideal placement of 

implementation teams was in central entities such as executive 

budget offi ces and legislative research/budget units, which 

typically had strong relationships with both executive and 

legislative offi cials as well as the ability to work collaboratively 

with agencies to analyze data. Given that Results First ’ s goal is 

to facilitate the use of evidence in the budget processes, we 

now strongly recommend that all partner jurisdictions locate 

teams in these central units. 

Need for prolonged engagement . It has become clear that 

promoting change in governments’ policy-making cultures requires 

several years to take root and a high level of understanding and 

support among diverse stakeholders. We have also learned that it is 

important to keep expectations reasonable, recognizing that 

evidence is only one of many factors that policy makers will 

consider when making choices, and it must be balanced with 

political and ideological considerations, constituent needs, and 

timing constraints (Bogenschneider and Corbett 2010 ; Jennings and 

Hall 2012 ). 

 Accordingly, Results First has sustained its work in all participating 

jurisdictions, and some of these partnerships have continued for 

over five years. For example, we began working with New Mexico ’ s 

Legislative Finance Committee in 2012, and this partnership has 

successfully informed legislative decisions to appropriate millions 

of dollars to evidence-based programs. Our work has expanded in 

recent years to focus on building capacity within the state ’ s executive 

branch agencies to use evidence tools to inform their program 

selection and implementation activities. 

 A growing number of our partner governments have adopted 

policy levers to institutionalize evidence-based policy making, 

recognizing that maintaining policy maker attention and support 

for any initiative can be difficult given the distractions inherent in 

the political process (Kingdon 1995 ). These actions include creating 

1526 The Promise of Evidence-Based Policymaking
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

4 Public Administration Review • xxxx | xxxx 2016

statutory definitions of evidence to create a common vocabulary and 

understanding among stakeholders, establishing funding preferences 

for evidence-based programs, and requiring that program 

inventories and return on investment analyses be incorporated into 

budget analyses. 

 For example, Mississippi has required agencies to report program 

inventories to the legislature, and it has required funding requests to 

be justified with answers to seven questions relating to effectiveness 

evidence, fidelity monitoring, and outcome assessment systems 

(Arinder 2016 ). New Mexico ’ s legislature has similarly required 

budget analysts to consider rigorous evidence and return on 

investment when considering budget requests. Several states have 

established mechanisms that target funding to evidence-based 

programs. For example, New York has established a grant program 

that gives preference to programs predicted to generate strong 

public safety and financial impacts; Oregon has required mental 

health, substance abuse, adult corrections, and juvenile justice 

treatment programs to allocate at least 75 percent of available 

funds to evidence-based programs. Collectively, the governments 

participating in Result First have used the approach to target over 

$158 million to evidence-based programs. 

 Our ongoing engagements have found that states and local 

governments pose different challenges in evidence-based policy 

making. In general, states have greater analytical capacity but their 

organizational scale and complex policy systems can require very longterm engagements to achieve meaningful impact. In contrast, local 

governments have more limited data capacity and require a higher 

level of technical assistance to implement evidence tools, but they can 

move much more quickly to adopt policy levers and shift funding 

once their smaller groups of policy makers reach agreement to do so. 

 A final challenge and lesson learned is that evidence-informed 

governance requires steps that are beyond the scope of Results First. 

While directing funding to evidence-based programs is a critical first 

step toward improving government outcomes, this goal will only be 

achieved if these programs are implemented with fidelity to their 

treatment models. Governments also need to create systems that 

regularly track program outcomes to determine if desired outcomes 

are being achieved, and they must rigorously evaluate new and 

innovative programs to continue to their build knowledge of “what 

works” (Pew Charitable Trusts 2014 ). Building such integrated 

policy-making systems will require partnerships with other 

initiatives that focus on these important steps, and we are actively 

seeking such collaborations in our partner jurisdictions. 

Conclusion 

 Our experience in the Pew-MacArthur Results First Initiative shows 

that it is possible to help governments increase their use of evidence 

in their budget processes by providing tools that make evidence 

easier to access, offering technical assistance to help their staff 

incorporate evidence and investment analyses into routine budget 

processes, and promoting knowledge and support for using evidence 

among policy makers to inform their policy and budget choices. 

While this work is challenging, participating governments are 

taking steps to increase evidence use, target funds to evidence-based 

programs, and create policy frameworks that will help sustain these 

efforts in the long term. 

 However, truly bridging the gap between researchers and policy 

makers will require sustained actions by both governments and the 

academy. Additional research is needed to expand the knowledge 

of “what works” and to test whether evidence-based programs 

achieve predicted outcomes when implemented in the field. 

The analytical tools that enable governments to readily use this 

information in the policy process must continue to be developed, 

and governments will continue to need training and technical 

assistance to implement these techniques. Further, evidencebased analytical and management techniques should become 

part of the public administration curriculum so that new public 

managers are equipped to effectively use these tools and deliver 

evidence-based programs with fidelity. These steps will not occur 

overnight. Nonetheless, the promise of evidence-based policy 

making—achieving materially better outcomes by targeting funds 

to those interventions that have been shown to be highly effective in 

achieving desired results—provides a compelling case for continuing 

these efforts. 

Notes 

 1 . See the Results First Clearinghouse Database at http://www.pewtrusts.org/en/

research-and-analysis/issue-briefs/2014/09/results-first-clearinghouse-database . 

 2 . The Results First model uses a cloud-based software platform that is based on 

the Microsoft Excel–based application developed and used by Washington State 

Institute for Public Policy. Once customized, the model can be used by multiple 

users in a jurisdiction to conduct analyses and has extended analyses and report 

production functions. 

References 

 Arinder , Max K. 2016 . Bridging the Divide between Evidence And Policy in Public 

Sector Decision Making: A Practitioner ’ s Perspective . Public Administration 

Review 76 ( 3 ): 394 – 98 . 

 Bansal , Pratima , Stephanie Bertels , Tom Ewart , Peter MacConnachie , and James 

 O ’ Brien . 2012 . Bridging the Research–Practice Gap . Academy of Management 

Perspectives 26 ( 1 ): 73 – 92 . 

 Bogenschneider , Karen , and Tom Corbett . 2010 . Evidence-Based Policymaking: 

Insights from Policy-Minded Researchers and Research-Minded Policymakers . 

 New York : Routledge . 

 Cairney , Paul . 2016 . The Politics of Evidence-Based Policymaking . London : Palgrave 

Macmillan . 

 Jennings , Edward T. , and Jeremy L. Hall . 2011 . State Agency Attention to Scientific 

Sources of information to Guide Program Operations . In Evidence-Based Public 

Management: Practices, Issues, and Prospects , edited by Anna Shillabeer , Terry F. 

 Buss , and Denise M. Rousseau , 199 – 212 . Armonk NY : M. E. Sharpe . 

 ——— . 2012 . Evidence-Based Practice and the Use of Information in State Agency 

Decision-Making . Journal of Public Administration Research and Theory 22 ( 2 ): 

 245 – 66 . 

 Kingdon , John W. 1995 . Agendas, Alternatives, and Public Policies , 2nd ed. New York : 

 Longman . 

 Martin , Steve . 2010 . Co-Production of Social Research: Strategies for Engaged 

Scholarship . Public Money and Management 30 ( 4 ): 211 – 18 . 

 Miller , David , and Mike Oliver . 2015 . Engaging Stakeholders for Project Success . White 

paper , Program Management Institute . http://www.pmi.org/~/media/PDF/

learning/engaging-stakeholders-project-success.ashx [accessed May 26, 2016]. 

 National Conference of State Legislatures (NCSL) . 2011 . State Budget Update: 

Fall 2011 . http://www.ncsl.org/portals/1/documents/fiscal/fallsbu2011final_

freeversion.pdf [accessed May 26, 2016]. 

 ——— . 2012 . State Budget Update: Summer 2012 . http://www.ncsl.org/portals/1/

documents/fiscal/SummerSBU2012_Free.pdf [accessed May 26, 2016]. 

The Promise of Evidence-Based Policymaking 1527
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

Bridging the Gap between Evidence and Policy Makers: A Case Study of The Pew-MacArthur Results First Initiative 5

 Newman , Joshua , Adrian Cherney , and Brian W. Head . 2016 . Do Policymakers 

Makers Use Academic Research? Reexamining the “Two Communities” Theory 

of Research Utilization . Public Administration Review 76 ( 1 ): 24 – 32 . 

 Neuhoff , Alex , Simon Axworthy , Sara Glazer , and Danielle Berfond . 2015 . The 

What Works Marketplace: Helping Leaders Use Evidence to Make Smarter Choices . 

 Bridgespan Group , April. http://www.bridgespan.org/getmedia/7761b06c-680d4a4a-a786–4de3ed528822/The-What-Works-Marketplace.aspx [accessed May 

26, 2016]. 

 Pew Charitable Trusts . 2014 . Evidence-Based Policymaking: A Guide for 

Effective Government . http://www.pewtrusts.org/en/research-and-analysis/

reports/2014/11/evidence-based-policymaking-a-guide-for-effective-government 

[accessed May 26, 2016]. 

 VanLandingham , Gary , and Elizabeth Drake . 2012 . Results First: Using EvidenceBased Policy Models in State Policy-Making . Public Performance and 

Management Review 35 ( 3 ): 550 – 63 . 

1528 The Promise of Evidence-Based Policymaking
1

New York’s Investment in 

Evidence-Based Policymaking

A case study from the Pew-MacArthur Results First Initiative Jan 2016

Overview

The Pew-MacArthur Results First Initiative can help states that are already committed to evidence-based 

policymaking take their work to the next level by integrating research and analysis into everyday decision-making. 

In New York, the Results First approach has been used to inform program and budget decisions as well as to 

enhance the state’s evidence-based alternatives to incarceration. Areas of focus include:

• Investing in analysis by creating a robust, customized Results First benefit-cost model that helped leaders 

identify cost-effective, evidence-based interventions that can reduce criminal recidivism and generate 

government savings, with some interventions projected to return more than $4 for each dollar spent.

Getty Images

The Promise of Evidence-Based Policymaking 1529
2 pewtrusts.org/resultsfirst

The development of New York’s Results First benefit-cost model, which DCJS analysts built in about two years with 

guidance from the Results First project, provided policymakers with a succinct list of evidence-based interventions 

that included detailed, state-specific calculations of how each program could affect the number of criminal 

victimizations and the amount of government spending. (See Appendix A.) In estimating program benefits, analysts 

chose to apply a relatively short period for calculating a return on investment (five years, versus the seven to 10 

years used by some other Results First states) and to focus on direct benefits accrued to state government, in order 

to ensure that results resonated with policymakers. “We knew an excessively long period of return on investment 

• Investing in evidence by securing more than $60 million over three years targeted to effective evidence-based 

programs through new grant initiatives, including about $50 million in state general funds and $12 million in 

“Pay for Success” funding from the U.S. Department of Labor.

• Investing in outcomes by requiring grant recipients to show that the programs are being implemented 

according to their original design—demonstrating fidelity—and that they are achieving expected outcomes.

Although a variety of factors account for New York’s successful application of the Results First approach, 

leaders point to five key lessons learned that bolstered their efforts and will be important for other jurisdictions 

embarking on this work: the need for dedicated staff, careful timing, strategic focus, relationship building, and 

ongoing commitment.

Investing in analysis

New York joined Results First in 2012 with considerable internal assets, including an advanced technical staff, 

a rich criminal justice data warehouse, and strong agency leadership committed to using evidence in funding 

decisions.1 These agency leaders wanted to better leverage their internal assets to develop a consistent, formal 

benefit-cost methodology that would strengthen the decision-making processes already in place2

 and help 

policymakers prioritize limited state resources toward evidence-based criminal justice programming. 

The state’s criminal justice leaders wanted to examine their investment in alternatives-to-incarceration 

programming, which consists of community-based interventions that focus on treatment. Proponents of such 

programs cite positive outcomes such as improved public safety and reduced corrections spending. But New 

York had large gaps in knowledge about what results the state could reasonably expect from the more than 

170 community justice programs it was funding.3

 “We had limited resources and wanted to reduce crime—to 

provide the best programs that get the most out of taxpayer dollars—but at the time we had no idea whether 

we were funding the right interventions for our population or what we were getting for our money,” said Michael 

C. Green, executive deputy commissioner of the New York State Division of Criminal Justice Services (DCJS), 

which oversees Results First in New York.4 “Before Results First, our primary role was ensuring recipients used 

their grant funds the way they said that they would, such as paying for salaries or providing services. Our funding 

decisions were often based on anecdote. There was little emphasis on long-term public safety outcomes or return 

on investment.”

Before Results First ... our funding decisions were often based on 

anecdote. There was little emphasis on long-term public safety 

outcomes or return on investment.”

—Executive Deputy Commissioner Michael C. Green, New York State Division of Criminal Justice Services

1530 The Promise of Evidence-Based Policymaking
pewtrusts.org/resultsfirst 3

Results First: A Model for Cost-Effective Policy Choices

The Results First Initiative, a project of The Pew Charitable Trusts and the John D. and Catherine T. 

MacArthur Foundation, works with states and localities to develop the tools that policymakers need 

to identify and invest in effective programs that yield high returns on investment. Using innovative 

methods that can be customized, Results First partners learn to:

• Create an inventory of currently funded programs.

• Review which programs work.

• Conduct benefit-cost analysis to compare programs’ likely return on investment. 

• Use evidence to inform spending and policy decisions.

These efforts have helped leaders improve public outcomes, reduce costs, and increase accountability 

by ensuring that resources go to effective, cost-beneficial approaches.

wouldn’t work in this state. Leaders want to see tangible results and cost savings that are relevant to the state’s 

budgeting cycle,” said Marc Schabses, the cost-benefit coordinator for DCJS.5 The agency also staggered the 

distribution of its two project reports—one detailing programs’ expected impact on criminal victimizations6

 and one 

examining the return on investment expected from those interventions7—to “ensure that the conversation focused 

first on community safety,” explained Deputy Commissioner Theresa E. Salo.8 “After all, the real benefit of our 

investment is not only avoided costs; it’s the impact of these programs on the safety of our state.”

New York’s Results First analysis identified several incarceration-based and community-based programs that 

were likely to produce a positive public safety and financial impact. Community-based employment programs 

emerged as a clear “safe bet,” generating as much as $2.58 in taxpayer benefits for every $1 invested (based on 

meta-analytic findings), as did cognitive behavioral interventions,9

 which were expected to return as much as 

$2.52 for every dollar invested. (See Appendix A.) Although evidence-based employment and cognitive programs 

had been priorities for a number of years, their now-quantifiable effect on crime and spending solidified their 

importance in the state’s criminal justice portfolio—and helped guide legislative funding. 

Investing in evidence

Around the time that New York was building its Results First model and beginning to examine its communitybased justice programs, the state also confronted a problem: Approximately 18 percent of funds ($3.5 million) 

for these programs came from federal stimulus money set to expire in 2012.10 This reduction in federal funding 

presented a perfect opportunity for the state to use its Results First work to rethink how and on what type of 

programming to allocate resources. DCJS requested $5 million from the state’s general fund to replace the 

expiring federal dollars, which it proposed to invest in evidence-based programs expected to return savings to 

the state. According to agency senior staff, early Results First benefit-cost calculations were critical to persuading 

decision-makers in the executive branch and Legislature to approve this request. “Our legislators want to make 

the most of taxpayer dollars, and Results First was the way to do that. They saw that a $5 million investment 

today would save them even more money in the long term,” said Commissioner Green. 11

The Promise of Evidence-Based Policymaking 1531
4 pewtrusts.org/resultsfirst

New York’s Key Players

• The Governor’s Office of Public Safety directs criminal justice-related policy and legislative matters 

on behalf of the governor and oversees eight public safety agencies: the Commission of Correction, the 

Department of Corrections and Community Supervision, the Division of Criminal Justice Services, the 

Division of Homeland Security and Emergency Services, the Division of Military and Naval Affairs, the 

Division of State Police, the Office for the Prevention of Domestic Violence, and the Office of Victim 

Services. 

• The Division of Criminal Justice Services, which oversees New York’s Results First work, is a criminal 

justice support agency with a variety of responsibilities, including the collection and analysis of 

statewide crime data and the administration of state and federal grants. Its Office of Probation 

and Correctional Alternatives oversees county probation departments and community correction 

programs.

• The Department of Corrections and Community Supervision is responsible for the confinement and 

habilitation of approximately 53,000 individuals in custody at 54 state facilities, along with 36,000 

parolees supervised by seven regional offices.

• The Commission of Correction promulgates standards for the management of correctional facilities; 

evaluates, investigates, and oversees correctional facilities; and assists in developing new correctional 

facilities.

• The Division of Budget assists the governor in preparing the executive budget proposal, offers fiscal 

policy advice to the governor’s office, and administers and monitors expenditures authorized by the 

enacted budget.

The infusion of state funds catalyzed a larger restructuring of DCJS investments in incarceration alternatives. The 

agency crafted two new funding strategies: a $5.1 million competitive grant for programs that sought to “deliver 

effective service interventions at a competitive unit cost per participant”12 and a $5.8 million noncompetitive 

continuation grant to service providers that had received stimulus funding in the previous fiscal year.13 Both 

of these funding initiatives required the implementation of cost-effective, evidence-based programs, with an 

emphasis on cognitive behavioral and employment interventions that demonstrate positive outcomes. Both 

also subjected the applicants to new performance standards and required that programs use a state-approved 

validated risk assessment instrument14 to target their services to higher-risk populations. For the fiscal year 2015-

16 budget process, the agency placed similar program requirements on state-funded county re-entry task forces, 

which traditionally had not provided direct evidence-based services.15

According to DCJS senior staff, the Results First process helped to inform the requirements in these grant 

solicitations and in the scoring of applications. Now, instead of being selected through legislator preference or 

outdated formulas, programs must demonstrate their adherence to evidence-based practice; show quantifiable, 

tangible results; and agree to submit to fidelity reviews. 

1532 The Promise of Evidence-Based Policymaking
pewtrusts.org/resultsfirst 5

FY12-13 FY13-14 FY14-15 FY15-16 Total

Total alternatives-to-incarceration 

appropriations $16.3 $24.6 $24.5 $24.9 $90.3

Competitive or conditioned 

evidence-based/targeted 

programming

$4.1 $15.1 $15.6 $18.7 $53.5 

Noncompetitive programming $12.2 $9.5 $8.9 $6.2 $36.8 

Percentage of funds targeted for 

evidence-based programming 25% 61% 64% 75% N/A

Source: Internal communication, New York State Division of Criminal Justice Services, 2015

© 2015 The Pew Charitable Trusts

Table 1

State General Fund Appropriations for Alternatives to Incarceration

Millions of dollars

New York also leveraged its Results First analysis to compete for and win a $12 million “Pay for Success” grant 

from the U.S. Department of Labor to expand evidence-based employment programs expected to generate 

cost savings.16 New York was one of only two recipients of this grant nationwide.17 State leaders attributed the 

successful bid to the strength of the Results First work, which quantified the financial and public safety value of 

investing in employment services for high-risk, recently released parolees. The state partnered with the Center 

for Employment Opportunities, which is using the award to expand its delivery of evidence-based employment 

programs to annually serve an additional 500 recently released, high-risk parolees with substantial employment 

needs.

The Department of Corrections and Community Supervision is also using Results First analyses to direct 

resources to programs that have proved effective and to rethink how to address problems for which research 

has yet to find a successful solution. One state that has taken such an approach is Iowa, where the Department 

of Corrections used its Results First analysis to replace an ineffective domestic violence program with a new 

model based on evidence-based practice.18 Like Iowa, New York was “surprised to see that existing domestic 

violence interventions were unlikely to work ... but also inspired to be creative, thinking about different 

strategies—like containment models—that could be applied for this population,” noted Deputy Commissioner 

Thomas J. Herzog.19 Acting Commissioner Anthony J. Annucci explained that his agency wants to “use a proven 

The Promise of Evidence-Based Policymaking 1533
6 pewtrusts.org/resultsfirst

methodology and translate data into a ‘dollars and cents’ message that the general public can understand. At the 

end of the day, we have to know what we are funding and what results we can expect from that investment.”20

New York’s investment of approximately $50 million from the state’s general fund for evidence-based 

alternatives to incarceration from fiscal 2013-14 to 2015-16 represented an increase in the percentage of state 

dollars dedicated to proven programs from about 25 percent in fiscal 2012-13 to 75 percent in fiscal 2015-16. 

(See Table 1.) Factoring in the grant from the U.S. Department of Labor, New York has allocated more than $60 

million for evidence-based programs in the three years since it joined Results First.

[Our agency wants to] use a proven methodology and translate 

data into a ‘dollars and cents’ message that the general public can 

understand. At the end of the day, we have to know what we are 

funding and what results we can expect from that investment.”

—Acting Commissioner Anthony J. Annucci, Department of Corrections and Community Supervision

Competitive or conditioned evidence-based/targeted programming Percentage of funds targeted for evidence-based programming

0

$10

$20

$30

$40

Millions

0

25%

50%

75%

100%

Percentage

FY12-13 FY13-14 FY14-15 FY15-16

$

4.1

$

15.1 $

15.6

$

18.7 25%

61% 64%

75%

Source: Internal communication, New York State Division of Criminal Justice Services, 2015

© 2015 The Pew Charitable Trusts

Figure 1

State General Fund Appropriations for Evidence-Based Alternatives 

to Incarceration 

1534 The Promise of Evidence-Based Policymaking
pewtrusts.org/resultsfirst 7

Investing in outcomes

To get the results it wanted—reduced crime as well as increased savings—New York needed to monitor the 

implementation and outcomes of the evidence-based programs in which it had invested. As Marc Schabses of 

DCJS explained: “Washington State’s early experiences were really eye-opening for us: programming decisions 

were made based on [benefit-cost] simulations, but the outcomes were not as expected because of poor 

program implementation. It demonstrated that making decisions or steering funding based on solid data was not 

necessarily enough.”21 In other words, it is vital that evidence-based programs are implemented according to their 

original design if the anticipated outcomes are to be realized.

DCJS reformed its existing performance-based contracting system to include case-level performance monitoring 

and regular fidelity reviews aimed at correcting issues early and ensuring that programs meet targeted outcomes.

• Performance monitoring: DCJS has engaged in performance-based contracting for a number of years, 

requiring providers to demonstrate achievement of performance milestones in order to receive funding. 

Although it is useful for holding programs accountable, the system did not track client demographics or 

risk. With the introduction of Results First and the increased focus on evidence, the quality of proposals and 

the use of data by grantees began to improve. The agency now requires programs to submit additional data 

on clients, which analysts match to the state’s criminal justice data warehouse to prepare quarterly reports 

with information on client criminal history and in-program criminal activity. According to DCJS staff, these 

new data have helped providers better understand, track, and address the needs of clients through targeted 

interventions, and they have helped agency staff to identify and resolve issues at an early stage without going 

through a formal, lengthy program review. The state is also using the newly collected case-level participant 

information to conduct long-term comprehensive evaluations of program participant recidivism with outcome 

measures, comparison groups, and follow-up periods specifically tailored to individual programs.

• Fidelity reviews: An ambitious new fidelity monitoring initiative—seeded with $128,000 in general fund 

money—is helping to ensure that programs achieve outcomes through implementation that is faithful to the 

original design. New York has contracted with the University of Cincinnati Corrections Institute to train teams 

from John Jay College of Criminal Justice and Rochester Institute of Technology, which are being paid by the 

state to conduct fidelity assessments of individual program sites using the institute’s Corrections Program 

Checklist. Once the reviews are completed, two new DCJS “action planners” provide intensive technical 

assistance to help programs correct issues identified during the review. Supplemental tools such as online 

trainings in evidence-based practices are being rolled out for use by all state-funded programs.

Leaders and staff within DCJS recognize that increased performance reporting and fidelity monitoring are timeand resource-intensive for both providers and agency staff but are also necessary for achieving outcomes. Leigh 

Bates, research manager at DCJS, explained the importance of this process to the state’s Results First work: 

“These reviews—and the intensive technical assistance that follows—are really the only way to bring this work 

full circle. We can’t tell providers that they are administering an evidence-based program wrong and then not 

offer support to change it.”22 The state will continue its investment in fidelity monitoring by using a portion of the 

local assistance appropriations.

The agency also committed to supplying providers with the resources they needed to become evidence-based, 

including a substantial focus on training in principles and specific programs. Providers accepted the new 

requirements and more intensive monitoring without the opposition one might expect. “Some of the providers 

The Promise of Evidence-Based Policymaking 1535
8 pewtrusts.org/resultsfirst

were just as excited as we were about the new requirements. They have the same goal—to get the best outcomes 

for the people they serve—and we wanted to put them in a position to succeed,” said Yvonne Behan, director of 

the Office of Program Development and Funding within DCJS.23

The state’s increased focus on accountability, support, and results has noticeably strengthened the quality of 

programs offered to reduce recidivism in New York. Deputy Commissioner Robert Maccarone, who directs the 

Office of Probation and Correctional Alternatives, said: “While providers had offered some evidence-based 

programming in the past, once they understood that DCJS was moving towards a higher level of rigor, they 

responded with great proposals that incorporated the evidence-based practices we were looking for.”24

Lessons learned

Although a variety of factors account for New York’s successful application of the Results First approach, agency 

leaders point to five key elements—staff, timing, focus, relationships, and commitment—that other states might 

consider when embarking on this work. 

• Dedicated staff: DCJS leaders underscore the importance of cultivating a technical and collaborative staff 

able to develop a robust benefit-cost model, which requires coordination of data from several agencies, 

and to translate its results into concrete policy and funding decisions that leaders trust. “Our agency has 

sophisticated and highly skilled technical staff to do the work who have earned the trust of both state 

leadership and providers. We also have strong support from state leadership, and a growing agency culture 

excited for change,’” said Commissioner Green.25 Where they did not have staff, agency leaders brought in 

external fidelity monitors and created two positions to ensure sufficient technical assistance.

• Careful timing: New York opted to approach its Results First work gradually, which senior staff members 

recommend to other states contemplating this work. Marc Schabses explained that the team “did not get 

ahead of itself. We did not commit to saving a specific amount of taxpayer dollars before our work began. 

Instead, we integrated our analysis into funding decisions thoughtfully, looking for opportunities that made 

sense and working collaboratively to implement changes.”26 In New York, this opportunity came in the form of 

the federal government’s expiring stimulus dollars, which presented the agency with the chance to rethink how 

it wanted to fund programs and to introduce new requirements for contracts. Although these opportunities 

will vary by state, agency senior staff members agree that they must not be forced, and successful results 

should not be overpromised. 

• Strategic focus: The team set an expected return on investment based on a short period of time (five years) 

and projected benefits accrued solely by state government. This decision was intended to produce realistic 

results that the agency felt it could actually achieve—and that legislators would trust. “We developed a 

conservative model that resonates with stakeholders, focusing on clear and objective information so that 

even the most cautious people would see that the analysis is solid,” said Deputy Commissioner Salo.27 The 

substantial investment of staff, time, and money into performance reporting and fidelity monitoring reflects 

DCJS’ commitment to achieving these outcomes. As Deputy Commissioner Maccarone said, “It is important 

that we keep up our end of the bargain by delivering results.”28

• Relationship building: Agency leaders reiterate the importance of engaging providers in the process, treating 

them as partners in a larger effort to achieve outcomes for clients. “Our agency was not only telling providers 

what we wanted them to do, but also saying that we would help get them there,” said Behan, of the DCJS 

Office of Program Development and Funding.29 In practice, that meant offering a series of information and 

1536 The Promise of Evidence-Based Policymaking
pewtrusts.org/resultsfirst 9

training sessions with current and prospective service providers to ensure that they understood and had 

opportunities to meet the new high standards. 

• Ongoing commitment: Leaders also speak of adopting a Results First “lifestyle” in which research and 

analysis informs both program selection and monitoring. “Results First has grown from just a tool to the way 

we do business,” Deputy Commissioner Salo said.30 “Every time we make a major decision about programs, 

we review data to better understand the population being served and review research to identify the costs 

and benefits of different interventions. We also monitor and evaluate those programs after we implement 

them to make sure we get the results we expected.” This process has been incorporated into the state’s 

decision-making business model for criminal justice programming, as shown below. 

Next steps for Results First in New York

The Division of Criminal Justice Services plans to incorporate evidence-based requirements into more of its 

funding streams, including juvenile justice programming. “The beauty of this approach is that it is very open and 

transparent. Everything that we fund must meet a high standard, and providers receive the tools they need to 

meet that standard. There are no surprises,” said Commissioner Green. “Equally important is the human side to 

this work. We want to use our resources in a way that gives people the best possible chance to break the cycle of 

recidivism and improve their lives.”31

Appendix

The New York Division of Criminal Justice Services used its Results First benefit-cost model to estimate the 

cost and impact of various criminal justice interventions. Agency leaders have used these data to identify which 

interventions are most likely to improve public safety and produce a positive return on investment. 

New York State’s Decision-Making Business Model for 

Criminal Justice Programming

1. Analyze population and program needs.

2. Recommend programming through cost-benefit analysis.

3. Implement programming.

4. Verify program quality (fidelity).

5. Evaluate program outcomes.

6. Confirm that results are as expected.

7. Use results to inform future funding decisions.

Source: Marc Schabses, “Cost Benefit Analysis for Criminal Justice: Deployment and Initial Application of the Results First Cost Benefit 

Model,” New York State Division of Criminal Justice Services (2013), http://www.criminaljustice.ny.gov/crimnet/ojsa/resultsfirst/

rftechnical_report_cba1_oct2013.pdf

The Promise of Evidence-Based Policymaking 1537
10

Population 

receiving 

programming

5-year cumulative 

reconviction rate

Gross monetary benefits

(per program participant Cost of 

programming 

per 

participant

Net monetary benefits (per program participant)

Reduction in 

victimizations 

(per 100 program 

participants)

Taxpayer only Total

Baseline 

recidivism

Recidivism 

with 

programming

Taxpayer 

benefits

Victim 

benefits

Total 

benefts

Benefits 

minus 

costs

Benefit-tocost ratio Benefits minus costs Benefit-to- cost ratio

State inmate (prison) programming modalitiesBasic skills

General education in prison Meta-analytic effect size: -0.238 Evaluations in meta-analysis: 11 Prison general 47% 36% $2,368 $458 $2,827 $1,493a,i $875 $1.59 $1,334 $1.89 15.0

Vocational education in prison Meta-analytic effect size: -0.238 Evaluations in meta-analysis: 11

Prison general 47% 36% $2,249 $435 $2,683 $2,295a -$46 $0.98 $388 $1.17 14.1

Prison high risk 69% 57% $3,145 $918 $4,062 $2,295a $850 $1.37 $1,767 $1.77 20.7

Correctional industries in prison Meta-analytic effect size: -0.078 Evaluations in meta-analysis: 9 Prison general 47% 43% $816 $157 $973 $185a,h $631 $4.41 $788 $5.26 5.2

Employment: Basic training/ job readiness

Meta-analytic effect size: -0.074 

Evaluations in meta-analysis: 16

Prison general 47% 43% $781 $151 $932 $940a -$159 $0.83 -$8 $0.99 4.9

Prison high risk 69% 65% $1,031 $301 $1,332 $940a $91 $1.10 $392 $1.42 6.6

Prison low risk 17% 15% $299 $110 $410 $940a -$641 $0.32 -$530 $0.44 1.4

Chemical dependency

Drug treatment while incarcerated: Therapeutic community

Meta-analytic effect size: -.119 

Evaluations in meta analysis: 18 Prison general 47% 41% $1,229 $238 $1,467 $1,828a -$599 $0.67 -$361 $0.80 7.9

Drug treatment while incarcerated: Residential or IOP Meta-analytic effect size: -.172 Evaluations in meta analysis: 6 Prison general 47% 39% $1,747 $338 $2,085 $1,292a $455 $1.35 $793 $1.61 10.9Drug treatment while incarcerated: Standard outpatient Meta-analytic effect size:-.173 Evaluations in meta analysis: 8 Prison general 47% 39% $1,743 $338 $2,081 $426a $1,317 $4.09 $1,655 $4.88 11.1

Behavioral

Cognitive behavioral intervention

Meta-analytic effect size: -.125 

Evaluations in meta analysis: 38

Prison general 47% 41% $1,279 $248 $1,526 $1,024a $255 $1.25 $502 $1.49 8.1

Prison high risk 69% 62% $1,772 $514 $2,285 $1,024a $748 $1.73 $1,261 $2.23 11.4

Prison low risk 17% 14% $478 $176 $654 $1,024a -$546 $0.47 -$370 $0.64 2.3

Domestic violence perpetrator treatment

Meta-analytic effect size: +.064 

Evaluations in meta analysis: 9 Prison general 47% 50% -$755 -$145 -$901 $1,638c -$2,393 -$0.46 -$2,539 -$0.55 -4.3

Other Work release Meta-analytic effect size: -.080 Evaluations in meta analysis: 7

Prison general 47% 43% $853 $165 $1,017 $170a $683 N/A $847 N/A 5.3

Prison high risk 17% 15% $309 $114 $423 $170a $139 N/A $253 N/A 1.5

Local inmate (jail) programming modalitiesBasic skills Employment: Basic training/ job readiness Meta-analytic effect size: -.074 Evaluations in meta analysis: 16 Jail general 51% 47% $999 $313 $1,312 $490f,g $509 $2.04 $822 $2.68 5.8 Jail under age 25 60% 57% $1,201 $459 $1,660 $490f,g $711 $2.45 $1,170 $3.39 7.9

Chemical dependency

Drug treatment while incarcerated: Residential or IOP

Meta-analytic effect size: -.172 

Evaluations in meta analysis: 6 Jail general 51% 42% $2,202 $688 $2,889 $2,998d -$796 $0.73 -$109 $0.96 13.3

Drug treatment while incarcerated: Standard outpatient

Meta-analytic effect size: -.173 

Evaluations in meta analysis: 8 Jail general 51% 42% $2,240 $696 $2,936 $1,094d $1,146 $2.05 $1,842 $2.68 13.4

Behavioral

Cognitive behavioral intervention

Meta-analytic effect size: -.125 

Evaluations in meta analysis: 38

Jail general 51% 45% $1,648 $515 $2,163 $653f,g $995 $2.52 $1,510 $3.31 9.9

Jail under age 25 60% 54% $2,004 $773 $2,777 $653f.g $1,351 $3.07 $2,124 $4.25 13.2

Domestic violence perpetrator treatment

Meta-analytic effect size: +.064 

Evaluations in meta analysis: 9 Jail general 51% 54% -$938 -$290 -$1,228 $1,638c -$2,576 -$0.57 -$2,866 -$0.75 -5.5

Other Work release Meta-analytic effect size: -.080 Evaluations in meta analysis: 7 Jail general 51% 47% $1,049 $328 $1,378 $170b $879 N/A $1,208 N/A 6.3

1538 The Promise of Evidence-Based Policymaking
11

Population 

receiving 

programming

5-year cumulative 

reconviction rate

Gross monetary benefits

(per program participant Cost of 

programming 

per 

participant

Net monetary benefits (per program participant)

Reduction in 

victimizations 

(per 100 program 

participants)

Taxpayer only Total

Baseline 

recidivism

Recidivism 

with 

programming

Taxpayer 

benefits

Victim 

benefits

Total 

benefts

Benefits 

minus 

costs

Benefit-tocost ratio Benefits minus costs Benefit- to-cost ratio

 State parolee programming modalitiesSupervision

Supervision with risk need and responsivity principles

Meta-analytic effect size: -.239 

Evaluations in meta analysis: 7

Parole general 47% 36% $2,397 $461 $2,858 $696c $875 $1.59 $1,334 $1.89 15.0

Parole high risk 69% 56% $3,335 $975 $4,310 $696c $2,639 $4.79 $3,614 $6.19 22.3

Intensive supervision: Surveillance only

Meta-analytic effect size: +.004 

Evaluations in meta analysis: 14

Parole general 47% 47% -$30 -$6 -$36 $10,449c -$46 $0.98 $388 $1.17 14.1

Parole high risk 69% 69% -$42 -$13 -$55 $10,449c $850 $1.37 $1,767 $1.77 20.7

Intensive supervision: Treatment

Meta-analytic effect size: -.205 

Evaluations in meta analysis: 17

Parole general 47% 37% $2,011 $389 $2,399 $11,405c -$9,394 $0.18 -$9,006 $0.21 8.8

Parole high risk 69% 58% $2,824 $826 $3,649 $11,405c -$8,581 $0.25 -$7,756 $0.32 18.4

Case management: Referral style

Meta-analytic effect size: -.074 

Evaluations in meta analysis: 13

Parole general 47% 43% $742 $142 $884 $1,066a -$324 $0.70 -$182 $0.83 4.6

Parole high risk 69% 65% $1,053 $304 $1,357 $1,066a -$13 $0.99 $291 $1.27 7.0

Parole low risk 17% 15% $277 $102 $379 $1,066a -$789 $0.26 -$687 $0.36 1.4

Case management: Swift and certain

Meta-analytic effect size: -.258 

Evaluations in meta analysis: 6 Parole general 47% 35% $2,521 $485 $3,006 N/Al N/A N/A N/A N/A 16.0

Electronic monitoring Meta-analytic effect size: -.264 Evaluations in meta analysis: 16

Parole general 47% 35% $2,590 $503 $3,093 $464a $2,126 $5.58 $2,629 $6.67 16.2

Parole high risk 69% 55% $3,678 $1,070 $4,748 $464a $3,214 $7.93 $4,284 $10.23 24.3

Basic skills

Employment: Basic training/ job readiness

Meta-analytic effect size: -.074 

Evaluations in meta analysis: 16

Parole general 47% 43% $776 $151 $927 $613a $163 $1.27 $314 $1.51 4.9

Parole high risk 69% 65% $1,025 $297 $1,321 $613a $412 $1.67 $708 $2.15 6.6

Parole low risk 17% 15% $297 $110 $407 $613a -$316 $0.48 -$206 $0.66 1.4

Employment: Transitional wage employment

Meta-analytic effect size: -.139 

Evaluations in meta analysis: 1

Parole general 47% 40% $1,426 $274 $1,700 $6,600a,k -$5,174 $0.22 -$4,900 $0.26 9.0

Parole high risk 69% 61% $1,897 $551 $2,449 $6,600a,k -$4,703 $0.29 -$4,151 $0.37 12.8

Parole low risk 17% 14% $533 $197 $730 $6,600a,k -$6,067 $0.08 -$5,870 $0.11 2.4

Behavioral Cognitive behavioral intervention Meta-analytic effect size: -.125 Evaluations in meta analysis: 38

Parole general 47% 41% $1,299 $253 $1,552 $919c,g $380 $1.41 $633 $1.69 8.1

Parole high risk 69% 62% $1,767 $512 $2,278 $919c,g $848 $1.92 $1,359 $2.48 11.4

Parole low risk 17% 14% $476 $176 $652 $919c,g -$443 $0.52 -$267 $0.71 2.3

Chemical dependency

Drug treatment in community: Residential or IOP

Meta-analytic effect size: -.048 

Evaluations in meta analysis: 5

Parole general 47% 45% $510 $99 $609 $2,738d -$2,228 $0.19 -$2,129 $0.22 3.2

Parole high risk 69% 66% $676 $198 $874 $2,738d -$2,062 $0.25 -$1,864 $0.32 4.5

Drug treatment in community: standard outpatientt

Meta-analytic effect size: -.076 

Evaluations in meta analysis: 4

Parole general 47% 43% $807 $157 $964 $992d -$185 $0.81 -$28 $0.97 5.0

Parole high risk 69% 65% $1,080 $316 $1,396 $992d $88 $1.09 $404 $1.41 6.9

The Promise of Evidence-Based Policymaking 1539
12

Population 

receiving 

programming

5-year cumulative 

reconviction rate

Gross monetary benefits

(per program participant Cost of 

programming 

per 

participant

Net monetary benefits (per program participant)

Reduction in 

victimizations 

(per 100 program 

participants)

Taxpayer only Total

Baseline 

recidivism

Recidivism 

with 

programming

Taxpayer 

benefits

Victim 

benefits

Total 

benefts

Benefits 

minus 

costs

Benefit-tocost ratio Benefits minus costs Benefit- to-cost ratio

 Local probationer programming modalitiesSupervision

Supervision with risk need and responsivity principles

Meta-analytic effect size: -.239 

Evaluations in meta analysis: 7

Probation general 28% 20% $1,809 $615 $2,424 $610c $1,199 $2.97 $1,814 $3.97 10.1

Probation under 

age 25 39% 29% $2,844 $1,081 $3,925 $610c $2,234 $4.66 $3,315 $6.43 17.6

Intensive supervision: surveillance only

Meta-analytic effect size: +.004 

Evaluations in meta analysis: 14

Probation general 28% 28% -$45 -$15 -$60 $9,387c -$9,432 $0.00 -$9,447 $0.01 -0.4

Probation under 

age 25 39% 39% -$58 -$24 -$81 $9,387c -$9,445 -$0.01 -$9,468 $0.01 -0.5

Intensive supervision: Treatment

Meta-analytic effect size: -.205 

Evaluations in meta analysis: 17

Probation general 28% 21% $1,577 $533 $2,110 $10,166c -$8,589 $0.16 -$8,056 $0.21 8.7

Probation under 

age 25 39% 30% $2,482 $944 $3,426 $10,166c -$7,684 $0.24 -$6,740 $0.34 15.1

Case management: Referral style

Meta-analytic effect size: -.074 

Evaluations in meta analysis: 13 Probation general 28% 25% $563 $192 $755 $1,309a,b -$746 $0.43 -$554 $0.58 3.5

Case management: Swift and certain

Meta-analytic effect size: -.258 

Evaluations in meta analysis: 6 Probation general 28% 20% $1,912 $650 $2,562 N/Al N/A N/A N/A N/A 10.5

Electronic monitoring Meta-analytic effect size: -.264 Evaluations in meta analysis: 16

Probation general 28% 19% $1,962 $665 $2,627 $405a,e $1,557 $4.84 $2,222 $6.49 10.9

Probation under 

age 25 39% 28% $3,143 $1,198 $4,341 $405a,e $2,738 $7.76 $3,936 $10.72 19.6

Basic skills Employment: Basic training/ job readiness Meta-analytic effect size: -.074 Evaluations in meta analysis: 16

Probation general 28% 25% $609 $207 $816 $236f,g $373 $2.58 $580 $3.46 3.4

Probation under 

age 25 39% 35% $948 $360 $1,308 $236f,g $712 $4.02 $1,072 $5.54 5.7

Behavioral Cognitive behavioral intervention Meta-analytic effect size: -.125 Evaluations in meta analysis: 38

Probation general 28% 24% $998 $340 $1,339 $553f,g $445 $1.80 $786 $2.42 5.7

Probation under 

age 25 39% 33% $1,577 $601 $2,178 $553f,g $1,024 $2.85 $1,625 $3.94 9.4

Chemical dependency

Drug treatment in community: Residential or IOP

Meta-analytic effect size: -.048 

Evaluations in meta analysis: 5

Probation general 28% 26% $379 $129 $508 $2,750d -$2,371 $0.14 -$2,242 $0.18 2.3

Probation under 

age 25 39% 36% $627 $238 $865 $2,750d -$2,123 $0.23 -$1,885 $0.31 3.8

Drug treatment in community: Standard outpatient

Meta-analytic effect size: -.076 

Evaluations in meta analysis: 4

Probation general 28% 25% $605 $205 $810 $1,004d -$399 $0.60 -$194 $0.81 3.5

Probation under 

age 25 39% 35% $984 $377 $1,360 $1,004d -$20 $0.98 $356 $1.35 6.0

1540 The Promise of Evidence-Based Policymaking
pewtrusts.org/resultsfirst 13

New York State Results First Net Program Impact Table

Explanation of table terms

Class/category of program intervention modality. A description of the type or class of intervention; not all types 

are currently offered in New York.

Meta-analytic evidence base of program evaluations. The standardized mean difference effect size and the 

number of evaluations that were found to be of acceptable rigor and utilized to compute the effect size. Effect 

sizes sourced from Washington State Institute for Public Policy April 2012 and December 2013 meta-analyses. 

See Washington State Institute for Public Policy, “Return on Investment: Evidence-Based Options to Improve 

Statewide Outcomes - April 2012 Update,” Document No. 12-04-1201 (April 2012); and Washington State 

Institute for Public Policy, “Inventory of Evidence-Based and Research-Based Programs for Adult Corrections,” 

Document No. 13-12-1901 (December 2013).

Population receiving programming. The population or subpopulation used to conduct the modeling; all cohorts 

based on felony offender data. 

Baseline recidivism. Cumulative five-year reconviction rate for each population without programming (New Yorkspecific).

Recidivism with programming. Expected cumulative five-year reconviction rate for the population when offered a 

specific program, assuming that the program is delivered competently and with fidelity to its model. 

Taxpayer benefits. Monetary benefits (in 2013 dollars) accrued by government due to reduced criminal justice 

costs from fewer reconvictions over a five-year period.

Victim benefits. Avoided tangible victimization costs such as medical bills and lost wages that are realized 

by society as a whole, not by government. Victim benefits based on the work of McCollister et al. See “The 

cost of crime to society: New crime-specific estimates for policy and program evaluation. Drug and Alcohol 

Dependence,” 108(2010) 98-109.

Notes 

a Based on existing New York state programming delivered in actual setting.

b Based on existing New York state programming delivered in comparable setting.

c Estimated cost based on program components.

d Based on the New York State Office of Alcoholism and Substance Abuse Services treatment reimbursement rates.

e Based on information received from local departments/providers.

f Based on proposed budgets submitted in response to December 2013 requests for proposals for Alternative to Incarceration programs.

g Based on information received from a certified practitioner trainer.

h Net cost taking into account program revenue and value of services provided.

i State cost only, additional 7 percent borne by federal government.

j Differential cost from standard incarceration.

k Based on the U.S. Department of Labor’s 2013 Pay for Success initiative.

l Modality not currently operating in New York state. Cost cannot be estimated at this time. No net benefits calculated.

Sources: New York State Division of Criminal Justice Services, Office of Justice Research and Performance, July 2014

© 2015 The Pew Charitable Trusts

The Promise of Evidence-Based Policymaking 1541
14 pewtrusts.org/resultsfirst

Endnotes 

1 New York State prioritized funds to evidence-based programs in previous grant solicitations, such as a 2009 request for proposals for 

residential stabilization centers, a 2010 RFP for special offender substance abuse programs (http://www.criminaljustice.ny.gov/opca/

pdfs/2011satsorfpfinal3.pdf), and a 2012 RFP for alternatives to incarceration for individuals with families whose income does not exceed 

200 percent of the federal poverty level (http://www.criminaljustice.ny.gov/ofpa/pdfdocs/200percentofpovertyrfp_final_8-7-12.pdf).

2 New York State Division of Criminal Justice Services, Using Data to Inform Evidence-Based Decision Making (January 2013), http://www.

ncja.org/sites/default/files/documents/Terry_Salo_Using_Data_to_Inform.pdf.

3 Updated program totals can be found online at http://www.criminaljustice.ny.gov/opca/ati_description.htm.

4 Pew-MacArthur Results First Initiative panel interview, April 28, 2015.

5 Ibid.

6 Marc Schabses, “Cost Benefit Analysis for Criminal Justice: Deployment and Initial Application of the Results First Cost Benefit Model” 

(October 2013), New York State Division of Criminal Justice Services, http://www.criminaljustice.ny.gov/crimnet/ojsa/resultsfirst/rftechnical_report_cba1_oct2013.pdf.

7 Report, 2014, unpublished.

8 Pew-MacArthur Results First Initiative panel interview, July 2, 2014.

9 According to the Washington State Institute for Public Policy, cognitive-behavior therapy emphasizes individual accountability and 

teaches offenders that cognitive deficits, distortions, and flawed thinking processes can cause criminal behavior (http://www.wsipp.

wa.gov/BenefitCost/Program/438).

10 Pew-MacArthur Results First Initiative panel interview, April 28, 2015.

11 Ibid.

12 New York State Division of Criminal Justice Services, “Request for Proposals: Alternatives to Incarceration” (2013), http://www.

criminaljustice.ny.gov/ofpa/pdfdocs/ATI-RFP-Due-Sep-16-2013.pdf.

13 New York State Division of Criminal Justice Services, “Alternatives-to-Incarceration (ATI) and Employment Programs: Application for 

Funding” (2013), http://www.criminaljustice.ny.gov/ofpa/downloadforms/ATI-2013-14-Application-June.17.2013.doc.

14 For more information on these tools, see Council of State Governments, “Risk Assessment Instruments Validated and Implemented 

in Correctional Settings in the United States” (2013), http://csgjusticecenter.org/reentry/publications/risk-assessment-instrumentsvalidated-and-implemented-in-correctional-settings-in-the-united-states/.

15 New York State Division of Criminal Justice Services, “Alternatives-to-Incarceration and Employment Programs: Application for Funding” 

(2013).

Total gross benefits. Total benefits to taxpayers and society that result from one unit of program participation. 

Cost of programming (per participant). Estimated cost of delivering the intervention to a single offender in New 

York state. 

Taxpayer net benefits. The net benefit from a governmental or budgeting perspective.

Taxpayer benefit-to-cost ratio. Ratio displaying the amount of governmental return (savings) for each dollar 

spent on programming.

Total benefits. The net benefit to taxpayers and to society.

Total benefit-to-cost ratio. Ratio displaying the total benefit returned to taxpayers and to society for each dollar 

spent on programming.

Reduction in victimizations. Estimated number of victimizations avoided (via reduced recidivism) when 

intervention is provided to 100 participants.

1542 The Promise of Evidence-Based Policymaking
pewtrusts.org/resultsfirst 15

16 See New York State, “Investing in What Works: ‘Pay for Success’ in New York State Increasing Employment and Improving Public Safety” 

(2014), https://www.budget.ny.gov/contract/ICPFS/PFSFactSheet_0314.pdf.

17 Massachusetts also received funding through this federal grant. For more information, see http://www.dol.gov/opa/media/press/eta/

ETA20131936.htm.

18 The Pew Charitable Trusts, “Iowa’s Cutting-Edge Approach to Corrections: A Progress Report on Putting Results First to Use” (December 

2013), http://www.pewtrusts.org/~/media/legacy/uploadedfiles/pcs_assets/2013/RFIBriefResultsFirstIowaProgressReportFINALpdf.pdf.

19 Pew-MacArthur Results First Initiative panel interview, April 28, 2015.

20 Ibid.

21 Cost-Benefit Knowledge Bank for Criminal Justice, “An Interview With Marc Schabses of the New York State Division of Criminal Justice 

Services” (July 25, 2013), http://cbkb.org/2013/07/an-interview-with-marc-schabses-of-the-new-york-state-division-of-criminal-justiceservices/.

22 Pew-MacArthur Results First Initiative panel interview, April 28, 2015.

23 Ibid.

24 Ibid.

25 Ibid.

26 Ibid.

27 Ibid.

28 Ibid.

29 Ibid.

30 Ibid.

31 Ibid.

The Promise of Evidence-Based Policymaking 1543
16 pewtrusts.org/resultsfirst

The Pew Charitable Trusts

901 E St. NW, Washington, DC 20004

pewtrusts.org

The Pew Charitable Trusts is driven by the power of 

knowledge to solve today’s most challenging problems. 

Pew applies a rigorous, analytical approach to improve 

public policy, inform the public, and invigorate civic life.

The John D. and Catherine T. 

MacArthur Foundation

140 S. Dearborn St. Chicago, IL 60603

macfound.org

The John D. and Catherine T. MacArthur Foundation 

supports creative people and effective institutions 

committed to building a more just, verdant, and 

peaceful world. In addition to selecting the MacArthur 

Fellows, the Foundation works to defend human rights, 

advance global conservation and security, make cities 

better places, and understand how technology is 

affecting children and society.

Contact: Gary VanLandingham, director, Pew-MacArthur Results First Initiative 

Email: gvanlandingham@pewtrusts.org Phone: 202-540-6207 

Pew-MacArthur Results First Initiative, a project of The Pew Charitable Trusts and the John D. and Catherine T. MacArthur Foundation, 

works with states to implement an innovative cost-benefit analysis approach that helps them invest in policies and programs that are 

proved to work.

1544 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t18-dmlw

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0137

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

Address:

Huffman, TX, 77336

Email: aktblackburn@gmail.com

General Comment

I am opposed to any sort of data base tracking in education intended to "follow the child"

anywhere outside of the student's enrolled school district (I am not referring to statistical

educational demographics.), or that would be used for anything other than that which is

necessary for national security. I believe our Constitutional rights to privacy are being

violated with such measures. 

1)It was against our Constitutional protections that a national database be constructed.

That measure would fall to the States. So what did our government do under the

direction of an Obama administration, they bribed our states with money for each state to

create its own database identical to every other state created databas. The purpose of

creating identical databases was for the data sharing ability. And so they have created a

National database right under the noses of Americans while saying it was "state led."

(Now where have we heard that before?? A: Common Core) Whether the intent to create

this nationalized database was criminal or not is not for me to decide; however, I believe

it is incredibly and unethically dishonest. 

The Promise of Evidence-Based Policymaking 1545
2)Additionally, our children's private information should NEVER have been

compromised through the gutting of FERPA or HIPAA by President Obama. These

measures were put in place to protect our children AND our families which he signed

away with wave of his pen. We can clearly see why. Someone is making money hand

over fist off our data, and it does not have the best interest of American citizens (students

or parents) in mind! 

3)Collecting and sharing personal data/information on our children (though not

exclusively to our children) alway runs the risk of profiling them, which will result in

great prejudices and injustices when it makes its way into the wrong hands; which it

surely will. The intent, I understand, is to collect data from early education on into the

workforce. So their employers are to have access to this information? My research tells

me Yes, they are. This should NEVER be allowed. This an full violation of privacy. 

Despite "security measures," our children's private information will be compromised, as

we have already had security breeches in a local Texas school district, Katy ISD. It took

two years for the data company to divulge the breech, and it was also disclosed that an

additional breech had occurred. The breech was said to have been internal and

accidental; but we can clearly see you cannot implement measures to protect us; and we

should not trust our private information to you. With the recent hacking of the highest

ranking government officials and political parties, as well as the violations of privacy

that were exposed in the NSA monitoring emails and phone conversations, can we even

allow ourselves to be fooled? The release of this information is child's play compared to

what the government is collecting on our students and on our families. 

This is clear and convincing evidence that we CANNOT be assured of protections for

ours or our children's data. 

4)Student portfolios should only be for purposes of education to follow the child within

the district which the student attends. Student portfolios should ALWAYS remain

private, under the authority and the property of parent/child, belonging to the school

district only so long as the student is enrolled in their district, and should never be

released publicly or privately without parent or adult student approval, barring a legal

criminal investigation. Our children's data in education is currently being targeted by

tech ed companies for massive profit, as is that of private individuals. It is not for fair

and reasonable profit, but for unconscionable gain and greed! Our children's data has

been coined by tech ed companies and Next Generation assessment commissions as

"gold," "the new gold," "a treasure trove," "human capital," etc. And we should not be

foolish enough to think the intention to profit off of all our citizens is not part of this

push for data collection. 

Sincerely,

Angela Blackburn

1546 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t18-vfkj

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0138

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

The new Commission on Evidence-based Policymaking being pushed by Speaker Paul

Ryan and Senator Patty Murray , and urged (by Bill Gates and the other

foundations/corporations that want access to more and more of our children's data) to

establish a national student-unit record system that would allow government tracking of

citizens from Pre- school throughout their careers is absolutely unconstitutional and must

be stopped dead in it tracks. American citizens have not given the government the power

to monitore our every waking moment. This system serves only the oligarchs and global

elites and is against how our government is designed to work.... for, of and by the

people.

Trump must choose a Secretary of Education prepared to dismantle the

unconstitutional/illegal DOE and soon be out of a job, while removing the strings

attached to education through the Department of Labor and the Department of health and

human services.

The Promise of Evidence-Based Policymaking 1547
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t18-hynv

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0139

Comment on FR Doc # 2016-22002

Submitter Information

Name: Alice Linahan

General Comment

Parents all across the nation understand, The Workforce Innovation and Opportunity Act

(WIOA) and the Every Student Succeeds Act (ESSA) had to get passed before the

election. This is what locked the GLOBAL transformation of education "SYSTEM" in

place, no matter if it is implemented at the local, state or federal level. With the goal of

tracking and collection of data for behavior modifying adaptive assessments for

workforce development. Which will be coming in with the Next Generation of

Assessments and Accountability. 

One major concern....

When the state and federal government are tracking students for workforce development

aligned to the United Nations model, this brings in the issue of national security. 

While parents across the country are becoming more and more concerned with the issue

of the Federal Government monitoring, tracking and collecting data on our children, it is

very alarming to learn that the Brookings Institute also has a Center for Universal

Education that partners with UNESCO for the UN education agenda where there is a

focus on "Global Tracking" and the need to support development of more robust systems

for assessing learning outcomes. 

1548 The Promise of Evidence-Based Policymaking
The establishment of a National Student-unit record system should remain illegal as it is

clearly unconstitutional. 

The Promise of Evidence-Based Policymaking 1549
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t19-kwaa

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0140

Comment on FR Doc # 2016-22002

Submitter Information

Name: Cynthia Forland

Address:

Lacey, WA, 98503

Email: cforland@esd.wa.gov

General Comment

See attached file(s)

Attachments

WA Forland comment re CEBP

1550 The Promise of Evidence-Based Policymaking
Page 1 of 4

Comments RE: Federal Register # 2016-22002

Cynthia Forland, Director, Labor Market and Performance Analysis

Washington State Employment Security Department

Member, Department of Labor Workforce Information Advisory Committee

November 14, 2016

COMMISSION QUESTION #1: Are there successful frameworks, policies, practices, and methods

to overcome challenges related to evidence-building from state, local, and/or international governments the Commission should consider when developing findings and recommendations regarding Federal evidence-based policymaking? If so, please describe.

Resource limitations related to data access, funding, IT, and/or staff capacity are typical challenges. In

Washington state, we have made great advances in developing an evidence-building infrastructure, with

corresponding frameworks, policies, practices and methods. We have not only developed longitudinal

administrative data sets, but also the infrastructure to convert and analyze the data to inform policy and

customer choices.

Other examples of successful approaches to developing data infrastructure include the numerous other

state longitudinal administrative data sets funded, in part, under the Workforce Data Quality Initiative

(WDQI) and State Longitudinal Data Systems (SLDS) grants. A good cross-state effort is the Western

Interstate Commission for Higher Education’s (WICHE) multi-state longitudinal data exchange (MLDE).

This latter effort is important for showcasing the value of tracking individuals across state lines to understand the impact of our education, training and other investments.

COMMISSION QUESTION #3: Based on identified best practices and existing examples, how

should existing government data infrastructure be modified to best facilitate use of, and access

to, administrative and survey data?

Streamline federal data initiatives, such as SLDS and WDQI, which may mean combining some initiatives

that have overlapping goals. Existing federal funding methods for these initiatives are unpredictable, uncoordinated, and tend to promote overlapping and uncoordinated projects. If the 21st century is about lifelong learning with people moving back and forth between education and employment, or participating in

employment and education simultaneously, we need to develop pre-K to career longitudinal systems, not

separate education and workforce program systems.

Invest in, and continuously improve, labor market information so that high-quality data relevant for the

current economy is available to guide policymakers and help customers make career, education and

training decisions. Episodic, competitive grants cannot provide the solid foundation needed for this key

piece of data infrastructure. For example, the last survey regarding the contingent workforce was conducted by the Bureau of Labor Statistics in February 2005 and estimates of the size of this population

today vary widely as a result.

COMMISSION QUESTION #4: What data sharing infrastructure should be used to facilitate data

merging, linking, and access for research, evaluation and analysis purposes?

The Promise of Evidence-Based Policymaking 1551
Page 2 of 4

A federated exchange system that does not involve developing and maintaining a single national data

repository.

COMMISSION QUESTION #5: What challenges currently exist linking state and local data to federal data? Are there successful instances where these challenges have been addressed?

Federal and state laws and regulations governing critical sources of data are a recurring challenge. Examples include the Family Educational Rights and Privacy Act (FERPA), IRS regulations and variable

state Unemployment Compensation laws.

COMMISSION QUESTION #6: Should a single or multiple clearinghouse(s) for administrative and

survey data be established to improve evidence-based policymaking?

A single federal clearinghouse is not an efficient or practical approach. A more federated system with the

ability to exchange information amongst states and federal agencies would be far more sustainable.

COMMISSION QUESTION #7: What data should be included in a potential U.S. government data

clearinghouse(s)? What are the current legal or administrative barriers to including such data in a

clearinghouse or linking the data?

Key data sources include administrative data on K-12 education, post-secondary education (including higher education, apprenticeships), traditional employment (e.g., Unemployment Compensation), alternative employment (e.g., IRS), and other data to assist with matching (e.g., state

driver and vehicle licensing records).

COMMISSION QUESTION #8: What factors or strategies should the Commission consider for how

a clearinghouse(s) should be self-funded? What successful examples exist for self-financing related to similar purposes?

The data and human infrastructure needed to build and maintain an evidence base for federal and state

programs are public goods. We question the premise that the ordered exchange of that data should be

self-funded, except that outside researchers and institutions should cover the marginal costs of their data

acquisitions. Government entities have a role and responsibility to maintain and be good stewards of program administrative data, and should receive adequate funding and support for the integrity of those systems and legal exchanges amongst them.

COMMISSION QUESTION #9: What specific administrative or legal barriers currently exist for accessing survey and administrative data?

Programs have been developed independently and in a siloed fashion, as have the legal requirements

around them, so there are numerous legal barriers. Also creating barriers are the many levels of government involved in program administration and the varied policies and practices associated with these programs.

COMMISSION QUESTION #10: How should the Commission define “qualified researchers and institutions?” To what extent should administrative and survey data held by government agencies

be made available to “qualified researchers and institutions?”

1552 The Promise of Evidence-Based Policymaking
Page 3 of 4

Just to be clear, states should be considered partners to any clearinghouses, not qualified researchers

and institutions. As governmental entities, state workforce agencies should have access to governmental

data needed for federally required performance reports and to conduct research and evaluations. Governmental entities, especially WIOA partner agencies at the federal and state level, should have priority

access to available data. The Commission could learn from state workforce agencies that provide data,

selectively, to outside researchers and institutions. Given resource constraints, such uses could be limited

to specific project requests that support the research agendas of federal and state agencies, as a first

priority.

QUESTION #11: How might integration of administrative and survey data in a clearinghouse affect

the risk of unintentional or unauthorized access or release of personally identifiable information,

confidential business information, or other identifiable records? How can identifiable information

be best protected to insure the privacy and confidentiality of individual or business data in a

clearinghouse?

Facing high demand for government data with constrained resources, many entities are currently approaching privacy and confidentiality in an ad hoc way that involves more risk than having an organized

approach with clear processes, safeguards and rules. And a carefully developed exchange system would

prevent the risks associated with a single federal data repository.

QUESTION #13: What technological solutions from government or the private sector are relevant

for facilitating data sharing and management?

See response to Question #1 above.

COMMISSION QUESTION #14: What incentives may best facilitate interagency sharing of information to improve programmatic effectiveness and enhance data accuracy and comprehensiveness?

• Create efficient and streamlined data collection, confidentiality, and data access processes,

with excellent privacy safeguards.

• Ensure that state agencies that collect, safeguard, clean, and share data are able to benefit

from the data soon after it becomes available.

• Provide state workforce agencies with adequate funding, training, and technical assistance as

they fulfill their responsibilities for collecting, safeguarding and sharing information.

• Provide state workforce agencies (and their state and federal partners) the critical funding and

other support needed to translate data into information that is useful to customers – including

policymakers, program managers, job seekers and employers. Otherwise, only the private sector will have the capacity to build an evidence base and develop customer tools and information, which is problematic in cases where it is more efficient to develop in-house and/or

cross-state solutions. 

• Ensure that evidence is used to inform policy. For example, despite 25 years of evidence by

federal and state partners that job search assistance and UI claimant reemployment services

are high return-on-investment strategies, funding for the major programs supporting these

strategies has declined significantly over the last 20 years.

COMMISSION QUESTION #15: What barriers currently exist for using survey and administrative

data to support program management and/or evaluation activities?

The Promise of Evidence-Based Policymaking 1553
Page 4 of 4

Limitations exist on the use of specific data sources by states. For example, some federal data sources

can be used for enforcement but not for research purposes. This includes IRS data. Given the flexibility

that is being asked of states to share items like UI wage records for a broader scope of evaluation (outside of just UI program administration), the same expansion should be asked of federal agencies.

COMMISSION QUESTION #16: How can data, statistics, results of research, and findings from

evaluation, be best used to improve policies and programs?

Based on recent history, federal, state, local, and even private sector money available for public interest

services have not, and likely will not, keep up with the expanding need. That means that we can keep

doing the same things but doing them for fewer people, or we can keep serving similar or expanding

numbers of people but simply do less for them, or we can analyze our data and learn how to be more effective. Ideally this would not simply mean looking at data to see which services work and which

don’t. Instead, we should be able to use data to better understand which services work best and for

whom. It is likely that most services have some effectiveness but that effectiveness varies by population

served and other factors, which calls for a customized, evidenced-based approach.

COMMISSION QUESTION #17: To what extent can or should program and policy evaluation be addressed in program designs?

Government should evaluate program and service effectiveness and use findings to improve effectiveness and efficiency. That obligation is to both taxpayers in general and those our services are intended to

benefit.

COMMISSION QUESTION #18: How can or should program evaluation be incorporated into program designs? What specific examples demonstrate where evaluation has been successfully incorporated in program designs?

Unemployment Insurance program “worker profiling models,” if timely and regularly updated, are an example of an approach that aids state evaluation of UI programs. These predictive models incorporate

changes in the economy that impact who is likely to exhaust benefits. They enable state workforce agencies to directly engage those claimants most likely to have long unemployment durations in effective

reemployment strategies early in their unemployment insurance spells.

COMMISSION QUESTION #19: To what extent should evaluations specifically with either experimental (sometimes referred to as “randomized control trials”) or quasi-experimental designs be

institutionalized in programs? What specific examples demonstrate where such institutionalization has been successful and what best practices exist for doing so?

The Commission should explicitly recognize there is a place for both experimental and quasi-experimental

design in evaluation work. While experimental design may be the “gold standard” of evaluation, it is extremely costly and also not the most appropriate research design in many cases. Also, quasiexperimental design is more easily accepted by those who deliver services. State workforce agencies

have used the results of quasi-experimental research to inform policy and practice, and should have the

flexibility to use research designs appropriate to their needs.

1554 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t19-3ebm

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0141

Comment on FR Doc # 2016-22002

Submitter Information

Name: Charles and Debbie Baxter

Address: United States,

General Comment

NO to collecting any information on my children, in school or anywhere else. This is a

direct intervention of our individuals rights to privacy and to raising our children.

Anything proposed along these lines is UnConstitutional .. both the Federal Constitution

and our State Constitution. If Paul Ryan and Patti Murphy want to live in a collectivist

society, then perhaps they should have Bill Gates recommend another country where

they don't honor a Bill of Rights. Just NO.

The Promise of Evidence-Based Policymaking 1555
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t19-r889

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0142

Comment on FR Doc # 2016-22002

Submitter Information

Name: Brian Halladay

General Comment

To Whom It May Concern,

In response to your request for comments for the Commission on Evidence-Based

Policymaking. Under the Constitution, government has no right to create policies

regarding education. Optimally, under the 10th Amendment, States would have enabled

to develop policies and frameworks specific to the needs of their population. 

As it is, a bunch of government entities, researchers, evaluators and contractors (aka "the

American Institutes for Research") (which is also integrated with the Bureau of Census)

will do whatever the hell they want, and be awarded whatever grant they apply for all

under the guise of "developing policy" just as they have for the past 55 years. 

Now, I realize that without a radical change in education policy this won't be, so I'll

address the question just to be on record as opposing this effort.

The ONLY factor to consider to ensure the security and privacy of admin and survey

data is to STOP gathering surveys and data. Ask yourselves, who does this benefit and

why do they need this data. How can we expect data to be private when so many

different technologies exist that collect data at school? What's to prevent his data being

1556 The Promise of Evidence-Based Policymaking
stored in a meta-databank? And why is this allowed? Do you really care about the

privacy of our students? I would contend no, you do not.

There should be no data-sharing infrastructure as it is too easy to link, merge and access

the different datasets.

With the increasing propensity of organizations (including the fed government) to have

data breaches the creation of a clearinghouse (which let's not fool ourselves, already

exists) increases the probability of misuse and risk of cybersecurity breaches.

Children should have a right to privacy, including privacy from their own federal

government. Not only does a clearinghouse provide untold uses to business and related

parties (especially behavioral data) , but how are you going to restrict access to such a

clearinghouse and manage the data once it's been compromised? How will you ensure

my children's data isn't compromised? It simply won't be done.

One of the most over-reaching surveys ever existed was Project Talent. The results of

this program served as the foundation of educational policy and programs for decades

afterwards, and still exists today. More harm has come from this than good. The federal

government should stay out of the data collection business, and allow education to be

controlled by the states instead of federal lobbyists.

The Promise of Evidence-Based Policymaking 1557
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t19-b2tg

Comments Due: December 14, 2016

Submission Type: API

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0143

Comment on FR Doc # 2016-22002

Submitter Information

Name: Kris Perry

Address:

Washington, DC, 20005

Email: frabin@ffyf.org

General Comment

The First Five Years Fund thanks the Commission on Evidence-Based Policymaking for

the opportunity to submit comments at this time. Please see the attached file

[FFYF_Comments_Evidence-basedPolicymaking Commission.docx] for our full

comments to the Commission. 

Attachments

FFYF_Comments_Evidence-based Policymaking Commission (2)

1558 The Promise of Evidence-Based Policymaking
Shelly Martinez 

Executive Director 

Commission on Evidence-Based Policymaking 

Washington, D.C. 

Re: Commission on Evidence-Based Policymaking Comments, Docket ID USBC-2016-0003-0001

Dear Ms. Martinez: 

The First Five Years Fund (FFYF) strongly supports the Evidence-Based Policymaking 

Commission’s (“Commission”) mission to identify effective strategies for producing and 

using evidence to support federal programs and policies. FFYF supports the core early 

learning and care programs carried out across the U.S. Department of Health and Human 

Services, the U.S. Department of Education, and the U.S. Department of Health Resources 

and Service Administration, and is committed to a well-funded, high-quality continuum of 

affordable early learning and care. Given our work, FFFY urges Commission Members to 

carefully consider the following recommendations when evaluating and identifying future 

Commission activities, including possibly developing findings and recommendations for 

Congress. 

Data Use in Program Design, Management, Research, Evaluation, and Analysis 

FFYF shares the Commission’s interest in identifying strategies for better using data, 

statistics, research, and findings from evaluation to improve policies and programs. Strong 

government programs adopt these strategies as part of a culture of continuous 

improvement. Effective programs benefit from embedded evaluations, as referenced in 

the Commission’s Request for Comments. FFYF encourages the Commission to examine 

ways to better and more routinely embed evaluation as a core element of federal 

programs. For example, Congress recently provided crosscutting program evaluation 

authority and funding to the Secretary of Education in the new Every Student Succeeds 

Act (ESSA, P.L. 114-95). As you examine such models, however, we also encourage you to 

ensure your work does not lead to an inadvertent over-investment in programs with 

previously demonstrated effectiveness to the detriment of innovative efforts to develop 

and test new ideas. Policymakers must continue to invest in innovation, while also 

supporting efforts to scale programs supported by greater levels of evidence. 

Furthermore, we recommend an approach to the evaluation of early learning and care 

program models that takes into consideration the varied factors in children and families’ 

lives outside the scope of program service delivery that impact child outcomes, and more 

specifically, that program evaluation is intentionally tied to a theory of change that 

identifies intended outcomes and the underlying capacities or mechanisms on which 

those outcomes rest. Poorly designed studies that only focus on third grade assessment 

The Promise of Evidence-Based Policymaking 1559
outcomes – using tests designed for State K12 accountability systems – do not satisfy this 

higher standard. 

Striking a thoughtful balance between stimulating promising innovations and investment 

in practices with a greater evidence base is particularly important to strengthening early 

learning systems. Recent studies, including work by Nobel Prize-winning economist 

Professor James Heckman and his colleagues at the Center for the Economics of Human 

Development, demonstrate that early learning has moved beyond rudimentary inquiries 

to more sophisticated challenges related to achieving greater scale and quality. Professor 

Heckman’s work, and other well-designed studies, demonstrates that the most effective 

early learning programs provide high quality, developmentally appropriate, 

comprehensive services for children from low-income families from birth through age 5. 

As we enter into the era of greater implementation based on this existing research base, 

we also need continued experimentation, research and working experience that informs 

practice, fosters innovation and provides elected officials and the public with even more 

ĐŽŶĨŝĚĞŶĐĞƚŽŝŶǀĞƐƚŝŶƐLJƐƚĞŵƐĂŶĚƉƌŽŐƌĂŵƐƚŚĂƚǁŽƌŬറ—റĂŶĚĐŽŶƚŝŶƵŽƵƐůLJŝŵƉƌŽǀĞ͘ Dr. 

Jack Shonkoff, M.D. at Harvard University’s Center on the Developing Child recently 

released a report, ‘From Best Practices to Breakthrough Impacts’, which highlights that 

while there exists a number of studies on the effectiveness of various early learning 

programs and interventions demonstrating the difference early childhood programs do 

indeed make, there is limited data available that could be used to inform replication and 

scalability that would result in improved outcomes at a population level. Furthermore, in 

evaluating program effectiveness, it is imperative that we graduate from using a broad 

brush in asking whether or not a program ‘works’, and begin asking ‘which features work 

for whom and why?’ This frame of program evaluation positions the field to advance 

replication of the elements that do lead to improved outcomes without being tethered to 

less effective elements of earlier intervention iterations. Dr. Shonkoff also recommends 

that in addition to conducting randomized control trials, which take several years to 

complete with no opportunities for mid-course corrections, program evaluation should 

include micro-trials that are small-scale, and short-duration field tests designed to 

catalyze rapid, shared learning across multiple projects in the field simultaneously. 

 Dr. Shonkoff and his colleagues summarized the particular challenge in the early learning 

field well when they wrote: 

 

“On the one hand, many leaders in the field are engaged in critically important 

efforts to improve the quality of programs, increase the effectiveness and 

efficiency of service delivery systems, enhance the skills and compensation of a 

highly diverse early childhood workforce, and encourage innovation. These 

efforts are happening at multiple levels across a variety of sectors—and they 

must be sustained. On the other hand, most decision makers urge funding 

1560 The Promise of Evidence-Based Policymaking
solely for programs with previously demonstrated effectiveness, regardless of 

the nature or magnitude of their impacts. This widespread preference for 

“evidence-based” programs, many of which have produced small effects on 

random categories of outcomes that have not been replicated, seriously limits 

the likelihood of achieving increasingly larger impacts at scale over time. 

Indeed, many of the most compelling challenges facing the early childhood 

field today are linked to the absence of sufficient professional and political 

incentives for developing and testing new ideas”1

As a result, FFYF encourages the Commission to explore evidence-based structures that 

support and encourage federal programs and investments across a continuum from 

promising innovations to investments with a more robust research base. Some existing 

programs, such as the U.S. Department of Education’s legacy Investing in Innovation Fund 

(i3), and ESSA’s new Education Innovation and Research Grants, specifically utilize a tiered 

approach to investment, based on an initiative’s evidence base. Exploring the efficacy of 

such evidence based models, while also establishing other evidence and evaluation 

structures that identify and document best practices for sharing, could help to 

fundamentally improve federal policy and practice. 

Thank you for providing this opportunity to help inform the Commission’s deliberations 

and work. We would be glad to answer any questions you may have about these ideas 

and FFYF’s work and would be pleased to participate in future Commission events. 

Sincerely, 

Kris Perry 

Executive Director 

First Five Years Fund 

1010 Vermont Avenue, NW, Suite 810 

Washington, D.C. 20005 

(202) 730-0941 

 1 Center on the Developing Child at Harvard University (2016). From Best Practices to Breakthrough Impacts: A Science-Based 

Approach to Building a More Promising Future for Young Children and Families. http://www.developingchild.harvard.edu

The Promise of Evidence-Based Policymaking 1561
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t19-qlac

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0144

Comment on FR Doc # 2016-22002

Submitter Information

Name: Philip Held

Address:

Walnut Creek, CA, 94597

Email: esrd00@gmail.com

General Comment

See attached file(s)

Attachments

PJH Platinum CEP 11 14 16docx

PJH Platinum PDF CEP 11 14 16docx

1562 The Promise of Evidence-Based Policymaking
November 14, 2016

Comments Submitted to the

“Evidence-Based Policymaking Commission” (CEP)

(Created by the ‘‘Evidence-Based Policymaking Commission Act of 2016’, Public Law 114–140

114th Congress). 

Submitted by: Randall Webb, MA, 

Philip J. Held, PhD,

Table of Contents

Page

Executive Summary 2

Statement of the Problem 3

Introduction 3

Background 3

NIDDK/NIH USRDS Creates a System to Share Data 4

Selected Statistics for the USRDS Database 7

Selected Portions of the USRDS Data Documentation 8

(Readily Available to the Research Community)

Data Use Agreement 9

The Promise of Evidence-Based Policymaking 1563
2

Executive Summary

Medical care in the US is in general, of exceptionally high quality although finding the 

resources to provide it is a never-ending public issue in economic, moral, and political terms. 

Data, not surprising, are a central part of the health care sector and are fundamental to:

• Experiments to test and improve new clinical approaches, drugs, procedures and 

policies

• Surveillance of health and disease tracking and providing quality assessment and 

guidance for population health improvement

• Guiding and advising patients as well as protecting confidentiality

• Compensating and regulating the millions of suppliers

• Maintaining trust and confidence among all sectors

This Commission on Evidence Based Policy Making is focused on the use, control, and access 

to data - presumably specifically defined as public data paid for by the US taxpayer that are of 

potential use in making policy decisions based on empirical information. The public concerns 

which generated this commission must have been focused on access to tax payer funded data 

as well as protection of confidentiality.

This comment for the CEP reports on the United States Renal Data System (USRDS) a very 

productive and vital system that generates, controls, and widely distributes an endless volume 

of data that is part of the federal system that funds, monitors, and controls, the care of nearly 

700,000 Americans with end stage renal disease (ESRD) – requiring dialysis therapy or kidney 

transplantation, as well as the millions of Americans with the preceding stages of chronic 

kidney disease (CKD), not at ESRD. The two primary institutions that run this program are the 

Centers for Medicaid and Medicare (CMS) and the National Institute of Diabetes, Digestive 

and Kidney Diseases (NIDDK) which is part of the National Institutes of Health (NIH). Both 

these institutions are part of the US Department of Health and Human Services.

The data system incrementally developed by the USRDS has been successfully made 

available for use by researchers – sharing, distributing standard analysis files (SAFs) and

protecting sensitive data, for over 25 years – without serious mishap. Incredible volumes of

detailed and extensive information, often of a confidential nature, are made available to 

researchers and policy makers both in and out of the government including profit and nonprofit institutions. This data distribution system has been central and vital to public policy, the 

setting of medical guidelines, and encouragement and nurturing of a research arena second to 

none in the world.

The system that provides these data at the heart of the treatment of kidney failure can provide 

a good model to the CEP.

1564 The Promise of Evidence-Based Policymaking
3

Statement of the Problem

Kidney disease is the 9th leading cause of death and a significant contributor to cardiovascular 

and diabetic deaths – it has been called both a ‘disease multiplier’ and a ‘cost multiplier’ and is 

a non-communicable disease that is more prevalent than diabetes mellitus in the US general 

population. There are about 500,000 patients on chronic dialysis therapy and nearly 100,000 

patients on the kidney transplant waitlist, which continues to grow. Medicare spends $32.8 

billion per year (USRDS 2016 Annual Data Report) on the treatment of patients with End Stage 

Renal Disease (ESRD) – 7.2% of total Medicare costs for ~1% of the population served by 

Medicare. Combining all sources of payment (Medicare, Medicaid, private insurance and

patient obligations), ESRD patients have costs of $125,000 per patient per year and a high 

mortality rate of approximately 15-20% per year. The National Institute of Digestive, Diabetes 

and Urologic Diseases (NIDDK) typical budget amounts to nearly $2 billion per year. 

Introduction

It is our understanding that the CEP is interested in developing policy recommendations to aid 

government policymaking. In developing policy one can either make decisions on ideological 

basis or on a scientific basis. If the scientific basis is chosen for such decision-making the 

following are minimum requirements:

• Relevant and sufficient data,

• Appropriate analytical methods and,

• Processes open to validation and reproducibility

Review of the CEPs Law’s mandate makes it clear that the CEP is to develop policies based 

on a scientific process with an emphasis on the issue of data which involves:

1. Determining which policy questions are to be addressed 

2. Identification of relevant data to address these questions

3. Identification of persons and institutions appropriate to have access to these data

4. Determining methods of accessing such data including the allocation of costs of making 

data available

5. When appropriate, determine procedures to protect confidentiality of sources, 

particularly patient-level identifiable information

6. Determining what are appropriate versus inappropriate uses of these data

Background 

The United States Renal Data System (USRDS) is run jointly by the National Institute of 

Digestive, Diabetes and Urologic Diseases (NIDDK) and the Centers for Medicare and

Medicaid Services (CMS). The USRDS was formed in response to a Congressional Mandate

requiring the creation of “a National Renal [Kidney] Registry” for the US and has been 

operated, under contract, for over 25 years and remains a powerful and successful example of 

how to fulfill the 6 requirements listed above. 

The Promise of Evidence-Based Policymaking 1565
4

Prior to the formation of the USRDS, almost all medical data funded in total or in part by the 

Federal Government were kept usually under the control of a contracting or grant receiving 

agency or institution that collected or assembled the data. The same could be said of 

numerous federal studies wherein the data were collected at great taxpayer expense, but 

typically the data were kept in vault-like circumstances where only a few selected internal 

investigators ever had access to these data.

NIDDK/NIH funded USRDS Creates a System to Share Data Widely (1988)

When NIDDK issued an RFP for the creation of the USRDS, the winning bidder’s proposal 

(Philip Held PI, and Randall Webb, Chief of Information Systems) contained a sophisticated 

and timely set of proposals to ensure that data and analyses performed in this project would 

be:

1. Transparent and built on widely available data handling systems. (Past 

experience in working with medical data stored on varied university systems was 

often based on locally developed systems that were dated, poorly documented 

and demanding of many technician hours to accomplish even minimal goals).

2. Transferable to other data systems both in and out of government. Oracle, a 

relatively new operation at that time, and SAS were the foundations of all project 

work although other relational databases did evolve over time.

3. Inclusive of all files including cleaned analytical files, designed to save users 

many person-years of effort, by circumventing the need for tedious data 

collection and processing efforts before real analysis could begin would be made 

available.

4. Files designed to be consistently defined over the years to allow longitudinal 

analyses.

5. Development of a Researchers Guide to guide researchers in the use of the 

data.

6. Incorporation of new data sources, notably from the CMS CROWNWeb and Part 

D data systems, into the database as they became available.

7. Open to requests for other formulations of the data.

8. Open to merging existing data with other data sources if privacy requirements 

can be met.

9. Pricing these files at cost of reproduction.

10.Determining what were appropriate and inappropriate uses of the data. Research 

in the public interest was the general rule. Research designed to discriminate 

against patients for commercial reasons was not appropriate research.

11.Involved of a Scientific Advisory committee to supplement and guide the 

contractor’s staff in insuring the purposes for which the data were released were 

intended for valid scientific purposes and in the public interest.

12.Inclusive of developing methods and a Data Use Agreement to protect 

confidentially of these medical data, which incorporated very sensitive 

information.

With the initiation of the USRDS, data were from two primary sources: CMS’s Medicare data 

and primary data designed by the USRDS but collected by the ESRD Networks, a series of 

institutions contracted by CMS to assist in the management of the ESRD program. Dialysis 

1566 The Promise of Evidence-Based Policymaking
5

and transplant units were also tapped into to collect relevant data.

It is not an over statement to note that these handling and distribution procedures by the 

USRDS contractor were quite radical at the time but were ultimately immensely popular with all 

sorts of “stakeholders” including the research community, the media, the NIDDK, CMS, and

the providers (both profit and not-for-profit).

NIDDK was pleased and proud of this new data access system:

1. NIDDK could help their grantees tackle a wider list of topics in a timely and costefficient manner.

2. NIDDK could gain real empirical data to assist in the design of clinical trials by 

use of these observational data.

3. The USRDS Annual Data Reports (ADRs) have been produced annually since 

1989. The ADRs have become the foundational national reference on the 

epidemiology of ESRD and CKD.

4. Because the same files are used for the ADR and by outside researchers, the 

researcher’s results can be benchmarked against the ADR.

5. They could carry to Capitol Hill results from a ‘real world’ clinical, empirical data 

system beyond presenting on projects mostly focused on laboratory based basic 

science efforts.

6. NIDDK could work jointly with CMS on a timely, highly visible and costly public 

health issue: the treatment of renal failure and chronic kidney disease.

CMS was pleased with the new data access system:

1. The system furthered the research into renal failure which was a pressing and 

highly visible costly part of Medicare

2. It furthered their (CMS) research program efforts in making research easier and 

less costly to support. And having qualified highly research focused data 

handlers provided supplemental efforts to the CMS data system which, of 

necessity, is focused on management and control of a multi-billion dollar billing 

and payment system.

3. CMS was justly proud that their incredible efforts at building, collecting, and 

managing a large powerful data system was receiving the recognition and credit 

that helped reward their efforts.

Policy makers, providers and regulatory agencies were pleased with the data access that 

resulted from this system:

1. More and more policy decisions could be made on the basis of hard data and not 

The Promise of Evidence-Based Policymaking 1567
6

just ad hoc information. For example Medicare Part D which was implemented 

many years later could draw on the real world processes facing renal failure 

patients and providers. Also a decision to bundle certain services provided in the 

dialysis process lead to substantial savings and could be monitored for 

unintended adverse consequences.

2. The success of these data access procedures was an essential ingredient when 

serious efforts were made to improve practice guidelines. (Research leading to 

this effort to improve patient care showed that more than 75 percent of treatment 

practices in the treatment of renal failure were based on hearsay, custom, and 

rumor). 

3. These data access procedures improved fact based medicine but also had some

positive unintended consequences. For example, the provider community saw 

there was benefit to allowing access to their own private patient data (with 

confidentiality protected) which of course added immensely to the type and detail 

of the data were available to the research community. In effect, the availability of 

privately collected data also made for a quality check on the federal data as well 

as providing an expansion of the data available from Federal sources. And the 

power of these data sources was shown to be contagious as the Veterans 

Administration subsequently joined in the efforts to assemble and use clinically 

relevant information system that could benefit by observing the CMS/USRDS 

efforts.

1568 The Promise of Evidence-Based Policymaking
7

Selected Statistics for the USRDS Database

Item Statistic (s) Comments

Founded

1988

Revolutionized Federal Medical 

Data Access

Architect:

Randall Webb

Data Sources

(For ESRD data)

Medicare;

Social Security System;

US Census;

Primary Data Collection

Organ Procurement 

Transplantation Network (OPTN)

Data Use Agreement between 

Medicare and NIDDK; also 

SRTR: Scientific Registry of 

Transplant Patients

Patient Counts

1996-2015

~ 2 Million prevalent ESRD 

patients

In 2014:

200,907 patients with a 

functioning kidney transplant

500,000 receiving chronic dialysis 

therapy

Chronic Kidney Disease (CKD)

For CKD, the USRDS uses

multiple data sources (Medicare 

5% sample, Clinformatics Data 

Mart, data from the VA health 

system, NHANES, etc.

Data Volume 

12/31/2015

25 TB for Producing ADR

1.6 TB for SAF and Linkage 

requests sent to researchers

ADR: Annual Data Report; 

SAF: Standard Analysis Files

(TB: terra bites)

Data and File 

Requests Fulfilled

2015:

US Government: 25

Helpdesk: 51

SAF: 70

SAF:

Standard Analysis Files

Data Files 

Released 

2015:

1,445

All SAF files released in 2015 –

not including separate files for 

individual data requests

Resulting Peer 

Reviewed Articles

2001 to 2015

90

Examples of Policy 

Based on 

Empirical Data

1. Payment for EPO drug

2. Bundling of Dialysis 

Services

3. Design of Clinical Trials 

and Demonstrations

The Promise of Evidence-Based Policymaking 1569
8

Selected Portions of the USRDS Data Documentation (Readily Available to 

the Research Community on the WWW)

2015 Researcher’s Guide to the

USRDS Database (571 Pages).

The entire Document is available at:

https://www.usrds.org/2015/rg/2015_USRDS_Researchers_Guide_15.pdf

Preface

The Researcher’s Guide is intended for investigators within and outside of the 

United States Renal Data System (USRDS) Coordinating Center (CC) who wish 

to undertake research projects using data from the USRDS database. This guide 

places particular emphasis on the USRDS Standard Analysis File (SAF) dataset, 

the primary means by which USRDS data are made available for use. The 

Researcher’s Guide includes information needed to help researchers select and 

use the appropriate SAFs for the intended project.

Introduction

USRDS website: www.usrds.org

The USRDS website provides users with access to the Annual Data Report 

(ADR), which is available in HTML and downloadable PDF, Excel, and 

PowerPoint files. ADR chapters and volumes are

available in PDF; Reference Tables and the data underlying the figures and 

tables for each chapter are available in Excel; and, all chapter figures and tables 

are available in PowerPoint slides. Because of the file size, downloading some 

portions of the ADR may require user patience.

1570 The Promise of Evidence-Based Policymaking
9

Data Use Agreement (available at:

https://www.usrds.org/2015/appx/3_1_USRDS_Agreement_For_Release_of_Data_15.pdf

United States Renal Data System (USRDS)

Merged Dataset Agreement for Release of Data

Project Title ___________________________________________________________________ 

In this agreement, “Requester Organization” means ___________________________________ 

A. The National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK), through the United 

States Renal Data System (USRDS) Coordinating Center, will provide the Requester data extracted 

from the USRDS research database (the “Data”), via download or on CDs, DVDs, or other media type. 

Prior to receiving USRDS data, the Requester will provide USRDS with a list of personally identifiable 

information (PII) so USRDS can report which of the Requester’s subjects are in the USRDS end-stage 

renal disease (ESRD) data. 

B. The sole purpose of providing the Data is the conduct of legitimate and approved biomedical, costeffectiveness, and/or other economic research by the Requester. 

C. USRDS shall not use or disclose the Requester’s data for any purpose other than to create the Data 

extracted from the USRDS database. In the event that the Requester’s data is used or disclosed for 

any purpose other than that covered by this agreement, USRDS will notify the Requester 

immediately and agree to work with Requester to address the use or disclosure. The USRDS will 

destroy the Requester’s dataset one year after the linkage is complete unless otherwise specified by 

the Requester in the research proposal. 

D. The Requester shall not combine or link the Data provided with any other collection or source of 

information that may contain information specific to individuals on the files, except where a waiver 

of authorization has been approved by the Requester’s IRB/Privacy Board and NIDDK. 

E. The Requester shall not use the Data for purposes that are not related to biomedical research, costeffectiveness, economic and/or other epidemiological research. Purposes for which the Data may 

not be used include, but are not limited to, 

• the identification and targeting of under- or over-served health service markets primarily for 

commercial benefit 

• the obtaining of information about providers or facilities for commercial benefit 

• insurance purposes such as redlining areas deemed to offer bad health insurance risks 

• adverse selection (e.g., identifying patients with high risk diagnoses) 

Any use of the Data for research not in the original proposal must be approved by the USRDS Project 

Officer (PO). 

The Promise of Evidence-Based Policymaking 1571
10

F. The Requester shall not publish or otherwise disclose the Data in the files to any person or 

organization unless the Data have been aggregated (that is, combined into groupings of Data such 

that the Data are no longer specific to any individuals within each grouping), and no cells 

(aggregates of Data) contain information on fewer than ten individuals or fewer than five providers 

or facilities. The Requester shall not publish or otherwise disclose Data that identify individual 

providers or facilities, or from which such identities could be inferred. However, the Requester may 

release Data to a contractor for purposes of data processing or storage if (1) the Requester specified 

in the research plan submitted to the USRDS Project Officer that Data would be released to the 

particular contractor, or the Requester has obtained written authorization from the PO to release 

the Data to such contractor, and (2) the contractor has signed a data release agreement with the PO. 

G. A copy of any aggregation of Data intended for publication shall be submitted to the PO for review 

for compliance with the confidentiality provisions of this agreement prior to submission for 

publication and, if not approved, shall not be published until compliance is achieved. The PO must 

respond within 30 days. 

The Approval Request Checklist may be found at: 

https://www.usrds.org/2015/appx/3_1_USRDS_Manuscript_Approval_Request_Checklist_15.pdf

H. Appropriate administrative, technical, procedural, and physical safeguards shall be established by 

the Requester to protect the confidentiality of the Data and to prevent unauthorized access to it. 

The safeguards shall provide a level of security outlined in OMB Circular No. A-130, Appendix III — 

Security of Federal Automated Information Resources, which sets forth guidelines for security plans 

for automated information systems in Federal agencies. 

I. No copies or derivatives shall be made of the Data in these files except as necessary for the purpose 

authorized in this agreement. The Requester shall keep an accurate written account of all such 

copies and derivative files, which will be furnished upon request to the PO. The USRDS Data files 

covered in this data use agreement (DUA) may be retained by the Requester until the date specified 

by the PO in the approval letter, at which time Requester may request renewal of this data use 

agreement to extend the retention period to comply with legal or institutional recordkeeping 

requirements or to maintain the integrity of the research or research publications. If at any time 

during the data retention period the DUA between USRDS and CMS is canceled, the Requester will 

be contacted to destroy the files in their possession. At the completion of the activities in the 

research plan, the file(s) and any derivative files and copies shall be destroyed. At that time, the 

Requester will inform the USRDS and the PO in writing that the files have been destroyed. 

J. For the purpose of inspecting security procedures and arrangements, authorized representatives of 

the NIDDK and/or of CMS will, upon request, be granted access to premises where the Data are 

kept.

1572 The Promise of Evidence-Based Policymaking
11

Standard Analysis Files (SAFs) requested:

܆ Core ܆ Dialysis Morbidity and Mortality Study (DMMS)

܆ Transplant ܆ Comprehensive Dialysis Study (CDS)

܆ Hospital ܆ Clinical Performance Measures

܆ CKD 5% Cohort Core ܆ Case Mix Adequacy (CMA)

܆ CKD 5% Cohort Hospital ܆ Active-Adipose Study (AAS) 

܆ CROWNWeb Clinical Data ܆ Medicare Claims Clinical Data 

For the following SAFs, indicate the claim year(s) requested as well:

܆ Institutional Claims (pre-1989 through 2013 available) _______________________ 

܆ Physician/Supplier Claims (1991–2013 available) _______________________ 

܆ Part D (2006–2013 available) _______________________ 

܆ Pre-ESRD Institutional Claims (incident years 1995-2013) _______________________ 

܆ Pre-ESRD Physician/Supplier Claims (incident years 1995-2013) _______________________ 

܆ Pre-ESRD Part D (incident years 2008-2013) _______________________ 

܆ CKD 5% Institutional Claims (1992–2013 available) _______________________ 

܆ CKD 5% Physician/Supplier Claims (1992–2013 available) _______________________ 

܆ CKD 5% Part D (2006–2013available) _______________________ 

Other:

܆ Provider Crosswalk ܆ Physician Crosswalk

IMPORTANT! Specify: 

܆Data ONLY on matched patients OR

܆Complete SAFs, including matched and unmatched patients 

 ____________________________________________________________________________________ 

 Requester Signature (for the Institutional Official for Data Assurance)

 ____________________________________________________________________________________ 

 Authorized Signatory (name, title & date) 

 ____________________________________________________________________________________ 

 Requester Address 

 ____________________________________________________________________________________ 

 Requester Telephone Number 

The Promise of Evidence-Based Policymaking 1573
12

Read and Acknowledged (for Primary Investigator and all co-investigators who will 

analyze data directly)

 _________________________________ ____________________________ _____________ 

 Investigator / Analyst signature Name Date 

 _________________________________ ____________________________ _____________ 

 Investigator / Analyst signature Name Date 

 _________________________________ ____________________________ _____________ 

 Investigator / Analyst signature Name Date 

 _________________________________ ____________________________ _____________ 

 Investigator / Analyst signature Name Date 

 (attach additional signature pages as necessary) 

USRDS Project Officer: Kevin C. Abbott, MD, NIDDK, NIH, Kevin.abbott@nih.gov

 

 

USRDS Project Officer Signature Date 

Checklist: 

DID YOU REMEMBER TO SEND: 

܆ Signed copy of your institutional IRB approval memo 

܆ Copy of your project proposal in recommended format 

at http://www.usrds.org/2015/appx/3/2_Outline_for_research_proposals_using_merged_

USRDS_data.pdf

܆ Copy of this Data Use Agreement signed by your institutional official, PI, and all 

active participants. 

Please note that any MODIFICATIONS or AMMENDMENTS, regardless of whether they 

require additional files, require a new IRB approval memo (1 above), copy of the 

original project proposal (2 above) with additional analyses/extractions highlighted, and 

a new signed Data Use Agreement (3). 

Please send ALL documents (including the research protocol) in PDF format 

(please save the research protocol as PDF within Microsoft Word when you have 

completed it). AND consolidate all files into a single PDF file (using the “PDF 

Portfolio” feature in Adobe) when sending to the NIDDK. 

1574 The Promise of Evidence-Based Policymaking
“Commission on Evidence-Based Policymaking Comments”

Docket ID USBC-2016-0003

November 14, 2016

Comments Submitted to the 

 “Evidence-Based Policymaking Commission” (CEP) 

(Created by the ‘‘Evidence-Based Policymaking Commission Act of 2016’, Public Law 114–140 

114th Congress). 

Submitted by: Randall Webb, MA, (randall.webb@att.net) 

 Philip J. Held, PhD, (esrd00@gmail.com)

Table of Contents 

 Page 

Executive Summary 2 

Statement of the Problem 3 

 

Introduction 3 

 

Background 3 

 

NIDDK/NIH USRDS Creates a System to Share Data 4 

 

Selected Statistics for the USRDS Database 7 

Selected Portions of the USRDS Data Documentation 8 

(Readily Available to the Research Community) 

Data Use Agreement 9 

 

The Promise of Evidence-Based Policymaking 1575
2 

Executive Summary 

Medical care in the US is in general, of exceptionally high quality although finding the 

resources to provide it is a never-ending public issue in economic, moral, and political terms. 

Data, not surprising, are a central part of the health care sector and are fundamental to: 

• Experiments to test and improve new clinical approaches, drugs, procedures and 

policies 

• Surveillance of health and disease tracking and providing quality assessment and 

guidance for population health improvement 

• Guiding and advising patients as well as protecting confidentiality 

• Compensating and regulating the millions of suppliers 

• Maintaining trust and confidence among all sectors 

This Commission on Evidence Based Policy Making is focused on the use, control, and access 

to data - presumably specifically defined as public data paid for by the US taxpayer that are of 

potential use in making policy decisions based on empirical information. The public concerns 

which generated this commission must have been focused on access to tax payer funded data 

as well as protection of confidentiality. 

This comment for the CEP reports on the United States Renal Data System (USRDS) a very 

productive and vital system that generates, controls, and widely distributes an endless volume 

of data that is part of the federal system that funds, monitors, and controls, the care of nearly 

700,000 Americans with end stage renal disease (ESRD) – requiring dialysis therapy or kidney 

transplantation, as well as the millions of Americans with the preceding stages of chronic 

kidney disease (CKD), not at ESRD. The two primary institutions that run this program are the 

Centers for Medicaid and Medicare (CMS) and the National Institute of Diabetes, Digestive 

and Kidney Diseases (NIDDK) which is part of the National Institutes of Health (NIH). Both 

these institutions are part of the US Department of Health and Human Services. 

The data system incrementally developed by the USRDS has been successfully made 

available for use by researchers – sharing, distributing standard analysis files (SAFs) and 

protecting sensitive data, for over 25 years – without serious mishap. Incredible volumes of 

detailed and extensive information, often of a confidential nature, are made available to 

researchers and policy makers both in and out of the government including profit and nonprofit institutions. This data distribution system has been central and vital to public policy, the 

setting of medical guidelines, and encouragement and nurturing of a research arena second to 

none in the world. 

The system that provides these data at the heart of the treatment of kidney failure can provide 

a good model to the CEP. 

 

1576 The Promise of Evidence-Based Policymaking
3 

Statement of the Problem 

Kidney disease is the 9th leading cause of death and a significant contributor to cardiovascular 

and diabetic deaths – it has been called both a ‘disease multiplier’ and a ‘cost multiplier’ and is 

a non-communicable disease that is more prevalent than diabetes mellitus in the US general 

population. There are about 500,000 patients on chronic dialysis therapy and nearly 100,000 

patients on the kidney transplant waitlist, which continues to grow. Medicare spends $32.8 

billion per year (USRDS 2016 Annual Data Report) on the treatment of patients with End Stage 

Renal Disease (ESRD) – 7.2% of total Medicare costs for ~1% of the population served by 

Medicare. Combining all sources of payment (Medicare, Medicaid, private insurance and 

patient obligations), ESRD patients have costs of $125,000 per patient per year and a high 

mortality rate of approximately 15-20% per year. The National Institute of Digestive, Diabetes 

and Urologic Diseases (NIDDK) typical budget amounts to nearly $2 billion per year. 

Introduction 

It is our understanding that the CEP is interested in developing policy recommendations to aid 

government policymaking. In developing policy one can either make decisions on ideological 

basis or on a scientific basis. If the scientific basis is chosen for such decision-making the 

following are minimum requirements: 

• Relevant and sufficient data, 

• Appropriate analytical methods and, 

• Processes open to validation and reproducibility 

Review of the CEPs Law’s mandate makes it clear that the CEP is to develop policies based 

on a scientific process with an emphasis on the issue of data which involves: 

1. Determining which policy questions are to be addressed 

2. Identification of relevant data to address these questions 

3. Identification of persons and institutions appropriate to have access to these data 

4. Determining methods of accessing such data including the allocation of costs of making 

data available 

5. When appropriate, determine procedures to protect confidentiality of sources, 

particularly patient-level identifiable information 

6. Determining what are appropriate versus inappropriate uses of these data 

Background 

The United States Renal Data System (USRDS) is run jointly by the National Institute of 

Digestive, Diabetes and Urologic Diseases (NIDDK) and the Centers for Medicare and 

Medicaid Services (CMS). The USRDS was formed in response to a Congressional Mandate 

requiring the creation of “a National Renal [Kidney] Registry” for the US and has been 

operated, under contract, for over 25 years and remains a powerful and successful example of 

how to fulfill the 6 requirements listed above. 

The Promise of Evidence-Based Policymaking 1577
4 

Prior to the formation of the USRDS, almost all medical data funded in total or in part by the 

Federal Government were kept usually under the control of a contracting or grant receiving 

agency or institution that collected or assembled the data. The same could be said of 

numerous federal studies wherein the data were collected at great taxpayer expense, but 

typically the data were kept in vault-like circumstances where only a few selected internal 

investigators ever had access to these data. 

NIDDK/NIH funded USRDS Creates a System to Share Data Widely (1988) 

When NIDDK issued an RFP for the creation of the USRDS, the winning bidder’s proposal 

(Philip Held PI, and Randall Webb, Chief of Information Systems) contained a sophisticated 

and timely set of proposals to ensure that data and analyses performed in this project would 

be: 

1. Transparent and built on widely available data handling systems. (Past 

experience in working with medical data stored on varied university systems was 

often based on locally developed systems that were dated, poorly documented 

and demanding of many technician hours to accomplish even minimal goals). 

2. Transferable to other data systems both in and out of government. Oracle, a 

relatively new operation at that time, and SAS were the foundations of all project 

work although other relational databases did evolve over time. 

3. Inclusive of all files including cleaned analytical files, designed to save users 

many person-years of effort, by circumventing the need for tedious data 

collection and processing efforts before real analysis could begin would be made 

available. 

4. Files designed to be consistently defined over the years to allow longitudinal 

analyses. 

5. Development of a Researchers Guide to guide researchers in the use of the 

data. 

6. Incorporation of new data sources, notably from the CMS CROWNWeb and Part 

D data systems, into the database as they became available. 

7. Open to requests for other formulations of the data. 

8. Open to merging existing data with other data sources if privacy requirements 

can be met. 

9. Pricing these files at cost of reproduction. 

10. Determining what were appropriate and inappropriate uses of the data. Research 

in the public interest was the general rule. Research designed to discriminate 

against patients for commercial reasons was not appropriate research. 

11. Involved of a Scientific Advisory committee to supplement and guide the 

contractor’s staff in insuring the purposes for which the data were released were 

intended for valid scientific purposes and in the public interest. 

12. Inclusive of developing methods and a Data Use Agreement to protect 

confidentially of these medical data, which incorporated very sensitive 

information. 

With the initiation of the USRDS, data were from two primary sources: CMS’s Medicare data 

and primary data designed by the USRDS but collected by the ESRD Networks, a series of 

institutions contracted by CMS to assist in the management of the ESRD program. Dialysis 

1578 The Promise of Evidence-Based Policymaking
5 

and transplant units were also tapped into to collect relevant data. 

It is not an over statement to note that these handling and distribution procedures by the 

USRDS contractor were quite radical at the time but were ultimately immensely popular with all 

sorts of “stakeholders” including the research community, the media, the NIDDK, CMS, and 

the providers (both profit and not-for-profit). 

NIDDK was pleased and proud of this new data access system: 

1. NIDDK could help their grantees tackle a wider list of topics in a timely and costefficient manner. 

2. NIDDK could gain real empirical data to assist in the design of clinical trials by 

use of these observational data. 

3. The USRDS Annual Data Reports (ADRs) have been produced annually since 

1989. The ADRs have become the foundational national reference on the 

epidemiology of ESRD and CKD. 

4. Because the same files are used for the ADR and by outside researchers, the 

researcher’s results can be benchmarked against the ADR. 

5. They could carry to Capitol Hill results from a ‘real world’ clinical, empirical data 

system beyond presenting on projects mostly focused on laboratory based basic 

science efforts. 

6. NIDDK could work jointly with CMS on a timely, highly visible and costly public 

health issue: the treatment of renal failure and chronic kidney disease. 

CMS was pleased with the new data access system: 

1. The system furthered the research into renal failure which was a pressing and 

highly visible costly part of Medicare 

2. It furthered their (CMS) research program efforts in making research easier and 

less costly to support. And having qualified highly research focused data 

handlers provided supplemental efforts to the CMS data system which, of 

necessity, is focused on management and control of a multi-billion dollar billing 

and payment system. 

3. CMS was justly proud that their incredible efforts at building, collecting, and 

managing a large powerful data system was receiving the recognition and credit 

that helped reward their efforts. 

Policy makers, providers and regulatory agencies were pleased with the data access that 

resulted from this system: 

1. More and more policy decisions could be made on the basis of hard data and not 

The Promise of Evidence-Based Policymaking 1579
6 

just ad hoc information. For example Medicare Part D which was implemented 

many years later could draw on the real world processes facing renal failure 

patients and providers. Also a decision to bundle certain services provided in the 

dialysis process lead to substantial savings and could be monitored for 

unintended adverse consequences. 

2. The success of these data access procedures was an essential ingredient when 

serious efforts were made to improve practice guidelines. (Research leading to 

this effort to improve patient care showed that more than 75 percent of treatment 

practices in the treatment of renal failure were based on hearsay, custom, and 

rumor). 

3. These data access procedures improved fact based medicine but also had some 

positive unintended consequences. For example, the provider community saw 

there was benefit to allowing access to their own private patient data (with 

confidentiality protected) which of course added immensely to the type and detail 

of the data were available to the research community. In effect, the availability of 

privately collected data also made for a quality check on the federal data as well 

as providing an expansion of the data available from Federal sources. And the 

power of these data sources was shown to be contagious as the Veterans 

Administration subsequently joined in the efforts to assemble and use clinically 

relevant information system that could benefit by observing the CMS/USRDS 

efforts. 

 

1580 The Promise of Evidence-Based Policymaking
7 

 

Selected Statistics for the USRDS Database 

Item Statistic (s) Comments

Founded 

1988 

Revolutionized Federal Medical 

Data Access 

Architect: 

 Randall Webb 

 

Data Sources 

(For ESRD data) 

Medicare; 

Social Security System; 

US Census; 

Primary Data Collection 

Organ Procurement 

Transplantation Network (OPTN) 

Data Use Agreement between 

Medicare and NIDDK; also 

SRTR: Scientific Registry of 

Transplant Patients 

Patient Counts 

1996-2015 

~ 2 Million prevalent ESRD 

patients 

In 2014: 

200,907 patients with a 

functioning kidney transplant 

500,000 receiving chronic dialysis 

therapy 

Chronic Kidney Disease (CKD) 

 For CKD, the USRDS uses 

multiple data sources (Medicare 

5% sample, Clinformatics Data 

Mart, data from the VA health 

system, NHANES, etc. 

Data Volume 

12/31/2015

25 TB for Producing ADR 

1.6 TB for SAF and Linkage 

requests sent to researchers 

ADR: Annual Data Report; 

SAF: Standard Analysis Files 

(TB: terra bites) 

Data and File 

Requests Fulfilled 

2015:

US Government: 25

Helpdesk: 51

SAF: 70

SAF: 

Standard Analysis Files 

Data Files 

Released 

2015:

1,445 

All SAF files released in 2015 – 

not including separate files for 

individual data requests 

Resulting Peer 

Reviewed Articles 

2001 to 2015 

90 

Examples of Policy 

Based on 

Empirical Data 

1. Payment for EPO drug 

2. Bundling of Dialysis 

Services 

3. Design of Clinical Trials 

and Demonstrations 

 

The Promise of Evidence-Based Policymaking 1581
8 

Selected Portions of the USRDS Data Documentation (Readily Available to 

the Research Community on the WWW) 

2015 Researcher’s Guide to the 

USRDS Database (571 Pages). 

The entire Document is available at: 

https://www.usrds.org/2015/rg/2015_USRDS_Researchers_Guide_15.pdf

Preface 

The Researcher’s Guide is intended for investigators within and outside of the 

United States Renal Data System (USRDS) Coordinating Center (CC) who wish 

to undertake research projects using data from the USRDS database. This guide 

places particular emphasis on the USRDS Standard Analysis File (SAF) dataset, 

the primary means by which USRDS data are made available for use. The 

Researcher’s Guide includes information needed to help researchers select and 

use the appropriate SAFs for the intended project. 

Introduction 

USRDS website: www.usrds.org 

The USRDS website provides users with access to the Annual Data Report 

(ADR), which is available in HTML and downloadable PDF, Excel, and 

PowerPoint files. ADR chapters and volumes are 

available in PDF; Reference Tables and the data underlying the figures and 

tables for each chapter are available in Excel; and, all chapter figures and tables 

are available in PowerPoint slides. Because of the file size, downloading some 

portions of the ADR may require user patience. 

 

1582 The Promise of Evidence-Based Policymaking
9 

Data Use Agreement (available at:

https://www.usrds.org/2015/appx/3_1_USRDS_Agreement_For_Release_of_Data_15.pdf

United States Renal Data System (USRDS) 

Merged Dataset Agreement for Release of Data 

*'",!,$====================================================================


&, !+*%&,37)-+,**&!2,!'&8%&+===================================

5  ,!'&$
&+,!,-,'!,+&!+,!.&!&1!+++>
?3, *'- , &!,

,,+&$,1+,%>?''*!&,!&&,*3/!$$(*'.!, )-+,*,0,*,

*'%, *+* ,+>, 7,8?3.!'/&$''*'&+3+3'*', *%!,1(5

*!'*,'*!.!&,3, )-+,*/!$$(*'.!/!, $!+,'(*+'&$$1!&,!!$

!&'*%,!'&>

?+'&*('*,/ ! ', )-+,*6++-",+*!&, &:+,

*&$!++>?,5

5  +'$(-*('+'(*'.!!&, ,!+, '&-,'$!,!%,&((*'.!'%!$3'+,:

,!.&++3&9'*', *'&'%!*+* 1, )-+,*5

5 + $$&',-+'*!+$'+, )-+,*6+,'*&1(-*('+', *, &,'*,, ,

0,*,*'%, ,+5
&, .&,, ,, )-+,*6+,!+-+'*!+$'+'*

&1(-*('+', *, &, ,'.*1, !+*%&,3/!$$&',!1, )-+,*

!%%!,$1&*,'/'*#/!, )-+,*,'*++, -+'*!+$'+-*5 /!$$

+,*'1, )-+,*6+,+,'&1*,*, $!&#!+'%($,-&$++', */!++(!!1

, )-+,*!&, *+* (*'('+$5

5  )-+,*+ $$&','%!&'*$!&#, ,(*'.!/!, &1', *'$$,!'&'*+'-*'

!&'*%,!'&, ,%1'&,!&!&'*%,!'&+(!!,'!&!.!-$+'&, !$+30(,/ */!.*

'-, '*!2,!'& +&((*'.1, )-+,*6+
9*!.1'*&
5

5  )-+,*+ $$&',-+, ,'*(-*('++, ,*&',*$,,'!'%!$*+* 3'+,:

,!.&++3'&'%!&9'*', *(!%!'$'!$*+* 5-*('++'*/ ! , ,%1

&',-+!&$-3-,*&',$!%!,,'3

• , !&,!!,!'&&,*,!&'-&*:'*'.*:+*. $, +*.!%*#,+(*!%*!$1'*

'%%*!$&!,

• , ',!&!&'!&'*%,!'&'-,(*'.!*+'*!$!,!+'*'%%*!$&!,

• !&+-*&(-*('+++- +*$!&!&*+%,''* $, !&+-*&*!+#+

• .*++$,!'&>553!&,!1!&(,!&,+/!,  ! *!+#!&'++?

&1-+', ,'**+* &',!&, '*!!&$(*'('+$%-+,((*'.1, *'",

!*>?5

The Promise of Evidence-Based Policymaking 1583
10 

5  )-+,*+ $$&',(-$!+ '*', */!+!+$'+, ,!&, !$+,'&1(*+'&'*

'*&!2,!'&-&$++, , .&*,>, ,!+3'%!&!&,'*'-(!&+',+- 

, ,, ,*&'$'&*+(!!,'&1!&!.!-$+/!, !& *'-(!&?3&&'$$+

>*,+',?'&,!&!&'*%,!'&'&/*, &,&!&!.!-$+'*/*, &!.(*'.!*+

'*!$!,!+5 )-+,*+ $$&',(-$!+ '*', */!+!+$'+,, ,!&,!1!&!.!-$

(*'.!*+'*!$!,!+3'**'%/ ! +- !&,!,!+'-$!&**5	'/.*3, )-+,*%1

*$+,,''&,*,'*'*(-*('++',(*'++!&'*+,'*!>D?, )-+,*+(!!

!&, *+* ($&+-%!,,,', *'",!*, ,,/'-$*$+,', 

(*,!-$*'&,*,'*3'*, )-+,* +',!&/*!,,&-, '*!2,!'&*'%, ,'*$+

, ,,'+- '&,*,'*3&>E?, '&,*,'* ++!&,*$+*%&,/!, , 5

5 '(1'&1*,!'&',!&,&'*(-$!,!'&+ $$+-%!,,,', '**.!/

'*'%($!&/!, , '&!&,!$!,1(*'.!+!'&+', !+*%&,(*!'*,'+-%!++!'&'*

(-$!,!'&&3!&',((*'.3+ $$&',(-$!+ -&,!$'%($!&!+ !.5 %-+,

*+('&/!, !&FC1+5

 ((*'.$)-+, #$!+,%1'-&,4

 ,,(+499///5-+*+5'*9ECDG9((09F=D==&-+*!(,=((*'.$=)-+,= #$!+,=DG5(

	5 ((*'(*!,%!&!+,*,!.3, &!$3(*'-*$3&( 1+!$+-*++ $$+,$!+ 1

, )-+,*,'(*',,, '&!&,!$!,1', ,&,'(*.&,-&-, '*!2++,'!,5

 +-*++ $$(*'.!$.$'+-*!,1'-,$!&!&!*-$*'5:DFC3((&!0


<

-*!,1'*$-,'%,
&'*%,!'&+'-*+3/ ! +,+'*, -!$!&+'*+-*!,1($&+

'*-,'%,!&'*%,!'&+1+,%+!&*$&!+5


5 ''(!+'**!.,!.++ $$%', ,!&, +!$+0(,+&++*1'*, (-*('+

-, '*!2!&, !+*%&,5 )-+,*+ $$#(&-*,/*!,,&'-&,'$$+- 

'(!+&*!.,!.!$+3/ ! /!$$-*&!+ -('&*)-+,,', 5 ,!$+

'.*!&, !+,-+*%&,>?%1*,!&1, )-+,*-&,!$, ,+(!!

1, !&, ((*'.$$,,*3,/ ! ,!%)-+,*%1*)-+,*&/$', !+,-+

*%&,,'0,&, *,&,!'&(*!',''%($1/!, $$'*!&+,!,-,!'&$*'*#(!&

*)-!*%&,+'*,'%!&,!&, !&,*!,1', *+* '**+* (-$!,!'&+5
,&1,!%

-*!&, ,*,&,!'&(*!', ,/&&!+&$3, )-+,*/!$$

'&,,,'+,*'1, !$+!&, !*('++++!'&5,, '%($,!'&', ,!.!,!+!&, 

*+* ($&3, !$>+?&&1*!.,!.!$+&'(!++ $$+,*'15,, ,,!%3, 

)-+,*/!$$!&'*%, &, !&/*!,!&, ,, !$+ .&+,*'15

5 '*, (-*('+'!&+(,!&+-*!,1(*'-*+&**&%&,+3-, '*!2*(*+&,,!.+'

, 
&9'*'/!$$3-('&*)-+,3*&,++,'(*%!++/ *, ,*

#(,5

1584 The Promise of Evidence-Based Policymaking
11 


#
$"

'* !$1+!+'*!!,1&'*,$!,1,-1>?

*&+($&, '%(* &+!.!$1+!+,-1>?

	'+(!,$ $!&!$*'*%&+-*+

GM' '*,'* +!0)-1>?

GM' '*,	'+(!,$ ,!.:!('+,-1>?

$!&!$, !*$!%+$!&!$,


!

#$"


&+,!,-,!'&$$!%+>(*:DJIJ, *'- ECDF.!$$? ======================= 

 1+!!&9-(($!*$!%+>DJJD;ECDF.!$$? ======================= 

*,>ECCH;ECDF.!$$? ======================= 

*:
&+,!,-,!'&$$!%+>!&!&,1*+DJJG:ECDF? ======================= 

*: 1+!!&9-(($!*$!%+>!&!&,1*+DJJG:ECDF? ======================= 

*:*,>!&!&,1*+ECCI:ECDF? ======================= 

GM
&+,!,-,!'&$$!%+>DJJE;ECDF.!$$? ======================= 

GM 1+!!&9-(($!*$!%+>DJJE;ECDF.!$$? ======================= 

GM*,>ECCH;ECDF.!$$? ======================= 

"

*'.!**'++/$#  1+!!&*'++/$#

	 

"

,
'&%, (,!&,+	

'%($,+3!&$-!&%, &-&%, (,!&,+

===================================================================================== 




>'*, 
&+,!,-,!'&$!!$'*,++-*&?

===================================================================================== -, '*!2!&,'*1>&%3,!,$@,?

===================================================================================== )-+,**++

=====================================================================================

)-+,*$( '&-%*

The Promise of Evidence-Based Policymaking 1585
12 






>'**!%*1
&.+,!,'*&$$':!&.+,!,'*+/ '/!$$

&$12,!*,$1?

================================== ============================   ==============


&.+,!,'*9&$1+,+!&,-* % ,

================================== ============================   ==============


&.+,!,'*9&$1+,+!&,-* % ,

================================== ============================   ==============


&.+,!,'*9&$1+,+!&,-* % ,

================================== ============================   ==============


&.+,!,'*9&$1+,+!&,-* % ,

>,, !,!'&$+!&,-*(++&++*1?

	

4.!&5',,33
3
	
	
	


 

 

*'",!*!&,-* ,





4

!&'(1'1'-*!&+,!,-,!'&$
((*'.$%%'

'(1'1'-*(*'",(*'('+$!&*'%%&'*%,,

 ,,(499///5-+*+5'*9ECDG9((09F9E=-,$!&='*=*+* =(*'('+$+=-+!&=%*=

=,5(

'(1', !+,+*%&,+!&11'-*!&+,!,-,!'&$'!!$3
3&$$

,!.(*,!!(&,+5

$+&',, ,&1


'*3**$++'/ , *, 1

*)-!*!,!'&$!$+3*)-!*&/
((*'.$%%'>D'.?3'(1', 

'*!!&$(*'",(*'('+$>E'.?/!, !,!'&$&$1++90,*,!'&+ ! $! ,3&

&/+!&,+*%&,>F?5

$++&

'-%&,+>!&$-!&, *+* (*',''$?!&'*%,

>($++., *+* (*',''$+/!, !&!*'+','*/ &1'- .

'%($,!,?5'&+'$!,$$!$+!&,'+!&$!$>-+!&, 7

'*,'$!'8,-*!&'?/ &+&!&,', 
5

1586 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t19-zl6w

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0145

Comment on FR Doc # 2016-22002

Submitter Information

Name: Rachel Zinn

Address:

Washington, DC, 20036

Email: rachelz@workforcedqc.org

General Comment

Workforce Data Quality Campaign (WDQC), a project of National Skills Coalition, is a

non-profit initiative that promotes inclusive, aligned and market-relevant education and

workforce data. Guided by a diverse group of national partners and state officials

representing stakeholders across the education and workforce spectrum, WDQC

encourages the use of data to ensure that all of our nation's education and training

programs are preparing students and workers to succeed in a changing economy.

The attached document contains our comments in response to the U.S. Bureau of the

Census (USBC) Notice: Request for Comments for the Commission on Evidence-Based

Policymaking (Docket USBC-2016-0003-0001). We appreciate the opportunity to offer

recommendations to the Commission.

Attachments

EBP_Commission_Comments_Nov2016-14fin

The Promise of Evidence-Based Policymaking 1587
To: Commission on Evidence-Based Policymaking 

From: Workforce Data Quality Campaign

Re: Docket ID USBC-2016-0003-0001 

Date: November 14, 2016 

Workforce Data Quality Campaign (WDQC), a project of National Skills Coalition, is a non-profit 

initiative that promotes inclusive, aligned and market-relevant education and workforce data. Guided 

by a diverse group of national partners and state officials representing stakeholders across the 

education and workforce spectrum, WDQC encourages the use of data to ensure that all of our 

nation’s education and training programs are preparing students and workers to succeed in a 

changing economy. 

The Commission on Evidence-Based Policymaking’s examination of federal administrative and survey 

data provides an exceptional opportunity to address the management and use of data for measuring 

postsecondary education and workforce outcomes. The Commission’s findings could go far beyond 

enabling the provision of better information for researchers and policymakers. It could also help 

educators, students, employers, and workers all make more informed decisions. 

Overarching Challenge 

The collection and use of education and workforce data can improve human capital policies that play 

a part in strengthening the national economy, but information still remains too disconnected to 

realize its full potential for improving workforce and economic development. 

We advise the Commission to consider the creation of a national clearinghouse that would facilitate 

linkages between employment data and data about postsecondary and workforce program 

participants. The system should enable public access to data aggregated or anonymized sufficiently 

to prevent tracing back information to individuals, but disaggregated sufficiently to allow useful 

program analysis and research. Absent the creation of a clearinghouse, further developing the use 

and linkages of these data can still improve program reporting, evaluation, and research. We also 

recommend aligning federal laws like the Workforce Innovation and Opportunity Act (WIOA), The 

Perkins Career and Technical Education Act (Perkins), and the Higher Education Act (HEA) to improve 

data collection. Finally, we recommend providing assistance and incentives for states to strengthen 

their roles in the collection, use, and sharing of data. 

The Commission also should issue an admonition, possibly in its final report, to remind audiences 

that data alone will not drive policy improvements, and that human factors will influence how data 

are interpreted and used. While policy improvements should become more likely because of better 

data, linkages, and access, we must continually work to create a culture of responsible data use. 

Federal data policies should be regularly assessed and refined as thinking and technology change. 

1588 The Promise of Evidence-Based Policymaking
2 

Below are recommendations organized according to questions that are the most pertinent to our 

remit of improving the collection and use of data for education and workforce policies and programs. 

Our comments incorporate input from multiple national and state experts. 

COMMISSION QUESTION #1: Are there successful frameworks, policies, practices, and methods 

to overcome challenges related to evidence-building from state, local, and/or international 

governments the Commission should consider when developing findings and recommendations 

regarding Federal evidence-based policymaking? If so, please describe.

Through our work with states and federal agencies over the past several years, we have observed the 

following effective practices for using data to inform workforce development policy: 

x Convene relevant stakeholders to identify high-priority policy questions, then develop a 

longitudinal data infrastructure to enable meaningful analysis to answer these questions. 

x Allow states (and their agents) to access this information to assist with required program 

reporting and evaluation, and further tailor information for their specific needs. 

x Outline the importance of data linkages, privacy, and security. Select systems that ensure 

privacy, security, and confidentiality. 

x Educate the public and policymakers on the merits of data collection and use, and 

particularly on the importance of labor market information. 

x Facilitate and encourage the incorporation of data into easy-to-use tools that fulfill specific 

stakeholder needs. 

x Invest in technical assistance and professional development to ensure that data managers 

know how to use data to inform policy choices. 

More detailed examples of these practices and proposals are included in answers to the questions 

below. 

COMMISSION QUESTION #2: Based on identified best practices and existing examples, what 

factors should be considered in reasonably ensuring the security and privacy of administrative 

and survey data?

Ensuring that privacy and security are integrated with the development of new data linkages or the 

creation of a national clearinghouse will be essential for building trust and support for the 

Commission’s recommendations. 

Policies established to guide system development and use should always address privacy and 

security. Moreover, these policies should be transparent, meaning that they should be published, and 

also communicated in a way that will be understood by relevant stakeholders.1 

 

1

 Several important federal laws and rules applicable to privacy and confidentiality were covered during the 

Commission’s meeting of September 9, 2016, including The Office of Management and Budget Circular A-130, 

the 1974 Privacy Act, the Confidential Information Protection and Statistical Efficiency Act (CIPSEA), and the 

Family Educational Rights and Privacy Act (FERPA). 

The Promise of Evidence-Based Policymaking 1589
3 

To reduce risk, the following basic practices should be followed: 

x Only collect data that are needed. This limits the opportunity for revealing personal 

information, and has the added benefit of reducing burden on data providers, collectors, and 

systems. 

x Ensure that publicly available data are aggregated or otherwise stripped of all information 

that could be used to identify particular individuals or employers. 

x Avoid willful disclosure or inappropriate use. Convey consequences to all relevant parties. 

We also encourage the use of the following best practices: 

x Promote research: Maintaining privacy, confidentiality and security is essential, but it should 

not be construed as a reason to pull back on research. Data policies should explicitly allow 

research, policy analysis, and program evaluation, in addition to publication of aggregate 

performance outcomes. 

x Transparency: Each agency that has administrative data should publish their process for datasharing with researchers and policymakers; including identifying what data the agency has 

and are available for sharing, the mechanism for requesting access, specific timelines for 

making decisions, and the specific individuals who are authorized to make decisions about 

access. The process should also identify who is eligible to access data and for what purposes, 

what security measures are required for maintaining the data that are ultimately received, 

and any fees imposed for preparing the data. 

x Create Memoranda of Understanding: Craft standard templates for memoranda of 

understanding (MOUs) to ensure that personnel are sharing and using information according 

to consistent guidelines. 

x Take into account different cultural perspectives: Residents have different levels of comfort 

about government and other entities having access to personal data. Sensitivity to different 

perspectives on privacy should shape how recommendations are made and how privacy is 

addressed in data collection. 

Technological advances should be explored and used, with any plan including provisions for 

regularized assessments to adopt improved practices and adaptation to evolving security threats. 

Currently, two technological innovations may be worth exploring in particular: 

Tiered access allows layers of access and aggregation using the same foundational data sets. This 

technological solution could also eliminate the need to hold different data sets in different locations 

or among disparate systems with varied levels of security. 

California provides a good example of tiered access. The California Community Colleges Chancellor’s 

Office career and technical education (CTE) “LaunchBoard” tool, which links education data and wage 

records, provides community college employees with login access to detailed dashboards that 

analyze outcomes of CTE programs. The same underlying data is used to create “Salary Surfer,” 2

which is a publicly-available tool that allows users to see the median salaries of postsecondary 

programs at California’s community colleges. 

 

2

 Cal-PASSPlus, “About the LaunchBoard,” https://www.calpassplus.org/Launchboard/About.aspx 

1590 The Promise of Evidence-Based Policymaking
4 

Texas has made some of the biggest strides in data integration, and provides different types of users 

with access to specific portions of its workforce database. The U.S. Government Accountability Office 

(GAO) described the Texas data infrastructure in its recent review of Workforce Innovation and 

Opportunity Act (WIOA) reporting. Texas agencies draw on multiple data systems that share 

information through regular, automated exchanges. The data from these systems are compiled in a 

separate system, called The Workforce Information System of Texas (TWIST) web reports, where UI 

wage data and other data are merged with participant information and then sent to the U.S. 

Department of Labor (DOL).3 TWIST uses a system providing different levels of restricted access so 

agencies can see individual-level outcomes for only their own participants.4

Synthetic data are artificial data sets that are similar to, but are not the raw, confidential data from 

which they are derived. A statistical model is created that adjust specific data fields so that individuallevel data no longer corresponds to real people, but the data set as a whole maintains the 

characteristics of the original. Because the data are not tied to real people, confidentiality rules do 

not apply. 

Analysts can access these data that have the properties of the real data, without facing the same 

restrictive barriers in addressing their research questions. The types of inferences that may be drawn 

from synthetic data, however, will be more limited. Therefore, the fact that these data sets are 

synthetic must be made clear so researchers and other users can take into account the limitations of 

these data sets for appropriate use.5

The Maryland Longitudinal Data System (MLDS) Center has embarked on a multi-year project to 

create synthetic data sets, a project which was funded by a portion of a grant from the U.S. 

Department of Education (ED).6 The U.S. Census Bureau’s Longitudinal Employment-Household 

Dynamics (LEHD) program uses partially synthetic data to examine workers’ residential patterns.7 

COMMISSION QUESTION #3: Based on identified best practices and existing examples, how should 

existing government data infrastructure be modified to best facilitate use of, and access to, 

administrative and survey data? 

Rationalizing the federal data landscape would require significant changes to multiple laws, but it 

could cut costs and reduce security risks by cutting down on the number of redundant data sets 

scattered across government, and could also make the creation of a national clearinghouse more 

manageable. In the answer to Question #7, we identify federal data sets that deserve consideration. In 

 

3

 U.S. General Accountability Office, “WIOA Performance and Accountability Data,” March 2016, p. 12, 

http://www.gao.gov/assets/680/675642.pdf 

4

 Barry Stern, “Stepping Up: State Developments in SNAP Employment & Training Data,” March 2015, Workforce 

Data Quality Campaign: Washington, DC, p. 5. 

5

 R. Jarmin, T. Louis, and J. Miranda, “Expanding the Role of Synthetic Data at the Census Bureau,” Center for 

Economic Studies, February 2014, ftp://ftp2.census.gov/ces/wp/2014/CES-WP-14-10.pdf 6

 University of Maryland, College of Education, “MLDS Center Investigates the Creation of Synthetic Data,” Web 

News Release, December 2015, http://www.education.umd.edu/CollegeNews/2015/wnrMLDSdata.html. 

7

 U.S. Census Bureau, “Confidentiality Protection,” OnTheMap Help and Documentation, 

http://lehd.ces.census.gov/applications/help/onthemap.html#!confidentiality_protection

The Promise of Evidence-Based Policymaking 1591
5 

particular, revising access policies for wage record sources would greatly improve the quality of 

information for performance reporting and analysis. For example, access to the Department of Health 

and Human Services’ National Directory of New Hires (NDNH) data, which includes actual hire dates in 

addition to quarterly wages, would help to inform workforce development programs on earnings and 

indicate how long it takes people to return to work. 

In addition to revising the organizational structure of data systems, the federal government should 

align data definitions and performance metrics across key U.S. laws that authorize education and 

workforce development programs. Consistent definitions would streamline data collection and 

provide comparable information, facilitating program collaboration and allowing policymakers to 

have a holistic view of human capital development. 

WIOA passed in 2014 with wide bipartisan approval to help job seekers succeed in the labor market by 

better coordinating access to employment, education, training, and support services. Since then, 

government agencies significantly increased their collaboration to issue rules and guidance on 

performance reporting, making WIOA a useful guidepost by which to align reporting requirements 

with other programs related to those services. We advise the Commission to take a closer look at the 

following laws and programs for alignment of definitions and metrics: 

Perkins – This law, which governs career and technical education, is currently due for 

reauthorization. The revised law should use the WIOA definition of “postsecondary recognized 

credential,” which includes degrees, licenses, and industry-recognized certificates and certifications. 

This inclusive definition reflects growing evidence that many types of credentials can give students 

career opportunities. In addition, Perkins should include employment outcome metrics aligned with 

WIOA for postsecondary students. 

HEA - While this law drives a large portion of postsecondary funding, the current law, last 

reauthorized in 2008, does not require performance reporting to address employment outcomes. ED 

currently publishes a College Scorecard with employment outcome information, using annual data on 

employment from the Department of the Treasury. Legislative language formalizing the ED College 

Scorecard should consider how that information aligns with other federally-mandated scorecards, 

especially training provider reporting required by WIOA. WIOA performance reporting requires 

quarterly metrics based primarily on UI wage records held by states, so the data source is different. 

Furthermore, WIOA focuses on short-term outcomes of less than one year, whereas the College 

Scorecard’s figures show median earnings 10 years after graduation. Having comparable data on 

education and training providers, with interval snapshots showing short-term and long-term 

outcomes, would provide a clearer, broader view for consumers. 

Supplemental Nutrition and Assistance Program Education and Training (SNAP E&T) - As an 

example of agency progress toward alignment, the Department of Agriculture’s Food and Nutrition 

Service (FNS) recently established four national metrics that all states must report for SNAP E&T 

participants, which closely align with core measures in WIOA.8 For example, outcomes metrics must 

 

8

 U.S. Department of Agriculture, Food and Nutrition Service, “Supplemental Nutrition Assistance Program 

(SNAP): Employment and Training Program Monitoring, Oversight and Reporting Measures,” Federal Register, 

March 24, 2016, https://www.federalregister.gov/documents/2016/03/24/2016-06549/supplemental-nutritionassistance-program-snap-employment-and-training-program-monitoring-oversight 

1592 The Promise of Evidence-Based Policymaking
6 

now include the number and percentage of E&T participants and former participants who are in 

unsubsidized employment during the second and fourth quarters after completion of participation in 

E&T; and the median quarterly earnings of all current and former participants who are in 

unsubsidized employment during the second quarter after completion of participation in E&T.9 FNS 

consulted with DOL when it created the new metrics. 

Temporary Assistance for Needy Families (TANF) - WIOA requires that TANF be a partner program in 

the federally-supported One-Stop work center system (if a governor does not opt out), and 

encourages states to include TANF as part of the comprehensive state workforce plans required under 

WIOA. The TANF program currently lacks performance requirements relating to participant outcomes. 

TANF reauthorization is overdue and will present an opportunity for alignment with WIOA. The 

Commission should recommend that states be authorized to at least have the option of negotiating 

performance rates for credential attainment, skills gains, employment, and earnings for work-eligible 

individuals to collect better data on outcomes and align these data with WIOA.10

COMMISSION QUESTION #5: What challenges currently exist in linking state and local data to 

federal data? Are there successful instances where these challenges have been addressed? 

Although having data linked across stages of education, training and employment holds an incredible 

amount of promise for informing education and workforce development, a number of persistent hurdles 

still prevent linkages of state and federal data. Legal barriers and lack of capacity have presented 

challenges for government agencies and institutions to collect, process, and use data in ways that 

make it possible to match data on a systematic basis. Moreover, with an inconsistent application of 

definitions and metrics across laws and programs, such as those discussed above, it will be hard to 

develop a set of common standards that could support an integrated clearinghouse where data could 

be linked. 

Two barriers worth revising through legislative action are the Family Educational Rights and Privacy 

Act (FERPA) and the HEA ban on a federal student record system. 

Family Educational Rights and Privacy Act (FERPA) – (20 U.S.C. 1232g; 34 CFR Part 99) – This law to 

protect student education records should be amended for clarity. Current federal regulations 

elaborating on statutory FERPA language allow education entities to designate an “authorized 

representative” to evaluate education (including job training) programs. Authorized representatives, 

such as workforce agencies, may receive individual-level student data. This regulation is crucial for 

allowing education and workforce data linkages. 

Although state education and workforce agencies are commonly able to collaborate and produce 

longitudinal analysis of education through to workforce outcomes, misinterpretation of FERPA 

 

9

 Loretta Robertson, “SNAP E&T Reporting Measures & Data Sources,” USDA – Food and Nutrition Service, 

webinar slides, September 14, 2016, 

http://www.workforcedqc.org/sites/default/files/files/SNAP%20ET%20Data%20Webinar%20Sept2016.pdf 10 “Temporary Assistance for Needy Families: Reauthorization Recommendations,” National Skills Coalition, 

Washington, DC: July 2016, http://www.nationalskillscoalition.org/resources/publications/file/TemporaryAssistance-for-Needy-Families-Reauthorization-Recommendations_July2016.pdf 

The Promise of Evidence-Based Policymaking 1593
7 

continues to cause roadblocks. Amending FERPA to incorporate the regulatory language into the 

statute itself would help to reassure public officials and facilitate data linkages.

The Higher Education Act (HEA) ban on a federal student record system - (P.L. 110-315, Sec. 113). 

The ban on a federal student record system took effect with the last reauthorization of HEA in 2008. A 

federal student record system used for all postsecondary students would allow for building a more 

complete picture of postsecondary paths, especially for transfer students and working adults who 

attend school part-time. Through linkages with employment data, a federal student record system 

also would allow more accurate aggregate reporting on graduates’ employment and wages. 11

Although the federal government is banned from retaining individual-level data, most of this 

information is already collected and held by a range of agencies and institutions. States have made 

strides in developing similar systems for their students, but with much effort and cost. More than half 

of states have mechanisms in place to show student progress in education and careers, but they lose 

precious information when these students go onto schools or find employment in other states. 

Given that states have faced hurdles in accounting for information on students who leave their state 

borders, and also for those who do not appear in state-held UI wage records (e.g. self-employed and 

military), several initiatives have helped to fill in these gaps. Examples of systems for states to directly 

share wage records include: 

x The Wage Record Interchange System I and II (WRIS/2) allows states to exchange quarterly 

UI wage records through an automated clearinghouse system that directs queries and 

manages flow of data between states. The DOL operates this exchange system through a 

cooperative agreement with Maryland. The system has been challenged by slow response 

times from states in returning queried data. 

Because WRIS/WRIS2 is not a central database, but rather a system that facilitates transfer of 

requested individual-level UI records, state officials must manually gather the wage records to 

respond to queries. This is instructive for the Commission when considering different models 

for a federal clearinghouse. A process that requires individual agencies to manually respond 

to data requests, even if those requests come through a central clearinghouse, may pose a 

capacity challenge and result in delays in data exchange. 

x The Western Interstate Commission for Higher Education’s (WICHE) Multistate 

Longitudinal Data Exchange (MLDE) created a data sharing arrangement between Hawaii, 

Idaho, Oregon, and Washington. Each state contributed to a data set containing information 

on over 190,000 students who graduated from a public high school and/or began at a public 

 

11 HEA: Sec. 113. Database of Student Information Prohibited. Part C of title I (20 U.S.C. 1015) is further amended 

by adding after section 133 (as added by section 112 of this Act) the following: ‘‘Sec. 134. Database of Student 

Information Prohibited. (a) Prohibition.—Except as described in subsection (b), nothing in this Act shall be 

construed to authorize the development, implementation, or maintenance of a Federal database of personally 

identifiable information on individuals receiving assistance under this Act, attending institutions receiving 

assistance under this Act, or otherwise involved in any studies or other collections of data under this Act, 

including a student unit record system, an education bar code system, or any other system that tracks 

individual students over time.” 

1594 The Promise of Evidence-Based Policymaking
8 

postsecondary institution in the state. The data set included information on enrollments, 

postsecondary credential attainment, and wage records so researchers could look at 

employment outcomes and subsequent enrollments of students after completing a program 

of study. About half of graduates had employment records in the same state where they 

earned a credential. MLDE enabled states to find wage records for an additional 9 percent to 

22 percent of students, depending on the state.12 The project is in the process of expanding to 

include over 10 states. 

In addition, there are two notable processes that allow states to access employment data held by 

federal agencies: 

x Federal Employment Data Exchange System (FEDES) - DOL operates an exchange system 

through a cooperative agreement with Maryland that allows states to query relevant wage 

records from the U.S. Department of Defense and Office of Personnel Management files. This 

system has helped states fill in UI wage record gaps for mandated performance reporting, but 

the participating federal agencies prohibit use of this data for broader research purposes. 

x LEHD data pilot – This Census program — which holds UI wage records, Office of Personnel 

Management files, and tax data for self-employed workers — has nationwide coverage except 

for Wyoming. Key data has been covered for over ten years and the collection continues to 

grow. Census recently agreed to conduct a pilot program with the University of Texas (UT) 

System to collect student records from UT and link them with LEHD employment data, to 

produce aggregate employment outcomes. UT already collects employment outcomes 

information about students who remain in Texas. This project is intended to show what 

happens with the students who move out of state, and will provide more information on the 

utility of harnessing LEHD data for larger-scale research projects on postsecondary and 

workforce outcomes. 

COMMISSION QUESTION #7: What data should be included in a potential U.S. government data 

clearinghouse(s)? What are the current legal or administrative barriers to including such data in 

a clearinghouse or linking the data?

The streamlining of information systems through the creation of a national clearinghouse could 

reduce the burden on agencies that have had to submit the same data multiple times, and strengthen 

data security and privacy by using a system with better privacy protection and security protocols. 

A national clearinghouse(s) should include employment data that allows for the calculation of reliable 

and comparable employment outcomes for students across education and workforce programs. It 

 

12 Brian Prescott and Patrick Lane, “Fostering State-to-State Data Exchanges,” Envisioning the National 

Postsecondary Data Infrastructure in the 21st Century, Postsecondary Data Collaborative paper series, Institute 

for Higher Education Policy, Washington, DC: May 2016, 

http://www.ihep.org/sites/default/files/uploads/postsecdata/docs/resources/fostering_state-tostate_data_exchanges.pdf. For more information about WICHE’s MLDE project, visit 

http://www.wiche.edu/longitudinalDataExchange 

The Promise of Evidence-Based Policymaking 1595
9 

also should include information on a wide array of postsecondary students and jobseekers – including 

part-time students, and all students at education and training providers that receive federal aid.

Because of the changing nature of education and career paths, the clearinghouse should include 

information on all meaningful credentials — certificates, licenses, certifications — not only two- or 

four-year degrees. The Commission should recommend overturning the HEA ban on a federal student 

record system and the ban on a federal database of WIOA data, so that we can have a nationwide, 

inclusive data set to show how people are moving through a variety of education pathways. 

A number of the most relevant, currently-available data collections include: 

x LEHD at the U.S. Census Bureau, which includes UI wage records submitted by all states, 

except for Wyoming, Office of Personnel Management files, and tax data for self-employed 

workers. 

x NDNH, under the Department of Health and Human Services, which also contains UI wage 

records submitted by states, employment data for most federal workers, and data that 

includes employees’ dates of hire. 

x Tax records from the Social Security Administration and the Treasury Department - in 

limited instances, agencies have found ways to use these data to show employment 

outcomes on programs. The LEHD program at Census has used records to fill in gaps on 

information for the self-employed. In an example provided during the Commission’s inaugural 

public event, Prof. Raj Chetty shared how he used Treasury data to show correlation between 

social mobility and where people grew up. 

x Student record information - ED’s Integrated Postsecondary Education Data System (IPEDS) 

collects information nationwide by institution, but has limited utility for linking with other 

data sets when students (not institutions) are the unit of analysis.13 Individual-level student 

data is kept at ED’s Federal Student Aid office, but only includes students receiving Title IV 

federal aid and is maintained in antiquated data systems that inhibit meaningful analysis. The 

College Scorecard links this financial aid student data with employment data from Treasury to 

calculate institutional employment outcomes. 

The National Student Clearinghouse (NSC), a non-profit organization, illustrates the potential 

for a federal student record system. It collects individual-level student data on a voluntary 

basis from institutions that use it to facilitate required reporting. The NSC operates under 

contract as an agent of each school to allow for FERPA-compliant data exchanges and has 

received approval to operate from ED’s Federal Student Aid office. NSC is not, however, 

 

13 Jamie Rorison and Mamie Voight, “Putting the Integrated Back into IPEDS: Improving the Integrated 

Postsecondary Data System to Meet Contemporary Data Needs,” Postsecondary Data Collaborative paper 

series, Institute for Higher Education Policy, Washington, DC: May 2016, 

http://www.ihep.org/sites/default/files/uploads/postsecdata/docs/resources/putting_the_integrated_back_int

o_ipeds.pdf 

1596 The Promise of Evidence-Based Policymaking
10 

obligated to meet the same privacy standards as ED or other federal agencies, 14 or a national 

clearinghouse that would be created by the federal government. In addition, NSC primarily 

serves institutions. Its central mission does not include offering transparency about higher 

education. 

Additional data collection would enhance the abovementioned resources, such as data on hours 

worked and occupation. DOL has been running a pilot program to assess the utility of adding hours 

worked and job titles to wage records. If successful, this additional information could fill in knowledge 

gaps in administrative data: whether people are earning at a full-time or part-time rate, and whether 

they are working in occupations related to their program of study. 

Some legal/administrative and technical barriers would need to be overcome to include data in a 

national clearinghouse: 

x Legal /administrative barriers – Several key legal barriers, or clarifications, would be 

necessary before incorporating data into a national system: 

o Lifting the ban on a federal student record system (2008 reauthorization of HEA), 

Higher Education Opportunity Act, P.L. 110-315, Sec. 113 currently prohibits the 

provision of a national database on student information, and would need to be 

overturned or amended to allow for inclusion of such information in a national 

clearinghouse.

o Linkages with employment data in LEHD or NDNH are currently limited. States 

voluntarily provide wage data to LEHD through a data sharing agreement that 

restricts how data may be used. Wyoming recently withdrew from the arrangement, 

illustrating that renegotiating the agreement with states to allow incorporation into a 

federal clearinghouse would be challenging. NDNH data may only be used for 

purposes specifically enumerated in the Social Security Act. 

o Lifting the ban on a national database under WIOA (Pub. L. 113-128, Sec. 501 (b) would 

be necessary to include data on workforce program participants in a national 

clearinghouse. 

x Informational barriers for linking data – linking employment data with student records 

usually requires matching personally identifiable information, such as Social Security 

Numbers (SSNs), and first and last names. Tax and wage records usually SSNs, but many 

student records do not. Some states have taken on the challenge of trying to match student 

 

14 Clare McCann and Amy Laitinen, “The College Blackout: How the Higher Education Lobby Fought to Keep 

Students in the Dark,” New America, Washington, DC: March 2014, 

https://s3.amazonaws.com/www.newamerica.org/downloads/CollegeBlackoutFINAL.pdf 

The Promise of Evidence-Based Policymaking 1597
11 

information to wage records without SSNs, for example, by using department of motor vehicle 

records to serve as a check that improves the probability of matching other data sources.15 

COMMISSION QUESTION #9: What specific administrative or legal barriers currently exist 

for accessing survey and administrative data? 

Legal and administrative barriers that obstruct the inclusion of employment and education data into 

a federal clearinghouse also restrict access to individual-level data by external entities for program 

evaluation and research. These issues include some of those addressed above in question 7. 

Linkages with employment data in LEHD or NDNH are currently limited. States voluntarily provide 

wage data to LEHD through a data sharing agreement that restricts how data may be used. Wyoming 

recently withdrew from the arrangement, illustrating that renegotiating the agreement with states to 

allow incorporation into a federal clearinghouse would be challenging. The original process of 

negotiating the sharing of state wage record data with Census took years, and in some instances 

required legislative changes in states. Substantial staff resources in federal and state government are 

dedicated to re-negotiating those arrangements as they expire in each state – a process which can be 

challenging. NDNH data may only be used for purposes specifically enumerated in the Social Security 

Act. 

State laws on UI confidentiality may make it challenging for states to contribute data to NDNH or 

LEHD if they are accessed for broader purposes through a federal clearinghouse.16 The Center for 

Regional Economic Competitiveness conducted a survey of all states and found variance in their 

interpretation of various federal and state laws applicable to wage record information.17 The variance 

in accessing information for matching data causes hurdles for reporting on employment and wage 

outcomes within states, and between states where certain agreements have been established for 

matching data across borders.18 

It would also be useful to amend FERPA to incorporate current regulatory language that allows an 

“authorized representative” to evaluate education programs, broadly defined as including job 

 

15 National Association of State Workforce Agencies (NASWA), Labor Market Information Subcommittee, “Access 

and Use of Drivers’ License Files by State Labor Market Information Offices; Advancing the Cause,” Washington, 

DC: March 2016. 

16 See Title 20, Section 603, of the U.S. Code of Federal Regulations (20 CFR § 603)—The Federal State 

Unemployment Compensation Program; Confidentiality and Disclosure of State Unemployment Compensation 

Information. State laws vary on how they address the sharing of education and workforce-related information. 17 “Balancing Confidentiality and Access: Sharing Employment and Wage Data for Policy Analysis and Research,” 

Center for Regional Economic Competitiveness and the Labor Market Information Institute, Washington, DC: 

May 2015, http://www.lmiontheweb.org/download/2015-05/Report--_Data_Confidentiality_and_Sharing_-

_CREC-LMI_Institute_-_May_2015.pdf 

18 Ibid. The Center for Regional Economic Competitiveness is currently assessing state laws and regulations that 

govern business revenue and UI wage data disclosure. The project will determine best practices for maintaining 

the safety of data and for expanding access to confidential data for policymakers and researchers. 

1598 The Promise of Evidence-Based Policymaking
12 

training. The “authorized representative” clause is important for allowing education and workforce 

data linkages. 

COMMISSION QUESTION #14: What incentives may best facilitate interagency sharing of 

information to improve programmatic effectiveness and enhance data accuracy and 

comprehensiveness? 

The federal government uses incentives to promote data sharing within states and between states 

and the federal government. These practices could inform further recommendations of the 

Commission, especially on the subject of creating a national clearinghouse. 

Tie greater funding to the provision of data – Within the context of greater funding for the 

development of state longitudinal data systems, the Commission could recommend that funding be 

tied to states providing data to a national clearinghouse. For example, states that apply for DOL’s 

Workforce Data Quality Initiative (WDQI) grants must agree to participate in the WRIS 2 system that 

facilitates wage record matching across states for education and workforce performance reporting. 

This incentive in the existing WDQI grant program has proven effective in motivating some states to 

join, so that 42 states, plus the District of Columbia and Puerto Rico, now participate in WRIS2. 

Invest in training data stewards and users – While tying funding to the use of data for policy, it will 

be important to invest in technical assistance and professional development to ensure that data 

managers know how to collect quality data, follow privacy and security rules, and effectively use data 

to inform policy choices. 

Promote public facing information – Encourage agencies to provide user-friendly information to 

help customers select the training and education programs that best suit their needs. Public and 

political support would be helpful in generating the will and resources for creating a national 

clearinghouse. Support for data collection and system development will strengthen as employers, 

workers, students, and policymakers increasingly benefit from the use of this information and as long 

as highly sensitive information remains confidential and secure. 

Conclusion

We would like to thank the Commissioners for their work on this subject, and the opportunity to 

comment through this written submission. Should you have questions or want to request additional 

information, please contact: 

Rachel Zinn, Director 

rachelz@workforcedqc.org 

202-223-8355 

Workforce Data Quality Campaign –a project of National Skills Coalition 

1730 Rhode Island Ave. NW, Suite 712 

Washington, DC 20036 

www.workforcedqc.org 

The Promise of Evidence-Based Policymaking 1599
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t19-fd1j

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0146

Comment on FR Doc # 2016-22002

Submitter Information

Name: Matthew Hill

Address:

Philadelphia, PA, 19104

Email: mjhill@upenn.edu

General Comment

See attached file(s)

Attachments

AISP-EPBC-161114

1600 The Promise of Evidence-Based Policymaking

 #&

Subject: Request for Comments 

To: Shelly Martinez, Executive Director/ Commission on Evidence-Based 

Policymaking, Department of Commerce 

From: Dennis Culhane, Co-PI; Matthew Hill, Executive Director/ Actionable 

Intelligence for Social Policy, University of Pennsylvania 

Date: November 12, 2016 

'##
&$% $

8.#%#$&$$&#( #$-! $-!#%$-% $%  '# $

#%% '2&# $%%- -1 #%#%  '#%$%

 $$ $ & $#(' !$# % $##

#'2$! *,	$ -!$$#.

,7!-"('%
'-%%" '(+("%(%"2?@1)+-'%+)(+-,+,,%%(-!,

",,.,7.+(.+1)+-)'%,?9
 %
,,.,'+&'-,4:9(/+''4(%"",'

+(.+,4:9-.%"-2'"'"&.&--'+,4:'9!'(%( 2'-

.+"-2:@(',",-(,--'%(%+)+,'--"/,0!(!/."%-',.,,.%%2()+-


'- +--2,-&,?
@'+)+-(-!
-0(+$4,0%%,&"1)+-,

"'-!,+,7!+)(+-,0"%%,+"-!)(%"",')+(.+,'-!- (/+''

,-+- ",-!-0"%%'%,--,'(.'-",-('/" --!)+(,,+*."+-(,.+%2

,-(+4%"'$4'+2)-''%23"'"/".%<%/%&"'",-+-"/-7!)'%,%,()+(/"

+(&&'-"(',(+-!)+(,,,(+/"0"' +,+!+*.,-,(+-4+'-"%"' '

.-!'-"-"' +,+!+,4&$"' -,,"%+&(-%2-!+(. !9-/"0,:0!"%

+,-+"-"' '%2,-,8,,-(-!-.%-4'.-(&-",%(,.++/"0(++,+!

+,.%-,7

9.$ %$%!#%$)$%)!$-(%% #$$ &

 $##$ *$&#%$&#%*!#'* $%#%'$&#'*%,

.+)'%(-!'(%( 2'-,.+"-21)+-,?(&).-+,"'-",-,4,--,2,-&

&"'",-+-(+,4-!'(%( ",-,@0"%%)+()(,"' ,.+))+(!(+,--,'%(%

 (/+'&'-,-(,!+-.,"' ,--(-!+-'+2)-"('4'-(+"' "'(.-,"-,-,

-!-(.%,.+%2%"'$7!++&'21&)%,-!-(.%+)%"-"'-!'

"'.+()0!++,+!+,'/"0'+&(-%2,,+,+!-,-,4+/"0,--",-"%

(.-).-4.-'(-!/,,-(-!-.%-7!,-,-,1",-(+-"&%"&"-)+"("'

,.+,!"('7

The Promise of Evidence-Based Policymaking 1601
4G;'"/7('',2%/'" G

!"%-!++&'2-!'(%( "%))+(!,-(',.+-,.+"-2')+"/24-!++

-!+(-!+(&"',-!-&.,-%,(+,,=% %4)+(.+%4')!2,"%,.+"-2

, .+,7!2,"%,.+"-2',,"&)%,&"'-"'"' %($(",'!+'0(+$

,--"(',-!-&-+%,.+"-2,-'+,7
 %%24&"'",-+-"/-&.,-)+(--

-!+(. !,+",( +&'-,-!-%+%2,-").%--.,',,'!(0-0"%%

!'%"'(&)%"'0"-!+%',--%0,7+(.+%, .+,,!(.%%,(

(.,4"'%."' +/"0(&&"--,(+- (/+''(+-!-))+(/,.,',,4

,-,+(.-"'-."-,4()+-,(+"' -(-+',)+'-.,"',,)+(.+,4,0%%,

%+%2"')+(,,,')+(.+,-!-++-'+/",-!+(. !(%%(+-"/

)+(,,()+-"")-"' ,-$!(%+,7

%	#$%#&%&#$$

:.$ %$%!#%$)$%)!$- ($ &)$% '#%

%#$%#&%&# % $%%%&$ $$% $%#%'

$&#'*%,

--!,--'%(%%/%4,--,,!(.%"''-"/"3-(,-(+-4&"'-"'-"%

((($,4'+-+(+%"'$ ,2,-&,-!-)+&"-.-!(+"3.,+,-(,,-!,-

"',.+'/"+('&'-7'"'  ."'+(&-!77)+-&'-(.-"('

?@,!(0,!(0.-"('-',-(+"',.!'
7F

"&"%+%24-!77)+-&'-

(	(.,"' '+'/%()&'-?	@!,.-!(+"3"-,"((%"2/%()&'-'

,+!?A@-('-+"'-(-
"',+&'-0"-!+,+!(+ '"3-"(',?"'%."' 


'- +--2,-&,@0!')(%"2<+%/'-+,+!*.,-"(','('%2',0+2

.,"' )+,('%%2"'-""%"'(+&-"('"'-!)(,,,,"('(	7G

+(&&'-!-,--,

 -,"&"%+ ."'+ +"' )+-&'-(
(+?
@0 +(+,'%%<2+	%-!

%"&,-4-0((-!&(,-"&)(+-'--,-,-!-+.++'-%2,,2)(%"2'%2,-,

'+,+!+,4.-('%2"'!'.%(,--,7--,+,$"' %+ ."'+(&+%

 '",-!-(.-%"',-!.-!(+"-2')+(.+,20!"!
'%%<2+-'

,-(+,.+%2'%"'$0"-!'+2)-"'-""+,4''('2&"3(+)+&",,"%

/%.-"('4."-')%''"' ).+)(,,7

;.%%2$##$%#&%&#$ &&$% %%%#--

$$ ##$#-'&% -*$$!&#! $$,

--!+%%/%40'(.+ -!/%()&'-('"'+,-+.-.+-!-0(.%'%

 (/+'&'- '",'+,+!+,-(,.))%2-,-,"',-'+(+&-40"-!-!

())(+-.'"-2-(,,'%"'$0"-!'-"('%+'"' ,+(+,'(-!++%,.+/2-7-

-!,--'%(%%/%40)+()(,-!/%()&'-(6?F@,-'+"3-"'+,-+.-.+



F "!%	0,4+(--"' -.'-+"/2!"%+-"' ',"' 
'- +--2,-&,6*."+&'-,

',-+-",7/%()"' "-,('+'7'"/+,"-2('',2%/'"77H4GEFI7

G !--),6;;0007!..,+7 (/;)(+-%;+,+!;)+>-<%"',7!-&%7

1602 The Promise of Evidence-Based Policymaking
4G;'"/7('',2%/'" H

&(%-!-0"%%%%(0 (/+'&'-,-(,.+%2%"'$4,-(+''%23'+2)-"'"/".%<%/%

&"'",-+-"/-4'0!"!0"%%,.'"(+&,)(,,"%+(,,,--,'(.'-",,

('+%%2&'-&"'"&.&-%&'-,',.+"-2,-'+,5'?G@,!+

/%.-"('-+',-"(',)%-(+&-!-0"%%'%#.+","-"(','/%.-(+,-()+()(,'

' (-"-+,+!)+(#-,4," '-., +&'-,'"''"%('-+-,4+.''%2,,'

(&&.'"-+,.%-,4'-+$)+(#-,4-(,-<""'-',%%%/%4+*."+"' &"'"&%

.+''!.&'+,(.+,&',+(&%+2(/+.+' (/+'&'- '",7

--,'%(% (/+'&'-,,!(.%'8-!/-(+"'/'--!0!%(++)%"--!)+(.+%

))+-.,',,+2-(+-,.!-"'+,-+.-.+'-+',-"(')%-(+&40!"!",0!2

0,. ,--!+-"('(%(0<(,-4,%%
(+,--,'(.'-",4',2,-&<0"

 (/+''(+(&)+",(,--'%(%(""%,-()+(/"(/+," !--(-!(+-7

<.%$&##%*)$%$%% %% #%,#%#

$&$$&$%$(#%$$'#$$,

.,,.%1&)%,(%"'$"' ,--'%(%--(+%-"'%.	8,-
"',

+&'-)+(,,0!"!%%(0,,--,-(,-(+	-''--(+)+<))+(/).+)(,,

'/%.-"(',?,H@7"-"('%%24-!8,)'"'  ."',!(0,!(0

.-"('-',-(+"''
?,H@7'(.+ ,"&"%+,(%.-"('(+-!%%

2+%"&,-,,?,@,0%%,,--%(+'0 -4(+'&)%(2&'-


',.+'?
@+(+,7

=. &$ #&%!# &$3$4 #$%#%'$&#'*%

$%$% !# ''2$! *,%%$ #%% $#*

%  &%#%#!!# ,

-!"'$,"' %4,!+%+"' !(.,(+,--'%(% (/+'&'-,",/",%?(+-!--

%,-(',-+-,,((',)(,,"%@4'-!-"-,!(.% (/+'2(&&"--(+

(+(#.+","-"('%+)+,'--"/,4+,+!+,4'(&).-+,"'-",-,7.!'-"('%

"'+,-+.-.+0(.%'8-+)%,--'(.'-2<%/%
"',-%%-"(',4.-0(.%%%(0-!,

%(%#.+","-"(',-(-)"'-(,!+,-(,-'+"3,.+"-2&!'",&,4()',(.+

-((%,4'%2-"))+(!,4',!+-+',-"(',)+(,,,2,-&-!-0(.%)+(/"

.-(&-)+(#-&' &'-'-+$"' 7!"%,--'%(% (/+'&'-,'-(

+-"'.-!(+"-2(/+-!"+(0'-4))+(/%()+(#-,4'+/"0'+%,(+,.%-,4

-!2,!(.%(+-!())(+-.'"-2-()+-"")-"',.!,!+,2,-&'-+',-"('

)%-(+&4-(+.-!"+(,-,'!.&'+,(.+,.+',7

>.%%$ &&! %%.. '#%%# &$3$4,%

#%&##% #$%#%'###$% &$&%# &$ #

%%,

-!"'$-!-&"'"&.&'.&+(-(&"','!" !*.%"-2-,(.+,,!(.%

"'%."'+% (/+'&'--%+"' !(.,7!,(&"',"'%.6+'"' +(+,4

,("%,.+"-2","%"-24+-"+&'-'&(+-%"-2-4""'"+%"&,+(+,4

The Promise of Evidence-Based Policymaking 1603
4G;'"/7('',2%/'" I

!(.,"' ,,",-'+")"'--4'+%)+",('&",,"(''",!+ +(+,7"-!"'

!(-!,(&"',-!++/+"%,-!-+."-+ .%+%2(++)(+-"' 4()+-"' 4

'"%%"' ).+)(,,7 .%++%"'('-!.+2(-!,-&',-!--!2+(

!" !+*.%"-2-!'(-!+&"'",-+-"/--!-+'(-(&&('%2.,(+,.!).+)(,,7

?.%% #$ #$%#%$$ &% $$  $# # (# &$3$4

 &$2&,%$&$$&)!$)$% #$2#%% $#

!&#! $$,

-!"'$-!--!%+"' !(.,(.%,%<.'-!+(. !('(-!(%%(0"' -!+

,'+"(,6?F@/+"%)+""' (+9)2)+.,:&(%4"'0!"!+,+!+,0(.%!+ 

(+!-+*.,-&-!+(. !-!%+"' !(.,7!(,-(,.!+*.,-,&" !-

/+2)'"' "-!+,+!,(. !--(%"'$-!"+(0',.+/2-.)0"-! (/+'&'--4

(+%-+'-"/%24"-!2,"&)%2,(. !--(,,1",-"'  (/+'&'--7?G@'''.%%"',

&(%4"'0!"!+,+!+,(+-!"+"',-"-.-"('; '20(.%)2,.,+")-"('(+

.'%"&"--+*.,-,&-!+(. !-!%+"' !(.,(/+,-)+"((-"&7?H@!2+"

&(%"'0!"!+,+!+,(+-!"+"',-"-.-"('; '20(.%)2,.,+")-"('4'-!'

'"-"('%)'"' ('-!'.&+(-+*.,-,&"',-)+"((-"&7

.,,.%&"'",-+-"/+,+!'-+,"'.+()''()+-('-!,-2),(

)+""' &(%,7

@.%$!$%#%' ####$&##%*)$% #$$$&#'*

$%#%'%,

$'(0('(&#(+% %++"+,-(,,"' ,.+/2'&"'",-+-"/-7!+%

%0,?'&(,-,--%0,@ (/+'"' &(,-!%-!4.-"('4!.&',+/"4!(.,"' '

+'"' ,+(+,4%%(0(+-!,!+"' (&"'",-+-"/-(+/+"(.,'%2-").+)(,,4

"'%."' ."-4/%.-"('4)%''"' '+,+!7(+"',-'4	
 ."%"',%%(0(+-!

.,(+(--	%-!
'(+&-"('?	
@(++,+!4-,-"' '/%.-"(',(%(' ,-!

+,+!('-+".-,-(9 '+%"3%$'(0% 4:-!+,+!+,!/ "'-!+*."+


))+(/%4'-!+,+!",++"(.-0"-!9%"&"--,-:-!-('-"','("+-

)+,('%"'-""+,7H

"&"%+%24-!)'"'  ."')+&"-,,!((%,-(%"'$-!"+

&"'",-+-"/-"''
,(%(' ,-!%"'$ ",(+% "-"&-.-"('%).+)(,4'

-!+,+!('.-",(+-!,!((%",-+"-'"'-'-("&)+(/-!*.%"-2(

"',-+.-"('7I

"&"%++,+!1)-"(',1",-(++%%(!(%'.,-'.,



H 
%"'-+,4-"('%(+.&('(.-!"(%'+/'-"('6	
+"/2.%(',"+-"(',7

/%()"' "-,('+'4(/7F4GEFF7%,(+*."+,-!--., +&'-," '-0'-!

+,+!+'-!-)+(/"+7

I "!%	0,4+(--"' -.'-+"/2!"%+-"' ',"' 
'- +--2,-&,6*."+&'-,

',-+-",7/%()"' "-,('+'7'"/+,"-2('',2%/'"77H4GEFI7

1604 The Promise of Evidence-Based Policymaking
4G;'"/7('',2%/'" J

+-&'-(+,?IG+-G@4-!	(&%,,' &'-
'(+&-"('2,-&?	
@4

'-!+"/2-?JJGG?@?L@@7J

87. ($ &% $$ /"&#$##$$%%&% $,0 (%

)%%$ &$%#%'$&#'*%* '#%$

'% /"&#$##$$%%&% $,0

!(&&",,"(',!(.%"'9*.%""+,+!+,'"',-"-.-"(',:,&"4'('<)+("-

(+ (/+'&'-+,+!+,(+-'%2,-,0"-! +.-?!;@-+"'"' "'-!,("%

,"',0!(!/
-+"'"' "'-!)+(--"('(!.&',.#-,'-!"%"-2-(',.+-!

,"'-"""'- +"-2(-!"++,+!2,.&"--"' "--()++/"0)+(,,7&"'",-+-"/

',.+/2-,!(.%&/"%%-(,.!+,+!+,"'"'-""(+ + -

(+&4-+-+*.,-)+()(,%!,"+,-'+/"02 (/+'"' (+(+"-,&+"-

',""%"-24,0%%,"-0"-!-!)+"(+"-",()+-".%+ (/+'&'- '27K

88. (%%#%  $%#%'$&#'*%# &$%%#$

 &%%  #&&% #+$$ ##$ !#$ *2% #% -

 %&$$$ #% - # %#%# #$, (%

 #% $%!# %%% $&#%!#'* %%* '& #

&$$$%# &$,

!"'- +-"('(-!-"'%+"' !(.,,!(.%-$)%(',.+,+/+(+"'

,.++--'/"+('&'-4'+,+!+,,!(.%)+,'-,,-(-!--

('%2"'"'-""';(+ + -(+&,(-!--!2'/+!/"+-,,-("'"/".%

+(+,(+"'-""-7,+!+,,!(.%.,-!%+"' !(.,-(,'*.+",-(-!

-4+&(-%2'%23"-4' -+,.%-,4-+"+,-"' .-!'-"-'+-""7,,

,!(.%"+'-"-4,.!-!-'()+,('(.%/+,"'-""4"'"/".%%/%+(+,4

',.!-!-(+ '"3-"('%(+.,"',,"'(+&-"('(.%,'"'"'-""&''+4

'$)-)+"/-0!"%/"0-+(+%/%7

89.	# &$(##%-(%%*!$ #$%#% $$ &! %&$$ 

%%# &$*/"&#$##$$%%&% $,0

"+,-4+,+!)+()(,%,.,"' )+(--"'"/".%+(+,,!(.%+/"0'))+(/

2- (/+'"' (+(',",-"' (+)+,'--"/,+(&-! (/+'&'- '",0"-!-!

% %.-!(+"-2(/+-!-7'%2)+(#-,-!-&-(-!-!!.&',.#-,+*."+&'-,

(-!+%/'-
4'-!-.,-("%+,)(',""%"-",(-! '",,!(.%))+(/

(++,+!7,+!-,-,,!(.%+-)+-!+,+!+*.,--!-+,('

'+2)-"'-""+,7,+!-,-,',-,"(++&(-,,-!+(. !,.+



J (!'-+"%4
 %
,,.,"'-!,(%-+('"-2,-&,(+("%"',+!4
'(!''-.33(

'''",.%!'?,7@7-"('%
'-%%" '6,"' 
'- +--2,-&,-(!"/(+-"/4

""'-'-!'"%(/+'&'-4GEFK6))7HN<LK7

K ?GEFJ@4(/+''+(,,4'"/+,"-2('',2%/'"7

The Promise of Evidence-Based Policymaking 1605
4G;'"/7('',2%/'" K

)(+-%(+-"&<%"&"-)+"(7,+!+,,!(.%+'-"%'.-!'-"-"'

(++-((-"',,7,+!+,,!(.%'(-)+&"---(!/+(+%/%/"0,(

"'"/".%--!-!,'(-'(-!+0",.%%2'('2&"37(+,-!-+,',"-"/4,.!

,.,"',,+(+,4.-+'(-,,',"-"/(+)+(--,

4&2/"0%--!+(+

%/%7	(0/+4%%+,.%-,,!(.% (-!+(. !",%(,.++/"0)+(,,(++%,-(

,,.+-!-'("'-""%"'(+&-"('",+/%7

8:.%%  $ &% $#  '#% #%!#'%$% ###'% #

%%%$#%,

/+%,--,'(.'-","'-!-!-%(' -(-!'-0(+$4%(' 0"-!,"-,"'

''.+()4!//%()
,2,-&+!"--.+,-!-)+&"--!,.+

-+',&",,"('(-4+%"%+(+&-!"' -!'"*.,4'-!'('2&"3"' ''+2)-"('

(--(+.-!+",$(+"'-""-"('7
'"-"('4-!,,2,-&,)+(/"+,+!+,

0"-!"-!++,-+"-('<,"-,,(+,.++&(-,,-(-/"0,?7 7,--",-"%

-%,@0"-!&"'"&.&%%,"34+."' -!+",$(+",%(,.+4'',.+"' -!--,,

",)+&"--('%2(+.-!'-"-.,+,7L

%%+%,- (-!+(. !",%(,.++/"0

)+(,,,0%%7/+"-2(-!'(%( 2,(%.-"(',+&"'-!,,2,-&,4"'%."' 

,.+'+2)-"(''+(+-+',&",,"('4%"'$ (!,!+(+,4'('2&"3-"('(-4

,2,-&-")+-.+-"('(-?"+'-"%)+"/2@4.-(&-",%(,.++/"04'(-!+

&-!(,7(&&" !-'"' "/',(%.-"('4.-'(-',,+"%2%%7

8;.%%'$*$%%%%#*$#  #% % !# '

!# #%%'$$%&#* !#$'$$,

+%4,--'%(% (/+'&'- '",,!(.%.%%2"''-"/"3-(.,-!"+)+( +&

&"'",-+-"/.',-()2(+/%.-"('('&(+(',",-'-'0",)+,",4.,"' 

&"'",-+-"/-7&"'",-+-"/-)+(/"'"&)(+-'-4/%.%'.'+.-"%"3

+,(.+4-!-+/"%%-%(0-('((,--(-!, '",?(,-,(+-,2,-&,+

%+2&"'",-+-"/%2.'+0+"--'"'&(,-,,@7+% '2'(.+ &'-(+/'

+*."+&'--!-&"'",-+-"/.',.,-(,.))(+-+" (+(.,)+( +&/%.-"('0(.%

)(-'-"%%2-+',(+&-"/7)"%+%)+( +&"'"-"-"/,(+&(',-+-"(',,!(.%

+*."+,)+-(,.,,.%))%"-"('-!-,--,(+%(% (/+'&'-,"'%.-!.,(

%"'$&"'",-+-"/-(+-!/%.-"('4"'%."' /'+*."+&'--!-+%.',

.,-(,.))(+--!/%()&'-('('< ("' "'+,-+.-.+(+-!,-(+ '%"'$ 

(&"'",-+-"/-(+-!,).+)(,,7+%+,+! '",?
	4
4
4	4

		'-"-",@,!(.%+-,)"")+( +&,(+)+"(+"-",(+-!.'"' (/%.-"(''

+,+!&(',-+-"(')+(#-,-!-.,%"'$&"'",-+-"/-(+-!-,.))(+--!



L (+"',-'4+"'(',-%?GEFI@7,,-.2(-!.+'('2&(.,
'(+&-"('
"'$ ?
@

-026)+"/2<)+(--"' +&(-,,,2,-&(+!%-!<+%-+,+!'/%.-"('7(.+'%(

"(&"%
'(+&-"('5'+,!'-.&+?GEFK@4'/+/"0(+!"--.+,'!'"*.,(+
'- +-

-2,-&,
&)%&'--"('4
'(!''-.33('''",.%!'?,7@7-"('%
'-%%" '6,"' 


'- +--2,-&,-(!"/(+-"/4""'-'-!'"%(/+'&'-4))7FEJ<FGI7

1606 The Promise of Evidence-Based Policymaking
4G;'"/7('',2%/'" L

+-"('('('< ("' "'+,-+.-.+"',.))(+-(-!",0(+$7.!"''-"/"3-"('0(.%%

-('-"('%%2,%"' .)("'- +--,2,-&,+%-"/%2*."$%2+(,,-!(.'-+27

"&.%-'(.,%24 '",,!(.%.+-!+'(.+ -(,,,,-!+%""%"-24/%""-24

.+24(&)%-',,'(/+ (!-%&'-"'-!"+)(,,,,"('4'-(,(+

-!&(+-!,"'-"""'- +"-27
''-"/,,!(.%)+(/"(+ '",-(,2,-&-"%%2

"&)+(/-!,"'-"""'- +"-2(-%&'-,4'-(+-'0,2,-&,(+%"%-

(%%-"(',',,+27

%$# #$-
%-$#-'&% -*$$

8<.%###$&##%*)$% #&$$&#'*$%#%'%% $&!! #%!# #

%1 #'&% %'%$,

0",)+?&",@)+)-"('1",-,-!-&"'",-+-"/-+'(-,,"%-(1-+'%

/%.-(+,(+(-!+ '",.-(% %++"+,4,.!,4-!+"/2-(+	
7


"&"-"'+,-+.-.+(+)+(,,"' +*.,-,%,(&',-!-+*.,-,%"' +(+%(' )+"(,

'+(-''/++/"0(+))+(/7+,(.- '21)(,.+(+)((+(+

"'(&)%--(-'%-(&.%-")%)+(.+%++"+,"' &(.'--(' "',-

"'*."+",7+,(.-' -"//%.-"('(.-(&,+-%"&-(0+"',,(.-

1-+'%/%.-(+,0!(+.'$'(0'40!(&2'(-!/-!"%%2(+0"-!,!+

)+-'+,!")-!(,+ +"' -!0(+$7%$()+,(''%1",--(&' -!,"-"/"-",

(+/%.-"('4"'%."' )+(,,"' +*.,-,4%"'$"' -!+(+,4",,."'  +&'-,4('-+-,

(+)+(.+&'-,4,',,+24'$)"' -+$()+(#-,'-!"++/"07

8=. (%-$%%$%$-#$&%$ #$#-$# '&% -$%&$% 

!# '! $!# #$,

"+,-4-4,--",-",4'+,+!+,.%-, '+-+(&,+")-"/)"&"(%( "%,-.2(

!" !<)+"(+"-2,("%)+(%&'.,-()+(/")().%-"('<,.'+,-'"' (

)+-".%+,("%)+(%&?7 741-'-(-!)+(%&4",-+".-"('+(,,$2)().%-"('

 +(.),4,("-%(,-,,,("-0"-!-!)+(%&4-7@7!",",-+.(+",,.,%"$+"&"'%

#.,-"4.-"('%!"/&'- ),4!" !(,-!%-!+.,+,4!"%0%+(.-(&,4

!(&%,,',,4(+()"-"-"('&(' (-!+,7+"-"/'%2-",'-!'.,-(

"'-"2-(+,',.)().%-"(',,,("-0"-!)(,"-"/(.-(&,(+-!(,-2

-!,)+(%&,4'-((')-.%"3-!(+",(!' + +"' -"/"'-+/'-"('

,-+- ",-!-'/%()7"'%%24-!"'-+/'-"(','-,-(+-!"+-"/',,

(+-!)().%-"(',-2-!)+(%&7!,'-$-!(+&(+'(&"3('-+(%

-+"%,-!-"'/(%/&.%-")%%(0<(,-"'/,-" -"(',-("'(.-0!-0(+$,,-(+0!(&4*.,"<

1)+"&'-,4(+(-!+/"'<."%"' (+-,7M



M (+&(+('-!",-!+<)+-))+(!,(!''-.33(-%?GEFL@7!
'- +--2,-&))+(!(+

(/+'&'-6+(."' -"('%
'-%%" '-!+(. !'' ("' +(%&<(%/"' +(,,7(+-!(&"' "'-!

(.+'%((%"2'%2,",'' &'-7

The Promise of Evidence-Based Policymaking 1607
4G;'"/7('',2%/'" M

8>. (%)%% #$ &!# #! *'&% #$$!# #

$$,

-!"'$-!-)+( +&';(+)(%"2/%.-"(',,!(.%"'%."''2)+( +&," '4

'-!--!2+$2-()+(,,(('-"'.(.,)+( +&"&)+(/&'--!-.,,-4

/"'4'/%.-"('-("&)+(/(.-(&,7(+(/+40/"0-!/%()&'-(

&"'",-+-"/-,2,-&,,$2-()+(."' !" !<,)4%(0<(,-/%.-"(',4,.!,

+'(&"3('-+(%-+"%,?,@-(-,-,("%)+( +&"''(/-"(',7N

!+,.%-,(,.!

/%.-"(','.,-(&("2'"&)+(/)+( +&," '-!+(. !('-"'.(.,)+(,,

(+(.-"'"3(+ '"3-"('%"'*."+24';(+-!+(. !)%''4%+ +,%,("%1)+"&'-,7

+( +&.'"' ,!(.%"'%.&"'",-+-"/,-!-,.))(+-+" (+(.,/%.-"(''.,(

"'- +-&"'",-+-"/-7

8?. ( #$ &!# #'&%  #! #%% !# #$$,%

$!)!$ $%#%(#'&% $$&$$&* #! #%

!# #$$,


' '+%49)2(+,.,,:(+-,!/'-"/"'+*."+"' +" (+(.,/%.-"('," ',

,)+-(-!"+/%()&'-'"&)%&'--"('7!.,()+"/-.',')+(+&'

'!&+$,-!--+" +)2&'-!,).-&#(+(.,('-!&,.+&'-'/%.-"('(

-!,(+-,7!+% (/+'&'-,!(.%,"&"%+%2+*."++" (+(.,/%.-"(','

)+(+&''!&+$,-(&,.++%"%2'('-"'.(.,%2-(,,.+)+( +&,+

&-"' -!"+(#-"/,4"'%."' (#-"/,?(+/',('+2'"-,@-!-&2

1)+"'-!+(. !(-!+ '2)+( +&,4'0!"!-!+(++*."++(.-"'+(+

%"'$ +(,, '","'(++-(&,.+72(+,.,,(+-,!/,!(0' +-

0"%%"' ',,-(,,,,-!,&.%-" '2"&)-,4'!/(+ '",-(,!+-"'

(++-(.'+,-''&,.+-!&7+( +&/%.-"('+*."+&'-,(+," '""'-

"''-"/,--!-(+%)+( +&,)'"' (.%'(.+ '0 '+-"('(

"''(/-"(''-,-"' (,("%)+( +&",4."%-.)('-!&.%-" '2/"0,)+(/"2

"'- +-&"'",-+-"/-,2,-&,7

8@. (%)%%$ &'&% $$!*(%%#)!#%3$ %$

###% $/# + %# %#$04 #"&$2)!#%$$$%%&% +

!# #$,%$!)!$ $%#%(#$&$%%&% +% $

$&$$&(%$%!#%$)$% # $ ,

-!"'$-!-/%.-"(',0"-!1)+"&'-%," ',?(++'(&"3('-+(%-+"%,(+,@

,!(.%"',-"-.-"('%"3"')+( +&,4'(&"'0"-!&"'",-+-"/-,2,-&,7!

.,("'- +-&"'",-+-"/-"',%%(0,+,+!-("'%.&(+"'(+&-"('('



N (+&(+('-!.,()+( +&/%.-"('-("&)+(/)+( +&(.-(&,')(%"2"''(/-"('4,6

!--)6;;0007",)7.)''7.;- ;)+( +&</%.-"(';7

1608 The Promise of Evidence-Based Policymaking
4G;'"/7('',2%/'" N

-!"'"/".%,"'%."'-!-+"%?(+"'&-!(&)+",(' +(.),(')+<1",-"' 

!+-+",-",(+(/+"-,@7.+-!+&(+4-!.,(%+2<(%%-&"'",-+-"/-

 +-%2+.,-!(,-,,,("-0"-!-!,,(+-,(+" (+(.,/%.-"(',4,)"%%20!'

,,-!+(. !"'- +--,2,-&,0"-!0%%<,-%",!4&.%-"<).+)(,

"'+,-+.-.+7,.,"' -!",-2)(-+.-!(,-()+( +&/%.-"('0!"%,-"%%

&"'-"'"' +" (+(.,,-'+,4&$"' "-)(,,"%(+ (/+'&'-(""%,-(.,,"'-""

/"'(.-90!-0(+$,:-("'+, (/+'&'--"/',,7!++'.&+(.,

1&)%,0!+,.!"',-"-.-"('%"3-"('(,!,',.,,.%"'"&)+(/"' /+"-2(

 (/+'&'-)+( +&,4"'%."' +"&"'%#.,-")+(-"('4!"%0%+,&' &'-

,+/",4)+'-"' "'-+/'-"(',4'0(+$(+/%()&'-)+( +&,&(' (-!+,7FE

''",7.%!'

'''+0-('+(,,(+(("%

(%"2

'"/+,"-2('',2%/'";G

HLEF
(.,-%$

!"%%)!"4FNFEI<KGFI

.%!'B.)''7.

--!07	"%%4!77

1.-"/"+-(+

-"('%
'-%%" '(+("%(%"2

G;'"/+,"-2('',2%/'"

HLEF
(.,-%$

!"%%)!"4FNFEI

&#!"%%B.)''7.

KFE<GEH<GMJE



FE (%"-"('(+/"'<,(%"2?GEFG@4	(0
(0<(,-'(&"3('-+(%+"%,+(,,"%"''2

+,(("%(%"25/"(+',-"'?GEFI@4
'-!
(' +('(/+-24&%%"-(+",!---+40(+$

"&,4'.+2M5/"
('!+-?GEFI@4!."-(/&'--($(/+'&'-"%
,,-'40(+$

"&,4.%2FJ7

The Promise of Evidence-Based Policymaking 1609
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t19-okqe

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0147

Comment on FR Doc # 2016-22002

Submitter Information

Name: Amanda Janice Roberson

Address:

Washinton, DC, 20006

Email: ajroberson@ihep.org

General Comment

Chairperson Katharine Abraham, University of Maryland

Co-Chair Ron Haskins, Brookings Institution

Commission on Evidence-Based Policymaking

Docket ID USBC-2016-0003

Dear Chairperson Abraham, Co-Chair Haskins, and Commissioners,

As members of the Postsecondary Data Collaborative (PostsecData), we appreciate the

opportunity to provide comments to the Commission on Evidence-Based Policymaking.

We seek to assist the Commission in achieving its statutory objectives to develop a

strategy for increasing the availability and use of data in order to build evidence about

government programs, while protecting privacy and confidentiality. The PostsecData

Collaborative is a coalition of organizations that advocates for the use of high-quality

postsecondary data to inform policymaking, institutional improvement, and consumer

choicewith the broader goal of improving outcomes for all students, while closing equity

gaps. Our group includes participants from a large swath of the higher education

community, including experts and organizations that represent students, states,

1610 The Promise of Evidence-Based Policymaking
researchers, advocates, the business community, and institutions and systems of higher

education. Through the Collaborative, we seek consensus among our diverse set of

organizations on key postsecondary data issues, particularly those that impact students. 

This letter intends to answer a subset of the questions posed by the Commission in its

request for comments, as well as address outstanding questions from the Public Hearing

on October 21, 2016. Some of our recommendations for Commission action apply to

multiple questions and are marked accordingly. The undersigned organizations

recommend the following: 

1. Promote best practices in privacy and security for interconnected data systems. (Q: 2) 

2. Leverage existing data to decrease burden, streamline reporting, and answer critical

stakeholder questions. (Q: 1, 7)

3. Make recommendations that address the administrative and legal barriers to data

linking and access. (Q: 5, 9)

4. Expand access to wage and labor market information for postsecondary outcomes. (Q:

1, 5)

5. Align definitions and metrics across federal laws. (Q: 3)

6. Recommend that Congress overturn the ban on a federal student-level data system. (Q:

9)

PostsecData partners agree that the federal government plays a key role in collecting and

reporting consistent, comprehensive postsecondary data in ways that promote student

success, especially considering the federal government's significant annual investment in

student financial aid. However, the current disconnected, duplicative, and incomplete

data systems do not allow for a cohesive postsecondary data ecosystem, where the data

that are collected can be used for policymaking and consumer information purposes.

PostsecData supports the Commission in its endeavor to streamline federal data

collections and data systems.

Please find attached our complete comments. 

Thank you for your focus on improving the structure and utility of the national data

infrastructure. If you have questions or would like to discuss these issues further, please

contact Mamie Voight, Vice President of Policy Research at the Institute for Higher

Education Policy (mvoight@ihep.org, 202-587-4967). To learn more about the

Postsecondary Data Collaborative, visit our website at www.ihep.org/postsecdata.

Sincerely,

Advance CTE

Association for Career & Technical Education

Association of Public and Land-grant Universities

Data Quality Campaign

Institute for Higher Education Policy

NASPA - Student Affairs Administrators in Higher Education

New America

The Education Trust

The Promise of Evidence-Based Policymaking 1611
The Institute for College Access & Success

Workforce Data Quality Campaign

Young Invincibles

Attachments

PostsecData CEP Comments 11.14.16 (1)

1612 The Promise of Evidence-Based Policymaking
1 

Chairperson Katharine Abraham, University of Maryland 

Co-Chair Ron Haskins, Brookings Institution 

Commission on Evidence-Based Policymaking 

Docket ID USBC-2016-0003 

Dear Chairperson Abraham, Co-Chair Haskins, and Commissioners, 

As members of the Postsecondary Data Collaborative (PostsecData), we appreciate the opportunity to 

provide comments to the Commission on Evidence-Based Policymaking. We seek to assist the Commission 

in achieving its statutory objectives to develop a strategy for increasing the availability and use of data in 

order to build evidence about government programs, while protecting privacy and confidentiality. The 

PostsecData Collaborative is a coalition of organizations that advocates for the use of high-quality 

postsecondary data to inform policymaking, institutional improvement, and consumer choice—with the 

broader goal of improving outcomes for all students, while closing equity gaps. Our group includes 

participants from a large swath of the higher education community, including experts and organizations 

that represent students, states, researchers, advocates, the business community, and institutions and 

systems of higher education. Through the Collaborative, we seek consensus among our diverse set of 

organizations on key postsecondary data issues, particularly those that impact students. 

This letter intends to answer a subset of the questions posed by the Commission in its request for 

comments, as well as address outstanding questions from the Public Hearing on October 21, 2016. Some 

of our recommendations for Commission action apply to multiple questions and are marked accordingly. 

The undersigned organizations recommend the following: 

1. Promote best practices in privacy and security for interconnected data systems. (Q: 2)1

2. Leverage existing data to decrease burden, streamline reporting, and answer critical stakeholder 

questions. (Q: 1, 7) 

3. Make recommendations that address the administrative and legal barriers to data linking and 

access. (Q: 5, 9) 

4. Expand access to wage and labor market information for postsecondary outcomes. (Q: 1, 5) 

5. Align definitions and metrics across federal laws. (Q: 3) 

6. Recommend that Congress overturn the ban on a federal student-level data system. (Q: 9) 

PostsecData partners agree that the federal government plays a key role in collecting and reporting 

consistent, comprehensive postsecondary data in ways that promote student success, especially 

considering the federal government’s significant annual investment in student financial aid. However, the 

current disconnected, duplicative, and incomplete data systems do not allow for a cohesive 

postsecondary data ecosystem, where the data that are collected can be used for policymaking and 

consumer information purposes. PostsecData supports the Commission in its endeavor to streamline 

federal data collections and data systems. 

1. Promote best practices in privacy and security for interconnected data systems. (Q: 2)

In its recommendations regarding data linkages, the Commission must address the importance of privacy, 

security, and confidentiality; all conversations about data require a critical examination of data privacy 

and security protocols and inclusion of practices vetted by data security experts. For postsecondary data 

in particular, the U.S. Department of Education’s National Center of Education Statistics (NCES) has 

established an excellent reputation for maintaining data security, as well as protecting and de-identifying 

student data for academic and policy research. Additionally, the Commission should review and 

The Promise of Evidence-Based Policymaking 1613
2 

incorporate federal standards under the Federal Information Security Management Act (FISMA) and the 

Family Educational Rights and Privacy Act (FERPA), as well as those set by the National Institute of 

Standards and Technology (NIST), as they represent the most robust standards for protecting data. It is 

critical that the Commission construct de-identification protocols for the elements included in an 

education record and standards to determine who has access to the data. 

Generally, policies recommended by the Commission should outline the importance of data linkages and 

transparency, while emphasizing the need to utilize best practices in privacy and security. The Commission 

should look to state system examples, like the University of Texas System Dashboard2 and California 

Community Colleges’ Launchboard,3 where these postsecondary and workforce data are integrated, with 

different users having varied access to the data. This user-based, tiered approach to data access secures 

the data, protects the privacy of the students included in the data set, and allows stakeholders to access 

aggregate data to use for decision-making and policymaking purposes. 

The Commission must frame all of its recommendations in ways that promote responsible data use, with 

clear consequences for failure to do so. All publicly available data should be presented in aggregate form, 

stripped of information that could be used to identify specific individuals or employers. Administrative 

data should not be sold or used for any purposes – commercial or non-commercial – outside of their 

intended use. Willful disclosure and inappropriate use of data must be avoided, with offenders fined 

and/or prosecuted. 

2. Leverage existing data to decrease burden, streamline reporting, and answer critical questions. 

(Q: 1, 7) 

In order to provide information to drive the decisions of policymakers, institutions, and students, existing 

administrative data should be leveraged and linked. Data from federal sources like U.S. Department of 

Education (ED), Social Security Administration (SSA), the Department of Defense (DoD), and Department 

of Veterans Affairs (VA), when linked, provide valuable data on important subgroups of students who are 

often overlooked, including Pell Grant recipients, student loan borrowers, and student veterans. Linking 

these data sources will illuminate opportunities to eliminate duplicative collections and decrease 

reporting burden for data providers. Without these data, policymakers, institutions, and students cannot 

answer critical questions about postsecondary access, success, post-college outcomes, and affordability:

o How many low-income, Pell Grant recipient, first-generation, veteran, adult, transfer, and 

part-time students, who make up the new majority on today’s campuses, attend each 

college? Do these students graduate? 

o How long does it take students, particularly students who enter with less academic 

preparation or fewer financial resources, to complete college? 

o Do the students who don’t graduate transfer, or do they drop out? 

o How much do different types of students borrow, and how do their repayment outcomes 

vary? 

o Can students find jobs in their chosen field, and how much do they earn? 

To answer these questions, with or without a student-level data system, there are a number of state and 

federal datasets that can and should be leveraged. These include: (1) the Department of Health and 

Human Services National Directory of New Hires (NDNH) and the U.S. Census Bureau’s Longitudinal 

Employer Household Dynamics (LEHD) program, which both include Unemployment Insurance (UI) wage 

records submitted by states to answer workforce and outcomes related questions; (2) SSA and the Internal 

Revenue Service (IRS), which have individual tax records that, in limited instances, agencies have found 

ways to use to report employment and earnings outcomes at the program level to better understand 

student outcomes in the workforce; (3) ED, which houses the National Student Loan Data System (NSLDS) 

and Integrated Postsecondary Education Data System (IPEDS), which include information on financial aid 

1614 The Promise of Evidence-Based Policymaking
3 

and student access and success that can be disaggregated to see progress for student subpopulations; 

and (4) the DoD and VA, which house data on student veterans, financial aid, and recruiting and can be 

used to answer these questions specifically for the veteran population. 

It is imperative that data collections are limited to those metrics that have a specific and valuable purpose 

to meeting administrative, policymaking, and research needs. These collections should also include a 

robust reporting function, returning data for benchmarking and improvement back to colleges and 

programs and adding value for the data providers. Any clearinghouse or data system solution that is 

considered by the Commission should focus on answering critical questions about student outcomes, 

balancing reporting burden with analytic value, and including outcomes data around employment and 

earnings for all students, within all postsecondary levels and workforce programs. 

3. Make recommendations that address the administrative and legal barriers to data linking and 

access. (Q: 5, 9) 

There are barriers that the Commission will need to consider when making recommendations related to 

federal data. First, political support is absolutely necessary to accomplish these goals. Whether it is the 

creation of the proposed national clearinghouse, linking federal administrative data in other ways, or 

recommending changes to federal law to create a student-level data system, policymakers must champion 

legislation and push for resource allocation. The Commission will need to mitigate concerns about the size 

and role of the federal government as a data steward, as well as fear that students’ privacy will be violated. 

PostsecData strongly supports adopting comprehensive privacy and security standards that are informed 

by best practices in the field. A tiered, user-based access system would provide each stakeholder with the 

data necessary for decision-making, while protecting the student data housed in the system. A deep 

exploration and understanding of these issues is paramount for success in leveraging federal 

administrative data for policymaking purposes. 

Second, the Commission should consider changes to statute that may be necessary to meet these goals, 

depending on the data included in a national clearinghouse. The Higher Education Opportunity Act of 

2008 includes a ban prohibiting the creation of a federal postsecondary student-level data collection; a 

similar ban exists in the Workforce Innovation and Opportunity Act of 2014. With these bans in place, a 

comprehensive, national approach will require linking other data sources at the federal and state levels. 

Data sharing around UI wage records varies from state to state, and some state laws do not currently 

allow for sharing across state lines. Many states have passed privacy and security legislation over the past 

few years4, creating barriers to sharing data across the local, state, and federal levels, as well as confusion 

as to when data sharing is permissible. 5

Finally, technical barriers may complicate the implementation of a robust national clearinghouse. Some 

of the states that passed privacy and security legislation ban collection of social security numbers with 

educational records, which complicates data matching. In many cases, employment and education data 

matching requires personally identifiable information, like Social Security Numbers (SSNs), and first and 

last names. Tax and wage records usually include this information, but student records vary from state to 

state and may not include all of the required information for complete matching. The Commission should 

consider the mechanisms for matching these data and processes for how to approach unmatched data. 

Addressing some of the aforementioned legal barriers around student-level data systems and data sharing 

could help to alleviate these problems. Access to IRS and SSA data is highly controlled for privacy and 

security, and understandably so. However, as the federal government moves toward a more streamlined 

federal data system – potentially through a national clearinghouse – privacy and security standards and 

The Promise of Evidence-Based Policymaking 1615
4 

processes, like role-based access, should be implemented to ensure quality data, accurate linkages, and 

complete security of all data. 

4. Expand access to wage and labor market information for postsecondary outcomes. (Q: 1, 5) 

Students and policymakers are particularly interested in better information about post-college outcomes 

and employment measures for schools and programs. Right now, data and metrics on employment and 

earnings are limited to voluntary initiatives,6 like College Measures,7 state dashboards, and the College 

Scorecard. The Commission should explore wage and labor market information datasets to understand 

the return on personal investment for students and families, as well as the economic return on federal 

and state investments in higher education. The federal government needs to create efficient, strategic 

processes for managing employment data. Sources for these data vary in completeness and require 

legislative or executive actions to link the data. First is the Census Bureau’s LEHD program or NDNH, which, 

as previously mentioned, utilize state UI wage records. When linked, UI wage records are able to show 

mobility, but exclude all federal employees and the self-employed. The Federal Employment Data 

Exchange System (FEDES) and Wage Record Interchange System (WRIS/WRIS2) datasets also provide state 

level data, but because of restrictions on data use for performance reporting, research, and evaluation, 

policymakers will need to renegotiate when those data are available to use for these purposes.8 A more 

complete federal option is to use SSA and IRS tax records, which should include records for all taxpayers. 

In the absence of a comprehensive national data solution, states are linking a limited set of education and 

labor market information data regionally, through efforts like the Western Interstate Commission on 

Higher Education’s (WICHE) Multistate Longitudinal Data Exchange (MLDE). Currently operating in four 

states (Hawaii, Oregon, Washington, and Idaho), state data exchange efforts like the MLDE require 

extensive memoranda of understanding between the states and require navigating data privacy and 

security laws at the state level. Through this system, states discover new information about the impact of 

state policies and provide needed evidence for institutional accountability and benchmarking. The 

challenge of this system is that when scaling to a 50-state system, incomplete geographic and institutional 

coverage constrains its utility, as many state systems do not include private institutions.9 State-based 

wage data contain information only on workers in that state, so linking data across states is necessary to 

capture student mobility. Without mobility data, efforts like College Measures are limited to post-college 

outcomes only for students who remain in state. Additionally, a lack of capacity and funding at the state 

and local levels has created challenges to collecting, processing, and using these data. The viability of a 

state-based solution is contingent upon a sustainable funding stream. Sources such as the State 

Longitudinal Data System Grant Program and Workforce Data Quality Initiative could provide this 

necessary support. 

5. Align definitions and metrics across federal laws. (Q: 3) 

Establishing common definitions for data metrics across federal laws would reduce administrative burden 

and create comparable outcomes across federal programs. Federal laws like the Higher Education Act, the 

Workforce Innovation and Opportunity Act, the Perkins Career and Technical Education Act, and the 

Elementary and Secondary Education Act contain similar metrics that should be streamlined. While these 

recommendations would require Congressional action, consistent definitions would cut costs and reduce 

security risks by minimizing the number of redundant data sets and collections across the federal 

government. The Commission should propose a legal framework that uses the proposed national 

clearinghouse to streamline current agency collections and data sets by aligning metrics around: 

x Student enrollment rate in colleges and programs 

x College Readiness 

x College and program completion 

1616 The Promise of Evidence-Based Policymaking
5 

x Employment rate or job placement rate 

x Earnings 

x Credential Attainment 

Having an aligned set of metrics across federal agencies would synergize existing reporting and allow 

consumers to make informed decisions about programs that span agencies. Standard definitions would 

also create opportunities to combine and expand data dashboards and reporting for additional purposes, 

including consumer information and regulatory compliance. For example, the College Scorecard could 

include data from the Department of Labor’s administration of WIOA reporting in order to show training 

program outcomes and serve as a resource for students to understand outcomes for different career 

pathways. High school scorecards could also include college matriculation rates to show how students in 

schools and districts progress on to higher education. These data have the potential to illustrate different 

education pathways and outcomes for all students.

6. Recommend that Congress overturn the ban on a federal student-level data system. (Q: 9) 

The statutory bans in the Higher Education Act and the Workforce Innovation and Opportunity Act on a 

federal student-level data system stifle the ability of policymakers to answer questions about our 

postsecondary system, limit the information available to consumers, and impose unnecessary burden 

onto institutions. The Commission should recommend that Congress overturn the ban and encourage ED 

and the Department of Labor to engage with the higher education community to design and implement a 

student-level data system. This system would create a nationwide, inclusive data set that shows how 

students move through higher education and their post-college outcomes. This system would allow for 

disaggregation by key student characteristics, like Pell Grant receipt, race/ethnicity, and others, and 

illuminate evidence for future policymaking around closing equity gaps and strengthening the federal 

investment in higher education and postsecondary programming. Given the sensitive nature of record 

level data, the Commission should recommend rigorous data privacy and security policies to govern this 

system, including all those mentioned in section 1. 

Thank you for your focus on improving the structure and utility of the national data infrastructure. If you 

have questions or would like to discuss these issues further, please contact Mamie Voight, Vice President 

of Policy Research at the Institute for Higher Education Policy (mvoight@ihep.org, 202-587-4967). To learn 

more about the Postsecondary Data Collaborative, visit our website at www.ihep.org/postsecdata. 

Sincerely, 

Advance CTE 

Association for Career & Technical Education 

Association of Public and Land-grant Universities 

Data Quality Campaign 

Institute for Higher Education Policy 

NASPA - Student Affairs Administrators in Higher Education 

New America 

The Education Trust 

The Institute for College Access & Success 

Workforce Data Quality Campaign 

Young Invincibles 

 

The Promise of Evidence-Based Policymaking 1617
6 

APPENDIX 

Questions addressed in the request for comments, taken from the Federal Register10: 

1. Are there successful frameworks, policies, practices, and methods to overcome challenges related to 

evidence-building from state, local, and/or international governments the Commission should consider 

when developing findings and recommendations regarding Federal evidence-based policymaking? If so, 

please describe. 

2. Based on identified best practices and existing examples, what factors should be considered in 

reasonably ensuring the security and privacy of administrative and survey data? 

3. Based on identified best practices and existing examples, how should existing government data 

infrastructure be modified to best facilitate use of and access to administrative and survey data? 

5. What challenges currently exist in linking state and local data to federal data? Are there successful 

instances where these challenges have been addressed? 

7. What data should be included in a potential U.S. government data clearinghouse(s)? What are the 

current legal or administrative barriers to including such data in a clearinghouse or linking the data? 

9. What specific administrative or legal barriers currently exist for accessing survey and administrative 

data? 

1

 Please see the appendix for an itemized list of the questions answered in this comment letter. 2

 For more information on the University of Texas System Dashboard, please visit: http://data.utsystem.edu/ 3

 For more information on the California Community College Launchboard, please visit: 

http://doingwhatmatters.cccco.edu/launchboard.aspx 

4

 The Data Quality Campaign tracks and synthesizes state privacy legislation annually. For information on 2015 and 

2016 legislation, please visit: http://dataqualitycampaign.org/resource/student-data-privacy-legislation-happened2015-next/; http://dataqualitycampaign.org/resource/2016-student-data-privacy-legislation/ 5

 For more information on limited access to and sharing data for policy research, please read “Balancing 

Confidentiality and Access: Sharing employment and wage data for policy analysis and research”: 

http://www.lmiontheweb.org/download/2015-05/Report--_Data_Confidentiality_and_Sharing_-_CRECLMI_Institute_-_May_2015.pdf 

6

 Over the past decade, voluntary data initiatives have collected data on student access, progress, and completion 

in an effort to guide federal, state, and institutional policy. In many cases, these collections supplement was is 

collected at the state and federal levels and seeks to include more students. For more information, please visit the 

PostsecData website: http://www.ihep.org/postsecdata/mapping-data-landscape/voluntary-data-initiatives 7

 To explore the 2 and 4 year college tools and the Economic Success Metrics, please visit: 

http://www.collegemeasures.org/ 

8

 For more information on utilizing federal employment data systems, please read “Employing WRIS2: Sharing age 

records across states to track program outcomes” by Rachel Zinn and John Dorrer: 

http://www.workforcedqc.org/sites/default/files/images/WRIS2%20Report%20May%202014.pdf 9

 For more information, please read “Fostering State-to-State Data Exchanges” by Brian Prescott and Patrick Lane: 

http://www.ihep.org/sites/default/files/uploads/postsecdata/docs/resources/fostering_state-tostate_data_exchanges.pdf 

10 https://www.regulations.gov/document?D=USBC-2016-0003-0001 

 

1618 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t1a-z668

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0148

Comment on FR Doc # 2016-22002

Submitter Information

Name: Leonie Haimson

Address:

New York City, NY, 10011

Email: leoniehaimson@gmail.com

General Comment

November 14, 2016

Docket ID: USBC-2016-0003-0001

The Commission on Evidence-Based Policymaking (CEP) 

To the Commission:

We, the undersigned organizations, respectfully submit the following comments to the

Commission on Evidence-Based Policymaking. While we applaud the ambitious charge

of the Commission to examine "strategies to increase the availability and use of

government data, in order to build evidence related to government programs and

policies, while protecting the privacy and confidentiality of the data," we strongly

oppose any proposal that would lead to the creation of a central federal clearinghouse or

linked data sets containing the personally identifiable information ("PII") of all students,

commonly referred to as a federal student unit-record system or national database.

The Promise of Evidence-Based Policymaking 1619
We cannot overstate the threat to student privacy that would be posed by the

development of such a database, including breach, malicious attack, or use of student PII

for purposes not initially intended. Ever since a federal student unit-record system was

first proposed by the Bush administration in 2005, and banned by the Higher Education

Act in 2008, the reasons against creating it have only become more persuasive in recent

years.

Please see the attached link for a more detailed discussion of our concerns. 

Yours,

Parent Coalition for Student Privacy

American Civil Liberties Union

Network for Public Education and NPE Action

Parents Across America 

Badass Teachers Association 

New York State Allies for Public Education 

Attachments

letter-to-CEP-w-signers-final-11.14.16

1620 The Promise of Evidence-Based Policymaking
www.studentprivacymatters.org 

info@studentprivacymatters.org 

@parents4privacy 

124 Waverly Place 

New York, NY 10011 

303.204.1272 

November 14, 2016 

Docket ID: USBC-2016-0003-0001 

The Commission on Evidence-Based Policymaking (CEP) 

To the Commission: 

We, the undersigned organizations, respectfully submit the following comments to the Commission on 

Evidence-Based Policymaking. While we applaud the ambitious charge of the Commission to examine 

“strategies to increase the availability and use of government data, in order to build evidence related to 

government programs and policies, while protecting the privacy and confidentiality of the data,” we 

strongly oppose any proposal that would lead to the creation of a central federal clearinghouse or 

linked data sets containing the personally identifiable information (“PII”) of all students, commonly 

referred to as a federal student unit-record system or national database. 

We cannot overstate the threat to student privacy that would be posed by the development of such a 

database, including breach, malicious attack, or use of student PII for purposes not initially intended. 

Ever since a federal student unit-record system was first proposed by the Bush administration in 2005, 

and banned by the Higher Education Act in 2008, the reasons against creating it have only become more 

persuasive in recent years. 

First, we are gravely concerned about the high probability of breaches and unauthorized access to the 

data. As a 2015 report by the U.S. Government Accountability Office (“GAO”) revealed, reports of 

security incidents involving breaches of personal information held by federal agencies rose from 10,481 

in 2009 to 27,624 in 2014 – an increase of 164 percent over five years -- for a total of 144,439 reported 

instances. 1

 The report also noted that these events can “adversely affect national security; [and] 

damage public health and safety” and yet federal agencies have failed to implement approximately 

nearly half of the recommendations made to them to improve security of their systems over the last six 

years. 

In addition to system breaches documented by the GAO, the Office of Personnel Management 

announced in June 2015 that the personnel records of about 22.1 million people had been maliciously 

hacked by foreign interests -- not only federal employees and contractors but also their families and 

friends, including highly sensitive information gathered for the purposes of security clearance.2

The US Department of Education has been found to have especially weak security standards in its 

collection and storage of student information, as reported by an audit released in November 2015 by 

the department’s Inspector General. This puts at risk the huge amount of data that the agency already 

holds, including student loan information involving information on more than 100 million individuals and 

 

1 http://www.gao.gov/assets/680/673678.pdf

2 https://www.washingtonpost.com/news/federal-eye/wp/2015/07/09/hack-of-security-clearance-systemaffected-21-5-million-people-federal-authorities-say/

The Promise of Evidence-Based Policymaking 1621
at least 39 million unique Social Security numbers.3

 A reported by the audit, staff in the IG office hacked 

into the Department’s main IT system and gained unfettered access to personal data without anyone 

noticing. Overall, the audit found significant weaknesses in four out of the five security categories.4

 In 

May 2016, the government scorecard created to assess how well federal agencies were implementing 

data security measures awarded the Education Department an overall grade of D.5

Second, K-12 student data currently collected by state departments of education in statewide 

longitudinal data systems (SLDS) that would potentially be shared with the federal database generally 

extend well beyond traditional administrative data to include upwards of 700 specific personal data 

elements, including students’ immigrant status, disabilities, disciplinary incidents, and homelessness 

status.6

Data collected ostensibly for the sole purpose of research but without the individual’s consent or 

knowledge would likely be merged with other federal agency data sets, to follow students into the 

workplace and beyond, and could include data from their military service, tax returns, criminal and 

health records. If this granular level of sensitive information were available in a universal U.S. student 

record database, it could quickly become a go-to repository for purposes that should never be allowed. 

A real-life example of the potential misuse of a system of this nature has just been reported in England. 

There, a similar student data repository called the National Pupil Database (“NPD”) was intended to be 

maintained “solely for internal departmental use for the analytical, statistical and research purposes.” 

But as Freedom of Information requests7

 recently revealed, the names and home addresses of 

thousands of students8

 in the NPD have been requested by police and the Home Office for various 

purposes over the last 15 months, including to curb “abuse of immigration control.”9

 A group of parents, 

teachers, and human rights campaigners has launched a national boycott to urge parents and schools to 

withhold their children’s country of birth and nationality, data which is being collected at national level 

for the first time.10

Finally, we are very concerned about recent revelations of the widespread surveillance on ordinary 

citizens by the federal government, and the way in which a national unit-record system could be used to 

expand tracking of students. While data holds promise to solve complex problems and may be used to 

improve our nation’s policies, we have a responsibility to our nation’s citizens to protect the privacy of 

their most personal information, especially that of vulnerable children. 

 

3 https://www2.ed.gov/about/offices/list/oig/auditreports/fy2015/a11o0001.pdf

4 http://federalnewsradio.com/cybersecurity/2015/11/government-testers-easily-bypassed-education-defensesrecent-cyber-audit/

5 http://www.nextgov.com/cio-briefing/2016/05/fitara-scorecard-fewer-agencies-get-failing-scores/128410/

6

 See NYS for example at http://www.p12.nysed.gov/irs/sirs/documentation/NYSEDstudentData.xlsx 7 https://www.whatdotheyknow.com/request/sharing_national_pupil_database?nocache=incoming878444#incoming-878444

8 http://www.parliament.uk/business/publications/written-questions-answers-statements/writtenquestion/Commons/2016-10-13/48635/

9

http://www.theregister.co.uk/2016/10/12/national_pupil_database_has_been_used_to_control_immigration/?

mt=1476378123415

10 https://www.liberty-human-rights.org.uk/news/press-releases-and-statements/government-facing-lordsopposition-over-widely-condemned-“foreign

1622 The Promise of Evidence-Based Policymaking
Any recommendation by the Commission to establish a federal data clearinghouse of student PII could 

effectively create life-long dossiers on nearly every individual in the nation. Instead, we strongly believe 

that the federal government should use aggregate, de-identified student information already 

maintained by states or districts for research or policy decisions. 

We strongly urge that members of the Commission to consider the threats to privacy that overturning 

the ban on a federal student unit-record clearinghouse would create. Once privacy is lost it is nearly 

impossible to restore, and we hold a moral and ethical obligation to our children – and our citizens -- to 

minimize this risk in any way possible. 

Yours, 

Parent Coalition for Student Privacy 

American Civil Liberties Union 

Network for Public Education and NPE Action 

Parents Across America 

Badass Teachers Association 

New York State Allies for Public Education 

The Promise of Evidence-Based Policymaking 1623
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t1a-sh02

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0149

Comment on FR Doc # 2016-22002

Submitter Information

Name: Marc Rotenberg

Address:

Washington, DC, 20009

Email: rotenberg@epic.org

General Comment

See attached file(s)

Attachments

EPIC-CEP-RFC

1624 The Promise of Evidence-Based Policymaking
Commission on Evidence-based Comments of EPIC 

Policymaking: RFC November 14, 2016 

1 

COMMENTS OF THE ELECTRONIC PRIVACY INFORMATION CENTER 

to the 

COMMISSION ON EVIDENCE-BASED POLICYMAKING 

Request for Public Comment 

November 14, 2016 

 By notice published on September 14, 2016, the Commission on Evidence-Based 

Policymaking (“CEP”) requests public comments on “how to increase the availability and use of 

government data in support of evidence-building activities related to government programs and 

policies, while protecting the privacy and confidentiality of such data.”1

 Pursuant to this notice, 

the Electronic Privacy Information Center (“EPIC”) submits these comments to (1) make clear 

that data can be used both for informed policy-making and for profiling, segmentation, and 

discrimination; (2) urge the Commission to promote privacy-enhancing techniques (“PETs”) that 

minimize or eliminate Personally Identifiable Information; and (3) propose data use schemes 

that leave the data with the custodial agencies instead of a central repository. 

 EPIC is a public interest research center in Washington, D.C. EPIC was established in 

1994 to focus public attention on emerging privacy and related human rights issues and to 

 

1 Request for Comments for the Commission on Evidence-Based Policymaking, 81 Fed. Reg. 63,166 (Sep. 

14, 2016) [hereinafter “Request for Comments]. 

The Promise of Evidence-Based Policymaking 1625
Commission on Evidence-based Comments of EPIC 

Policymaking: RFC November 14, 2016 

2 

protect privacy, the First Amendment, and constitutional values. EPIC has a particular interest in 

safeguarding personal privacy and preventing harmful data practices. For example, EPIC 

routinely submits comments to federal agencies, urging them to uphold the Privacy Act and 

protect individual privacy in mass government databases.2

 EPIC is also a leading consumer 

advocate before the Federal Trade Commission (“FTC”). EPIC has a particular interest in 

protecting consumer privacy, and has played a leading role in developing the authority of the 

FTC to address emerging privacy issues and to safeguard the privacy rights of consumers.3

 In 

2014, EPIC submitted extensive comments to the White House Office of Science and 

Technology Policy, warning of the enormous risk to Americans that current "big data" practices 

present, and recommending the adoption of privacy-enhancing techniques.4

 EPIC also maintains 

a webpage on practical privacy tools.5

 

2 See, e.g., EPIC et al., Comments on the Terrorist Screening Database System of Records, Notice of Privacy Act 

System of Records and Notice of Proposed rulemaking, Docket Nos. DHS 2011-0060 and DHS 2011-0061 (Aug. 5, 

2011), available at http://epic.org/privacy/airtravel/Comments_on_DHS-2011-0060_and_0061FINAL.pdf; EPIC, 

Comments on Secure Flight, Docket Nos. TSA-2007-28972, 2007-28572 (Sept. 24, 2007), available at

http://epic.org/privacy/airtravel/sf_092407.pdf; EPIC, Secure Flights Should Remain Grounded Until Security and 

Privacy Problems are Resolved, Spotlight on Surveillance Series (August 2007), available at

http://epic.org/privacy/surveillance/spotlight/0807/default.html; Passenger Profiling, EPIC, 

http://epic.org/privacy/airtravel/profiling.html (last visited Apr. 3, 2014); Secure Flight, EPIC, 

http://epic.org/privacy/airtravel/secureflight.html (last visited Apr. 3, 2014); Air Travel Privacy, EPIC, 

http://epic.org/privacy/airtravel/ (last visited Apr. 3, 2014). 

3 See, e.g., Letter from EPIC Executive Director Marc Rotenberg to FTC Commissioner Christine 

Varney, EPIC (Dec. 14, 1995) (urging the FTC to investigate the misuse of personal information by the 

direct marketing industry), http://epic.org/privacy/internet/ftc/ftc_letter.html; DoubleClick, Inc., FTC File 

No. 071-0170 (2000) (Complaint and Request for Injunction, Request for Investigation and for Other 

Relief), http://epic.org/privacy/internet/ftc/DCLK_complaint.pdf; Microsoft Corporation, FTC File No. 

012 3240 (2002) (Complaint and Request for Injunction, Request for Investigation and for Other Relief), 

http://epic.org/privacy/consumer/MS_complaint.pdf; Choicepoint, Inc., FTC File No. 052-3069 (2004) 

(Request for Investigation and for Other Relief) , http://epic.org/privacy/choicepoint/fcraltr12.16.04.html. 

4

 [cite to comments] 

5

 EPIC Online Guide to Practical Privacy Tools, http://epic.org/privacy/tools.html. 

1626 The Promise of Evidence-Based Policymaking
Commission on Evidence-based Comments of EPIC 

Policymaking: RFC November 14, 2016 

3 

1. Data is the basis of research, innovation, economic growth, and informed policy 

decisions, but data is also the basis for profiling, tracking, segmentation, and 

discrimination 

Although increased use of administrative and survey data has the potential to improve 

informed policymaking, there are real risks in combining this data and making it more easily 

available. Data that is improperly protected can be used by the government and in the private 

sector for profiling, tracking, and discrimination. The potential use of personal information to 

make automated decisions and segregate individuals based on secret, imprecise and oftentimes 

impermissible factors presents clear risks to fairness and due process. 

A. Government collection and abuse of data 

Today, Americans are in more government databases than ever. Government agencies 

routinely amass personally-identifiable information (“PII”) but absolve themselves of any legal 

duties or responsibilities to safeguard individual privacy. For example, the Federal Bureau of 

Investigation’s Data Warehouse System hoards individual information, including: 

biographical information (such as name, alias, race, sex, date of birth, place of 

birth, social security number, passport number, driver's license, or other unique 

identifier, addresses, telephone numbers, physical descriptions, and photographs); 

biometric information (such as fingerprints); financial information (such as bank 

account number); location; associates and affiliations; employment and business 

information; visa and immigration information; travel; and criminal and 

investigative history, and other data that may assist the FBI in fulfilling its 

national security and law enforcement responsibilities.6

Incredibly, the agency has exempted itself from Privacy Act requirements that the FBI 

maintain only “accurate, relevant, timely and complete” personal records.7

 The FBI has also 

exempted itself from Privacy Act requirements permitting individuals to access and amend 

 

6

 Privacy Act of 1974; System of Records, 77 Fed. Reg. 40,630, 40,631 (July 10, 2012), available at

http://www.gpo.gov/fdsys/pkg/FR-2012-07-10/pdf/2012-16823.pdf. 

7

 28 C.F.R. §16.96 (v). 

The Promise of Evidence-Based Policymaking 1627
Commission on Evidence-based Comments of EPIC 

Policymaking: RFC November 14, 2016 

4 

inaccurate records.8 Other agencies, like the Department of Homeland Security and the National 

Security Agency, have exempted databases containing detailed, sensitive personal information 

from well-established Privacy Act safeguards. 9 EPIC has routinely objected to agencies 

gathering personally identifiable information while eschewing privacy protections, noting: 

It is inconceivable that the drafters of the Privacy Act would have permitted a 

federal agency to propose a profiling system on U.S. citizens and be granted 

broad exemptions from Privacy Act obligations. Consistent and broad application 

of Privacy Act obligations are the best means of ensuring accuracy and reliability 

of the data used in a system that profoundly affects millions of individuals as they 

travel throughout the United States on a daily basis.10

The government also uses predictive analytics to the detriment of millions of individuals. 

For example, the Department of Homeland Security’s TSA PreCheck program collects vast 

amounts of PII including biometric information to perform a “security threat assessment” of “law 

enforcement, immigration, and intelligence databases, including a fingerprint-based criminal 

history check conducted through the Federal Bureau of Investigation.” 11 The TSA uses 

automated data processing to determine which individuals will be scrutinized upon traveling 

throughout the United States. 12 The decisions are completely opaque and lack an effective 

recourse option. Remarkably, the TSA itself has lost sensitive personal information that it has 

 

8 Id. 9 See, e.g., EPIC et al., Comments on the Department of Defense Privacy Program (Oct. 21, 2013), 

available at https://epic.org/privacy/nsa/Coal-DoD-Priv-Program-Cmts.pdf; see also supra note 3, 

Comments Urging the Department of Homeland Security To (A) Suspend the “Automated Targeting 

System”.

10 EPIC, Comments on TSA PreCheck Application Program System of Records Notice and Notice of 

Proposed Rulemaking and TSA Secure Flight System of Records Notice, 5 (Oct. 10, 2013), available at

http://epic.org/apa/comments/TSA-PreCheck-Comments.pdf. 11 Privacy Act of 1974: Implementation of Exemptions; Department of Homeland Security Transportation Security 

Administration, DHS/TSA-021, TSA PreCheck Application Program System of Records, 78 Fed. Reg. at 55,657 

(proposed Sept. 11, 2013), available at http://www.gpo.gov/fdsys/pkg/FR-2013-09-11/pdf/2013-22069.pdf. 

12 Privacy Act of 1974; Department of Homeland Security Transportation Security Administration--

DHS/TSA—019 Secure Flight Records System of Records, 78 Fed. Reg. 55,270, 55,271 (proposed Sept. 

10, 2013), available at http://www.gpo.gov/fdsys/pkg/FR-2013-09-10/pdf/2013-21980.pdf. 

1628 The Promise of Evidence-Based Policymaking
Commission on Evidence-based Comments of EPIC 

Policymaking: RFC November 14, 2016 

5 

collected from its employees.13 The TSA lost a portable drive containing the bank account 

numbers, Social Security numbers, names and birth dates of more than 100,000 people who 

worked at the TSA over a three-year period. 

It is vitally important to ensure that any data clearinghouse minimizes collection, secures 

the information that is collected, and prevents abuses of collected data through the use of 

predictive analytics. 

B. The 1965 National Data Center Proposal and the Privacy Act of 1974 

This Commission’s current efforts echo in many ways the goals of the proposed National 

Data Center in the 1960s. As Rebecca Kraus wrote: 

Computer technology had improved the efficiency and affordability of research 

with large data sets, and the expansion of government social programs called for 

more data and research to inform public policy. As a result, in 1965 social 

scientists recommended that the federal government develop a national data 

center that would store and make available to researchers the data collected by 

various statistical agencies.14

A 1965 report prepared by the SSRC Committee on the Preservation and Use of 

Economic Data noted that federal government statistics were highly decentralized and held by 

agencies that collected the underlying data as a “by-product of the regulatory process.”15 It 

recommended the creation of a “Federal Data Center” with the authority to obtain data 

“produced by all federal agencies.”16 The report also recommended the development of an 

 

13 Thomas Frank, TSA Seeks Hard Drive, Personal Data on 100,000, USA TODAY, May 5, 2007, 

available at http://usatoday30.usatoday.com/news/washington/2007-05-04-harddrive-tsa_N.htm?csp=1. 

14 Rebecca S. Kraus, Statistical Dèjá Vu: The National Data Center Proposal of 1965 and Its 

Descendants, 5 J. Privacy & Confidentiality 1, 1 (2013). 

15 RICHARD RUGGLES, RICHARD MILLER, EDWIN KUH, STANLEY LEBERGOTT, GUY ORCUTT & JOSEPH 

PECHMAN, REPORT OF THE SSRC COMMITTEE ON THE PRESERVATION AND USE OF ECONOMIC DATA

(1965). 

16 Id. at 1. 

The Promise of Evidence-Based Policymaking 1629
Commission on Evidence-based Comments of EPIC 

Policymaking: RFC November 14, 2016 

6 

organization that could provide a “clearing house and coordination of requests for data made by 

individual scholars from Federal agencies.”17 

The proposal was met with public outrage.18 Congress held hearings at which proponents 

of the national data center appeared to downplay privacy concerns.19 In 1973, a federal advisory 

committee released its report on Records, Computers, and the Rights of Citizens.20 As EPIC 

President Marc Rotenberg explained in 2000: 

The purpose was benign. It was believed that such a databank would be very 

useful to social scientists and others, but the implications were severe. People 

understood that the collection of these permanent profiles, made possible by 

computerized automation, would pose a threat to the privacy and liberty of 

American citizens. The proposal for the National Data Center was withdrawn and 

over time a comprehensive legal framework—the Privacy Act of 1974—was 

established to safeguards the rights of American citizens. The Privacy Act 

imposed on all federal agencies essential privacy rights and responsibilities—

“Fair Information Practices”—that would limit would federal agencies could do 

with personal information and gave every American the right to see the 

information about them that was collected.21

The Privacy Act incorporates the Code of Fair Information Practices that the Health, 

Education, and Welfare Advisory Committee on Automated Data Systems issued in 1973.22 The 

Code of Fair Information Practices (“FIPs”) sets out five obligations for all organizations that 

collect personal data: 

1. There must be no personal data record-keeping systems whose very existence is secret. 

 

17 Id. at 2. 

18 Id. at 13–17; VANCE PACKARD, THE NAKED SOCIETY (Ig Publishing 2014) (1964). 

19 Invasions of Privacy: Hearings before the Subcomm. on Administrative Practice and Procedure of the 

Senate Comm. on the Judiciary, 89th Cong. (1966), https://www.epic.org/privacy/hew1973report/; see 

also Kraus, supra note 14, at 11. 

20 SECRETARY'S ADVISORY COMM. ON AUTOMATED PERS. DATA SYS, RECORDS, COMPUTERS AND THE 

RIGHTS OF CITIZENS (1973), https://www.epic.org/privacy/hew1973report/. 

21 Internet Privacy and Profiling: Hearing before the S. Comm. on Commerce, Sci. & Transp., 106th 

Cong. (2000) (statement of Marc Rotenberg, Director, Electronic Privacy Information Center). 

22 The Code of Fair Information Practices, EPIC, http://epic.org/privacy/consumer/code_fair_info.html. 

1630 The Promise of Evidence-Based Policymaking
Commission on Evidence-based Comments of EPIC 

Policymaking: RFC November 14, 2016 

7 

2. There must be a way for a person to find out what information about the person is in a 

record and how it is used. 

3. There must be a way for a person to prevent information about the person that was 

obtained for one purpose from being used or made available for other purposes without 

the person's consent. 

4. There must be a way for a person to correct or amend a record of identifiable information 

about the person. 

5. Any organization creating, maintaining, using, or disseminating records of identifiable 

personal data must assure the reliability of the data for their intended use and must take 

precautions to prevent misuses of the data.23

 In passing the Privacy Act of 1974, Congress found that: (1) individual privacy is 

“directly affected by the collection, maintenance, use, and dissemination of personal information 

by Federal agencies”; (2) big data in the government sector “greatly magnified the harm to 

individual privacy”; (3) misuse of government data can threaten “the opportunities for an 

individual to secure employment, insurance, and credit, and his right to due process”; (4) privacy 

is a constitutionally-protected “personal and fundamental right”; and (5) “in order to protect the 

privacy of individuals identified in information systems maintained by Federal agencies, it is 

necessary and proper for the Congress to regulate the collection, maintenance, use, and 

dissemination of information by such agencies.”24 

 The United States has been slow to update its privacy laws and companies have been 

reluctant to implement privacy enhancing technologies—neither an appropriate legal framework 

 

23 U.S. Dep't. of Health, Education and Welfare, Secretary's Advisory Committee on Automated Personal 

Data Systems, Records, computers, and the Rights of Citizens viii (1973). 

24 Public Law 93-579, 93rd Congress, S.3418, Privacy Act, Section 2 (a) (Dec. 31, 1974). 

The Promise of Evidence-Based Policymaking 1631
Commission on Evidence-based Comments of EPIC 

Policymaking: RFC November 14, 2016 

8 

or technical framework have been implemented to consistently safeguard individual privacy 

through the FIPs. 

 The FIPs appear in various privacy laws and frameworks, such as the Organization for 

Economic Cooperation and Development (“OECD”) Privacy Guidelines,25 the Privacy Act of 

1974, 26 and the European Commission’s recent Data Protection Regulation.27 In the United 

States, the Consumer Privacy Bill of Rights (“CPBR”) is a flexible and adaptable instantiation of 

the FIPs. 

 The CPBR provides a comprehensive framework that lists seven substantive privacy 

protections for consumers: Individual Control, Transparency, Respect for Context, Security, 

Access and Accuracy, Focused Collection, Accountability.28 This Commission’s efforts to make 

administrative and survey data available for use in evidence-based policymaking while 

preserving privacy protections should focus on technology that facilitates the implementation of 

the privacy protections listed in the CPBR. 

The reaction to the proposed National Data Center contains several lessons for this 

Commission. First, privacy must be an integral component of any effort to streamline access to 

administrative and survey data. Second, the importance of privacy to the project must be clearly 

communicated to the public. Third, because the idea of a centralized repository is particularly 

worrisome, any clearinghouse should leave data with the custodial agencies. And finally, a 

 

25 OECD Guidelines on the Protection of Privacy and Transborder Flows of Personal Data, available at

http://www.oecd.org/document/18/0,3343,en_2649_34255_1815186_1_1_1_1,00.html. 

26 Privacy Act of 1974, 5 USC § 552a. 

27 Proposal for a Regulation of the European Parliament and the Council on the protection of individuals 

with regard to the processing of personal data and the free movement of such data (General Data 

Protection Regulation), E.C. COM (2012) final, (Jan. 25, 2012), available at 

http://ex.europa/eu/justice/data-protection/document/review2012/com_2012_11_en.pdf 

28 Id.

1632 The Promise of Evidence-Based Policymaking
Commission on Evidence-based Comments of EPIC 

Policymaking: RFC November 14, 2016 

9 

clearinghouse for government data must operate within the protections provided by the Privacy 

Act. 

2. The Commission should encourage the development and use of privacy-enhancing 

techniques to maximize the benefits and minimize the risks of greater data access 

The Commission should focus on Privacy Enhancing Techniques 29 (“PETs”) that 

“minimize or eliminate the collection of personally identifiable information.”30 The Commission 

can support and further the work of computer scientists that have created various privacy 

enhancing mechanisms. Techniques that help obtain the advantages of big data while minimizing 

privacy risks should be encouraged, but these techniques must be robust, scalable, provable, and 

practical. We discuss some relevant privacy-enhancing techniques below. 

A. Data Minimization 

The Commission should incorporate data minimization requirements based on those 

described by the CPBR. The principles that call for federal agencies to “collect only as much 

personal data as they need to accomplish purposes specified” and “securely dispose of or deidentify personal data once they no longer need it, unless they are under a legal obligation to do 

otherwise”31 applies equally to any use or disclosure of agency data. Data minimization protects 

the confidentiality of consumer data and also serves important data security purposes. Limiting 

the amount of personal data that agencies collect, retain, and make available also limits the harm 

that results from possible data breaches. 

 

29 We use the word “techniques” instead of the more common “technologies” here to reflect the fact that 

privacy-enhancing methods do not necessarily have to be technological. 

30 Testimony and Statement for the Record of Marc Rotenberg, Executive Director, EPIC, Hearing on 

Privacy in the Commercial World, Before the Committee on Commerce, Trade, and Consumer Protection 

(Mar. 1, 2001), http://epic.org/privacy/testimony_0301.html; See also Herbert Burkert, Privacy 

Enhancing Technologies: Typology, Critique Vision in PHIL E AGRE AND MARC ROTENBERG,

TECHNOLOGY AND PRIVACY: THE NEW LANDSCAPE 125-42 (MIT Press 1998). 

31 White House, CPBR. 

The Promise of Evidence-Based Policymaking 1633
Commission on Evidence-based Comments of EPIC 

Policymaking: RFC November 14, 2016 

10 

Two examples show how evidence-based policymaking can be done without using any 

personally identifiable information: the U.S. Courts’ federal wiretap reports32 and the National 

Oceanic and Atmospheric Administration’s (“NOAA’s”) weather data. 

The wiretap reports are annual reports to Congress “concerning intercepted wire, oral, or 

electronic communications” pursuant to federal and state wiretap laws. Federal law requires the 

Administrative Office of the United States Courts to report the number of federal and state 

wiretap applications, authorizations, and denials.33 The reporting requirement provides a 

common data set that allows researchers, advocates, and government officials to describe the 

scope of lawful electronic surveillance in the United States. Because the reports are mandated by 

law, not voluntary or dependent on private sector data sources such as “transparency reports,” the 

reports are regularly reported and stable over time. The methodology for the reports is 

transparent, the data is provable, and the reports pose no privacy risk because PII is neither 

collected nor published. 

The NOAA uses weather forecasting data, climate data, and satellite imagery extensively. 

Its reports as used by fishing, shipping, agriculture, and many associated industries. Its data also 

supports mission-critical functions, emergency services, and local and state governments. None 

of this data is PII. 

B. Anonymization or “De-Identification” of Data 

The Commission should ensure that a clearinghouse uses anonymization techniques that 

adequately de-identify data so that data cannot be combined with other information for reidentification. Because not all de-identification techniques adequately anonymize data, it is 

 

32 U.S. COURTS, WIRETAP REPORTS, http://www.uscourts.gov/statistics-reports/analysis-reports/wiretapreports (last visited Nov. 14, 2016). 

33 Omnibus Crime Control and Safe Streets Act of 1968, Pub. L. No. 90-351, § 802, 82 Stat. 197 (codified 

at 18 U.S.C. § 2519). 

1634 The Promise of Evidence-Based Policymaking
Commission on Evidence-based Comments of EPIC 

Policymaking: RFC November 14, 2016 

11 

important that the process employed is robust, scalable, transparent, and shown to provably 

prevent the identification of consumer information.34 

Many companies claim to anonymize or de-identify personal information by aggregating 

it or assigning pseudonyms to it. Behavioral advertising companies routinely claim that the use 

of pseudonymous identifiers renders personal information anonymous.35 Data brokers also rely 

on the aggregate nature of their marketing data as a defense against criticism of their privacy 

practices. However, these claims of anonymization are often deceptive. Widely-publicized 

anonymization failures have shown that even relatively sophisticated techniques have still 

permitted researchers to identify particular individuals in large data sets.36 

EPIC favors techniques to de-identify user data,37 and many scholars are performing 

valuable research on various de-identification techniques, 38 but greater clarification and 

standardization is needed. For example, Distinguished Scientist at Microsoft Research Cynthia 

Dwork has espoused “differential privacy” as a “privacy-preserving analysis.” 39 Differential 

privacy “ensures that the removal or addition of a single database item does not (substantially) 

 

34 See generally EPIC, Re-identification, http://epic.org/privacy/reidentification/. 35 DMA Interest-Based Advertising (IBA) Compliance Alert & Guidelines for Interest-Based Advertising, 

Direct Marketing Assoc, http://www.dmaresponsibility.org/privacy/oba.shtml (“Relevant Ads Using 

Anonymous Data. IBA relies on anonymous, aggregated data to deliver an ad to a computer based on the 

computer browser’s activity, not the activities of a specific individual. Companies use cookies to make 

this happen.”). 

36 See, e.g., Latanya Sweeney, Simple Demographics Often Identify People Uniquely

http://dataprivacylab.org/projects/identifiability/paper1.pdf; Paul Ohm, Broken Promises of Privacy: 

Responding to the Surprising Failure of Anonymization, 57 UCLA L. REV. 1701, 1704 (2010) (“Data can 

be either useful or perfectly anonymous but never both.”). 

37 See generally Re-identification, ELEC. PRIVACY INFO. CTR., 

https://epic.org/privacy/reidentification/ (last visited Nov. 19, 2012). 

38 See, e.g., Cynthia Dwork, Differential Privacy: A Survey of Results, in THEORY AND 

APPLICATIONS OF MODELS OF COMPUTATION 1, 3 (Manindra Agrawal et al. eds., 2008); see 

also Latanya Sweeney, k-anonymity: A Model for Protecting Privacy, INT’L J. ON UNCERTAINTY, 

FUZZINESS AND KNOWLEDGE-BASED SYSTEMS, 10(5), 2002; 557- 570. 

39 Cynthia Dwork, Differential Privacy: A Survey of Results, 1, 2008, 

http://www.cs.ucdavis.edu/~franklin/ecs289/2010/dwork_2008.pdf. 

The Promise of Evidence-Based Policymaking 1635
Commission on Evidence-based Comments of EPIC 

Policymaking: RFC November 14, 2016 

12 

affect the outcome of any analysis.” 40 Although not an “absolute guarantee of privacy,” 

differential privacy “ensures that only a limited amount of additional risk is incurred by 

participating in the socially beneficial databases.”41 

Jeff Jonas, Chief Scientist for the IBM Analytics Groups, describes the need to “bake in” 

privacy protection by, for example, “the ability to anonymize the data at the edge, where it lives 

in the host system, before you bring it together to share it and combine it with other data.”42 The 

Commission should focus on improving anonymization techniques to not only increase its 

effectiveness but also to expand the use cases for anonymization. 

3. A clearinghouse should leave the data with the custodial agencies instead of storing 

data in a central repository 

The Commission asks in question 11: 

How might integration of administrative and survey data in a clearinghouse affect 

the risk of unintentional or unauthorized access or release of personallyidentifiable information, confidential business information, or other identifiable 

records? How can identifiable information be best protected to ensure the privacy 

and confidentiality of individual or business data in a clearinghouse? 

EPIC addresses this point to stress that a data clearinghouse should not be a central 

repository of data. A central database would increase the risk of data breach and insider misuse. 

It would also be more likely to lead to the kinds of perceptions that led to the demise of the 1965 

National Data Center. 

The 2015 data breaches at the Office of Personnel Management (OPM), which 

compromised the personal data of 21.5 million people, including 1.8 million people who did not 

 

40 Id. at 2. 

41 Id. at 2-3. 

42 IBM’s Jeff Jonas on Baking Data Privacy into Predictive Analytics, Data Informed, Nov. 20, 2013,

http://data-informed.com/ibms-jeff-jonas-baking-data-privacy-predictiveanalytics/#sthash.hBM0lg1N.dpuf 

1636 The Promise of Evidence-Based Policymaking
Commission on Evidence-based Comments of EPIC 

Policymaking: RFC November 14, 2016 

13 

apply for background checks,43 illustrate the dangers of holding administrative and survey data in 

a single location. The OPM breach exposed sensitive background investigation data spanning 

three decades.44 OPM warns on its website: 

If you underwent a Federal background investigation in 2000 or afterwards 

(which occurs through the submission of forms SF-86, SF-85, or SF-85P for 

either a new investigation or a reinvestigation), it is highly likely that you are 

impacted by the incident involving background investigations. If you underwent a 

background investigation prior to 2000, you still may be impacted, but it is less 

likely.45

 

The fingerprints of 5.6 million people were also stolen in the data breach.46 

Though it may be difficult to imagine, the OPM breach could have been worse if the 

OPM had held the disparate types of information contemplated in a clearinghouse of 

administrative and survey data. The more information a database holds, and the more 

information that resides in the same place, the greater the amount of information that will be 

disclosed in a breach. 

Unauthorized insider access is also a greater threat when data sets are combined into a 

central location. Criminal dockets contain numerous examples of government employees prying 

for entertainment or profit. Police officers and deputy sheriffs,47 customs officers,48 corrections 

 

43 Dan Goodin, Call it a “Data Rupture”: Hack Hitting OPM Affects 21.5 Million, ARS TECHNICA (July 9, 

2015), http://arstechnica.com/security/2015/07/call-it-a-data-rupture-hack-hitting-opm-affects-21-5-

million/. 

44 Andrea Shalal & Matt Spetalnick, Data Hacked from U.S. Government Dates Back to 1985: U.S. 

Official, REUTERS (June 5, 2015), http://www.reuters.com/article/us-cybersecurity-usaidUSKBN0OL1V320150606. 

45 Office of Personnel Management, Cybersecurity Resource Center, 

https://www.opm.gov/cybersecurity/cybersecurity-incidents/. 46 Andrea Peterson, OPM Says 5.6 Million Fingerprints Stolen in Cyberattack, Five Times as Many as 

Previously Thought, WASH. POST (Sep. 23 2015), https://www.washingtonpost.com/news/theswitch/wp/2015/09/23/opm-now-says-more-than-five-million-fingerprints-compromised-in-breaches/. 

47 See, e.g., United States v. Black, No. 1:14-cr-00012 (D. Colo. 2014) (running license plates against 

motor vehicle databases to help a drug-dealing relative determine whether certain vehicles were 

unmarked police cars); United States v. Cave, No. 8:12-cr-00417 (D. Neb. 2013) (running state Criminal 

Justice Information Systems (CJIS) searches on behalf of car dealerships seeking to repossess vehicles); 

United States v. Nowlin, No. 1:12-cr-00513 (D. Md. 2013) (police officer accessing a motor vehicle 

The Promise of Evidence-Based Policymaking 1637
Commission on Evidence-based Comments of EPIC 

Policymaking: RFC November 14, 2016 

14 

officers,49 Veterans Administration employees,50 Social Security Administration employees,51

and many IRS employees52 have been convicted of access to data for unauthorized purposes. If a 

clearinghouse of administrative and survey data is created, it is a certainty that someone will look 

at it despite criminal penalties for doing so. If the data is spread out among the custodial agencies, 

inaccessible to a single login, the risk of disclosure from insider prying will be minimized. 

Finally, the specter of a single database collecting all the government’s data about a 

person is exactly the kind of proposal that led to the demise of the National Data Center and the 

enactment of the Privacy Act of 1974. Even if data is de-identified—and de-identification would 

be much more difficult when all data is collected together instead of subsets—many will fear, 

justifiably so, the uses that such a database might be put to. 

4. Conclusion 

The use of administrative and survey data has great potential for informed, fact-based 

policymaking. But it also has the potential to harm privacy and liberty interests. EPIC asks the 

Commission to encourage the development and use of PETs, including data minimization and 

robust de-identification of data, in any plan for a data clearinghouse. EPIC also urges the 

Commission to adopt data use schemes that leave the data with the custodial agencies instead of 

a central repository. 

 

database on behalf of a drug dealer); United States v. Green, No. 4:10-cr-00059 (S.D. Tex. 2010) 

(sheriff’s deputy selling information from the National Crime Information Center (NCIC) database). 

48 See, e.g., United States v. Ben-Shabat, No. 4:09-cr-02180 (D. Ariz. 2010) (customs officer accessing 

databases to gather information on a company with whom the officer was involved in a legal dispute); 

United States v. Yanez-Camacho, No. 3:09-cr-02755 (S.D. Cal. 2009). 

49 See, e.g., United States v. Barone, No. 3:08-cr-00174 (D. Conn. 2009). 

50 See, e.g., United States v. Dubree, No. 1:09-cr-00067 (D. Md. 2009) (Veterans Administration 

employee accessing a co-worker’s medical records). 

51 See, e.g., United States v. Wilson, No. 1:09-cr-00662 (D. Md. 2010) (accessing Social Security 

Administration records for the information necessary to take out a credit card in someone else’s name). 

52 See, e.g., United States v. Harris, No. 5:14-cr-00120 (N.D. Tex. 2015); United States v. Krien, No. 

2:08-cr-20148 (Kan. 2009); United States v. Supple, No. 3:08-cr-00029 (D. Conn. 2008); United States v. 

Orr, No. 2:07-cr-00016 (E.D. Ky. 2007); United States v. Jones, No. 1:06-cr-00169 (W.D. Mo. 2006). 

1638 The Promise of Evidence-Based Policymaking
Commission on Evidence-based Comments of EPIC 

Policymaking: RFC November 14, 2016 

15 

 Respectfully Submitted, 

 /s/ Marc Rotenberg 

 Marc Rotenberg 

 EPIC President and Executive Director 

 

 /s/ James Graves 

 James Graves 

 EPIC Law and Technology Fellow 

The Promise of Evidence-Based Policymaking 1639
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t1b-ka5q

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0150

Comment on FR Doc # 2016-22002

Submitter Information

Name: Stanley Hartzler

Address:

Luling, TX, 78648

Email: HartzlerSJ@gmail.com

General Comment

I am resolutely opposed to the work of CEP. This is Big Brother and Brave New World

on rocket fuel or speed or steroids - - take your pick.

1640 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t1b-5ia7

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0151

Comment on FR Doc # 2016-22002

Submitter Information

Name: John Byrd

General Comment

See attached file from MAPPS, the national association of mapping and geospatial firms.

Docket ID USBC-2016-0003

"Commission on Evidence-Based Policymaking Comments"

Attachments

MAPPS Comments to Census Bureau for the Commission on Evidence-Based

Policymaking (11-14-2016)

The Promise of Evidence-Based Policymaking 1641
MAPPS Comments to the Bureau of the Census for the 

Commission on Evidence-Based Policymaking 

Docket ID USBC-2016-0003 

"Commission on Evidence-Based Policymaking Comments" 

Formed in 1982, MAPPS (www.mapps.org) is the only national association exclusively comprised of 

private sector firms in the remote sensing, spatial data and geographic information systems field in the 

United States. The MAPPS membership spans the entire spectrum of the geospatial community, including 

Member Firms engaged in satellite and airborne remote sensing, surveying, photogrammetry, aerial 

photography, LIDAR, hydrography, bathymetry, charting, aerial and satellite image processing, GPS, and 

GIS data collection and conversion services. MAPPS also includes Associate Member Firms, which are 

companies that provide hardware, software, products and services to the geospatial profession in the 

United States and other firms from around the world. Independent Consultant Members are sole 

proprietors engaged in consulting in or to the geospatial profession, or provides a consulting service of 

interest to the geospatial profession. MAPPS provides its member firms opportunities for networking and 

developing business-to-business relationships, information sharing, education, public policy advocacy, 

market growth, and professional development and image enhancement. 

MAPPS Recommendation:

The Commission on Evidence-Based Policymaking should establish a priority on Federal government 

collection and application of geospatial data, particularly the National Spatial Data Infrastructure (NSDI), 

to address national policies and priorities. 

Background 

The United States is a world economic power in spite of its surveying and mapping, not because of it. 

According to the Congressional Research Service (CRS) and the Federal Geographic Data Committee 

(FGDC), there are estimates that as much as 90% of government information has a geospatial component. 

Executive Order 12906, issued in 1994, called for the development of the NSDI, but it has languished. 

Accurate surveying and mapping, via the NSDI, can be a matter of life or death. 

An independent review team tasked by the Coalition for Geospatial Organizations (COGO) gave the 

NSDI a report card grade of "C" in 2014. As a result, the United States would rank behind some 15 other 

countries. That’s the conclusion of the former Wyoming Governor Jim Geringer who pointed to the 

United Arab Emirates as an example of a nation that has an excellent spatial data infrastructure and uses it 

in almost every government decision and program. 

According to Geringer, NSDI is “not complete and not well governed” and called for a “move into a

coordinated and integrated data set.”

John M. Palatiello, Executive Director 

1856 Old Reston Avenue, Suite 205, Reston, Virginia 20190 

P (703) 787-6996 F (703) 787-7550 E info@mapps.org www.mapps.org 

1642 The Promise of Evidence-Based Policymaking
PAGE 2 - MAPPS Comments 

The COGO report card encourages government agencies to improve the NSDI to better support efficient 

government operations at all levels. “The goal of the report card evaluation is to bring attention to the 

need for current and accurate geospatial data for the nation,” he said. As governor, geospatial framework 

data was “my reference” when making policy or doing analysis, Geringer explained. “You don’t even 

know you’re making an erroneous assumption,” when using dated, incomplete or inaccurate spatial data. 

That “impacts the quality of a decision.” He went on to say spatial data affects “decision-making, policy 

and economics.”

Why is our nation’s surveying and mapping infrastructure as important as its roads, bridges, airports, 

waterworks and other physical infrastructure? Let’s look at the ways:

x The Affordable Care Act, commonly known as “Obamacare,” includes 814 provisions requiring 

location/geographic/place-based data for implementation. The lack of data, or a geospatial 

management office in the Department of Health and Human Services, could be a factor in the 

failure of websites, exchanges and other methods of delivering quality, affordable medical 

attention to those in need. 

x The federal government wastes $2 billion a year on some 77,000 unneeded buildings, and the 

Government Accountability Office (GAO) cites the fact Uncle Sam lacks a current, accurate 

inventory of the land and buildings it owns, finding existing data is “unreliable and of limited 

usefulness” and “not current or reliable.”

x The lack of uniform national parcel data in United States means no government agency could 

properly track real estate trends or access an “early warning system” that could have prevented, or 

at least minimized, the trillion-dollar mortgage foreclosure crisis. 

x Based on data from NOAA's Digital Coast project, rising sea levels threaten coastal watershed 

counties that are home to 163.8 million Americans — approximately 52 percent of the nation’s 

population — with the number expected to increase by more than 15 million by 2020. However, 

accurate data and integrated information to enable coastal communities to address many climate, 

environmental and emergency management issues does not exist. There is no accurate shoreline 

surveying and mapping data to measure, monitor, verify or validated the alleged effects of climate 

change. This seriously affects the coastal zone, which is the home of over half of the nation’s 

economic productivity. 

x When Congress sought to reauthorize MAP-21, the Federal highway law enacted in 2012, as well 

as the recently enacted FAST Act, essential were and continue to be surveying, mapping and 

other location-based services to plan, design, inventory, assess, operate and maintain highways 

and transit systems. Vehicle to vehicle (V2V) or “connected vehicle” technology to enable 

vehicles to communicate potential risks to drivers and avoid rear-end, lane change and 

intersection crashes requires accurate spatial data. The transportation layer received the COGO 

report card’s lowest grade of D. MAP-21 Reauthorization, via the FAST act, provided an 

opportunity for Congress to not only leverage investments and introduce new geospatial 

technology, data, products and services, but also to reduce costs, and enhance safety and 

efficiency in our nation’s transportation systems.

x Pipelines in the United States could encircle Earth 25 times. It is estimated an underground utility 

line is hit somewhere in the nation every 60 seconds. There were approximately 335,000 

The Promise of Evidence-Based Policymaking 1643
PAGE 3 - MAPPS Comments 

underground excavation damages in 2013. Improved underground infrastructure location data 

would enhance public safety, environmental protection and the economy. President Obama in 

June signed into law the "Protecting our Infrastructure of Pipelines and Enhancing Safety" or 

"PIPES Act" of 2016 as Public Law 114-183. The law extends the Pipeline and Hazardous 

Materials Safety Administration (PHMSA), makes changes to PHMSA safety policies and gives 

the Department of Transportation more power in the event of pipeline emergencies. The law also 

includes an issue promoted by MAPPS to ensure coordination and collaboration on pipeline 

mapping, research, development, and technology between PHMSA, industry, and public 

stakeholders, including provisions strengthening geolocation data for pipelines and other 

underground utility infrastructure through enhanced underground utility location data 

requirements. The Department of Energy's (DOE) Quadrennial Energy Review also highlighted 

pipeline safety as an issue for the nation's energy infrastructure. Trends in pipeline accidents 

suggest there continues to be opportunity for safety improvement. The DOE and PHMSA have 

since announced an interagency task force to "initiate regulatory actions to help ensure the safety 

of natural gas storage facilities." "Improving data collection and reporting, including geospatial 

data" was one area highlighted in the January 2015 study on integrity management by the 

National Transportation and Safety Board (NTSB). In September 2015, NTSB Chairman 

Christopher Hart testified before the Senate Commerce Committee emphasizing the importance 

of, and increased need for, geospatial data collection as part of improving location data and 

"integrity management" for underground utilities and pipelines connected to PHMSA 

Reauthorization. 

x The federal government’s National Flood Insurance Program (NFIP) is at least $24 billion in debt 

to U.S. taxpayers. These losses are in part due to inadequate mapping data, and result in frequent 

flooding or unwise construction. Current, accurate elevation and structure data, and better use of 

surveying technology, would help bring fairness, loss prevention and lower costs to NFIP. 

Fortunately to help address this problem in a coordinated process, FEMA and several other 

Federal agencies are working closely to collect enhanced elevation data nationwide via the US 

Geological Survey's 3D Elevation Program, commonly known as 3DEP. 

x In 2015, NBC News reported on technical flaws in E-911 systems resulting in inaccurate location 

and untimely dispatching of ambulances and emergency medical personnel. 

MAPPS Conclusion:

To reiterate, the Commission on Evidence-Based Policymaking should establish a priority on Federal 

government collection and application of geospatial data, particularly the NSDI, to address national 

policies and priorities. 

1644 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t1b-14k9

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0152

Comment on FR Doc # 2016-22002

Submitter Information

Name: John Byrd

General Comment

Please find the attachment from the National Society of Professional Surveyors (NSPS).

Docket ID USBC-2016-0003

"Commission on Evidence-Based Policymaking Comments"

Attachments

NSPS Comments to Census Bureau for the Commission on Evidence-Based

Policymaking (11-14-2016)

The Promise of Evidence-Based Policymaking 1645
 National Society of Professional Surveyors

 5119 Pegasus Court, Suite Q, Frederick, MD 21704 

 Phone: 240-439-4615 * Fax: 240-439-4952 

 Curtis W. Sumner, PLS, Executive Director 

 www.nsps.us.com

NSPS Comments to the Bureau of the Census for the 

Commission on Evidence-Based Policymaking 

Docket ID USBC-2016-0003 

"Commission on Evidence-Based Policymaking Comments" 

Having incorporated the American Congress on Surveying and Mapping (ACSM) within the National Society of 

Professional Surveyors ((NSPS) www.nsps.us.com) in 2012, NSPS is the preeminent national professional society 

working in affiliation with the respective state societies of professional surveyors and along with persons trained, 

registered, or interested in the profession of surveying and mapping. NSPS advances the sciences and disciplines within 

the profession and strives to establish and further common interests, objectives, and political efforts to help bind the 

surveying profession into a unified body in the United States. NSPS is the interface for the surveying profession with a 

multitude of national organizations in related fields, and with federal government entities. 

NSPS believes the Commission on Evidence-Based Policymaking should establish a priority on Federal government 

collection and application of geospatial data, particularly the National Spatial Data Infrastructure (NSDI), to address 

national policies and priorities. 

According to studies by the Congressional Research Service (CRS), Federal Geographic Data Committee (FGDC), and 

others, it is estimated that as much as 90% of government information has a geospatial component. Executive Order 

12906, issued by President Clinton in 1994, (and renewed by President George W. Bush in section 25 of Executive Order 

13286) called for the development of the NSDI. 

Federal agencies, acting through the FGDC, are responsible for coordinating the development and use of geospatial data, 

pursuant to Office of Management and Budget Circular A-16. As noted in numerous Government Accountability Office 

(GAO) reports, as recently as 2015, this process has numerous and serious shortcomings. Several Congressional hearings 

have come to the same conclusion. 

An independent review conducted by the Coalition for Geospatial Organizations (COGO) produced a “report card” that 

assigned an overall grade of "C" to the NSDI in 2014. 

Decision-making, policy, economics and other important governmental activities are dependent on place-or-locationbased geographic information. Such spatial or geospatial data are critical to effective evidence-based policymaking. 

A spatial data infrastructure is as important to the commerce and governance of our Nation as are the roads, bridges, 

airports, waterworks and other physical infrastructure. 

National issues and priorities as varying as implementation of the Affordable Care Act, better utilization of government 

land and buildings, rising sea level to new vehicle to vehicle (V2V) or “connected vehicle” technology, pipeline and other 

underground utility location to the federal government’s National Flood Insurance Program (NFIP), emergency response 

to E-911 systems are all dependent on current, accurate geospatial data. 

NSPS respectfully urges the Commission on Evidence-Based Policymaking establish a priority on Federal government 

collection and application of geospatial data, particularly the NSDI and data collected by professional surveyors licensed 

by the states, to address national policies and priorities. 

1646 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t1b-o2gn

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0153

Comment on FR Doc # 2016-22002

Submitter Information

Name: Jennifer Goodpaster

General Comment

My child is under 18, and the government does not have the right to track him as a

minor. Please shut down the National Student-unit record system.

The Promise of Evidence-Based Policymaking 1647
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t1c-zb1k

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0154

Comment on FR Doc # 2016-22002

Submitter Information

Name: mandy hampson

General Comment

Against efforts to establish a federal "unit record" database that would create an

individual data dossier of college students linked to their employment, with the strong

likelihood that it would be connected to the K-12 longitudinal system. 

Students information should have privacy!!!

1648 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t1c-4u5s

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0155

Comment on FR Doc # 2016-22002

Submitter Information

Name: David Holt

Address:

Houston, TX, 77098

Email: dholt@consumerenergyalliance.org

General Comment

See attached file(s)

Attachments

CEA Comments on Evidence-Based Policymaking

The Promise of Evidence-Based Policymaking 1649
 

 2211 Norfolk Street, Suite 410 

Houston, Texas 77098 

P 713 337 8800 

F 866 273 8998 

November 14, 2016 

Nick Hart 

Policy and Research Director 

Commission on Evidence-Based Policymaking 

Bureau of the Census 

U.S. Department of Commerce 

4600 Silver Hill Road 

Washington, D.C. 20233 

RE: Request for Comments on Evidence-Based Policymaking (USBC-2016-0003) 

Dear Mr. Hart: 

Consumer Energy Alliance (CEA) welcomes the opportunity to provide comments on the Commission on 

Evidence-Based Policymaking’s (Commission) examination of how to strengthen evidence-building to 

inform program and policy design and implementation. 

As the Voice of the Energy Consumer, CEA is a nationwide association of energy consumers who 

advocate for balanced policies that support access to affordable, reliable energy. In addition to our 

nearly 300 company and association members that represent nearly every sector of the U.S. economy, 

CEA’s membership includes more than 400,000 individual citizens across the country. 

The United States is home to abundant natural resources that can secure the nation’s long-term energy 

security, to the benefit of families and small businesses across the country. However, policy decisions 

based on undue precaution, speculation about risks, theoretical scenarios involving mere possibilities, or 

inadequate cost-benefit analysis threaten to unnecessarily block this historic opportunity. 

To that end, policies and regulations based on evidence are necessary for the nation to best realize the 

economic, social, and environmental benefits associated with the development of this critical domestic 

energy, and CEA requests that the Commission address the importance of evidence-based federal 

policies and regulations as they pertain specifically to energy activities and projects and their purported 

environmental and societal implications. 

In seeking public input, among other things, the Commission has requested feedback on how data, 

statistics, results of research, and findings from evaluation can be best used to improve policies and 

programs (Question 16). Regrettably, recent federal actions related to domestic energy leasing, 

development, and transportation demonstrate that such information is insufficiently considered, if 

considered at all. 

For example, the U.S. Interior Department earlier this year announced its decision to remove its 

previously-proposed Mid/South Atlantic oil and natural gas lease sale from any further consideration, 

1650 The Promise of Evidence-Based Policymaking
 

citing a variety of factors that were not supported by the record.1 Similarly, important questions have 

been raised about the extent to which energy regulations may not further and could even inhibit safety 

and environmental protection,2 as well as the sufficiency of cost-benefit analysis for major proposed 

rules.3 In one instance that some have indicated could be a solution in search of a problem, an overhaul 

of air quality regulations pertaining to offshore energy is being proposed without a clear demonstration 

that revised rules are necessary.4

In yet another example, even though the federal government has acknowledged that seismic 

exploration surveys have taken place for decades without any evidence of adverse impacts on marine 

animal populations, commercial fishing, or coastal communities,5 applications for permits and related 

approvals to conduct seismic studies to better understand the Atlantic energy resource base have been 

inexplicably pending with the Interior and Commerce Departments since being requested as far back as 

2014.6 

Troublingly, the Interior Department has recently proposed new restrictions that could impact the ability 

to conduct seismic surveys in the Gulf of Mexico -- to the detriment of conventional and renewable 

energy development and coastal restoration activities – even though the Interior Department’s analysis 

has found that there would not be any added overall benefits.7 

 

1 See 2017-2022 Outer Continental Shelf Oil and Gas Leasing Proposed Program, March 2016, available at https://www.boem.gov/2017-2022-

Proposed-Program-Decision/, and “Interior Department Announces Next Step in Offshore Oil and Gas Leasing Planning Process for 2017-2022,” 

March 15, 2016, available at https://www.doi.gov/pressreleases/interior-department-announces-next-step-offshore-oil-and-gas-leasingplanning-process (alleging insurmountable conflicts with defense, fishing, and tourism activities, widespread public opposition). 2 See e.g. U.S. Bureau of Safety and Environmental Enforcement, Well Control Rule, available at https://www.bsee.gov/guidance-andregulations/regulations/well-control-rule, and “Cassidy Comments on Expected Well Control Rule,” available at

http://www.cassidy.senate.gov/newsroom/press-releases/cassidy-comments-on-expected-well-control-rule. See also U.S. Bureau of Safety and 

Environmental Enforcement, Arctic Drilling Rule, available at https://www.bsee.gov/guidance-and-regulations/regulations/arctic-rule, and 

National Petroleum Council’s 2015 Report on “Arctic Potential: Realizing the Promise of U.S. Arctic Oil and Gas Resources,” available at

http://npcarcticpotentialreport.org/pdf/AR-Executive_Summary-Final.pdf. 3 See e.g. U.S. Bureau of Safety and Environmental Enforcement Proposed Rule on Oil and Gas and Sulphur Operations in the Outer Continental 

Shelf –Blowout Preventer Systems and Well Control (estimating ~$883 million in total costs over 10 years), available at

https://www.gpo.gov/fdsys/pkg/FR-2015-04-17/pdf/2015-08587.pdf, and Quest Offshore and Blade Energy Partners’ BSEE Proposed Well 

Control Rule Cost and Economic Analysis (estimating over $32 billion in costs over 10 years), available at 

www.regulations.gov/contentStreamer?documentId=BSEE-2015-0002-

0154&attachmentNumber=1&disposition=attachment&contentType=pdf. See also U.S. Bureau of Ocean Energy Management Proposed Rule 

on Air Quality Control, Reporting, and Compliance (estimating $290 million in costs over 10 years), available at 

https://www.gpo.gov/fdsys/pkg/FR-2016-04-05/pdf/2016-06310.pdf, and Joint Trades Comments on Air Quality Control, Reporting and 

Compliance (estimating over $3.4 billion in costs over 10 years), available at

https://www.regulations.gov/contentStreamer?documentId=BOEM-2013-0081-

0074&attachmentNumber=1&disposition=attachment&contentType=pdf. 4 See U.S. Bureau of Ocean Energy Management Proposed Rule on Air Quality Control, Reporting, and Compliance, available at 

https://www.gpo.gov/fdsys/pkg/FR-2016-04-05/pdf/2016-06310.pdf, and June 20, 2016 Comment Letter from Texas Commission on 

Environmental Quality Executive Director Richard Hyde to the U.S. Bureau of Ocean Energy Management, available at

https://www.regulations.gov/contentStreamer?documentId=BOEM-2013-0081-

0080&attachmentNumber=1&disposition=attachment&contentType=pdf (“The BOEM should withdraw the proposed rule until it has 

completed the appropriate studies to determine whether the revised regulations are necessary. The BOEM has not completed the necessary 

studies to determine whether sources in the OCS actually have an air quality impact on states with regard to the current NAAQS. The studies 

the BOEM intends to use for establishing emission exemption thresholds (EET) may inform many aspects of the proposed rule or even call into 

question whether the proposed rule revisions are actually needed. Such information is necessary for affected lessees and operators, potentially 

impacted states, and other interested parties to adequately comment on the BOEM’s proposal. The BOEM should withdraw the proposed rule 

until all studies are completed in order to justify its proposed actions.”). 

5 See U.S. Bureau of Ocean Energy Management Science Notes, August 22, 2014, available at https://www.boem.gov/BOEM-Science-NoteAugust-2014/. 6 See U.S. Bureau of Ocean Energy Management, Currently submitted Atlantic OCS Region Permits, available at

https://www.boem.gov/Currently-submitted-Atlantic-OCS-Region-Permits/. 7 See U.S. Bureau of Ocean Energy Management, Draft Programmatic Environmental Impact Statement for Geological and Geophysical Activities 

on the Gulf of Mexico Outer Continental Shelf, available at https://www.boem.gov/Gulf-of-Mexico-Geological-and-Geophysical-ActivitiesProgrammatic-EIS/#Draft. 

The Promise of Evidence-Based Policymaking 1651
 

Another recent action of note includes the proposed expansion of the Flower Garden Banks National 

Marine Sanctuary, which contradicts the recommendation of the Sanctuary Advisory Council8 and which 

was proposed without having accounted for the area’s undiscovered resource potential (among other 

things), even though the U.S. Bureau of Ocean Energy Management had specifically requested that such 

analysis be conducted in order to better understand the full range of potential impacts of the proposed 

Sanctuary expansion.9 

Other federal decisions pertaining to the Keystone XL and Dakota Access pipelines, Clean Power Plan, 

proposed Waste Prevention, Production Subject to Royalties, and Resource Conservation Rule (Venting 

and Flaring Rule), and postponement of the October 19, 2016 oil and gas lease sale near Chaco Culture 

National Historical Park all serve to highlight the failure to adequately consider the impact of federal 

decisions on consumers. In addition, they also fail to sufficiently assess environmental costs that could 

be incurred as a result of a greater reliance on alternative mechanisms and sources for transporting and 

producing energy. 

All of the examples described above illustrate the need for more evidence-based decision-making, 

including through the use of data, statistics, research results, and findings from evaluation for 

improving policies, programs, and regulations. 

Furthermore, recognition of the need to best use such information and the myriad of laws that require 

use of the best available science both underscore the importance of avoiding decisions based on mere 

speculation through the improper use and application of approaches like the precautionary principle. 

Evidence-based decision-making presents an opportunity to avoid the improper use and application of 

approaches such as the precautionary principle by helping to ensure that decisions are made based on 

scientific information grounded in reason and practicality and more rational, effective, and realistic 

cost-benefit analysis. 

As to the Commission’s inquiry on the extent to which data held by government agencies should be 

made available to qualified researchers and institutions (Question 10), the examples described above 

also highlight the critical importance of ensuring that such information is made available to the public 

– not just to qualified researchers and institutions – for review and comment, which will enable more 

informed public comments that in turn facilitate better-informed decisions. 

Federal decisions related to energy significantly impact American families, citizens, and businesses 

across the economic spectrum. Thus, in response to Question 17 on the extent to which program and 

policy evaluation can or should be addressed in program designs, it is vital that metrics designed to 

evaluate the impacts of federal decisions to be made pertaining to energy be proposed and subject to 

public review and comment at the earliest stages of the development of the applicable policy, 

program, or regulation. Developing such metrics in an open manner early on will also significantly aid 

the effectiveness of retrospective reviews of previously-issued regulations. 

 

8 See Flower Garden Banks Proposed Sanctuary Expansion, available at http://flowergarden.noaa.gov/management/expansiondeis.html. 9 See August 5, 2016 Letter from U.S. Bureau of Ocean Energy Management Gulf of Mexico OCS Region Regional Director Michael Celata to 

NOAA Office of National Marine Sanctuaries Southeast Atlantic, Gulf of Mexico, and Caribbean Region Regional Director Dr. Billy Causey, 

available at https://www.regulations.gov/contentStreamer?documentId=NOAA-NOS-2016-0059-

1305&attachmentNumber=1&disposition=attachment&contentType=pdf. 

1652 The Promise of Evidence-Based Policymaking
 

As noted in the House Report that accompanied the law authorizing and directing this Commission’s 

work, “[w]ithout evidence, the federal government is an ineffective fiduciary on behalf of the taxpayer. 

Unfortunately, in many instances, federal decision-makers do not have access to the data necessary to 

best inform decisions. In such instances, agencies are unable to show the benefits or impacts of the 

programs they administer and cannot determine what, if any, unintended consequences are created by 

programs, or whether programs can be improved.”10

In that regard, and in light of the significant and adverse impact that ill-informed decisions based on 

undue precaution, risk speculation, theoretical scenarios, and inadequate cost-benefit analysis can have 

on the energy and economic security of American families, citizens, and businesses, CEA respectfully 

urges the Commission to address these important issues in its findings and recommendations to be 

developed for Congress and the President. 

Sincerely, 

David Holt 

President 

 

Cc: 

The Honorable Paul Ryan, Speaker of the U.S. House of Representatives 

The Honorable Mitch McConnell, U.S. Senate Majority Leader 

The Honorable Rob Bishop, Chairman, U.S. House Natural Resources Committee 

The Honorable Lisa Murkowski, Chairman, U.S. Senate Energy and Natural Resources Committee 

The Honorable Fred Upton, Chairman, U.S. House Energy and Commerce Committee 

The Honorable Harold Rogers, Chairman, U.S. House Appropriations Committee 

The Honorable Thad Cochran, Chairman, U.S. Senate Appropriations Committee 

The Honorable Jim Inhofe, Chairman, U.S. Senate Environment and Public Works Committee 

The Honorable Jason Chaffetz, Chairman, U.S. House Oversight & Government Reform Committee 

The Honorable Ron Johnson, Chairman, U.S. Senate Homeland Security & Governmental Affairs 

Committee 

 

10 See H. Rept. 114-211, available at https://www.congress.gov/114/crpt/hrpt211/CRPT-114hrpt211.pdf. 

The Promise of Evidence-Based Policymaking 1653
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t1d-loin

Comments Due: December 14, 2016

Submission Type: API

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0156

Comment on FR Doc # 2016-22002

Submitter Information

Name: Emmanuel Saez

Address:

Berkeley, CA, 94720

Email: saez@econ.berkeley.edu

General Comment

Authorizing research on evidence-based policy making using tax data

Federal income tax data are uniquely valuable for research that would contribute to

evidence-based policy making in the United States. Tax data provide comprehensive

annual measures of many important outcomes necessary to evaluate the impacts of

policies. These data also have a natural longitudinal structure, with 20 years of

population-wide data currently available, so they permit both short- and long-run

analyses (including of the effects of policies aimed at parents on the long-run outcomes

of children). 

The use of administrative tax data for research purposes is governed by 26 U.S. Code

6103 - Confidentiality and disclosure of returns and return information (hereafter Section

6103). Section 6103 imposes strict data protections, by specifying who can access the

data and for which purpose. Currently, most research that contributes to evidence-based

policy occurs under subsections (h) and (n), which allow the use of tax data for tax

administration purposes, including the development and formulation of Federal tax

1654 The Promise of Evidence-Based Policymaking
policy relating to existing or proposed internal revenue laws, related statutes, and tax

conventions (6103(b)(4)(A)(ii)). More specifically, subsection (h) allows such use by US

Treasury employees, while subsection (n) allows similar use by other authorized persons.

The current definition of tax administration unnecessarily limits the scope for research

that is highly relevant to economic policy. It is of course reasonable to impose some

limits on the topics of research that use such data, but the current definition is both

arbitrary and too narrow. It is arbitrary because many policies that currently fall outside

the limits could be administered (in part) through the tax system. It is too narrow because

many policies that could not be administered through the tax system are critical to

understanding both spending and revenue projections, and there is no reason to

specifically encourage research on some policies while barring research on other, equally

important policies simply because of the government agency responsible for

implementing them. For instance, research studying the labor supply effects of programs

such as the Supplemental Nutrition Assistance Program (SNAP), Medicaid, or Medicare

which are not administered through the Federal income tax code would provide valuable

evidence for policymaking but falls outside the current definition.

We therefore recommend amending the definition of tax administration in Section 6103

to further include explicitly the analysis of the effects of proposed and existing laws on

federal tax revenue. This simple change would authorize evidence-based policy research

more broadly. Because the tax data are superior to any other data, this simple

amendment could be transformative for research on evidence-based policy in the United

States, while fully maintaining all the existing thorough protections for the data,

including the prohibition against public disclosure in a form that would allow

identification of a particular taxpayer.

Other simple changes to the tax code would also accomplish the goal of explicitly

authorizing evidence-based policy research for a broader range of policies, as well as

improve the functioning of other governmental statistical functions. For instance,

subsection 6103(j)(1) Statistical Use allows IRS to provide the Bureau of Economic

Analysis (BEA) and the Census Bureau with a few narrow categories of variables only to

the extent necessary in the structuring of censuses and national economic accounts and

conducting related statistical activities authorized by law. Broadening this authority to

include data for more general purposes would not only produce more research for

evidence-based policy but could also result in better statistics. Similarly, Section 6108

Statistical Publications and Studies allows for special statistical studies or compilations,

which might encompass evidence-based policy research, but work on those studies is

generally limited to Treasury employees. Broadening this authority to include authorized

other persons outside Treasury as in subsection 6103(n) is another route to expanding

evidence-based policy research using federal income tax data.

Alan Auerbach, University of California-Berkeley

David Card, University of California-Berkeley

Raj Chetty, Stanford University

Amy Finkelstein, Massachusetts Institute of Technology

The Promise of Evidence-Based Policymaking 1655
John Friedman, Brown University

Nathan Hendren, Harvard University

Lawrence Katz, Harvard University

Emmanuel Saez, University of California-Berkeley

Danny Yagan, University of California-Berkeley

Attachments

comments-commission_final

1656 The Promise of Evidence-Based Policymaking
Request for comments from Evidence-Based Policymaking Commission

https://www.federalregister.gov/documents/2016/09/14/2016-22002/requestfor-comments-for-the-commission-on-evidence-based-policymaking

Authorizing research on evidence-based policy making using tax data 

Federal income tax data are uniquely valuable for research that would contribute to 

evidence-based policy making in the United States. Tax data provide comprehensive 

annual measures of many important outcomes necessary to evaluate the impacts of 

policies. These data also have a natural longitudinal structure, with 20 years of 

population-wide data currently available, so they permit both short- and long-run 

analyses (including of the effects of policies aimed at parents on the long-run 

outcomes of children).1 

The use of administrative tax data for research purposes is governed by 26 U.S. Code 

§ 6103 - Confidentiality and disclosure of returns and return information (hereafter 

Section 6103). Section 6103 imposes strict data protections, by specifying who can 

access the data and for which purpose. Currently, most research that contributes to 

evidence-based policy occurs under subsections (h) and (n), which allow the use of 

tax data for “tax administration” purposes, including “the development and 

formulation of Federal tax policy relating to existing or proposed internal revenue 

laws, related statutes, and tax conventions” (6103(b)(4)(A)(ii)). 2

The current definition of “tax administration” unnecessarily limits the scope for 

research that is highly relevant to economic policy. It is of course reasonable to 

impose some limits on the topics of research that use such data, but the current 

definition is both arbitrary and too narrow. It is arbitrary because many policies 

that currently fall outside the limits could be administered (in part) through the tax 

system. It is too narrow because many policies that could not be administered 

through the tax system are critical to understanding both spending and revenue 

projections, and there is no reason to specifically encourage research on some 

policies while barring research on other, equally important policies simply because 

of the government agency responsible for implementing them. For instance, 

research studying the labor supply effects of programs such as the Supplemental 

Nutrition Assistance Program (SNAP), Medicaid, or Medicare – which are not 

administered through the Federal income tax code – would provide valuable 

evidence for policymaking but falls outside the current definition. 

We therefore recommend amending the definition of “tax administration” in 

Section 6103 to further include explicitly “the analysis of the effects of 

proposed and existing laws on federal tax revenue.” This simple change would 

 

1 Population wide individual tax return data are available since 1996. Most information returns start 

in 1999. 

2 More specifically, subsection (h) allows such use by US Treasury employees, while subsection (n) 

allows similar use by other authorized persons. 

The Promise of Evidence-Based Policymaking 1657
authorize evidence-based policy research more broadly. Because the tax data are 

superior to any other data, this simple amendment could be transformative for 

research on evidence-based policy in the United States, while fully maintaining all 

the existing thorough protections for the data, including the prohibition against 

public disclosure in a form that would allow identification of a particular taxpayer. 

Other simple changes to the tax code would also accomplish the goal of explicitly 

authorizing evidence-based policy research for a broader range of policies, as well 

as improve the functioning of other governmental statistical functions. For instance, 

subsection 6103(j)(1) “Statistical Use” allows IRS to provide the Bureau of Economic 

Analysis (BEA) and the Census Bureau with a few narrow categories of variables 

“only to the extent necessary in ... the structuring of censuses and national economic 

accounts and conducting related statistical activities authorized by law.” 

Broadening this authority to include data for more general purposes would not only 

produce more research for evidence-based policy but could also result in better 

statistics. Similarly, Section 6108 “Statistical Publications and Studies” allows for 

special statistical studies or compilations, which might encompass evidence-based 

policy research, but work on those studies is generally limited to Treasury 

employees. Broadening this authority to include authorized “other persons” outside 

Treasury – as in subsection 6103(n) – is another route to expanding evidence-based 

policy research using federal income tax data. 

Alan Auerbach, University of California-Berkeley 

David Card, University of California-Berkeley 

Raj Chetty, Stanford University 

Amy Finkelstein, Massachusetts Institute of Technology 

John Friedman, Brown University 

Nathan Hendren, Harvard University 

Lawrence Katz, Harvard University 

Emmanuel Saez, University of California-Berkeley 

Danny Yagan, University of California-Berkeley 

1658 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t1d-b9x8

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0157

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

Attached is the Association of American Medical Colleges (AAMC) comment letter to

the Commission on Evidence-Based Policymaking.

Attachments

AAMC Comment to Commission on Evidence Based Policy

The Promise of Evidence-Based Policymaking 1659
November 14, 2016 

Shelley Martinez 

Executive Director 

Commission on Evidence-Based Policymaking 

Commission on Evidence-Based Policymaking Comments 

Docket ID USBC-2016-0003 

Submitted electronically via www.regulations.gov

The Association of American Medical Colleges (AAMC) appreciates the Commission on 

Evidence-Based Policymaking’s commitment to explore the impact, feasibility, and necessity of 

using available and newly collected data to propose, implement, and evaluate government 

programs and policies and is pleased to provide this response to the request for comments to 

guide the Commission’s future activities, findings, and potential recommendations. The AAMC 

is a not-for-profit association representing all 147 accredited U.S. medical schools, nearly 400 

major teaching hospitals and health systems, and more than 80 academic and scientific societies. 

Through these institutions and organizations, the AAMC represents nearly 160,000 faculty 

members, 83,000 medical students, 115,000 resident physicians, and thousands of graduate 

students and postdoctoral trainees in the biomedical sciences. 

The comments below provide the Commission with: examples of recent reports that support the 

need for evidence-based policymaking; a model the AAMC has used to evaluate the effect of a 

revised regulation and which could be applied to other policies and programs; and the 

Association’s thoughts on when evidence-generation should be considered or required in the 

context of new policies. As an organization that works to support the tripartite mission of 

research, clinical care, and medical education at our member institutions, the AAMC supports 

the increased collection and use of evidence throughout the policymaking and implementation 

process, from the proposal of new policies and programs through the evaluation of the 

effectiveness of those policies and programs. Without data to understand the need for or likely 

impact of new initiatives, regulations or policies may be ineffective, inefficient, or unduly 

burdensome without achieving intended aims. One area that has received significant attention 

recently and for which there are limited but promising mechanisms and proposed frameworks for 

incorporating evidence into policies and programs is in biomedical research. 

A well-documented increase in regulatory burden for biomedical research underscores the 

need for thoughtful, deliberative, evidence-based policymaking. Recent studies and reports 

1660 The Promise of Evidence-Based Policymaking
2 

that have sought to catalogue the cumulative effect of this burden raise concerns that the failure 

to engage in systematic, evidence-based assessment of current, possibly outdated, and proposed 

 policies is potentially diminishing research productivity and advances. The 2012 Federal 

Demonstration Partnership Faculty Workload Survey of researchers found that on average, 42% 

of their research time was spent fulfilling administrative duties instead of conducting research.1

Since that time, the number of new and proposed regulations and policies has only increased, 

placing significant stress on researchers and academic institutions. Recent reports from the 

National Academies of Sciences, Engineering, and Medicine (the Academies),2

 the Government 

Accountability Office (GAO),3

 and the AAMC4 identify a need for federal agencies to harmonize 

regulations, reduce workload and costs, and consider evidence-based regulatory approaches. 

Determining the effectiveness of government policies and programs is hindered by a lack of 

data to inform the rulemaking process and limited evidence-based mechanisms to evaluate 

whether agency goals are being met. The Academies’ report on optimizing the nation’s 

investment in academic research examined many regulations governing federally funded 

research, finding that the expansion of the regulatory system is diminishing the effectiveness of 

the U.S. research enterprise. Notably, the Academies’ committee discovered there is “little 

rigorous analysis or supporting data precisely quantifying the total burden and cost to 

investigators and research institutions of complying with federal regulations specific to the 

conduct of federally funded research.” The report also highlights the committee’s “difficulty 

finding data calculating the opportunity costs associated with diverting time, expertise, resources, 

and potential away from the conduct of basic and applied research to meet regulatory demands.”5

The report cited the AAMC Conflict of Interest (COI) Metrics Project as an existing effort to 

quantify the impact and burden of research regulations on academic institutions.

The AAMC COI Metrics Project (www.aamc.org/metricsproject) was designed to measure the 

cost and outcomes of the 2011 revised regulations on financial conflicts of interest in Public 

Health Service funded research. The study, which collected data from AAMC member 

institutions over a course of three years (the year before the August 24, 2012 implementation 

date for the revised regulations and the first two years after implementation), concluded that 

academic institutions incurred significant costs beyond their ongoing program administration 

costs to fully implement the regulations. Notably, these regulations made discrete changes to an 

existing framework without changing the underlying structure of the rule, meaning that at the 

 

1

 Federal Demonstration Partnership Faculty Workload Survey (2012). 

http://sites.nationalacademies.org/cs/groups/pgasite/documents/webpage/pga_087667.pdf

2

 “Optimizing the Nation’s Investment in Academic Research, A New Regulatory Framework for the 21st Century,” (2016). 

https://www.nap.edu/catalog/21824/optimizing-the-nations-investment-in-academic-research-a-new-regulatory. 3

 See “Federal Research Grants: Opportunities Remain for Agencies to Streamline Administrative Requirements,” (2016) 

http://www.gao.gov/assets/680/677949.pdf. 4

 AAMC Analysis in Brief, “Implementing the Regulations on Financial Conflicts of Interest, Results from the AAMC Conflict 

of Interest Project”, April 2015.

https://www.aamc.org/download/429214/data/april2015implementingtheregulationsonfinancialconflictsofintere.pdf 5

 “Optimizing the Nation’s Investment in Academic Research, A New Regulatory Framework for the 21st Century,” (2016). p 2. 

The Promise of Evidence-Based Policymaking 1661
3 

time that the new rule was issued each of these institutions already had the infrastructure, 

policies, and personnel in place to comply with the existing regulations. Nonetheless, to come 

into compliance with the revised requirements of the regulations, 71 institutions invested almost 

$23 million ($22,557,744) in total, and the average number of full time equivalent employees 

needed to administer the requirements of the rule increased from 1.9 to 2.7. The number of 

“significant financial interests” (not financial conflicts of interest) collected by institutions rose 

by 45% as a result of new thresholds and requirements, requiring additional resources to review 

them for potential financial conflicts of interest. This substantial investment of resources resulted 

in relatively modest increases in the identification of conflicts of interest that required further 

review and reporting, with less than half reporting any increase in the number of identified 

financial conflicts of interest and all but 5 institutions of those that did see increases reported 

identifying fewer than 20 more conflicts of interest the year after the rule was implemented than 

the year before. 

Based on AAMC’s findings in addition to data from surveys by the Council on Government 

Relations and the National Science Board’s Task Force on Administrative Burden, the 

Academies’ report concluded that these surveys “call into question whether the new COI rule is 

accomplishing its intended goal of protecting the integrity of the scientific process and the 

welfare of research subjects, especially given the documented increases in administrative burden 

to institutions and investigators in the year following implementation of the rule.”6

Efforts like the AAMC COI Metrics Project not only demonstrate how objective, rigorous, 

and systematic evaluation can be used as a framework for evidenced-based review of 

government programs, policies, and regulations, but also suggest how such a model could 

be employed prospectively to assess the likely impact of proposed regulation and policy 

(Questions 1, 16, and 17). The key to effective evidence generation in this context is the early 

consideration of whether the type of policy being considered would benefit from a prospective 

pilot or data collection and identification of the types of data that will best demonstrate the 

impact and the effectiveness of the policy or program. Partnership with those served by or 

affected by the initiative can be an efficient and effective way to answer these threshold 

questions. Recognizing that the required evidence will not always come from existing 

government data, pilot programs or collaboration with institutions, associations, and 

communities can both increase engagement in the process and enhance the changes that the 

initiative will accomplish its desired goals. 

A 2013 memorandum to federal agency and department heads from the Office of Management 

and Budget captured the desire to “deliver[] a smarter, more innovative, and more accountable 

government for citizens,” one component of which is for government agencies to “continually 

improve program performance by applying existing evidence about what works, generating new 

knowledge, and using experimentation and innovation to test new approaches to program 

 

6

 “Optimizing the Nation’s Investment in Academic Research, A New Regulatory Framework for the 21st Century,” (2016). p. 91.

1662 The Promise of Evidence-Based Policymaking
4 

delivery.”7 This call for applying evidence to program delivery can be seen as a welcome and 

logical extension of the 2011 mandate to streamline regulations through retrospective review by 

agencies and increase interagency coordination to harmonize regulations.8 However, as the GAO 

report concludes, with respect to the administration of federal grants, “opportunities exist... to 

further reduce universities’ administrative workload and costs,” and “efforts to standardize 

requirements have not fully addressed variations in agency implementation of requirements.”9

 

Agencies should integrate robust evidenced based evaluation mechanisms into the 

proposed rulemaking process to ensure that regulatory decisions are made using the best 

and most current evidence available (Question 17). The failure to gather and use data about 

impact, including burden and outcomes, can lead to protracted or ultimately unsuccessful 

policymaking processes where the stated objectives of the policy are poorly reflected in its issued 

form or not achieved in its implementation. A failure to determine in advance how that policy 

will be evaluated leaves agencies and the regulated communities equally uninformed about 

which policies are working as intended. As a current example, the Academies report discussed at 

length the Department of Health and Human Services’ Notice of Proposed Rulemaking (NPRM) 

for the “Common Rule” which governs federally funded research with human subjects. The 

Academies made the recommendation to suspend the Common Rule NPRM, concluding that the 

proposed rule was “marred by omissions, the absence of essential elements, and a lack of 

clarity.”10 Among other concerns, the report highlights the committees’ concern with the 

proposed revision prohibiting all research with deidentified biospecimens without written 

consent, a concern echoed which was also echoed in AAMC’s comment letter to HHS, which 

stated that the revision as proposed would place “extraordinary stresses [...] on the research 

community as a whole” and “without increasing meaningful understanding [...] or protection of 

human subjects.”11 If the rule is implemented as proposed, it appears that there is no specific plan 

to evaluate its cost, impact, or success. Given the significant costs that would be needed to 

implement just this one aspect of the proposed rule and the potential chilling effect on essential 

research with biospecimens, the lack of a plan to gather evidence to evaluate the policy is 

problematic.

The Commission has asked if program or policy evaluation can or should be incorporated in 

program designs. If the evidence and criteria on which the success of a proposed policy would be 

measured were included as a routine and reviewable component of the policymaking process, 

agencies would have the benefit of stakeholder input on the evaluation process, best sources of 

 

7

“Next Steps in the Evidence and Innovation Agenda,” Memorandum to the Heads of Departments and Agencies (July 26, 2013), 

https://www.whitehouse.gov/sites/default/files/omb/memoranda/2013/m-13-17.pdf. 

8

 Executive Order 13563: “Improving Regulation and Regulatory Review,” 76 Fed Reg. 3821 (Jan. 18, 2011). 9 “Federal Research Grants: Opportunities Remain for Agencies to Streamline Administrative Requirements,” (2016), pg. 3. 10 “Optimizing the Nation’s Investment in Academic Research, A New Regulatory Framework for the 21st Century,” (2016) 

p.167, The Academies report also recommends the establishment of a new commission to reconsider the process for protecting 

and engaging human research subjects, p.168. 

11 AAMC Comment Letter, January 4, 2016 (available at 

https://www.aamc.org/download/451896/data/aamcsubmitscommentstohhsonthecommonrulenprm.pdf). 

The Promise of Evidence-Based Policymaking 1663
5 

data to support the evaluation, and whether the proposed metrics are likely to provide the agency 

and the public with meaningful evidence about a program or policy’s effectiveness. This would 

set a precedent for an unparalleled and productive level of engagement between agencies and 

stakeholders, demonstrating not only accountability, but a commitment to the shared goal of 

ensuring that federal policies are implemented for the right reasons and meet clear, articulated 

objectives. 

AAMC is supportive of the Commission’s interest in using data to build evidence to inform 

program and policy design. The AAMC would be happy to work with the Commission on any of 

the issues discussed in our letter, provide additional information about the methodology or 

findings from the AAMC COI Metrics Project, or discuss the evidence-based policymaking 

across any areas that affect the academic research community. For more information, please 

contact me or Heather Pierce, Senior Director, Science Policy and Regulatory Counsel 

(Association of American Medical Colleges, 655 K Street NW, Suite 100, Washington DC 

20001, (202) 478-9926, hpierce@aamc.org). 

Sincerely, 

Ross E. McKinney, Jr, MD 

Chief Scientific Officer 

1664 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t1f-tkon

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0158

Comment on FR Doc # 2016-22002

Submitter Information

Name: Michael Leuzzi

General Comment

I oppose the continuation or increase of federal collection of individual student data.

We should not have a federal unit record system of student data.

Tracking autonomous free individuals through most of their lives in the name of better

information for the benefit of others may be justifiable, but its extremism should at the

very least be acknowledged and addressed.

Thank you for listening!

The Promise of Evidence-Based Policymaking 1665
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t1f-4nzw

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0159

Comment on FR Doc # 2016-22002

Submitter Information

Name: Wendy Hart

General Comment

I would like to go on record opposing this expansion of federal powers. Data is the new

gold. It will not be able to be protected, and any "benefit" will not be seen by those

providing, in many cases, unwittingly, access to their data. Data is person and property at

the same time. It is the sworn duty of this government to protect it, not to amass it or

exploit it. Please leave the prohibitions in place, consistent with Article 1 Section 8 of

the Constitution and the 10th Amendment.

1666 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t1g-dss5

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0160

Comment on FR Doc # 2016-22002

Submitter Information

Name: Shelley White

General Comment

Please, for the privacy, safety, and security of our children and citizens, no national

student database. The US is not a communist dictatorship. Let us live our lives freely

without complete government oversight. Thank you.

The Promise of Evidence-Based Policymaking 1667
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t1g-h2nc

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0161

Comment on FR Doc # 2016-22002

Submitter Information

Name: Brian Wee

Address:

Washington, DC, 20009

Email: bwee@neptuneinc.org

General Comment

(The attached PDF includes Figures 1 and 2 that are referred to in our response)

Docket ID: USBC-2016-0003

RFC Subject: Commission on Evidence-Based Policymaking Comments

Question #16: How can data, statistics, results of research, and findings from evaluation,

be best used to improve policies and programs?

To evaluate the effectiveness of federal programs and tax expenditures, we* recommend

that evidence based policy making using data and statistics should be embedded in a

Structured Decision Making (SDM: Figure 1) framework, and ideally implemented using

a Decision Analysis Support System (DASS: Figure 2) that closely reflects the SDM

approach. SDM is highly applicable to monitoring and evaluation processes that help

improve programmatic performance and achieve results.

SDM is a class of methods used to help individuals and groups think through

multidimensional choices constrained by science, stakeholders, and trade-offs, while

1668 The Promise of Evidence-Based Policymaking
taking into consideration uncertainties and value-judgments that are inherent in this

process (Gregory et al 2012). A DASS may be thought of as a computer-based decision

support system that incorporates these multidimensional choices by using statistical

decision tools such as decision analysis, sensitivity analysis, and value of information

analysis (Black and Stockton 2009).

Incorporating data and statistics using an SDM approach begins with a comprehensive

understanding of the decision landscape (e.g., programmatic objectives, desired

outcomes, possible implementation options, regulatory aspects of the decision). The

decision landscape may be thought of as desired program improvements and associated

policy changes that improve the government's fiduciary responsibilities on behalf of the

taxpayer. The values and preferences of stakeholders are made explicit and translated

into decision objectives. Data and models are used where appropriate to quantify the

effects of various implementation options. Probabilistic consequence modeling can then

be used for forward and backward reasoning, sensitivity analyses used to identify

influential variables, and value of information analyses used to address reducing sources

of uncertainty.

Data, statistics, results of research, and findings from evaluation, when evaluated using

an SDM approach and implemented using a DASS, facilitate defensibility, traceability

and transparency of the decision process for evaluating the effectiveness of federal

programs and tax expenditures, thus leading to improved and more cost-effective

policies and programs.

*Neptune and Company, Inc. is a small, employee-owned business that specializes in (1)

statistical analysis and quality assurance, (2) development of decision analysis

methods/software to facilitate all forms of risk management.

References

R. Gregory, L. Failing, M. Harstone, G. Long, T. McDaniels, D. Ohlson (2012)

Structured Decision Making: A Practical Guide to Environmental Management Choices.

Hoboken, NJ. John Wiley & Sons.

P. Black, T. Stockton (2009). Basic Steps for the Development of Decision Support

Systems. In Decision Support Systems for Risk-Based Management of Contaminated

Sites (pp. 1-27). Boston, MA: Springer US. http://doi.org/10.1007/978-0-387-09722-0_1

Attachments

2016-11-14.CommisionEvidenceBasedPolicyMaking.v1.5

The Promise of Evidence-Based Policymaking 1669
Page 1 of 2 

Docket ID: USBC-2016-0003 

RFC Subject: Commission on Evidence-Based Policymaking Comments 

Question #16: How can data, statistics, results of research, and findings from evaluation, be best used 

to improve policies and programs? 

To evaluate the effectiveness of federal programs and tax expenditures, we* recommend that evidence 

based policy making using data and statistics should be embedded in a Structured Decision Making 

(SDM: Figure 1) framework, and ideally implemented using a Decision Analysis Support System (DASS: 

Figure 2) that closely reflects the SDM approach. SDM is highly applicable to monitoring and evaluation 

processes that help improve programmatic performance and achieve results. 

SDM is a class of methods used to help individuals and groups think through multidimensional choices 

constrained by science, stakeholders, and trade-offs, while taking into consideration uncertainties and 

value-judgments that are inherent in this process (Gregory et al 2012). A DASS may be thought of as a 

computer-based decision support system that incorporates these multidimensional choices by using 

statistical decision tools such as decision analysis, sensitivity analysis, and value of information analysis 

(Black and Stockton 2009). 

Incorporating data and statistics using an SDM approach begins with a comprehensive understanding of 

the decision landscape (e.g., programmatic objectives, desired outcomes, possible implementation 

options, regulatory aspects of the decision). The decision landscape may be thought of as desired 

program improvements and associated policy changes that improve the government’s fiduciary 

responsibilities on behalf of the taxpayer. The values and preferences of stakeholders are made explicit 

and translated into decision objectives. Data and models are used where appropriate to quantify the 

effects of various implementation options. Probabilistic consequence modeling can then be used for 

forward and backward reasoning, sensitivity analyses used to identify influential variables, and value of 

information analyses used to address reducing sources of uncertainty. 

Data, statistics, results of research, and findings from evaluation, when evaluated using an SDM 

approach and implemented using a DASS, facilitate defensibility, traceability and transparency of the 

decision process for evaluating the effectiveness of federal programs and tax expenditures, thus leading 

to improved and more cost-effective policies and programs. 

*Neptune and Company, Inc. is a small, employee-owned business that specializes in (1) statistical 

analysis and quality assurance, (2) development of decision analysis methods/software to facilitate all 

forms of risk management. 

References 

x R. Gregory, L. Failing, M. Harstone, G. Long, T. McDaniels, D. Ohlson (2012) Structured Decision 

Making: A Practical Guide to Environmental Management Choices. Hoboken, NJ. John Wiley & 

Sons. 

x P. Black, T. Stockton (2009). Basic Steps for the Development of Decision Support Systems. In 

Decision Support Systems for Risk-Based Management of Contaminated Sites (pp. 1–27). 

Boston, MA: Springer US. http://doi.org/10.1007/978-0-387-09722-0_1 

1670 The Promise of Evidence-Based Policymaking
Page 2 of 2 

Figure 1. The components of structured decision making. 

Figure 2. An example of a decision analysis support system for structured decision making.

The Promise of Evidence-Based Policymaking 1671
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 14, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t1g-jlfh

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0162

Comment on FR Doc # 2016-22002

Submitter Information

Name: Louis Jacobson

Address:

McLean, VA, 22101

Email: new.horizons.lj@gmail.com

General Comment

see attached file

Attachments

Response to Federal Register Notice Docket Number 160907825

1672 The Promise of Evidence-Based Policymaking
1 

Response to Federal Register Notice Docket Number 160907825-6825-01 

Request of Comments on Evidence-Based Policy Making 

Louis S. Jacobson, PhD, President, New Horizons Economic Research & George Washington University 

Institute of Public Policy (GWIPP), Research Professor 

1867 Massachusetts Avenue, McLean, VA 22101 

New.Horizons.LJ@gmail.com – 703 241-3757 

These comments describe a pathway to using data to improve outcomes in one especially important 

policy area that could be emulated in other areas. They do not address every issue raised by the Federal 

Register Notice (FRN). This is because, in my view, focusing on integrating all of the elements needed to 

resolve a major problem of national significance, not just assembling the data, will lead to constructive 

action. In contrast, partitioning the problem into separate elements and addressing each element 

individually too often fails to produce desired results because not all of the necessary elements are in 

place. 

In other words, there are states, such as Florida and Washington, that have integrated data systems that 

can be used to produce useful information to improve policy effectiveness, but often do not lead to 

improvements because these data are not used to produce relevant measures, or useful information is 

not effectively disseminated, or insufficient attention is given to assessing why the information is not 

effectively used. 

The comments presented below primarily are drawn from a paper published in April 2013 by the 

Hamilton Project, Brookings Institution: “Using Data to Improve the Performance of Workforce Training” 

by Louis S. Jacobson & Robert J. LaLonde. They focus on how the federal government can effectively 

work with states and researchers to improve education and training outcomes by creating and 

disseminating information that would actually (rather than hypothetically) improve the choices made by 

ordinary citizens seeking to increase their earnings, the institutions providing the education and training, 

and the policy-makers overseeing those institutions. 

There are five key elements to reaching policy-oriented goals: 

1. Defining the problem to be addressed. 

2. Assembling relevant data. 

3. Determining what measures should be used with the data. 

4. Disseminating the information to the relevant decision makers. 

5. Obtaining feedback to assess how well the system met its goals, what could be done to 

overcome impediments, and sustain cost effective programs. 

The idea presented in the Hamilton Project paper is to create a national competition open to consortium 

of states that would create entities that would be responsible for executing all of the above elements. 

The goal would be to help individuals obtain the education and training needed to substantially raise 

their earnings. Most states already have the required data to help identify high return programs that 

are likely to be completed by various individuals. Many would need help organizing the data such as 

that on high school and college course taking and linking those data to earnings data. Some would need 

legislative changes that would permit the linking of the data. 

However, a major issue is how to use the data to produce measures that would help improve individual 

choices. In particular, it is not sufficient to simply describe the number of students starting programs, 

completing programs, and earnings following completion. Additional information is needed to help 

The Promise of Evidence-Based Policymaking 1673
2 

potential enrollees determine if they have the academic and other qualifications needed to complete 

the program, and in many cases, help individuals understand the characteristics of jobs for which they 

would qualify. 

Similarly, a major issue is how to disseminate the information so it can be effectively used. Often it is 

assumed that the individuals who need to make decisions about education and training can use 

websites with various type of “report card” information. However, even if the data are necessary to 

make solid choices, they would be insufficient if extensive assessment and counseling was not also 

provided. Indeed, CUNY’s ASAP program which has proven to double the rate of community college 

credentialing does not depend on use of data to help students select programs, but on having 10 times 

the ordinary availability of counselors to help students get through their programs. 

The final key component is to have an evaluation system in place to assess how well the goals are being 

met, what are the factors that impede reaching goals, and how those impediments can be overcome. 

One underlying premise is that developing a complete system is a complex task that requires careful 

monitoring to determine that all components are working effectively together. A second premise is that 

if an information system is demonstrated to be cost effective it would receive the long-term support it 

deserves. In particular, it is likely that improved education and training choices would lead to raising the 

return on these investments to more than pay for the costs of the system used to improve those 

choices. 

In summary, the view expressed here is that the best option for the federal government is to focus on 

development of entities that would encourage the federal government, states, and researchers to work 

together to develop data development and dissemination systems that provide the information needed 

to improve policy outcomes, not simply resolve the problems that have only allowed a few states to 

assembly the types of data required to improve policy outcomes. 

In large part, this position stems from recognizing that the Florida system used to produce a great deal 

of highly useful policy-oriented research stemmed from convincing key decision-makers in the executive 

and legislative branches that the information was useful in reaching goals held in common. If the data 

system was developed in isolation from demonstrating its value to decision-makers, it is doubtful that 

the system would have been given the resources needed to create and sustain the system. 

With respect to the specific FRN questions the points above are most relevant to Q3 & Q4—describing 

the most appropriate infrastructure; Q6—suggesting competitions to create several multistate 

consortiums would be most appropriate for creating systems that would end-up improving decisionmaking; and Q8—suggesting sustained funding should be secured by demonstrating that the 

information has led to increasing the cost-effectiveness of government programs that pay for the 

systems many times over. 

I have many ideas related to the other specific questions. However, I strongly believe that the key to 

seeing the desired improvement in policy-making depends on holding a competition that offers 

sufficient funds to cause states already well on their way to create the entities described above to 

demonstrate the effectiveness of these entities. 

1674 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 15, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t1h-btqu

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0163

Comment on FR Doc # 2016-22002

Submitter Information

Name: Felice J. Levine

Address:

Washington, DC, 20005

Email: flevine@aera.net

General Comment

See attached file(s)

Attachments

AERA Comments to CEP_FINAL_11-14-16

The Promise of Evidence-Based Policymaking 1675
1430 K Street, NW x Washington, DC 20005 x (202) 238-3200 

Facsimile (202) 238-3250 x http://www.aera.net

November 14, 2016 

Mr. Nick Hart 

Policy and Research Director 

Commission on Evidence-Based Policymaking 

Nicholas.r.hart@census.gov

Docket ID: 160907825-6825-01 

Commission on Evidence-Based Policymaking Comments 

Dear Mr. Hart, 

On behalf of the American Educational Research Association (AERA), I am pleased to have the 

opportunity to offer comments in response to the request from the Commission on EvidenceBased Policy to inform the Commission’s work and provide recommendations of core questions. 

The AERA is the major national scientific association of more than 25,000 faculty, researchers, 

graduate students, and other distinguished science professionals dedicated to advancing 

knowledge about education, encouraging scholarly inquiry related to education, and promoting 

the use of research to improve education and serve the public good. Founded in 1916, the 

association is committed to the highest standards of research rigor, integrity, and responsibility 

for research scientists, as reflected in such policy documents as AERA’s research standards 

and code of ethics as well as longstanding programs to encourage data sharing and access 

consonant with responsible use. 

Our members use and analyze federal statistics and data in their research and rely on the 

objectivity and trustworthiness of this information. AERA members are interested in increasing 

access to and usefulness of impartial and accurate information to best improve policy and 

practice decision making. Our division of Education Policy counts over 2,500 members and our 

Measurement and Research Methods division more than 3,000 research science. Also, AERA 

members participated in 25 established Special Interest Groups focused on various aspects of 

evaluations and statistics. 

AERA applauds the Commission on Evidence-Based Policymaking for the comprehensive 

examination, careful consideration and inclusive approach. AERA shares the belief that 

increased access to reliable data can dramatically bolster our ability to best guide and drive the 

most effective and informed development and evaluation of policy. 

Given the considerable expertise on the panel and the extensive feedback from the community 

both to these questions and in meetings and public hearings, we are responding selectively to 

questions only where AERA has specific examples in education to offer or a unique observation 

based on our own work in this area. We have tried to highlight some of the most useful 

examples from the decades of guidance and practice addressing these important questions. 

The Commission can play a critical leadership role establishing how to expand access to 

administrative, survey and linked data consonant with appropriate concerns for privacy and 

protecting confidentiality. 

1676 The Promise of Evidence-Based Policymaking
The time is especially ripe for a fresh look under the current circumstances of expanded 

electronic access to wide-ranging information, growing capacity to examine this information 

efficiently, and rising costs in time and money in implementing major surveys. We have seen in 

just this past decade promising uses of rapidly collected digital information, the transformation of 

administrative information largely in paper form to digitized administrative data systems, and a 

deeper appreciation that quality administrative data, well and widely utilized, can make for more 

robust research that can speak to policy and program development and implementation. 

Fortunately there has been considerable experience in the federal government, in particular in 

the federal agencies, with using administrative data systems, linking data, and devising 

mechanisms for secure access and use. There has also been considerable guidance about how 

best to expand use of administrative data systems with appropriate mechanisms for data 

protection and access commensurate with the level of risk. For over 30 years, the National 

Academies, most typically at the request of federal agencies, has examined and provided 

guidance on access to federal data assets aligned with privacy protection, confidentiality, and 

data security. In those works, including those since the turn of the century, are useful examples, 

observations, and recommendations that can inform the Commission’s work. See Appendix A. 

This comment responds specifically to questions raised by the Commission. Nonetheless, we 

wish also to make four general points that might be helpful as the Commission proceeds with its 

essential work. 

1. Define ‘evidence’ and ‘effectiveness’ broadly to account for the spectrum of outcomes 

significant to assessing program and policy goals. 

2. Ensure a robust understanding of the methodologies essential to studying effectiveness, 

short- and long-term consequences, and unintended effects. These would include but not be 

limited to experimental and quasi-experimental methods, longitudinal designs, statistical 

matching, and so forth. 

3. Examine and invest in making accessible federal data assets, including administrative 

information, under institutional arrangements and data use agreements that maximize the 

capacity to examine policies and programs consonant with privacy provisions and confidentiality 

protections. Review current data use agreements and data management plans to maximize 

access under conditions of data security. 

4. Evaluate the leadership of statistical agencies, maximizing autonomy to allow for expert 

advice based on sound evidence and to safeguard statistical agencies from political influence. 

Leadership should reflect technical expertise and understanding of data use. 

These are guiding principles relevant to the Commission’s three sets of questions. They also 

are useful for devising a framework to inform evidence-based policy and evaluation. 

Overarching Questions 

1. Are there successful frameworks, policies, practices, and methods to overcome challenges 

related to evidence-building from state, local, and/or international governments the Commission 

should consider when developing findings and recommendations regarding Federal evidencebased policymaking? If so, please describe. 

The Promise of Evidence-Based Policymaking 1677
There is a history of practice and guidance from the federal statistical agencies and from the 

Office of Management and Budget (OMB) Office of Information and Regulatory Affairs (OIRA) 

directed to promoting the quality of federal data and statistical information to enhance evidencebased policies and programs. The leadership of OIRA’s Statistical and Policy Office has been 

instrumental in the federal and international arenas that could be usefully extrapolated in 

considering how best to strengthen data quality, access, and use with state and local data. 

In the domain of education programs and policy, the State Longitudinal Data Systems (SLDS) 

Grant Program, authorized by the Educational Technical Assistance Act of 2002, Title II of the 

statute that created the Institute of Education Sciences (IES), is arguably, a successful 

framework for state educational evidence-building. 

Thanks to federal support, the program has enabled the successful design, development, 

implementation, and expansion of K12 and P-20W (early learning through the workforce) 

longitudinal data systems in nearly every state. In fact, 47 states having legislated state funds to 

continue the operations after using federal funds to build the infrastructure. As articulated by 

Robert Swiggum in his testimony in front of the House Education and Workforce Committee, 

SLDS enable states to efficiently and accurately manage, analyze, and use education data. In 

Georgia, SLDS have led to data-informed decisions to improve student learning and outcomes. 

As part of a joint conference directed to developing model guidelines for use of longitudinal 

administration systems convened by AERA and the Organisation for Economic Co-operation 

and Development (OECD) in December 2015, representatives from administrative data systems 

described how linkages in data sets can provide for robust findings with policy implications. One 

model for the U.S. is the United Kingdom’s Administrative Data Research Network, a repository 

of administrative linked and de-identified data sets made available to social science researchers 

under secure conditions. Data linked among multiple sources for approved research projects 

have provided relevant information for policy decisions with the goal of benefiting society. For 

example, the National Pupil Database – which connects data sources for exam results, 

attendance records, name of the school a child is attending with a student identifier – allows for 

decisions on how much money from the national education budget is given to particular local 

authorities and schools. Another project in process is linking data on unemployment benefits 

and successive sanctions with Scottish data on school attendance to determine whether there 

are unintended consequences to children when parents’ unemployment benefits are stopped 

with the aim to encourage them to return to work. 

2. Based on identified best practices and existing examples, what factors should be considered 

in reasonably ensuring the security and privacy of administrative and survey data? 

The National Center for Education Statistics (NCES) has pioneered making available data sets 

with personally identifiable information (PII) to researchers, through restricted-use data licenses. 

Authorized users are subject to the laws, regulations, and penalties that apply to the NCES use 

of confidential data. The NCES Statistical Standards Program monitors the licensing process 

and inspections. Wide access to NCES data is balanced by stringent sanctions for violation. The 

NCES website has extensive materials on data access to public use and restricted-use data, 

including a Restricted-Use Data Procedure Manual (NCES 2007 at 

http://nces.ed.gov/pubs96/96860rev.pdf) 

The Interuniversity Consortium for Political and Social Research (ICPSR) has established a 

protocol for preserving respondent confidentiality that starts with the depositors of data, 

1678 The Promise of Evidence-Based Policymaking
requiring documentation for information that could identify respondents, which could establish 

restricted use to the data. ICPSR offers four levels of restricted use: Traditional Restricted Data, 

Physical Data Enclave, Restricted Online Analysis, and Delayed Dissemination. 

Data Infrastructure and Access 

3. Based on identified best practices and existing examples, how should existing government 

data infrastructure be modified to best facilitate use of and access to administrative and survey 

data? 

4. What data-sharing infrastructure should be used to facilitate data merging, linking, and 

access for research, evaluation, and analysis purposes? 

When considering best practices regarding data infrastructure, I encourage you to consult the 

OMB Guidance for Providing and using Administrative Data for Statistical Purposes released on 

February 14, 2014. This document provides tools and detailed guidance on the interaction of the 

use of administrative data for statistical purposes with the Privacy Act requirement. In addition, 

Sharing Data While Protecting Privacy (M-11-02 of November 3, 2010), Open Data PolicyManaging Information as an Asset (M-13-13 of May 9, 2013), and Next Steps in the Evidence 

and Innovation Agenda (M-13-17 of July 26, 2013) are three useful OMB memoranda. 

5. What challenges currently exist in linking state and local data to federal data? Are there 

successful instances where these challenges have been addressed? 

Looking specifically at NCES, efforts to access data in the state longitudinal data systems 

(SLDS) must be negotiated with each individual state. Even for states inclined to make every 

effort to share state level data, laws and regulations regarding protections of student data 

privacy are sufficiently ambiguous leading states to err on the side of caution deciding against 

sharing data. 

8. What factors or strategies should the Commission consider for how a clearinghouse(s) could 

be self-funded? What successful examples exist for self-financing related to similar purposes? 

While still too early to tell if effective, the recently passed bipartisan reauthorization of the 

Elementary and Secondary Education Act includes Sec. 8601. Evaluations. This section 

requires IES to do an evaluations of each program authorized in the Act. The legislation allows 

the Secretary to reserve .5 percent of the program account for the evaluation costs and 

dissemination of findings. In addition the section provides guidance on the design of 

evaluations. 

(a) RESERVATION OF FUNDS. – Except as provided in subsection (b) and (e), the 

Secretary, in consultation with the Director of the Institute of Education Sciences, may 

reserve not more than 0.5 percent of the amount appropriated for each program 

authorized under this Act to carry out activities under this section. (ESSA) 

In addition to the cost of conducting evaluations, I encourage the Commission to think about the 

workforce capacity to most instructively conduct and interpret evaluations. IES has developed 

tremendous technical and capacity in this regard, thanks to hiring flexibility permitted by their 

accepted service hiring authority. 

The Promise of Evidence-Based Policymaking 1679
9. What specific administrative or legal barriers currently exist for accessing survey and 

administrative data? 

As mentioned in previous responses, numerous legal, regulatory and operational barriers 

prevent federal agencies from linking to state data. Census and NCES have been required to 

approach each state individually to initiate data linkages. Even when states are motivated to 

share data, they are often advised against doing so due to legal concerns. 

10. How should the Commission define “qualified researchers and institutions?” To what extent 

should administrative and survey data held by government agencies be made available to 

“qualified researchers and institutions?” 

Administrative and survey data held by government agencies should be maximally available to 

qualified researchers and institutions. Users might be required to provide data management 

plans that would give insight into the researcher's intentions for their data both during and after 

the research project. (ICPSR) 

14. What incentives may best facilitate interagency sharing of information to improve 

programmatic effectiveness and enhance data accuracy and comprehensiveness? 

Interagency information sharing is presumably beneficial to all involved agencies. Removing 

obstacles might be a kin to providing incentives. In addition to alleviating concerns about the 

legality of sharing information, agencies engaged in high level analysis of data would benefit 

from the flexibility in hiring high qualified staff to maximize the benefit of increasing access to 

data and effective data management plans. 

Data Use in Program Design, Management, Research, Evaluation, and Analysis 

15. What barriers currently exist for using survey and administrative data to support program 

management and/or evaluation activities? 

Currently, administrative and legal barriers prevent the use of survey and administrative data. 

For example, researchers looking to determine how certain P-12 programs may effect wages 

later in life are unable to access wage information readily available with IRS or the Social 

Security Administration. 

Another barrier to using survey and administrative data is the variation in definitions. A 

successful effort to standardize terms was the Common Education Data Standards (CEDS) 

voluntarily developed common data standards for a key set of education data elements to 

streamline the exchange, comparison, and understanding of data within and across early 

learning through postsecondary and workforce settings. 

19. To what extent should evaluations specifically with either experimental (sometimes referred 

to as “randomized control trials”) or quasi-experimental designs be institutionalized in 

programs? What specific examples demonstrate where such institutionalization has been 

successful and what best practices exist for doing so? 

AERA supports the use of the most rigorous methods consonant with the research issues and 

contexts of study and program evaluation. We appreciate the importance of randomized control 

1680 The Promise of Evidence-Based Policymaking
trials and experimental and quasi-experimental designs. We encourage, however, rather than 

institutionalizing any single method, embracing the importance of using methods appropriate to 

the research or the program being evaluated. RCT designs are valuable methodologies in 

isolating effects when appropriate; nevertheless, many interventions and programs cannot be 

introduced under conditions that would differentially provide known benefits or potentially 

introduce risks that would negatively affect individuals. 

Please refer to the AERA volume, Estimating Causal Effects: Using Experimental and 

Observational Designs (2005) and, Appendix B, the AERA Definition of Scientifically Based 

Research. See also the Institute of Education Sciences and the National Science Foundation 

guidance. Common Guidelines for Education Research and Development, August 2013. 

AERA very much supports the Commission’s efforts and the openness that you bring to this 

complex and ambitious task. We think the progress you can making in promoting sound policy 

and programs through greater secure and responsible use of data systems is enormous. We 

welcome helping and supporting you in that effort. 

Sincerely, 

Felice J. Levine, PhD

Executive Director 

flevine@aera.net

202.238.3201 

American Educational Research Association 

1430 K Street, NW 

Suite 1200 

Washington, DC 20005 

 

The Promise of Evidence-Based Policymaking 1681
Appendix A 

Reports on Protecting Participants and Facilitating Research 

from the National Academies 

With Additions from document initially prepared in October 2011 

by Dr. Connie Citro, Committee on National Statistics 

The challenge of protecting human subjects in biomedical and social and behavioral sciences 

research while facilitating responsible research and access to research data has engaged the 

attention of federal agencies, the National Academies, and the scientific community for 

decades. Below is a chronological list of major reports from the National Research Council and 

the Institute of Medicine. 

National Research Council. (1979). Privacy and confidentiality as factors in survey response. 

Washington, DC: National Academy of Sciences. 

National Research Council. (1985). Sharing research data. Washington, DC: National Academy 

Press. 

National Research Council. (1993). Private lives and public policies: Confidentiality and 

accessibility of government statistics. Washington, DC: National Academy Press 

National Research Council. (2000). Improving access to and confidentiality of research data: 

Report of a workshop. Washington, D.C.: National Academy Press. 

Institute of Medicine. (2000). Protecting data privacy in health services research. Washington, 

DC: National Academy Press. 

Institute of Medicine. (2001). Preserving public trust: accreditation and human research 

participant protection programs. Washington, DC: National Academy Press. 

Institute of Medicine. (2002). Responsible research: a systems approach to protecting research 

participants. Washington, DC: The National Academies Press. 

National Research Council. (2003). Protecting participants and facilitating social and behavioral 

sciences research. Washington, DC: The National Academies Press. 

Institute of Medicine. (2004). The ethical conduct of clinical research involving children. 

Washington, DC: The National Academies press. 

National Research Council and Institute of Medicine. (2005). Ethical considerations for research 

on housing-related health hazards involving children. Washington, DC: The National 

Academies Press. 

National Research Council. (2005). Expanding access to research data: Reconciling risks and 

opportunities. Washington, DC: The National Academies Press. 

Institute of Medicine. (2006). Effect of the HIPAA privacy rule on health research: Proceedings 

of a workshop presented to the National Cancer Policy Forum. Washington, DC: The National 

Academies Press. 

1682 The Promise of Evidence-Based Policymaking
National Research Council. (2006). Improving business statistics through interagency data 

sharing: Summary of a workshop. Washington, DC: The National Academies Press. 

National Research Council. (2007). Putting people on the map: Protecting confidentiality with 

linked social-spatial data. Washington, DC: The National Academies Press. 

National Research Council. (2007). Engaging privacy and information technology in a digital 

age. Washington, DC: The National Academies Press. 

Institute of Medicine. (2007). Ethical considerations for research involving prisoners. 

Washington, DC: The National Academies Press. 

National Research Council. (2007). Understanding business dynamics: An integrated data 

system for America’s future. Washington, DC: The National Academies Press. 

Institute of Medicine. (2009). Beyond the HIPAA privacy rule: enhancing privacy, improving 

health through research. Washington, DC: The National Academies Press. 

National Research Council. (2009). Ensuring the integrity, accessibility and stewardship of 

research data in the digital age. Washington, DC: The National Academies Press. 

National Research Council. (2009). Principles and practices for a federal statistical agency. 

Washington, DC: The National Academies Press. 

National Research Council. (2009). Protecting student records and facilitating education 

research: A workshop summary. Washington, DC: The National Academies Press. 

National Research Council. (2010). Conducting biosocial surveys: Collecting, storing, 

accessing, and protecting biospecimens and biodata. Washington, DC: The National Academies 

Press. 

National Research Council. (2010). Protecting and accessing research data from the Survey of 

Earned Doctorates: a research summary. Washington, DC: The National Academies Press. 

National Research Council (2014). Proposed revisions to the common rule for the protection of 

human subjects in the behavioral and social sciences. Washingto, DC: The National Academies 

Press. 

 

The Promise of Evidence-Based Policymaking 1683
Appendix B: Alternate Definition of Scientifically Based Research (SBR) 

AERA offers the following definition of scientifically based research (SBR) provides a broader 

definition grounded in scientific standards and principals. It was developed by an expert working 

group convened by the American Educational Research Association (AERA) in June 2008. 

Alternate Definition of Scientifically Based Research (SBR) 

Supported by AERA Council, July 11, 2008

I. The term “principles of scientific research” means the use of rigorous, systematic, and 

objective methodologies to obtain reliable and valid knowledge. Specifically, such 

research requires: 

A. development of a logical, evidence-based chain of reasoning;

B. methods appropriate to the questions posed;

C. observational or experimental designs and instruments that provide reliable 

and generalizable findings;

D. data and analysis adequate to support findings;

E. explication of procedures and results clearly and in detail, including specification 

of the population to which the findings can be generalized;

F. adherence to professional norms of peer review;

G. dissemination of findings to contribute to scientific knowledge; and

H. access to data for reanalysis, replication, and the opportunity to build on findings.

II. The examination of causal questions requires experimental designs using random 

assignment or quasi-experimental or other designs that substantially reduce plausible 

competing explanations for the obtained results. These include, but are not limited to, 

longitudinal designs, case control methods, statistical matching, or time series analyses. 

This standard applies especially to studies evaluating the impacts of policies and 

programs on educational outcomes.

III. The term “scientifically based research” includes basic research, applied research, and 

evaluation research in which the rationale, design, and interpretation are developed in 

accordance with the scientific principles laid out above. The term applies to all 

mechanisms of federal research support, whether field-initiated or directed.

1684 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 15, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t1j-j86m

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0164

Comment on FR Doc # 2016-22002

Submitter Information

Name: David Massey

Address:

Juneau, AK, 99801

Email: dgmassey@gci.net

General Comment

Please don't implement the National Student-unit record system! We don't need another

nationalized invasion of our children's privacy posing as an improvement in our

educational system. Enough already. Return education to the states and local

communities where it belongs.

The Promise of Evidence-Based Policymaking 1685
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 15, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t1m-px7f

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0165

Comment on FR Doc # 2016-22002

Submitter Information

Name: Robin Brooks

Address:

Topsham, ME, 04086

Email: robinellenb@gmail.com

General Comment

I am a teacher, parent, and citizen and wish to express my strong opposition to the

creation of any sort of national student database. Teaching is a local enterprise best

governed by school boards and communities. Local communities can benefit from broad

general guidance and funding support from the federal and state governments to assure

that basic civil rights laws are enforced and that there is equity in funding. Teachers and

parents want a strong locally controlled system of truly public schools. 

Data mining our students does not serve the national interest of a free and democratic

society. The new Commission on Evidence-based Policymaking being pushed by

Speaker Paul Ryan and Senator Patty Murray , and urged (by Bill Gates and the other

foundations/corporations that want access to more and more of our children's data) to

establish a national student-unit record system that would allow government tracking of

citizens from Pre- school throughout their careers is invasive and anti-democratic. It

takes our children and turns them into commodities to be tracked. In a free and fair

society this cannot happen and I wish to reiterate my strong opposition to such a system.

1686 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 15, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t1v-8vis

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0166

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

Please shut down the establishment of a National Student-unit record system. This is not

necessary... you do not need cradle to the grave information on my children. I do not see

why you are allowing Bill Gates, the self appointed Education Czar, to push this through.

The Promise of Evidence-Based Policymaking 1687
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 16, 2016

Status: Posted

Posted: November 16, 2016

Tracking No. 1k0-8t2d-ilyd

Comments Due: December 14, 2016

Submission Type: API

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0167

Comment on FR Doc # 2016-22002

Submitter Information

Name: Todd Hall

Address:

Cambridge, MA, 02139

Email: thall@povertyactionlab.org

General Comment

See attached file(s)

Attachments

2016.11.16 CEP Public Comment

1688 The Promise of Evidence-Based Policymaking
1 

November 16th 2016

Submitted Electronically 

Shelly Martinez,

Executive Director of the Commission on Evidence-Based Policymaking

Bureau of the Census

RE: U.S. Census Bureau Request for Comments for the Commission on Evidence-Based 

Policymaking (Federal Register Doc. 2016-22002)

Introduction

J-PAL North America (J-PAL NA), based in the Department of Economics at the Massachusetts 

Institute of Technology, leverages scholarship from 143 affiliated professors to generate and 

disseminate rigorous evidence about anti-poverty policies. J-PAL NA provides pro-bono

technical support, capacity building, and matchmaking with researchers to government agencies 

and nonprofits seeking to design and implement randomized evaluations, many of which rely 

extensively on administrative data. Affiliates in our network have conducted 154 ongoing or 

completed randomized evaluations in North America across sectors such as health care, housing, 

criminal justice, education, and labor markets. J-PAL NA also creates training materials to build 

research capacity, including a comprehensive, practical guide to obtaining and using 

administrative data for randomized evaluations.1 We appreciate the opportunity to submit 

comments to the Commission on Evidence-Based Policymaking. 

J-PAL affiliated researchers have relied heavily on administrative data to conduct policy-relevant 

research. Data from IRS tax records enabled an almost 20-year follow-up of families involved in 

the Moving to Opportunity Experiment. The follow-up study demonstrated that young children 

who moved to low-poverty neighborhoods increased their college attendance and expected 

lifetime earnings.2 Data from the U.S. Department of Education, the Ohio Board of Regents, and 

the National Student Clearinghouse collectively enabled a randomized controlled trial showing 

that simplifying the financial aid application process increased college attendance and 

persistence.3 Data from hospitals in the Portland area revealed that Medicaid insurance, for 

which opportunities to apply were allocated through a lottery in Oregon, increased emergency 

room usage by 40 percent.4 Access to administrative data was critical to generating these 

insights.

1 Feeney, Laura, Jason Bauman, Julia Chabrier, Geethi Mehra, and Michelle Woodford. “Using Administrative Data for 

Randomized Evaluations.” J-PAL North America. December 2015. 

https://www.povertyactionlab.org/sites/default/files/documents/AdminDataGuide.pdf 2 Chetty, Raj, Nathaniel Hendren, and Lawrence F. Katz. "The Effects of Exposure to Better Neighborhoods on Children: New 

Evidence from the Moving to Opportunity Experiment." NBER Working Paper #21156, May 2015.

3 Bettinger, Eric P., Bridget Terry Long, Philip Oreopoulos, and Lisa Sanbonmatsu. 2012. "The Role of Application Assistance 

and Information in College Decisions: Results from the H&R Block Fafsa Experiment." The Quarterly Journal of Economics 

127(3):1205-42.

4 Taubman, Sarah, Heidi Allen, Bill Wright, Katherine Baicker, Amy Finkelstein, and the Oregon Health Study Group. 2014. 

"Medicaid Increases Emergency Department Use: Evidence from Oregon's Health Insurance Experiment." Science 

343(6188):263-8.

The Promise of Evidence-Based Policymaking 1689
2 

Executive Summary 

This comment reflects J-PAL’s expertise concerning randomized evaluations, administrative data 

access, and collaboration between government agencies and external researchers. It incorporates 

recommendations from an open letter penned by several leading economists, including multiple 

J-PAL affiliates5

, and a short paper published by a subset of the same authors.6 This comment 

elaborates on these key recommendations:

x Establish clear data documentation and standard data request forms, building on the 

example set by the Centers for Medicare and Medicaid Services. 

x Expand secure access to real microdata to qualified researchers, prioritizing secure 

remote connections while also increasing capacity at Census Research Data Centers.

x Develop a data clearinghouse within the Census Bureau for currently hard-to-access data, 

particularly microdata on earnings and income, and link the data across agencies.

x Avoid flat per-user fees for data access to encourage validation and double-checking of 

data analysis.

x Clearly articulate program objectives and build ongoing process evaluation into every 

program to lay the foundation for impact evaluation. 

x Institutionalize a process for identifying questions for program evaluation and appropriate 

conditions for randomized evaluations, focusing on three cases:

o Demand for a program exceeds capacity to supply the program.

o Gradual roll out of a program to different individuals or locations over time.

o Refinement or reconsideration of eligibility criteria for a program.

Responses to Specific Questions

3. Based on identified best practices and existing examples, how should existing 

government data infrastructure be modified to best facilitate use of and access to 

administrative and survey data?

Build on the example of the Centers for Medicare and Medicaid Services to establish clear 

data documentation and standard data requests. 

Existing government data infrastructure should incorporate standard data request forms with 

clear data dictionaries, using the Centers for Medicare and Medicaid Services (CMS) Research 

Data Assistance Center (ResDAC) data documentation as a model of best practices. The 

ResDAC system allows researchers to understand specifically what variables are available and to 

submit requests with data protection plans. Because the ResDAC system allows CMS to review 

those requests systematically as opposed to on an ad hoc basis, ResDAC facilitates routine,

 

5 Card, David, Raj Chetty, David Cutler, Steven Davis, Martin Feldstein, William Gale, Jonathan Gruber, Michael Greenstone, 

Caroline Hoxby, Lawrence Katz, and Emmanuel Saez. 2010. “An Open Letter on Expanding Access to Administrative Data for 

Research in the United States. https://eml.berkeley.edu/~saez/dataaccess_final.pdf 6 Card, David, Raj Chetty, Martin S. Feldstein, and Emmanuel Saez. "Expanding access to administrative data for research in the 

United States." American Economic Association, Ten Years and Beyond: Economists Answer NSF's Call for Long-Term 

Research Agendas (2010). https://eml.berkeley.edu/~saez/card-chetty-feldstein-saezNSF10dataaccess.pdf 

1690 The Promise of Evidence-Based Policymaking
3 

secure access to administrative data that culminates in several hundreds of medical studies each

year.7

Applying the ResDAC model to an administrative data clearinghouse or other data repositories

would allow researchers to see exactly what variables they are permitted to request, along with a 

brief description of each variable, before submitting a request. This explicit listing of available

variables would enable data discovery and save program administrators and researchers hours of 

valuable time by avoiding long correspondences about whether the desired data exist.

Publicizing exactly which variables exist and what agency houses the data increases transparency 

with no risk of revealing personally identifiable information. Better data documentation can thus 

facilitate use of and access to administrative and survey data without raising concerns for data 

security and privacy protection. 

Moreover, sensitive variables that would trigger additional levels of review or security could be 

clearly labeled as sensitive as part of this clearer data documentation. Currently, researchers may 

request a variable that is not central to their analysis, without realizing that it captures sensitive 

information. This could delay or jeopardize the entire request or allow access to sensitive data 

that, with clearer data documentation, would not have been requested from the agency.8

4. What data-sharing infrastructure should be used to facilitate data merging, linking, and 

access for research, evaluation, and analysis purposes?

Expand secure access to microdata to qualified researchers through remote and on-site 

connections rather than creating synthetic data.

An optimal infrastructure for integrating administrative, survey, and statistical data to facilitate 

research and evaluation while ensuring data security and privacy will provide secure 

environments where qualified researchers can directly access microdata. Microdata enable 

researchers to perform more informative analyses by controlling for individual characteristics 

(such as educational attainment or race) to better determine the impact of a program. Microdata 

also allow researchers to evaluate how a program affects specific subpopulations, such as lowincome individuals. Researchers can use microdata to validate and adjust their analysis as they 

learn from the data in real time—a crucial step in the research process. Moreover, for analysis in 

rigorous randomized evaluations, researchers require microdata to link individuals to their 

treatment status. 

There are currently twenty-four Federal Statistical Research Data Centers (RDCs), which are 

physical, secure environments established through partnerships between the Census Bureau and 

research institutions where researchers who have undergone special sworn status can access 

restricted microdata. However, capacity in these RDCs is limited, both physically and according 

to Census bandwidth, and access is artificially restricted to researchers based on geographic 

proximity rather than on the merit of their research proposal.9 Similar constraints apply to 

researchers working with statutorily restricted tax data through contracts with the IRS Statistics 

 

7 Ibid.

8 Feeney et al., “Using Administrative Data for Randomized Evaluations,” 2015. 

9 Card et al., "Expanding access to administrative data,” 2010b.

The Promise of Evidence-Based Policymaking 1691
4 

of Income Division (SOI)—the type of arrangement that enabled the Moving to Opportunity 

follow-up study. The SOI is small, has a limited budget, and can accommodate few research 

projects at a time. 10 More secure, direct access to microdata should be provided in two ways:

(1) preferably through remote, secure connections such as the “flexiplace” systems used by 

federal employees who work with restricted data from home, but also (2) through expansion of 

on-site secure environments in the form of additional space and funding for Research Data 

Centers and similar centers at other statistical agencies.11

Synthetic data, one alternative to expanding remote and on-site secure connections to restricted 

data, are a far inferior option for enabling policy-relevant research and program evaluation. 

Synthetic data are constructed to mimic certain features and aggregate characteristics of real data 

without containing real individual-level information. Although this appears—on its surface—to 

enable research while protecting privacy, synthetic data suffer severe disadvantages relative to

real microdata. Synthetic data may be incompatible with randomized evaluations and other

rigorous program evaluations because researchers must be able to link individuals to their 

treatment status—i.e., whether a particular person received a program or not. Furthermore, 

synthetic data make it difficult or impossible to study subpopulations, such as low-income 

individuals, which may be of particular policy interest. Researchers would have to specify each 

subpopulation they intend to study and all necessary contents of the data in advance. This may be 

impossible, in part because researchers often revise their analyses to address observations they 

learn from the raw data.12 Meanwhile, data administrators would have to create new synthetic

datasets for each request to study a specified subpopulation, which would require significant 

infrastructure and personnel.

7. What data should be included in a potential U.S. government data clearinghouse? What 

are the current legal or administrative barriers to including such data in a clearinghouse or 

linking the data?

Establish a data clearinghouse within the Census Bureau for currently hard-to-access data, 

particularly microdata on earnings and income. 

For reasons discussed in response to question 4, the data should be real microdata rather than 

aggregated, de-identified, synthetic, or perturbed data. The clearinghouse should prioritize data 

that do not already benefit from strong infrastructure for access. Specifically, a clearinghouse 

should be developed for federal income and earnings microdata and focus on enabling 

researchers to link these data to the extent legally possible.

Income and earnings data have less well-developed access infrastructure and face several legal

barriers to use for program evaluation, meaning that the clearinghouse would not be redundant.13

For example, individual states maintain their own data system for Unemployment Insurance (UI) 

 

10 Office of Management and Budget, Barriers to Using Administrative Data for Evidence-Building (white paper submitted for 

the Commission on Evidence-Based Policymaking, July 15, 2016). https://www.whitehouse.gov/sites/default/files/omb/mgmtgpra/barriers_to_using_administrative_data_for_evidence_building.pdf

11 Card et al., "Expanding access to administrative data,” 2010b. 12 Ibid. 13 Ibid.

1692 The Promise of Evidence-Based Policymaking
5 

records, with individual discretion and statutory protections on providing access to this data. The 

Department of Labor does not store the data in a central location. Although the Census Bureau 

has made a significant contribution to accessing state UI data through the Longitudinal Employer 

Household Dynamics Program (LEHD), the LEHD program requires that researchers be on-site 

at a designated Research Data Center.14

With infrastructure secure enough for the highly restricted data from UI and tax records 

established, the clearinghouse should then focus on facilitating linkage of these data with other, 

less restricted data. Federal data on income, namely tax records or Unemployment Insurance 

records, are in high demand because income can serve as a key outcome variable for many 

government programs or policies in education, job training, criminal justice, and place-based 

interventions.15 For example, earnings—as measured by tax records—was a key outcome 

variable in the follow-up study of the Moving to Opportunity Experiment.

8. What factors or strategies should the Commission consider for how a clearinghouse(s) 

could be self-funded? What successful examples exist for self-financing related to similar 

purposes?

Avoid flat per-user fees for data access to encourage validation and double-checking of 

data analysis. 

As is customary, a clearinghouse may charge fees for accessing data, such as a fee per project, a 

fee for sets of users, or an initial fee for the first user followed by much smaller fees for 

additional users. It should not charge the same flat fee per person for accessing the data because 

this severely discourages the double checking crucial to correcting human errors. Despite its 

clear data request process, CMS charges a fixed fee of $25,000 per person who accesses

identifiable data through their Virtual Research Data Center.16 This may create problems because 

researchers often need multiple people to work with the data to ensure accuracy—including 

people who effectively proofread to correct for human coding errors. With fixed per-person 

costs, researchers either pay a large inflexible sum of money for someone to double check the 

analysis or—facing tradeoffs given limited research funds—forego a set of “fresh eyes” to 

double check the analysis at risk of making mistakes. Therefore, by creating a high marginal cost 

to adding additional users, charging a fixed fee per data user effectively institutionalizes 

mistakes. 

18. How can or should program evaluation be incorporated into program designs? What 

specific examples demonstrate where evaluation has been successfully incorporated in 

program designs?

Clearly articulate program objectives and build process evaluation into every program to 

lay the foundation for impact evaluation.

 

14 Ibid. 15 Office of Management and Budget, Barriers to Using Administrative Data, 2016. 16 Research Data Assistance Center. “Fee List for RIFs: Physical Research Data Requests.” Centers for Medicare and Medicaid 

Services. https://www.resdac.org/sites/resdac.umn.edu/files/CMS%20Price%20List%20for%20Research%20Files_23.pdf

Accessed October 31, 2016.

The Promise of Evidence-Based Policymaking 1693
6 

Process evaluation is always needed and constitutes a critical prerequisite for impact evaluation. 

It is not sensible to ask whether the program is succeeding or failing to deliver outcomes without

first knowing whether the program itself is being delivered with fidelity. In some cases, 

important questions about how a program can or should function may be sufficiently answered 

by process evaluations, needs assessments, or literature reviews. Rigorous impact evaluation,

particularly randomized evaluation, should be pursued when the benefits in terms of knowledge 

generated would likely outweigh the costs of the evaluation, and when planning during program 

design can facilitate impact evaluation. 17

Many components that aid process and impact evaluation should be developed during program 

design:

x Precisely articulated program objectives. 

x A needs assessment clearly articulating the problem that the program will address.

x Standard outcome measures used in research literature about similar programs that allow 

potential impact evaluation results to be compared to those in other studies and used in 

cost-effectiveness analyses.

x A plan for data collection and flow from program practitioners to administrators. This 

includes planning in advance to collect identifying information, such as Medicaid ID 

numbers, to enable later matching of program-level records to administrative records for 

impact evaluation. 

As an example of incorporating evaluation into program design, Benefits Data Trust (BDT) is 

working with J-PAL North America in an ongoing randomized evaluation of different outreach 

strategies to increase enrollment in the Supplemental Nutrition Assistance Program (SNAP) 

among eligible but unenrolled individuals in Pennsylvania. BDT had a clear grasp of the need its 

program addressed: despite awareness among eligible households that SNAP exists, many people 

could not imagine navigating the enrollment process alone. BDT had the clear program objective 

of increasing benefits enrollment, and change in program enrollment is a standard outcome that 

could be compared across different studies. BDT and researchers agreed that based on a review

of the existing research, there was little rigorous evidence about what interventions can increase 

SNAP enrollment. 

Although BDT was already providing enrollment assistance and sending outreach, BDT worked 

with researchers to design and test two distinct outreach activities—one high-touch intervention 

including a letter plus enrollment assistance and one low-touch intervention including a letter 

only. The researchers also worked with BDT to design a new letter for the evaluation based on 

marketing and psychology literature. Seeing quickly that this newly designed letter was more 

effective, BDT plans to incorporate this letter design in other states outside of Pennsylvania. 

Ultimately, the impact of the different outreach strategies will be measured using administrative 

data, which can be accessed according to a data use agreement with the Pennsylvania 

Department of Human Services. 

 

17 Glennerster, Rachel, and Kudzai Takavarasha. Running randomized evaluations: A practical guide. Princeton University Press, 

2013.

1694 The Promise of Evidence-Based Policymaking
7 

19. To what extent should evaluations specifically with either experimental (sometimes 

referred to as “randomized control trials”) or quasi-experimental designs be 

institutionalized in programs? What specific examples demonstrate where such 

institutionalization has been successful and what best practices exist for doing so?

Institutionalize a process for developing specific research questions and determining the 

appropriate conditions for randomized control trials or other evaluation methods.

Federal agencies should institutionalize a process of developing high-priority research questions 

and determining the most appropriate evaluation methodology, following the precedent set by

the Social and Behavioral Sciences Team (SBST). SBST launches demonstration projects—

usually in the form of randomized evaluations—to rapidly evaluate low-cost applications of 

behavioral science to achieve desired outcomes, such as increasing workplace savings plan 

enrollment among military service members or increasing the rate at which indebted graduates 

apply for income-based loan repayment plans.18

When properly designed and implemented, randomized evaluations rigorously demonstrate the

causal impact of a program by establishing the counterfactual—what outcomes would exist for

program participants if they had not received the program. Random assignment ensures that, 

with a large enough sample, the group that receives the program and the group that does not are 

similar on average before the start of the program. Therefore the impact estimate from a 

randomized evaluation offers confidence that any differences in outcomes between the two 

groups are a result of the program. The ability to isolate program impact from self-selection or 

other confounding factors is why randomized evaluations are widely recognized as a highly 

credible method for estimating program impact. Where there is little internal experience 

implementing randomized evaluations, agencies should seek partnerships with external or 

academic researchers who are vested in similar questions.

Randomized evaluations can only occur when randomization is built into the program design.

However, randomization should not be incorporated indiscriminately; rather, randomization

should be incorporated into programs to facilitate randomized evaluations where appropriate on

three grounds:

x The current evidence for answering the well-defined research question is non-existent, 

insufficient, or inconclusive.

x There is a clear unit of randomization—individual program participants, schools, clinics, 

etc.—for which there is a large enough sample size and a clear means of tracking 

outcomes for both the treatment group and the control group.

x Randomization is feasible and ethical. Although not an exhaustive list, the following 

conditions offer opportunities where randomization may be feasible and ethical:

o Demand for a program exceeds capacity to provide the program. A lottery may be 

a fairer alternative than allocating slots on a first come, first served basis—

particularly when a goal of the program is equity of access—and offers an

 

18 Social and Behavioral Sciences Team. 2016 Annual Report. Executive Office of the President, National Science and 

Technology Council. September 15, 2016. https://sbst.gov/assets/files/2016%20SBST%20Annual%20Report.pdf

The Promise of Evidence-Based Policymaking 1695
8 

alternative to imposing increasingly narrow eligibility criteria under funding 

constraints. 

o A program is being expanded by gradually offering it to individuals, schools, or 

districts until full coverage is reached. A lottery can be used to randomly assign

the order in which individuals or units receive the program. The individuals or 

units that have not yet received the program serve as the control group until all 

units receive the program.

o A new intervention—such as a financial incentive or care coordination services—

will be added to an existing program. Program participants can be randomized to 

receive different versions of the program, e.g., with or without the added 

intervention, to isolate the impact of the new intervention.

o Program eligibility criteria are being refined or reconsidered. People just

above/below the eligibility cutoff can be randomly assigned to receive or not 

receive the program to determine whether it is effective for this marginal group.

Meanwhile, those well within the program eligibility cutoff, automatically receive 

the program, and those well outside the cutoff do not qualify for the program.

As a specific example of institutionalizing a randomized evaluation, the South Carolina 

Department of Health and Human Services is partnering with J-PAL North America to 

incorporate a randomized evaluation in its expansion of a nurse home-visiting program for lowincome mothers delivered by the nonprofit organization Nurse-Family Partnership (NFP). The 

specific, high-priority research question is whether a new, less expensive version of the NFP 

program that South Carolina is expanding will be as effective as the pre-existing version that has 

been rigorously evaluated before.

A randomized evaluation was found to be an appropriate method for answering this question 

given excess demand for the program. Although South Carolina is expanding this less expensive 

version of NFP to thousands of mothers through an innovative pay-for-success initiative, the 

program does not have sufficient resources to serve all of the women who are eligible.

Applicants will be randomly assigned, on a rolling basis from 2016 to 2020, to either a treatment 

group that is offered access to the program, or to a control group that is not.19 We will assess the 

effect of NFP on a range of short- and long-run maternal and child outcomes using 

administrative data that will be available for all members of both treatment and control groups. 

This will yield useful evidence for South Carolina and for policy makers nationally, who are 

interested in the broader health and financial consequences of expanding Medicaid to include 

similar services.

 

19 The Abdul Latif Jameel Poverty Action Lab. “The Impact of a Nurse Home Visiting Program on Maternal and Child Health 

Outcomes in the United States.” Accessed October 24, 2016. https://www.povertyactionlab.org/evaluation/impact-nurse-homevisiting-program-maternal-and-child-health-outcomes-united-states

1696 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 17, 2016

Status: Posted

Posted: November 17, 2016

Tracking No. 1k0-8t30-8u9g

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0168

Comment on FR Doc # 2016-22002

Submitter Information

Name: Diane Freese

Address:

Merrritt Island, FL, 32953

Email: caligirl005@yahoo.com

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

Our family has openly, reasonably protested expansive, invasive national student data

collection since 2005 by opting our students out of internet use at school. Our local

agency (School District) retaliated against our students and family due to our protests,

(non-conformance) regarding internet use. We reported the District's informal policies of

retaliation and more retaliation followed. 

We do not believe that students are obligated to provide data in exchange for their

Uniform, Safe, Secure, Efficient and High Quality public education. In fact, we believe

internet use at school and the accompanying data collection makes them unsafe. We

strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all students, commonly

referred to as a student unit-record system or national database.

The Promise of Evidence-Based Policymaking 1697
The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008, the reasons against creating it

have only become more compelling.

In the past few years, much highly personal data held by federal agencies has been

hacked, including the release of the records of the Office of Personnel Management

involving more than 22 million individuals, not only federal employees and contractors

but also their families and friends.

The US Department of Education has been found to have especially weak security

standards in its collection and storage of student data, and received a grade of D for its

security protections.

In addition, preK-12 student data currently collected by state departments of education

that would potentially be shared with the federal government include upwards of 700

highly sensitive personal data elements, including students' immigrant status, disabilities,

disciplinary records, and homelessness data.

We are also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from PreK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

We urge you to strongly oppose the creation of any centralized federal data system

holding students' personally identifiable information and to support the continuation of

the ban in the report you provide to Congress.

1698 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 17, 2016

Status: Posted

Posted: November 17, 2016

Tracking No. 1k0-8t34-5vgq

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0169

Comment on FR Doc # 2016-22002

Submitter Information

Name: Chad Green

Address:

Leesburg, VA, 20175

Email: chadgreen@verizon.net

General Comment

Today's societal demands require a fundamental structural shift from bureaucratic

government to networked governance, so how can these "successful frameworks,

policies, practices, and methods" facilitate the process?

For example, see: 

Society Is Too Complicated to Have a President, Complex Mathematics Suggest

http://motherboard.vice.com/read/society-is-too-complicated-to-have-a-presidentcomplex-mathematics-suggest

and:

From "Government" to "Governance" in Public Administration: Differences, Concepts

and Theories

http://patheory.net/docs/2016ConferenceDocs/Qi_H.pdf

The Promise of Evidence-Based Policymaking 1699
Best,

Chad

1700 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 17, 2016

Status: Posted

Posted: November 18, 2016

Tracking No. 1k0-8t3a-fmvn

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0170

Comment on FR Doc # 2016-22002

Submitter Information

Name: Mary Jo Hoeksema

Address: United States,

Email: maryjo@popassoc.org

General Comment

See attached.

Attachments

PAA APC comments to Evidence Based Policy Commission Final

The Promise of Evidence-Based Policymaking 1701
1 

 November 14, 2016 

Commission on Evidence-Based Policymaking 

Comments: Docket ID USBC-2016-0003 

To whom it may concern: 

On behalf of the Population Association of America 

(PAA) and the Association of Population Centers 

(APC), we are pleased to respond to the Request for 

Comments for the Commission on Evidence-Based 

Policymaking. 

The PAA and APC are two affiliated organizations 

that together represent over 3,000 social and 

behavioral scientists, including demographers, 

economists, and sociologists, and the approximately 

40 federally-supported population research centers 

nationwide. PAA and APC members conduct 

interdisciplinary scientific research and research 

training to further understanding about the 

implications of population change. 

Many PAA members and affiliates of APC members 

are engaged in research that informs policy decision 

making and our understanding of the impacts of 

public policy on both the U.S. population and 

populations worldwide. Our members’ research is 

decidedly evidence-based, making use of the best 

data sources, including those collected by the U.S. 

statistical agencies and those from other countries 

and international organizations. The PAA maintains a 

standing Committee on Population Statistics whose 

purpose is to continually monitor and provide input to 

U.S. statistical agencies on the quality of data 

collected by these agencies and their availability to 

the research community. Furthermore, many of our 

members have been at the forefront of developing 

statistical methods that are used to evaluate public 

policy questions in such areas as population health, 

fertility and mortality, and other demographic, social 

1702 The Promise of Evidence-Based Policymaking
2 

and economic outcomes central to so many policy issues. 

PAA and APC applaud passage of the Evidence-Based Policymaking 

Commission Act of 2016, which created the bipartisan Commission on 

Evidence-Based Policymaking and charged this Commission with: 

x determining how to integrate administrative and survey data and to 

make those data available to facilitate research, evaluation, analysis, 

and continuous improvement while protecting privacy and 

confidentiality; 

x recommending how data infrastructure, database security, and statistical 

protocols should be modified to best fulfill the integration and 

increased availability of data as described above; 

x recommending how best to incorporate rigorous evaluation into 

program design; and, 

x considering whether a federal clearinghouse should be created for 

government survey and administrative data. 

We are pleased to continue supporting the Commission by responding to 

the recent request for comments published in the Federal Register on 

September 14. Our responses address questions (#3-7, 9, and 10) in the 

notice dealing with data infrastructure and access-- two issues that are 

central to the missions of both of our organizations and to the population 

sciences. Further, some of our comments are outside the scope of the 

questions, reflecting issues that our organizations hope the Commission 

will consider. 

Comments: Data Infrastructure and Access 

A. There are important benefits to use of administrative data, especially 

when linked, for conducting policy-relevant research. Administrative 

records have been used in a variety of research areas and provide an 

essential source of data for conducting important policy-relevant research. 

For example, such records have been used to study participation in and 

impacts of social programs (e.g., welfare programs, manpower training, 

food stamps, the earned income tax credit, etc.) on various outcomes. Often 

these outcomes are measured with linked administrative data, such as wage 

earnings (from linked unemployment insurance wage records), health 

conditions (from linked Medicaid records) or fertility (from linked birth 

certificate records). The availability of administrative records from federal, 

state or local sources provide a cost-effective way of supporting 

evaluations of these programs, regardless of whether the evaluations made 

The Promise of Evidence-Based Policymaking 1703
3 

use of randomized designs for allocation of program participants to 

different “treatments,” or other studies that have made use of nonexperimental designs. 

But social program evaluation is not the only place where administrative 

records can and will be the primary source of data to monitor particular 

programs and/or evaluate particular policies or “treatments.” Furthermore, 

they do not only use government records. Here we reference two examples. 

First, biomedical research, including research that is relevant to policies 

affecting health-related behaviors, such as smoking bans or regulation of 

the nutritional content of foods, uses increasingly electronic health records 

(EHRs) from public and private health care systems to measure the health 

effects of variation in such policies. Second, administrative records from 

private firms that construct credit scores for use by financial institutions 

have been used by researchers, including the research division of the New 

York Federal Reserve Bank, to monitor and conduct policy-relevant 

research on student loan debt in the U.S. In both of these areas, 

administrative records support important policy-relevant research in a way 

that is both cost-effective and potentially more accurate than data collected 

via other means, e.g. surveys. 

B At the same time, there are important legal and other constraints that 

limit the use of administrative records and the ability of researchers to link 

records from different administrative sources. In particular, different 

sources of administrative data are subject to varying and divergent laws and 

regulations that can inhibit their use. For example, administrative records 

from social programs administered at the state or local level (e.g., TANF 

programs) are often subject to laws and regulations that make it hard for 

one agency to share their records with another agency. And, as noted in the 

NRC report on the Reengineering of the Survey of Income and Program 

Participation (SIPP), existing state laws that cover the privacy and access 

of administrative records from TANF, Medicaid, unemployment insurance, 

and the workers’ compensation programs make it very difficult, if not 

impossible, for these programs to share their data with the Census Bureau 

(or other) surveys like the SIPP.1

 1 Constance Citro and John Karl Scholz, eds., Reengineering the Survey of Income 

and Program Participation, National Research Council, 2009. 

1704 The Promise of Evidence-Based Policymaking
4 

Historically, this issue has complicated the conduct of biomedical research 

that makes use of electronic (or non-electronic) health care records of 

individuals as institutional review boards (IRBs) have required studies to 

obtain informed consent from subjects in these studies for any follow-up 

use of subjects’ EHRs and/or updating of these records. Recently proposed 

revisions to the Common Rule2

 will reduce and/or eliminate this reconsenting requirement for certain types of studies and types of 

administrative records so long as subjects are provided with a clear 

statement regarding potential future use of administrative records as part of 

their initial consent process. Many population scientists welcome this 

change and suggest it may represent a model for the Commission to 

examine as it considers how to facilitate access to records like EHRs while 

still providing participants with the opportunity to make informed decisions 

about research access to their records. 

More generally, we strongly urge the Commission to investigate the 

various laws and regulations governing access to administrative records for 

research purposes. In particular, we urge the Commission to look closely at 

the laws affecting access to state and local government data and policies 

restricting record linkage across various federal agencies. 

C. To facilitate the conduct of evidence-based, policy-relevant research, we 

urge the Commission to examine and improve access to administrative 

records to qualified researchers outside and inside the government. We 

understand and appreciate that there are important confidentiality and 

security concerns that necessarily limit access of researchers to various 

types of government-based administrative records and/or restricted-use data 

sources. Furthermore, we appreciate why restrictions on the access of nongovernmental researchers may need to be different, and possibly more 

restrictive, than that applied to researchers employed by authorized 

government agencies. But, at times, these restrictions have made access to 

such data very difficult for academic and non-governmental researchers. 

Over the last 20 years, U.S. statistical agencies, initially led by the U.S. 

Census Bureau, have made great strides in improving access to restricteduse versions of federal data sources through the Federal Statistical 

 

2 HHS–OPHS–2015–008, Federal Policy for the Protection of Human Subjects, 

Federal Register, 80(173), Sept 8, 2015, 53933-54058. 

The Promise of Evidence-Based Policymaking 1705
5 

Research Data Centers (RDCs) program. This program now allows access 

to data products from 12 different federal statistical agencies for qualified 

governmental and non-governmental researchers in 20 different centers 

around the country. While some the research covered by the data 

agreements approved for use of these centers is often not directly related to 

policymaking, much of it is. 

A similar effort for providing access to data from the Internal Revenue 

Service (IRS) under the Joint Statistical Program of the Statistics of Income 

(SOI) Division of the IRS has enabled qualified researchers to submit 

proposals for access to IRS data and to link it to various data for research 

purposes. This program has facilitated a number of highly visible and 

widely cited lines of research by Professors Raj Chetty (Stanford) and 

Emmanuel Saez (UC Berkeley). For example, Chetty and co-authors 

analyzed the association between income and the life expectancy of 

individuals in the U.S. since 2000 by linking IRS tax records on income 

with Social Security Administration death records.3

 The findings of this 

research, especially the finding of differences by area in the associations of 

longevity by income, raises important questions about the sources of these 

disparities and how to alleviate these differences. Such research could not 

have been conducted without this program. 

A large body of research shows that geography (e.g. neighborhoods) affects 

the social and economic well-being and health of individuals and families. 

But, state and local policymakers, researchers, and program officials often 

lack the data needed to measure differences in community environments, to 

isolate how neighborhood characteristics shape micro-level outcomes, or to 

test the efficacy of neighborhood-level interventions. Most survey data files 

lack such key contextual information, while most administrative data lack 

key demographic, socioeconomic, behavioral, and outcome information. 

While individual-level record linkage of survey and administrative data 

could provide such critical data for state and local-level evidence-based 

policymaking, most state and local researchers/program evaluators lack the 

resources to submit proposals and conduct these types of linkages and 

research within a RDC. The Commission should also encourage statistical 

 

3 Chetty, R. et al. (2016). “The Association between Income and Life Expectancy 

in the United States, 2001-2014, Journal of the American Medical Association, 

315(15): 1750-1766. 

1706 The Promise of Evidence-Based Policymaking
6 

agencies and other researchers to create spatially-linked administrative and 

survey data that could be provided to state and local researchers/program 

evaluators outside of RDCs to increase evidence-based policymaking at the 

state and local level. 

We urge the Commission to encourage expanding access of data and 

records from federal, state and local sources to qualified non-governmental 

and governmental researchers. This expansion should include state and 

local government researchers, whose access to data can provide support for 

accurately assessing needs, creating programs to address those needs, and 

delivering services in more cost-effective ways. While such efforts may 

include expanding the RDC and/or IRS’s Joint Statistical Programs or 

similar programs, they should also include expanding access to spatiallylinked administrative and survey data that could be provided outside of 

RDCs. Efforts also may include providing more funding for merit-based 

grants to undertake these projects, especially in light of the limited 

resources available to researchers in local governments and those working 

in non-governmental settings. 

We also urge the Commission to consider recommending any necessary 

legal revisions that would allow federal statistical agencies to share data 

with researchers conducting evidence based research. For example, the 

Census Bureau’s authorizing regulation, Title 13, does not explicitly 

recognize the use of sensitive data for conducting scientific research, be it 

policy-relevant or not, as a “benefit to the Bureau.” Rather, Title 13 only 

supports data access to improve the quality of Census (and other) data 

products. A more explicit acknowledgment that qualified research projects 

can be conducted for scientific purposes would allow Census to approve 

studies using confidential data that are primarily designed to replicate 

existing studies and/or determine the robustness of findings from previous 

research. Such changes would help ensure the legitimacy of research uses 

of these data and give greater credibility to the findings based upon these 

data. 

D. We encourage greater attention be given to the population 

representativeness of the policy-relevant research produced using 

administrative records and population-based surveys to better assess and 

characterize the population-representativeness of findings from 

administrative data. Many studies use administrative records to “evaluate” 

the impact of some particular policy or program. As we have argued above, 

the administrative records provide a potentially cost-effective way of 

conducting such evaluations—particularly when compared to the 

The Promise of Evidence-Based Policymaking 1707
7 

alternative of collecting survey data that is collected from a sample 

representative of the population relevant for the study. However, such 

benefits of using administrative records in evaluative research does not 

mitigate the importance of assessing the sampling properties of this data 

source. 

Consider the following example regarding the design of the Precision 

Medicine Initiative (PMI). One of the key components of the PMI’s initial 

plan is to assemble a million-person sample of individuals who would 

provide access to their Electronic Health Records (EHRs) as a condition of 

the study. Access to EHRs on this large sample would provide data to study 

a wide range of health conditions, including conditions that are relatively 

rare and only affect population subgroups. One of the study’s recruitment 

strategies was to use social media and other methods to attract participants 

who would grant access to their EHRs and undergo one or more physical 

examinations. 

While the goals of the PMI are important and have the potential to provide 

evidence-based assessments of health conditions relevant for U.S. health 

policy, as population scientists, we are concerned about lack of attention to 

the properties of what amounts to a “volunteer” sample of people with 

EHRs, even if the sample includes data on one million participants. In 

public comments, PAA and APC raised these concerns and strongly 

suggested that the NIH leadership consider using existing population-based 

health studies to form at least part of the PMI cohort to assess the 

population-representativeness of the recruitment strategy based on 

volunteers. In developing both policies and best practices for policyrelevant research, we urge the Commission to advocate for the designs of 

data collection that explicitly account for the sampling properties and 

population-representativeness of its studies. 

Lastly, we encourage the Commission to ensure that populationrepresentative data sources collected by the Federal government continue to 

be viewed as an important source of data for policy-relevant research, both 

as a way to monitor behaviors and phenomena relevant to public policy. 

For example, data sources like the Current Population Survey (CPS), the 

American Community Survey (ACS) and the Survey of Income and 

Program Participation (SIPP) all play roles in the monitoring and 

implementing public policy in the U.S. The CPS is the populationrepresentative data that enables the BLS to construct estimates of 

unemployment and labor force participation rates of the U.S. population on 

a monthly basis. The ACS provides data on poverty rates at the lowest 

1708 The Promise of Evidence-Based Policymaking
8 

levels of geography, such as school districts and communities, which are 

used to allocate funding for programs such as the USDA’s National School 

Lunch Program and State Children’s Health Insurance Program (SCHIP). 

The SIPP has facilitated a broad range of research on the distribution of 

income and participation in a range of social programs using a survey that 

is designed to be population representative for most states in the U.S. These 

surveys, and others, are important components of the U.S. data 

infrastructure and are needed to support evidence-based policymaking. 

Thank you for considering our comments and for the important work of the 

Commission. Please do not hesitate to contact the Director of Government 

and Public Affairs, Mary Jo Hoeksema (maryjo@popassoc.org), if our 

organizations can be of further assistance. 

Sincerely, 

 

 Judith A. Seltzer 

President, Population Association of America 

 

 Lisa Berkman 

President, Association of Population Centers 

The Promise of Evidence-Based Policymaking 1709
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 19, 2016

Status: Posted

Posted: November 21, 2016

Tracking No. 1k0-8t4k-udei

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0171

Comment on FR Doc # 2016-22002

Submitter Information

Name: Florida Parent

General Comment

In the interest of our Children's Constitutional Rights we must stop Common Core. The

schools need to be managed and controlled at the local level, overseen by the state. It is

completely against everything our Founding Fathers had in mind when they organized

this country. Common Core is Communism - it does not belong in the USA - it is ANTIAMERICAN. STOP Common Core NOW!!!

1710 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 19, 2016

Status: Posted

Posted: November 21, 2016

Tracking No. 1k0-8t4r-qi7w

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0172

Comment on FR Doc # 2016-22002

Submitter Information

Name: Timothy Law

General Comment

I am against the ever increasing thirst for data on the population in general. I am

particularly against that information on our children. That belongs in the local school,

not in a national database.

If you need information to make decisions, then take a sample data study. You don't

need ever intrusive data on every child or every person. In the end, that becomes a way

around the 5th Amendment. 

Timothy Law

Minnetonka, MN

The Promise of Evidence-Based Policymaking 1711
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 20, 2016

Status: Posted

Posted: November 21, 2016

Tracking No. 1k0-8t52-zlqs

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0173

Comment on FR Doc # 2016-22002

Submitter Information

Name: James Poulsen

General Comment

Common Core is big government fools brainwashing every chiold in America as they

have since 1965 only now they will add Marxism to the curricula.Schools should be a

state and local issue nothing should be big government it all ends up l;ike Detroit or

Baltimore.

1712 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 20, 2016

Status: Posted

Posted: November 21, 2016

Tracking No. 1k0-8t56-gws4

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0174

Comment on FR Doc # 2016-22002

Submitter Information

Name: Chris Adams

General Comment

I find it deplorable that what was supposed to be an accountability registry has grown to

encompass a cradle to grave

" Human Capital". tracking system. This needs to be stopped.

The Promise of Evidence-Based Policymaking 1713
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 20, 2016

Status: Posted

Posted: November 21, 2016

Tracking No. 1k0-8t57-xvme

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0175

Comment on FR Doc # 2016-22002

Submitter Information

Name: Aaron Hsu

General Comment

I strongly oppose measures by the government to perform long term tracking of

individuals, whether for the benefit of society or not.

We must strongly resist the urge to collect and acquire such data.

1714 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 21, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t5p-7m4d

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0176

Comment on FR Doc # 2016-22002

Submitter Information

Name: Cheryl Pedone

General Comment

We believe that student privacy and parental consent should always be considered preeminent compared to the research

desires of the government or private sector, especially in the realm of psychological

profiling.

The government has no constitutional, statutory, or moral right to collect data, especially

highly personal and sensitive 

socioemotional data on our children.

The Promise of Evidence-Based Policymaking 1715
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 21, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t5t-7ifj

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0177

Comment on FR Doc # 2016-22002

Submitter Information

Name: Frank Caprio

General Comment

I am against against the efforts to establish a federal "unit record" database. The idea that

the government should track children from their preschool years throughout their

academic life is a repulsive one. The Department of Education is an unconstitutional

entity. We need to eliminate it not give it a bigger agenda. Our 4th Amendment protected

rights to be secure in our "person, houses. papers, and effects" is also at stake here. There

is no proper reason for the government to collect this data. The prohibition that exists is

there for a good reason and needs to stay in force. As a matter of fact, ALL of Common

Core should be eliminated.

1716 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 21, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t5u-9io5

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0178

Comment on FR Doc # 2016-22002

Submitter Information

Name: Sharon Kraus

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive.

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

standards in its collection and storage of student data, and recently received a grade of

"D" for its security protections.

The Promise of Evidence-Based Policymaking 1717
In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban.

Yours,

Sharon Kraus

1718 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 21, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t5u-psem

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0179

Comment on FR Doc # 2016-22002

Submitter Information

Name: Suzanne Lanier Philips

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive.

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

standards in its collection and storage of student data, and recently received a grade of

"D" for its security protections.

The Promise of Evidence-Based Policymaking 1719
In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban.

Yours,

Suzanne Lanier Philips, Florida

1720 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 21, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t5u-wwvc

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0180

Comment on FR Doc # 2016-22002

Submitter Information

Name: Heather Aaron

General Comment

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive.

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

standards in its collection and storage of student data, and recently received a grade of

"D" for its security protections.

In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

The Promise of Evidence-Based Policymaking 1721
highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban.

Yours,

Heather Aaron

New Jersey

1722 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 21, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t5u-zjj7

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0181

Comment on FR Doc # 2016-22002

Submitter Information

Name: Tawnie Cisneros

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database. 

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive. 

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

standards in its collection and storage of student data, and recently received a grade of

"D" for its security protections. 

The Promise of Evidence-Based Policymaking 1723
In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness. 

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation. 

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban. 

Yours,

Tawnie Cisneros, Illinois

1724 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 21, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t5u-o1iz

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0182

Comment on FR Doc # 2016-22002

Submitter Information

Name: Tanya Pollard

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive.

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

standards in its collection and storage of student data, and recently received a grade of

"D" for its security protections.

The Promise of Evidence-Based Policymaking 1725
In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban.

Yours,

Tanya Pollard, Brooklyn College, CUNY, New York, NY

1726 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 21, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t5u-imxc

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0183

Comment on FR Doc # 2016-22002

Submitter Information

Name: Adam Trowbridge

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive.

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

standards in its collection and storage of student data, and recently received a grade of

"D" for its security protections.

The Promise of Evidence-Based Policymaking 1727
In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban.

Yours,

Adam Trowbridge

1728 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 21, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t5u-sifr

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0184

Comment on FR Doc # 2016-22002

Submitter Information

Name: Susan Gold

Address:

Malta, NY, 12020

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive.

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

The Promise of Evidence-Based Policymaking 1729
standards in its collection and storage of student data, and recently received a grade of

"D" for its security protections.

In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban.

1730 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 21, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t5u-z18o

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0185

Comment on FR Doc # 2016-22002

Submitter Information

Name: Wendy V

General Comment

I am writing to ask you to oppose the creation of any centralized federal data system

holding students' personally identifiable information and to support the continuation of

the ban.

I, along with several members of my family, have been involved in technical areas

touching on issues of privacy and security for many years. I have worked for over 11

years in web design / development and have spent MUCH of that time educating clients

on safety, security, and privacy of client information. My father worked for over 34

years at IBM, much of which was in security. He's currently the administrator of a new

college specifically training in cybersecurity. Risks to information held in a federal

database are real, not imaginary... think "when" not "if" a breach would happen. I realize

we don't live in the 1950s anymore, but the US Department of Education recently

received a grade of "D" for its security protections. I do not believe we have the *right*

to make this decision regarding the next generation of adults' information. 

This information, which would and could include upwards of 700 highly sensitive

elements, includes information such as students' immigrant status, disabilities,

disciplinary records, and homelessness. Aside from the threat of security breach, I

STRONGLY disagree with this information being collected on private citizens. 

The Promise of Evidence-Based Policymaking 1731
The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban.

1732 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 21, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t5u-nvqk

Comments Due: December 14, 2016

Submission Type: API

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0186

Comment on FR Doc # 2016-22002

Submitter Information

Name: C. Michael Barton

Address:

Tempe, AZ, 85287

Email: michael.barton@asu.edu

General Comment

CLEARINGHOUSE FOR INTEGRATIVE HUMAN ANALYTICS AND DATA

SYNTHESIS -- 

As a growing community of scientists who study human social and behavioral systems

(listed in attached document), we strongly support the goals of this commission to make

more effective use of rapidly expanding administrative and survey data, to strengthen

evidence-building to inform program and policy design and implementation. We

strongly endorse a principle of open access to all federal data, with exceptions only for

sensitive data sets as mandated by law, where privacy and/or national security are at

issue. Where these exceptions occur, we propose development of methods that maintain

privacy but still allow their use for public policy making and program development. To

support these goals and improve efficiency in our use of scarce government resources,

we envision a national clearinghouse for integrative Human Analytics and Data

Synthesis (iHADS) to: 

1) Increase the value of government data by facilitating access to and use of

administrative and survey data from multiple sources; 

The Promise of Evidence-Based Policymaking 1733
2) Promote community-wide standards and best practices for data management and

access to reduce administrative costs and create incentive for data sharing across

agencies;

3) Harmonize diverse data definitions and formats to allow analytical syntheses across

information from multiple sources; 

4) Develop cyberinfrastructure and security to enable next generation analytics,

synthesis, and modeling that enables evidence-based policy making; 

5) Provide an environment that enables exemplary research that uses government data to

support evidence-based policymaking to benefit the American public; 

6) Negotiate legal agreements between data managers and credentialed users to lower

regulatory barriers to proper data use while ensuring their safety; 

7) Protect private, confidential, and identifiable information, when needed, through

standards-based credentialing of qualified users and differential privacy safeguards

matched with user credentials and analyze the risks associated with the release of that

information to the credentialed user. 

A national iHADS clearinghouse will make the current dispersed enterprise more

effective and generate significant cost savings for agencies that must manage data,

identify qualified users of data, and provide data to those users while protecting sensitive

information. It could also offer significant cost savings to potential users by reducing

administrative and regulatory barriers, improving data discoverability, and facilitating

linkage across multiple datasets. The latter is especially important for increasing the

return on investment to the American public that has already paid with their tax dollars

for the collection and management of the information in federal databases, and has

earned the opportunity to derive multiple benefits from these data. 

Even sensitive data are powerful engines of innovation and economic value when

appropriately curated. The proposed clearinghouse promotes innovative applications of

data for diverse purposes across the private and public sectors and for scientific research

that can contribute to policy-making. The creation of new cyberinfrastructure to facilitate

the use and synthesis of diverse data sources about human systems--including federal

administrative and survey data--has the potential for important benefits beyond better

policy-making, including facilitating personalized health care, targeting economic

development goals, and accelerating innovation. Because human decisions and actions

are major drivers of many changes in Earth system today, next generation science of

human systems, supported by advanced data synthesis and analytics can offer new

insights into natural systems that can impact human life and well-being. We recommend

that the Commission consider a study by an independent scientific organization as a

guide for possible organization and implementation of such a national clearinghouse.

In the attached document, we list the 37 contributors to these comments and their

affiliations, provide an overview of our response, and respond in detail to the Questions

for Response #1-17, posed by the Commission.

Attachments

1734 The Promise of Evidence-Based Policymaking
CommentsforCEBPNov2016

The Promise of Evidence-Based Policymaking 1735
	




	





	
	





 !"
#$%"&'



()*(
 


"("
$)(	

















$"
"+"
$" 

 		!


"#



$ "
"!

%	

	
&	
'! 
(
"	
")
 


*"" +

"#

)
,$$	
 	!
''
,-





!"&%



!


-"$
!"('&!	
!

' 

 
'


	
 	(".
'

/0)# 	!
-



'
'


"
!,
'!
'


%)

*.
!*('
1%
!

*"(
!(1	
"
 
"
'2


%





"$
!(2%
	
'0!!


	


	


'"


-


"2(
!*3" 	'
3"
	"


0
 
"	

"
*"
"4"'$"'
0)#'
%
!%





4	



$%
!") " 	',!

5"
"*$'! 
%
!'

-


5-"
"	
,

5 


#(
, 

 


	
'
 	!'




%
!

6"-$
"$	
"

0!! 

'



"$
"%$
06 

 

)(
(	
-	
'	
',)







	"&5&''
-
	


4




"("
	77(--
!
 


-
 2'

6 

0)#

5"
8
5"&	
'!&%
!'

*-
6'!"".
'



"
"	

	
%
!'



-


$"
$(06 & 	


 



1736 The Promise of Evidence-Based Policymaking
	
	
 

"(
"
(
*2'&!	
!

' 

'

'!
"'
7'1% 
4 	(".
'



(
1!'
1%
!

"(
%! 
%


'
%
	
0)#

 !	



5"'*
*	
1
 
'	%




'!
-
'

 !&!	

0!'	

0)#

#"("
")(%(!0)#'
%
!%



4	



$
(" 
, 

6
	
,

-
'

 2'6 

0)#

("$
8
$	
"!!
 
!	
	

%




2 	


6-
#(2 

'8
)9 !	





9$$"
" -
	

	! 
 	(


4!
0 
'



"
'(0
-!!	0!&'	

%
!


#"("
( "&" 	'
3"
	"


0
 
"	

6"-(
-
":(! !

"

1

'

5"
%&%
!"
"


 4!

 		!
'!
	-
'	 %


,2

4		

!$$%
*((
','






".
'



-"2
9"'	
,
,%
	
'


 ,%'9

%


2 	



(2

	






	
	
















	






















	

		





	


	




	





 	








	


	








	



	





!



	



	




	

	





















	





		




	

	




	





	

		

"



	

	



	


#


	






	


	

		
	
	











	





	





"
	
	







		








	



		



	


		





$
		



%	
	

	





	

	
	

		





&		






		



'


	

	
(


The Promise of Evidence-Based Policymaking 1737
	
	
 

		







	





	
	

















)
*



	







!


	












		


	










	

	


)
	
%











	









	

	





		

		


	
		





	
	



	

*



	




	



	



	

	

	



+
	





	












	

,




		
	



!

	












	






	


	

*	
	

	



	







	






	

	
	

&&




	
	












	







	



			






	


	








	
-	
	


	












	
	





		




%







		





	



		


			

	






	






	

	



	


	

	







		

	

	


	





	




&

&

	
	

)
	

		




	





			
	









		

		




	

	


	
	
.









	



	



















	
	
	
	






	



	



	

	

	



	
















	







	
"
	








	

		

	

	




			
"
	
	

	


	



		






	



	


	


/	



&









	




0


		




	






!



	







	


!

	


	


)

	
	











	





	

			
	

	


23
#


 	










		


	



	




	




		,

 4

&

		

	

	




		






	

		







	

		

	,

 .


	




	

	




			
		








		,

1738 The Promise of Evidence-Based Policymaking
	
	
 

 +

	

	



!



	
		
	






	

&	



,

 4





	
!
	

		





	

&	







*


,

 5


	


	



		






	



	


	



	,

 4


















		&	




%


		






		





	

	




		
		




	











	 


*




	






		

	







	



	
	
	


	

	


2


		




3



%


		







	
		




		









	

	



	
	
	



		







	




	





	











		


		
+


	


	













	




		

)	
	







	




	
		
		






		


	





	






&

6



		


	

	

	



		




	


*




	







!


	














		















	

	



)





	





	

		
	


	



		


			&&





	


	
&&	









	



&




	















-	


	
	




	

7

	




8	

	

			

!


	





			
	



		
	


	




	
	




98
			








&

*	
	








		

:	

;<






		

	


	






	






	




		











	




	

-

	


:	
	

=		
;&>
	



		


The Promise of Evidence-Based Policymaking 1739
	
	
 !


("(-'
;*

	
		

	
			



	


	




				

	
	

	
	
 !	
	

	

)
 

	








&








	




	

	
		
	





,












	"
	

*

!	







		







	







	



&	







#




	

2#'''
'!3





+
		


4

#$%&	'

2	#''3

%	

	(			
	

2	#'''	'3
	)


 
?



 
2#''!		3




	
	
	


2	#''			3
	*



 
		
2#''		'3




		


-

@
 
	
	
2#''	''3

(	


2	#''A'
'3

	
2	#'''''3

		


(

4

*
 



!	





	
	




2	#''
		'
'3


5


B


$	
'
2	#''	'	
	'	'	AC&	3



.








		





	
		












	
		


		
6
!

	


	










		


	


	
		#
3















	




	



3












	








	
*
!


			




		


	

	

.

=

 


2#''	

	
3

"

	
	









D 


2	

D


D


 
	3
5+
25

+
!3

 *
2 



 

*

	
3

*



	

			














	
	



 


	



	





.

=

 
		













		










		
	





	



















	
	


		

9



8







+	



	











*
			
!


	




2





	3

	
	




	



	



8	






+





.

	




	



		






&








	





	
)	
	



	
		






	














	


	





1740 The Promise of Evidence-Based Policymaking
	
	
 +





	




$
	

	


	






	



		








&









	
	
)
	


	
	

%











	







	


		
D

		


	












	










	









	&&











		


	

	






	

	









	
2

'

	
	
			3



=
	



	








	






	




	




			


	











	



		







		
.




	





	






	


	












	
		


	

	
*
	

	






















	


	




		







		
6
	
		

		




	



			
		


2






	







		



		




		

	






	









	




	



	


		


	

	




2

3








	




	






%	

	
	


	



		
	



	

	








	&

				






	







		


	

	


	
	

"#			$	$
			



		
	
	
		


 

)


	





	
	

		







	

	
		




	




		
	


	








	


	

*
	



%






	
			,(
			








	







	






	








	
		




		
 





	







	

	

		


%







		







@

	

	(	

	





	


	
		


	

	





	







	










	
	
	






	





		

%




		



	
		


	


	
























	


	
		
*

		


























	

	








	


The Promise of Evidence-Based Policymaking 1741
	
	
 -

	
			







	



	



	






		







2

3

	




	







	










2
 

 

5	
	
	


	
			3
.	
(		




	




&		





	











	






	

%





	


	

	
@

	
	

	


	













	




	




	
	

	






		



	
	





"

		





		
	


	







		














		


	

		





((	
	



	





	


	

	












		

		





D









	










		


		
	



		









	



2

/03

	&&





	


		



	



7



	


		
%	






	

	


		

		






		




		


	


	


	


	


	






	
.


	















			
		


23

	





	




		







	


	

 














	

	&&&
		


















	

&

	

	
4







	



	





	


	
		
		



		

		




	










		




%










		



	
	

	

		


6

	








	






	







		





	


	
	




	

		







	




 











		
	








	
		







	



	
		








	


""
7("(**(
"


%#			$	$
		

$	

	

	

			

		
 

)





	











	






	




	

*



	






	


		



1742 The Promise of Evidence-Based Policymaking
	
	
 /



	


	

!
	




	
	




	
	






	




	


	


	
	


-
	

	

		


	






	






	



	


		

	






		
6

	






	














7
	



	




	

6

		


	

	
	













		










	


	

	

%




		




	

		







	
















	







	







	

	


	




		
2	
	
:	

;E3
)
	
		




	


		



	







		






	

	












	
%


		



	




































	
2	
	

:	
	
;F

;3


)
	


	



	

	
&



		





		

		








	







		



	




	




		





	

	




		
 
	
	
		
	
	







	






	




	
		




	






	


		





		
		

7
	


	
	



	


	











	




	


	

	






 




	


	






 





	

	






	









		


	


	






	
=



	



		








	



		

		



	




	




	






	

	








	













)

!

	


	


	


	



	




!

	


	







	



		

&&










	

	




	
2	
	
:	

;F3

=


	


	





	
%

	


		




*



		



	

	
		




	

	




	
	

*






	

	

	






	






		
"

	
		
2
	





		

:	



;3



	


	





!	



!	






		
)
+
	





	











		



	
	


	



		






		







!
	






5

	







		


	
	


	





		




The Promise of Evidence-Based Policymaking 1743
	
	
 0


		
)

!
		









	






	







		

	

	




	

















		



%
	


6
!



	

?
 

	




%







	



	







	

&	

!	

?


2	#''	'
''	'
	
;
!
3
-	






		

	

	




	

!

?
 





!

9
8


		



	

	









	
	
	







		

	

	

 








		






!	

	






	

*

&








	

		



	




	


	


	
&





	


	




			
%
	


	
		


		
*	





		



:	
	
;

;F


		










2


	3















	
	


	

	



	
)
	
%
	

	








	
	
2
!

	








	



3
	



		


7
)
	





	

	




	
2

	



3


	
	

	


2%



	



	

	&&	


:	
	
;

;G3
	&		
	




	
*









	






	
2
 :@
+-6
 H

!

$?=
I@ 




	3
@




		











%








			
'
'!

	



	
"




	


'
'!



	







	











	






	
	
	

		

	
		


2



	



	



		
	
	







	


		




		


&'		

	


	



			


		
		:

)

	










&	

	












%


	






	

		
!
	



	
	
	


	


	

	


	





6









	
%
	
!
	





	

	
		


	
	








	

-

	

	







6



	



		







	
	




!
	



	





	







		
D

	

















	


1744 The Promise of Evidence-Based Policymaking
	
	
 1

.
		








		





	&&



	
	
	







	


	







	



*	










			




%	
	


	




	

		

	




	







	






$


	
	



	



	

		





		


	8	










	
.
	




	






	

%


&




2.43

	











		
	
	
			





	





	

	
)








2



3


	&&



	
&&	














*





	
	

			


		


2

	









	




		
	
2

	











	

3





	



	



	
	




		

*	








	
	
	
	



	








	

	



	












	

	







)
	




	

!




		




	



*





	







	



&



	
	





%		


 



	
		




	


	
	




		


	


	
	

		





	










	
	


	







	
	






	
&
)

	

	


		
	

'

	

	,



	

	

	




		
	
	

!

	




	






	
2
4'J

=
	3

	



		
+

	

	




&	









		





	

	
	

		






)	
	


		



	


	

H

+	
	


2H+3

	

		









	

	








	
	





	

	
 

	
	







	


		




	

	



























		8

	
(	
	








	

	





	

		


!


	
		
	






	

	

	






	

	



The Promise of Evidence-Based Policymaking 1745
	
	
 

('

	

$	
	


 

	
		

			

			 

	




	









	









		







	

	



	



	
		


	




			
		
	


7
	

	


)

	






	
	















		

&

2
		





	
	

	

	



	3
=


	


	



	
		





	



		

	


	


	



	
		


	



	





		













	

















	





	




	


		











)
	
	





	


	

		





	




	
.






	
	

	
		


	






	




	



	




D
	


	


%
	



	














%


	
)
	

	


















%


		&&




%





%


	




D


















	
	
		

	
2
.H

	
	
	



	






	


	

D				3
*






		

	








	






	


		


	



		







		

)	

	






			

		







		


	


$
!

	
2	3		2D*I3


		
2	#''	'	&	
	
	&&&			'&&&

			'

		
'!

3

D





	
		


		











	&

	


D




	


	
)
	



	




	




	

D







		






		





		
	

	









*
!





		

:	



	

	



8	








+



.
)



	


		




	
		




		

















	


	









	






*	


	




	






	

		










&







	
	
)


5
B


+

.
	

	


		









1746 The Promise of Evidence-Based Policymaking
	
	
 

	








	


2




3



	


2	



			3




	


















		
2
3
	

)
	


	



		
	



			










		




	


	










	

	

	




	




	




	




	


	









	






	

	


6
!
	



	





!
	







	








	

	
	




	
D














	







!






&	












	

	

	



	

*

	












	
	








		

	



	


	



	




	




	


+

	

	



	




		
)
	



	


	













		












	

%



		


		
-	



	

	

	


	

	






	
	



	











	



	


!


	
	
	

	



	
*	



	




	





	


		




		




	






)*

	





	+	,		


	
		
 '	
	




 

)













	

		

	
		


23
 


	










	





	





		



	






	


	
)
	

	

		
!
	












	7	

			
%


			

	
		





	




	


	


		



	

		
)



	


	
	


	
	



	




		

	



	












	





	





	

		



	


)



	

	




	




	






	



	

*
	


	



	

















	
	



	










		




	

	


	




	
	
6






	


The Promise of Evidence-Based Policymaking 1747
	
	
 

	

!

	

	

&	






	







*


	

	

	
	
!
	

		




	

	



	

		











D	

	
	








	

	



	
2

3
@


		


2	

			


:	

;>3




&


	





	









	




	
.






	




	



	
	

	
	


	




	
	
	

	

	

		


	

	







)


	



	

	






	






















	






	
		

	




	





	
-	



	


		



	





	





	


		

	
	

	


	
	





		



	








		






	

	

"




		

	


	




	








	
	

5

*
	

 
	
(



D


	
















	




	





	
)
	











	







4		

*		

	

6
 
	
	
2$4*6 3
	


 
6




*

(

*		







		




	





&	










-'	




.*

	+	, 

'


		

	




	
 

	

	










6
 
	

 	


	








	
	
	

	







		




	






	



	











	













(!	


 
		
-





	

	
			








		



		
	




	
		

	



	




 
?

 
&

5

$




*	

*

	




	

	


		



5* *


			
	



	




7












	


	






	



	
2


		



	

3






&




		
*

	



	







	



	






	
	
		





	



	
	


*

	


1748 The Promise of Evidence-Based Policymaking
	
	
  



2


		
3





	
	
			



	

	


(!	








	

D


D


 
	


5

+
!
 

 

*

	

6
-

	





+

J	

2




3
.	



+
2	
3


.

.
 
	
2

	

	
	3





5




 

2


3

*



	



	



		






	







		


	
		

	








	


		

	

	
	

	







=






	
	



	
2





	&

	&&	
3





	



		
	

	







	
		








$








		





!
		
2

3





	



	





	
 

	

		



	
		





	

	





		

&		











	







		













		











	

	




	
	




	


	
2	
:	

;A3

	












		

	

	



		

	
)
	


	







	



	








	
	




	




		








	

	





		
	
=


	





%




	


		




	

	

	
		









	






	
		



		




	

	

	


!



	


	


*	

			







:	

;C

	





	





	
		
	



	





		


	


7
	


	
	

	





		

	


	

	





&
		




	

	






		



.
	




		


	



!







	

			
		


23

	








	





		

/'				

			



	+	,

	

 '	
		

$
	$		



	

		 

"
	
	







		







&

		




	


		



+


	

	

	










		






		

	



The Promise of Evidence-Based Policymaking 1749
	
	
 !



		

)	

		



	


	
	











&

)
*



	







!
	
















		,
	












	




	








	

*




	





	



	
	
	





	




	


	

	

		






%



		







	
		




		







	



	



	
	
	



		





	




	







	









		


		
*	

			




			



%	
	



	





















	


		







	


	


		


	


	


*





)





	





	

		
	


	



		


			&&





	


	
&&	









	



&




	














-	


	
	




	

7

	




8	

	

			

!


	





			
	



		
	


	




	
	




98
			








&



*	
	
	
	


	





		







!

		



	






		







	
















	
D


	



	







	



		

2	
:	

;3

$




	





	

		





	
	

	

	



	








	















	

0'		

	

$				


	 

+







	

	

	





	





!

		
*
	





	


		


	











!




6

	
	


	






	

%



	8	





	
	






	


)
	






















	



		




	


	
'


	

	
	



	



		




		






6

	







			



		


		








	

		









	





%




	
*

	


	
	
D


	
	



1750 The Promise of Evidence-Based Policymaking
	
	
 +

	


	



	






	
	







	8






		





	
)




D


D


 
	


	

	
		







	




	




	


D
	


		
-
%
	






	












	

	
	

 








	
	
	








	8
	

	
	
	





	

	


		
%


	


	

	













		

		


)


	








	
		

		





	
	
*


	





	
	


	



			
		


23

!

	





D











	
		



!





	








		
	





	


	







	




		



	









	


	








		
	

	
	

%




	






	




	















	
	




		










12	

		34

			
	 56

$	

		

	



34

			
	 5

=		

	

	

	



	











"&
		

		
	

	





)



	
%












	

	







		



	
	





	

		

	

	









		







	




		&&







	

	

	












	





	




	

	



5

 



6

5

	
	

.

5

(


.

	










%


		


	

	
	


%





	














	
	


!


		
)	
		












	

	




	



	

	


	




	

"
		









				


		


23

	

	




	
		




%


		


	

	


4


	






			




	
%







	



	

	


	


5 6

5.




	

	




	
*
	















		

	








%


		


	

	


The Promise of Evidence-Based Policymaking 1751
	
	
 -

)
	
	


				






	

	
		




	

	




		


	

	

	











	


	
	



		



		
	













	

	
	


		






	
+


	

	

		




%





	



		

	

		
"
		






	



	











	







		













		

2		


	

	



7		
		






			
	 2


		






			

	 





	
	
	



		&&


		

&&






	


	

)
	

		

	






	


	



 


&








		





		
$

	





	
		


		









6)4
		



=@
B






	

		









		








		



		

	

			
	






	



	


*








	
			
		



23

	





	
	


		
		


	












	




	




	



	



	


'
	
%	

		
-
	









	
	



!

	

	


2


3


	


	



	








	
	



	




	



	





	


D
	

			



	

%	

;G




	

	




	
	





	
	
	



	






%	
	









	

!
	

		




	






	




	





		



		



	






	














	

)

	



	







	












	

	


		









		







	







	




	







	




*


1752 The Promise of Evidence-Based Policymaking
	
	
 /



	



	










	





















	



	



"!

					





		

	34

			
	 5

+


	


















	




		

)









	



	
			
		


2
	













	
	
		

	
		







	



	

		

	

%





	


	

	

	

	




	














		










)

	


	
	
	




	


		

	




	


	
	






!



 


	


	
			

	

	

	K


	


	
	







&	




*	





		

:	

;E

		

&





	


%


		
		


















	



.*+ 
D





	


&&

		

	

		%



















%		












		


	





	

		

*	




		

:	

;C






		
























	
	


	







		



		


	

	




	


	






	



	


	


	





	













	





	



	
	,
	





	



	
	



		




%'

	

		



	 







	

			


		


%	
	









	
	








		



	








		







	











	





6

	

	

	









		





	

	

	





	
		



	


	

	

		

	






	
		
$









	
	

		


	

$


+


6
	'(!
2!
3

5 6&





	






The Promise of Evidence-Based Policymaking 1753
	
	
 0


		
	

	
%

	
	
	




	

		






	
		








$+I


	

	


















	
	

	


+

L	
)	

	&







	

	

&		&















	




	
	



	
		




	









		



	



		

	
)
	
	



	







	

		&









	
	


	
	




)

	

	










	


	

	
	


)
	
	
	







		

	




6

	



J4D
	
	



	





+

	















		








	








	








	

)	
		

	
	






	





		



		







	



			
		


23





	



&'		
	

		
			 



6

	



	
	

	

*



"


		







	









	







	
	
	



		



!
		



	



		


	

	



.





	



	







		



&



K
	

	











		
6



	

	
	

%




	





	






	



	
)	
	
	



	










	



		




		





		







		







		
)
	
%
	
	

!






	







&
	
	


	


	

	






	
		


	
.

	
	





	



	



		


	




	



	
	








	

	
	



















%


	


		

	


*



	



	
			
		



23






	







	
*	











	



		
		
	




!





		
	





	




		




	



1754 The Promise of Evidence-Based Policymaking
	
	
 1




	


	




		



	


	

2	





			

:	
	
;
A

M3
*	

			


	
			

	





		


	



	








""


#('("
'<
""'<
"(-<
"$*"<
"
"$%

('	

$	
		
		




	 



	


	

	

	



	










	
	
	

		


	


			


:	
	
;
A
F
>

C








	


		

%


		









	



	





	

	


	









			




	

	
	








	
*

	

			


			


%	
	
		&	
	



	
	

	







	

	
	
		

	
		



4



	

	






)	






	













	


			
	



			


			


%	
	




	





	
	






 
	


	



	







			

	





%

		



)2				

			

	


	
		 

)




	

	




		










	






	

	
 


	
=		












	






!






	


	




	




7
	K





	





	
	





	





	

	



'
)












	


	


	

:


	
	









%


		


	






	

		







	






				





	

	



(



	

	












				


	


%
	



	





	


	







%	




	


	


	
	
*		


	


%	
	

		






	













)

	















	



	




		






	
	
	




	

	




	


	
		

	

%	
	

	




	

	



The Promise of Evidence-Based Policymaking 1755
	
	
 


	






				
 
	
	



	

	
	



	


	


	


	
	









	

	
)



	

	


	








*

	



	
			


		


23





	











	


	


			

	
	

		
	



-6$	




		

		 

(











*

	



	


	


	


		








	
*	

	




		






	

	

		

	
		

4





	







		

!
	






	

	










	




2	
		

:	

;<3










!
	




	
	

	



	


	
	

	7



)

	




	








	




	

*


	

	



	


	








	










	
	
2	
		

:	

;<3

1756 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 21, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t5v-s5hf

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0187

Comment on FR Doc # 2016-22002

Submitter Information

Name: Terra Fisk

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive.

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

standards in its collection and storage of student data, and recently received a grade of

"D" for its security protections.

The Promise of Evidence-Based Policymaking 1757
In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban.

Yours,

Terra Fisk

Colorado Springs, Colorado

1758 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 21, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t5v-e2m3

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0188

Comment on FR Doc # 2016-22002

Submitter Information

Name: Terence Tryon

General Comment

Just as gun owners support their right to arms and as justification to ultimately protect

their rights from potential government oppression, I believe we need to be protective of

our rights through the protection of our private data. Especially, the data of our students.

I am strongly opposed to the elimination of the ban on student data. This data needs to be

protected, like the rights of our children.

Terence Tryon

The Promise of Evidence-Based Policymaking 1759
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 21, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t5v-weiy

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0189

Comment on FR Doc # 2016-22002

Submitter Information

Name: Gail Janensch

Address:

Bridgeport, CT, 06604

Email: gailj2@optonline.net

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

As a grandmother of 2 elementary-grade grandsons and as a retired Connecticut certified

high school teacher and as a constituent of Sen. Richard Blumenthal, a strong privacy

advocate, I oppose the creation of a centralized, federal clearinghouse of the personally

identifiable information of all public school students, commonly referred to as a student

unit-record system or national database. I will be letting Sen. Blumenthal know of this

communication to the Commission. Under the Higher Education Act of 2008, there is a

ban on such collection. Keep this ban. States already collect enough data on students.

The Federal government for policy purposes can get aggregated data on issues to

facilitate education planning. There is no need for the US government to have details on

students by name, only about groups of students to identify trends and suggest helpful

initiatives useful to students by age in general. Big corporations say they want to help

individual students learn better. And that they could help with individualized instruction

if they had the individual data. No these big corporations want to make MONEY off

1760 The Promise of Evidence-Based Policymaking
students. And if such a database were created and it was breached, young lives who

cannot vote might be harmed because of adult actions. Keep the ban. Protect each child's

privacy while they are so young and developing in their own way day by day.

The Promise of Evidence-Based Policymaking 1761
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 21, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t5w-5cyq

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0190

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive.

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

1762 The Promise of Evidence-Based Policymaking
standards in its collection and storage of student data, and recently received a grade of

"D" for its security protections.

In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban.

Yours,

Carol W. Heinsdorf

The Promise of Evidence-Based Policymaking 1763
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 21, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t5x-k4fy

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0191

Comment on FR Doc # 2016-22002

Submitter Information

Name: David Dobosz

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive.

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

standards in its collection and storage of student data, and recently received a grade of

"D" for its security protections.

1764 The Promise of Evidence-Based Policymaking
In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban.

Yours,

David Dobosz

NYC retired Teacher

The Promise of Evidence-Based Policymaking 1765
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 21, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t5x-6qxo

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0192

Comment on FR Doc # 2016-22002

Submitter Information

Name: Susan Schmitz, MS, LMHC

General Comment

As a citizen of the US, resident of Florida, and Licensed Mental Health Counselor, I AM

AGAINST the efforts to establish a federal "unit record" database that would create an

individual data dossier of college students linked to their employment, with the strong

likelihood that it would be connected to the K-12 longitudinal system.

This is a very dangerous idea. The beauty of the United States of America is the idea that

one can constantly improve and make oneself "better". Employers of adults should

NEVER have access to K-12 records. These years are sacred gardens of cultivation. The

individual's privacy is imperative to this process. If an initiative such as this comes to

pass, greater anxiety and mental health problems will increase in our youth. They already

feel as if they live in fish bowls with social media. This kind of tracking will only make

their self consciousness and evaluation pressure worse.

Sincerely,

Susan Schmitz, MS, LMHC

Wesley Chapel, FL

1766 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 21, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t5y-gc7r

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0193

Comment on FR Doc # 2016-22002

Submitter Information

Name: Adam Weiner

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense,

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive.

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

standards in its collection and storage of student data, and recently received a grade of

"D" for its security protections.

The Promise of Evidence-Based Policymaking 1767
In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban.

Sincerely, Adam Weiner, Brookline, MA

1768 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 21, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t61-7lgw

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0194

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database. 

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive. 

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

standards in its collection and storage of student data, and recently received a grade of

"D" for its security protections. 

The Promise of Evidence-Based Policymaking 1769
In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness. 

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation. 

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban. 

Yours,

Victoria Haddock, concerned parent

1770 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 21, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t61-1fsy

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0195

Comment on FR Doc # 2016-22002

Submitter Information

Name: Erin Rafferty

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive.

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

standards in its collection and storage of student data, and recently received a grade of

"D" for its security protections.

The Promise of Evidence-Based Policymaking 1771
In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban.

Yours,

Erin Rafferty

1772 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 22, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t65-vcmr

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0196

Comment on FR Doc # 2016-22002

Submitter Information

Name: Renee Crofford

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database. 

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive. 

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

standards in its collection and storage of student data, and recently received a grade of

"D" for its security protections. 

In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

highly sensitive elements, including students' immigrant status, disabilities, disciplinary

The Promise of Evidence-Based Policymaking 1773
records, and homelessness. 

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation. 

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban. 

A Worried Parent in Colorado

1774 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 22, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t66-wa6d

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0197

Comment on FR Doc # 2016-22002

Submitter Information

Name: Hannah Lotton

Address:

AK,

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive.

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

The Promise of Evidence-Based Policymaking 1775
standards in its collection and storage of student data, and recently received a grade of

"D" for its security protections.

In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban.

1776 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 22, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t6a-4v9e

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0198

Comment on FR Doc # 2016-22002

Submitter Information

Name: David M

General Comment

I am a parent and a teacher: A parent of a 'gifted' student whom was tested and thus

eligible to be in a gifted program due to her exceptional ability, and a teacher in a low

socioeconomic area with one of the lowest aggregate scores for a district in the entire

state of New York. I have a degree in Information Technology, and have relatives that

own multi-billion dollar corporations.

As a result of the above, I understand the need for information, yet I understand the need

for privacy. It is important to be able to focus resources where they are most needed.

Unfortunately, this is not the only way such data may be used. It would be nice that if we

were to pass a law prohibiting the use/exploitation of such information for political gain,

or for categorizing people to make decisions that may negatively impact a person's

opportunities in life, that such a law would be observed. To say that such laws would not

be broken, would be extremely naive and perhaps even dishonest. To claim that such a

treasure trove of information would never be hacked would be ignorant at best. We know

that the most logical laws are broken by many, and that individuals in all places of our

country have broken laws - even by those who repeatedly represent themselves as honest

and are trusted by a great many.

As a teacher, I can unequivocally state that many of my students need more time to

The Promise of Evidence-Based Policymaking 1777
develop their knowledge, skills, and attitudes as compared to a wide variety of other

students. As a result, they can easily be judged comparatively and put at a disadvantage

since their indicators will be lower than others in the same age group, etc. Just because

the guidance they receive at home is far less than other households in a higher

socioeconomic bracket does not mean they should be further inhibited or targeted. If

targeting is done by numbers alone, how can child abuse be recognized, parental

incarceration as a factor be considered, and developmental stunting due to malnutrition

and the resultant diminished brain development be accurately identified by numbers

alone? Many factors that affect attainment cannot be tested, and if they were, honest

answers would not be received because many students walk around with extreme shame

because they perceive themselves to be lesser than a 'normal' person (the statistical

average characteristics of a person). We could say that we could consider those items in

the statistics and adjust for them, but we also know that such adjustments can not be

accurate without accurate harvesting of all factors and that lack of information can skew

results away form certain populations. Results can be culturally biased. We could state

that we can adjust for that, but we can't even design tests so that they are not culturally

biased: It is impossible since different communities have different language and values

(see Paulo Freire's "Pedagogy of the Oppressed" regarding this subject). There are way

too many 'soft' factors to judge what works, that can not be integrated into an algorithm

to yield a couple of factors to target for greater return on resources. Think what that

means... resources for those in greater number, excluding those that are outliers for

whatever reason. Unfortunately, we tend to dichotomize and simplify matters so that the

public may understand them. If we institute such a system, this can only further

categorize populations for further separation/simplification/injustice. Statistics can be

allies in ignorance when misused. Misuse can not be eliminated as a possibility.

As a father of a gifted child, I do not want her to be tracked so that her opportunities may

be skewed. Her total free will cannot be undervalued. Her personal information should

never be put at risk, NO MATTER WHAT THAT INFORMATION IS! The redrawing

of political boundaries/districts should not be redrawn based upon her address and

likelihood to vote on an issue one way or the other that unbalances the value of her vote.

I am fully aware of the data warehousing and multi-dimensional databases that are

already in place for the harvesting of customer resource management, and the extensive

analysis that occurs. These databases have proven to increase profits and increase return

on investment. Unfortunately, a business model does not work for government simply

because a business operates in a market niche. The government should not choose what

niche it looks to invest in based upon an incomplete and inaccurate view. Such databases

can serve to provide targets for the most profitable areas in which to build private

schools, thereby diminishing resources for many.

Lastly, I am at best disappointed that there is a push for a database such as this, to

analyze data on public schools, yet not require private schools to release their statistics.

Data from in the TRUE aggregate can provide more accurate information. Additionally,

there is one truth: There are lies, damn lies, and the worst of all... statistics!

1778 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 22, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t6c-kuaa

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0199

Comment on FR Doc # 2016-22002

Submitter Information

Name: Amadee Anonymous

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive.

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

standards in its collection and storage of student data, and recently received a grade of

The Promise of Evidence-Based Policymaking 1779
"D" for its security protections.

In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban.

Yours,

Mrs. Amadee Safar, South Dakota

1780 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 22, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t6d-ib2r

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0200

Comment on FR Doc # 2016-22002

Submitter Information

Name: Meg Bakich

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive.

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

standards in its collection and storage of student data, and recently received a grade of

"D" for its security protections.

The Promise of Evidence-Based Policymaking 1781
In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban.

Yours,

Meg Bakich

1782 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 22, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t6e-2g1n

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0201

Comment on FR Doc # 2016-22002

Submitter Information

Name: LIsa Vavrik

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all students, commonly

referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008, the reasons against creating it

have only become more compelling.

In the past few years, much highly personal data held by federal agencies has been

hacked, including the release of the records of the Office of Personnel Management

involving more than 22 million individuals, not only federal employees and contractors

but also their families and friends.

The US Department of Education has been found to have especially weak security

The Promise of Evidence-Based Policymaking 1783
standards in its collection and storage of student data, and received a grade of D for its

security protections.

In addition, preK-12 student data currently collected by state departments of education

that would potentially be shared with the federal government include upwards of 700

highly sensitive personal data elements, including students' immigrant status, disabilities,

disciplinary records, and homelessness data.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from PreK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban in

the report you provide to Congress.

Yours,

Lisa Vavrik

1784 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 22, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t6e-xpan

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0202

Comment on FR Doc # 2016-22002

Submitter Information

Name: Timothy Kenyon

General Comment

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive.

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

standards in its collection and storage of student data, and recently received a grade of

"D" for its security protections.

In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

The Promise of Evidence-Based Policymaking 1785
highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban.

Sincerely,

Timothy Kenyon

NYS Teacher and parent

1786 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 22, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t6h-hovh

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0203

Comment on FR Doc # 2016-22002

Submitter Information

Name: David Rosenberg

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive.

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

standards in its collection and storage of student data, and recently received a grade of

"D" for its security protections.

The Promise of Evidence-Based Policymaking 1787
In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban.

Yours,

David Rosenberg

1788 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 22, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t6i-qcx9

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0204

Comment on FR Doc # 2016-22002

Submitter Information

Name: Kristy Hazel

Address:

Baltic, 57003

Email: kristyhazel@live.com

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database. Student

privacy is at risk. The risk that such a federal database would pose to student privacy is

immense; including the very real possibility of breach, malicious attack, or the use of

this information for purposes not initially intended. Ever since a federal student unitrecord system was banned by the Higher Education Act in 2008 the reasons against

creating it have only become more persuasive.

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

The Promise of Evidence-Based Policymaking 1789
standards in its collection and storage of student data, and recently received a grade of

"D" for its security protections.

In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban.

Yours,

Kristy Hazel

South Dakota 

1790 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 22, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t6k-48lm

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0205

Comment on FR Doc # 2016-22002

Submitter Information

Name: Nicole Barcliff

Address:

Washington DC, DC, 20006

Email: nbarcliff@lisc.org

General Comment

Date: November 22, 2016

Attn: Shelly Martinez

Executive Director of the Commission on Evidence-Based Policymaking

Cc: Nick Hart, Policy and Research Director for the Commission on Evidence-Based

Policymaking

nicholas.r.hart@census.gov

Re: Docket ID USBC-2016-0003

Commission on Evidence-Based Policymaking Comments

Submitted by: The Local Initiatives Support Corporation (LISC)

Thank you for the opportunity to offer comments on the future activities and scope of

work to be undertaken by the Commission on Evidence-Based Policymaking. The Local

Initiatives Support Corporation (LISC) views this as an opportunity to positively

influence how data is used to improve outcomes for communities with persistent

economic and social challenges. We hope that LISC's comments (see attached) - which

are informed by our experience in funding and facilitating the collaborative efforts of

The Promise of Evidence-Based Policymaking 1791
local constituencies and the public and private sectors - will be useful to the

Commission.

Please contact us if you have any questions or if LISC can serve as a resource to the

Commission.

ABOUT LISC

Established in 1979, the Local Initiatives Support Corporation (LISC) is a national

nonprofit and Community Development Financial Institution (CDFI) dedicated to

helping community residents transform distressed neighborhoods into healthy places of

choice and opportunity - good places to work, do business and raise children. LISC

mobilizes corporate, government and philanthropic support to provide local community

development organizations with loans, grants and equity investments; local, statewide

and national policy support; and technical and management assistance. LISC has local

offices in 31 cities and partners with more than 70 organizations serving rural

communities throughout the country. We focus our activities across strategic community

revitalization goals, including: expanding investment in housing and other real estate;

increasing family financial stability; stimulating economic development; improving

access to quality education; and supporting healthy environments and lifestyles.

For more than three decades, LISC has developed programs and raised investment

capital to help local groups revive their neighborhoods. The following feedback and

recommendations are rooted in our organization's experience in collaborating with

community groups to use research and data to identify the priorities and challenges of

their neighborhoods, and deliver the most appropriate support to meet local needs. 

FEEDBACK & RECOMMENDATIONS

LISC is pleased to offer feedback and recommendations to the following questions:

Overarching Question(s): 1. 

Data Infrastructure and Access Questions: 3., 4., 5., 11., 14.

Data Use in Program Design, Management, Research, Evaluation, and Analysis

Questions: 15., 16., 19.

Attachments

LISC EBPC Public Comments 11-22

1792 The Promise of Evidence-Based Policymaking
Page 1 of 5

Date: November 22, 2016

Attn: Shelly Martinez

Executive Director of the Commission on Evidence-Based Policymaking

Cc: Nick Hart, Policy and Research Director for the Commission on Evidence-Based 

Policymaking

nicholas.r.hart@census.gov

Re: Docket ID USBC-2016-0003

Commission on Evidence-Based Policymaking Comments

Submitted by: The Local Initiatives Support Corporation (LISC)

__________________________________________________________________________________

Thank you for the opportunity to offer comments on the future activities and scope of work to be 

undertaken by the Commission on Evidence-Based Policymaking. The Local Initiatives Support 

Corporation (LISC) views this as an opportunity to positively influence how data is used to improve 

outcomes for communities with persistent economic and social challenges. We hope that LISC’s 

comments – which are informed by our experience in funding and facilitating the collaborative efforts 

of local constituencies and the public and private sectors – will be useful to the Commission. 

ABOUT LISC

Established in 1979, the Local Initiatives Support Corporation (LISC) is a national nonprofit and

Community Development Financial Institution (CDFI) dedicated to helping community residents 

transform distressed neighborhoods into healthy places of choice and opportunity – good places to 

work, do business and raise children. LISC mobilizes corporate, government and philanthropic 

support to provide local community development organizations with loans, grants and equity 

investments; local, statewide and national policy support; and technical and management assistance. 

LISC has local offices in 31 cities and partners with more than 70 organizations serving rural 

communities throughout the country. We focus our activities across strategic community revitalization 

goals, including: expanding investment in housing and other real estate; increasing family financial 

stability; stimulating economic development; improving access to quality education; and supporting 

healthy environments and lifestyles.

For more than three decades, LISC has developed programs and raised investment capital to help local 

groups revive their neighborhoods. The following feedback and recommendations are rooted in our 

organization’s experience in collaborating with community groups to use research and data to identify

the priorities and challenges of their neighborhoods, and deliver the most appropriate support to meet 

local needs.

FEEDBACK & RECOMMENDATIONS

LISC is pleased to offer feedback and recommendations to the following questions:

Overarching Questions

1. Are there successful frameworks, policies, practices, and methods to overcome challenges 

related to evidence-building from state, local, and/or international governments the 

Commission should consider when developing findings and recommendations regarding 

Federal evidence-based policymaking? If so, please describe.

The Promise of Evidence-Based Policymaking 1793
Page 2 of 6

CHALLENGES

As federal agencies adopt evidence-based policies to become better stewards of limited 

taxpayer resources and improve outcomes for communities, we encourage them to be mindful 

of the following challenges related to practical implementation in local neighborhoods:

Local organizations lack the capacity and financial support to develop the ability to 

effectively implement data-driven programming.

Many organizations serving high need neighborhoods were established to address specific 

challenges, and did not evolve from structured research. As such, the application of evidencebased principles to policies that govern programs targeting low-income communities is still 

fraught with challenges. In our experience, most local organizations have limited familiarity 

with collecting data, lack data system infrastructures, and do not have adequate funding to 

address these deficiencies – making it extremely challenging to implement and evaluate 

rigorous programs. If local organizations are to be successful in implementing evidence-based 

programs and practices, they need financial support and technical assistance to build their 

organizational capacity. 

Investing in research summaries and the production of additional materials and trainings that 

aid “data translation” would help local policymakers and other leaders, many of whom might

not have the resources for a significant, ongoing academic research partnership, to pursue 

evidence-based practices. For example, the CrimeSolutions.gov database of evidence 

summarizes important criminal justice research in a relatively accessible way. It could be 

improved through modifications to capture important findings from research studies beyond 

the top-level classification of whether particular strategies were successful. 

Local organizations that are developing capacity take longer to show evidence of 

effectiveness toward achieving outcomes in communities. 

We know that very few organizations serving high need communities have the technical 

expertise, capacity and funding to decide which data is important to collect and develop 

evaluations that assess effectiveness toward achieving desired outcomes. Many service 

providers are just beginning to shift from collecting process data to collecting data that can be 

used to show program effectiveness and progress for participants. Accountability measures,

particularly those that might be used to limit or eliminate programmatic funding based upon 

progress toward achieving outcomes, need to take the limitations of local organizations 

making the shift to evidence-based programs into consideration. Specifically, organizations 

should be given a reasonable timeline to integrate data-driven practices and, if they do not 

meet the benchmarks associated with desired programmatic outcomes, they should be guided 

to adopt quality improvement measures to improve their results.

Policies that encourage the adoption and implementation of evidence-based programs 

often do not include sufficient support for innovative promising approaches. Promising 

programs and innovations should be incubated and helped along the road to becoming 

evidence-based. 

It is encouraging to see policymakers move in the direction of ensuring that publicly 

supported programs achieve their intended outcomes and improve the quality of life in 

struggling communities. Program implementers face significant pressures to demonstrate that 

the approaches adopted improve participant outcomes, are cost-effective and have a return on 

investment. The pressure to show progress toward achieving desired programmatic outcomes

coupled with unrealistic timeframes to achieve significant outcomes may inadvertently lead to 

abandoning programs that work, but do not yet have a robust evidence base. As previously 

mentioned, local program providers often lack the funding and capacity to demonstrate 

evidence of effectiveness. In addition to supporting replication of proven programs, it is 

important that federal efforts promote and incubate innovation, and allow sufficient time for

demonstration of outcomes.

1794 The Promise of Evidence-Based Policymaking
Page 3 of 6

Evidence-based policies and community-oriented / community-driven decision-making

are often mistakenly perceived as mutually exclusive.

We have observed that some policymakers view evidence-based decision-making and 

community-oriented or community-driven decision-making as mutually exclusive. There is a

flawed assumption that evidence-based decisions happen when policymakers gather with 

experts behind closed doors to parse significant amounts of research and emerge with policy 

decisions; in community-oriented processes, residents are invited to participate in setting 

priorities based wholly on their own perceptions and experience, with no consideration of 

evidence. We see this as a false choice that leads to important decisions being made in a topdown fashion, with smaller decisions left to superficially indulge communities.

SUCCESSFUL POLICIES 

Policies that support collaboration between the public and private sectors hold great promise for 

addressing community need. Below are several successful programs that are instructive to efforts that 

seek to promote data driven investments, support innovation, and employ cross-sector partnerships. 

- Corporation for National and Community Service (CNCS) Social Innovation Fund

(SIF)

Authorized by the Edward M. Kennedy Serve American Act of 2009, SIF catalyzes the 

use of data-driven approaches with evidence of effectiveness in local communities. 

Through an intermediary structure, the program mobilizes public and private sector 

resources to address local and national challenges in three priority areas: economic 

opportunity, healthy futures, and youth development. Since 2010, the SIF has invested 

$800 million in compelling community solutions, allocating more than $295 million in 

federal grants and leveraging more than $582 million in non-federal match commitments.

o The SIF Classic Program provides awards to eligible grant making institutions 

or partnerships to select, fund, support, and evaluate community-based 

nonprofits seeking to grow innovative, evidence-based solutions in the areas of 

economic opportunity, healthy futures and/or youth development. SIF Classic 

resources are enabling LISC to grow, scale, replicate, and enhance the Financial 

Opportunity Center (FOC) model. FOCs provide low-income individuals with 

integrated services across three critical areas: employment services and career 

planning, financial coaching, and income supports. SIF resources enabled 

evaluation of the model. The research revealed that clients who accessed 

combined services were 50% more likely to land a well-paying job than people 

receiving employment services only, and that long term employment almost 

doubles when financial coaching reinforces the work of employment counseling. 

o The SIF Pay for Success (PFS) Program awards grants to eligible nonprofit 

organizations to leverage upfront philanthropic and private dollars to fund social 

service programs that demonstrate success through measurable outcomes. LISC 

was awarded PFS resources to help social service providers design effective 

programs, raise private capital and produce the metrics needed to demonstrate

positive outcomes. Three to four high-quality projects in need of transaction 

structuring services for youth development, economic opportunity and healthy 

futures will be supported by the award. This program may to revolutionize the 

way that social services are funded and delivered.

One challenge facing innovations funded by the SIF is identifying a pathway to 

integrating, scaling and expanding approaches with evidence of effectiveness in a manner 

that aligns with existing federal efforts to address national and local challenges. In fact, 

despite the SIF’s widespread success, many federal agencies are unaware that SIFsupported innovations leverage significant private capital, and have the potential to help 

improve existing programs, yielding better outcomes for vulnerable communities. If 

seriously considered by federal agencies, SIF innovations have the potential to help 

agencies tackle pressing social and economic issues. We believe that the SIF program 

The Promise of Evidence-Based Policymaking 1795
Page 4 of 6

evaluations, data, and experience of partner organizations are of particular value to 

federal agencies tasked with providing effective services in the areas of economic 

opportunity, healthy futures, and youth development. As the Commission builds out its 

priorities and scope of work, our Coalition suggests that you encourage federal agencies 

to adopt relevant data-driven programs/strategies with evidence of effectiveness currently 

supported by the Social Innovation Fund at the Corporation for National and Community 

Service (CNCS).

- Department of Justice (DOJ) Byrne Criminal Justice Innovation Program (BCJI)

We encourage the Commission to recognize the importance of investing in action 

research models tied to evidence-based decision-making to address problems that are 

known to be shaped significantly by context. In the case of community safety programs, 

the social cohesion of a high crime community is known to have complex compounding 

factors – like crime drivers that are inextricably linked to poverty and disinvestment. The

Byrne Criminal Justice Innovation (BCJI) Program recognizes that context is important, 

and supports the use of data to address crime contextually. 

The Byrne Criminal Justice Innovation (BCJI) Program brings together diverse partners 

including local law enforcement, researchers and residents to analyze crime drivers and 

pursue strategies that reduce crime, spur revitalization and build community-police 

collaboration. Administered by the Department of Justice Bureau of Justice Assistance, 

BCJI is part of the “Smart Suite” of programs that support partnerships between 

researchers and practitioners to develop data-driven and evidence-informed strategies.

The Program creates a unique structure that supports the development, implementation, 

and evaluation of promising innovative approaches to crime reduction in areas 

historically plagued by persistent crime. 

BCJI is an important data driven program not only because of its support of innovative 

practices, but also because of the distinctive role of technical assistance (TA). The

program requires cross-sector partnerships to use data and research to adopt and 

implement crime reduction strategies. Because the cross-sector partnerships include local 

community members who may have no experience with the development of researchdriven programming, and researchers who have may have no experience in authentic 

community engagement, the technical assistance providers are essential to facilitating 

dialogue that leads to the adoption of a data-driven crime reduction implementation 

strategy. The technical assistance provider is essentially the intermediary glue that binds 

research and residents.

Data Infrastructure and Access

3. Based on identified best practices and existing examples, how should existing government 

data infrastructure be modified to best facilitate use of and access to administrative and 

survey data?

In facilitating the use of and access to administrative and survey data, we believe that existing 

government data infrastructure should consider the implications for individuals who are very 

transient or are victims of identity theft. Assumptions about using information like address or 

birth date to link individuals’ records across datasets may be less applicable to some 

populations.

4. What data-sharing infrastructure should be used to facilitate data merging, linking, and 

access for research, evaluation, and analysis purposes?

The Data-Driven Justice Initiative (DDJ) and similar efforts that seek to support local 

governments in sharing data for broader public use may prove instructive regarding the

facilitation of data merging. DDJ was launched with a bipartisan coalition of 67 city, county, 

and state governments committed to using data-driven strategies to divert low-level offenders 

with mental illness out of the criminal justice system and change approaches to pre-trial 

incarceration, so that low-risk offenders no longer stay in jail simply because they cannot 

1796 The Promise of Evidence-Based Policymaking
Page 5 of 6

afford a bond. The initiative brings data together from across criminal justice and health 

systems to identify the individuals with the highest number of contacts with police, 

ambulance, emergency departments, and other services, and link them to health, behavioral 

health, and social services in the community, with a goal of reducing overreliance on 

emergency healthcare and encounters with the criminal justice system. 

5. What challenges currently exist in linking state and local data to federal data? Are there 

successful instances where these challenges have been addressed?

Many of the challenges that exist in linking state and local data to federal data also exist in 

trying to link similar data across federal agencies. Challenges exist in the following areas: 

• In some cases, data collection efforts for programs serving the same individuals 

either overlap or are duplicative. Local programs may have to collect the same data

points on the same participants but enter that data into different platforms because of 

disjointed data collection requirements across agencies and programs. 

• Similar terms are defined differently because different levels of government and 

agencies have different objectives for the use of data.

11. How might integration of administrative and survey data in a clearinghouse affect the risk 

of unintentional or unauthorized access or release of personally-identifiable information, 

confidential business information, or other identifiable records? How can identifiable 

information be best protected to ensure the privacy and confidentiality of individual or 

business data in a clearinghouse?

One strategy to protect identifiable information to ensure the privacy of individual or business 

data is to amend the Common Rule governing the mandate and invocation of Institutional 

Review Boards (IRBs) and to be more diligent about privacy impact in the context of use of 

clearinghouse (or all administrative/survey) data.

14. What incentives may best facilitate interagency sharing of information to improve 

programmatic effectiveness and enhance data accuracy and comprehensiveness?

We are pleased that the Commission has an interest in improving interagency sharing of best 

practices. The Commission should consider the sharing of information and the use of 

information to improve programmatic effectiveness as two related, but distinct objectives. For 

example, agencies may currently be utilizing platforms for sharing research and data of 

innovations, but the shared information may only be targeting research divisions of agencies, 

which often exist in silos, separate from staff responsible for programmatic implementation.

As such, interagency sharing improvements should not only improve the platform used for 

information sharing, but also influence the way that research divisions of agencies 

communicate with an influence programmatic divisions. 

An incentive program for the adoption of shared information in an effort to improve programs 

may be useful in encouraging agencies to pilot or integrate innovations. Incentives could 

include additional funding or special national recognition from the White House. 

Data Use in Program Design, Management, Research, Evaluation, and Analysis

15. What barriers currently exist for using survey and administrative data to support program 

management and/or evaluation activities?

Often, survey and administrative data is not used to support program management or 

evaluation activities at the local level because providers have funding limitations (most 

programs do not have enough operating capital to meet the needs of their communities, much 

less to purchase data systems), and lack the capacity to make decisions about research.

As a national intermediary that works in very specific cities & neighborhoods, we perceive

several specific challenges. 

- Survey data is generally not available at the small neighborhood-level geographies where 

we actually implement our work - results are only significant at city/state/metro/national 

level depending on the dataset. 

The Promise of Evidence-Based Policymaking 1797
Page 6 of 6

- Administrative data, is more likely to be available for small geographies but is often not 

consistent or reconcilable across geographies. For example, it may be feasible to use 

administrative data for a study that is taking place only in Chicago, but it becomes harder 

to find & access parallel data for multiple cities at once.

16. How can data, statistics, results of research, and findings from evaluation, be best used to 

improve policies and programs?

Data, statistics, the results of research and evaluation findings should be used to improve 

policies and programs, but programs should also be allowed to innovate on the ground in 

order to adapt to and address environmental circumstances. The heavy emphasis on 

experimental and quasi-experimental designs make it much more difficult to use evidence to 

inform programs, because of the lengthy time horizons for those studies and because they

require a rigid “lock-in” of program design/model in order to draw conclusions about design 

features. Prioritizing experimental findings actually decreases the chance that evaluation will 

be useful on the ground for program design – or that utility will come at a much slower pace. 

19. To what extent should evaluations specifically with either experimental (sometimes referred 

to as “randomized control trials”) or quasi-experimental designs be institutionalized in 

programs? What specific examples demonstrate where such institutionalization has been 

successful and what best practices exist for doing so?

From LISC’s perspective, institutionalizing evaluations within programs would likely prove

costly and siphon program time and energy from actual program delivery. Even if data 

collection is always in place for the treatment group, programs will have to incur the cost of 

finding a reasonable comparison group and implementing data collection. For example, in the 

social impact bond and Pay For Success context, results of the embedded experimental 

evaluation provide the basis for valuation of returns to the capital investor. This structuring is 

accompanied by large transaction costs. 

Using experimental or quasi-experimental designs as the standard for evidence based policy 

rules out interventions that work at a unit of analysis that are too big for reasonable control 

groups to be assembled given standard statistical assumptions. If the impact is projected for 

individuals, then experimental designs make sense. If the impact is happening at the 

neighborhood level, which is the case of a lot of LISC’s investments, then the logic of 

experimental or quasi-experimental design is much less applicable.

CLOSING

Thank you for the opportunity to weigh in on these important issues. Please contact Nicole Barcliff, 

Sr. Policy Officer at nbarcliff@lisc.org or (202) 739-9296 if you have any questions or if LISC can 

serve as a resource to the Commission. 

1798 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 22, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t6k-8pae

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0206

Comment on FR Doc # 2016-22002

Submitter Information

Name: Stefanie Fuhr

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

Please leave my children be. There is no positive reason that you would need their

information.

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database. 

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive. 

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

standards in its collection and storage of student data, and recently received a grade of

The Promise of Evidence-Based Policymaking 1799
"D" for its security protections. 

In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness. 

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation. 

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban. 

Yours,

Stefanie Fuhr, Colorado

1800 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 22, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t6l-rw8h

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0207

Comment on FR Doc # 2016-22002

Submitter Information

Name: Edgar Figueroa, MD, MPH

Address:

New York, NY, 10034

Email: edfig99@yahoo.com

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all students, commonly

referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008, the reasons against creating it

have only become more compelling.

In the past few years, much highly personal data held by federal agencies has been

hacked, including the release of the records of the Office of Personnel Management

involving more than 22 million individuals, not only federal employees and contractors

The Promise of Evidence-Based Policymaking 1801
but also their families and friends.

The US Department of Education has been found to have especially weak security

standards in its collection and storage of student data, and received a grade of D for its

security protections.

In addition, preK-12 student data currently collected by state departments of education

that would potentially be shared with the federal government include upwards of 700

highly sensitive personal data elements, including students' immigrant status, disabilities,

disciplinary records, and homelessness data.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from PreK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

As a physician we don't have national health records on patients - it's been frequently

dismissed as intrusive and the harms of privacy breaches have been identified as very

real. I see no difference with student records.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban in

the report you provide to Congress.

Yours,

1802 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 22, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t6r-ieth

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0208

Comment on FR Doc # 2016-22002

Submitter Information

Name: Rosalie Friend

General Comment

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all students, commonly

referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008, the reasons against creating it

have only become more compelling.

In the past few years, much highly personal data held by federal agencies has been

hacked, including the release of the records of the Office of Personnel Management

involving more than 22 million individuals, not only federal employees and contractors

but also their families and friends.

The US Department of Education has been found to have especially weak security

standards in its collection and storage of student data, and received a grade of D for its

security protections.

The Promise of Evidence-Based Policymaking 1803
In addition, preK-12 student data currently collected by state departments of education

that would potentially be shared with the federal government include upwards of 700

highly sensitive personal data elements, including students' immigrant status, disabilities,

disciplinary records, and homelessness data.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from PreK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban in

the report you provide to Congress.

1804 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 23, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t7a-zidz

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0209

Comment on FR Doc # 2016-22002

Submitter Information

Name: Joshua Seff

Address:

McKinney, TX, 75070-5815

Email: mv9508@sbcglobal.net

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all students, commonly

referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008, the reasons against creating it

have only become more compelling.

In the past few years, much highly personal data held by federal agencies has been

hacked, including the release of the records of the Office of Personnel Management

involving more than 22 million individuals, not only federal employees and contractors

The Promise of Evidence-Based Policymaking 1805
but also their families and friends.

The US Department of Education has been found to have especially weak security

standards in its collection and storage of student data, and received a grade of D for its

security protections.

In addition, preK-12 student data currently collected by state departments of education

that would potentially be shared with the federal government include upwards of 700

highly sensitive personal data elements, including students' immigrant status, disabilities,

disciplinary records, and homelessness data.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from PreK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban in

the report you provide to Congress.

1806 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 23, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t7a-a3mq

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0210

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

I understand your goal is to identify ways in which the collection of unit record data can

be expanded while overcoming challenges to its potential for privacy violation. The

purpose for my comments is to persuade you that the collection of data for any use

outside of its original explicit intended transactional use should be abandoned entirely.

To that end, please consider the attached statements of both fact and opinion. Thank you.

Attachments

Comments online to CEB re Unit Record Data 11.2016

The Promise of Evidence-Based Policymaking 1807
https://www.regulations.gov/comment?D=USBC-2016-0003-0001

I understand your goal is to identify ways in which the collection of unit record data can be expanded 

while overcoming challenges to its potential for privacy violation. The purpose for my comments is to 

persuade you that the collection of data for any use outside of its original explicit intended transactional 

use should be abandoned entirely. To that end, please consider the following statements of both fact 

and opinion: 

Outside of family, there are perhaps two relationships compare in their intimacy, and trust: those we 

have with our doctors, and with our children's teachers. There is an implicit acceptance of the trade we 

make with each of these people. We share information with our doctors that we may not even tell our 

spouse, in exchange for his or her promise to use it for our benefit. Teachers, who by definition are with 

our children for a third of their waking hours, get to know them very well indeed. They know when 

they’re tired, happy, distracted, upset, or bored. They also know where they are academically strong, 

and challenged. We trust that the teacher will use their knowledge and understanding of our kids to 

help them be their best, both socially and academically. When intervention is warranted, we trust the 

teacher’s guidance and more often than not, support his/her recommendations in the best interest of 

our children. 

That is the implicit agreement. We agree to divulge, and be vulnerable. They agree to use what they 

know about us to help us, and only us. There never was an agreement - implicit, explicit, or otherwise - 

to turn these private relationships into Petri dishes for state and federal government. And yet it was 

done, and continues to occur. My concerns and recommendation are as follows: 

1. Access to data doesn't give state or federal government any natural, implicit or explicit rights to 

use it in any way outside the purview of the transaction that originally generated it 

2. Government has collected data about its citizens, but not in the abstract and certainly not by 

disclosing its eventual use for analytics, or targeting, by the government. 

3. The data government has collected was a by-product of transactions with citizens over time, 

which are either required by law or necessary in order for citizens to utilize privileges awarded 

exclusively by government – i.e., tax returns, voting, drivers licenses, passports. In every 

instance, the citizen’s only option was to supply the requisite information. 

4. If compliance with the information requested in all these transactions was 100% voluntary, 

participation would unquestionably fall short of 100%. Therefore, the government cannot 

rationally argue, after the fact, that the data it acquired was relinquished voluntarily in the first 

place. If given a choice, a substantial percent of the population would limit the data they give to 

the government. 

5. If given the choice, a substantial percent of the population would limit the data they give to the 

government. So, if the information that has already been provided was done so only because it 

was required in order to complete a transaction, how can the government now expand its 

intended use above and beyond its original permission? Furthermore, how can it justify keeping 

and using it for multiple and as-yet undefined applications, while at the same time removing the 

1808 The Promise of Evidence-Based Policymaking
law requiring the owner of the data to provide written permission for each explicit use of the 

data? (FERPA Executive Action 2011). 

6. Poor awareness, or lack of understanding by the public of the implications of this Commission’s 

goal, is no excuse for ignoring common sense. And common sense suggests that if 100% of the 

population understood the initiative, and it was given ample public debate, most of the 

population would disapprove. 

7. The continued expansion of citizen databases has far-reaching implications for every single 

person in America. Longitudinal studies of students at the federal level are reminiscent of a 

post-WWII Germany in which students were channeled into aptitude-indicated careers whether 

they liked it or not. Overlays of public and private data including parents’ education, religion, 

voting history, medical history, employment history, ethnicity, beg the question to government 

to justify this massive theft of their privacy. 

8. A central database with no accountability to the owners of the data is an invitation to corruption. 

For example, let’s say an administration decides to overlay carbon footprint onto each 

household in the country. Individuals could then be targeted based on suppositions like how 

many cars or appliances they own, square footage of their homes, gas consumption. Or perhaps 

an individual of Mexican descent has voted outside the majority of his ethnicity. A federal 

government could siphon off his information to either political party for direct targeting. Last, it 

is determined that a student whose standardized test scores fall within two arbitrary numerical 

data points should be guided toward an Engineering college rather than a liberal arts school as 

he prefers. Does the student who agrees to study engineering receive the student loan, as the 

one who opts for liberal arts is penalized? The list of unfair, discriminatory, racist, political, 

financial applications of the data is infinite. This leads to my next point. 

9. Government assurances of protections have become largely meaningless in the wake of 

revelations about data abuse at the highest levels. The IRS broke privacy laws when it admitted 

targeting 501C3 applicants based on their political affiliation. Aside from the resignation of its 

public face, no one was held accountable. The privacy disclosures which were written into law 

and considered sacrosanct were eviscerated with the stroke of a pen (FERPA, Execution Action, 

2011). External hacks by other nations into government agency databases, and high-level 

government employees’ emails. All of these examples and more have revealed what is logical 

and true – that data is never private; that laws are ignored and broken without retribution; that 

a central repository of data as vast and deep as you are considering will become an instant 

target for a multitude of unknown assailants with greater data theft capabilities than you can 

know of today, and unimaginable nefarious intentions. If this risk is even infinitesimal, which it 

is not, then why would the government do it? 

Government’s role is to protect its citizenry, not analyze it, target it, or try to affect its behavior based 

on what it knows about individuals. The continued pursuit of this entire project is essentially a theft, 

and a massive flip of a finger to whatever remains of public trust in its government. I urge you to 

recommend that the entire database objective be abandoned entirely. Only in that end will you have 

done your absolute best for the American people. 

The Promise of Evidence-Based Policymaking 1809
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 23, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t7e-l2do

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0211

Comment on FR Doc # 2016-22002

Submitter Information

Name: Joseph P. Bottini

General Comment

Stop! Anything that controls the life and personal information of a child is ludicrous in a

Democratic Republic. This makes a cloud gathering of students data dangerous to the

well being of our culture.

No Child Left Behind, Race To The Top and Common Core are among the disastrous

programs that have been fostered upon local school districts by the government.

A "one size fits all education fits no one." 

National blanket requirements of instruction and learning models are educationally

unsound and fundamentally detrimental to children. 

The 10th amendment to the United States Constitution dictates a federal/national

education department unconstitutional/illegal.

I have arrived at the above opinions based on 35 years a public school teacher and 8

years a local Board of Education member.

1810 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 23, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t7g-30wc

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0212

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive.

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

standards in its collection and storage of student data, and recently received a grade of

The Promise of Evidence-Based Policymaking 1811
"D" for its security protections.

In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban.

Yours,

D. Martinez, NYS, parent

1812 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 24, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t7s-2q4v

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0213

Comment on FR Doc # 2016-22002

Submitter Information

Name: Tiffany Rios

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all students, commonly

referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008, the reasons against creating it

have only become more compelling.

Our children need to be protected and not treated as data in a database. They are entitled

to privacy too.

Thank You for your time.

The Promise of Evidence-Based Policymaking 1813
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 24, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t7w-oyth

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0214

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

Get out of the school system period. We don't like big brother looking over our shoulders

and certainly NOT over the shoulders of our children. Common Core is an education

buster not a boon! Everyone is NOT equal or the same when it comes to education.

1814 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 25, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t8r-d00w

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0215

Comment on FR Doc # 2016-22002

Submitter Information

Name: Ilana Spiegel

Address:

Englewood, CO, 80111

Email: ilanadspiegel@gmail.com

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all students, commonly

referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008, the reasons against creating it

have only become more compelling.

In the past few years, much highly personal data held by federal agencies has been

hacked, including the release of the records of the Office of Personnel Management

involving more than 22 million individuals, not only federal employees and contractors

The Promise of Evidence-Based Policymaking 1815
but also their families and friends.

The US Department of Education has been found to have especially weak security

standards in its collection and storage of student data, and received a grade of D for its

security protections.

In addition, preK-12 student data currently collected by state departments of education

that would potentially be shared with the federal government include upwards of 700

highly sensitive personal data elements, including students' immigrant status, disabilities,

disciplinary records, and homelessness data.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from PreK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban in

the report you provide to Congress.

Yours, Ilana Spiegel, Grassroots Education Activist

1816 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 26, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t94-zyxf

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0216

Comment on FR Doc # 2016-22002

Submitter Information

Name: Lisa Cohen

General Comment

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive.

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

standards in its collection and storage of student data, and recently received a grade of

"D" for its security protections.

In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

The Promise of Evidence-Based Policymaking 1817
highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban.

1818 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 27, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8t9y-pnwu

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0217

Comment on FR Doc # 2016-22002

Submitter Information

Name: Stephanie Carver

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive.

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

standards in its collection and storage of student data, and recently received a grade of

"D" for its security protections.

The Promise of Evidence-Based Policymaking 1819
In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban.

Sincerely,

Stephanie Carver

Minnesota

1820 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 28, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8tad-cryi

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0218

Comment on FR Doc # 2016-22002

Submitter Information

Name: Janene Hasan

General Comment

I am concerned that this data could be used for unethical purposes, no matter how good

the intentions are in creating it. We must protect our children's privacy.

The Promise of Evidence-Based Policymaking 1821
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 28, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8taf-wocx

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0219

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

As a citizen who had a forty-one year career in education as an early childhood edcuatro,

I am deeply troubled by the intrusion by the federal government on the privacy rights of

young persons not old enough to protect themselves. As such, I strongly oppose any

proposal that would lead to the creation of a centralized, federal clearinghouse of the

personally identifiable information of all public school students, commonly referred to as

a student unit-record system or national database. There is truly no legitimate reason

why the government needs to have access to much of this private student data.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive.

In the past few years, data held by federal agencies has been hacked, including the

1822 The Promise of Evidence-Based Policymaking
personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

standards in its collection and storage of student data, and recently received a grade of

"D" for its security protections. 

In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness. 

As an educator who had been tasked with protecting student private and confidential

information from person(s) or group(s) not entitled to such personal information, I am

also very concerned about recent revelations of the widespread surveillance on ordinary

citizens by the federal government, and the way in which a national student data system

would be used to expand the tracking of students from preK into high school, college,

the workforce and beyond. A federal data clearinghouse of student information could

effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban.

Sincerely,

Mary E. Burnham

Cheshire, CT

Director of The Children's Center - retired 

Educational Consultant - retired

The Promise of Evidence-Based Policymaking 1823
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 28, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8tai-oelu

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0220

Comment on FR Doc # 2016-22002

Submitter Information

Name: James Scanlan

Address:

Washington, DC, 20007

Email: jps@jpscanlan.com

General Comment

COMMISSION ON EVIDENCE-BASED POLICYMAKING COMMENTS,

DOCKET ID USBC-2016-0003, QUESTION 16 (Nov. 28, 2016)

Attached are comments that follow up on comments submitted on November 14, 2016.

The earlier comments are available here:

http://www.jpscanlan.com/images/Comments_of_J_Scanlan_for_Comm_on_EvidenceBased_Policymaking_Nov._14,_2016_.pdf

James P. Scanlan

Attorney at Law

1529 Wisconsin Ave., NW

Washington, DC 20007

jps@jpscanlan.com

(202) 338-9224

1824 The Promise of Evidence-Based Policymaking
Attachments

Comments for the Commission on Evidence-Based Policymaking (Nov. 28, 2016)

The Promise of Evidence-Based Policymaking 1825
COMMISSION ON EVIDENCE-BASED POLICYMAKING COMMENTS, 

DOCKET ID USBC–2016–0003, QUESTION 16 

(Nov. 28, 2016) 

 Submitted by: 

 James P. Scanlan 

 Attorney at Law 

 1529 Wisconsin Ave., NW 

 Washington, DC 20007 

 jps@jpscanlan.com 

 (202) 338-9224 

These comments are a follow-up to comments submitted on November 14, 2016. The earlier 

comments are available here: 

http://www.jpscanlan.com/images/Comments_of_J_Scanlan_for_Comm_on_EvidenceBased_Policymaking_Nov._14,_2016_.pdf

1826 The Promise of Evidence-Based Policymaking
i 

Table of Contents 

 Page 

A. The Failure of the Scientific Community to Understand That Relative 

Differences in Mortality and Relative Differences in Survival Commonly Yield 

Opposite Conclusions About the Comparative Size of Demographic Disparities ............. 1 

B. The Mistaken Belief That Generally Reducing Discipline Rates Will Tend to 

Reduce Relative Demographic Differences in Discipline Rates and the Proportions 

More Susceptible Groups Comprise of Persons Disciplined ...................................... 3 

The Promise of Evidence-Based Policymaking 1827
 These comments are a follow-up to my Comments for the Commission on EvidenceBased Policymaking1

 submitted on November 14, 2016. The instant comments discuss two 

recent matters that illustrate points in the earlier comments regarding (a) the failure of persons 

analyzing demographic differences in outcome rates to understand that relative differences in 

adverse outcomes and relative differences in the corresponding favorable outcomes commonly 

yield opposite conclusions about the comparative size of racial and other demographic 

differences at different points in time, among different populations/subpopulations, and with 

regard to different conditions and (b) the failure of federal government agencies, including the 

Departments of Education and Health and Human Services, to understand that reducing adverse 

outcomes tend to increase, not decrease, relative demographic differences in rates of 

experiencing those outcomes and the proportions groups most susceptible to the outcomes make 

up of persons experiencing them. 

A. The Failure of the Scientific Community to Understand That Relative 

Differences in Mortality and Relative Differences in Survival Commonly Yield Opposite 

Conclusions About the Comparative Size of Demographic Disparities 

 At page 17 or the earlier comments, after presenting a number of examples of the pattern 

whereby the rarer an outcome the greater tends to be the relative difference in experiencing it and 

the smaller tends to be the relative difference in avoiding it, I stated: 

With only minor exception, the above patterns are utterly unknown in the law and the 

social and medical sciences. In fact, particularly with regard to infant and cancer 

outcomes, researchers will often refer to survival and mortality interchangeably, often 

stating they are examining relative differences in the former while in fact examining 

relative differences in the latter. They do so without recognizing the possibility, much 

less the likelihood, that the two relative differences will provide opposite results as to 

whether some demographic disparity is increasing or decreasing over time or is larger in 

one setting than another or with regard to one condition than another. See the Mortality 

and Survival page of jpscanlan.com. 

 

 A useful illustration of the failure of understanding in this area may be found in a study in 

the June 20, 2016 issue of the Journal of Clinical Oncology by Holowatyj et al. titled 

“Racial/Ethnic Disparities in Survival Among Patients With Young-Onset Colorectal Cancer.”

2

 

The study was highlighted in a May 25, 2016 article on the National Cancer Institute (NCI) 

website by NCI staff titled “Survival Disparities Identified in Young African Americans with 

Colorectal Cancer.” The study was funded by Wayne State University and arms of the National 

Institutes of Health including the NCI. 

 

1

 To facilitate consideration of issues raised in documents such as this I include links to referenced materials in 

electronic copies of the documents. An electronic copy of this document is available by means of the Measurement 

Letters page of jpscanlan.com. 

2 Holowatyj AN, Ruterbusch JJ, Rozek L, et al. Racial/ethnic disparities in survival among patients with youngonset colorectal cancer. J Clin Oncol 34(18):2148-2156, 2012. 

1828 The Promise of Evidence-Based Policymaking
2 

 Notwithstanding the titles of the study and the NCI article describing it, the study 

analyzed racial/ethnic differences in cancer outcomes in terms of relative differences in mortality 

rather than relative differences in survival. And, as with probably every other published study 

of demographic differences in cancer outcomes – regardless of the measure employed – it did so 

without showing any understanding that whether one in fact analyzed relative differences in 

mortality or relative difference in survival could yield opposite conclusions about a demographic 

disparity much less that this would commonly be the case when overall outcome rates vary 

substantially. 

 The failure of understanding affected much the Holowatyj study’s analysis. A variation 

on a near universal misunderstanding regarding the effects of generally reducing mortality on 

relative differences in mortality may be found in the study’s first paragraph. After noting recent 

reductions in cancer mortality, the study states (at 2148): “However, racial disparities in survival 

rates have grown more pronounced.” (Citations omitted). 

 The earlier comments and their references discuss the widespread notion that general 

reductions in mortality should reduce relative differences in mortality. The comments explain 

that, in fact, while general reductions in mortality will tend to reduce relative differences in 

survival, they will tend to increase relative differences in mortality (as is repeatedly observed in 

reality). In the case of the Holowatyj study, the word “however” in the quoted language suggests 

an expectation that general reductions in mortality should reduce relative differences in survival.3

 

This is a correct expectation. But it would be an incorrect expectation as to the relative 

difference in mortality that the Holowatyj study in fact employs to measure racial disparities in 

cancer outcomes and that the references the Holowatyj study cites as showing increased 

disparities in survival also employed. 

 The Holowatyj study (at 2151) describes as novel a finding that racial differences in 

survival were most pronounced for among individuals with stage II and stage III cancers given 

that prior studies have found the greatest survival disparities among individuals with advanced 

stage cancers. Such finding was also highlighted in the NCI article. 

 I have not examined what the other studies in fact measured. But what the Holowatyj 

study in fact found – i.e., larger relative differences in mortality (not survival) for less advanced 

cancers – is precisely what one should expect in the circumstances given the higher overall 

survival (and lower overall mortality) among persons with less advanced cancers. In fact, the 

Holowatyj study’s Figure 1 (at 2153) appears to show that, as one ought to expect in the 

circumstances, the relative difference in survival is larger for stage IV cancer for the less 

advanced cancers. 

 This is not to suggest that the data in Holowatyj study or its references will invariably 

comport with the patterns described in the earlier comments and their references.4

 But only with 

 

3

 A similar suggestion may be found in the statement in the Discussion section (at 2151) that racial disparities in 

survival “have actually worsened since 2000.” (Citations omitted.) 

4

 The study suggests that, contrary to the described patterns, the relative difference in mortality is greater for stage 

III than stage II cancer. That could reflect random variation of it could reflect something meaningful. Examples of 

The Promise of Evidence-Based Policymaking 1829
3 

an understanding of those patterns will one be able to derive useful information, or draw sound 

inferences about processes and mechanisms, from such data. 

 The Holowatyj study, however, is merely symptomatic of the fact that in all likelihood 

not a single person analyzing racial differences in cancer outcomes – or funding, providing peer 

review of, or publishing analyses of such differences – is aware that relative differences in 

survival and relative differences mortality can (or typically will) yield opposite conclusions 

about directions of changes in disparities or regarding the comparative size of disparities as to 

different populations/subpopulations or as to different conditions. 

 But while research into disparities in cancer outcomes provides a striking example of the 

confusion in the analyses of demographic differences (like the extreme examples discussed at 

pages 34-35 of the initial comments), similar failures of understanding undermine virtually all 

analyses of demographic differences involving outcomes rates. 

B. The Mistaken Belief That Generally Reducing Discipline Rates Will Tend to 

Reduce Relative Demographic Differences in Discipline Rates and the Proportions More 

Susceptible Groups Comprise of Persons Disciplined 

 A key point of Part I of the earlier comments involved the fact that federal civil rights 

enforcement policies regarding lending, school discipline, criminal justice, and employment are 

based on the mistaken premise that generally reducing the frequency of adverse outcomes would 

tend to reduce relative demographic differences in rates of experiencing those outcomes and the 

proportions groups more susceptible to those outcomes comprise of persons experiencing the 

outcomes. As explain in Part I and its references, exactly the opposite is the case. While 

reducing the frequency of an outcome tends to reduce relative differences in the corresponding 

favorable outcome, it tends to increase relative differences in rates of experiencing the outcome 

itself. Further, while reducing the frequency of an adverse outcome tends to increase the 

proportion groups more susceptible to the outcome make up of persons experiencing the 

corresponding favorable outcome (hence, reducing all measures of differences between the 

proportions such groups make up of persons potentially experiencing either outcome (the pool) 

and the proportions such groups make up of persons experiencing the favorable outcome), it 

tends also to increase the proportion such groups make up of persons experiencing the adverse 

outcome itself (hence, increasing all measures of differences between the proportions such 

groups make up of the pool and the proportions they make up of persons experiencing the 

adverse outcome). 

 The earlier comments also specifically recommended (at 46) that the Commission 

“recommend that Congress take all steps necessary to ensure that no federal law enforcement 

actions are based the belief that reducing the frequency of an adverse outcome tends to increase 

relative demographic differences in rates of experiencing the outcome or the proportion 

disadvantaged groups make up of persons experiencing those outcomes.”

 observed patterns of relative differences in morality and survival with respect to different overall survival rate 

situations may be found in the Mortality and Survival page of jpscanlan.com and Tables 10 and 11 of “Measuring 

Health and Healthcare Disparities,” Proceedings of the Federal Committee on Statistical Methodology 2013 

Research Conference (March 2014) 

1830 The Promise of Evidence-Based Policymaking
4 

 A recent development highlighting this issue with regard to one of the areas where the 

misunderstanding of the effects of the prevalence of an outcome on disparity measures has some 

of its most pernicious consequence may be found in the August 2016 award of a $1 million grant 

from the Department of Education (DOE) and the Department of Health and Human Services 

(HHS) to the Technical Assistance Center on Positive Behavioral Interventions and Supports 

(PBIS) to implement the Pyramid Equity Project. A document released by the Pyramid Equity 

Project in September 2016 titled “The Pyramid Equity Project: Promoting Social Emotional 

Competence and Addressing Disproportionate Discipline in Early Childhood Programs” (PEP 

fact sheet) reflects the belief, which has previously been expressed by DOE, HHS, and many 

other public and private entities, that generally reducing discipline rates will tend to reduce (a) 

relative differences between the discipline rates of groups with higher discipline rates and groups 

with lower discipline rates and (b) the proportions groups with higher discipline rates comprise 

of persons who are disciplined. The grant, which is focused on racial and gender disparities in 

preschool discipline, highlights both the failure to understand the effects of generally reducing 

discipline on measures of discipline disparities and the failure to recognize that racial disparities 

in preschool suspensions are seemingly large precisely because suspensions are rare in 

preschool. In fact, data cited in Pyramid Equity Project document suggest that a majority of 

school districts had no preschool suspensions. 

 The matter is explained more fully in a November 26, 2016 letter to the principal staff 

and consultants of the Pyramid Equity Project. The letter also urges the leadership of the 

Pyramid Equity Project to explain to the granting agencies that a central premise of the grant is 

the opposite of reality and to address with the funding agencies whether, in light of recognition 

of such fact, the agencies wish to reconsider the grant. 

 The suggestion that the Pyramid Equity Project address this issue with the funding 

agencies is entirely reasonable and involves actions that Pyramid Equity Project leadership, once 

understanding the statistical issues, ought to recognize as manifestly appropriate. It nevertheless 

warrants mention that there is something incongruous about a research regime where a citizen 

must explain to a federal grant recipient that a premise of the grant is false and urge the recipient 

to explain such fact to the granting agencies. Rather, federal agencies should understand these 

issues themselves before they issue grants. Thus, I suggest, there are compelling reasons for the 

Commission to follow the recommendation quoted above. 

 There are similarly compelling reasons for the Commission to follow the other 

recommendations set out at pages 45-46 of the earlier comments. 

The Promise of Evidence-Based Policymaking 1831
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 28, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8taj-x5jt

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0221

Comment on FR Doc # 2016-22002

Submitter Information

Name: John Bestor

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

As a citizen who had a forty-one year career in public education as a school

psychologist, I am deeply troubled by the intrusion by the federal government on the

privacy rights of young persons not old enough to protect themselves. As such, I strongly

oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database. There is truly

no legitimate reason why the government needs to have access to much of this private

student data.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive.

1832 The Promise of Evidence-Based Policymaking
In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

standards in its collection and storage of student data, and recently received a grade of

"D" for its security protections. 

In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness. 

As a school professional who had been tasked with protecting student private and

confidential information from person(s) or group(s) not entitled to such personal

information, I am also very concerned about recent revelations of the widespread

surveillance on ordinary citizens by the federal government, and the way in which a

national student data system would be used to expand the tracking of students from preK

into high school, college, the workforce and beyond. A federal data clearinghouse of

student information could effectively create life-long dossiers on nearly every individual

in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban.

Sincerely,

John Bestor

Cheshire, CT

National Association of School Psychologists - Retired

Connecticut Education Association - Retired

The Promise of Evidence-Based Policymaking 1833
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 28, 2016

Status: Posted

Posted: November 28, 2016

Tracking No. 1k0-8tam-5jr7

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0222

Comment on FR Doc # 2016-22002

Submitter Information

Name: Chris S

General Comment

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all students, commonly

referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008, the reasons against creating it

have only become more compelling.

In the past few years, much highly personal data held by federal agencies has been

hacked, including the release of the records of the Office of Personnel Management

involving more than 22 million individuals, not only federal employees and contractors

but also their families and friends.

The US Department of Education has been found to have especially weak security

standards in its collection and storage of student data, and received a grade of D for its

security protections.

1834 The Promise of Evidence-Based Policymaking
In addition, preK-12 student data currently collected by state departments of education

that would potentially be shared with the federal government include upwards of 700

highly sensitive personal data elements, including students' immigrant status, disabilities,

disciplinary records, and homelessness data.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from PreK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban in

the report you provide to Congress.

Thanks,

Chris S. 

Des Moines WA 98148

The Promise of Evidence-Based Policymaking 1835
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 28, 2016

Status: Posted

Posted: November 29, 2016

Tracking No. 1k0-8taq-kztl

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0223

Comment on FR Doc # 2016-22002

Submitter Information

Name: Carol Vyhonsky

Address:

Melbourne, FL, 32901

Email: cvyhonsky@earthlink.net

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive.

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

1836 The Promise of Evidence-Based Policymaking
Personnel Management. The US Department of Education has especially weak security

standards in its collection and storage of student data, and recently received a grade of

"D" for its security protections.

In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban.

The Promise of Evidence-Based Policymaking 1837
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 30, 2016

Status: Posted

Posted: November 30, 2016

Tracking No. 1k0-8tbq-a34r

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0224

Comment on FR Doc # 2016-22002

Submitter Information

Name: Laura Bowman

Address:

Roanoke, VA, 24018

Email: lnbowman@hotmail.com

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive.

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

1838 The Promise of Evidence-Based Policymaking
Personnel Management. The US Department of Education has especially weak security

standards in its collection and storage of student data, and recently received a grade of

"D" for its security protections.

In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban.

Thank you,

Laura Bowman

The Promise of Evidence-Based Policymaking 1839
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: November 30, 2016

Status: Posted

Posted: December 01, 2016

Tracking No. 1k0-8tc4-cv6t

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0225

Comment on FR Doc # 2016-22002

Submitter Information

Name: Miron Straf

Address:

Arlington, VA, 22203

Email: mstraf@vbi.vt.edu

General Comment

My comments in the attached document address the section on data infrastructure of the

Commission's request, specifically questions 3 and 4 on data infrastructure.

Miron L. Straf

Attachment1_<Remarks to the Commission on Evidence-Based Policymaking>

Attachments

COMMISSION-REMARKS-30N

1840 The Promise of Evidence-Based Policymaking
Remarks to the Commission on Evidence-Based Policymaking

The singular accomplishment the Commission could achieve is to provide responsible 

access by entrusted researchers to policy-relevant information that is now kept from them by

confidentiality legislation and government agency practices. Doing so will enable data to be 

linked in innovative–sometimes unpredictable–ways to provide needed evidence of the 

consequences of public policies and the means to improve them. 

Such access cannot be done by a data clearinghouse. No institution could house all the 

data that now flow in torrents from public and private sources. Restricting a clearinghouse to 

administrative data seriously hampers our ability to learn from vast resources of other data.

It cannot be done through controlled access in “data enclaves.” That would prevent 

downloads and linkages to important data outside the enclave. Even where researchers can 

upload data files for linkages and analyses, such as through virtual data enclaves, many 

important sources of data may not be available to the enclave. Moreover, the results may not be 

accessible if they pose a risk to confidentiality. A repository of data on specific programs that is 

accessible to researchers can lead to many program improvements, but, to fully evaluate the 

consequences of policies and programs, additional data are often required.

It cannot be done by agencies offering to analyze their data with a researcher’s model and 

providing only the results, not the data. This practice prevents researchers from making 

important, relevant discoveries by exploring patterns in data without a specific model in mind. 

Confirmatory data analysis is not the same as exploratory data analysis.

It cannot be done by altering or “de-identifying” data. That would impede, if not prevent, 

linkages and compromise the utility of the data.

It cannot be done by replacing the data with replicated or “synthetic” data. That would 

lose many characteristics of the original data, cast doubt on many findings, and limit discoveries.

It cannot be done by relinquishing control to individuals of what can be done with their 

personal data. That would seriously erode the utility of what data would be accessible. 

Although individuals may have the right to control their personal information, they do not have 

the right to control statistics and research findings derived from that information.

And it certainly cannot be done by requiring data to be used only for the purposes for 

which they were originally collected. That would eviscerate the value of data as a national 

resource.

How then can needed access be enabled while protecting the confidentiality of personal 

information? What are required are clear guidelines and enforceable standards that enable 

federal, state, and local government agencies to share data they currently do not with researchers 

for public policy purposes, including identification and description of societal conditions and the 

formulation, implementation, evaluation, and improvement of policies and programs.

The Promise of Evidence-Based Policymaking 1841
2

For this purpose, I suggest that the Commission recommend to the Congress and federal, 

state, and local agencies the following:

• First, establish clear legal standards for researchers and institutions to have access 

to confidential data.

Put the onus of responsibility to protect confidentiality on researchers and their 

institutions. Make intentional breaches of confidentiality of government-collected information a 

serious offense with serious penalties. For unintentional breaches, require researchers to report 

them and to maintain the confidentiality of those so identified. Put both researchers and 

institutions at risk of losing federal grants and contracts for improper practices.

As a first step in developing common legal standards that are clear, harmonize the many 

that now exist, for example, in the Confidential Information Protection and Statistical Efficiency 

Act, the Federal Information Security Management Act, the Health Insurance Portability and 

Accountability Act, and the Family Educational Rights and Privacy Act.

• Second, for data released to the public, establish a standard of reasonable care not 

zero tolerance to avoid inadvertent identifications that breach confidentiality.

The standard should apply not only to researchers and their institutions but also to 

statistics agencies and other governmental units that release public-use data files. Zero tolerance 

can be a standard for the public release of personal information, but, for research and statistical 

data derived from the information, the standard must be to exercise reasonable care so that the 

risk of identification is very small. The utility of data for analysis is greatly increased for a very 

small risk of inadvertent identification.

• Third, establish procedures for a researcher to become entrusted with confidential 

data.

Researchers should be authorized and their institutions accredited with high standards for 

access to confidential data. The standards should apply to facilities and practices to keep data 

secure. And the standards must require rigorous, effective training of researchers in the 

stewardship of confidential data.

Taking care to safeguard confidential data is part of the ethos of science. As such, it must 

be part of the education and training of future researchers and become accepted practice at all 

research institutions.

Miron L. Straf

Visiting Scholar, Social and Decision Analytics Laboratory (SDAL) 

The Biocomplexity Institute of Virginia Tech 

900 North Glebe Road 

Arlington, VA 22203-1890

1842 The Promise of Evidence-Based Policymaking
3

mstraf@vbi.vt.edu

Office: 571-858-3109

These remarks express the views of the author and are not intended to express the views of the 

Social and Decision Analytics Laboratory, Virginia Tech, or other organizations.

The Promise of Evidence-Based Policymaking 1843
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 01, 2016

Status: Posted

Posted: December 01, 2016

Tracking No. 1k0-8tch-n07b

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0226

Comment on FR Doc # 2016-22002

Submitter Information

Name: Chad Green

Address:

Leesburg, VA, 20175

Email: chadgreen@verizon.net

General Comment

16. How can data, statistics, results of research, and findings from evaluation, be best

used to improve policies and programs?

This question concerns the need to collect ongoing feedback from the environment in

order to ensure the resilience of both the federal government and the ecosystems to

which it is accountable (i.e., panarchy).

I am an internal evaluator located in the research office of a local school district in

Northern Virginia. Lately we've found that policymakers are requesting shorter review

cycles for improvement initiatives. Similar needs were reported in a national study of

school districts conducted by the Directors of Research and Evaluation (attached, see

findings on final slide; source: http://www.redirectors.org/recent-news-report-links ). 

This emerging shift to a short-cycle continuous improvement focus reflects the need to

both capture innovative evaluation practices and formulate new policies to support their

ongoing development. More fundamentally, I also see the importance of developing

1844 The Promise of Evidence-Based Policymaking
flexible systemic structures of governance which are more responsive to today's societal

challenges (i.e., "wicked problems"). 

How can government be more flexibly structured? One framework which I have found

very useful over the years is the Competing Values Framework:

http://www.valuebasedmanagement.net/methods_quinn_competing_values_framework.html

. I view it as a meta-sensemaking tool because I can align many other frameworks with

it. It can be used to manage program lifecycles and organizational functions, as well as to

facilitate dialogue on preferences for organizational culture (e.g., centralized,

decentralized, networked). For example, programs or offices within a government

department can be assessed as a portfolio to ensure a healthy mix of entities across the

four major models of organization: human relations, open systems, rational goal, and

internal process. 

Best,

Chad

Attachments

District Program Evaluation Study FINAL

The Promise of Evidence-Based Policymaking 1845




Martha Abele Mac Iver & Douglas J. Mac Iver

(With Robert Balfanz, Charlene Pryseski,

Jo Fennessey, & Vaughan Byrnes)

Everyone Graduates Center

Johns Hopkins University

April 2014

With special thanks to Joe O’Reilly and the

Directors of Research and Evaluation (DRE)

for helping facilitate the study

1846 The Promise of Evidence-Based Policymaking
www.extension.org

The Promise of Evidence-Based Policymaking 1847
Sites.temple.edu

1848 The Promise of Evidence-Based Policymaking


• What is the capacity (human resources and training, technological

capacity, data resources etc.) of Research and Evaluation Offices to conduct program evaluations? How does this vary by contextual

characteristics? What are the technical and capacity needs for

conducting program evaluations?

• What are the major barriers to acquiring timely program evaluation

findings?

• To what extent are technological solutions seen as helpful in

addressing barriers?To what extent are technological solutions

being used?

• How does the evaluation environment and demand for evaluation

vary across school districts?To what extent are there idea

champions urging use of data and evaluation results to guide

improvement efforts?

• To what extent are evaluations used in decisionͲmaking about

improving district initiatives?What factors contribute to low levels

of using program evaluation to inform decisionͲmaking?

The Promise of Evidence-Based Policymaking 1849


ȋ
Ȃ ʹͲͳͶȌ

• Measurement of publically available program evaluation reports

from district websites (Council of Great City Schools member

districts)

• Surveys of Directors of Research and Evaluation in local districts

• Telephone interviews with a subgroup of invited survey participants

1850 The Promise of Evidence-Based Policymaking





The Promise of Evidence-Based Policymaking 1851


Each of the member districts of the Council of Great City Schools (CGCS)

was selected for analysis:

• in 2004 (60 districts from previously conducted study)

• in 2014 (66 districts)

• 57 districts were CGCS members in both years

We counted how many program evaluation reports were posted by the 57

districts in each of those two years.

(Districts may have conducted program evaluations but not posted them

online.)

1852 The Promise of Evidence-Based Policymaking
ǡ

ʹͲͲͶʹͲͳͶ

26%

12%

21%

40%

Percent of Districts Posting Evaluation Reports

Both years

2004 only

2014 only

Neither year

The Promise of Evidence-Based Policymaking 1853




32.4% 67.6%

Of the 34 Districts with a posted report in either year

More reports posted in 2004 More reports posted in 2014

1854 The Promise of Evidence-Based Policymaking

ͳͲ

ʹͲͳͶȋȌ

Austin, TX

Dallas Independent, TX

Houston Independent, TX

Albuquerque, NM

Philadelphia, PA

Jefferson County, KY

Broward County, FL

Palm Beach County, FL

Minneapolis, MN

Hillsborough County, FL

Los Angeles Unified, CA

Portland, OR

MiamiͲDade County, FL

Denver, CO

CharlotteͲMecklenburg, NC

Clark County, NV

Charleston County, SC

Metropolitan Nashville, TN

St Paul, MN

Fresno Unified, CA

San Francisco Unified, CA

The Promise of Evidence-Based Policymaking 1855


ǣǡǡ	

As in 2004,public posting of evaluation reports in 2014 was

particularly prominent in:

• Texas (4 districts with at least one report)

3 districts have roughly 300 or more each (from multiple years)

• California (5 districts with at least one report)

• Florida (5 districts with at least one report)

1856 The Promise of Evidence-Based Policymaking


The Promise of Evidence-Based Policymaking 1857


Directors of Research/Evaluation from all member districts of the

Council of Great City Schools were invited to participate in the survey,

together with other members of the Directors of Research and

Evaluation (n=132).

A subgroup of these (n=26) were invited to participate in a followͲup

telephone interview.

1858 The Promise of Evidence-Based Policymaking


Districts

Invited for survey 132

Responded to survey 75 (57%)*

Responded to most

survey questions 61 (46%)

Invited for interview 26

Characteristics

Large districts from

all over U.S., 24/26

members of CGCS

Scheduled interview 13

Responded to

interview 13 (50%)

*Note:A few districts may have had multiple respondents.

The Promise of Evidence-Based Policymaking 1859


12%

15%

18%

23%

32%

Under 10,000 students

10,001Ͳ20,000 students

20,001Ͳ50,000 students

50,001Ͳ100,000 students

More than 100,000 students

District Size

1860 The Promise of Evidence-Based Policymaking


Poverty

(% Free/Reduced Price Lunch)

9%

45%

36%

10%

Under

25%

25Ͳ50%

51Ͳ75%

More than

75%

Urbanicity

44%

25%

23%

8%

Urban

FirstͲring

outside

major city

Suburban

Rural

The Promise of Evidence-Based Policymaking 1861
	

1862 The Promise of Evidence-Based Policymaking
Dzdz

Evaluation Products

• How many evaluations conducted?

• In what time frame?

• What types of programs evaluated?

• What designs used and types of data collected?

Human Resources

• Staffing

• Staff Training

Data Systems

• Data warehouse

• Linkage of student and teacher data

• Availability of programͲrelated data

The Promise of Evidence-Based Policymaking 1863


• 85% (64/75) of the total respondents reported

conducting a program evaluation within the past 2 years.

• Range of 0 to 50

• Mean of 8

• Median of 4

1864 The Promise of Evidence-Based Policymaking
ǣ



% of Respondents Mentioning Each

Program Type

Curricular Program... 89%

at elementary level 68%

at high school level 68%

at middle grades level 60%

at PreͲK level 38%

Program for ELL population 42%

Teacher professional development 40%

School reform initiative... 36%

secondary 32%

elementary 28%

After school program 34%

Dropout prevention program 28%

Mentoring program 23%

Summer school program 23%

The Promise of Evidence-Based Policymaking 1865


ͻ

16%

56% 28%

Percent of evaluations conducted in the last 2 years that

took...

less than 3 months to complete

3 to 9 months to complete

More than 9 months to complete

Analyses involved aggregating thetotal number of conducted evaluations

reported, within timeframe categories

1866 The Promise of Evidence-Based Policymaking


ǥ

• Programs need to run their course over the year

• Collection of qualitative data for formative feedback (from

interviews, observations, etc.) takes time

• Districts often interested most in the impact on the yearly

high stakes assessment

Interview respondents think it unlikely that technology could

speed up the process for many evaluations.

Districts may be open to more discussion about shortͲcycle

continuous improvement efforts, but our interview respondents

did not emphasize current district efforts in this direction.

The Promise of Evidence-Based Policymaking 1867
Ƭ

ǣ

Use student level outcome data drawn from multiple data systems 96%

Involve survey data collection 89%

Involve qualitative data collection (classroom observations, interviews, etc.) 78%

Involve online or electronic survey data collection 78%

Do Not Include a Comparison Group 71%

Rely on pre/post outcome measures without a comparison group 71%

Include a Comparison School or a Comparison Group 76%

Use comparison schools matched to treatment schools on

demographics and/or prior outcome measures

64%

Use a matching method other than propensity score matching to

identify a matched comparison group of students

42%

Use propensity score matching to identify a matched comparison

group of students

40%

Use a regression discontinuity design 16%

Use random assignment of students to treatment and control groups 9%

Use random assignment of schools to treatment and control schools 7%

Percentages represent % of respondents checking the response.Districts conduct multiple types of evaluations, both with and without

comparison groups.Rigorous methods to identify comparison groups are much less common.

1868 The Promise of Evidence-Based Policymaking




35%

11%

37%

11%

6%

How Many Program Evaluations Were Conducted for the

Respondent's District by External Research Groups

In Past 2 Years?

Zero

One

Two or Three

Four or Five

More than Five

The Promise of Evidence-Based Policymaking 1869
	



45%

33%

18%

5%

Sources Used to Fund These Evaluations

External Evaluation By Grants

Alone

External Evaluation By District

Sources Alone

External Evaluation By Combining

District Sources and Government

Grants

External Evaluation By Combining

District, Government, Foundation,

and Corporate Sources

1870 The Promise of Evidence-Based Policymaking
Ƭ



• District size is a strong predictor of the size of its Research and Evaluation

Office (ɴ=.67).

• The largest districts (more than 100,000 students) averaged more than 6Ͳ

10 staff members.

• The next largest districts (50,001 to 100,000 students) averaged about 6Ͳ

10 staff members.

• Most of the staff in research and evaluation offices devote at least part of

their time to duties other than program evaluation (such as generating

school by school accountability reports).

• Supplementary analyses reveal that research and evaluation offices with

more than 3 staff members typically have at least one staff member who

focuses exclusively on program evaluation.Large research and evaluation

offices (with “more than 10” or “more than 15” staff) typically have 3 or 4

staff whose fullͲtime focus is on program evaluation.

The Promise of Evidence-Based Policymaking 1871
Ƭ



38%

16%

25%

13%

8%

How many total staff members do you have in your

Research and Evaluation office?

Fewer than 3

3Ͳ5

6Ͳ10

11Ͳ15

More than 15

1872 The Promise of Evidence-Based Policymaking


	Ǧ

54%

19%

13%

14%

About how many of these staff members are devoted

fullͲtime to program evaluation?

None

1Ͳ2

3Ͳ4

5 or more

The Promise of Evidence-Based Policymaking 1873





6%

21%

18%

13%

42%

About what percent of your professional research and

evaluation staff members have graduate level training in

conducting high quality evaluations?

0%

1Ͳ25%

26Ͳ50%

51Ͳ75%

More than 75%

1874 The Promise of Evidence-Based Policymaking





0%

10%

20%

30%

40%

50%

60%

70%

80%

Advanced

statistical

methodology

(HLM, Propensity

Score Matching,

etc.)

Training in using

statistical

software (STATA,

SPSS, etc.)

Training in

classroom

observation

Tips for writing

up/presenting

results

General statistical

methodology

Training in

conducting focus

groups

Percent Identifying

Type of Need

The Six "Most Pressing" Professional Development

Needs Related to Program Evaluation

The Promise of Evidence-Based Policymaking 1875


Percent of respondents reporting:

66% 31% 3%

Multiple sources of student level data

available in a single warehouse

Data warehouse in process but not complete

No data warehouse

1876 The Promise of Evidence-Based Policymaking




48%

25%

16%

11%

To what extent is student level data linked to teacher data

in your district data files?

Linked but requires extensive roster

validation and/or data processing

WellͲlinked to teacher identification

number only

Student data wellͲlinked to teacherͲ

level variables

Not at all

The Promise of Evidence-Based Policymaking 1877


34%

15% 17%

56%

66%

52%

10%

19%

32%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Program or intervention

enrollment data

Program or intervention

attendance/dosage data

Intervention process

monitoring data

To what extent are the following types of student level

data available electronically in your district?

Not available

Available for some

programs/interventions

Available for most

programs/interventions

1878 The Promise of Evidence-Based Policymaking
ǣ

Dzdz

Limited staff time 54%

Timing of outcome data release 20%

Difficulty gathering appropriate outcome data 13%

Technological capacity 6%

Difficulty in identifying recipients of interventions 4%

Staff need additional training 4%

What would you say is the CHIEF BARRIER to

acquiring timely program evaluation findings?

The Promise of Evidence-Based Policymaking 1879




Aspect of Program Evaluation

Current Use of Technology

Solution

Projected Use of Technology

Solution

Use offͲtheͲ

shelf

technology

Use customͲ

created

technology

Do not use a

technology

solution (but

WOULD use, if

we had one)

WOULD NOT

use a

technology

solution for

this

To catalog available programs or interventions 17% 17% 50% 15%

To gather qualitative outcome data 13% 17% 48% 21%

To efficiently analyze qualitative outcome data 21% 19% 48% 12%

To identify valid comparison cohorts 18% 30% 38% 14%

To track program or intervention

attendance/dosage 21% 40% 34% 6%

To track program or intervention enrollment 24% 39% 33% 4%

To identify or flag students or staff for a particular

intervention or program

26% 42% 28% 4%

To deͲidentify or anonymize data to protect data

privacy without impeding the informative analysis

of studentͲlevel records

36% 44% 16% 4%

To link student data to teachers and/or to schoolͲ

level characteristics 40% 42% 15% 2%

To evaluate the outcome results of a program and

explore detailed outcome data from aggregate

results

46% 35% 12% 8%

1880 The Promise of Evidence-Based Policymaking


Ȃ 

33%

53%

10%

4%

A great deal

To some extent

Not very much

Not at all

To what extent do you think technology could be helpful in

addressing barriers to acquiring timely program evaluation

findings?

The Promise of Evidence-Based Policymaking 1881





Findings from Interviews

• Need good ways to visualize data (some districts could use

help in building dashboards)

• Some districts have data in different silos so that looking at

relationships among variables difficult

• Problems with human data entry

• takes lots of time

• errors common

• automation/technology would be useful

1882 The Promise of Evidence-Based Policymaking




• Collecting/managing data on:

• all district programs in various schools

• program attendance/dosage

• intervention details and/or process

• For example, could technology help in coding/analyzing

videos of classrooms?

In general, however, district respondents felt that the program

evaluation needs were more related to organizational factors than

potentially solvable through technology.Evaluations with useful

formative qualitative data take time – and technology cannot really

reduce the time required.

The Promise of Evidence-Based Policymaking 1883


• Who are the advocates of using program evaluation?

• To what extent do evaluation findings shape decisionͲmaking?

• Why do evaluation findings not influence decisionͲmaking as

much as they might?

1884 The Promise of Evidence-Based Policymaking


	



75% 73%

61%

43%

38% 38%

13%

Leaders in

Curriculum &

Instruction

Superintendent/CEO Assistant

Superintendents

Outside groups

(funders, community

groups, vendors)

Leaders in Student

Support Services

School Principals Leaders in Human

Resources

Which of the following would you say are strong advocates of using

program evaluation in your district?

The Promise of Evidence-Based Policymaking 1885
ǡ	






2%

33%

38%

10%

17%

What proportion of leaders in your district office (excluding

school principals) advocate the use of program evaluation

results to guide improvement efforts?

0%

1Ͳ25%

26Ͳ50%

51Ͳ75%

More than 75%

1886 The Promise of Evidence-Based Policymaking
	

ǦǦǦ 

13%

53%

27%

7%

To what extent have program evaluation findings shaped

district decisionͲmaking in the last two years?

To a large extent

Somewhat

Not very much

Not at all

The Promise of Evidence-Based Policymaking 1887
	

Ǧ

Political or financial considerations trumped evaluation findings 60%

Few program evaluations conducted 54%

Decisions had to be made before findings were available or

findings not acquired in a timely enough way

46%

District leaders did not agree with evaluation findings 15%

Evaluation findings were not actionable 15%

Reasons Why Program Evaluation Findings Have Not

Shaped District DecisionͲMaking as Much as They Could Have

(Multiple responses possible)

Percent of Respondents (n=52) Citing Reasons

Related Finding:The number of evaluations conducted in districts is

related not just to capacity, but also to the commitment of district

leaders to learn from evaluation findings.

1888 The Promise of Evidence-Based Policymaking
	

Ǧ



• Decisions need to be made before evaluation results ready (but

respondents generally did not think technology solutions could

make a significant impact on this)

• Pressure for decisionͲmaking does not allow many

programs/interventions to run their course and have a chance to

have an impact(Programs/interventions and assessments generally

not aligned to a shortͲcycle evaluation approach)

• Some district leaders defensive and/or not open to negative

evaluation findings

• Communication of program evaluation results to all stakeholders

does not always occur and/or is not always clear

• Political and financial considerations often trump evaluation findings

The Promise of Evidence-Based Policymaking 1889
	



Ǧ

• Lack of a strategic plan or overarching approach to the use of

evaluation in continuous improvement efforts raised by some

districts

• Evaluation occurs when required by funding sources, but not

necessarily done for most important programs

• Planning process does not take into account what is needed to

ensure collection of data and evaluation results needed for

informed decisionͲmaking

• This could be linked to high turnover in district leadership at top and

middle levels, mentioned by many district respondents

1890 The Promise of Evidence-Based Policymaking


The Promise of Evidence-Based Policymaking 1891


Ǧ	

• District leaders are asking for findings before most traditional

program evaluations can be completed.

• The term “program” has been traditionally associated with a multiͲ

component initiative designed to be implemented over a period of

time and aimed at broadly improving a global outcome (e.g. literacy,

high school graduation, supporting ELL students etc.).

• As a result, district evaluation efforts have typically focused on

conducting a small numberof longer evaluationsrather than a

shortͲcycle improvement process that focuses on discrete

interventions and their impacts or the impact of larger programs on

leading indicators.

• Districts and states are only in the very early stages of thinking

about a shortͲcycle continuous improvement process and how to

organize data collection and analysis to support this.

• More focused collaboration among district leaders about such a

shortͲcycle improvement process would be productive.

1892 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 02, 2016

Status: Posted

Posted: December 02, 2016

Tracking No. 1k0-8td0-v84j

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0227

Comment on FR Doc # 2016-22002

Submitter Information

Name: Elizabeth Jordan

Address:

Bethesda, MD, 20814

Email: ejordan@childtrends.org

General Comment

See attached file for comments from Child Trends

Attachments

Child Trends response to Request for Comments Evidence-Based Policymaking

The Promise of Evidence-Based Policymaking 1893
1 

Response to Request for Comments for the 

Commission on Evidence-Based Policymaking 

December 2, 2016 

Child Trends 

7315 Wisconsin Ave, Suite 1200W 

Bethesda, MD 20814 

www.childtrends.org 

Introduction 

On behalf of Child Trends, a nonpartisan, nonprofit research organization focused exclusively on 

improving the lives and prospects of children, youth, and their families, we would like to extend our 

thanks to the Commission on Evidence-Based Policymaking and its Commissioners. We appreciate their 

attention to such an important issue and the opportunity to participate in that conversation. We are 

pleased to see and hear so much support for developing and supporting evidence-based policymaking in 

the Commission's meetings and hearings and look forward to reviewing the resulting report in 2017. 

Child Trends brings 37 years of experience conducting research about children, families, and the 

programs that serve them. A major component of our work involves building an evidence base around 

“what works” for our most vulnerable children and families. Our staff of over 120 researchers brings a 

wealth of methodological experiences that we draw from for these comments: from implementation 

studies, to conducting multi-year randomized-controlled trials, to technical assistance for state and nonprofit leaders, to linking and analyzing state administrative data, to incorporating big data in our 

analyses. 

With this experience in mind, today we provide responses to questions 7, 10, 15, and 16, the questions 

that are most relevant to our daily work. We would be happy to provide additional details or submit 

answers to other questions if it would be helpful to the Commission. 

 

1894 The Promise of Evidence-Based Policymaking
2 

7. What data should be included in a potential U.S. government data clearinghouse(s)? What are the 

current legal or administrative barriers to including such data in a clearinghouse or linking the data? 

To have a comprehensive federal data clearinghouse, it is essential to incorporate available data on 

children and families. In 2016, the federal government will spend $309 billion on children.1

 This 

constitutes a significant investment in the public's money and supports a broad range of programs and 

services for families to support their health, education, nutrition, and other needs. Although a significant 

investment, the need is also great. Policymakers require high-quality and timely data to ensure that 

funds are used in an efficient manner, and that as many children as possible receive the services that 

will yield the best possible outcomes. To provide policymakers with the type of rich information they 

need to make decisions, data must be readily available and regularly analyzed. 

Creating a federal clearinghouse of data would overcome one the major obstacles that researchers and 

policymakers face now in using federal data: the "siloed" nature of data. Currently, data are housed in 

different agencies or warehouses by program or information on one family is located in several different 

data systems. Without access to multiple datasets, we can only tell a limited story about children and 

families – the story of their involvement in a particular program or service. For example, we may know 

much about the numbers, demographics, and service utilization of children in the child welfare system. 

However, without linking to other data, we cannot know if one family receives duplicate services, or if 

families access multiple programs. We also cannot link service receipt to long term outcomes in other 

areas such as education or justice. We are unable to paint a clear picture of what is happening to 

families in a single system or how to most efficiently and effectively serve them. 

Much work must be done to ensure that the data in the federal clearinghouse is as useful as possible. 

From our work, we specifically recommend the Commission consider the following: 

- Establish guidelines to ensure data quality so that researchers, policymakers and the public 

have confidence in the information gleaned from the clearinghouse. This includes making sure 

all those who enter the data are consistently defining data terms across states and communities 

and that data is collected in a timely manner. To achieve such high standards, states may need 

additional support and technical assistance to understand new requirement or terms. 

- Invest in training to ensure a federal workforce is skilled in data analysis. Establish partnerships 

among federal and state agencies, universities, and researchers so that the data in the 

clearinghouse is fully used. Simply having the data will not be enough to inform policymaking. 

Data must be analyzed and incorporated into evaluations so that the information can better 

inform decision making. 

- Support convenings of various stakeholders to discuss how best to translate the research into 

understandable and actionable information to shape decisions on programs and services. To 

provide raw numbers or technical analysis is not enough. 

- Identify and establish ways to link state and federal data to maximize utility of the 

data. Currently the federal government holds only a fraction of the data on services and 

participants in services. A large portion of data needed to get a full picture of service provision 

 

1

 Children’s Budget 2016. First Focus. Available at https://firstfocus.org/wp-content/uploads/2016/07/ChildrensBudget-2016.pdf

The Promise of Evidence-Based Policymaking 1895
3 

and participant outcomes for federally-funded programs are collected by state agencies. A 

federal-level data clearinghouse would need to either include state-held data or facilitate 

linkages between federal and state data to understand program outcomes. We recommend the 

latter. The Commission could make a recommendation on standardizing Memoranda of 

Understanding across state and federal agencies to share data, providing secure methods of 

linking data across datasets irrespective of whether they are federal or state, and having a 

shared data governance structure over administrative data held by the federal or state 

governments. 

10. How should the Commission define “qualified researchers and institutions?” To what extent 

should administrative and survey data held by government agencies be made available to “qualified 

researchers and institutions”? 

We recommend the Commission make federal administrative and survey data available to external 

researchers. The research community is broad. It includes universities, think tanks, professional research 

organizations, and other groups. Over the years, Child Trends and other independent organizations 

have interpreted various government datasets to shine a light on issues of importance to children and 

families. Our work, like that of others organizations, has long been valued by policymakers at all levels of 

government and across the political spectrum. This work has expanded the knowledge on evidencebased programs serving children and youth in areas such as child welfare, teen pregnancy prevention, 

bullying prevention, and poverty reduction. 

External research organizations can provide a level of independence to the analyses and interpretation 

of the results that lends credibility to the findings. 

15. What barriers currently exist for using survey and administrative data to support program 

management and/or evaluation activities? 

As discussed above in our response to question 7, data quality (specifically clear and consistent data 

definitions for data entry and data timeliness) is a barrier to using survey and administrative data for 

program management and/or evaluation activities. Additionally, the Commission should be aware of the 

following challenges: 

- Access to data: Researchers need streamlined processes to access data, while protecting data 

security. Currently, access to data can be an arduous process, taking months to years. To have 

research that is timely enough to support policy decision-making, researchers need swift access 

to data to answer questions before they are moot. Strategies like those of the National Center 

for Education Statistics or the National Center for Health Statistics, which set up secure access to 

data, may be useful for the Commission to explore. 

- Siloed nature of data: As discussed above, data is currently siloed in one system or the other 

and at the federal or state level. Linking or connecting data is critical to answering the types of 

questions we need to inform decision-making. 

- Timeliness of data/evaluation: Evidence-based policymaking and programmatic decisionmaking often requires data to be available and analyzed quickly to answer questions and 

respond to issues on a timely basis. Currently, accessing data is not a timely or efficient process. 

When programs are administered by local communities, there is often a natural lag between 

1896 The Promise of Evidence-Based Policymaking
4 

when data are reported to the state agency, cleaned and prepared and then reported to the 

federal government. This process often can take multiple years so data currently available at the 

federal level may be years old (e.g. Medicaid or birth records). For more timely evaluations, 

researchers are often faced with negotiating data sharing with each individual state, which can 

take 6 to 18 months to secure an agreement. 

- Dissemination of research to policymakers: In our work partnering with states and childserving organizations, we have learned the importance of distilling and translating research into 

actionable information for use in policymaking and programmatic decision-making. However, 

researchers are incentivized to publish in peer-reviewed journals or lack funds to make research 

more broadly accessible. Organizations such as Child Trends can be effective partners in 

disseminating knowledge to policymakers and others who are making decisions on programs 

and services for children and families. . 

16. How can data, statistics, results of research, and findings from evaluation, be best used to improve 

policies and programs? 

Data and research must be a continuous part of policy and programmatic work. It is not as simple as 

conducting an evaluation at some static moment after a program gets up and running. Evaluation, 

research, and data are all parts of a process that begins as policymakers and community members 

identify a problem that needs to be addressed and continues throughout a program’s life. Though 

discussed below as a linear process, our experience indicates the process of integrating research and 

evaluation into policies and programs is iterative and overlapping. The process may begin with a needs 

assessment using the most up to date administrative data to identify the issues faced by a community 

and population that is the focus of intervention efforts. It continues through intervention identification, 

using research and systematic reviews to identify available effective models. Implementing 

organizations often develop a theory of change and a logic model based on research and 

implementation science. A performance management system should be developed and data regularly 

collected so that program inputs, outputs, and outcomes can be tracked, reported and used for 

monitoring progress and continuous improvement. Research and evaluation may be used at multiple 

points in time to inform program design and implementation. Descriptive evaluations of participant 

outcomes and implementation can support high-quality programs through documenting activities and 

outcomes. Once a program is mature, impact evaluations may occur to determine if the program as a 

whole or individual pieces of the program are effective. Programs are all at different stages of 

implementation and need different types of research and evaluation support to guide evidence in their 

decision-making. 

Research on the use of evidence in policymaking suggests leadership and staff use people and 

organizations they trust to learn about relevant research. Without trust and ongoing partnership, 

research often goes unused. Ongoing research-to-practice partnerships are one tool to ensure research 

and evaluation remains relevant, timely, and integrated to policy and practice. 

The work of the Commission is exciting and has the potential to advance the adoption of evidence-base 

d policymaking. To maximize these investments we hope the Commission will make sure data is fully 

accessible and available on a timely basis; fund training to improve data analysis and expertise among 

stakeholders; and encourage partnerships to ensure the wide application of the data to improve the 

efficiency and effectiveness of all government services and programs. 

The Promise of Evidence-Based Policymaking 1897
5 

Conclusion 

We would be happy to provide further information on any of these recommendations or address 

specific questions. Please feel free to contact: Elizabeth Jordan, senior policy analyst, Child Trends at 

ejordan@childtrends.org or 202-520-9090. 

1898 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 03, 2016

Status: Posted

Posted: December 05, 2016

Tracking No. 1k0-8tdq-gih9

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0228

Comment on FR Doc # 2016-22002

Submitter Information

Name: Steven Carnes

Address:

Tea, SD, 57064

Email: stevesauto@iw.net

General Comment

It would take a team of 10 attorneys to decipher this written garbage. This is all about

data sharing on our children and others, it is un-needed. I would like to know what

special interest groups brought this pre-written garbage to you. Stop it already, do not

pass this.

The Promise of Evidence-Based Policymaking 1899
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 03, 2016

Status: Posted

Posted: December 05, 2016

Tracking No. 1k0-8tdx-7hn7

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0229

Comment on FR Doc # 2016-22002

Submitter Information

Name: Melissa Aberle

General Comment

I urge this commission to use its power to strengthen local control of data, 

meaning parental and teacher stewardship over student data, instead of 

aiming to broaden the numbers of people with access to personally identifiable 

student information to include government agencies and/or educational sales/research

corporations

such as Pearson, Microsoft, or the American Institutes for Research.

I urge this commission to use any influence that it has to promote safekeeping 

of unit-record data at the parental and teacher level, where that authority rightly belongs.

Please protect the privacy of our children.

1900 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 04, 2016

Status: Posted

Posted: December 05, 2016

Tracking No. 1k0-8te7-8nt4

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0230

Comment on FR Doc # 2016-22002

Submitter Information

Name: Elaine Simons

General Comment

Dear Commission on Evidence-Based Policymaking,

I love the American concept of voter-based, Constitution-based, elected representativebased, policymaking. It's why I live in America. In contrast to voter-based policymaking

there is evidence-based policymaking, which I don't love because it implies that one

entity's "evidence" trumps individuals' consent to new policy changes.

Former Secretary of Agriculture Ezra Taft Benson said something about education that

also applies to educational data and policymaking:

"The best way to prevent a political faction or any small group of people from capturing

control of the nation's educational system is to keep it decentralized into small local

units... This may not be as efficient as one giant super educational system (although

bigness is not necessarily efficient, either) but it is far more safe. There are other factors,

too, in favor of local and independent school systems. First, they are more responsive to

the needs and wishes of the parents and the community. The door to the school

superintendent's office is usually open to any parent [or teacher]... But the average

citizen would be hard pressed to obtain more than a form letter reply from the national

Commissioner of Education in Washington, D.C."

The Promise of Evidence-Based Policymaking 1901
Local control, and consent of the governed, are two foundational principles in our great

nation.

Because the CEP is not an elected body, it does not hold authority to collect, or to

recommend collection, of student-level evidence, or of any evidence, without written

consent; and, for the same reasons, neither does the Department of Education.

Because the fifty, federally-designed, evidence-collecting, State Longitudinal Database

Systems never received any consent from the governed in any state to collect data on

individuals (as the systems were put into place not by authority, but by grant money) it

follows that the idea of having CEP study the possible removal of barriers to federal

access of those databases, is an egregious overstep that even exceeds the overstep of the

State Longitudinal Database Systems.

Because federal FERPA regulations altered the original protective intent of FERPA, and

removed the mandate that governments must get parental (or adult student) consent for

any use of student level data, it seems that the idea of having CEP study and possible

influence removal of additional "barriers" to federal use of data, is another egregious

overstep.

As a mother of children who currently attend public, as a patriot who believes in

"consent of the governed" and in the principles of the U.S. Constitution; I feel that my

letter represents the will of many who stand opposed to the study of the removal of

protective barriers on student-level data, which the CEP's website has outlined it will do.

I urge this commission to use its power to strengthen local control of data, meaning

parental and teacher stewardship over student data, instead of aiming to broaden the

numbers of people with access to personally identifiable student information to include

government agencies and/or educational sales/research corporations such as Pearson,

Microsoft, or the American Institutes for Research.

To remove barriers to federal access of student-level data only makes sense to a socialist

who agrees with the Marc Tucker/Hillary Clinton 1998 vision of a cradle-to-grave nanny

state with "large scale data management systems" that dismiss privacy as a relic in

subservience to modern government. It does not make sense to those who cherish local

control.

It is clear that there is a strong debate about local control and about consent of the

governed, concerning data and concerning education in general. NCEE Chair Mark

Tucker articulated one side of the debate when he said: "the United States will have to

largely abandon the beloved emblem of American education: local control. If the goal is

to greatly increase the capacity and authority of the state education agencies, much of the

new authority will have to come at the expense of local control."

Does that statement match the philosophical stand of this commission? I hope not. Local

control means individual control of one's own life. How would an individual control his

1902 The Promise of Evidence-Based Policymaking
or her own destiny if "large scale data management systems" in a cradle-to-grave system,

like the one that Tucker and Clinton envisioned, override the right to personal privacy

and local control? It is not possible.

I urge this commission to use any influence that it has to promote safekeeping of unitrecord data at the parental and teacher level, where that authority rightly belongs.

Sincerely,

Elaine Simons

The Promise of Evidence-Based Policymaking 1903
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 05, 2016

Status: Posted

Posted: December 05, 2016

Tracking No. 1k0-8tf2-hukp

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0231

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

Based on Arnstein's ladder of citizen participation, this mechanism for providing

feedback by the public would be categorized as level 4 consultation, a form of tokenism:

http://www.citizenshandbook.org/arnsteinsladder.shtml .

To move up the ladder (level 5 placation), the 15-member Commission on EvidenceBased Policymaking would take steps to engage key stakeholders, such as from AEA

and other VOPEs, in the decision-making process itself through the formation of

subcommittees. Nevertheless, this step is only a more sophisticated form of tokenism.

Level 6 (partnership), where power in the planning and decision-making process is

shared, would be a more flexible organizational arrangement assuming the commission

is authorized (and amenable) to playing this leadership role.

1904 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 06, 2016

Status: Posted

Posted: December 06, 2016

Tracking No. 1k0-8tfw-qtpx

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0232

Comment on FR Doc # 2016-22002

Submitter Information

Name: Frank Farrow

General Comment

The Center for the Study of Social Policy (CSSP) is pleased to provide comments to the

Commission on Evidence-Based Policymaking. We appreciate the opportunity to

participate in this important process and thank you for your consideration.

Attachments

CSSP Comments to the CEP

The Promise of Evidence-Based Policymaking 1905
1575 Eye Street, NW x Suite 500 x Washington, DC x 20005 x p202.371.1565 

50 Broadway x Suite 1504 x New York, NY x 10004 x p212.979.2369 

www.cssp.org 

December 6, 2016 

Re: Commission on Evidence-Based Policymaking Comments Docket ID USBC–

2016–0003 

Attention: Commission on Evidence-Based Policymaking 

The Center for the Study of Social Policy (CSSP) is pleased to provide comments to the 

Commission on Evidence-Based Policymaking (hereafter, the Commission) regarding the 

use of data in program design, management, research, evaluation and analysis. We thank 

the Commission for focusing on the availability and use of government data in support of 

evidence-building activities related to government programs and policies as these 

practices have the potential to improve outcomes for vulnerable groups, particularly 

children and families. With limited public and private resources available, it is imperative 

that we use evidence to make decisions aimed at achieving better results, at scale. As 

such, we appreciate the opportunity to provide comment on this matter. 

CSSP is a national policy organization recognized for its leadership in shaping policy, 

reforming public systems and building the capacity of communities with a focus on 

promoting racial equity and securing equal opportunities and better futures for children, 

youth and families. We are also the founding member of Friends of Evidence – a group 

of distinguished leaders across diverse fields, disciplines and sectors – concerned with the 

limitations of prevailing approaches to obtaining evidence and focused on defining and 

promoting more effective approaches to gathering and using evidence in support of better 

results. CSSP’s policy positions are informed by our extensive experiences helping early 

childhood, child welfare and other public human services systems advance their work and 

achieve improved outcomes for the children and families in their care. Through our work 

over the years we’ve found that simply funding or implementing initiatives based on 

currently available lists of evidence-based programs will not achieve the goals we all 

share. Initiatives and their component programs must be implemented with a clear 

attention to measuring and understanding results, using that information to make wellconceived adaptations and mid-course corrections when initiatives fall short of achieving 

desired results and insuring that processes are in place for continuous learning and 

accountability to funders, taxpayers and service recipients. 

Establishing a results focus and a rigorous approach to generating and applying credible 

evidence – through multiple techniques and methods and from multiple sources – should 

be institutionalized in program design and implementation. However, acceptable 

1906 The Promise of Evidence-Based Policymaking
2 

evidence should not be limited to that derived from one methodology or one source of 

information. Experimental and quasi-experimental evaluation methodologies play an 

important role in building general knowledge, but they cannot provide the only 

information that is valued. In fact, rigor demands that an evaluation approach match the 

questions being asked. As OMB has stated, “Rigor is not restricted to impact 

evaluations.” Therefore, rigorous and robust evidence requires an infrastructure is one 

that supports both internal mechanisms for continuous improvement that are aligned with 

external evaluation efforts. Yet creating this internal infrastructure often does not receive 

enough allocation of time and resources. Adopting a comprehensive approach to data 

gathering and analysis enables implementers to learn whether the selected program or the 

newly designed initiative is having the intended impact for the target population and 

allows others to understand how that impact is being achieved. Administrative data and 

surveys are two sources of evidence that should be employed, although not exclusively, 

in designing, implementing and evaluating policies, programs and multi-discipline 

complex initiatives. Findings from administrative data and surveys have the ability to 

reveal population needs as well as a particular practice or program’s effectiveness with a 

target population. It also serves as a catalyst to making timely adjustments to improve 

practice or population outcomes. 

The remainder of our statement will focus on the five questions addressed in the section 

of the Federal Register Notice on the use of data in program design, management, 

research, evaluation and analysis. We have organized the questions into two groups and 

will specifically address topics concerning (1) the importance and limitations of 

administrative data and surveys in improving policies and programs – questions 15 and 

16 – and (2) incorporating evaluation into program design – questions 17, 18 and 19. 

The importance and limitations of administrative data and surveys in improving 

policies and programs 

Evidence generated from multiple sources is essential for policymakers, advocates and 

service providers to make decisions that will have the greatest impact on children and 

families. Administrative data and surveys should be employed in designing, 

implementing and evaluating policies, programs and complex initiatives. Administrative 

data in particular should be readily available to be regularly reviewed by the agencies 

collecting the data in a continuous improvement discipline as it can serve as a catalyst to 

making timely adjustments to improve practice or population outcomes. 

However, too often there are technical and capacity barriers to using both administrative 

data and surveys as part of a continuous evidence-building process. This is because: 

x Databases are incomplete, “siloed” and apply different definitions. In our 

experience assessing child welfare systems and the outcomes achieved, we can 

assert that on their own, federally financed child welfare administrative data bases 

often do not contain the information that would be most meaningful for assessing 

child and family well-being outcomes. For example, until recently at the federal 

level we could not say how often case workers visited children without reading 

and coding narrative records. Furthermore, we do not know how many of our 

The Promise of Evidence-Based Policymaking 1907
3 

foster care youth are expectant or parenting, nor do we know the developmental 

or educational attainment of these youth in many states because the information is 

not captured in the administrative data. Finally, states do not consistently capture 

information about a family’s housing needs, and the actual living conditions of 

“homeless” families often differs from the definition applied by the Department 

of Housing and Urban Development (HUD).1

 Therefore, child welfare 

administrative data often needs to be to be linked with other data bases – such as 

health care, education and homeless systems – in order to have a more complete 

picture of the child’s needs and outcomes.

x Current child welfare data bases are typically not configured in a way to facilitate 

longitudinal analysis without the additional expertise needed to effectively link 

records – such as the work of the Chapin Hall Center for Children with the 

Multistate Foster Care Archive. These technical barriers make it difficult for 

initiatives aiming to improve well-being and track progress over time to have the 

knowledge base for effective decision-making. Furthermore, initiative managers 

sometimes lack the analytical skills to make effective use of available 

administrative data. 

Free standing surveys produce point-in-time snapshots of perceptions and conditions, but 

can be a costly means of gathering information that can quickly become dated. In our 

work, one-time surveys are useful for learning directly from those affected about the 

problems to be addressed. However, unless surveys can be repeated over time at regular 

intervals with similar populations so that the results of multiple surveys can reveal trends, 

they will not be effective for assessing progress toward outcome achievement or be an 

efficient tool for jurisdictional-level continuous evidence-building and improvement 

operations. 

The application of both administrative data and surveys requires statistical and analytical 

skills not always available within service delivery organizations at the state and local 

levels. An example where such skill building has been a focus is in New Jersey. The New 

Jersey Department of Children and Families (DCF) implemented a “Data Fellows 

Program” that provided 100 middle-management DCF staff with the opportunity to 

utilize administrative data to support improved case practice and outcomes for the 

children and families they serve in their areas or local offices all while developing 

analytical and presentation skills for management and decision-making, leadership, and 

team building.2

 

 

1

 Currently, HUD considers a family to be homeless if they are living on the street or in a shelter and 

prioritizes services to chronically homeless populations. This excludes many homeless families and youth 

as they are often “doubled up” with other family members or friends or living in one-bedroom motels. For 

more information on definitions of homelessness, please refer to materials by First Focus regarding the 

Homeless Children and Youth Act of 2015. 

2

 Initial funding for this program came from the Children’s Bureau through the Northeast and Caribbean 

Implementation Center. 

1908 The Promise of Evidence-Based Policymaking
4 

Incorporating evaluation into program designs. 

Our work in systems reform and with place-based initiatives tackling complex and 

interdependent social challenges underscores the need for a broader evaluation focus –

one that focuses more on systems and population outcomes across communities rather 

than on individual programs. It has also demonstrated to us the value of a culture of 

continuous learning and improvement as a means of reengineering systems3

 and creating 

greater opportunities for achieving equitable population outcomes. Before discussing 

specific methodologies, it is important to define the kinds of evidence that we think 

should be gained by incorporating evaluation and other measurement techniques into 

program design. The recent federal budget included a statement from OMB that offers the 

following definition, “Broadly speaking, “evidence” is the available body of facts or 

information indicating whether a belief or proposition is true or valid.” OMB states that 

“The best government programs use a broad range of analytical and management tools, 

which collectively comprise an “evidence infrastructure,” to learn what works (and what 

does not) for whom and under what circumstances, as well as improve results.” We 

support this definition. Evidence is derived from several sources: program management 

and monitoring as well as formal evaluations that employ a range of data collection and 

analytical methods and research that have relevance to achieving the desired results. In 

addition, the rigor and credibility of evidence is not established by a single evaluation 

method. Rigor and credibility are determined by the use of multiple methods 

appropriately matched to what we want to learn and need to know. 

Programs and complex initiatives, therefore, should be required to incorporate rigorous 

robust evidence-building mechanisms in design and implementation. The evaluation 

methodology should fit the purpose of the inquiry rather than a methodology driving the 

program design. Therefore, multiple evaluation methods and the application of multiple 

sources of evidence should be institutionalized in programs along with investment in key 

infrastructure components such as data systems, research expertise, and analytical skills 

that support an “evidence culture”. To illustrate what we mean, we offer two examples 

of complex initiatives that are greater than the sum of bundled program components and 

are achieving improved outcomes for the populations they serve by using an inclusive 

and expanded approach to generating and applying evidence. 

x First, The Northside Achievement Zone (NAZ) is a collaborative working to close the 

achievement gap and end multigenerational poverty in a 13- by 18- block area on the 

north side of Minneapolis. NAZ brings together service providers, schools and 

families to move families through a cradle-to-career “pipeline” of supports for 

children from birth through college and ultimately career. From the design stage, 

NAZ has drawn upon multiple sources of evidence to steer and adapt its strategies 

and to create the interventions that address the unique needs of a particular 

community. It “fits the tool to the task,” using different evaluation tools, including 

randomized control trials, for different purposes. It has created model systems, 

infrastructure and processes that regularly engage a broad group of stakeholders in 

 

3

 See Jeffrey B. Liebman (2013) Building on Recent Advances in Evidence-Based Policymaking. 

The Promise of Evidence-Based Policymaking 1909
5 

analyzing and using data to continually improve. NAZ has demonstrated significant 

increases in school readiness among the population it serves. 

x Second, The Carnegie Foundation for the Advancement of Teaching is nurturing a 

networked improvement community to address the extraordinarily high failure rates 

among the half-million community college students annually assigned to 

developmental math in the US. Failing developmental math can seriously derail a 

student from taking full advantage of higher education opportunities. Carnegie Math 

Pathways (CMP) is a multi-pronged strategy designed tackle this problem and 

employs improvement science to continually assess its progress, learn for whom the 

strategies are working and where and how the strategies are coming up short. CMP 

is being implemented by a network of college faculty, researchers, designers, students 

and content experts who work together, using both research and practitioner 

knowledge, to create a new system to increase student success in development 

mathematics. The multiple elements of this systems-level intervention include a redesigned curriculum, changes in the way remedial math classes are conducted, new 

ways of preparing students by changing attitudes about whether they can be 

successful at math and the development of support networks among students. Perhaps 

its most important single design feature is an ongoing, data-rich quality improvement 

effort. CMP has tripled the success rates of community college students in remedial 

math courses in half of the time it normally takes students to complete them. 

Moreover, CMP students consistently outperformed their non-CMP peers across 27 

colleges that participated in CMP in School Year 2013-2014, despite the varying 

conditions and resources across these schools. The fact that these improvements have 

occurred for every racial, ethnic and gender subgroup and virtually every college 

participating in the initiative should give decision-makers the assurance they need 

about both the process and its impact.

CSSP commends the Commission for seeking feedback regarding the availability and use 

of government data in support of evidence-building activities related to government 

programs and policies. We hope that the comments provided above help to inform the 

Commission’s future activities and aids in the fulfillment of the Commission’s duties. We 

thank you for your continued commitment to building evidence to continuously improve 

public programs and policies and strengthening evidence-building to inform program and 

policy design and implementation. 

If you have any questions, please don’t hesitate to contact Frank Farrow, (202) 371-1565; 

frank.farrow@cssp.org and thank you in advance for consideration of our comments and 

recommendations. 

Sincerely, 

 Frank Farrow 

Director 

Center for the Study of Social Policy

1910 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 06, 2016

Status: Posted

Posted: December 07, 2016

Tracking No. 1k0-8tfz-8sm9

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0233

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all students, commonly

referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008, the reasons against creating it

have only become more compelling.

In the past few years, much highly personal data held by federal agencies has been

hacked, including the release of the records of the Office of Personnel Management

involving more than 22 million individuals, not only federal employees and contractors

but also their families and friends.

I urge you to strongly oppose the creation of any centralized federal data system holding

The Promise of Evidence-Based Policymaking 1911
students' personally identifiable information and to support the continuation of the ban in

the report you provide to Congress.

If you decide to propose this centralized database of all children in our country, you need

to have strong "OPT IN" requirements as part of this. 

The federal government should not be in the business of collecting and tracking children

through their lives. When they become adults, they can decide what data relating to their

persons is appropriate for others to have. 

The government should crack down on private companies compiling massive amounts of

data on US citizens.

1912 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 07, 2016

Status: Posted

Posted: December 07, 2016

Tracking No. 1k0-8tge-w7m9

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0234

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

Stop the invasive, ubiquitous intrusion of private citizens lives. "Evidence-Based" and

"Best Practices" are buzzwords used to justify top-down control. Individual Freedom is

dead; you helped kill it.

Shame on all of you politicians, bureaucrats, think-tankers and researchers for

agrandizing yourselves while enslaving the rest of us.

The Promise of Evidence-Based Policymaking 1913
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 08, 2016

Status: Posted

Posted: December 08, 2016

Tracking No. 1k0-8th8-8cgc

Comments Due: December 14, 2016

Submission Type: API

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0235

Comment on FR Doc # 2016-22002

Submitter Information

Name: Paul Lingenfelter

Address:

Westminster, CO, 80234

Email: paul@sheeo.org

General Comment

Dear Co-Chairs Abraham and Haskins, and Members of the Commission:

Noting that you have extended the period for submitting comments to the Commission, I

decided to review the comments you have received, including my own of November 1,

2016. Please forgive me for submitting a second comment. 

The attached file comments on the challenges and goals facing the Commission in view

of the inputs you have received. 

Thanks for your service.

Paul E. Lingenfelter

Attachments

1914 The Promise of Evidence-Based Policymaking
PEL Comment #2

The Promise of Evidence-Based Policymaking 1915
PAUL E. LINGENFELTER

President Emeritus, State Higher Education Executive Officers 

December 8, 2016 

Dear Co-Chairs Abraham and Haskins, and Members of the Commission: 

Noting that you have extended the period for submitting comments to the 

Commission, I decided to review the comments you have received, including my 

own of November 1, 2016. Please forgive me for submitting a second comment. I’ll 

do my best to be brief. 

My quick review of 220 comments submitted to the Commission found 158 

comments submitted by the general public that were uniformly and often quite 

strongly negative. Many of these used identical or similar language, suggesting an 

organized effort, but I have no doubt that views fearing the loss of privacy and 

governmental intrusion into personal life are widespread. 

The 62 comments (slightly less than 30% of the total) submitted by organizations or 

people with a professional interest in the topic, such as myself, were generally 

positive about the task of the commission, even though most had suggestions for 

improving governmental policy and practice. The gulf between the views of the 

general public and the “professionals” is striking. As one who believes the public 

good depends on evidence-based policy and practice, it is also disturbing. It is clear 

that all of us who seek to use data to improve human lives need to do a better job of 

using data effectively and demonstrating through our work that the way evidence is 

used is beneficial to individuals and society as a whole. 

The Commission, as a bi-partisan, deliberative body, is well situated to make an 

important contribution. I respect the members of the Commission and the difficulty 

of your task, and I look forward to your report. Reflecting on my previous comments 

and those submitted by others, I believe it will be helpful if the Commission report: 

1. Articulates your own consensus on both the proper role of policy and its 

practical and ethical limitations in shaping practice in education, in medicine, 

and in other fields affecting individual lives. Obviously, it is not easy to draw a 

bright line between governmental over-reach and a proper role for the 

federal government (or state governments) in supporting research and 

practice and in regulating or providing incentives to shape practice. But it 

would be helpful for the Commission to articulate principles to guide the 

drafting of legislation and administrative regulations. 

2. Articulates the differences and similarities between the use of evidence in 

practice and in policy, including: a) that both policy and practice need robust, 

1916 The Promise of Evidence-Based Policymaking
commonly accepted measures, and b) that the data required to guide policy are 

much more coarse-grained than those required for practice. The creation of 

massive, fine-grained data sets on individuals at the national level or at the 

state level unavoidably raises concerns about inappropriate governmental 

use. 

While longitudinal, individual data are essential for analyzing policies such as 

the impact of student assistance on participation, retention, and graduation, 

only a modest number of data elements are usually needed. For example, the 

much-maligned Department of Education proposal to create a student record 

data system for higher education included only 44 data elements. These data 

elements would have been sufficient for creating all the existing IPEDS 

institutional reports as well as meaningful state and national data on student 

enrollment, progression, and graduation. They would have been quite helpful 

in illuminating the impact of adequate or inadequate financial aid on student 

progress. 

I agree with the private citizens who argue that there is no legitimate reason 

for detailed personal data on students (such as grades, disciplinary actions, 

etc.) to be in a data base accessible to people other than those directly 

responsible for assisting students in achieving personal educational goals. 

3. Specifies the techniques necessary to assure absolute privacy of individual data 

used in research and recommends legislation to mandate employing such 

safeguards, including significant penalties for violating them. Although I am 

confident that it is possible to de-identify individual records while making 

them accessible for appropriate research, I know this is not a simple matter. 

These techniques need to be described in a way that builds confidence in the 

appropriate use of administrative data, and the public needs to see evidence 

that the research uses of data are beneficial at the same time that privacy 

rights are fully protected. 

In making this recommendation I note that experience demonstrates that 

there will always be some potential for a breach of security in any database, 

public or private. But not collecting and using data is an unacceptable 

remedy. We simply need to constantly improve the tools for managing the 

problem. One tool is not collecting more data at high governmental levels 

than are needed to serve the legitimate purposes of government at that level. 

4. Acknowledges that complexity, variability, and human agency pose challenges 

for policy and practice that evidence may address, but can never entirely 

overcome. Experience suggests it is generally impossible to obtain conclusive 

evidence about the effectiveness of any single policy or programmatic 

intervention across varying situations and circumstances. Evidence-based 

policy and practice should be conducted with humility, and programs or 

The Promise of Evidence-Based Policymaking 1917
policies should not be required to meet impossible standards of “proven 

effectiveness.” At the same time, programs and policies should be required to 

collect evidence of effectiveness and use it for continuing improvement. 

The difficulty the federal government has had in implementing its own 

policies for improving regulation (which are elaborated in the comment from 

the George Washington University Regulatory Studies Center) illustrates the 

challenges of evidence-based policy. The suggestions of the Regulatory 

Studies Center for addressing these problems reflect a good understanding of 

their complexity, without, in my view, offering a fully satisfactory solution. 

Ironically federal efforts to improve the effectiveness of regulation seem 

often to result in more red tape that further reduces the agility and adaptive 

capabilities of government. 

Some people suggest that the difficulty of assuring the effectiveness of 

governmental policies and programs is, in itself, an argument against 

governmental efforts to improve the lives of citizens. While I am quite aware of 

the difficulties, the contributions of the state and federal governments to the 

quality of life in our country refutes this claim. The strength of the United States 

among the nations of the world is the result of freedom combined with the 

efforts of the national, state, and local governments to create through law and 

policy the conditions necessary for opportunity, fairness, safety, and security. 

We simply need to recognize that sophisticated, multi-faceted approaches (with 

checks and balances) have always been necessary for dealing with complex 

problems. 

Thank you for inviting comment. 

 

 

1918 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 09, 2016

Status: Posted

Posted: December 09, 2016

Tracking No. 1k0-8thu-3035

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0236

Comment on FR Doc # 2016-22002

Submitter Information

Name: Jonathan Mintz CEO, CFE Fund

Address:

New York City, NY, 10005

Email: jmintz@cfefund.org

General Comment

The Cities for Financial Empowerment (CFE) Fund's mission is to leverage municipal

engagement to improve the financial stability of low and moderate income households

by embedding financial empowerment strategies into local government infrastructure. To

this end, the CFE Fund provides grants and technical assistance to local governments

and their partners. Simply put, evidence of effectiveness is critical to local governments:

they need to allocate limited resources, their constituents hold them accountable for the

quality of services, and local programs seek sustainability across changes in

administration.

Please see attached document for the remainder of the comment.

Attachments

CFE Fund re Evidence Based Policy

The Promise of Evidence-Based Policymaking 1919
1920 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 1921
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 09, 2016

Status: Posted

Posted: December 09, 2016

Tracking No. 1k0-8thx-nvc0

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0237

Comment on FR Doc # 2016-22002

Submitter Information

Name: Tom Gallagher

General Comment

See attached file(s)

Attachments

Comments_Docket_160907825-6825-01

1922 The Promise of Evidence-Based Policymaking
State of Wyoming

Department of Workforce Services

Research & Planning

P.O. Box 2760, Casper, WY 82602

(307) 473-3807

http://doe.state.wy.us/LMI

Matthew H. Mead

Governor

John Cox

Director

Lisa M. Osvold

Deputy Director

December 9, 2016

Tom Gallagher

PO Box 2760

Casper, WY 82602

307-473-3801

tom.gallagher@wyo.gov

Subject: Response to the “Request for Comments for the Commission on Evidence-Based Policy 

Making,” Department of Commerce [Docket Number 160907825-6825-01] published in the Federal 

Register Vol. 81, No. 178, September 14, 2016 pages 63166-63168.

Labor Market Information (LMI) offices within State Workforce Agencies are major producers of high 

quality data and analysis in the nation-wide US Department of Labor (DOL) state/federal Bureau of Labor 

Statistics (BLS) and Employment and Training Administration (ETA) Workforce Information Grant 

(WIG) funded systems. State LMI offices use funding from ETA to produce industry and occupational 

projections, the principle source of occupational in-demand information intended to drive employment

and training programs under the Workforce Innovation and Opportunity Act (WIOA). By seeking to 

amend the Wagner-Peyser Act to add LMI office consultation with the providers of services under the 

Carl D. Perkins Career and Technical Education Act, Congress also assumes that in-demand occupational 

information will be available to support employer need and career guidance requirements. Current reauthorizing Perkins language (H.R. 5587) includes appropriations of over $1.1 billion in Fiscal Years 

2017 and 2018, climbing to $1.2 billion by FY2022. BLS resources sustain the statistical building blocks 

for state workforce LMI offices to produce most of the market demand information, or “evidence” used in 

guiding career choices for students, dislocated workers, returning veterans, and others as well as by those 

designing education and training curriculum. They achieve this goal, in part, through the use of 

administrative records from the Unemployment Insurance (UI) tax system.

A real barrier to these core data has been stagnating federal funding for these data collection and analysis 

activities through both BLS and ETA. States can use statewide set-aside funds under certain WIOA 

grants, but these and other major sources of funding for workforce programs have also stagnated, and 

competition for those funds is fierce. 

State workforce agencies are typically comprised of Employment and Training (E&T), UI, and LMI 

offices. Located within these offices are E&T performance and UI quality control measures and 

sometimes, research and evaluation functions. (Frequently, other workforce services are also included in 

state workforce agencies.) Most state workforce agencies contain the WIOA administrative services for 

single state area delivery systems, or for sub-state self-governing workforce boards. 

The purpose of this communication is to address the Commission’s first question; “Are there successful 

frameworks, policies, practices, and methods to overcome challenges related to evidence building ... the 

Commission should consider when developing findings and recommendations regarding Federal 

evidence-based policy making?”

The Promise of Evidence-Based Policymaking 1923
The answer is that the state/federal cooperative statistical programs operated by the BLS have served the 

country and the states well for the past several decades. Cooperative agreements are designed to meet the 

mutual needs of the states and federal government and have done so since the mid-1980s. The system is 

known for its integrity and continuity of purpose. As a federal statistical agency, BLS attempts to meet 

required statistical standards and practices which include investing in state training and education, 

maintenance and updating program documentation, establishing security and confidentiality standards, 

and ensuring timely publication of statistics which hold the public sector accountable for its policies and 

actions while keeping the public informed about the outcomes of private sector market forces.

Slowness in adapting to change, recognize opportunities where major gains could be made for marginal 

investment, capitalize on state innovation, and the failure to fund state statistical infrastructure are chief 

among its failings. Funding for the states has remained basically unchanged with Federal Unemployment 

Insurance Tax Act appropriations remaining at $65 million for the past four years. A result is that BLS 

uses funds for state operations to modernize the IT system to maintain and upgrade cooperative agreement 

statistical programs. 

Cooperative agreements work well to the extent that they are adequately funded and the states have the 

authority and organizational acumen to maintain a balance in the establishment of goals for the system.

ETA’s WIG lacks the federal bureaucratic restrictions associated with a federal statistical agency. On the 

other hand, it creates the opportunity and provides the flexibility for innovation in state LMI operations in 

their attempts to meet state and local need for “in-demand” information. LMI offices accomplish this task 

through the use of products from the BLS cooperative programs such as the Occupational Employment 

Statistics estimates of employment and wages by occupation. However, while the demand for information 

grows, for example through programs to re-employ unemployment insurance claimants, WIG grant funds 

have remained at $32 million for the states during the past ten years.

In addition to nominal funding insufficient to meeting the bare minimum of state and local customer 

expectations, is ETA’s lack of investment in infrastructure such as training for the E&T community in the 

use of labor market information. As a non-statistical agency, ETA lacks credible oversight and assistance 

to the states. Moreover, it has yet to demonstrate the capacity to integrate such program activity as WIOA 

program evaluation, which incorporates the role of understanding how the labor market works with such 

programs as the Workforce Data Quality Initiative, whose purpose is to support quasi-experimental design 

(in addition to producing descriptive statistics about the market) in WIOA program evaluation.

Despite the fact that both the cooperative agreement and WIG are programs administered by DOL, DOL 

exhibits little evidence of intentionally fostering a synergy between the two programs at either the state or 

federal level. With so many programs federally funded, but state administered, the Commission may want 

to explore implementing accountability systems that require state reporting directly to Congress on the 

performance of federal funding authorities.

Sincerely, 

Tom Gallagher

Manager, Research & Planning

1924 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 10, 2016

Status: Posted

Posted: December 12, 2016

Tracking No. 1k0-8tij-axmu

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0238

Comment on FR Doc # 2016-22002

Submitter Information

Name: Jane Carrell

Address:

Roscoe, 61073

Email: roscoeguys@gmail.com

General Comment

I agree completely with Comment ID: USBC-2016-0003-0233

I could not express my concerns more perfectly. It is not the business of the federal

government to compile records on its citizens, cradle to grave, or even higher education

to employer.

The Office of Personnel Management hack, reported in July 2015, which compromised

records of 22.1 million government employees and contractors, including security

clearance files, is illustrative of the problem.

https://www.washingtonpost.com/world/national-security/chinese-hack-of-governmentnetwork-compromises-security-clearance-files/2015/06/12/9f91f146-1135-11e5-9726-

49d6fa26a8c6_story.html

The Promise of Evidence-Based Policymaking 1925
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 10, 2016

Status: Posted

Posted: December 12, 2016

Tracking No. 1k0-8tik-5p66

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0239

Comment on FR Doc # 2016-22002

Submitter Information

Name: Gerald Olson

General Comment

I am opposed to the additional information that this would gather on citizens which

would only be an additional unjustified intrusion into the lives of Americans

1926 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 10, 2016

Status: Posted

Posted: December 12, 2016

Tracking No. 1k0-8tik-pvgk

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0240

Comment on FR Doc # 2016-22002

Submitter Information

Name: Cheryl Hartman

General Comment

I oppose the tracking aka womb to tomb tracking. Thanks you.

The Promise of Evidence-Based Policymaking 1927
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 10, 2016

Status: Posted

Posted: December 12, 2016

Tracking No. 1k0-8tiq-mqdt

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0241

Comment on FR Doc # 2016-22002

Submitter Information

Name: Ann Thanawalla

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all students, commonly

referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008, the reasons against creating it

have only become more compelling.

In the past few years, much highly personal data held by federal agencies has been

hacked, including the release of the records of the Office of Personnel Management

involving more than 22 million individuals, not only federal employees and contractors

but also their families and friends.

The US Department of Education has been found to have especially weak security

1928 The Promise of Evidence-Based Policymaking
standards in its collection and storage of student data, and received a grade of

Devastating "D" for its security protections. What part of human safety and security

matters to this commission? 

In addition, preK-12 student data currently collected by state departments of education

that would potentially be shared with the federal government include upwards of 700

highly sensitive personal data elements, including students' immigrant status, disabilities,

disciplinary records, and homelessness data. 

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from PreK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation. This

data is already being combined with city data via MOU's and city grants which conceal

the third party access of aggregated data before the so-called "research data" funnels into

the university stream. The cracks are wide and the cracks are deep. Data is also being

shared with other countries, deepening the need for more stringent control, not less.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban in

the report you provide to Congress.

Yours,

Chair of the Santa Monica Privacy Coalition, Santa Monica CA

The Promise of Evidence-Based Policymaking 1929
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 13, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tkl-9e54

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0242

Comment on FR Doc # 2016-22002

Submitter Information

Name: Bryce Ebeling

Address:

Oceanside, CA, 92056

Email: bebeling@gmail.com

General Comment

I believe in informed consent and oppose non-consensual data mining. Stop the mining

of our kids personal data NOW and forever.

1930 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 13, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tkl-6gf6

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0243

Comment on FR Doc # 2016-22002

Submitter Information

Name: Thomas Ultican

General Comment

I believe in informed consent. I oppose non-consensual data mining. Stop this madness.

This looks like the road to tyranny.

The Promise of Evidence-Based Policymaking 1931
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 13, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tkl-azuk

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0244

Comment on FR Doc # 2016-22002

Submitter Information

Name: Julie Behunin

General Comment

You are treading on sacred ground. You do not have the right or consent to continue

obtaining information from minors, the parents need to give explicit consent. I am emailing Mike Lee and requesting his involvement in disbanding the CCE, as well as our

President elect.

1932 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 13, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tkm-odjn

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0245

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

I believe in informed consent. I oppose non-consensual data mining. Stop this immoral,

unethical behavior.

The Promise of Evidence-Based Policymaking 1933
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 13, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tkn-fvca

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0246

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

I believe in informed consent. I oppose non-consensual data mining.Stop this madness.

1934 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 13, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tkn-bic7

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0247

Comment on FR Doc # 2016-22002

Submitter Information

Name: Katherine Evans

General Comment

Comment by Katherine Smith Evans, Executive Director,

Council of Professional Associations on Federal Statistics

to the 

Commission on Evidence Based policymaking

Disclaimer: This set of comments reflects my views and experience, but does not

necessarily represent the views or opinions of the Members of the Council of

Professional Associations on Federal Statistics (COPAFS). 

The comments in the uploaded file address the infrastructural, administrative and

governance aspects of the establishment of a national clearing house for federal

administrative data. It is informed by my 5 years of experience as the Administrator of

one of the 13 Principal Statistical Agencies of the U.S. (the Economic Research Service

of USDA), my administration of research programs that evaluated social programs in

food assistance (SNAP, WIC, school lunch) and years of personal research and program

evaluation utilizing administrative data from USDA program agencies, including

objective evaluation of the Conservation Reserve Program, Farmland Protection

Program, and aspects of USDA water quality programs.

The Promise of Evidence-Based Policymaking 1935
A main point is that the construct, administration, and governance of a clearinghouse for

researcher access to federal administrative data may best be managed by a partnership

among academic institutions, the federal government, and state and private owners of

administrative data related to federal programs. It is further suggested that the locus of

operations of a clearinghouse could be a university or universities, and that brick and

mortar facilities need not be the only alternative for private, highly secure, and

supervised access to confidential administrative data.

Details are given in the attached file

Attachments

Comment by Katherine Smith Evans to CEP

1936 The Promise of Evidence-Based Policymaking
Comment by Katherine Smith Evans, Executive Director, 

Council of Professional Associations on Federal Statistics 

to the 

Commission on Evidence Based policymaking 

Disclaimer: This set of comments reflects my views and experience, but does not necessarily represent 

the views or opinions of the Members of the Council of Professional Associations on Federal Statistics 

(COPAFS). 

These comments address the infrastructural, administrative and governance aspects of the 

establishment of a national clearing house for federal administrative data. It is informed by my 5 years 

of experience as the Administrator of one of the 13 Principal Statistical Agencies of the U.S. (the 

Economic Research Service of USDA), my administration of research programs that evaluated social 

programs in food assistance (SNAP, WIC, school lunch) and years of personal research and program 

evaluation utilizing administrative data from USDA program agencies, including objective evaluation of 

the Conservation Reserve Program, Farmland Protection Program, and aspects of USDA water quality 

programs. 

A main point is that the construct, administration, and governance of a clearinghouse for researcher 

access to federal administrative data may best be managed by a partnership among academic 

institutions, the federal government, and state and private owners of administrative data related to 

federal programs. It is further suggested that the locus of operations of a clearinghouse could be a 

university or universities, and that brick and mortar facilities need not be the only alternative for private, 

highly secure, and supervised access to confidential administrative data. 

1. Challenges to federal agency management and operation of a clearinghouse of data we expect to 

grow over time include the following: 

• Uncertain funding of federal agencies makes long-term planning and planned growth difficult. 

Statistical agencies of the U.S. have not received appropriations consistent with their 

programmatic needs. Over the last 5 fiscal year, statistical agencies have generally experienced 

stagnant or declining inflation-adjusted appropriations, even as: 

o New data collections are legislated (as in the case of the Bureau of Transportation 

Statistics); 

o Costs for the collection, processing, storage, protection, and dissemination of data and 

statistics rise with inflation; 

o Agencies budgets are taxed, heavily and without recourse, for support of centralized IT 

systems; and 

o Some Departments substantially tax agencies to support Departmental administrative 

functions. 

Furthermore, budget actions such as “sequestration” i

 have cut programs indiscriminately for 

the purpose of overall deficit reduction. 

• Static and declining funding forces decisions about what programs to cut (unless an 

administrator jeopardizes the quality of all programs by instituting across-the-board cuts). As a 

rule, statistical agencies must first protect all programs mandated by law. Second and critical 

priorities are programs that support decision making (such as rate-setting) for other federal 

government units. Third priorities are those that further the stated mission of the agency but 

The Promise of Evidence-Based Policymaking 1937
are not mandated and do not support mandated activity. By necessity, at the bottom of the list 

for program maintenance are voluntary programs that are not directly supportive of an agency’s 

mission. Under current law, no agency is mandated to provide confidential access to 

administrative data that it does not generate.

• All federal agencies are subject to laws that can impede the efficiency of data operations and 

could potentially, under future circumstances, lead to breaches in confidentiality. Examples 

include the Federal Information Technology Acquisition and Reform Act, which gives Chief 

Information Officers at Cabinet Department levels to prescribe or approve IT hardware or 

software licensing. Although statistical agencies are accommodated by the law, experience to 

date is that the law can substantially lengthen and complicate a variety of operational 

processes. The Cybersecurity Act of 2015 requires all agencies to use malware protection 

software operated by the Department of Homeland Security (DHS). The law mandates that all 

internet traffic into and out of agencies be monitored by DHS. It also states that the only 

purpose for DHS access is for cybersecurity, but there is no specified penalty for failure to limit 

DHS use to cybersecurity,

2. Inadequate involvement of data users in data clearinghouse governance can mean that program 

evaluation partnership opportunities are overlooked. Many agencies do work closely with advisory 

committees that provide valuable input to agency decision making. But there is a large difference 

between giving advice and being involved directly in such processes as assurance of accountability, 

prioritization, allocation of funding, and strategic planning. 

3. There are successful, alternative models for administrative data access and linkage for research and 

program evaluation purposes. For example: 

• The Administrative Data Research Network (ADRN) of the United Kingdom 

(https://adrn.ac.uk/about/general-information/what-we-do/) is a group of universities and 

national statistics agencies spread throughout the UK, that work together to help researchers 

get access to de-identified, linked administrative data. It is funded via a government agency, the 

Economic and Social Research Council, which is the UK's largest organization for funding 

research on economic and social issues. It is governed by a Board of Directors that is chaired by 

the UK Statistics Authority and includes representatives of partnering universities, government 

departments and agencies, funders and the wider research community. The ADRN’s 

Administrative Data Service: helps researchers prepare their projects for assessment by an 

Approvals Panel; processes all applications and provides project support; trains researchers to 

use administrative data safely, legally and responsibly; negotiates with data custodians, such as 

government departments and agencies, for access to administrative data; assures standards of 

data security to protect privacy; and informs the public about the social benefits of the research 

it facilitates. 

• ICPSR is an international consortium of more than 750 academic institutions and research 

organizations, that provides leadership and training in data access, curation, and methods of 

analysis for the social science research community. ICPRS maintains a data archive of more than 

250,000 files of research in the social and behavioral sciences as well as themed sets of 

statistical data from federal agencies and other sources. The Consortium receives funding from 

1938 The Promise of Evidence-Based Policymaking
both public and private sources and is governed by elected representatives from among its 

members. 

4. Private interests can provide equivalent or superior service to public agencies with respect to data 

privacy and the confidentiality of respondents. For example, NORC at the University of Chicago 

manages a data enclave that a number of federal agencies use to allow secure and remote access to 

confidential data. If one envisions private administrative data sets (for instance, John Deere’s 

proprietary precision agricultural data that could productively be linked with federal conservation 

program administrative data), being included in an administrative data clearinghouse, it is 

conceivable that private businesses could view private data services as more secure than federally 

operated services. This is especially true given uncertainties about a uniform, DHS-operated 

cybersecurity system. 

These comments should not be read as a condemnation of federal administrative data access 

management. Indeed, the federal data research center system and individual agencies have a 

magnificent history of maintaining data privacy and confidentiality. And many agencies are responsive 

to the needs of those who would use administrative data for research and program evaluation purposes. 

I do, however, think it is important to acknowledge that alternative models for a secure and highly 

functional administrative data clearing house can be imagined. 

 

i

Budget sequestration is a procedure in U.S. law that limits the size of the federal budget. It involves the 

Congressional Budget Committee setting a hard cap on the amount of government spending within broadly 

defined categories; if Congressional Appropriations Committees enacts annual appropriations that exceed these 

caps, an across-the-board spending cut is automatically imposed on these categories, affecting all departments 

and programs by an equal percentage. 

The Promise of Evidence-Based Policymaking 1939
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 13, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tkn-mbmg

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0248

Comment on FR Doc # 2016-22002

Submitter Information

Name: Pamela Budge

General Comment

4. What data-sharing infrastructure should be used to facilitate data merging, linking, and

access for research, evaluation, and analysis purposes?

ANSWER: Data should NOT be merged or linked or accessed unless individual

informed consent is expressly given from people whose personal data will be used for

research, evaluation, or analysis purposes. 

5. What challenges currently exist in linking state and local data to federal data?

ANSWER: State and Local data should NOT be linked to Federal data.

6. Should a single or multiple clearinghouse(s) for administrative and survey data be

established to improve evidence-based policymaking? 

ANSWER: None.

7. What data should be included in a potential U.S. government data clearinghouse(s)?

ANSWER: None.

1940 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 13, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tkn-s6in

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0249

Comment on FR Doc # 2016-22002

Submitter Information

Name: Kathy Hsu

General Comment

Please kindly find attached comments from Socrata. Thank you !

Attachments

Socrata Response _ Commision on Evidence Based Policymaking

Socrata Response _ Commision on Evidence Based Policymaking

The Promise of Evidence-Based Policymaking 1941
Bureau of the Census (USBC)

Proposal for Commission on Evidence-Based Policymaking 

Submitted To: 

Bureau of the Census (USBC)

c/o Nick Hart, Policy and Research Director for the Commission on Evidence-Based Policymaking, 

U.S. Census Bureau

4600 Silver Hill Road

Washington, DC 20233

Email: nicholas.r.hart@census.gov

Submitted By: 

Kathy Hsu, Federal Account Consultant 

705 5th Ave S #600, 

Seattle, WA 98104

Phone: (925) 719-9707

Email: Kathy.hsu@socrata.com

1942 The Promise of Evidence-Based Policymaking
 © Copyright 2016 1

Executive Summary 

The Commission on Evidence-Based Policymaking (i.e., the Commission)’s authorizing legislation directs the 

Commission to consider how administrative and survey data “may be integrated and made available to facilitate 

program evaluation, continuous improvement, policy-relevant research, and cost-benefit analyses by qualified 

researchers and institutions.” Through this directive, the Commission is exploring ways in which to increase access 

to, and transparency of, government data to foster greater awareness and understanding by the general public, and to 

promote more informed policy making overall.

In the pages that follow, we outline our thoughts and recommendations in response to each question presented in the 

Bureau of the Census (USBC)’s Request for Information (RFI). We include common open data challenges facing 

agencies today, and we discuss typical best practices that can help overcome those challenges - including workflow,

data governance, the monetization of certain data sets, and the protection of sensitive data. We also offer examples 

of how our technology products - specifically Socrata’s Open Data Platform - can improve accessibility and 

transparency of the Commission’s data, by offering robust visualizations and narratives to help users more clearly 

understand the data and make educated decisions based upon that information. 

Socrata’s suite of open data products provides administrators and end users with:

ޮ A world class data discovery experience. We firmly believe that users should be able to find the data they 

are looking for easily, regardless of their technical sophistication and familiarity with robust datasets. 

Across each initiative we engage in, our team ensures that accessibility and ease-of-use are of top-most 

priority.

ޮ A Collaboration Environment. The Socrata Data Platform facilitates collaboration around data, through 

commenting, sharing, and regular updates. This promotes a thriving data-marketplace that firmly connects 

data producers with data consumers.

ޮ Core Integration Capabilities. Socrata has mastered the automation of data and metadata workflows, and 

integration to pull data from a wide variety of systems, and programmatically distribute it to tools and 

software for using data. The ability to combine many shapes and sources of data into a central platform will 

be a main predictor of the Commission’s ability to use data intuitively.

ޮ Continuously Improving Cloud-based Delivery Model. Our data analysts and developers are constantly 

creating and adding new and improved capabilities, and our technologies today go far beyond what they 

were as recently as 12 months ago. That trend will continue, meaning our customers continuously benefit 

from a future-proof infrastructure that continues to evolve as technology improvements open up new 

innovation possibilities in the future.

Technology, People, and Process to Harness the Power of the Commission’s Data 

Since our founding, Socrata has worked with over 1,300 governments, NGOs, non-profits, and multilaterals 

organizations globally. This includes virtually all major cities in the United States, dozens of counties, roughly half 

of all states, 19 Federal customers, and multiple international groups. 

Our team provides a unique combination of capabilities and expertise for accessing and aggregating data across 

disparate sources inside the enterprise. We also excel at making the data discoverable and understandable to a wide 

The Promise of Evidence-Based Policymaking 1943
 © Copyright 2016 2

range of different user types, via downloadable export options, data visualizations, dashboards, and tailored data 

exploration experiences. 

Socrata is the nation’s leading government data platform in number of active deployments, revenue and dedicated 

engineers, customer success managers, apps, and innovators.

Our mission statement is unleashing the power of government data to improve society, and we work to achieve this 

by combining smart technology, people, and processes:

ޮ Technology. Socrata provides a software as a service (Saas) platform powered by the industry leading 

Amazon AWS infrastructure. Our product is Commercial Off-the-Shelf (COTS) software, meaning there is 

no custom development or software consulting deliverables required to implement or maintain the solution. 

ޮ People & Process. Our team has outstanding hands-on experience planning, implementing and executing 

public sector open data and performance management initiatives. Our staff members include former 

government performance analysts, program managers, data analysts, and systems engineers - all of whom 

work exclusively on local government projects. Socrata brings best practices in both Open Data and 

Performance Indicators Management for government. And, we utilize a proven, Seven Step Implementation 

Methodology to ensure each project is designed around repeatable industry best practices.

We look forward to discussing this important initiative in greater detail with your team, and we’d welcome an 

opportunity to demonstrate the flexibility of our solution.

Sincerely, 

Kathy Hsu, Federal Account Consultant

Phone: (925) 719-9707

Email: Kathy.hsu@socrata.com

Overarching Questions

The following sections includes Socrata’s feedback on all questions outlined in USBC’s RFI. Wherever possible, we 

have provided specific examples of where our technology products can help the Commission achieve its survey and 

administrative requirements, and we’ve illustrated specific projects on which we’ve achieved similar results for our 

customers. 

1944 The Promise of Evidence-Based Policymaking
 © Copyright 2016 3

1. Are there successful frameworks, policies, practices, and methods to overcome challenges 

related to evidence-building from state, local, and/or international governments the Commission 

should consider when developing findings and recommendations regarding Federal evidencebased policymaking? If so, please describe.

The Commission should consider taking a look at the Pew MacArthur Results First Initiative, which is a project of 

Pew Charitable Trusts and the John D. and Catherine T. MacArthur Foundation. The initiative works with states and 

localities to develop the tools policymakers need to invest in programs that will likely yield high impact. In working 

with 20 states and six (6) counties, the initiative is focused on: 1) creating an inventory of currently funded programs 

in a particular government, 2) reviewing programs which work, 3) conducting cost-benefit analysis to compare 

programs’ likely return on investment, and 4) using evidence to inform spending and policy decisions. The initiative 

also comprises the Results First Clearinghouse Database, which is one-stop online resource that provides 

policymakers with an easy way to find information on the effectiveness of a variety of interventions.

Socrata has had a number of discussions with Pew recently given that a few of our customers, most notably 

Montgomery County, MD, are focused on evidence-based policymaking as well as data publishing and data 

storytelling. The Commission should consider the policies and practices of leveraging technology to support this 

work, for everything from data collection, to data governance, to data quality, and ultimately data-driven decision 

making and communication of results. We dive into each of these aspects in greater detail within Questions 3 and 

13.

2. Based on identified best practices and existing examples, what factors should be considered in 

reasonably ensuring the security and privacy of administrative and survey data?

Centralized access to administrative and survey data is critical to promoting evidence-based policymaking. 

However, it is equally crucial that whomever you select as your partner is equipped to ensure the security and 

privacy of that data as it is published for researchers, institutions, and the general public. 

Socrata follows industry best practices for ensuring the security and privacy of data that we host on our platforms, 

including: 

ޮ Hosting data in a secure system that offers role based access control - i.e., only those on a “need to know” 

basis are given access;

ޮ Establishing data governance and lifecycle practices with customers like the Centers for Disease Control, 

UN Capital Development Fund, and other organizations that ensure data conforms to federal law around 

personally identifiable information 

ޮ Creating views on top of private data that can be shared with wider audiences, yet hide information that is 

meant to be internal only;

ޮ Maintaining physical data security measures, which includes replicating data between three different AWS 

Availability Zones, in each of two geographically separated AWS data centers. Each zone can power the 

entire Socrata platform, as well as replicate the database write-ahead logs in near-real-time and take full 

backups every three days. Backups are retained for 30 days, and Socrata has an RTO target of 96 hours in 

the event of catastrophic failure of the AWS environment. 

Furthermore, Socrata fully complies with FedRAMP regulations for responsible disclosure and notification, in the 

case of breach, leakage, or security incident, and we maintain an Information Security Contingency Plan that covers 

business continuity, as well as incident response procedures. This plan is reviewed and tested annually, and the plan 

and test results can be made available upon request. We maintain a FISMA-Low ATO, and we inherit our physical 

security measures from AWS. In addition, Socrata is currently in process to receive FedRAMP certification. We 

The Promise of Evidence-Based Policymaking 1945
 © Copyright 2016 4

have completed our audit and are working with a cabinet-level agency to finalize the process and achieve our 

Authority To Operate (ATO). To see the latest on our current status, please visit the Socrata CSP page on the 

FedRAMP website.

Data Infrastructure and Access

3. Based on identified best practices and existing examples, how should existing government 

data infrastructure be modified to best facilitate use of and access to administrative and survey 

data?

One of the biggest transformations in modern government has been the pivot to data transparency and efficiency. 

Traditional data was often siloed - locked up within internal government networks and servers, making highly 

valuable information inaccessible to other departments, researchers, policy makers, and public users who could 

benefit from it most. The move towards data visibility has led to sweeping improvements across the government, 

providing the public and key stakeholders with instant access to critical data and performance measures that can 

guide more informed decision making. 

Foundational and policy-specific evidence are vital to the creation of specific programs and legislation, and this 

evidence relies heavily on survey and administrative data, including: large-scale administrative data; programspecific administrative data; general purpose survey data; and study-specific survey data. 

Best Practices for Opening Survey Data 

Because of survey complexity, often this data remains in formats that are inaccessible or unusable to the general 

public. Existing government data infrastructures can be modified to facilitate access to, and sharing of, 

administrative and survey data by: 

ޮ Uploading it to a centralized platform that enables easy sharing among internal teams and external users, as 

needed;

ޮ Ensuring data is captured in a machine-readable, API-enabled format;

ޮ Increasing data accessibility through easy-to-use search features, and proper metadata classification for 

filtering;

ޮ Promoting scalability - making it easy to build on top of the display data in a visual, easy-to-understand 

format. 

Socrata Open Data Platform for Optimized Transparency & Accessibility 

Socrata offers a unique combination of capabilities and expertise for accessing and aggregating data across disparate 

sources. On every data initiative we engage in with our clients, our focus is on making data discoverable and 

understandable to a wide range of different user types. We do this through sophisticated data visualizations, easy-touse dashboards, contextual narratives, flexible download and export options, and highly-tailored exploration 

experiences.

The Socrata Data Platform is a cloud-based, turnkey, software-as-a-service (SaaS) solution designed from the 

ground up to simplify the process of data publishing and data access. 

1946 The Promise of Evidence-Based Policymaking
 © Copyright 2016 5

High-level Socrata Data Platform architecture.

It empowers end users with tools for ad-hoc data consumption, exploration and visualization, and provides 

capabilities that enable the sharing of that information across multiple channels. The platform is delivered as a web 

portal that allows users to:

ޮ Perform robust searches for data and information.

ޮ Interact with that data by sorting, filtering, and performing group-by and roll-up operations.

ޮ Create real-time visualizations on top of the data with charts, graphs and maps and save these visualizations 

to user profiles.

ޮ Share data and saved views in context rich reports, dashboards and stories to extend its reach and utility.

ޮ Integrate, extract and cleanse a variety of shapes and sizes of source system data in real time and present 

that information in a consumption formats.

ޮ Empower advanced users and developers with machine readable data, APIs and tools to create 

sophisticated services, robust interfaces and dynamic mobile applications.

For USBC, this platform would offer a number of important benefits across the data lifecycle, including: acquiring 

data; preparing the data; providing data; and monitoring usage and outputs. We see these benefits as including: 

ޮ A world class data discovery experience. Socrata’s world class data discovery experience would allow 

USBC users to find the data they are looking for easily, regardless of their technical sophistication and 

familiarity with robust survey results and datasets. 

ޮ A Collaboration Environment. The Socrata Data Platform facilitates collaboration around data, through 

commenting, sharing, and regular updates. It promotes a thriving data marketplace that quickly connects 

data producers with data consumers to foster informed action as a result of that data. 

ޮ Core Integration Capabilities. Socrata’s products allow for the automation of data and metadata 

workflows, including the integration of data from a wide variety of systems.

ޮ Continuously Improving Cloud-based Delivery Model. New and improved capabilities and features are 

constantly being added to the Socrata suite of products, which means USBC administrators would benefit 

The Promise of Evidence-Based Policymaking 1947
 © Copyright 2016 6

from a future-proof infrastructure that continues to evolve as technology improvements open up new 

innovation possibilities in the future.

Socrata Work Samples Relevant to USBC’s Data Initiative 

Socrata’s work spans federal, state, and local government projects, and we have highlighted a specific example 

below from the CDC - which centers on the hosting and administration of survey data: 

CDC’s Chronic Disease and Health Promotion Data & Indicators 

Increasing and improving access to CDC’s chronic disease and health promotion data and indicators is a key 

component of addressing the burden of chronic diseases in the United States. Chronic diseases are responsible for 7 

of 10 deaths each year, and treating people with chronic diseases accounts for 86% of our nation’s health care costs.

In January 2015, the National Center for Chronic Disease Prevention and Health Promotion at the CDC 

launched Chronic Disease and Health Promotion Data & Indicators that aims to enable quick access to data and 

indicators across health areas such as tobacco use, reproductive health, oral health, cancer, legislative actions, and

behavioral risk factors – to name only a few.

As an early adopter in the Center, the Office on Smoking and Health migrated the current and historical state-level 

data on tobacco use prevention and control housed in the State Tobacco Activities Tracking and Evaluation 

(STATE) System to the new data Tobacco Use Portal and opened the door to using a new technology portfolio to 

reinvent the STATE System for 2015.

Now in Socrata, users such as data analysts, policy staff, and researchers can quickly view data in colorful charts, 

graphs and maps, create custom filters to subset data, download data in a variety of formats, and even receive 

updates when data of interest has been updated. The featured dataset below displays a map of US states with 

comprehensive smokefree air laws, which protect the public from exposure to secondhand smoke.

4. What data-sharing infrastructure should be used to facilitate data merging, linking, and access 

for research, evaluation, and analysis purposes?

The real power of data to drive evidence-based policymaking is unleashed when researchers and policymakers can 

combine data and present it to legislators and to the public in novel ways. In order to facilitate this, data needs to be 

on a platform that allows users to easily build on top of that information to provide greater insight - going beyond 

simply presenting raw data, to offer insight into why it matters, and ensuring users understand its accuracy and 

integrity. One way the underlying data infrastructure can facilitate the merging and linking of data is to provide 

programmatic access to data through application programming interfaces (APIs). APIs allow developers to build 

tools that extract data from multiple datasets or other web services and merge them using the attributes of the data 

itself, or more complex methodologies. For researchers who may not be developers or as technically sophisticated, 

APIs allow them to feed data into commonly used statistics and visualization tools, like Tableau and SAS. 

The Socrata Open Data API provides an open, standards-based, RESTful application programming interface for 

datasets. Any dataset uploaded to the Socrata platform is automatically provided with a RESTful JSON endpoint, 

allowing developers to leverage data within the platform through a familiar and powerful API built on open 

standards. The Socrata platform also has libraries for development languages like R, which are commonly used in 

the statistics community. 

Socrata’s open APIs give users the power to:

1948 The Promise of Evidence-Based Policymaking
 © Copyright 2016 7

ޮ Upload data in a central platform, allowing users to build on top of it;

ޮ Merge and link data through visualizations;

ޮ Combine data with APIs and and build custom tools on top of it;

ޮ Take into account the fact that governments cannot account for every single use of data, thereby leveraging 

APIs to give end users flexibility to build the apps they need;

ޮ Make data accessible via endpoints, allowing common visualization and analysis tools to be used;

ޮ Unlock machine processing of datasets; and

ޮ Empower developers to combine datasets with other web services for new mashups and applications.

5. What challenges currently exist in linking state and local data to federal data? Are there 

successful instances where these challenges have been addressed?

Data challenges that exist with inter-agency data sharing - both between federal agencies themselves, and between 

federal and state agencies - are typically self-imposed. Often, agencies at all levels have adopted formal and 

informal policies (e.g., sharing of personally identifiable information, other sensitive information, etc.) over the 

years - sometimes with limited understanding of how these policies may contribute to the difficulty around sharing 

within government or between governments. These policies are often complicated further by technical complexities, 

including differences in data schemas and inconsistencies in common data terms used. 

We recommend that the Commission lead a review of all existing policies - with strong federal, state, and local 

executive leadership - to identify archaic data sharing policies, and create an action plan to refine these policies to 

promote greater accessibility and use.

6. Should a single or multiple clearinghouse(s) for administrative and survey data be established 

to improve evidence-based policymaking? What benefits or limitations are likely to be 

encountered in either approach?

Socrata’s recommendation is to centralize all data on a single platform to ease administration and data management. 

However the decision to establish a single warehouse, or multiple clearinghouses for administrative and survey data, 

should be based on the administration model of that clearinghouse. It is likely that the owners, curators, and 

publishers of the data will be different from the consumers - and not all will have the ability to edit and/or view 

every dataset. If a clearinghouse does not support the unique administration of assets, it may be valuable to 

segregate data into multiple clearinghouses with different permission models. A data platform like Socrata that 

enables roles based access control can house all data in one, centralized clearing house and maintain administrative 

boundaries. 

The Socrata platform limits who can see and perform actions based on defined end user permissions, and depending 

on whether a dataset is recorded as “private” (i.e., internal only) or “public” (i.e., published externally for 

researchers and institutions). A role defines the types of actions that someone can perform across all datasets on the 

domain.

The Promise of Evidence-Based Policymaking 1949
 © Copyright 2016 8

Administrators on a Socrata-powered data site are able to assign any number of roles to other users, allowing for 

delegation of tasks on the site and enabling other users to create and manage datasets. Socrata users can have roles 

on a domain, which define the types of actions they can perform across all datasets on that domain, as well as highlevel operational control of the domain itself. Examples of these actions include publishing data, adding features, or 

managing users. Socrata users can also be granted specific user permissions per dataset. A permission defines the 

types of actions that someone is able to perform on a specific dataset. Examples include editing that dataset, viewing 

that dataset, and publishing that dataset. 

7. What data should be included in a potential U.S. government data clearinghouse(s)? What are 

the current legal or administrative barriers to including such data in a clearinghouse or linking 

the data?

The data that the Commission should consider including really depends on the problems it is trying to solve, and the 

related programs and policies where greater evidence building will be used to drive more impactful practices. 

Whatever data the Commission decides to include, it should be filterable and searchable in a number of ways, such 

as by broad policy area, issue area, department, data type, data source, and perhaps some type of rating as to its 

usefulness - and in the case of an intervention, whether it’s proven. Each dataset should also be API-enabled for 

greater utility across applications.

In terms of data specifically, the Commission should first consider broad themes in the areas of: economy, health, 

education, transportation, agriculture, environment, housing, and security, and then drill deeper into specific policy 

areas under each broad theme. The Commission should emphasize data that has multiple applications, can be used 

by multiple parties with few barriers, and has strong re-use value. 

8. What factors or strategies should the Commission consider for how a clearinghouse(s) could 

be self-funded? What successful examples exist for self-financing related to similar purposes?

Socrata’s recommendation for self-funding its clearinghouse is to offer full access to specific data sets for free, but 

charge users for access to more detailed reports, data visualizations, or highly-valuable analysis and narratives (e.g. 

specific benchmarking reports and analysis etc.). We work with one customer in particular - the Carbon Disclosure 

Project (CDP) - who offers free data across a number of key areas, but charges users a specific fee for access to its 

premium data sets. 

1950 The Promise of Evidence-Based Policymaking
 © Copyright 2016 9

CDP’s purchasing page for access to its premium data. 

Premium access for CDP users includes entry into a larger collection of primary environmental data, maximum 

flexibility to use its raw data, and historical data going back five years, providing greater transparency into how 

companies have evolved throughout the years. If the Commission can identify specific data that is uniquely 

valuable, offering a similar model is likely the easiest way to at least partially self-fund.

9. What specific administrative or legal barriers currently exist for accessing survey and 

administrative data?

In the White House Commission on Evidence-Based Policymaking’s white paper, Barriers to Using Administrative 

Data for Evidence - Building, it states that “Agencies need infrastructure, such as trained personnel, formal written 

procedures, and information technology tools, to consider data sharing requests and to develop and implement data 

sharing agreements” to overcome barriers for data use. Many of these barriers also exist for accessing survey and 

administrative data. In order to use data, it needs to be collected, curated, centralized, published, and shared between 

organizations so that researchers can access and use it effectively. It also requires a methodology that addresses all 

these stages in a complete and robust way. 

Our projects are typically divided into three high-level phases that include: plan, build, and handoff. Throughout 

the project the Socrata team focuses on: 

ޮ Establishing goals and success measures, and working closely with stakeholders to develop clearly defined 

(and ideally measurable) goals.

ޮ Defining the proper user experience (UX) for end users. Techniques such as wireframing and rapid 

prototyping are employed to elicit user feedback and suggestions to inform the UX design.

ޮ Working with data owners to establish a dataset on-boarding process, including: 

ߦ Surveying and identifying relevant datasets,

The Promise of Evidence-Based Policymaking 1951
 © Copyright 2016 10

ߦ Cleansing and formatting data, and

ߦ Processing metadata. 

ޮ Building and managing negotiations and trusted relationships with key data providers, with respect to 

public information, privacy, data sharing, and level of access. 

ޮ Managing roll-out and an ongoing enhancement strategy, and maintaining documentation. 

ޮ Promoting industry knowledge and leadership regarding innovation trends and emerging market shifts. 

ޮ Effectively maintaining key contacts related to survey and administrative data, community engagement, 

and data integration.

Coupling a strong, repeatable program methodology with a technology platform that removes current barriers to 

accessing survey and administrative data is key to the success of a good data program. The Socrata platform 

provides data publishers with the ability to curate metadata and any supporting documentation, and equips users 

with multi-level catalog search and filtering capabilities to find appropriate data and then make user of it.

10. How should the Commission define “qualified researchers and institutions?” To what extent 

should administrative and survey data held by government agencies be made available to 

“qualified researchers and institutions?”

Socrata has elected not to answer this question, as it is outside the scope of our expertise. Should this RFI move to a 

Request for Proposal, we will partner with a trusted company in order to supplement any areas of work that we 

cannot deliver. 

11. How might integration of administrative and survey data in a clearinghouse affect the risk of 

unintentional or unauthorized access or release of personally-identifiable information, 

confidential business information, or other identifiable records? How can identifiable 

information be best protected to ensure the privacy and confidentiality of individual or business 

data in a clearinghouse?

When dealing with sensitive data that may include personally-identifiable information or other confidential 

information, it is of critical importance to establish robust data governance and lifecycle management practices. 

Socrata’s data consultants work with organizations on a daily basis to establish data practices that ensure PII and 

confidential information is not transferred from source data systems to clearinghouses that are open to researchers 

and other public with access. 

According to best practices data is processed through an extract, transform, load (ETL) workflow that scrubs it of 

any sensitive information before it is published to a clearinghouse. Separating the data that is housed in the 

clearinghouse from the source systems prevents any accidental contamination and dissemination. 

Before data is published on the Socrata platform it also goes through an approval process. Datasets that are set to be 

published go through stages, or gates, with configurable settings for approval stages with assigned gatekeepers for 

each stage and controlled notifications. This approval queue allows individuals to approve or reject datasets and 

provides view of all the datasets currently in the publishing lifecycle. Any sensitive information that accidentally 

made it past the ETL process can be flagged by administrators before it is published on the platform.

1952 The Promise of Evidence-Based Policymaking
 © Copyright 2016 11

12. If a clearinghouse were created, what types of restrictions should be placed on the uses of 

data in the clearinghouse by “qualified researchers and institutions?”

Socrata believes data access should be restricted by user role, and by “need to know.” For example, researchers 

should only have access to data that is relevant to their project(s), and even then, only if their role on the project 

team requires that access. Adhering to this concept of “least privilege” minimizes the risk that potentially sensitive 

project data will be unduly exposed. 

13. What technological solutions from government or the private sector are relevant for 

facilitating data sharing and management?

Socrata’s suite of open data products are highly relevant for the facilitating of data sharing and data management. 

We provide multiple tools to ensure our customers’ data is provided to the right audiences at the right time, and our 

solutions center on promoting the highest levels of data discoverability, visualization and context, and sharing and 

usability: 

Data Discoverability 

Socrata’s Open Data solution provides better access to quality data and is used extensively by elected officials and 

policymakers, government employees, residents and businesses, and developers and advocates. Using Open Data 

Portal allows users to: 

ޮ Publish datasets on any topic (and these data sets can be up to hundreds of millions of rows long), 

including: geospatial data, financial data, public safety data, and much more;

ޮ Use a global search interface to find the data, and take advantage of built-in SEO optimization to enable 

data to be found via standard web browser searches;

ޮ Provide insight into your most popular datasets with a summary page that captures important information 

about your uploaded content.

Visualization and Context 

Charts and graphs would allow USBC staff to transform data into visualizations to promote better understanding of 

trends and gaps. Users can easily create and maintain customized charts, graphs, and other data visualizations for 

public data use, without having to build and maintain a data infrastructure on their own. We offer 13 different 

visualization formats available out-of-the-box, which can be dynamically applied to datasets for maximum 

flexibility in displaying critical information. These include:

Area chart | Bar chart | Bubble chart | Column chart | Donut chart | Line chart | Pie chart | Stacked bar chart | Stacked 

column chart | Timeline | & Tree map 

Furthermore, we have developed our own charting library using the D3.js toolkit, which provides a wide range of 

configuration options, dramatic colors, as well as great performance and stability. Charts will always support the full 

set of values calculated and rendered, with no limitations to scrolling across the full dataset in a chart, in order to 

display every value. They will auto-scale dynamically depending on type, and the axis placement can be dragged 

around on the page to provide the best display on your screen. Visualization legends also support conditional 

formatting. 

The Promise of Evidence-Based Policymaking 1953
 © Copyright 2016 12

Visualization options using Open Data Portal.

Socrata Perspectives augments the Open Data Portal by helping users connect the dots between policy and 

outcomes. With Socrata Perspectives, users are able to easily and quickly produce narrative reports and stories with 

drag-and-drop functionality that allows for text, images, maps, and data visualizations. The screenshots below show 

a sample dashboard, before context and data is added (left), and and after (right). 

1954 The Promise of Evidence-Based Policymaking
 © Copyright 2016 13

USBC employees can leverage a drag-and-drop toolkit of 

images, visualizations, text, video and banners to create 

compelling reports and dashboards based off of live data. 

Resulting reports pair data with text and will 

track updates as they are published from the SSP 

and EHRI Data Warehouses.

Sharing and Usability

Maximizing the value of data means making it available to as many people as possible. Socrata provides several 

tools to empower data users to easily access and utilize the data in creative ways.

ޮ Share any data asset on Socrata’s platform with others directly through the built-in data catalog or on social 

media.

ޮ Embed content within existing websites or build a customized, attractive open data homepage.

ޮ Syndicate datasets from other nearby localities to increase the context and meaning of the data.

ޮ Give technical developers, innovators, and entrepreneurs easy access to the data through the Socrata Open 

Data API (SODA).

14. What incentives may best facilitate interagency sharing of information to improve 

programmatic effectiveness and enhance data accuracy and comprehensiveness?

There are a number of ways to incentivize the facilitation of interagency data sharing, including:

ޮ Making data sharing a condition of federally-funded research;

The Promise of Evidence-Based Policymaking 1955
 © Copyright 2016 14

ޮ Working with OMB to develop a fund that will support these efforts across agencies through the budget 

process;

ޮ Setting collaborative goals for improving program performance through evidence-based practices that cut 

across mission areas and departmental silos and that require shared data to drive progress;

ޮ Rewarding researchers for releasing and sharing data through proven citation systems so they receive credit 

and exposure;

ޮ Providing unique opportunities for “data sharers” from agencies across government to discuss promising 

practices in a variety of fora with opportunities to lead a series of studies, sponsored by the Commission 

and perhaps the White House, on programmatic impact through evidence-based policymaking; and

ޮ Providing a common data platform, available to all who participate as “data sharers”, for data sharing 

governance, management, and analysis.

Data Use in Program Design, Management, Research, 

Evaluation, and Analysis

15. What barriers currently exist for using survey and administrative data to support program 

management and/or evaluation activities?

Lack of documentation and appropriate metadata are two barriers to effective data use, after that data has been 

published publicly. It is critical to augment raw data with metadata and accompanying documentation so that users 

can determine if and how that data is appropriate for them to use. As the Barriers to Using Administrative Data for 

Evidence - Building states: “Data curation also includes the maintenance of relevant metadata and other 

documentation. Prospective data users need at least minimal documentation to assess fitness for a given use, and 

more detailed documentation enables more informed and appropriate use.” A platform that allows data publishers to 

curate and publish metadata is crucial to data accessibility and understanding, and the Socrata technology stack and 

methodology makes metadata a priority. 

Socrata contains standard metadata fields and also allows for custom definition of required and optional metadata 

fields. Specific fields, drop-down menus, and other metadata entry points can be added by administrators to ensure 

that all uploaded data follows an existing schema, and is not uploaded without complete metadata entry and 

definition. 

In addition to being able to configure all metadata content and fields during upload, any number of files can be 

attached to a datasets to allow for full documentation to accompany these datasets. A data dictionary, or a guiding 

document with detailed descriptions of the dataset and its metadata can be attached in this way. Any attached 

documentation will always accompany the dataset, and administrators can also configure the metadata templates to 

ensure all data sets are documented accurately. 

1956 The Promise of Evidence-Based Policymaking
 © Copyright 2016 15

16. How can data, statistics, results of research, and findings from evaluation, be best used to 

improve policies and programs?

Socrata has demonstrated excellence in developing high-profile open data portals, and in addressing administrative, 

usability, and ongoing support needs. Our clients include federal, state, and local governments, and in the process of 

completing over 1,300 projects, we have seen how data can help organizations increase transparency, encourage 

engagement and, in turn, improve policies and program. Below are a few examples:

Centers for Medicare and Medicaid Services

As the biggest single payer for health care in the U.S., the Centers for Medicare and Medicaid Services (CMS) is 

playing a leading role in making the costs of healthcare more transparent and easier to compare across geographies 

and providers. Its sites, data.cms.gov and data.medicare.gov, share data on the costs for procedures at different 

hospitals, as well as data on quality and efficacy, such as readmission rates by facility within 30 days after 

discharge.

The release of data is a critical component of CMS’s strategy to drive improvements in the U.S. healthcare system. 

"Data transparency is a key aspect of transformation of the health care delivery system,” says CMS Administrator 

Marilyn Tavenner.

By improving the availability of data on cost and quality, CMS has catalyzed competition in a largely opaque 

market. Their open data sites empower both consumers and payers and provide information in a variety of formats 

that serve many audiences. For example, access to the raw, machine-readable data available behind any visualization 

allows sophisticated users, such as researchers and data scientists, do in-depth analysis. It also allows private sector 

providers to benchmark themselves and understand how they compare against their competitors, hospitals, insurance 

companies, and individual providers.

Under the Tavenner’s leadership, CMS has made releasing data a key part of its strategy to make the US healthcare 

system more transparent, affordable, and accountable. CMS has been releasing cost and quality data to the public 

domain since 2011.

“Data is the lifeblood of our healthcare system, and for too long it has not been optimized by its full potential,” says 

Tavenner.

City of Boston

The City of Boston had been publishing their budget in multi-volume PDFs for years. While there’s a wealth of 

information contained in these volumes, it can be difficult for the public to understand where the City is allocating 

resources and for what purposes. In 2014, Boston Mayor Marty Walsh issued an executive order to improve the 

City’s open data efforts. Three months later, the City released a web-based interactive application – Socrata’s Open 

Budget -to publish both their operating and capital budgets. Open Budget has improved the City’s financial 

transparency and given everyone – city employees, reporters and researchers, and the public – easy access to the 

City budget.

Open Budget allows the City to publish to single portal that anyone can access. Because the budget data is public 

and easy to navigate, more audiences – citizens, reporters, academia – are able to review the exact same datasets that 

the government uses.

The Promise of Evidence-Based Policymaking 1957
 © Copyright 2016 16

The City had a fairly painless migration from our original datasets to the Open Budget data schema. “We were 

lucky, our data was mostly well-formed,” said Dwelley “We had a few items that needed customization, but Socrata 

handled that for us.”

There have been immediate internal benefits from the City’s open data program. Fewer requests for information free 

up staff time and consistent reporting across departments helps with information sharing. But Boston is also excited 

about what citizens can do with the data now that it’s so easy for them to access and understand.

Code for Boston hosts a weekly hack night that encourages “developers, designers, urban planners, and data geeks” 

to use technology and publicly available data to solve community problems. The Mayor’s Office of New Urban

Mechanics taps the data to help understand problems as diverse as school transportation and improving the quality 

of Boston’s roads.

City of Seattle

The Seattle Police Department, the birthplace of bicycle patrol units, has a track record of innovation and a forwardthinking approach in its publishing of public safety data. In 2012, a federal investigation over charges of excessive 

use of force, as well as related concerns regarding officer supervision and internal investigations, resulted in an 

agreement between the Seattle Police Department and the U.S. Department of Justice. The two organizations crafted 

a reform plan to boost the SPD’s oversight, training, and reporting.

The department achieved extraordinary progress in the reform areas over the following two years, including 

pioneering the publishing of public safety data. Momentum increased in 2014 with the appointment of Chief 

Kathleen O’Toole, as well as the launch of SeaStat, the SPD public safety data portal created in partnership with 

Socrata.

The SPD joined forces with the city’s Chief Technology Officer to cosponsor the City of Seattle’s privacy initiative. 

Together, they built upon the city’s race

and social justice initiative, promoted a set of guiding privacy principles, and crafted strategies to increase 

transparency, while also protecting privacy rights of citizens. These efforts will help city departments assess privacy 

issues around open data.

17. To what extent can or should program and policy evaluation be addressed in program 

designs?

Program and policy evaluations include policy-specific evidence that allow organizations to determine whether a 

particular program or piece of legislation is achieving its objective. Socrata is a strong proponent of data-driven, 

evidence based government. When appropriate, these evaluations should include performance measures and 

indicators that can be addressed and evaluated in program design. Socrata has worked with organizations at the state, 

local, and federal level in collecting and analyzing evaluation data and presenting it to the public to drive 

transparency into program and policy effectiveness. We provide specific examples below.

18. How can or should program evaluation be incorporated into program designs? What specific 

examples demonstrate where evaluation has been successfully incorporated in program designs?

Cook County’s performance management program began in January of 2011, when the County passed an ordinance 

that required each office and agency to set goals and collect operational data around those goals. The County began 

a program to create a strategic plan around the use of open data for decision-making and communications. Andrew 

1958 The Promise of Evidence-Based Policymaking
 © Copyright 2016 17

Schwarm, Chief Performance Officer of Cook County, was the project lead tasked with finding a performance 

measurement tool.

“We faced two main challenges when we began this program,” Schwarm says. “The ordinance requires weekly 

meetings to review our goals and measure performance. There is also a reporting requirement.”

The County started by publishing reports quarterly in PDF format. “Quickly, we saw it was difficult for residents to 

dig into these dense reports and digest the information in a useful way,” Schwarm explains. “We needed a tool that 

would allow us to take the wonderful data we collect quarterly, around 750 data points over 50 agencies across the 

County, and publish it in a usable way. Performance Data Cloud (formerly known as Open Performance) jumped out 

as the solution that made the most sense for us.”

An early adopter of the GovStat platform, Cook County chose a two-phase implementation process. First, the 

County replicated the PDF data and created reports for each department. “GovStat allowed us to take the data 

already gathered and put it on a more flexible, user-friendly, open, and transparent platform,” Schwarm says.

Once the data was updated and made available to the entire organization, Cook County entered phase two of the 

implementation: using data to drive decisions. “[Data-driven decision-making] is now part of our culture and the 

way we do business,” explains Schwarm. What’s more, accurate data allows Cook County to tell accurate stories.

One early goal emerged from the Finance and Administration Department. “We started by measuring employee time 

and attendance. Using GovStat and the data we collect helped us identify policy abuse or data entry errors, reducing 

average employee sick leave per month from over seven hours in 2012 down to 4.66 hours as of December 2013,” 

Schwarm says.

GovStat also allows for the dynamic evolution and improvement of goals. “The County does environmental 

inspections of industrial facilities in the suburbs,” Schwarm explains. “Two years ago, our main goal was to make 

sure 100 percent of the facilities were inspected. After GovStat showed we were consistently meeting that goal, we 

are now able to focus on the quality and outcomes of those inspections,” Schwarm says.

Schwarm is excited to track other county initiatives, including public safety and health care goals. “We plan to roll 

out one goal per month and continue to report on its progress,” Schwarm says. He continues, “The use of good, 

timely, accurate data, especially for a government facing tough fiscal situations, allows us to make decisions and 

prioritize at a high level.”

The Promise of Evidence-Based Policymaking 1959
 © Copyright 2016 18

19. To what extent should evaluations specifically with either experimental (sometimes referred 

to as “randomized control trials”) or quasi-experimental designs be institutionalized in 

programs? What specific examples demonstrate where such institutionalization has been 

successful and what best practices exist for doing so?

Socrata has elected not to answer this question, as it is outside the scope of our expertise. Should this RFI move to a 

Request for Proposal, we will partner with a trusted company in order to supplement any areas of work that we 

cannot deliver. 

1960 The Promise of Evidence-Based Policymaking
Bureau of the Census (USBC) 

Proposal for Commission on Evidence-Based Policymaking 

Submitted To: 

Bureau of the Census (USBC) 

c/o Nick Hart, Policy and Research Director for the Commission on Evidence-Based Policymaking, 

U.S. Census Bureau 

4600 Silver Hill Road 

Washington, DC 20233 

Email: nicholas.r.hart@census.gov 

Submitted By: 

Kathy Hsu, Federal Account Consultant 

705 5th Ave S #600, 

Seattle, WA 98104 

Phone: (925) 719-9707 

Email: Kathy.hsu@socrata.com

The Promise of Evidence-Based Policymaking 1961
Executive Summary 

The Commission on Evidence-Based Policymaking (i.e., the Commission)’s authorizing legislation 

directs the Commission to consider how administrative and survey data “may be integrated and made 

available to facilitate program evaluation, continuous improvement, policy-relevant research, and 

cost-benefit analyses by qualified researchers and institutions.” Through this directive, the Commission 

is exploring ways in which to increase access to, and transparency of, government data to foster 

greater awareness and understanding by the general public, and to promote more informed policy 

making overall. 

In the pages that follow, we outline our thoughts and recommendations in response to each question 

presented in the Bureau of the Census (USBC)’s Request for Information (RFI). We include common 

open data challenges facing agencies today, and we discuss typical best practices that can help 

overcome those challenges - including workflow, data governance, the monetization of certain data 

sets, and the protection of sensitive data. We also offer examples of how our technology products - 

specifically Socrata’s Open Data Platform - can improve accessibility and transparency of the 

Commission’s data, by offering robust visualizations and narratives to help users more clearly 

understand the data and make educated decisions based upon that information. 

Socrata’s suite of open data products provides administrators and end users with: 

ޮ A world class data discovery experience. We firmly believe that users should be able to find 

the data they are looking for easily, regardless of their technical sophistication and familiarity 

with robust datasets. Across each initiative we engage in, our team ensures that accessibility 

and ease-of-use are of top-most priority. 

ޮ A Collaboration Environment. The Socrata Data Platform facilitates collaboration around 

data, through commenting, sharing, and regular updates. This promotes a thriving 

data-marketplace that firmly connects data producers with data consumers. 

ޮ Core Integration Capabilities. Socrata has mastered the automation of data and metadata 

workflows, and integration to pull data from a wide variety of systems, and programmatically 

distribute it to tools and software for using data. The ability to combine many shapes and 

sources of data into a central platform will be a main predictor of the Commission’s ability to 

use data intuitively. 

ޮ Continuously Improving Cloud-based Delivery Model. Our data analysts and developers are 

constantly creating and adding new and improved capabilities, and our technologies today go 

far beyond what they were as recently as 12 months ago. That trend will continue, meaning 

our customers continuously benefit from a future-proof infrastructure that continues to evolve 

as technology improvements open up new innovation possibilities in the future. 

 © Copyright 2016 1 

1962 The Promise of Evidence-Based Policymaking
Technology, People, and Process to Harness the Power of the Commission’s Data 

Since our founding, Socrata has worked with over 1,300 governments, NGOs, non-profits, and 

multilaterals organizations globally. This includes virtually all major cities in the United States, dozens 

of counties, roughly half of all states, 19 Federal customers, and multiple international groups. 

Our team provides a unique combination of capabilities and expertise for accessing and aggregating 

data across disparate sources inside the enterprise. We also excel at making the data discoverable and 

understandable to a wide range of different user types, via downloadable export options, data 

visualizations, dashboards, and tailored data exploration experiences. 

Socrata is the nation’s leading government data platform in number of active deployments, revenue and 

dedicated engineers, customer success managers, apps, and innovators.

Our mission statement is unleashing the power of government data to improve society, and we work 

to achieve this by combining smart technology, people, and processes: 

ޮ Technology. Socrata provides a software as a service (Saas) platform powered by the industry 

leading Amazon AWS infrastructure. Our product is Commercial Off-the-Shelf (COTS) software, 

meaning there is no custom development or software consulting deliverables required to 

implement or maintain the solution. 

ޮ People & Process. Our team has outstanding hands-on experience planning, implementing 

and executing public sector open data and performance management initiatives. Our staff 

members include former government performance analysts, program managers, data 

analysts, and systems engineers - all of whom work exclusively on local government projects. 

Socrata brings best practices in both Open Data and Performance Indicators Management for 

government. And, we utilize a proven, Seven Step Implementation Methodology to ensure each 

project is designed around repeatable industry best practices. 

We look forward to discussing this important initiative in greater detail with your team, and we’d 

welcome an opportunity to demonstrate the flexibility of our solution. 

Sincerely, 

Kathy Hsu, Federal Account Consultant 

Phone: (925) 719-9707 

 © Copyright 2016 2 

The Promise of Evidence-Based Policymaking 1963
Email: Kathy.hsu@socrata.com 

Overarching Questions

The following sections includes Socrata’s feedback on all questions outlined in USBC’s RFI. Wherever 

possible, we have provided specific examples of where our technology products can help the 

Commission achieve its survey and administrative requirements, and we’ve illustrated specific projects 

on which we’ve achieved similar results for our customers. 

1. Are there successful frameworks, policies, practices, and methods to overcome challenges 

related to evidence-building from state, local, and/or international governments the Commission 

should consider when developing findings and recommendations regarding Federal 

evidence-based policymaking? If so, please describe. 

The Commission should consider taking a look at the Pew MacArthur Results First Initiative, which is a 

project of Pew Charitable Trusts and the John D. and Catherine T. MacArthur Foundation. The initiative 

works with states and localities to develop the tools policymakers need to invest in programs that will 

likely yield high impact. In working with 20 states and six (6) counties, the initiative is focused on: 1) 

creating an inventory of currently funded programs in a particular government, 2) reviewing programs 

which work, 3) conducting cost-benefit analysis to compare programs’ likely return on investment, and 

4) using evidence to inform spending and policy decisions. The initiative also comprises the Results

First Clearinghouse Database, which is one-stop online resource that provides policymakers with an 

easy way to find information on the effectiveness of a variety of interventions. 

Socrata has had a number of discussions with Pew recently given that a few of our customers, most 

notably Montgomery County, MD, are focused on evidence-based policymaking as well as data 

publishing and data storytelling. The Commission should consider the policies and practices of 

leveraging technology to support this work, for everything from data collection, to data governance, to 

data quality, and ultimately data-driven decision making and communication of results. We dive into 

each of these aspects in greater detail within Questions 3 and 13. 

2. Based on identified best practices and existing examples, what factors should be considered in 

reasonably ensuring the security and privacy of administrative and survey data? 

Centralized access to administrative and survey data is critical to promoting evidence-based 

policymaking. However, it is equally crucial that whomever you select as your partner is equipped to 

ensure the security and privacy of that data as it is published for researchers, institutions, and the 

general public. 

Socrata follows industry best practices for ensuring the security and privacy of data that we host on 

our platforms, including: 

ޮ Hosting data in a secure system that offers role based access control - i.e., only those on a 

“need to know” basis are given access; 

ޮ Establishing data governance and lifecycle practices with customers like the Centers for 

Disease Control, UN Capital Development Fund, and other organizations that ensure data 

conforms to federal law around personally identifiable information 

 © Copyright 2016 3 

1964 The Promise of Evidence-Based Policymaking
ޮ Creating views on top of private data that can be shared with wider audiences, yet hide 

information that is meant to be internal only; 

ޮ Maintaining physical data security measures, which includes replicating data between three 

different AWS Availability Zones, in each of two geographically separated AWS data centers. 

Each zone can power the entire Socrata platform, as well as replicate the database write-ahead 

logs in near-real-time and take full backups every three days. Backups are retained for 30 days, 

and Socrata has an RTO target of 96 hours in the event of catastrophic failure of the AWS 

environment. 

Furthermore, Socrata fully complies with FedRAMP regulations for responsible disclosure and 

notification, in the case of breach, leakage, or security incident, and we maintain an Information 

Security Contingency Plan that covers business continuity, as well as incident response procedures. 

This plan is reviewed and tested annually, and the plan and test results can be made available upon 

request. We maintain a FISMA-Low ATO, and we inherit our physical security measures from AWS. In 

addition, Socrata is currently in process to receive FedRAMP certification. We have completed our audit 

and are working with a cabinet-level agency to finalize the process and achieve our Authority To 

Operate (ATO). To see the latest on our current status, please visit the Socrata CSP page on the 

FedRAMP website. 

Data Infrastructure and Access

3. Based on identified best practices and existing examples, how should existing government data 

infrastructure be modified to best facilitate use of and access to administrative and survey data? 

One of the biggest transformations in modern government has been the pivot to data transparency 

and efficiency. Traditional data was often siloed - locked up within internal government networks and 

servers, making highly valuable information inaccessible to other departments, researchers, policy 

makers, and public users who could benefit from it most. The move towards data visibility has led to 

sweeping improvements across the government, providing the public and key stakeholders with 

instant access to critical data and performance measures that can guide more informed decision 

making. 

Foundational and policy-specific evidence are vital to the creation of specific programs and legislation, 

and this evidence relies heavily on survey and administrative data, including: large-scale administrative 

data; program-specific administrative data; general purpose survey data; and study-specific survey 

data. 

Best Practices for Opening Survey Data 

Because of survey complexity, often this data remains in formats that are inaccessible or unusable to 

the general public. Existing government data infrastructures can be modified to facilitate access to, and 

sharing of, administrative and survey data by: 

ޮ Uploading it to a centralized platform that enables easy sharing among internal teams and 

external users, as needed; 

ޮ Ensuring data is captured in a machine-readable, API-enabled format; 

 © Copyright 2016 4 

The Promise of Evidence-Based Policymaking 1965
ޮ Increasing data accessibility through easy-to-use search features, and proper metadata 

classification for filtering; 

ޮ Promoting scalability - making it easy to build on top of the display data in a visual, 

easy-to-understand format. 

Socrata Open Data Platform for Optimized Transparency & Accessibility 

Socrata offers a unique combination of capabilities and expertise for accessing and aggregating data 

across disparate sources. On every data initiative we engage in with our clients, our focus is on making 

data discoverable and understandable to a wide range of different user types. We do this through 

sophisticated data visualizations, easy-to-use dashboards, contextual narratives, flexible download and 

export options, and highly-tailored exploration experiences. 

The Socrata Data Platform is a cloud-based, turnkey, software-as-a-service (SaaS) solution designed 

from the ground up to simplify the process of data publishing and data access. 

High-level Socrata Data Platform architecture. 

It empowers end users with tools for ad-hoc data consumption, exploration and visualization, and 

provides capabilities that enable the sharing of that information across multiple channels. The 

platform is delivered as a web portal that allows users to: 

 

ޮ Perform robust searches for data and information. 

ޮ Interact with that data by sorting, filtering, and performing group-by and roll-up operations. 

ޮ Create real-time visualizations on top of the data with charts, graphs and maps and save these 

visualizations to user profiles. 

ޮ Share data and saved views in context rich reports, dashboards and stories to extend its reach 

and utility. 

ޮ Integrate, extract and cleanse a variety of shapes and sizes of source system data in real time 

and present that information in a consumption formats. 

 © Copyright 2016 5 

1966 The Promise of Evidence-Based Policymaking
ޮ Empower advanced users and developers with machine readable data, APIs and tools to create 

sophisticated services, robust interfaces and dynamic mobile applications. 

For USBC, this platform would offer a number of important benefits across the data lifecycle, including: 

acquiring data; preparing the data; providing data; and monitoring usage and outputs. We see these 

benefits as including: 

ޮ A world class data discovery experience. Socrata’s world class data discovery experience 

would allow USBC users to find the data they are looking for easily, regardless of their technical 

sophistication and familiarity with robust survey results and datasets. 

ޮ A Collaboration Environment. The Socrata Data Platform facilitates collaboration around 

data, through commenting, sharing, and regular updates. It promotes a thriving data 

marketplace that quickly connects data producers with data consumers to foster informed 

action as a result of that data. 

ޮ Core Integration Capabilities. Socrata’s products allow for the automation of data and 

metadata workflows, including the integration of data from a wide variety of systems. 

ޮ Continuously Improving Cloud-based Delivery Model. New and improved capabilities and 

features are constantly being added to the Socrata suite of products, which means USBC 

administrators would benefit from a future-proof infrastructure that continues to evolve as 

technology improvements open up new innovation possibilities in the future. 

Socrata Work Samples Relevant to USBC’s Data Initiative 

Socrata’s work spans federal, state, and local government projects, and we have highlighted a specific 

example below from the CDC - which centers on the hosting and administration of survey data: 

CDC’s Chronic Disease and Health Promotion Data & Indicators 

Increasing and improving access to CDC’s chronic disease and health promotion data and indicators is 

a key component of addressing the burden of chronic diseases in the United States. Chronic diseases 

are responsible for 7 of 10 deaths each year, and treating people with chronic diseases accounts for 

86% of our nation’s health care costs. 

In January 2015, the National Center for Chronic Disease Prevention and Health Promotion at the CDC 

launched Chronic Disease and Health Promotion Data & Indicators that aims to enable quick access to 

data and indicators across health areas such as tobacco use, reproductive health, oral health, cancer, 

legislative actions, and behavioral risk factors – to name only a few. 

As an early adopter in the Center, the Office on Smoking and Health migrated the current and 

historical state-level data on tobacco use prevention and control housed in the State Tobacco Activities 

Tracking and Evaluation (STATE) System to the new data Tobacco Use Portal and opened the door to 

using a new technology portfolio to reinvent the STATE System for 2015. 

Now in Socrata, users such as data analysts, policy staff, and researchers can quickly view data in 

colorful charts, graphs and maps, create custom filters to subset data, download data in a variety of 

formats, and even receive updates when data of interest has been updated. The featured dataset 

below displays a map of US states with comprehensive smokefree air laws, which protect the public 

from exposure to secondhand smoke. 

 © Copyright 2016 6 

The Promise of Evidence-Based Policymaking 1967
4. What data-sharing infrastructure should be used to facilitate data merging, linking, and access 

for research, evaluation, and analysis purposes? 

The real power of data to drive evidence-based policymaking is unleashed when researchers and 

policymakers can combine data and present it to legislators and to the public in novel ways. In order to 

facilitate this, data needs to be on a platform that allows users to easily build on top of that 

information to provide greater insight - going beyond simply presenting raw data, to offer insight into 

why it matters, and ensuring users understand its accuracy and integrity. One way the underlying data 

infrastructure can facilitate the merging and linking of data is to provide programmatic access to data 

through application programming interfaces (APIs). APIs allow developers to build tools that extract 

data from multiple datasets or other web services and merge them using the attributes of the data 

itself, or more complex methodologies. For researchers who may not be developers or as technically 

sophisticated, APIs allow them to feed data into commonly used statistics and visualization tools, like 

Tableau and SAS. 

The Socrata Open Data API provides an open, standards-based, RESTful application programming 

interface for datasets. Any dataset uploaded to the Socrata platform is automatically provided with a 

RESTful JSON endpoint, allowing developers to leverage data within the platform through a familiar 

and powerful API built on open standards. The Socrata platform also has libraries for development 

languages like R, which are commonly used in the statistics community. 

Socrata’s open APIs give users the power to: 

ޮ Upload data in a central platform, allowing users to build on top of it; 

ޮ Merge and link data through visualizations; 

ޮ Combine data with APIs and and build custom tools on top of it; 

ޮ Take into account the fact that governments cannot account for every single use of data, 

thereby leveraging APIs to give end users flexibility to build the apps they need; 

ޮ Make data accessible via endpoints, allowing common visualization and analysis tools to be 

used; 

ޮ Unlock machine processing of datasets; and 

ޮ Empower developers to combine datasets with other web services for new mashups and 

applications. 

5. What challenges currently exist in linking state and local data to federal data? Are there 

successful instances where these challenges have been addressed? 

Data challenges that exist with inter-agency data sharing - both between federal agencies themselves, 

and between federal and state agencies - are typically self-imposed. Often, agencies at all levels have 

adopted formal and informal policies (e.g., sharing of personally identifiable information, other 

sensitive information, etc.) over the years - sometimes with limited understanding of how these 

policies may contribute to the difficulty around sharing within government or between governments. 

 © Copyright 2016 7 

1968 The Promise of Evidence-Based Policymaking
These policies are often complicated further by technical complexities, including differences in data 

schemas and inconsistencies in common data terms used. 

We recommend that the Commission lead a review of all existing policies - with strong federal, state, 

and local executive leadership - to identify archaic data sharing policies, and create an action plan to 

refine these policies to promote greater accessibility and use. 

6. Should a single or multiple clearinghouse(s) for administrative and survey data be established to 

improve evidence-based policymaking? What benefits or limitations are likely to be encountered in 

either approach? 

Socrata’s recommendation is to centralize all data on a single platform to ease administration and data 

management. However the decision to establish a single warehouse, or multiple clearinghouses for 

administrative and survey data, should be based on the administration model of that clearinghouse. It 

is likely that the owners, curators, and publishers of the data will be different from the consumers - 

and not all will have the ability to edit and/or view every dataset. If a clearinghouse does not support 

the unique administration of assets, it may be valuable to segregate data into multiple clearinghouses 

with different permission models. A data platform like Socrata that enables roles based access control 

can house all data in one, centralized clearing house and maintain administrative boundaries. 

The Socrata platform limits who can see and perform actions based on defined end user permissions, 

and depending on whether a dataset is recorded as “private” (i.e., internal only) or “public” (i.e., 

published externally for researchers and institutions). A role defines the types of actions that someone 

can perform across all datasets on the domain. 

Administrators on a Socrata-powered data site are able to assign any number of roles to other users, 

allowing for delegation of tasks on the site and enabling other users to create and manage datasets. 

Socrata users can have roles on a domain, which define the types of actions they can perform across all 

datasets on that domain, as well as high-level operational control of the domain itself. Examples of these 

actions include publishing data, adding features, or managing users. Socrata users can also be granted 

specific user permissions per dataset. A permission defines the types of actions that someone is able 

to perform on a specific dataset. Examples include editing that dataset, viewing that dataset, and 

publishing that dataset. 

7. What data should be included in a potential U.S. government data clearinghouse(s)? What are 

the current legal or administrative barriers to including such data in a clearinghouse or linking the 

data? 

The data that the Commission should consider including really depends on the problems it is trying to 

solve, and the related programs and policies where greater evidence building will be used to drive 

more impactful practices. Whatever data the Commission decides to include, it should be filterable and 

searchable in a number of ways, such as by broad policy area, issue area, department, data type, data 

source, and perhaps some type of rating as to its usefulness - and in the case of an intervention, 

whether it’s proven. Each dataset should also be API-enabled for greater utility across applications. 

In terms of data specifically, the Commission should first consider broad themes in the areas of: 

economy, health, education, transportation, agriculture, environment, housing, and security, and then 

drill deeper into specific policy areas under each broad theme. The Commission should emphasize 

 © Copyright 2016 8 

The Promise of Evidence-Based Policymaking 1969
data that has multiple applications, can be used by multiple parties with few barriers, and has strong 

re-use value. 

8. What factors or strategies should the Commission consider for how a clearinghouse(s) could be 

self-funded? What successful examples exist for self-financing related to similar purposes? 

Socrata’s recommendation for self-funding its clearinghouse is to offer full access to specific data sets 

for free, but charge users for access to more detailed reports, data visualizations, or highly-valuable 

analysis and narratives (e.g. specific benchmarking reports and analysis etc.). We work with one 

customer in particular - the Carbon Disclosure Project (CDP) - who offers free data across a number of 

key areas, but charges users a specific fee for access to its premium data sets. 

CDP’s purchasing page for access to its premium data. 

Premium access for CDP users includes entry into a larger collection of primary environmental data, 

maximum flexibility to use its raw data, and historical data going back five years, providing greater 

transparency into how companies have evolved throughout the years. If the Commission can identify 

specific data that is uniquely valuable, offering a similar model is likely the easiest way to at least 

partially self-fund. 

9. What specific administrative or legal barriers currently exist for accessing survey and 

administrative data? 

In the White House Commission on Evidence-Based Policymaking’s white paper, Barriers to Using

Administrative Data for Evidence - Building, it states that “Agencies need infrastructure, such as trained 

personnel, formal written procedures, and information technology tools, to consider data sharing 

requests and to develop and implement data sharing agreements” to overcome barriers for data use. 

Many of these barriers also exist for accessing survey and administrative data. In order to use data, it 

needs to be collected, curated, centralized, published, and shared between organizations so that 

 © Copyright 2016 9 

1970 The Promise of Evidence-Based Policymaking
researchers can access and use it effectively. It also requires a methodology that addresses all these 

stages in a complete and robust way. 

Our projects are typically divided into three high-level phases that include: plan, build, and handoff. 

Throughout the project the Socrata team focuses on: 

 

ޮ Establishing goals and success measures, and working closely with stakeholders to develop 

clearly defined (and ideally measurable) goals. 

ޮ Defining the proper user experience (UX) for end users. Techniques such as wireframing and 

rapid prototyping are employed to elicit user feedback and suggestions to inform the UX 

design. 

ޮ Working with data owners to establish a dataset on-boarding process, including: 

ߦ Surveying and identifying relevant datasets, 

ߦ Cleansing and formatting data, and 

ߦ Processing metadata. 

ޮ Building and managing negotiations and trusted relationships with key data providers, with 

respect to public information, privacy, data sharing, and level of access. 

ޮ Managing roll-out and an ongoing enhancement strategy, and maintaining documentation. 

ޮ Promoting industry knowledge and leadership regarding innovation trends and emerging 

market shifts. 

ޮ Effectively maintaining key contacts related to survey and administrative data, community 

engagement, and data integration. 

Coupling a strong, repeatable program methodology with a technology platform that removes current 

barriers to accessing survey and administrative data is key to the success of a good data program. The 

Socrata platform provides data publishers with the ability to curate metadata and any supporting 

documentation, and equips users with multi-level catalog search and filtering capabilities to find 

appropriate data and then make user of it. 

10. How should the Commission define “qualified researchers and institutions?” To what extent 

should administrative and survey data held by government agencies be made available to 

“qualified researchers and institutions?” 

Socrata has elected not to answer this question, as it is outside the scope of our expertise. Should this 

RFI move to a Request for Proposal, we will partner with a trusted company in order to supplement 

any areas of work that we cannot deliver. 

 © Copyright 2016 10 

The Promise of Evidence-Based Policymaking 1971
11. How might integration of administrative and survey data in a clearinghouse affect the risk of 

unintentional or unauthorized access or release of personally-identifiable information, confidential 

business information, or other identifiable records? How can identifiable information be best 

protected to ensure the privacy and confidentiality of individual or business data in a 

clearinghouse? 

When dealing with sensitive data that may include personally-identifiable information or other 

confidential information, it is of critical importance to establish robust data governance and lifecycle 

management practices. Socrata’s data consultants work with organizations on a daily basis to establish 

data practices that ensure PII and confidential information is not transferred from source data systems 

to clearinghouses that are open to researchers and other public with access. 

According to best practices data is processed through an extract, transform, load (ETL) workflow that 

scrubs it of any sensitive information before it is published to a clearinghouse. Separating the data that 

is housed in the clearinghouse from the source systems prevents any accidental contamination and 

dissemination. 

Before data is published on the Socrata platform it also goes through an approval process. Datasets 

that are set to be published go through stages, or gates, with configurable settings for approval stages 

with assigned gatekeepers for each stage and controlled notifications. This approval queue allows 

individuals to approve or reject datasets and provides view of all the datasets currently in the 

publishing lifecycle. Any sensitive information that accidentally made it past the ETL process can be 

flagged by administrators before it is published on the platform. 

12. If a clearinghouse were created, what types of restrictions should be placed on the uses of 

data in the clearinghouse by “qualified researchers and institutions?” 

Socrata believes data access should be restricted by user role, and by “need to know.” For example, 

researchers should only have access to data that is relevant to their project(s), and even then, only if 

their role on the project team requires that access. Adhering to this concept of “least privilege” 

minimizes the risk that potentially sensitive project data will be unduly exposed. 

13. What technological solutions from government or the private sector are relevant for facilitating 

data sharing and management? 

Socrata’s suite of open data products are highly relevant for the facilitating of data sharing and data 

management. We provide multiple tools to ensure our customers’ data is provided to the right 

audiences at the right time, and our solutions center on promoting the highest levels of data 

discoverability, visualization and context, and sharing and usability: 

Data Discoverability 

Socrata’s Open Data solution provides better access to quality data and is used extensively by elected 

officials and policymakers, government employees, residents and businesses, and developers and 

advocates. Using Open Data Portal allows users to: 

 © Copyright 2016 11 

1972 The Promise of Evidence-Based Policymaking
ޮ Publish datasets on any topic (and these data sets can be up to hundreds of millions of rows 

long), including: geospatial data, financial data, public safety data, and much more; 

ޮ Use a global search interface to find the data, and take advantage of built-in SEO optimization 

to enable data to be found via standard web browser searches; 

ޮ Provide insight into your most popular datasets with a summary page that captures important 

information about your uploaded content.

Visualization and Context 

Charts and graphs would allow USBC staff to transform data into visualizations to promote better 

understanding of trends and gaps. Users can easily create and maintain customized charts, graphs, 

and other data visualizations for public data use, without having to build and maintain a data 

infrastructure on their own. We offer 13 different visualization formats available out-of-the-box, which 

can be dynamically applied to datasets for maximum flexibility in displaying critical information. These 

include: 

Area chart | Bar chart | Bubble chart | Column chart | Donut chart | Line chart | Pie chart | Stacked 

bar chart | Stacked column chart | Timeline | & Tree map 

Furthermore, we have developed our own charting library using the D3.js toolkit, which provides a 

wide range of configuration options, dramatic colors, as well as great performance and stability. Charts 

will always support the full set of values calculated and rendered, with no limitations to scrolling across 

the full dataset in a chart, in order to display every value. They will auto-scale dynamically depending 

on type, and the axis placement can be dragged around on the page to provide the best display on 

your screen. Visualization legends also support conditional formatting. 

Visualization options using Open Data Portal. 

 © Copyright 2016 12 

The Promise of Evidence-Based Policymaking 1973
Socrata Perspectives augments the Open Data Portal by helping users connect the dots between policy 

and outcomes. With Socrata Perspectives, users are able to easily and quickly produce narrative 

reports and stories with drag-and-drop functionality that allows for text, images, maps, and data 

visualizations. The screenshots below show a sample dashboard, before context and data is added 

(left), and and after (right). 

USBC employees can leverage a drag-and-drop toolkit of 

images, visualizations, text, video and banners to create 

compelling reports and dashboards based off of live 

data. 

Resulting reports pair data with text and will 

track updates as they are published from the 

SSP and EHRI Data Warehouses. 

Sharing and Usability

Maximizing the value of data means making it available to as many people as possible. Socrata 

provides several tools to empower data users to easily access and utilize the data in creative ways. 

ޮ Share any data asset on Socrata’s platform with others directly through the built-in data 

catalog or on social media. 

ޮ Embed content within existing websites or build a customized, attractive open data homepage. 

ޮ Syndicate datasets from other nearby localities to increase the context and meaning of the 

data. 

 © Copyright 2016 13 

1974 The Promise of Evidence-Based Policymaking
ޮ Give technical developers, innovators, and entrepreneurs easy access to the data through the 

Socrata Open Data API (SODA).

14. What incentives may best facilitate interagency sharing of information to improve 

programmatic effectiveness and enhance data accuracy and comprehensiveness? 

There are a number of ways to incentivize the facilitation of interagency data sharing, including: 

ޮ Making data sharing a condition of federally-funded research; 

ޮ Working with OMB to develop a fund that will support these efforts across agencies through 

the budget process; 

ޮ Setting collaborative goals for improving program performance through evidence-based 

practices that cut across mission areas and departmental silos and that require shared data to 

drive progress; 

ޮ Rewarding researchers for releasing and sharing data through proven citation systems so they 

receive credit and exposure; 

ޮ Providing unique opportunities for “data sharers” from agencies across government to discuss 

promising practices in a variety of fora with opportunities to lead a series of studies, sponsored 

by the Commission and perhaps the White House, on programmatic impact through 

evidence-based policymaking; and 

ޮ Providing a common data platform, available to all who participate as “data sharers”, for data 

sharing governance, management, and analysis. 

Data Use in Program Design, Management, 

Research, Evaluation, and Analysis

15. What barriers currently exist for using survey and administrative data to support program 

management and/or evaluation activities? 

Lack of documentation and appropriate metadata are two barriers to effective data use, after that data 

has been published publicly. It is critical to augment raw data with metadata and accompanying 

documentation so that users can determine if and how that data is appropriate for them to use. As the 

Barriers to Using Administrative Data for Evidence - Building states: “Data curation also includes the 

maintenance of relevant metadata and other documentation. Prospective data users need at least 

minimal documentation to assess fitness for a given use, and more detailed documentation enables 

more informed and appropriate use.” A platform that allows data publishers to curate and publish 

metadata is crucial to data accessibility and understanding, and the Socrata technology stack and 

methodology makes metadata a priority. 

Socrata contains standard metadata fields and also allows for custom definition of required and 

optional metadata fields. Specific fields, drop-down menus, and other metadata entry points can be 

added by administrators to ensure that all uploaded data follows an existing schema, and is not 

 © Copyright 2016 14 

The Promise of Evidence-Based Policymaking 1975
uploaded without complete metadata entry and definition. 

In addition to being able to configure all metadata content and fields during upload, any number of 

files can be attached to a datasets to allow for full documentation to accompany these datasets. A data 

dictionary, or a guiding document with detailed descriptions of the dataset and its metadata can be 

attached in this way. Any attached documentation will always accompany the dataset, and 

administrators can also configure the metadata templates to ensure all data sets are documented 

accurately. 

16. How can data, statistics, results of research, and findings from evaluation, be best used to 

improve policies and programs? 

Socrata has demonstrated excellence in developing high-profile open data portals, and in addressing 

administrative, usability, and ongoing support needs. Our clients include federal, state, and local 

governments, and in the process of completing over 1,300 projects, we have seen how data can help 

organizations increase transparency, encourage engagement and, in turn, improve policies and 

program. Below are a few examples: 

Centers for Medicare and Medicaid Services 

As the biggest single payer for health care in the U.S., the Centers for Medicare and Medicaid Services 

(CMS) is playing a leading role in making the costs of healthcare more transparent and easier to 

compare across geographies and providers. Its sites, data.cms.gov and data.medicare.gov, share data 

on the costs for procedures at different hospitals, as well as data on quality and efficacy, such as 

readmission rates by facility within 30 days after discharge. 

The release of data is a critical component of CMS’s strategy to drive improvements in the U.S. 

healthcare system. "Data transparency is a key aspect of transformation of the health care delivery 

system,” says CMS Administrator Marilyn Tavenner. 

By improving the availability of data on cost and quality, CMS has catalyzed competition in a largely 

opaque market. Their open data sites empower both consumers and payers and provide information 

in a variety of formats that serve many audiences. For example, access to the raw, machine-readable 

data available behind any visualization allows sophisticated users, such as researchers and data 

scientists, do in-depth analysis. It also allows private sector providers to benchmark themselves and 

understand how they compare against their competitors, hospitals, insurance companies, and 

individual providers. 

Under the Tavenner’s leadership, CMS has made releasing data a key part of its strategy to make the 

US healthcare system more transparent, affordable, and accountable. CMS has been releasing cost and 

quality data to the public domain since 2011. 

“Data is the lifeblood of our healthcare system, and for too long it has not been optimized by its full 

potential,” says Tavenner. 

City of Boston 

The City of Boston had been publishing their budget in multi-volume PDFs for years. While there’s a 

wealth of information contained in these volumes, it can be difficult for the public to understand where 

the City is allocating resources and for what purposes. In 2014, Boston Mayor Marty Walsh issued an 

 © Copyright 2016 15 

1976 The Promise of Evidence-Based Policymaking
executive order to improve the City’s open data efforts. Three months later, the City released a 

web-based interactive application – Socrata’s Open Budget -to publish both their operating and capital 

budgets. Open Budget has improved the City’s financial transparency and given everyone – city 

employees, reporters and researchers, and the public – easy access to the City budget. 

Open Budget allows the City to publish to single portal that anyone can access. Because the budget 

data is public and easy to navigate, more audiences – citizens, reporters, academia – are able to review 

the exact same datasets that the government uses. 

The City had a fairly painless migration from our original datasets to the Open Budget data schema. 

“We were lucky, our data was mostly well-formed,” said Dwelley “We had a few items that needed 

customization, but Socrata handled that for us.” 

There have been immediate internal benefits from the City’s open data program. Fewer requests for 

information free up staff time and consistent reporting across departments helps with information 

sharing. But Boston is also excited about what citizens can do with the data now that it’s so easy for 

them to access and understand. 

Code for Boston hosts a weekly hack night that encourages “developers, designers, urban planners, 

and data geeks” to use technology and publicly available data to solve community problems. The 

Mayor’s Office of New Urban Mechanics taps the data to help understand problems as diverse as 

school transportation and improving the quality of Boston’s roads. 

City of Seattle 

The Seattle Police Department, the birthplace of bicycle patrol units, has a track record of innovation 

and a forward-thinking approach in its publishing of public safety data. In 2012, a federal investigation 

over charges of excessive use of force, as well as related concerns regarding officer supervision and 

internal investigations, resulted in an agreement between the Seattle Police Department and the U.S. 

Department of Justice. The two organizations crafted a reform plan to boost the SPD’s oversight, 

training, and reporting. 

The department achieved extraordinary progress in the reform areas over the following two years, 

including pioneering the publishing of public safety data. Momentum increased in 2014 with the 

appointment of Chief Kathleen O’Toole, as well as the launch of SeaStat, the SPD public safety data

portal created in partnership with Socrata. 

The SPD joined forces with the city’s Chief Technology Officer to cosponsor the City of Seattle’s privacy 

initiative. Together, they built upon the city’s race 

and social justice initiative, promoted a set of guiding privacy principles, and crafted strategies to 

increase transparency, while also protecting privacy rights of citizens. These efforts will help city 

departments assess privacy issues around open data. 

17. To what extent can or should program and policy evaluation be addressed in program designs?

Program and policy evaluations include policy-specific evidence that allow organizations to determine 

whether a particular program or piece of legislation is achieving its objective. Socrata is a strong 

proponent of data-driven, evidence based government. When appropriate, these evaluations should 

include performance measures and indicators that can be addressed and evaluated in program design. 

Socrata has worked with organizations at the state, local, and federal level in collecting and analyzing 

 © Copyright 2016 16 

The Promise of Evidence-Based Policymaking 1977
evaluation data and presenting it to the public to drive transparency into program and policy 

effectiveness. We provide specific examples below. 

18. How can or should program evaluation be incorporated into program designs? What specific 

examples demonstrate where evaluation has been successfully incorporated in program designs? 

Cook County’s performance management program began in January of 2011, when the County passed 

an ordinance that required each office and agency to set goals and collect operational data around 

those goals. The County began a program to create a strategic plan around the use of open data for 

decision-making and communications. Andrew Schwarm, Chief Performance Officer of Cook County, 

was the project lead tasked with finding a performance measurement tool. 

“We faced two main challenges when we began this program,” Schwarm says. “The ordinance requires 

weekly meetings to review our goals and measure performance. There is also a reporting 

requirement.” 

The County started by publishing reports quarterly in PDF format. “Quickly, we saw it was difficult for 

residents to dig into these dense reports and digest the information in a useful way,” Schwarm 

explains. “We needed a tool that would allow us to take the wonderful data we collect quarterly, 

around 750 data points over 50 agencies across the County, and publish it in a usable way. 

Performance Data Cloud (formerly known as Open Performance) jumped out as the solution that 

made the most sense for us.” 

An early adopter of the GovStat platform, Cook County chose a two-phase implementation process. 

First, the County replicated the PDF data and created reports for each department. “GovStat allowed us 

to take the data already gathered and put it on a more flexible, user-friendly, open, and transparent 

platform,” Schwarm says. 

Once the data was updated and made available to the entire organization, Cook County entered phase 

two of the implementation: using data to drive decisions. “[Data-driven decision-making] is now part of 

our culture and the way we do business,” explains Schwarm. What’s more, accurate data allows Cook 

County to tell accurate stories. 

One early goal emerged from the Finance and Administration Department. “We started by measuring 

employee time and attendance. Using GovStat and the data we collect helped us identify policy abuse 

or data entry errors, reducing average employee sick leave per month from over seven hours in 2012 

down to 4.66 hours as of December 2013,” Schwarm says. 

GovStat also allows for the dynamic evolution and improvement of goals. “The County does 

environmental inspections of industrial facilities in the suburbs,” Schwarm explains. “Two years ago, 

our main goal was to make sure 100 percent of the facilities were inspected. After GovStat showed we 

were consistently meeting that goal, we are now able to focus on the quality and outcomes of those 

inspections,” Schwarm says. 

Schwarm is excited to track other county initiatives, including public safety and health care goals. “We 

plan to roll out one goal per month and continue to report on its progress,” Schwarm says. He 

continues, “The use of good, timely, accurate data, especially for a government facing tough fiscal 

situations, allows us to make decisions and prioritize at a high level.” 

 © Copyright 2016 17 

1978 The Promise of Evidence-Based Policymaking
19. To what extent should evaluations specifically with either experimental (sometimes referred to 

as “randomized control trials”) or quasi-experimental designs be institutionalized in programs? 

What specific examples demonstrate where such institutionalization has been successful and 

what best practices exist for doing so? 

Socrata has elected not to answer this question, as it is outside the scope of our expertise. Should this 

RFI move to a Request for Proposal, we will partner with a trusted company in order to supplement 

any areas of work that we cannot deliver. 

 © Copyright 2016 18 

The Promise of Evidence-Based Policymaking 1979
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 13, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tkp-ad03

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0250

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

I am strongly apposed to the data mining being collected on school age people. This is

communist style education and a enemy toward a free and fair state for all. We need to

bring back local control.

1980 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 13, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tkp-th9l

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0251

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

Ibelieve in informed consent. I oppose non-consensual data mining.Stop thismadness!

The Promise of Evidence-Based Policymaking 1981
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 13, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tkp-878v

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0252

Comment on FR Doc # 2016-22002

Submitter Information

Name: SARA ALLEN

Address:

Indianapolis, IN, 46234

Email: scubatsa@aol.com

General Comment

Non-consentual information must never be collected. This must stop.

1982 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 13, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tkq-as40

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0253

Comment on FR Doc # 2016-22002

Submitter Information

Name: Shari Ollis

General Comment

I believe in informed consent.

The Promise of Evidence-Based Policymaking 1983
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 13, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tkq-ah5a

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0254

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Hamilton

General Comment

Stop treating people like numbers. Treat them and their privacy with respect. Their

children are not your children, and raising them is NOT your God-granted stewardship.

It is so arrogant to assume such a role.

It is not the business of government to intrude in people's lives. Furthermore, you will

spend billions of dollars on data collection to find out that it is useless because it you

can't accurately characterize the complexity of people's lives with a few data points.

1984 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 13, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tkq-etzs

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0255

Comment on FR Doc # 2016-22002

Submitter Information

Name: GN Olson

General Comment

Thank you for considering the intelligent and reasoned points made by commenter

USBC-2016-0003-0233, with whom I agree strongly. Government agencies are at best

moribund, at worst, criminal, in their handling of private information. Please help the

United States return to its status as one of the most free countries in the world. Do not

approve this initiative. Thank you.

The Promise of Evidence-Based Policymaking 1985
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 13, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tkr-ihq6

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0256

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

Tired of being lied to and manipulated. I am against data mining. Enough. Our children

deserve a better world, a free world not the world described in "1984"

.

1986 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 13, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tkr-ps5g

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0257

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

I oppose data mining without informed consent. Have some integrity and stop womb to

tomb tracking.

The Promise of Evidence-Based Policymaking 1987
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 13, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tkr-639l

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0258

Comment on FR Doc # 2016-22002

Submitter Information

Name: Deborah Guebert

General Comment

Anyone who knows how governments, including ours, have misused dossiers of

information on private citizens in the past, will have no confidence in allowing the

federal government to compile and retain individually recognizable data files on

innocent people. Memories of the crimes perpetrated by HItler's Third Reich motivate

the German people, even to this day, to adamantly oppose the general collection of data

even by internet operators such as Google. Those who do not pay attention to history are

foolish indeed. Human nature has not changed, and the temptation to abuse power is still

one that very few are able to resist. As the saying goes, "Power corrupts. Absolute power

corrupts absolutely." We neglect that wisdom to our peril.

1988 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tkt-gu08

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0259

Comment on FR Doc # 2016-22002

Submitter Information

Name: Paul Adcock

General Comment

I think this is a bad idea, to say the least. I can see where all of this is going, to a control

economy like in China where the government picks the occupation of people. All of this

federal College and Career Ready, Social-Emotional Learning, Career Technical

Education, Common Core, etc are just codewords for setting up a cradle to career

controlled system.

The Promise of Evidence-Based Policymaking 1989
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tkt-1412

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0260

Comment on FR Doc # 2016-22002

Submitter Information

Name: Oak Norton

General Comment

Here's an idea. Stop data mining without the consent of those whose data you are

gathering. Stop encroaching on personally identifiable information. It's personal. That's

the point.

1990 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tkt-v4c7

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0261

Comment on FR Doc # 2016-22002

Submitter Information

Name: Michael Bird

General Comment

Our personal information is not for your use. Keep your hands off of it. This is just plain

wrong. Stop it.

The Promise of Evidence-Based Policymaking 1991
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tkt-i5z6

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0262

Comment on FR Doc # 2016-22002

Submitter Information

Name: JaKell Sullivan

General Comment

American taxpayers should not foot the bill to lose their privacy. The loss of personal

privacy equals the loss of liberty. If the CEP, being funded by taxpayers, is working to

make databases interoperable, than the taxpayer deserves to be able to make an informed

decision about whether they want to be tracked. If it's our private information, it is our

private property and the United States government was, in part, founded on the principle

that property is sacred--that the conscience is sacred. Please protect and preserve

individual liberty by opposing all efforts to interlink government databases. And, by

consulting with taxpayers about what privacy should look like in the 21st century. First,

do no harm.

1992 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tku-crji

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0263

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

Data based information!?? Are you kidding? Because the DMV is a bastion of customer

service oriented, friendly people? Because the USPS is known for its business acumen

and employees are overcome with joy? Because colleges and universities are focused on

knowledge and using that knowledge to make the world a better place for everyone.

Everyone or you are not making it a better place. Stay away from kids. The mandatory

six hours a day translates to maybe a half hour of education. Not better. Far worse.

The Promise of Evidence-Based Policymaking 1993
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tkv-l7l6

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0264

Comment on FR Doc # 2016-22002

Submitter Information

Name: Shelley Metzenbaum

Address:

Concord, MA, 01742

Email: smetzenbaum@outlook.com

General Comment

See attached file(s)

Attachments

Metzenbaum to CEP121416

1994 The Promise of Evidence-Based Policymaking
Metzenbaum comments to Commission on Evidence-Based Policymaking – Dec. 14, 2016 p. 1 

Commission on Evidence-Based Policymaking Comments 

Docket Number 160907825-6825-01 

December 14, 2016 

Submitted by Shelley H. Metzenbaum, Ph.D. 

Former OMB Associate Director, Performance and Personnel Management 

Independent Consultant and Senior Fellow, Volcker Alliance 

 

The Promise of Evidence-Based Policymaking 1995
Metzenbaum comments to Commission on Evidence-Based Policymaking – Dec. 14, 2016 p. 2 

1. Are there successful frameworks, policies, practices, and methods to overcome 

challenges related to evidence-building from state, local, and/or international 

governments the Commission should consider when developing findings and 

recommendations regarding Federal evidence-based policymaking? If so, please 

describe. 

Let me preface my discussion of successful frameworks, policies, practices, and methods by 

clarifying what I mean by the term “successful.” Successful data and evidence-rich frameworks, 

policies, practices, and methods (hereafter referred to as “Frameworks etc.”) are those used by a 

wide variety of users for a wide variety of uses. Data, analytics, and research findings inform 

decision-making, action, and communication that lead to better performance on multiple 

dimensions, including outcomes/impact, return on spending, fairness, understandability, 

democratic accountability, trust, and individual consumer choice. Further, successful Frameworks 

etc. are constructive; they motivate people in government, grantees, goal allies, and others to want 

to use and communicate data and other information for decision-making. They do not make them 

fear the data or worse, feel tempted to manipulate or implode the measurement system and distort 

or discredit evaluation findings. 

Unfortunately, many government Frameworks etc. are not successful. Some produce data and 

evidence that are of limited usefulness or, even, useless. Others produce relevant data and 

research findings that, independent of concerns about privacy or confidential business 

information, are underutilized or misused because few know they exist, know now how to 

analyze or interpret the information, or can easily and affordably access the information. 

Success necessitates that people are aware that useful information exists, understand how to 

access and interpret it, easily and affordably access it, feel motivated to apply the data and 

research findings, regularly use them to inform decisions, and readily contribute to data and 

evidence generation to fill critical knowledge gaps. Successful Frameworks etc. produce data, 

analyses, evaluations, root cause analyses, and other insights that are routinely used by a wide 

variety of data users (see List 1, re likely users, prior to the cases below) for a wide variety of 

uses (see List 2 re kinds of uses.) Successful Frameworks etc. can be national in scope, but they 

can also be local, state, regional, sector-specific, or problem/opportunity-specific in scope to 

inform real-time, field-based decision-making. 

Succinctly stated, successful frameworks use data and other evidence to: 

• illuminate (what works, what works better, what doesn’t, why); 

• nominate issues for government attention;

• motivate; 

• allocate; and 

• communicate to enlist, engage, share, learn, inform individual choice, and strengthen 

democratic accountability. 

Successful Frameworks etc. don’t just answer the questions, “What works?” and “What works 

where, when, and for whom?” They also answer the questions, “What works better and why?” 

and “How can we accomplish the same or more at a lower cost?” In addition, they use data and 

other evidence to inform the selection of priorities, to decide the ambitiousness of goals, and to 

1996 The Promise of Evidence-Based Policymaking
Metzenbaum comments to Commission on Evidence-Based Policymaking – Dec. 14, 2016 p. 3 

design the path to progress. Plus, they use data and evaluations to improve service (including 

regulatory) quality to make interactions with government more understandable, timely, 

predictable, and fair and to prevent unwanted side effects. 

Successful frameworks consider and design what and how to collect, store, analyze, present, and 

share evidence to make it more useful to more users. In addition, they take great care to make 

sure those being measured do not feel overly motivated to manipulate or implode the 

measurement system. 

The remainder of the answer to this questions provides 2 lists, one of likely users and one of 

kinds of uses, as well as 10 short cases that illustrate both useful, constructive, purposeful data 

and evaluation Frameworks etc. in the U.S. government and in non-federal organizations, as well 

as problematic Frameworks etc. 

• The first three cases (the U.S. National Highway Traffic Safety Administration’s approach to 

traffic fatality data, the U.S. Coast Guard’s use of data to work with a high-risk sector to 

reduce sea-based fatalities, the multi-decade effort of the U.S. Department of Health and 

Human Services using data to set goals to catalyze national and sub-national action to improve 

human health) address a common Federal government challenge: preventing bad things from 

happening and keeping costs as low as possible when they do. The first and third focus on 

specific problems, while the second focuses on a specific sector. Lessons from these three 

Frameworks – all of which count and characterize harmful incidents in a way that allows 

assessment of their causes, patterns, and relative import – can be applied to the data and 

evaluation systems of the large number of Federal programs that try to prevent and reduce 

harmful incidents such as fires, accidents, explosions, spills, non-compliance (inspection 

findings), and even poor customer/constituent interaction. More challenging (but still 

measurable and manageable by counting and characterizing near misses and early warning 

indicators) are programs that try to prevent low-frequency, high-consequence events, such as 

major chemical accidents. Even more challenging are programs that try to reduce hard-tocount harmful incidents, such as drug-running and human trafficking, although progress is 

being made even in these areas using predictive analytics comparing what should be (e.g., 

registered boat movement) and comparing with what is (observed boat movement.) 

• The fourth case describes efforts by one regional office of the U.S. Environmental Protection 

Agency to use data, and a clear, well-framed outcomes-focused goal to accelerate the cleanup 

of the Charles River in Massachusetts. In this case, observational data (water quality) is used 

rather than incident data and the data are collected by volunteers. 

• The fifth and sixth cases use transactional data, sales data in a supermarket and timeliness data 

for a package delivery company, while the seventh uses data that not only tracks but also 

supports the work of welfare caseworkers. 

• The eighth, ninth, and tenth cases describe unsuccessful Frameworks etc. Case 8 discusses 

why organizational report cards failed for hospitals, education, and motor cycle helmet laws, 

while the ninth and tenth cases briefly touch on less successful data collection efforts for Head 

Start and data analysis efforts for managing federal human resources. 

My response to this question is long, and I apologize for that, and assure you that my answers to 

the other questions are shorter. I nonetheless hope you will give these cases serious attention. , 

They support many of the other responses and my recommendations. Also, my overriding 

objective with these cases is to underscore the need for the Commission, in its recommendations, 

The Promise of Evidence-Based Policymaking 1997
Metzenbaum comments to Commission on Evidence-Based Policymaking – Dec. 14, 2016 p. 4 

to pay careful attention not just to the what and how of data and other evidence, but also to the 

users (both inside and outside government), uses, and mechanisms that motivate the generation 

and use of federal data and other evidence. 

List 1: Likely Users 

Federal information systems tend to be organized around agencies, not people, problems, or 

places. To make data more useful to more people to solve problems and pursue opportunities, 

federal agencies, working individually and across agencies, should adopt human-centered design 

principles and give intentional and continual attention to identifying, connecting with, and getting 

feedback from key audiences for federal information throughout all phases of information 

management. 

We often think about program managers, policy makers, and budget officials using data but data 

users also and especially need to include: 

• Those on the front line delivering public goods or services, in government and in delivery 

partners, including grantees in other levels of government and non-profit organizations; 

• Supervisors, regional managers, suppliers, technical assistance providers, trainers, logistical 

support, and others who strengthen or weaken front-line capacity; 

• Goal allies who want to advance the same or similar goals to the ones government seeks to 

advance or who want government to embrace other goals, such as the global sustainable 

development goals; and 

• Those who receive goods and services from government, especially those who can choose 

among providers or the timing or location of receipt. 

List 2: Kinds of Uses 

Government agencies should be expected to analyze, visualize, and disseminate their data and 

evidence to make them useful for multiple purposes, including: 

• detecting and understanding problems and opportunities; 

• informing priorities; 

• framing and setting goals; 

• supporting the search for effective and ineffective practices to speed uptake of the former and 

discourage repetition of the latter; 

• triggering discovery of the causal factors government and its delivery partners can influence 

to inform treatment and delivery design; 

• helping front-line workers and others learn from their own and others’ experience, while 

motivating continual improvement and collaboration on measured trials to discover better 

practices; 

• enlisting and engaging goal allies; 

• motivating continual improvement; 

• informing individual and organizational choices; 

• supporting private sector decision-making, thriving markets, and entrepreneurship; and 

• strengthening accountability to the American people by building understanding of what 

government tries to do, why, and how well. 

1998 The Promise of Evidence-Based Policymaking
Metzenbaum comments to Commission on Evidence-B etzenbaum comments to Commission on Evidence-Based Policymaking – Dec. 14, 2016 ased Policymaking – Dec. 14, 2016 p. 5

Case 1: The National Highway Traffic Safety Administration 

The experience of the federal road (now highway) program illustrates the potential of measurement as a tool of influence. When the 

first federal road office was established in 1904, it made information the center of its strategy for working with states. It gathered 

information about road conditions, mileage, and program characteristics in every county and conducted experiments to find effective 

road- and bridge-building practices and then frequently disseminated its findings to other levels of government. Information 

collection, analysis, and dissemination was and is a core federal highway function. In 2000 the Federal Highway Administration 

modernized its knowledge management role by launching online versions of its publications and databases to reach more interested 

users. 

Congress requires the secretary of transportation to report annually on the condition and performance of the nation’s roads but does 

not mandate inclusion of state-specific data in the annual report. The Federal Highway Administration opted to establish state 

reporting requirements through regulation. 

Despite the absence of federal law mandating state reporting, states have willingly delivered data to the federal government for more 

than fifty years. Why? According to one agency official, “We have been doing highway statistical information for 50 to 60 years, and 

have turned it back [to the states] from the beginning. We have always done some value-added work when we turned it back to the

states. It lets them see what other states were doing, and they see the data as a really valuable resource.” By functioning as a 

knowledge manager serving states and localities, the federal road agency built a performance-improving partnership with states that 

has flourished more than 100 years. 

As automobile use increased, so did accidents. In 1966 Congress responded by adopting a new federal goal: highway traffic safety. It 

required every state to establish a highway safety program in accordance with uniform federal standards. The National Highway 

Traffic Safety Administration (NHTSA) funds state employees in every state to review and code incident reports, such as police 

crash and coroner reports, collected for local purposes. State staff record data about traffic-related fatalities before, during, and after 

each accident, noting key characteristics of physical, social, and environmental conditions associated with the accident such as the 

state of the operator, type of equipment, and accident costs. They submit the information to a national highway fatality database. A 

dozen states voluntarily supplement the NHTSA’s fatality database with their own data about nonfatal crashes. 

The NHTSA not only supports and collects measurement, it also functions as the scientist in Brandeis’s laboratories of democracy. It 

studies the data it collects to look for patterns of problems, such as accident levels correlated with driver age and alcohol use. It 

determines the most prevalent problems needing attention. It also looks for anomalies and tries to understand their causes. For

example, the NHTSA helped one state understand why it had a higher right-angle crash rate than other states: as it turned out, more 

driveways in the state feed directly onto major thoroughfares than in other states. When states change their laws, the NHTSA 

compares changes in fatality rates in changed and unchanged states. Studying the effects of changes in state laws enabled the 

NHTSA to discover that when states allow police to pull people over to check safety belt use (instead of checking only when police 

stop drivers for other reasons), it increases belt use and lowers fatality rates. 

The NHTSA also uses measurement to help states find and replicate effective practices. North Carolina identified an interesting 

program in Canada that combined a media campaign with a “stop-and-check” blitz to increase seat belt use. North Carolina 

replicated the program, with good results. South Carolina wanted to try it but lacked primary enforcement (“stop-and-check”) 

authority, so the state sought help from the NHTSA to adapt the North Carolina program. The NHTSA developed audience-focused 

outreach materials to increase public awareness of the importance of wearing safety belts, including sample materials packaged for 

key target audiences such as schools (for example, morning announcements), law enforcement officials, parents, and others. The 

adaptation worked; after adoption, belt usage increased 9 percent in South Carolina and other southeastern states. The NHTSA then 

rolled the approach out nationally, simultaneously promoting state adoption of primary enforcement laws. It used grants to recruit 

volunteer states to participate in a controlled, measured experiment. Ten states tested the NHTSA Click It or Ticket campaign, four 

states served as a control group that did nothing, and four states tested programs of their own design. The NHTSA funded observers 

to measure belt use before and after the campaign, using a common measurement methodology. Belt use increased 8.2 percent in full 

implementation states, 2.7 percent in states using programs of their own design, and 0.5 percent in the control group of states.

The NHTSA’s data-rich, audience-focused work with states and localities dramatically improves outcomes. Automobile fatality rates 

have fallen in all but two of the past twenty-five years, and safety belt use climbed from 58 percent in 1994 to 81 percent in 2006. 

From: Shelley H. Metzenbaum, “From Oversight to Insight: Federal Agencies as Learning Leaders in the Information 

Age,” in Timothy J. Conlan and Paul L. Posner, editors, Intergovernmental Management for the 21st Century, Brookings, 2008, pp. 

209-242. Footnotes are available in original text, available upon request. 

After decades of decline, traffic fatality rates increased in 2012 and again in 2016. Work is underway to understand the 

causes of that increase and how to address it. 

NHTSA occasionally conducts measured trials, such as a multi-year trial to test a communication/enforcement campaign to 

reduce distracted driving, tested first in 2 small communities and since tried in 2 larger ones. This trial complemented many other 

studies diagnosing the causes and importance of the problem, as well as possible prevention methods. 

The Promise of Evidence-Based Policymaking 1999
Metzenbaum comments to Commission on Evidence-Based Policymaking – Dec. 14, 2016 p. 6

Case 2: U.S. Coast Guard 

In 1994, the U.S. Department of Transportation (DOT) piloted a project at the Coast Guard to focus its management system on 

performance outcomes, such as safety and health, rather than activity goals, such as the number of inspections. Within one year, 

the Coast Guard halved the fatality rate of towboat workers. By redefining its goal from measuring activities (e.g. vessel 

inspections) to measuring outcomes (worker fatality rates), the Coast Guard was able to enter into a partnership with the industry it 

regulates to fix the problem. When it pursued activity goals, agency progress imposed direct costs on the regulated community 

without a clear purpose. When it shifted to an outcome-focused goal, agency successes aligned more closely with those of the 

industry, its workers, and the public.... 

The Coast Guard pilot performance management project on marine safety, security, and environmental protection opted for fiveyear goals. The longer time frame afforded the organization the ability to choose challenging stretch targets. It adopted a goal of 

reducing accidental deaths and injuries from maritime casualties by twenty percent. The longer time frame also allowed greater 

flexibility in the selection of strategies to meet the goal....

By breaking the fatality rates for all vessels down into fatality rates for different types of vessels, the Coast Guard was able to 

determine which maritime sectors had the highest risks. This information allowed the Coast Guard to design an intervention 

strategy attentive to the culture of the industry group whose actions needed to change. It also allowed it to direct resources to the 

areas with the highest potential “safety return on investment.” And, in pursuit of the highest safety return on investment, DOT was 

subsequently able to shift some attention from commercial vessels to passenger vessels because so much progress had been made 

toward improving commercial vessel safety.... 

From Kennedy School Executive Session on Public Sector Performance Management, “Get Results through Performance 

Management: An Open Memorandum to Government Executives” Harvard Kennedy School of Government, 2001. As the Coast 

Guard/American Waterways Operators 2012 and 2016 report and industry trade press article attest, this outcomes-focused, datarich safety partnership continues to drive down waterway fatalities. 

the U.S

2000 The Promise of Evidence-Based Policymaking
Metzenbaum comments to Commission on Evidence-Based Policymaking – Dec. 14, 2016 p. 7

Case 3: U.S. Healthy People (page 1 of 2)

... The federal government issued the first Healthy People report in 1979, providing a health vision for the country by setting 

five specific national outcome targets, one each for five distinct age groups, to be achieved within ten years. This goal-setting 

report was followed one year later by a report setting specific targets for fifteen priority program areas, such as immunization 

and accident prevention, to reach the five national goals. The federal government has updated the health goals and objectives 

for the American people each successive decade, setting targets for the decade to follow.

Healthy People goals and objectives are set based on a review of the evidence about deaths and illness, their causes, and the 

effectiveness of prevention efforts. The reports select high-level goals, such as the current Healthy People goals of increasing 

quality and years of healthy life and reducing health disparities, and specific objectives for specific focus areas. Data on risks 

associated with illness and death guide the selection of objectives and focus areas. Health care system problems, behavioral 

factors, environmental hazards, and human biological factors are also considered in the selection of objectives. The reports 

classify preventable risks and tally them to find the best opportunities for risk reduction. 

Healthy People goals do not have the force of law, only the power of persuasion. Nonetheless, they have proved contagious, 

guiding priorities, informing decisions and influencing government spending. Other federal agencies, states, localities, and the 

private sector have embraced these goals. The New York City Department of Health and Hygiene uses Healthy People goals 

to guide and gauge its own performance. Forty-four states, the District of Columbia, and Guam, have replicated the Healthy 

People model and adopted their own health promotion and disease prevention objectives to guide local health initiatives. 

Congress incorporated Healthy People objectives into several laws and programs, including the Indian Health Care 

Improvement Act, the Maternal and Child Health Block Grant, and the Preventive Health and Health Services Block Grant. 

Indeed, the positive congressional response to Healthy People goals suggests that federal agencies can use goal setting as a 

way to start (or continue) a conversation with Congress about appropriate priorities. 

The federal government did not assume other levels of government would embrace Healthy People targets; it worked hard to 

build support. It offered technical assistance to encourage states and localities to undertake their own tailored Healthy People 

efforts and created the Healthy People Consortium, made up of 350 national membership organizations in addition to state and 

territorial health agencies. 

The Healthy People initiative also stimulated nongovernmental financial support to advance its goals. The Robert Wood 

Johnson Foundation, for example, awarded a grant in 1997 to direct the attention of American businesses to Healthy People 

goals. A Healthy People Business Advisory Council was created to encourage attention to Healthy People goals in the 

workplace. 

The intergovernmental trajectory of one Healthy People objective suggests how federal goal setting can influence others. The 

1979 Healthy People report identified cigarette smoking as the single most preventable cause of death and adopted a smoking 

cessation target. Since that time and without central coordination, other federal agencies, states, and local governments have 

adopted a wide variety of actions to prevent smoking. These include taxes, lawsuits, warning labels, and bans on smoking in 

public places. Uncoordinated goal-focused intergovernmental action, led at different times by different levels of government, 

has driven U.S. smoking levels down dramatically since the mid-1960s, when the surgeon general first issued a report warning 

of the dangers of smoking. 

(Continued on next page.)

The Promise of Evidence-Based Policymaking 2001
Metzenbaum comments to Commission on Evidence-Based Policymaking – Dec. 14, 2016 p. 8

Case 3: U.S. Healthy People (page 2 of 2) 

Well-selected federal goals, these vignettes suggest, have a powerful persuasive effect when they deal with issues 

that concern people and are backed by evidence showing the relative importance of a problem and the existence of 

effective prevention practices. They inspire effort and investment by others. Experience with the Healthy People 

initiative suggests that federal agencies can use the federal bully pulpit to set goals that persuade, even without 

incentives. 

Is Healthy People an exception or does it offer a model for other policy areas? Arguably, four replicable attributes 

of the initiative caused states and localities to align voluntarily with federally nominated objectives. First, its goals 

focused on issues that concern the public. Second, goal selection was informed by accumulation, analysis, and 

publication of data about the relative seriousness of problems needing attention. Third, Healthy People reports tally 

not only problems but also their preventable causes, suggesting a path for problem reduction. Fourth, the federal 

government used the goals to reach out broadly to recruit experts and implementers whose independent actions and 

decisions could improve health outcomes. 

It has been suggested that Healthy People is unique because the health field is rich with data and is an area with 

high goal congruence across government levels. However, Healthy People lacked data for about one-third of the 

objectives it set when it started. A decade later, it had reduced that gap to 20 percent because agencies started to 

generate the data they needed. In addition, data shortages in federal agencies are not always as desperate as 

claimed. Many agencies collect reams of data they never analyze, forgoing opportunities to understand the relative 

import of problems, their causes, and prevention possibilities. 

With regard to concern about goal congruence, intergovernmental goal dissonance undoubtedly exists. Differing 

values and the cost of goal pursuit make debates about government’s goal selection and the appropriate portfolio of 

goals common. Those conflicts intensify across levels of government, especially when goals set by one level 

impose costs on another. Goal dissonance is less an issue with hortatory federal goals. Contention rises when 

Congress makes state or local goal adoption mandatory and threatens penalties. It can also rise when goal adoption 

is required as a new condition added to existing intergovernmental grants routinely awarded every year. 

From: Shelley H. Metzenbaum, “From Oversight to Insight: Federal Agencies as Learning Leaders in the 

Information Age,” in Timothy J. Conlan and Paul L. Posner, editors, Intergovernmental Management for the 21st

Century, Brookings, 2008, pp. 209-242. Footnotes are available in original text, available upon request. HHS is 

currently working on setting 2030 goals. Based on data about causes of health problems, it also continues to set 

specific problem-solving goals, such as the goal to reduce harmful hospital-acquired conditions, embraced in the 

Partnership for Patients campaign. 

2002 The Promise of Evidence-Based Policymaking
Metzenbaum comments to Commission on Evidence-Based Policymaking – Dec. 14, 2016 p. 9

Case 4: U.S. EPA Cleaning Up the Charles River in Massachusetts 

In 1995 on the eve of the world-famous Head of the Charles regatta, EPA New England’s Regional 

Administrator John DeVillars publicly announced a goal: the lower Charles River would be swimmable in 

ten years. At the time, the river was so badly polluted that boaters who fell in were advised to get a tetanus 

shot. 

DeVillars and his staff realized that EPA would need better data to figure out how to meet this ambitious 

target. Fortunately, the non-profit Charles River Watershed Association (CRWA) had recently begun a 

volunteer monitoring effort. CRWA had trained volunteers to collect fecal coliform and dissolved oxygen 

samples, and they collected them every month at 37 points along the 80-mile stretch of the river. CRWA 

posted these data almost immediately in an online spreadsheet. EPA attorney Ken Moraff recognized that 

these data were more “actionable” than anything else EPA or the state had. Although volunteer-gathered, 

they were valid enough to trigger focused follow-up questions. 

Each month, the EPA team analyzed the CRWA data to look for anomalies and patterns. When found, they 

tried to understand the likely cause. In one case, when EPA saw a downstream reading worse than an 

upstream one that could not be explained by a permitted discharger between the two monitoring points, EPA 

called the Boston Water and Sewer Commission (BWSC) to see if it could explain the anomaly. BWSC 

could not, so the two teams joined together to “walk the pipes.” Doing so, they discovered an illicit hookup 

to the storm drain sending untreated waste that should have been directed to the sanitary sewers for 

treatment flowing instead through the storm sewers and directly into the Charles. 

The EPA team wondered if more illicit hook-ups existed. Brainstorming, someone suggested lifting 

manhole-covers over storm sewers on dry days to look for running water. EPA asked Massachusetts cities 

and towns to do this and to eliminate illicit hook-ups they found. This worked remarkably well. Within five 

years, the Charles was safe for boating 92% of the time (up from 39%) and safe for swimming 59% of the 

time (up from 19%). 

EPA did not meet its ten-year target. What is interesting is that this has never become a political issue. 

Perhaps that is because EPA shares its progress, problems, and strategies with the public every year, usually 

around Earth Day. One sign of the political viability and sustainability of this transparent goal-setting, datarich approach is that multiple regional administrators working for several U.S. Presidents from different 

political parties have continued the goal of a swimmable Charles. In September 2014, nineteen years after 

the goal was first set, EPA released its annual grade for the river, an A-. The river was safe for boating 96% 

of the time, and swimmable 70% of the time, the best result to date. 

It is hardly intuitive to think of water quality data as an effective way to measure compliance and 

enforcement activities, but the Charles River example shows just how powerful it can be – stimulating 

innovative thinking and revealing ways to improve. Beyond that, the case shows that useful measurement is 

not always costly to gather and can come from multiple sources, not just from government. 

From “Environmental Compliance and Enforcement Measurement: Why, What, and How? 

Forthcoming in Markell and Paddock, Ed. (Edward Elgar: forthcoming.) This excerpt was adapted and 

updated from “Measurement that Matters: Cleaning Up the Charles River,” in Donald F. Kettl, ed., 

Environmental Governance: A Report on the Next Generation of Environmental Policy (Brookings: 20020, 

pp. 58-117. Footnotes are available in original text, available upon request. This experience has informed 

the design of EPA’s nationwide Urban Waters program.

The Promise of Evidence-Based Policymaking 2003
Metzenbaum comments to Commission on Evidence-Based Policymaking – Dec. 14, 2016 p. 10

Case 5: Engaging Employees in Controlled Trials at a Spanish Supermarket 

Some well-run private sector companies encourage what one researcher describes as a “scientific 

mind” in all employees, encouraging every worker to generate hypotheses, conduct experiments, 

collect data, review results, and implement change. When more employees possess a “scientific 

mind” and combine that know-how with a robust performance measurement system, it encourages 

ongoing innovation and improvement, as this private sector example illustrates. “An employee 

suggested it would be better if there were a package of stew vegetables near the meat section so that 

the ... customers could get all the ingredients for stew quickly. When Mercadona gets a suggestion 

like that from an employee, it first tests the idea at one store to quantify its effects on various 

performance measures such as number of transactions, kilos of products sold, or reductions in kilos of 

waste generated. If there is an improvement in performance, they test it in an area... If the 

performance improvement is still there and if higher management supports the innovation, then 

Mercadona rolls it out across the entire chain.” 

 

 From Zeynep Ton, The Good Jobs Strategy (New Harvest: 2014), p. 57 

Cases 6 and 7: Information that Informs Front-Line Decisions – UPS and Case 

Commons/Casebook for Child Welfare 

Making information available and understandable so people in the field can readily access and use 

it when and where they need it greatly contributes to the success of data systems and 

complementary evaluations and root cause analyses. Consider two example: the private sector 

United Parcel Service (UPS) and Case Commons/Casebook, the child welfare information system 

improvement effort of the Annie E. Casey Foundation, initially working with the state of Indiana. 

Case 6 - UPS. UPS regularly uses data and tests new practices to increase productivity, profits, 

driver pay, and safety levels. It analyzes data from drivers, their handheld devices, and their trucks 

and then returns that information to drivers in a way that helps them find faster routes, anticipate 

dangerous dogs, and avoid lost packages. At the same time, it collects new data from drivers to 

build company and colleagues’ knowledge. In addition, UPS constantly asks questions, observes, 

brainstorms, and then works to test, assess, and adjust different designs for better equipment, such 

as smart key fobs and streamlined doors, and smarter practices, such as better driving behaviors. 

Complementing this technology is a negotiated union agreement preventing UPS from using the 

information it collects solely from technology for disciplinary purposes except in cases of 

dishonesty, a commitment to keep workers informed about the data being collected, and driver pay 

pay levels that have risen along with corporate financial gains from analyses. Sources: NPR, 

Mental_Floss.

Case 7 - Case Commons/Casebook. The Annie E. Casey Foundation is working to improve data 

systems and data used to help children in foster care. Reaching out to frontline social workers and 

state program managers to understand their information needs and work problems, it saw a problem 

in need of fixing. Caseworkers often write out their case notes in notebooks; if they move on, the 

notebooks go with them and their successors are left without the child’s case history. Even when 

caseworkers stay, though, the state cannot look across caseworkers’ notes to detect serious or 

common problems or find promising practices worth testing and promoting for broader adoption. 

Working closely with frontline workers as well as state program managers in Indiana, the 

Foundation launched Case Commons to design, test and refine an electronic notebook and data 

system that serves caseworkers and also generates information useful to supervisors, program 

managers and, eventually, researchers and policy makers. Early returns are promising, indicating 

that, at a minimum, the systems are improving interaction levels between caseworkers and families. 

2004 The Promise of Evidence-Based Policymaking
Metzenbaum comments to Commission on Evidence-Based Policymaking – Dec. 14, 2016 p. 11

Case 8: Hospitals, Nursing Homes, NHTSA, and Military 

Experience Suggest What It Takes to Make a Data Systems Last 

As William Gormley and David Weimer found in their study of organizational report cards, even those as 

illustrious as Florence Nightingale, for hospitals, and Horace Greeley, for schools, could not create 

sustainable data systems because constituencies fearful of being measured were able to organize to 

dismantle the measurement system, without encountering a sufficiently strong oppositional constituency.

NHTSA similarly encountered opposition when it tied state data on motorcycle accidents to high stakes 

incentives, restricting its ability to influence state helmet use laws. The military, in contrast, was able to 

build a comparative educational performance measurement system that survived over time, because it so 

strongly needed comparative student educational readiness data to meet its hiring needs, producing data 

that other well-organized political voices such as colleges and universities also deemed useful.

Organizations sometimes so fear the consequences of measurements that they organize to dismantle or 

prevent the establishment of measurement systems. Data system destruction tends to occur most 

frequently when a poor relative standing threatens those being compared and the political power of those 

threatened by comparison exceeds that of those using comparative information. This has frequently been a 

problem plaguing recurring efforts to introduce measurement into hospitals, education, and the 

environment. 

For comparative organizational measurement systems to survive, the political strength of the audience 

using the measurements needs to exceed that of those at risk by being compared. Agencies will be well 

served if they think about how to strengthen the political power of the consumers of measurement as a 

strategic issue so that measurement can survive and effectively motivate, illuminate, communicate, and 

inform choices. Also, measurements are infuriating rather than performance-improving when inaccurate 

information is used to guide consumer choices, regulatory action, or program or provider funding 

decisions. 

From Shelley Metzenbaum, “Performance Accountability Expectations: The Five Building 

Blocks and Six Essential Practices,” IBM Center for the Business of Government, 2006, citing research 

by Gormley and Weimer, Organizational Report Cards (Harvard: 1999) on hospitals, nursing homes, and 

the military. On motorcycle helmet laws, see “Strategies for Using State Information: Measuring and 

Improving Performance,” IBM Endowment for the Business of Government, December 2003. 

The Promise of Evidence-Based Policymaking 2005
Metzenbaum comments to Commission on Evidence-Based Policymaking – Dec. 14, 2016 p. 12

Cases 9 and 10: Lots of Data Not Used or Underutilized– Head Start and Federal Employee Statistics 

Case 9 – Head Start. [Until new regulations were promulgated in the fall of 2016, Head Start grantees were monitored 

on their compliance with 1,400 requirements, the Head Start Program Performance Standards.] The Designation Renewal 

System (DRS), mandated by the Improving Head Start for School Readiness Act of 2007 (Head Start Act), creates a form 

of accountability by requiring grantees that hit one of seven “triggers” to compete to renew their grants. 

[Unfortunately, little analysis of the data pertaining to these 1400 standards has been done, except to determine 

compliance and trigger a requirement that a grantee compete to renew. One likely reason that so little analysis has been 

done is that that the standards are primarily structured as yes/no questions, rather than actual data, and must be 

complemented by extensive paper documentation of meetings, plans, and other compliance requirements.] ... Only the 

last requirement on the trigger list, CLASS scores, emphasizes program quality beyond mere compliance, and focuses on 

a very narrow domain of Head Start performance. Further, the DRS does not differentiate between deficiencies that 

indicate serious shortcomings in programs’ health and safety practices, finances, or quality and deficiencies that reflect 

minor compliance problems. As a result, designation renewal has intensified the culture of compliance in Head Start, 

rather than encourage analysis of the data to find areas of strength and weakness. 

Designation renewal is grounded in sound principles — identify grantees that consistently fail to achieve desired results; 

take action to correct underperformance; and, where necessary, create opportunities for higher performers to replace the 

underperforming grantees. Data and reports from the field also suggest that designation renewal has led to the 

replacement of historically underperforming grantees — roughly 5 percent of all Head Start grantees have lost their grants 

in the past three years — and has stimulated improvement in underperformers. But the system’s reliance on imperfect and 

inadequate measurement has also resulted in identifying some high-performing grantees for competition, while failing to 

identify others that are low-performing. 

Further, the DRS ... identifies only those programs that fall short of standards, not exemplary performers.... To really 

support continuous improvement, Head Start needs a fairer and more results-based and robust way of measuring grantee 

quality. This is a necessary precondition for both identifying and learning from programs that are producing exemplary 

results, and for achieving the intended goals of the DRS. ...

Adapted from Mead and Libetti Mitchell, Moneyball for HeadStart: Using Data, Evidence, and Evaluation to 

Improve Outcomes for Families and Children (Results for America, Bellwether Education Partners, National Head Start 

Association, The Volcker Alliance: 2016), supplemented by commenter’s notes based on complementary information.

Case 10 – Federal Employee Statistics. Reams of survey and transactional data are collected about federal employees, 

including information about job openings, applicants, screening, hiring, retention, pay, promotion, employee views, and 

retirement. With the exception of the federal employees’ views, however, little of these data are routinely analyzed to find 

ways to improve, nor are they communicated in ways that will engage others in solving problems, pursuing opportunities, 

or motivating action. One noteworthy exception is the Federal Employee Viewpoint Survey (FEVS). Its administration 

was transformed over the last eight years from a statistically valid biennial snapshot of the viewpoints for each major 

agency and department that answered a question that was not very “actionable” -- whether or not progress was being 

made at the agency level on each question every 2 years – to a far more useful resources done annually at far great detail. 

The survey is now administered to everyone in all but the smallest organizational units of every agency and department in 

the federal government or to a statistically valid sample for each unit. The shift from a bi-annual to an annual survey and 

to statistical validity for organizational units as they are managed has rendered the FEVS a powerful management and 

motivational tool. It both enables and presses agencies to identify areas needing attention, and points to places where 

progress is being who may have practices others want to adopt. External attention to the FEVS from the Partnership for 

Public Service, which releases its own annual report on the best places to work, as well as the U.S. Office of Management 

and Budget (OMB) and the Performance Improvement Council may have encouraged increased attention to and analysis 

of the FEVS, as well as increased support to help agencies access and use the FEVS data, by the U.S. Office of Personnel 

Management (OPM). In contrast, a vast array of other rich HR data that could spotlight HR problem and areas with 

progress – such as cross-agency comparisons of employee application rates, hiring times, and retention rates for similar 

positions or HR staffing ratios – has yet to be done routinely. Perhaps this is because no one outside government, in 

Congress, or at OMB have yet pressed for these analyses to be done, at least not in a way that is being shared with the 

public, despite their high potential value to agencies. Source: commenter’s experience as former Associate Director for 

Performance and Personnel Management at the U.S. Office of Management and Budget. 

2006 The Promise of Evidence-Based Policymaking
Metzenbaum comments to Commission on Evidence-Based Policymaking – Dec. 14, 2016 p. 13 

2. Based on identified best practices and existing examples, what factors should be 

considered in reasonably ensuring the security and privacy of administrative and 

survey data? 

I can offer little expertise in this area, but recent efforts from Washington to develop a privacy 

modeling tool might be helpful: https://watech-beta.herokuapp.com/ 

Data Infrastructure and Access 

3. Based on identified best practices and existing examples, how should existing 

government data infrastructure be modified to best facilitate use of and access to 

administrative and survey data? 

To facilitate use and access to administrative and survey data, return data to data suppliers with 

value added through analyses, and help users do their own analyses by offering raw data, report 

generators, and analytic models and examples illustrating how to do and apply analyses. 

Also, support and, where necessary, create continuous learning and improvement communities 

(CLICs) that bring frontline workers, their supervisors, and others in the delivery system together 

with each other, with high-quality researchers, and with policy makers to learn from experience 

and discover new paths for improvement. 

 

4. What data-sharing infrastructure should be used to facilitate data merging, linking, 

and access for research, evaluation, and analysis purposes? 

In addition to supplying report generators and analytic models to states and others, as well as 

examples illustrating how to do and apply analyses, try building researcher/practitioner 

partnerships in a variety of ways. Increase academic stints in government agencies’ data shops 

using the Intergovernmental Personnel Act (IPAs), and create inter-agency analytic internships 

for students (who can career into a federal government position in any agency after working 320 

hours as an intern, if they have a B+ average, or after 640 hours, if they do not.) Also, consider 

testing research partnerships where government offices can turn to individual researchers for 

literature reviews. 

Also, build problem-focused websites similar to HealthyPeople.gov and http://www.nhtsa.gov/, 

both of which provide ready access to data, trends analyses, evaluations, and promising practices 

worth further testing. Both make data available nationwide and at the sub-national level, 

facilitating the search for positive outliers whose practices might be worth testing to produce 

comparable gains, as well as negative outliers to trigger the search for their root causes. 

In addition, increase Federal employees familiarity with APIs and their skills and knowledge 

about how and when to use them. 

5. What challenges currently exist in linking state and local data to federal data? Are 

there successful instances where these challenges have been addressed? 

As noted in some of the case studies and the writings cited, numerous challenges currently 

impede efforts to link state and local data to federal data. Aside from the push/pull between 

centralization and states’ rights that goes back to our founding fathers, states, local governments, 

and other grantees may resist linking their data to federal data for a variety of reasons. These 

include cost, disagreement about data standards and structure, recent investment in data systems 

The Promise of Evidence-Based Policymaking 2007
Metzenbaum comments to Commission on Evidence-Based Policymaking – Dec. 14, 2016 p. 14 

that the federal government wants to render outdated by establishing new reporting standards, 

disagreement about metrics, reporting burdens not counterbalanced by reporting value, fear of 

looking bad compared to others, fear of being penalized for poor performance or poor relative 

position, and, ironically, occasional fear of looking too good compared to other states, which 

might cause a state legislature to cut an agency budget or limit its authority because of concerns 

about interstate economic competitiveness. 

Ill-structured incentive systems, both the prospect of earning a bonus and the threat of penalties, 

can be especially problematic. As briefly discussed in Case 8, when the stakes get raised too high, 

those being measured often organize to dismantle the data system. In other cases, they game it. 

In truth, the push/pull between standardization and local discretion is not unique to government, a 

topic Zeynep Ton explores in her book cited in Case 5. 

A few cases have been offered here, most notably the National Highway Traffic Safety 

Administration and the Healthy People data sets, where local, state, and federal data have been 

linked, in part because of robust data collection framework was established (see, for example, the 

Haddon Matrix), and in part because the federal agencies collect data that are useful at the local 

level and also return data analyses back to data suppliers with value added through analyses. I am 

told that the Substance Abuse and Mental Health Services Administration (SAMSHA) may also 

be a successful instance of data and evidence sharing. 

6. Should a single or multiple clearinghouse(s) for administrative and survey data be 

established to improve evidence-based policymaking? What benefits or limitations 

are likely to be encountered in either approach? 

Data.gov seems like a good start as a clearinghouse for the nearly 200,000 federal government 

data sets. 

Despite a lot of hard work, except for PubMed.gov which although highly technical is 

nonetheless readily searchable and somewhat understandable even to the lay person, the evidence 

clearinghouses I have seen in other policy areas, both public and private, seem impenetrable for 

most audiences. Their primary audience seems to be researchers, not practitioners, but where can 

the interested teacher, principal, superintendent, social worker, inspector, benefits processor, etc. 

go to find relevant research and statistics? When I have asked practitioners that question, even 

those with Ph.D.’s in their field, most say they don’t have a good place to go. One Ph.D. K-12 

educator told me that he had tried the clearinghouses and found the New York Times more 

helpful in informing his curriculum purchases. Not surprisingly, the medical clearinghouses seem 

to be the most useful. 

Let me suggest testing out multiple clearinghouses, not a single clearinghouse, organized around 

multiple themes: 

• purpose (e.g., healthy people, thriving children, thriving veterans, employment and 

training, clean water); 

• processes (e.g., benefits processing, permits, regulatory compliance, risk reduction, r&d); 

• change mechanisms applicable across policy areas (e.g., awareness, understanding, 

acceptance, behavior change, contagion); 

• people or sectors (e.g., children, aviation). 

2008 The Promise of Evidence-Based Policymaking
Metzenbaum comments to Commission on Evidence-Based Policymaking – Dec. 14, 2016 p. 15 

HealthyPeople.gov may be a good model for a purpose-focused clearinghouse. Before embracing 

it in its entirety, however, the Commission should reach out to the people who work on it and 

who use it to determine its strengths, weaknesses, and planned next steps. 

Also, more closely integrate Data.gov, Performance.gov, and the social indicators in the 

President’s budget or elsewhere (e.g., sdg.data.gov) with each other and with the evidence 

clearinghouses. 

7. What data should be included in a potential U.S. government data 

clearinghouse(s)? What are the current legal or administrative barriers to including 

such data in a clearinghouse or linking the data? 

Let me suggest that this question be asked and answered through a systematic, annual process by 

revamping the annual Information Collection Budget (ICB) required by the Paperwork Reduction 

Act (PRA.) Historically, the ICB has focused on reducing the reporting burden placed on the 

public by the federal government. Information has been widely recognized as a major strategic 

asset over the last decade, yet the ICB continues to focus on information as a burden rather than a 

valuable asset. The ICB should be revised to, as its name implies, be a budget that reviews 

options and decides federal information management priorities within and across agencies. This 

change would fully align with the full set of legislated purposes of the PRA, including, as laid out 

in the law: 

• ensure the greatest possible public benefit from and maximizing the utility of information 

created, collected, maintained, used, shared and disseminated by or for the Federal 

Government; 

• improve the quality and use of Federal information to strengthen decisionmaking, 

accountability, and openness in Government and society; ... 

• strengthen the partnership between the Federal Government and State, local, and tribal 

governments by minimizing the burden and maximizing the utility of information created, 

collected, maintained, used, disseminated, and retained by or for the Federal Government; 

• provide for the dissemination of public information on a timely basis, on equitable terms, and 

in a manner that promotes the utility of the information to the public and makes effective use 

of information technology;... 

• ensure that information technology is acquired, used, and managed to improve performance 

of agency missions, including the reduction of information collection burdens on the 

public;”.... 

This revamp should also look at and align efforts to produce the Annual Inventory of Major 

Information Systems required by another law (FISMA) and the Enterprise Roadmaps agencies 

may be asked to do to prioritize legacy system upgrades and make sure they are user-centered and 

purpose-focused. 

8. What factors or strategies should the Commission consider for how a 

clearinghouse(s) could be self-funded? What successful examples exist for selffinancing related to similar purposes? 

If the ICB were actually treated as a budget, as suggested in my response to question 7, the 

options it raises and priorities it sets could be incorporated into the federal budget process, ideally 

with multi-year funding authority provided. In addition, as Cases 1 (NHTSA) and 9 (Head Start) 

suggest, grants can be a good mechanism for funding data generation, collection, analysis, and 

sharing, but this only works well when data are returned to data suppliers/grantees in ways that 

The Promise of Evidence-Based Policymaking 2009
Metzenbaum comments to Commission on Evidence-Based Policymaking – Dec. 14, 2016 p. 16

helps them make progress, such as by detecting problems and by identifying, through 

benchmarking (normalized comparisons) and other means, ways to do better. 

9. What specific administrative or legal barriers currently exist for accessing survey 

and administrative data? 

The need to keep both health information (HIPAA) and business information confidential is 

important, so it is neither surprising nor unwarranted to restrict their use. The problem is that 

agencies too often are overly restrictive, even to other agencies. Years ago, for example, the New 

Jersey environmental protection department tried to get health data from the state health 

department to inform the environmental agency’s enforcement, permitting, and rule-setting 

priorities. The environmental agency was unable to access the information and the health agency 

would not make the kind of analysis the environmental bureau wanted done a priority, precluding 

more impactful, efficient allocation of government resources. 

Agency policy can also be problematic. Selected academic researchers have long tried to 

negotiate with OPM to get access to federal human resource data with sufficient detail to do 

interesting, useful analyses. Last I talked to one of the leading researchers in this field, he had 

given up on the effort. 

It is worth noting that the lack of timely data can limit its value, even when available. This had 

been the case with OPM’s release of the FEVS data described in Case 10 (possibly since 

corrected.) The lag time between data collection and data release makes it hard for agencies to 

decide in which areas where they need to focus their HR actions to do better on the following 

year’s survey. To address this problem, some agencies have commissioned additional surveys. 

10. How should the Commission define “qualified researchers and institutions?” To 

what extent should administrative and survey data held by government agencies be 

made available to “qualified researchers and institutions?” 

First, let me suggest that you expand this question to ask not just about researchers and 

institutions, but also about the exchange and analysis of data across government agencies. (See 

New Jersey vignette in response to question 9.) 

Second, consider testing whether a user-centered approach would work as a way to qualify, 

researchers and institutions that want data. Have them answer several questions, including what 

they want to learn from the data, how they plan to share results of their findings, possible 

problems that could arise from their use of the data, and whether and how they plan to prevent the 

problems. Then, solicit selected members of the public and potential users and measured parties, 

not just the supplying agency, to react to the answers. I don’t know if this will work, but it seems 

worth a try. 

Third, consider flipping this question to ask and answer, “Who should not get specific federal 

data, what data should not be public, and why?” 

11. How might integration of administrative and survey data in a clearinghouse affect 

the risk of unintentional or unauthorized access or release of personally-identifiable 

information, confidential business information, or other identifiable records? How can 

identifiable information be best protected to ensure the privacy and confidentiality of 

individual or business data in a clearinghouse? 

2010 The Promise of Evidence-Based Policymaking
Metzenbaum comments to Commission on Evidence-Based Policymaking – Dec. 14, 2016 p. 17

I defer to the expertise of others on this subject. 

12. If a clearinghouse were created, what types of restrictions should be placed on 

the uses of data in the clearinghouse by “qualified researchers and institutions?” 

Clearly, an individual’s private health conditions/income/employment history/benefits 

history/taxes paid should not be discernable by employers, neighbors, family, friends, enemies or 

others without the individual’s permission. Researchers must de-identify data at a high enough 

level of aggregation to afford these protections before making their findings public. Similarly, to 

the extent possible, business secrets should not be made available to competitors, although tradeoffs between knowledge needed for public health and the need to keep business information 

confidential may be needed. 

You did not ask this question, but please think about the possibilities and possible consequences 

of allowing individuals to give permission to release their data, as so many of us now do when we 

agree to the terms and conditions of using a website? Should individuals similarly be allowed to 

give permission, up front, for researchers to use and possibly share information about them? 

Related to that, should individuals be allowed to authorize, up front, the use of their personal data 

if that use will benefit that person? Organizations such as Code for America are doing pathbreaking work improving government processes, such as its current work to improve the 

California Food Stamp application process. Should Code be allowed to ask Food Stamp 

applicants for permission to use their data to determine their eligibility for other programs, 

helping the individuals? Should they also ask applicants, at the time of application, if their data 

can be shared, or should a more paternalistic approach be used? 

13. What technological solutions from government or the private sector are relevant 

for facilitating data sharing and management? 

As Cases 6 and 7 illustrate, providing front-line workers, such as the foster care caseworker and 

the UPS driver, handheld devices that help them do their jobs and make decisions in real time 

greatly facilitates data sharing and management, especially when the central office analyzes the 

data and shares the insights with the front line worker, while supporting queries and decisionmaking of supervisors and higher level managers. 

Other promising technical solutions include: visualization (geographic and time-tagged), APIs, 

iterative user-centered design principles, remote sensing, electronic filing and applications, 

electronic diaries, and even electronic attendance keeping (which, I am told, proved very helpful 

in homelessness research.) All of these technologies make information potentially more relevant, 

useful, and affordable. 

Also, check out how Google shows use patterns at different locations by time of day and the day 

of the week, and imagine doing that at Social Security Offices or TSA clearance locations. 

Solutions need not always be technical, however. Case 5 about the Spanish supermarket chain 

suggests not only the value of a strengthening the scientific mind of front-line workers, but also 

the importance of structuring incentives in ways that encourage innovation and teamwork. The 

NPR story referenced in Case 6, about the UPS, elaborates on the analytic and experimental 

mindset of the central office data team, another critical factor, and suggests the need for such a 

team. The Coast Guard has similarly embraced the right skill set and mind set not just for tow 

boat worker safety, but also to reduce oil spills. For example, because it noted the timing of all 

The Promise of Evidence-Based Policymaking 2011
Metzenbaum comments to Commission on Evidence-Based Policymaking – Dec. 14, 2016 p. 18 

marine-based oil spills, one of its regional offices noticed that most spills occurred at night. After 

shifting some inspections, all of which had previously occurred during regular business hours, to 

the times when most of the spills were happening, the number of spills fell. 

14. What incentives may best facilitate interagency sharing of information to improve 

programmatic effectiveness and enhance data accuracy and comprehensiveness? 

My response to this question addresses two aspects of the question: what facilitates data sharing 

and what facilitates interagency data sharing. 

Making data, analyses, and evaluations useful, understandable, timely, and available at the likely 

point of use (which is much easier than in the past thanks to handhelds and screens everywhere) 

facilitates program improvement, especially, as Case 8 discusses, when those being measured do 

not fear the information will be used to punish them unfairly or when advocates for good data 

have more political power than those resisting. Making information not only readily available and 

understandable but also easy to supply (again, tapping new mobile technologies) enhances data 

accuracy. 

Unfortunately, much federal data collection, for grants but also for some regulatory and other 

programs, gives more attention to compliance and payment errors than to finding ways to do 

better and motivating improvement, as Case 9 illustrates. The Head Start program has long 

required grantees to fill out yes/no checklists on 1400 performance standards and submit 

documentation of meetings, plans, and other compliance requirements. It has not, however, 

collected or analyzed data about child health and development outcomes, safety incidents, or 

costs. This makes it difficult for grantees to learn from experience, their own and others, how to 

improve. Happily, recently released revisions to the Head Start performance standards aim to fix 

this problem, but critical information collection decisions must still be made by the federal 

government, grantees, and the vendors who sell information systems to grantees to enhance the 

usefulness of collected data. 

Other federal grant programs face similar challenges. Their reporting requirements fail to 

generate insights the front-line can use to make better decisions; their funding practices, often 

inadvertently, favor approving antiquated over more useful information systems; and 

government-wide policies for information collection complicate and impede needed adjustments 

to information collection practices. 

Interagency data sharing is facilitated by a clearly articulated shared goal (or goals), an easily 

found and accesses repository that shares or points to relevant information, frequent data-rich 

reviews, and someone steering the improvement effort (a goal leader.) See, for example, 

HealthyPeople.gov, multi-sectoral and interagency efforts to reduce homelessness; 

intergovernmental efforts to control inappropriate drug use; and the effort to clean up the Charles 

River. 

Data Use in Program Design, Management, Research, Evaluation, and 

Analysis 

15. What barriers currently exist for using survey and administrative data to support 

program management and/or evaluation activities? 

2012 The Promise of Evidence-Based Policymaking
Metzenbaum comments to Commission on Evidence-Based Policymaking – Dec. 14, 2016 p. 19 

A number of barriers impede the use of survey and other data (e.g., transactional, observational, 

performance measurement, process measurement.) These include fear, the absence of 

responsibility for analyzing and using data and evaluations, weak skills and capacity to use data 

and other evidence and interpret them accurately, the lack of awareness of data and evidence, 

access to and the affordability of data, a compliance mindset, and the lack of external drivers or 

motivators to analyze and use data comparable to the competitive pressures of the market place. 

For example, as discussed in Case 8 and the response to question 5, fear is one of the biggest 

barriers to using survey and administrative data. When people fear the data and evaluations will 

be used to penalize them, they often work to implode or manipulate the measurement system. 

Another barrier, as suggested in the discussion in Case 10 about underutilized federal human 

resources data, is the absence of responsibility to analyze data, use data analyses and other 

research findings in decision-making, and identify knowledge gaps to fill with new data set 

and/or evaluations/experiments. The naming of Chief Operating Officers in every agency and 

charging them with running quarterly data-rich reviews on priority goals, as required by the 

Government Performance and Results Act Modernization Act of 2010 (GPRA 2010), helps to 

address that weakness but only for a small set of government goals. The annual strategic reviews 

required by the same law and recently started are also expected to help close that gap. 

Much federal data collection, especially for grants but also for some regulatory and other 

programs, directs more attention to compliance and payment errors than to finding ways to do 

better and motivating improvement. 

16. How can data, statistics, results of research, and findings from evaluation, be best 

used to improve policies and programs? 

Past experience suggests a number of practices that encourage the use of data, statistics, and 

research results. These include: 

• Setting and managing clear, well-framed, resonant, outcomes-focused goals complemented by 

frequent and routine data-rich reviews led by organizational leaders and a regular cadence of 

public reporting that explains focus areas, strategies to drive progress, why both were chosen, 

progress, problems, and planned next steps; 

• User-centered design; 

• User-centered reporting, slicing and dicing data into league tables or using other ways to 

normalize data for fair comparisons (think Consumer Reports tables) to reveal trends, patterns, 

and divergences; 

• Supporting (and creating, where necessary) continuous learning and improvement 

communities (CLICs) that help grantees, field offices, and others learn from their own and 

others’ experience and encourage collaboration on problem assessment and measured trials to 

find better ways to do business;

• Making information available, discoverable, and usable while protecting security and 

respecting individual privacy while adhering to Open Data principles -- public, accessible, 

described, reusable, complete, timely, and managed post-released; 

• Visualization and presentation that provides context and makes it relevant and available at 

time of decision-making; 

The Promise of Evidence-Based Policymaking 2013
Metzenbaum comments to Commission on Evidence-Based Policymaking – Dec. 14, 2016 p. 20 

• Well-structured incentive systems, applying (and further developing) evidence about effective 

and ineffective motivational mechanisms1; 

• Recognition that data used for fast feedback, program adjustment, and to trigger focused 

follow-up such as root cause analyses need not be of the same quality as that used for 

enforcement actions and program evaluations, as Case 4 on the Charles River illustrated;

• Packaging materials to make them more useful to target audiences, as NHTSA has done. 

17. To what extent can or should program and policy evaluation be addressed in 

program designs? 

Similar to the way the Spanish supermarket chain in Case 5 and UPS in Case 6 integrate 

evaluations into their operations, federal agencies can and should integrate measured trials 

seamlessly into program operations for a wide variety of purposes. These can be as varied as 

testing to find unbiased ways to administer police exams to sending out regulatory compliance 

notices to behavioral change campaigns, such as NHTSA’s test of its campaign to reduce 

distracted driving, to testing different individual and organizational incentive structures. 

One challenge that should not be overlooked is figuring out a way to develop and share 

knowledge that is not unique to the policy area, but applies to processes, mechanisms, and 

audiences across different policy areas. For example, Wogalter’s and others’ research on 

warnings, risk communication, and human information processing, as well as the extensive body 

of research on cognitive bias, have long been integrated into FDA’s regulatory decision-making. 

This knowledge would likely be useful to many parts of government. 

18. How can or should program evaluation be incorporated into program designs? 

What specific examples demonstrate where evaluation has been successfully 

incorporated in program designs? 

The GPRA framework and timetable – including the quarterly reviews for priority and other 

goals, the annual strategic review and performance reporting and planning process, and the 

strategic planning process that takes place every four years – afford an excellent vehicle for 

integrating decisions about data and evaluation priorities into agency decision-making with a 

regular cadence. The Chief Evaluation Officer at the Department of Labor has regularly 

participated in these meetings, and has complemented them with office-specific research agendas, 

listing needs and priorities. 

The Cross-Agency Priority Goal goal-setting and quarterly review process can also be useful, as 

could a revamped ICB, as described in the response to question 7. 

19. To what extent should evaluations specifically with either experimental 

(sometimes referred to as “randomized control trials”) or quasi-experimental designs 

be institutionalized in programs? What specific examples demonstrate where such 

institutionalization has been successful and what best practices exist for doing so? 



Pink, Daniel (2011) Drive. New York: Riverhead. This book provides a good overview for the non-researcher on research findings 

about motivation and incentives affecting individuals. See also, the chapter on Building Block 5, in Metzenbaum, Shelley H. (2006) 

“Performance Accountability Expectations: The Five Building Blocks and Six Essential Practices,” Washington, DC: IBM Center for

the Business of Government. (http://www.businessofgovernment.org/sites/default/files/Performance%20Accountability.pdf)

2014 The Promise of Evidence-Based Policymaking
Metzenbaum comments to Commission on Evidence-Based Policymaking – Dec. 14, 2016 p. 21 

Medicine is, of course, one area where RCTs and quasi-experimental designs have been 

institutionalized. Part of the “secret sauce” here, I would argue, is that many on the front-line 

(doctors) keep up on the literature in their field, websites are readily available both for doctors 

and patients that share research findings, and many of the best medical journals report the 

information in a way that succinctly summarizes at the beginning of the article relevance and 

applicability. For example, “in line with the Knowledge Translation movement across the globe, 

the International Journal of Health Policy and Management has adopted an initiative by which all 

original articles are required to have Key Messages under two separate headings 

namely: Implications for Policy Makers and Implications for Public.” Similarly, JAMA (Journal 

of the American Medical Association) requires “a statement of relevance indicating implications 

for clinical practice or health policy... may also indicate whether additional study is required 

before the information should be used in clinical settings.”

The Promise of Evidence-Based Policymaking 2015
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tkz-jp6t

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0265

Comment on FR Doc # 2016-22002

Submitter Information

Name: Stephanie Mathisen

Address:

London, United Kingdom, EC1R 0DP

Email: smathisen@senseaboutscience.org

General Comment

See attached file(s)

Attachments

Sense about Science Commission on Evidence-Based Policymaking comments Docket

ID USBC-2016-0003-0001 2016 Dec 14

Attachment 1 Evidence transparency framework

Attachment 2 SaS-Transparency-of-Evidence-2016-Nov

Attachment 3 Missing-Evidence

2016 The Promise of Evidence-Based Policymaking
Docket ID USBC-2016-0003-0001 

Commission on Evidence-Based Policymaking Comments 

Response to overarching question 1: “Are there successful frameworks, policies, practices, and 

methods to overcome challenges related to evidence-building from state, local, and/or 

international governments the Commission should consider when developing findings and 

recommendations regarding Federal evidence-based policymaking? If so, please describe.” 

Sense about Science is the UK independent charity that challenges the misrepresentation of 

science and scientific evidence in public life. We advocate for openness and honesty about 

research findings, and work to ensure the public interest in sound science and evidence is 

represented and recognised in public discussion and policymaking. 

Our recent work has identified two significant problems relating to evidence building and use in 

policymaking which may be of interest to the Commission: a lack of transparency about the use of 

evidence in policy development, and that there is no comprehensive account of research 

commissioned by the UK government from external sources. We enclose three documents relating 

to these issues: a framework to assess the transparency of evidence used in policymaking; the 

report of the first ever assessment of whether the UK government is transparent about its use of 

evidence when developing policies; and the report of an inquiry into the delayed publication of 

research commissioned by the UK government. 

Transparency of evidence use in policy development 

In 2015, at the suggestion of the UK government’s What Works National Adviser (Dr David 

Halpern), Sense about Science, the Institute for Government (IfG) and the Alliance for Useful 

Evidence accepted the challenge of developing a rapid assessment tool to rate how well 

government departments use evidence in making policy decisions. Initial testing revealed a prior 

challenge: that it is often not possible to see what evidence has been used or the basis for the 

assumptions being made in a policy proposal. Transparency is a first and necessary step in 

enabling the quality of the evidence used by a government department to be assessed. To address 

this we developed a framework to assess the transparency of evidence used in policymaking. This 

was published by IfG in October 2015.1

The framework represents an approach to testing evidence transparency that can be applied 

rapidly, does not require subject matter expertise and allows comparison between different policy 

 

 

1 Show your workings: Assessing how government uses evidence to make policy, Jill Rutter and Jen Gold, Institute for 

Government (http://www.instituteforgovernment.org.uk/publications/show-your-workings) 

The Promise of Evidence-Based Policymaking 2017
areas and departments. The framework has been used2 by the UK parliament’s House of 

Commons Science and Technology Select Committee, which scrutinises government activity “to 

ensure that government policy and decision-making are based on good scientific and engineering 

advice and evidence”3 and will be used in Peru. 

Sense about Science, in partnership with IfG and the Alliance for Useful Evidence, then led testing 

and revision of the framework and used it to assess 13 domestic UK government departments’ 

transparency of evidence use.4 Transparency of evidence highlights good and bad practice and sets 

out what departments need to improve to perform well in the systematic ranking of departments 

we will perform next year. 

Publication of government commissioned research 

A spate of media stories in the UK about government commissioned research reports being 

suppressed or delayed, allegedly because the findings were politically awkward, led to concern 

among researchers and the public about: 

x Failure to make publicly funded research available to the public. 

x A lack of transparency about the basis of government decisions and the role that evidence 

played in reaching them. 

x A reduced willingness of researchers to assist in policymaking.

In response Sense about Science initiated an inquiry led by former High Court Judge Sir Stephen 

Sedley into the delayed publication of government commissioned research and into possible 

remedies.5 Sir Stephen found cases of research being suppressed. However, of far greater concern 

was the finding that no comprehensive account existed of how much research is commissioned by 

the UK government, nor of how much of it is published and when. Sir Stephen recommended that 

all government departments should record commissioned research in a standardised public 

register and report its publication so that this information is available, and continues to be 

available, to the rest of government, parliament, the research community and the public. 

 

 

2

 http://www.parliament.uk/business/committees/committees-a-z/commons-select/science-and-technologycommittee/science-and-technology-evidence-check-forum/ (accessed 13th December 2016) 3

 http://www.parliament.uk/business/committees/committees-a-z/commons-select/science-and-technologycommittee/role/ (accessed 13th December 2016) 4 Transparency of evidence: an assessment of government policy proposals May 2015 to May 2016, Sense about 

Science, November 2016 (http://senseaboutscience.org/wp-content/uploads/2016/11/SaS-Transparency-of-Evidence2016-Nov.pdf) 

5 Missing evidence: An inquiry into the delayed publication of government-commissioned research, the Right 

Honourable Sir Stephen Sedley, June 2016 (http://senseaboutscience.org/wp-content/uploads/2016/11/MissingEvidence.pdf) 

2018 The Promise of Evidence-Based Policymaking
We hope that the Commission will find the attached reports useful in its work. We would be happy 

to expand on any of the above points. 

Signed 

Dr Síle Lane 

Director of campaigns and policy, Sense about Science 

14th December 2016 

Attached: 

1. Evidence transparency framework (July 2016); Sense about Science, Institute for 

Government, Alliance for Useful Evidence; Appendix 2 in Transparency of evidence: An 

assessment of government policy proposals, May 2015 to May 2016, Sense about 

Science, November 2016 

2. Transparency of evidence: An assessment of government policy proposals, May 2015 to 

May 2016, Sense about Science, November 2016

3. Missing evidence: An inquiry into the delayed publication of government-commissioned 

research, the Right Honourable Sir Stephen Sedley, June 2016

Contact 

Dr Stephanie Mathisen, campaigns and policy officer, Sense about Science 

smathisen@senseaboutscience.org 

+44 (0)20 7490 9590 

Sense about Science 

14A Clerkenwell Green 

London, UK 

EC1R 0DP 

The Promise of Evidence-Based Policymaking 2019
2020 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 2021
TRANSPARENCY 

OF EVIDENCE:

An assessment of government policy 

proposals May 2015 to May 2016

2022 The Promise of Evidence-Based Policymaking
CONTENTS

1 Introduction 3

 1.1 Why look at transparency about evidence? 3

 1.2 The transparency framework 4

 1.3 The sample 5

 1.4 Summary of findings 8

2 Sections of the framework 9

 2.1 Diagnosis 9

 2.2 Proposal 12

 2.3 Implementation 15

 2.4 Testing and Evaluation 21

3 Cross-cutting issues 25

 3.1 Referencing and sharing work done 25

 3.2 Gesturing vs referencing 28

 3.3 Absent or weak evidence 31

 3.4 Values-based policies 34

4 Next steps 37

Acknowledgements 38

Appendix 1 Evaluating transparency of evidence 39

Appendix 2 The revised transparency framework, July 2016 43

Appendix 3 The original framework 45

Appendix 4 Policy documents reviewed 48

Appendix 5 Testable claims. When is evidence expected? 58

This appendix contains all the policy 

documents we reviewed, listed by department.

CONTENTS 2

The Promise of Evidence-Based Policymaking 2023
INTRODUCTION

1.1 Why look at transparency about evidence?

8LIRIIHJSVFIXXIVYWIERHWLEVMRKSJXLIIZMHIRGIMRTSPMG]QEOMRKMW[MHIP]EGORS[PIHKIHMRTSPMG]GMVGPIW8LIɄ

review of the Civil Service Reform Plan made a commitment to “publish more of the evidence base that supports 

policymaking”.

1

8LI3TIR+SZIVRQIRX%GXMSR4PERWIXSYXGLERKIWXSXLITYFPMWLMRKMRJVEWXVYGXYVIXLEX

would support this. This sits alongside wider moves to promote the use of evidence in policymaking, including the What 

;SVOWGIRXVIWERHXLI[MHIV;LEX;SVOWREXMSREPQSZIQIRXPIHF]XLI'EFMRIX3JƤGI&YXMXMWLEVHXSXIPP[LIXLIVXLSWI

MRRSZEXMSRWEVIɄMRGVIEWMRKXLIYWISJIZMHIRGIMRTSPMG]

-RXLI-RWXMXYXIJSV+SZIVRQIRX7IRWIEFSYX7GMIRGIERHXLI%PPMERGIJSV9WIJYP)ZMHIRGIXSSOJSV[EVHEWYKKIWXMSR

by the What Works National Adviser to develop a rapid assessment tool to rate government departments on the use 

of evidence behind policy decisions. Our approach to doing this was published in the report Show your Workings.2

-XɄIWXEFPMWLIHXLEXMRSVHIVXSIZEPYEXITSPMG]IZMHIRGIERHXLIIJJIGXMZIRIWWSJXLIWIMRMXMEXMZIWKSZIVRQIRXŭWYWI

SJɄIZMHIRGIRIIHWXSFIQSVIXVERWTEVIRX ŰXVERWTEVIRG]MWEƤVWXERHRIGIWWEV]WXITMRIREFPMRKXLIUYEPMX]SJE

HITEVXQIRXŭWIZMHIRGIFEWIHHIGMWMSRQEOMRKXSFINYHKIHű

;LIRXLIIZMHIRGIXLEXLEWFIIRYWIHXSNYWXMJ]ERHWLETIETSPMG]TVSTSWEPMWXVERWTEVIRX

 • The evidence can be evaluated and improved upon.

 • The public are better able to understand and engage with the reasoning for policy interventions.

 • Further government initiatives and policy evaluation can build on it.

In Show your Workings we set out a draft transparency framework — an approach to testing evidence transparency that 

GSYPHFIETTPMIHVETMHP]HMHRSXVIUYMVIWYFNIGXQEXXIVI\TIVXMWITVSHYGIHQIERMRKJYPERHGSRWMWXIRXVIWYPXWERHEPPS[IH

comparison between different policy areas and departments. Sense about Science committed to leading an assessment 

SJɄTSPMG]TVSTSWEPWXSWIILS[HMJJIVIRXHITEVXQIRXWVEROIHSRIZMHIRGIXVERWTEVIRG]

%W[IIQFEVOIHSRXLMW[IFIGEQIMRGPMRIHXS[EVHWERI\TIVMQIRXEP]IEVXSXIWXXLIJVEQI[SVOERHMHIRXMJ]XLIVERKI

SJɄTVEGXMGIW[IWLSYPHFIPSSOMRKEXEWWMWXIHF]HMWGYWWMSR[MXLHITEVXQIRXEPEREP]WXWERHXLI'EFMRIX3JƤGI;LEX;SVOW

8IEQ*SPPS[MRKXLIGLERKIWMRXLIPIEHIVWLMTERHQEGLMRIV]SJKSZIVRQIRXMR.YP][IHIGMHIHXLEXXLIVIZMI[

SJɄXLI]IEVXS1E]WLSYPHEMQXSMHIRXMJ]KSSHERHFEHTVEGXMGIERHXLEX[I[MPPMRWXIEHWXEVXVEXMRKHITEVXQIRXW

MRɄ8LMWLEWEPWSIREFPIHYWXSIPEFSVEXISRXLIJVEQI[SVOTEVXMGYPEVP]MRXLIGSRXI\XSJIEVP]WXEKIERRSYRGIQIRXW

8LMWVITSVXWLEVIW[LEX[IJSYRH-XLMKLPMKLXWI\EQTPIWSJLS[XVERWTEVIRG]EFSYXXLIYWISJIZMHIRGIMWFIMRKEGLMIZIH

ERH[EVRWSJXLIOMRHWSJTVEGXMGIWXLEX[MPPPIEHXSPS[WGSVIWMRRI\X]IEVŭWEWWIWWQIRX

&SXLXLIEWWIWWQIRXI\IVGMWIERHXLIHIZIPSTQIRXSJXLIJVEQI[SVOLEZIFIRIƤXIHJVSQHITEVXQIRXWŭVIWTSRWIW

ERHGPEVMƤGEXMSRWVIKYPEVHMWGYWWMSR[MXLHITEVXQIRXEPHMVIGXSVWSJEREP]WMWERHERMRMXMEPXIWXMRKI\IVGMWIMRZSPZMRK

ɄZSPYRXIIVWJVSQZEVMSYWTSPMG]FEGOKVSYRHW%KVERXJVSQXLI2YJƤIPH*SYRHEXMSRWYTTSVXIHEHIHMGEXIHVIWIEVGLIV

for the assessment and the partnership between Sense about Science, the Institute for Government and the Alliance for 

9WIJYP)ZMHIRGITVSZMHIHSZIVWMKLXERHVIZMI[-RXLI2I\X7XITWWIGXMSREXXLIIRH[IWIXSYXWSQIJYVXLIVSTTSVXYRMXMIW

XSHMWGYWWXLIWIƤRHMRKWERHHIZIPSTXLIQIXLSHWJSVXLIJYPPVIZMI[MRERH[I[SYPH[IPGSQIJYVXLIVJIIHFEGO

1

 Civil Service Reform Plan: One Year On Report, July 2013 https://www.gov.uk/government/uploads/system/uploads/

EXXEGLQIRXCHEXEƤPI'76C3=3C03;C6)7C4(*THJ

2

7LS[]SYV[SVOMRKW%WWIWWMRKLS[XLIKSZIVRQIRXYWIWIZMHIRGIXSQEOITSPMG].MPP6YXXIV
.IR+SPH3GXSFIV

http://www.instituteforgovernment.org.uk/publications/show-your-workings

INTRODUCTION 1.1 

2024 The Promise of Evidence-Based Policymaking
1.2 The transparency framework

4YXWMQTP]XLIUYIWXMSR[ILEZIEWOIHSJHITEVXQIRXWMWcould someone outside government see what you’re 

proposing to do and why? 8LIJVEQI[SVOPSSOWEXXLMWEGVSWWXLIJSPPS[MRKEVIEW

 DIAGNOSIS (The issue that will be addressed) 

 8LIHSGYQIRXW
WLSYPHI\TPEMR• what policymakers know about the issue, its causes, effects, and scale.

 PROPOSAL8LIKSZIVRQIRXŭWGLSWIRMRXIVZIRXMSR


 8LIHSGYQIRXW
WLSYPHI\TPEMR• why the government has chosen this intervention • what evidence, if any, 

that choice is based on • how policymakers have assessed the evidence base, including what has been tried 

before and whether that worked • whether there are other options and why they have not been chosen • what 

XLIɄKSZIVRQIRXTPERWXSHSEFSYXER]TEVXSJXLIMRXIVZIRXMSRXLEXLEWRSX]IXFIIRHIGMHIHYTSR• what the costs 

ERHFIRIƤXWEVIIWXMQEXIHXSFIERHXLIEWWYQTXMSRWFILMRHXLSWIGEPGYPEXMSRW

 IMPLEMENTATION (How the intervention will be introduced and run) 

 8LIHSGYQIRXW
WLSYPHI\TPEMR• why this method for delivering the intervention has been chosen • what 

evidence, if any, that decision is based on • whether there are other methods and if so the reasons for not 

choosing them • if the way to deliver the intervention is still being decided, what the method is for deciding •

[LEXɄXLIGSWXWERHFIRIƤXWEVIIWXMQEXIHXSFIERHXLIEWWYQTXMSRWFILMRHXLSWIGEPGYPEXMSRW

 TESTING AND EVALUATION (How we will know if the policy has worked) 

 8LIHSGYQIRXW
WLSYPHI\TPEMR• any testing that has been or will be done • plans to measure the impact of the 

policy, and the outcomes that will be measured • plans to evaluate the effects of the policy, including a timetable • 

plans for using further inputs.

*SPPS[MRKEZEPMHEXMSRI\IVGMWI[MXLTSPMG]ZSPYRXIIVWMR1E][IEQIRHIHXLIMRMXMEPJVEQI[SVO[LMGLLEHEWITEVEXI

WIGXMSRSRZEPYIJSVQSRI]XSMRGPYHIXLMWYRHIVXLI4VSTSWEPERH-QTPIQIRXEXMSRLIEHMRKWERHI\TERHIHXLIHIƤRMXMSR

SJɄ8IWXMRKERH)ZEPYEXMSRXSMRGPYHI[LIXLIVGSRWYPXEXMSRWXEKITVSTSWEPW[IVIXVERWTEVIRXEFSYXHITEVXQIRXWŭTPERW

XSɄYWIXLIQEXIVMEPKEXLIVIH3

 

%PXLSYKL[ILEZIRSXGVIEXIHHITEVXQIRXEPWGSVIWXLMW]IEVXLIJVEQI[SVOŭWWGSVMRKPEHHIV[EWYWIHXSPSSOEXEWEQTPI

SJTSPMGMIWJVSQIEGLHITEVXQIRX8LIX[SYRHIVP]MRKUYIWXMSRWYRHIVIEGLLIEHMRK[IVI

 Can you tell what evidence has been used?

 Can you tell how the government has assessed or used this evidence?

&VSEHP]XLIWGSVIPIZIPW[IVIEWJSPPS[W[MXLWSQIEHNYWXQIRXXSWYMXHSGYQIRXX]TIW


  2SXWYJƤGMIRXP]JSVPIZIP

  )ZMHIRGIMWQIRXMSRIH[MXLWSQIMRHMGEXMSRSJ[LEXMXMWERHLS[MXLEWFIIRYWIH

  %WMRPIZIPERHXLIWYTTSVXMRKIZMHIRGIMWQSWXP]PMROIHXSXLIVIPIZERXTEVXWSJXLITSPMG]TVSTIVP]GMXIH

ERHƤRHEFPIERHXLIVIMWHMWGYWWMSRSJLS[MXLEWFIIRYWIH

  7YTTSVXMRKIZMHIRGIMWGSRWMWXIRXP]PMROIHXSXLIVIPIZERXTEVXWSJXLITSPMG]TVSTIVP]GMXIHERHƤRHEFPI

and there is assessment of uncertainties and contradictions in the evidence base.

3

 The original version of the framework, as published in Show your Workings, is replicated in Appendix 3. 

8LIEQIRHIHZIVWMSRYWIHXSKIRIVEXIXLIƤRHMRKWHIWGVMFIHLIVIETTIEVWMR%TTIRHM\

INTRODUCTION 1.2 4

The Promise of Evidence-Based Policymaking 2025
1.3 The sample

;IHIƤRIHETSPMG]EWEWTIGMƤGMRXIVZIRXMSRXSGLERKIXLIWXEXYWUYSŪ[LEXXLITYFPMG[SYPHMRXYMXMZIP]XLMROSJEWE

policy and the usual way that policies are presented in announcements. Several policies are sometimes grouped under 

SRIɄMRMXMEXMZIWYGLEWEŬWXVEXIK]ŭ8LIWIEVIXVIEXIHEWWITEVEXITSPMG]TVSTSWEPW;LIVIETVSTSWEPLEWGSQTSRIRXWXLEX

share similar evidence propositions, we treated it as a single policy.

8LIPMWXSJTSPMGMIW[EWEWWIQFPIHF]KEXLIVMRKEPPTSPMG]VIPEXIHHSGYQIRXWGSRWYPXEXMSRW[LMXITETIVWMQTEGX

assessments, bills and announcements) from the period and then cross referencing them.

;IGLSWIXSPSSOEXHSQIWXMGTSPMG]HITEVXQIRXW;II\GPYHIHXLI*SVIMKR3JƤGIXLI1MRMWXV]SJ(IJIRGI

ERHɄXLIɄ(ITEVXQIRXJSV-RXIVREXMSREP(IZIPSTQIRX(J-(
JVSQXLIEWWIWWQIRXFIGEYWIXLIVI[IVIRSTSPMGMIWXLEX

ƤXXIHSYVɄHIƤRMXMSR;IRSXIXLEX(J-(LEWFIIRYWMRKSYVHVEJXJVEQI[SVOWMRGIMXWTYFPMGEXMSRMR3GXSFIVXSVIƥIGX

SRɄXLIɄMRJSVQEXMSRGSRXEMRIHMRMXWEMHVIZMI[W

*SVXLMWƤVWXWXEKI[IPSSOIHEXTSPMGMIWJVSQ;LMXILEPPHITEVXQIRXW;IHMHRSXKEXLIVTSPMGMIWJVSQXLI;IPWL7GSXXMWL

SVɄ2SVXLIVR-VMWLKSZIVRQIRXW;IEVIOIIRXSI\TERHXLIYWISJXLIJVEQI[SVOERHXLIIZMHIRGIXVERWTEVIRG]EKIRHE

MRɄXLIJYXYVI;IVIQSZIHTSPMGMIWMQTPIQIRXIHF]I\IGYXMZIEKIRGMIWWYGLEWXLI)RZMVSRQIRX%KIRG]8LMW[EWHYI

XSɄFSXLXMQIGSRWXVEMRXWERHXLIJEGXXLEXXLIWIFSHMIWEVIQSWXP]MQTPIQIRXMRKTSPMGMIWWIXF]GIRXVEPHITEVXQIRXW

-RXLI]IEVJVSQXLIKIRIVEPIPIGXMSRXS1E][IJSYRHHMWGVIXITSPMG]TVSTSWEPWF]HSQIWXMG

TSPMG]ɄHITEVXQIRXW

;IEWWIQFPIHXLIHSGYQIRXWEZEMPEFPIEXXLITSMRX[LIRXLIKSZIVRQIRXƤVWXWIXSYXETSPMG]TYFPMGP]8LMWMWMQTSVXERXŪ

XLMWMW[LIRXLITYFPMGTEVPMEQIRXERHXLIQIHMEƤVWXLEZIXLIGLERGIXSEWWIWWERI[TVSTSWEPŪERH[LIRMXMWMQTSVXERX

XLEXXLIKSZIVRQIRXI\TSWIWXLIIZMHIRGIFILMRHMXWMRMXMEPXLMROMRKMRSVHIVXSTVSQSXIMRJSVQIHIRKEKIQIRX

;IQEHIWSQII\GITXMSRW;LIVIXLIƤVWXHIWGVMTXMSRSJETSPMG][EWETVIWWVIPIEWISVFYHKIXERRSYRGIQIRXXLMW[EW

HMWVIKEVHIHMJERSXLIVTVIMQTPIQIRXEXMSRWXEKIHSGYQIRX[EWEZEMPEFPI;LIVIERMRMXMEPERRSYRGIQIRXMRHMGEXIHEJYPPIV

proposal to be published very shortly after, eg the following month, this later document was assessed.

&IGEYWISJXLIREXYVISJXLIEWWIWWQIRXŪPSSOMRKEXLS[XLITYFPMGGERKVEWTXLIFEWMWJSVETSPMG][LIRMXMWƤVWXWIX

out — most of the policies we looked at were early stage. Consultations made up around half of the total policy proposals 

MHIRXMƤIHERHNYWXSZIVLEPJSJXLSWIVIZMI[IH;IRSXMGIHXLEXETSPMG]FIMRKJYVXLIVHS[RXLIPMRISJHIZIPSTQIRXHMH

RSXEYXSQEXMGEPP]QIERMXWGSVIHLMKLIV,S[IZIVXLIVI[IVIWSQITSPMGMIW[LIVI[INYHKIHXLEXMX[SYPHFIERSZIV

ETTPMGEXMSRXSJYPP]ETTP]XLIJVEQI[SVOERHXLIWIEVII\TPEMRIHMRXLIJYVXLIVHIXEMPEFSYXQIXLSHWMR%TTIRHM\

Department Number of policies

'EFMRIX3JƤGI 

(ITEVXQIRXJSV&YWMRIWW-RRSZEXMSRERH7OMPPW 

(ITEVXQIRXJSV'SQQYRMXMIWERHɄ0SGEP+SZIVRQIRX 

(ITEVXQIRXJSV'YPXYVI1IHMEERH7TSVX 

(ITEVXQIRXJSV)HYGEXMSR 

(ITEVXQIRXJSV)RZMVSRQIRX*SSHERH6YVEP%JJEMVW 24 

(ITEVXQIRXJSV8VERWTSVX 22 

(ITEVXQIRXJSV;SVOERH4IRWMSRW 

(ITEVXQIRXSJ)RIVK]ERH'PMQEXI'LERKI 

(ITEVXQIRXSJ,IEPXL 

,18VIEWYV] 

,SQI3JƤGI 

1MRMWXV]SJ.YWXMGI 22 

Total 593

INTRODUCTION 1.3 

2026 The Promise of Evidence-Based Policymaking
%XXIQTXMRKXSEWWIQFPIEGSQTVILIRWMZIPMWXSJTSPMGMIWGSRƤVQIHXLEXKSZYOMWI\XVIQIP]PMQMXIHEWETPEXJSVQJSVTYFPMG

engagement with policymaking. The “policies” section is a collection of press releases, policy papers, speeches, guidance 

ERHSXLIVHSGYQIRXWYRHIVZIV]FVSEHLIEHPMRIWWYGLEWŰ'SYRXIVXIVVSVMWQűŰ8SYVMWQűERHŰ7QSOMRKű 4

 The image 

FIPS[WLS[WXLIVIWYPXSJEWIEVGLJSVXLIŰ;LEXXLIKSZIVRQIRXŭWHSMRKEFSYXIGSRSQMGKVS[XLMRVYVEPEVIEWű8LIVI

MWɄEɄGSPPIGXMSRSJɄHSGYQIRXW[MXLRSGSLIVIRGI 5

4

;IJSYRHEWMQMPEVTVSFPIQ[LIRHIZIPSTMRKXLIJVEQI[SVOJSV Show your Workings, see http://www.instituteforgovernment.

SVKYOWMXIWHIJEYPXƤPIWTYFPMGEXMSRW	-*+		7LS[MRK	]SYV	[SVOMRKW	ZFTHJT

5

LXXTW[[[KSZYOKSZIVRQIRXTSPMGMIWIGSRSQMGKVS[XLMRVYVEPEVIEWEGGIWWIHXL2SZIQFIV


;IXLIVIJSVILEHXSGVIEXISYVQEWXIVPMWXSJTSPMGMIWF][SVOMRKXLVSYKLEPPPMWXWERHVIJIVIRGIWXLEX[IGSYPHƤRH

JSVɄKSZIVRQIRXHSGYQIRXW4PEGIW[LIVI[IGSYPHƤRHTSPMGMIWSVGPYIWEFSYXXLIMVI\MWXIRGIMRGPYHIH

 • Consultations

 • Government bills

 • White papers 

 ŵ&YHKIXERHWTIRHMRKVIZMI[ERRSYRGIQIRXW

 • Impact assessments

 • Announcements on gov.uk and press releases

 ŵ'SRWIVZEXMZI4EVX]1ERMJIWXS

 • Single departmental plans

*VSQXLIVIWYPXMRKQEWXIVPMWXSJTSPMGMIW[IWIPIGXIHERMRMXMEPPMWXSJTIVHITEVXQIRX8LIWI[IVITMGOIHSRXLI

basis of departmental priorities, as set out in their single departmental plans, but were also designed to cover the breadth 

SJEHITEVXQIRXŭW[SVO;LIVIXLIVI[EWEGLSMGIFIX[IIRTSPMGMIWŪWYGLEWQYPXMTPITSPMGMIWGSRXEMRIHMRSRIWXVEXIK]

— we selected those that the public were most likely to be interested in (a new driving test rather than a small administrative

GLERKIXSXLIPMGIRWMRKFSH]JSVI\EQTPI
*MREPP][IGSQTEVIHXLIWIPIGXMSRWXSQEOIWYVIXLEXXLI][IVIVIEWSREFP]

IUYMZEPIRXJSVIEGLHITEVXQIRX8LITSPMGMIWERHEWWSGMEXIHTSPMG]HSGYQIRXW[IEWWIWWIHEVIPMWXIHF]HITEVXQIRX

MRɄ%TTIRHM\

INTRODUCTION 1.3 

The Promise of Evidence-Based Policymaking 2027
The list was shared with departments. A group of volunteer scorers and the steering group and research team then 

EWWIWWIHXLIWM\TSPMGMIWTIVHITEVXQIRX8LIVIWYPXMRKWGSVIWERHGSQQIRXW[IVIVIZMI[IHF]XLIVIWIEVGLXIEQ

*VSQɄXLMW[ILEZIHVE[RI\EQTPIWSJKSSHERHFEHTVEGXMGIMRIZMHIRGIXVERWTEVIRG]

While the scores have not been collated for an assessment of overall departmental performance, nor moderated to give 

EɄƤREPWGSVIEWXLI][MPPFIMR[LIVIMXMWLIPTJYPXLIOMRHWSJWGSVIWEGLMIZIHF]HMJJIVIRXTSPMGMIWERHTVEGXMGIWLEZI

been indicated.

8LIEWWIWWQIRXTVSGIWWLEWVEMWIHUYIWXMSRWXSFIHIGMHIHYTSRRI\X]IEVWYGLEWLS[FIWXXSVIƥIGXXLIFVIEHXL

SJɄHITEVXQIRXEP[SVOMRXLIWEQTPI-RXLIWTMVMXSJGSRWYPXEXMSRXLEXLEWMRJSVQIHXLMWTVSNIGX[I[MPPGSRZIRIEQIXLSHW

review group that will help resolve these issues, and possibly lead to further amendments to the framework.6

6

 A more thorough outlining of how we evaluated transparency of evidence in government can be found in Appendix 1.

INTRODUCTION 1.3 

2028 The Promise of Evidence-Based Policymaking
1.4 Summary of findings

Readers are reminded that our aim here is to improve use of evidence and engagement with the evidence used 

MRɄTSPMG]QEOMRKXLVSYKL increasing transparency. 8VERWTEVIRG]MWETVIVIUYMWMXIXSEWWIWWMRKUYEPMX]8LMWVIZMI[

EWWIWWIWXVERWTEVIRG]RSXXLIUYEPMX]SJXLIIZMHIRGIYWIHSVXLIQIVMXWSJXLITSPMG]%[IPPJSYRHIHTSPMG]ERHETSSVP]

founded policy may both score well for transparency. A transparent evidence base enables a better conversation about 

XLIɄTVSWERHGSRWSJER]TSPMG]

3YVEWWIWWQIRXWLS[IHXLEX[LMPIXLIVI[IVIWSQII\EQTPIWSJZIV]KSSHTVEGXMGI[LMGLWLS[WXLIWXERHEVHW[IEVI

setting are attainable), there were some general shortcomings in departmental approaches to evidence transparency. 

-RɄTEVXMGYPEV

1. Sharing work done

 (ITEVXQIRXWWLSYPHQEOIEZEMPEFPIERHGPIEVP]VIJIVIRGI
XLIVIWIEVGLERHVIZMI[WXLEXXLI]LEZIGSRHYGXIH

XSɄMRJSVQETSPMG]WSXLEXXLITYFPMGGERYRHIVWXERHXLIVEXMSREPIJSVMX

2. Poor referencing

 6IJIVIRGMRKRIIHWXSFIQSVIWTIGMƤGERHYWIJYP7SQITSPMGMIWPMROIHXSWMKRMƤGERXHSGYQIRXWWLS[IHGPIEVP]

[LMGLTSMRXW[IVIVIJIVIRGIHXS[LMGLWSYVGIWERHLS[XLIWI[IVIVIPIZERXXSXLIKSZIVRQIRXŭWGSRGPYWMSRW

1SWXHMHRŭX;IGSYPHRŭXPSGEXIKYMHERGISRVIJIVIRGMRKJSVKSZIVRQIRXHSGYQIRXW[LMGLQMKLXLIPT

3. A clear chain of reasoning

 4ISTPIWLSYPHFIEFPIXSJSPPS[XLIXLMROMRKFIX[IIR(MEKRSWMW4VSTSWEP-QTPIQIRXEXMSRERH8IWXMRK


Ʉ)ZEPYEXMSR8LIQSWXXVERWTEVIRXTVSTSWEPWHIQSRWXVEXIHXLIGLEMRSJVIEWSRMRKEWXS[LEXXLITVSFPIQ[EW

and why the policy was the chosen response, and included discussion about the limitations of the evidence. 

4. Manifesto-derived policy commitments can be transparent

 Policies that originated in manifesto commitments featured among the best and the worst for transparency. 

8LMWɄWIIQWXSLEZIFIIRMRƥYIRGIHF][LIXLIVXLI]EVIGSRGIVRIH[MXLSYXGSQIWIRHW
SV[MXLWTIGMƤG

measures to achieve them (means), and whether departments consulted on their development. 

5. Budget announcements

 4SPMGMIWERRSYRGIHMRXLI&YHKIXSV%YXYQR7XEXIQIRX[IVIWMKRMƤGERXP]PIWWXVERWTEVIRXEFSYXXLIYRHIVP]MRK

evidence than other policies.

6. Alternatives

 There was very little transparency overall about the consideration given to other policy options, even though this 

MWɄEVIUYMVIQIRXMRMQTEGXEWWIWWQIRXW

7. Modelling policy impacts

 Some departments have found clear and impressive ways to share their modelling and the assumptions 

FILMRHɄQSHIPW

8. Testing and evaluation

 There is a lot of scope to improve the description of plans for testing and evaluation and for what consultations 

will do with inputs. Few policies scored well on this. Policies with clear testing and evaluation plans tended to be 

clearer about the evidence for the scope and scale of the issues they were addressing. 

1ER]SJXLITVSFPIQWVEMWIHEVIIEWMP]VIQIHMIHERHMJEHHVIWWIH[SYPHQEOIEWYFWXERXMEPHMJJIVIRGIXSKSZIVRQIRXŭW

TIVJSVQERGISRIZMHIRGIXVERWTEVIRG]ELIEHSJRI\X]IEVŭWJYPPVEROMRKSJHITEVXQIRXW8LMWVITSVXPSSOWƤVWXEXƤRHMRKW

in relation to the four areas of the transparency framework and then at practices relevant to all parts of the policy proposal.

INTRODUCTION 1.4 

The Promise of Evidence-Based Policymaking 2029
SECTIONS OF THE 

FRAMEWORK

-RXLIJSPPS[MRKTEKIW[ILEZIWIXSYXKSSHERHFEHI\EQTPIWSJXVERWTEVIRG]EKEMRWXXLIJSYV

sections of the framework and where useful we have indicated how our scorers rated them. 

8LIɄI\EQTPIWEVIRSXI\LEYWXMZIERHWSQISJXLITSPMGMIW[IEWWIWWIHGSYPHLEZIFIIRQIRXMSRIH

YRHIVERYQFIVSJLIEHMRKWFYX[ILEZIWSYKLXXSWLS[EVERKISJI\EQTPIWEGVSWWHMJJIVIRXEVIEW

of policy. 

2.1 Diagnosis

The starting point of a robust policy is an understanding of the problem it is trying to address. This helps people understand 

the need for it and any relevant information they should raise. It also makes it possible to assess whether the policy is likely 

XSLEZIMXWMRXIRHIHIJJIGXERHPEXIV[LIXLIVMXLEWLEHXLEXIJJIGX(ITEVXQIRXWWLSYPHHIWGVMFI[LEXXLI]ORS[EFSYX

XLIɄMWWYIMXWGEYWIWIJJIGXWERHWGEPIERHXLI]WLSYPHWIXSYXXLIWSYVGIWJVSQ[LMGLXLI]LEZIHVE[RXLEXORS[PIHKI

;IJSYRHWSQIKSSHI\EQTPIWSJHITEVXQIRXWHSMRKXLMW4SPMGMIWXLEX[IVIEXEZIV]IEVP]WXEKISJHIZIPSTQIRXWXMPP

HMH[IPP[LIVIXLIHITEVXQIRXWMRUYIWXMSRLEHHMEKRSWIHXLITVSFPIQXVERWTEVIRXP]ERHFIIRGPIEVEFSYXLS[XLIɄRI\X

WXEKIW[SYPHFIHIZIPSTIH*SVXLI(ITEVXQIRXJSV8VERWTSVX(J8
ŭWTSPMG]SR2I[&YW*VERGLMWMRK4S[IVWJSV0SGEP

%YXLSVMXMIW[LMGL[SYPHFIɄTEVXSJXLI&YW7IVZMGIW&MPP[IPSSOIHEXXLIZIV]IEVP]WXEKITETIVGVIEXIHJSVMXWFYWVIJSVQ

workshops.7

8LMWTETIVLEHEɄXLSVSYKLPSSOEXXVIRHWMRFYWYWIF]KISKVETLMGEVIE*SVXLIQSSXIHJVERGLMWMRKTSPMG]

MXɄWIXSYXTVMRGMTPIWERHɄHMJJIVIRXQSHIPWSJJVERGLMWMRKETTVSTVMEXIJSVERIEVP]WXEKIHSGYQIRX-X[EWGPIEVEFSYX[LMGL

TSMRXW[IVIXSɄFIɄHIGMHIHSR

8LI'EFMRIX3JƤGIŭWTVSTSWEPXS)WXEFPMWL'SQQSR1IEWYVIWSJ7SGMS)GSRSQMG&EGOKVSYRHEPWSZIV]IEVP]WXEKI

was accompanied by thorough discussion of the problem and uncertainties in the evidence base. Studies that showed 

XLIɄFIRIƤXWSJEHMZIVWI[SVOJSVGI[IVIGPIEVP]GMXIH 8

7

&YW6IJSVQ;SVOWLSTW&EGOKVSYRH(SGYQIRX1SZMRK&VMXEMR%LIEH7ITXIQFIV

8

)RKEKIQIRX(SGYQIRX(IZIPSTMRKE'SQQSR7IXSJ1IEWYVIWJSV)QTPS]IVWSRXLI7SGMS)GSRSQMG&EGOKVSYRHW

SJɅXLIMVɅ;SVOJSVGIERH%TTPMGERXW1E]T

SECTIONS OF THE FRAMEWORK 2.1 

2030 The Promise of Evidence-Based Policymaking
8LI,SQI3JƤGIŭWGSRWYPXEXMSRSRɄ-RXVSHYGMRKE7XEPOMRK4VSXIGXMSR3VHIVI\TPEMRIHERHɄIPWI[LIVIVIJIVIRGIH


MXWɄZMI[ɄSRɄXLIPMQMXEXMSRWSJɄI\MWXMRKQIEWYVIW 9

8LI&EGOKVSYRHGLETXIVSJXLI(ITEVXQIRXSJ'YPXYVI1IHMEERH7TSVX('17
ŭWGSRWYPXEXMSRSR6IUYMVMRK(MVIGX

1EVOIXMRK'EPPIVWXS4VSZMHI'EPPMRK0MRI-HIRXMƤGEXMSRWLS[IHXLIGLEMRSJVIEWSRMRK[MXLVIJIVIRGIWXSɄXLIIZMHIRGIFEWI

XLVSYKLSYXJVSQHMEKRSWMRKXLITVSFPIQXSɄHIQSRWXVEXMRK[L]GYVVIRXQIEWYVIWEVIɄRSXɄIJJIGXMZI 10

8LI(ITEVXQIRXJSV'SQQYRMXMIWERH0SGEP+SZIVRQIRX('0+
ŭW4VSQSXMRK7YTTP]SJɄ7XEVXIV,SQIWTEVXSJXLI,SYWMRK

ERH4PERRMRK&MPPWGSVIHWJSV(MEKRSWMW[MXLMXWɄFEGOKVSYRHIZMHIRGIWIGXMSR7GSVIVWJIPXXLEXXLI]YRHIVWXSSHXLIFEWMW

SJXLIKSZIVRQIRXŭWGSRXIRXMSRXLEXXLIVIMWɄEɄWLSVXJEPPMRXLIGSRWXVYGXMSRERHɄGSQTPIXMSRSJɄRI[LSYWIW 11

9

-RXVSHYGMRKE7XEPOMRK4VSXIGXMSR3VHIVũEGSRWYPXEXMSR(IGIQFIVTTEVE

106IUYMVMRK(MVIGX1EVOIXMRK'EPPIVWXS4VSZMHI'EPPMRK0MRI-HIRXMƤGEXMSR.ERYEV]TT

11,SYWMRKERH4PERRMRK&MPP-QTEGX%WWIWWQIRX3GXSFIVT

SECTIONS OF THE FRAMEWORK 2.1 

The Promise of Evidence-Based Policymaking 2031
However, where several measures are linked in the same bill or strategy, departments need to be clear about the 

HMEKRSWMWJSVIEGLMWWYI-R('0+ŭW&VSEHIRMRKXLI(IƤRMXMSRSJ%JJSVHEFPI,SYWMRKMRXLIWEQIFMPPWGSVIVWLEHHMJƤGYPX]

working out which issue that policy was trying to address. After several readings it appeared to be that there are new 

QSHIPWSJɄEJJSVHEFPILSQIS[RIVWLMTERHXLIGYVVIRXTPERRMRKJVEQI[SVOHSIWRŭXVIGSKRMWIXLIQFYXXLMW[EWLEVH

XSɄHMWIRXERKPIJVSQXLISXLIVMWWYIWWIXSYXMRXLIFMPP

%QSRKXLIPS[IVWGSVIWJSV(MEKRSWMWXLI'EFMRIX3JƤGIŭWTVSTSWEPJSVER%RXM0SFF]MRK'PEYWIMR+SZIVRQIRX+VERX

%KVIIQIRXWŪWIXSYXMRETVIWWVIPIEWIERHEWYQQEV]SJXLIQIEWYVIũGSRXEMRIHNYWXERYRVIJIVIRGIHQIRXMSR

SJɄI\XIVREPGSQQIRXEV]EFSYXXLIMWWYIERHKEZIRSMRHMGEXMSREWXS[LEXKSZIVRQIRXFIPMIZIHXSFIMXWWGEPIGEYWI

SVɄIJJIGXW 12

It referred to a successful pilot of the new rule in another department but the link it provided went straight to another 

government press release, which in turn referenced a speech by the minister. We note that the lack of transparency 

about the issue being addressed in this case led to public speculation about the reason for the policy, some of which 

[EWɄTVSFEFP]YRJSYRHIH

8LI(ITEVXQIRXJSV;SVOERH4IRWMSRW(;4
ŭWTVSTSWEPSR0MQMXMRKXLI'LMPH)PIQIRXSJ9RMZIVWEP'VIHMXERH8E\'VIHMXW

[EWMRXVSHYGIHMRXLI7YQQIV&YHKIXERHFVSYKLXJSV[EVHF]XLI;IPJEVI6IJSVQERH;SVO&MPPWXVEMKLXEJXIV[EVHW

It scored very poorly for transparency. Claims were made about the problem and the impact of the policy without any 

discussion about where these assumptions came from, and no evidence was mentioned or referenced. While it is concerned 

[MXLJEMVRIWWMXQEHIEPSXSJXIWXEFPIGPEMQWXLEXWLSYPHLEZIEWWSGMEXIHIZMHIRGISVERI\TPEREXMSREFSYXXLIMVWSYVGI

8LI(ITEVXQIRXSJ)RIVK]ERH'PMQEXI'LERKI(IGG
ŭW'SRXVEGXWJSV(MJJIVIRGIJSV'EVFSR'ETXYVIERH7XSVEKI

[LMPIEGLMIZMRKEFEWMGPIZIPSJXVERWTEVIRG]ERHWSQIKSSHWGSVIWJSVVIJIVIRGMRKHMHRSXI\TPEMRXLIWSYVGISJMXW

GPEMQXLEXXLIGYVVIRXVIKYPEXMSRW[IVIHIƤGMIRXXLSYKLXLMWTSMRX[EWVITIEXIHSJXIRMREZEVMIX]SJ[E]W7GSVIVW

WXVYKKPIHXSɄYRHIVWXERHXLITVSFPIQXLITSPMG][EWMRXIRHMRKXSɄƤ\(IGGŭWHVEJXPIKMWPEXMSR4VSZMHMRK3JKIQ[MXL

4S[IVWXSɄ-QTPIQIRX7[MXGLMRKERH7IXXPIQIRX6IJSVQWLEHEWMQMPEVPEGOSJXVERWTEVIRG]EFSYXXLIWSYVGISJMXWGPEMQ

XLEXXLIɄTS[IVWEVIRIIHIH[LMGLGSRXVEWXW[MXLXLIHMWGYWWMSREFSYXYRGIVXEMRXMIWEFSYXXLITVSNIGXMSRWERHSXLIV

transparent aspects of the policy.

8LI8VIEWYV]ŭWGSRWYPXEXMSRSR%FSPMWLMRKXLI'EVFSR6IHYGXMSR'SQQMXQIRXEPWSPEGOIHXVERWTEVIRG]MRWIXXMRKSYXXLI

evidence behind its diagnosis. This consultation scored poorly on transparency in the other sections of the framework too.

8LI(ITEVXQIRXJSV)RZMVSRQIRX*SSHERH6YVEP%JJEMVW(IJVE
ŭW7MRKPI%RMQEP)WXEFPMWLQIRX0MGIRGI[EWUYMXIX]TMGEP

MRɄWIXXMRKSYXEPSKMGEPVEXMSREPIFYXRSXTVSZMHMRKEɄXVERWTEVIRXWSYVGIEFPIEGGSYRXSJXLIWGEPIERHWMKRMƤGERGIMRXLMW

GEWISJXLITVSFPIQWSJQYPXMTPIPMGIRGIW13

12+SZIVRQIRXERRSYRGIWRI[GPEYWIXSFIMRWIVXIHMRXSKVERXEKVIIQIRXW'EFMRIX3JƤGITVIWWVIPIEWIXL*IFVYEV]LXXTW[[[KSZYO

KSZIVRQIRXRI[WKSZIVRQIRXERRSYRGIWRI[GPEYWIXSFIMRWIVXIHMRXSKVERXEKVIIQIRXWEGGIWWIHɅXL2SZIQFIV


13'SRWYPXEXMSRSRXLIVIZMI[SJERMQEPIWXEFPMWLQIRXWPMGIRWMRKMR)RKPERH(IGIQFIVTTEVE

SECTIONS OF THE FRAMEWORK 2.1 

2032 The Promise of Evidence-Based Policymaking
2.2 Proposal

Once departments have diagnosed a problem, they need to develop a clear hypothesis about how an intervention might 

LIPTEHHVIWWMXERHHIPMZIVXLIKSZIVRQIRXŭWSFNIGXMZIW(ITEVXQIRXW[SYPHYWYEPP]HVE[SRIZMHIRGIŪJVSQTEWX

EXXIQTXWXSEHHVIWWEWMQMPEVTVSFPIQIZEPYEXMSRWSJXLSWIEXXIQTXWMRXIVREXMSREPSVSXLIVNYVMWHMGXMSREPI\TIVMIRGI

SVɄEGEHIQMGVIWIEVGLŪXSNYWXMJ][L]XLI]LEZIVIEWSRXSFIPMIZIXLIMVMRXIVZIRXMSRQMKLX[SVO

;II\TIGXIHXSFIEFPIXSXIPP[L]XLIKSZIVRQIRXLEHGLSWIRMXWMRXIVZIRXMSRERH[LEXSXLIVWMXLEHGSRWMHIVIH[LEX

evidence, if any, that choice was based on and how policymakers had assessed the evidence base, including what had 

FIIRXVMIHFIJSVIERH[LIXLIVXLEX[SVOIH;IEPWSI\TIGXIHXSWII[LEXXLIKSZIVRQIRXTPERRIHXSHSEFSYXER]XLMRK

MXɄLEHRSX]IX[SVOIHSYX[LEXXLIGSWXWERHFIRIƤXWEVIIWXMQEXIHXSFIERHXLIEWWYQTXMSRWFILMRHXLSWIGEPGYPEXMSRW

;LEXŭWEZEMPEFPI[MPPWSQIXMQIWFISJZIV]PMQMXIHVIPIZERGIŪFYXMRWYGLGEWIW[I[SYPHI\TIGXHITEVXQIRXWXSFIGPIEV

SRXLMWERHI\TPEMRLS[XLI]EVIXEOMRKXLEXMRXSEGGSYRXEWXLITSPMG]TVSGIIHWF]JSVI\EQTPITMPSXMRKERHIWXEFPMWLMRK

mechanisms for rapid feedback and adaptation.

Overall we noted an unsurprising relationship between the care and attention given to communicating the basis of the 

diagnosis and the transparency of conclusions about the proposal.

8LI(ITEVXQIRXSJ,IEPXL(,
ŭW2I[%PGSLSP+YMHIPMRIWTVSTSWEPWGSVIHLMKLP]JSVXVERWTEVIRG]SRFSXL(MEKRSWMW

ERHɄ4VSTSWEPXLIQEXIVMEPXLEXXLIHITEVXQIRXLEHYWIHJSVXLI(MEKRSWMWERH4VSTSWEP[IVIGPIEVERHXLIKSZIVRQIRXŭW

ZMI[SRMXWWXVIRKXLWERH[IEORIWWIW[EWTVIWIRXIHERHJVEROIKSRIEVP]TVIKRERG]IJJIGXWŰ6IPIZERXKSSHUYEPMX]

WXYHMIWEVIJI[űWIGXMSR
14 8LEXXVERWTEVIRG]LEWIREFPIHEPMZIP]HIFEXIXSIRWYIEFSYX[LIXLIVXLIIZMHIRGINYWXMƤIW

the content of the guidelines the department issued.

8LI(ITEVXQIRXJSV)HYGEXMSR(J)
ŭWTVSTSWEPJSVE2I[7SGMEP;SVO6IKYPEXSV[EWTSSVSRXLIXVERWTEVIRG]SJMXW

(MEKRSWMWWGSVMRKQSWXP]WERHW-XɄGMXIHERHPMROIHXSXLVIIVITSVXWSRWSGMEP[SVOIHYGEXMSRERHFYVIEYGVEG]MRXLI

W]WXIQFYXKEZIRSI\TPEREXMSRSJLS[XLIWIVITSVXWMRJSVQIHMXWTVSTSWEPWI\TERHMRKI\MWXMRKTVSKVEQQIWWIXXMRK

TVSJIWWMSREPWXERHEVHWERHWIXXMRKYTERI[VIKYPEXSV]FSH]
3XLIVHITEVXQIRXW[MXLTSPMGMIWEXɄEɄWMQMPEVIEVP]WXEKI

managed to do much better than this.

;LMPIEPMROFIX[IIRXVERWTEVIRG]SJ(MEKRSWMWERH4VSTSWEP[EWKIRIVEPP]IZMHIRXXLIVI[IVIRSXEFPII\EQTPIW[LIVI

ZIV]HIRWIERHHIXEMPIHQEXIVMEPMRXLI(MEKRSWMWXLIRQEHIMXHMJƤGYPXXSWIILS[XLI4VSTSWEPIQIVKIHJVSQMX;LIR

XV]MRKXSI\TPEMRSVYRHIVWXERH[LEXXLIKSZIVRQIRXMWXV]MRKXSHSERH[L]I\TPEMRMRKXLIVIPIZERGIERHMQTEGX

SJɄQEXIVMEPSRXLI[E]XLIHITEVXQIRXLEWHIZIPSTIHXLITSPMG]TVSTSWEPMWQSVIXVERWTEVIRXXLERLIETMRKSRJYVXLIV

QEXIVMEP8LMWWIGXMSRSJ(,ŭWTVSTSWEPJSV-RJIGXIH&PSSH6IJSVQSJ7YTTSVXMWSRII\EQTPISJWLEVMRKVIEWSRMRK

transparently, carrying through points from its diagnosis of the issue, including the limitations of that evidence, to the 

TVSTSWEPERHMXWGEPGYPEXMSRWSJZEPYIJSVQSRI]MX[SYPHLEZIWGSVIHE[MXLFIXXIVVIJIVIRGMRKXLSYKLMX[SRŭX[MRER]

TVM^IWJSVGPIEVTVSWI
 15

14%PGSLSP+YMHIPMRIW6IZMI[Ū6ITSVXJVSQXLI+YMHIPMRIWHIZIPSTQIRXKVSYTXSXLI9/'LMIJ1IHMGEP3JƤGIVW.ERYEV]T

15(IZIPSTMRKERI[W]WXIQSJƤRERGMEPERHSXLIVWYTTSVXJSVTISTPIMRJIGXIH[MXLLITEXMXMW'ERHSV,-:XLVSYKLFPSSH

ERHFPSSHTVSHYGXWMRXLI9/.ERYEV]T

See next page [p13].

SECTIONS OF THE FRAMEWORK 2.2 

The Promise of Evidence-Based Policymaking 2033
SECTIONS OF THE FRAMEWORK 2.2 

2034 The Promise of Evidence-Based Policymaking
16)RKEKIQIRX(SGYQIRX(IZIPSTMRKE'SQQSRWIXSJ1IEWYVIWJSV)QTPS]IVWSRXLI7SGMS)GSRSQMG&EGOKVSYRHWSJXLIMV;SVOJSVGI

ERHɅ%TTPMGERXW1E]%TTIRHM\4SWWMFPIQIEWYVIWSJWSGMSIGSRSQMG

172I[,SQIW&SRYW7LEVTIRMRKXLI-RGIRXMZI8IGLRMGEP'SRWYPXEXMSR(IGIQFIVT

8LI8VIEWYV]ŭWTVSTSWEPJSV7XVIRKXLIRMRKXLI+SZIVRERGI%GGSYRXEFMPMX]ERH8VERWTEVIRG]SJXLI&EROSJ)RKPERH[EW

one of the most consistent policies, scoring 2s across the framework. It referenced the evidence behind its proposal 

ERHɄLEHEWXVEMKLXJSV[EVHIEW]XSJSPPS[HIWGVMTXMSRSJXLIFEWMWJSVMXWMHIRXMƤGEXMSRSJETVSFPIQXLITVSTSWEPXSHIEP

with it and assumptions about how it would be implemented.

-X[EWVEVIJSVHITEVXQIRXWXSWLS[XLI]LEHGSRWMHIVIHEPXIVREXMZIETTVSEGLIWEPXLSYKLXLIVI[IVII\GITXMSRW

8LIɄ'EFMRIX3JƤGIŭWIRKEKIQIRXHSGYQIRXSR)WXEFPMWLMRK'SQQSR1IEWYVIWSJ7SGMS)GSRSQMG&EGOKVSYRH

transparently laid out a range of potential measures, along with its assessment of the pros and cons of each. These were 

PMROIHXSIZMHIRGIEWWIIRMRXLMWI\EQTPIVIPEXMRKXSTEVIRXEPMRGSQISV[IEPXLSRTEKI 16

('0+ŭWTVSTSWIHVIJSVQWXSXLI2I[,SQIW&SRYW[IVIEQSHIPJSVWIXXMRKSYXXLIVEXMSREPIFILMRHGLSMGIW

8LIɄWXVIRKXLWERH[IEORIWWIWSJEPXIVREXMZISTXMSRW[IVIGSRWMHIVIHMRHIXEMPERHVIEWSRWJSVRSXXEOMRKXLIQ

[IVIɄXVERWTEVIRX 17

SECTIONS OF THE FRAMEWORK 2.2 

The Promise of Evidence-Based Policymaking 2035
2.3 Implementation

Once the government has proposed an intervention, it needs to work out the ways to make that happen. There are often 

GLSMGIWSRXLIFIWX[E]XSHIPMZIVXLITSPMG]ERHHITEVXQIRXWWLSYPHFIEFPIXSI\TPEMR[L]XLI]LEZIGLSWIRSRI[E]

SZIVERSXLIVERH[LEXIZMHIRGIXLI]LEZIYWIHEW[IPPEWGSWXERHFIRIƤXGEPGYPEXMSRWERHɄXLIɄEWWYQTXMSRWFILMRH

them. There should also be some discussion of the opportunity cost of the new intervention if it is likely to divert resources 

JVSQI\MWXMRKEGXMZMX]&YXSJXIRMQTPIQIRXEXMSRMWWYIWŪ[LMGLEVIXLIVSSXSJQER]TSPMG]JEMPYVIWŪEVIRSXXLSVSYKLP]

GSRWMHIVIH[LIRETSPMG]MWTVSTSWIH8VERWTEVIRG]EFSYXXLIIZMHIRGIFILMRHTPERWXSɄMQTPIQIRXTSPMGMIW[EWEREVIE

SJɄ[IEORIWWJSVQER]TSPMG]HSGYQIRXWMRGPYHMRKXLSWIXLEXLEHWGSVIH[IPPSRSXLIVTEVXWSJɄXLIɄJVEQI[SVO

(,ŭW(IEXL'IVXMƤGEXMSR6IJSVQW[EWERI\EQTPISJKSSHTVEGXMGIMRXVERWTEVIRG]8LITPERRIHMQTPIQIRXEXMSR[EW

PMROIHXSPIWWSRWJVSQTMPSXTVSKVEQQIWEWWIIRSRTSJXLIGSRWYPXEXMSRHSGYQIRX 18

A “lessons from the pilots” document accompanied the consultation, providing further evidence. The consultation also 

MHIRXMƤIHSYXWXERHMRKUYIWXMSRWJSVMQTPIQIRXMRKXLITSPMG]WYGLEWXLMWI\EQTPI 19

7GSVIVWHMHRSXƤRHQYGLQIRXMSRSJXLIIZMHIRGIFILMRHSTXMSRWJSVMQTPIQIRXMRKXLI%TTVIRXMGIWLMTW0IZ][LMGL

[EWETVSTSWEP[LIVIXLI(ITEVXQIRXJSV&YWMRIWW-RRSZEXMSRERH7OMPPW&-7
PIHSRXLIMQTPIQIRXEXMSRSJETSPMG]ƤVWX

ERRSYRGIHMRXLI&YHKIX8LI]JSYRHRSRIMRXLIMQTPIQIRXEXMSREWTIGXWSJXLI8VIEWYV]ŭW,IPTXS7EZI[LMGLHMWGYWWIH

X[SɄEPXIVREXMZIQIXLSHWSJMQTPIQIRXEXMSRSR[LMGLXLIKSZIVRQIRX[EWLSTMRKXSGSRWYPX(J8ŭWERH8VERWTSVX

JSVɄ0SRHSR8J0
ŭWNSMRXTVSWTIGXYWSRE2I[4EVXRIVWLMTJSV6EMP4EWWIRKIV7IVZMGIWMRXLI7SYXL)EWXI\TPSVIH[E]W

SJMQTVSZMRKWIVZMGIWWYGLEWRI[QIXVSWX]PIXVEMRWFYXMX[EWZEKYIEWXSXLIMRJSVQEXMSRXLIWI[IVIHVE[RJVSQ

%RɄETTIRHM\TVIWIRXIHHMJJIVIRXSTXMSRWJSVMQTPIQIRXEXMSRFYXPMXXPIMRXLI[E]SJGSWXWERHFIRIƤXWSV[LEX[EWFIMRK

looked at in relation to each option.

18-RXVSHYGXMSRSJ1IHMGEP)\EQMRIVWERH6IJSVQWXS(IEXL'IVXMƤGEXMSRMR)RKPERHERH;EPIW4SPMG]ERH(VEJX6IKYPEXMSRWŪ

'SRWYPXEXMSR1EVGLTTEVE

19-FMHT

SECTIONS OF THE FRAMEWORK 2.3 

2036 The Promise of Evidence-Based Policymaking
Showing the department’s modelling

8VERWTEVIRXMQTPIQIRXEXMSRTPERWMRGPYHIH[LIVIVIPIZERXXLIHITEVXQIRXŭWQSHIPPMRKSJGSWXWERHFIRIƤXW;IJSYRH

EɄRYQFIVSJVIEPP]KSSHI\EQTPIWSJXLMW

8LI1MRMWXV]SJ.YWXMGI1S.
ŭW6EXMSREPMWMRKXLI'SYVXERH8VMFYREP)WXEXIEGLMIZIHEQMHPIZIPWGSVIJSVWLEVMRK

VIƥIGXMSRWSRMXWGEPGYPEXMSRW 20

8LIFIWXI\EQTPIWSJXLMWWIXSYXXLIGEPGYPEXMSRWYWIHEW[IPPEWXLIQIXLSHWYWIHXSQEOIXLIQERHXLIEWWYQTXMSRW

FILMRHMRTYXƤKYVIW3RII\EQTPISJXLMW[EW(IJVEŭWTVSTSWEPJSV'LERKIWXS4PEWXMGERH+PEWW4EGOEKMRK6IG]GPMRK

&YWMRIWW8EVKIXW8LIGSRWYPXEXMSRTVSZMHIHƤZISTXMSRWERHMRHMGEXIHXLIKSZIVRQIRXŭWTVIJIVVIHSRI-XGPIEVP]HMWTPE]IH

XLIQSHIPPMRKJSVXLIGSWXWERHFIRIƤXWSJIEGLSTXMSRT
 21

8LIEGGSQTER]MRKMQTEGXEWWIWWQIRXTVSZMHIHEFVIEOHS[RSJXLIGSWXWERHFIRIƤXWSJIEGLSTXMSRJSVEQIRHIH

KPEWWERHTPEWXMGVIG]GPMRKXEVKIXW8LIVI[EWEREP]WMWSJXLIGSWXWERHFIRIƤXWRSXNYWXSJXLILIEHPMRIXEVKIXW

FYXɄEPWSWYFMWWYIWWYGLEWGEVFSRWEZMRK8LIHSGYQIRXWIXSYXXLIEWWYQTXMSRWFILMRHMXWQSHIPPMRKERHTVSZMHIH

EɄWIRWMXMZMX]EREP]WMWXLEXSYXPMRIWEWWYQTXMSRWQEHIF]XLIHITEVXQIRXERHLS[XLI]EJJIGXXLIGEPGYPEXMSRW

TɄSJɄXLIɄMQTEGXɄEWWIWWQIRX
 22

20-QTEGX%WWIWWQIRXSR,IV1ENIWX]ŭW'SYVXWERH8VMFYREPW7IVZMGITVSTSWEPWSRXLITVSZMWMSRSJGSYVXWWIVZMGIW

MRɅ)RKPERHɅERHɅ;EPIW.YP]T

21'SRWYPXEXMSRSRGLERKIWXSXLITPEWXMGTEGOEKMRKVIG]GPMRKFYWMRIWWXEVKIXWJSVERHRI[XEVKIXWJSVTPEWXMG

ERHKPEWWJSVɅ2SZIQFIV

224PEWXMG
+PEWW4EGOEKMRK6IG]GPMRK&YWMRIWW8EVKIXW2SZIQFIVT

Continues on next page [p16].

SECTIONS OF THE FRAMEWORK 2.3 

The Promise of Evidence-Based Policymaking 2037
(IJVEŭW&ERRMRKXLI&YVRMRKSJ;EWXI3MPEWE*YIPMR,IEXIVWEGLMIZIHWPMKLXP]PS[IVQMHPIZIPWGSVIWJSV-QTPIQIRXEXMSR

LS[IZIVFIGEYWIMXHMHRSXMRHMGEXIXLIWSYVGIWJSVMXWGEPGYPEXMSRW(J)ŭW7GLSSPW2EXMSREP*YRHMRK*SVQYPEWGSVIHLMKLP]

for referencing the material in its tables and showing what lay behind its assumptions.

(IGGŭWTVSTSWEPJSV6IJSVQWXSXLI;EVQ,SQI(MWGSYRX7GLIQI[EWERSXLIVI\EQTPISJXVERWTEVIRG]SJQSHIPPMRK

8LIMQTEGXEWWIWWQIRXLEHEGSQTVILIRWMZIWIXSJQSHIPWI\TPSVMRKXLIGSWXWERHFIRIƤXWSJXLITVSTSWEPW%VERKI

of factors, such as the impact on energy demand and changes to household bills were taken into account (along with 

VIJIVIRGIWXSXLIWSYVGIWXLEXMRJSVQIHXLIHITEVXQIRXŭWEWWYQTXMSRW
8LIVIWYPXWSJ(IGGŭWEREP]WMW[IVITVIWIRXIH

MRɄXEFPIWPMOIXLISRIFIPS[ 23

23-QTEGX%WWIWWQIRX;EVQ,SQI(MWGSYRX)\XIRWMSR%TVMPT

SECTIONS OF THE FRAMEWORK 2.3 

2038 The Promise of Evidence-Based Policymaking
8LIVMWOWERHWIRWMXMZMXMIWXLEXGSYPHEJJIGX(IGGŭWQSHIPW[IVIEPWSEHHVIWWIHERHTVIWIRXIH

(;4ŭWTVSTSWIH'ETSR)EVP])\MX'LEVKIWJSV1IQFIVWSJ3GGYTEXMSREP4IRWMSR7GLIQIW[EWEJYVXLIVLMKLWGSVMRK

I\EQTPI[MXLMXWXVERWTEVIRXHMWTPE]SJXLIGSWXWERHFIRIƤXWGEPGYPEXIHF]XLIHITEVXQIRXYRHIVƤZIHMJJIVIRXWGIREVMSW

7IIXLMWI\EQTPISJXLIGEPGYPEXMSRWJSVGLEVKIWJEGIHYRHIVE	GETTSJXLIMQTEGXEWWIWWQIRX
 24

24-RXVSHYGMRKEGETSRIEVP]I\MXGLEVKIWMRXVYWXFEWIHSGGYTEXMSREPTIRWMSRWGLIQIW1E]T

SECTIONS OF THE FRAMEWORK 2.3 

The Promise of Evidence-Based Policymaking 2039
-R('0+ŭWXIGLRMGEPGSRWYPXEXMSRSR6IJSVQWXSXLI2I[,SQIW&SRYWXLIETTIRHIHXIGLRMGEPGSRWYPXEXMSRTVSZMHIH

EɄ[SVOIHI\EQTPISJEL]TSXLIXMGEPPSGEPEYXLSVMX]YRHIVXLIRI[FERHW7IITJSVI\EQTPI 25

%KEMRXLIQSHIPPMRKHMHRSXNYWXWLS[XLILIEHPMRIGSWXWERHFIRIƤXWFYXEPWSXLIMQTEGXSJXLITSPMG]YRHIVWIZIVEP

different scenarios, such as when the local authority had seen planning decisions appealed.

(IJVEŭWTVSTSWEPWJSV'PIER%MV>SRIW[LMGL[IVITEVXSJXLI(VEJX%MV5YEPMX]4PERWPEMHSYXGSWXERHFIRIƤXQSHIPW

MREɄWMRKPIEGGIWWMFPIHSGYQIRXER)ZMHIRGI%RRI\8LIVI[EWERI\TPSVEXMSRSJXLIGSWXWERHFIRIƤXWXSWSGMIX]

ŰGSQTEVMRKMQTVSZIQIRXWMREMVUYEPMX]EKEMRWXXLIEWWSGMEXIHGSWXWSJMQTPIQIRXEXMSRűWIITT
 26

1SWXSJXLII\EQTPIWSJQSHIPPMRK[IVIJSYRHMRMQTEGXEWWIWWQIRXW

252I[,SQIW&SRYW7LEVTIRMRKXLI-RGIRXMZI8IGLRMGEP'SRWYPXEXMSR(IGIQFIVT

26(VEJX)ZMHIRGI%RRI\%WWIWWQIRXSJXLITPERWXSMQTVSZIEMVUYEPMX]MRXLI9/7ITXIQFIVT

SECTIONS OF THE FRAMEWORK 2.3 

2040 The Promise of Evidence-Based Policymaking
Impact assessments: do they make a difference?

Impact assessments have to be provided by departments alongside proposals for a particular kind of policy — those with 

a regulatory impact on business or community organisations. They must provide a rationale for government intervention, 

XLISTXMSRWGSRWMHIVIHERHXLIGSWXWERHFIRIƤXWSJXLITSPMG]

278LI6IKYPEXSV]4SPMG]'SQQMXXII64'
ŭWKYMHERGI

SRɄMQTEGXEWWIWWQIRXWIRGSYVEKIWHITEVXQIRXWXSTVSHYGIVIPMEFPIIZMHIRGIJSVGSWXWERHFIRIƤXWERHIRWYVIXLI]LEZI

substantive evidence. 28 These are submitted to the RPC and given red, amber or green ratings depending on the evidence 

presented. Our assessment suggests the rigours of producing an impact assessment improve transparency in relation 

XSɄGSWXERHFIRIƤXQSHIPPMRK

The calculations of value for money, and the evidence behind these, tended to be stronger in documents with impact 

EWWIWWQIRXW;MXLMRXLIWIEKSSHI\EQTPI[EW(,ŭWTSPMG])\XIRHMRK'LEVKIWJSV2,77IVZMGIWJSV3ZIVWIEW:MWMXSVW

ERHɄ1MKVERXW[LMGLWIXSYXHIXEMPWSJGSWXWERHFIRIƤXWWSXLEXMXWZEPYIJSVQSRI]GEPGYPEXMSRWGSYPHFIPSSOIH

EXɄJSVɄIEGLSJXLISTXMSRWI\TPEMRIHXLIFEWIPMRIJSVXLIWIERHXLIEWWYQTXMSRWFIMRKQEHI-RQSWXSJXLISXLIVX]TIW

SJɄTSPMG]TVSTSWEPWXLMWMRJSVQEXMSR[EWTEXGL]MJMXI\MWXIH

1SVIFVSEHP]XLIVI[EWPMXXPIXSHMWXMRKYMWLSYVXSTERHFSXXSQTIVJSVQIVWSZIVEPP[MXLVIWTIGXXSLEZMRKERɄMQTEGX

EWWIWWQIRXSVRSXEPXLSYKLXLIVIUYMVIQIRXXSTVSHYGIERMQTEGXEWWIWWQIRXWIIQIHXSVEMWIXLIXVERWTEVIRG]

WXERHEVHSJJXLIFSXXSQPIZIPŪRSRISJSYV[SVWXTIVJSVQIVW[IVITSPMGMIW[LIVIHITEVXQIRXWLEHFIIRVIUYMVIH

XSɄTVSHYGIERɄMQTEGXEWWIWWQIRX%XXLISXLIVIRHXLIVI[IVII\EQTPIWSJLMKLWGSVMRKXVERWTEVIRXTSPMGMIWXLEXLEH

not been obliged to produce impact assessments. A look at the scores allotted to each policy showed that those with 

MQTEGXEWWIWWQIRXWETTIEVIHXSHSWPMKLXP]FIXXIVSRXVERWTEVIRG]JSV(MEKRSWMW4VSTSWEPERH-QTPIQIRXEXMSRXLSYKL

RSXɄ8IWXMRKERH)ZEPYEXMSR

8LMWWYKKIWXWXLEXXLIHMWGMTPMRISJLEZMRKXSTVSHYGIERMQTEGXEWWIWWQIRX[MXLXLIRIGIWWEV]GSWXFIRIƤXEREP]WMW

ERHɄGSRWMHIVEXMSRSJEPXIVREXMZISTXMSRWHSIWGSQTIPHITEVXQIRXWXSWIXSYXXLIIZMHIRGIYWIH,S[IZIV[IVIGSKRMWI

XLEXXLIGSRXVMFYXMSRSJGPIEVGSWXFIRIƤXEREP]WIWXSSYVXVERWTEVIRG]WGSVIWEKEMRWX4VSTSWEPERH-QTPIQIRXEXMSRQE]

QEWOVIPEXMZIP]PIWWXVERWTEVIRXFILEZMSYVMRSXLIVEWTIGXW3YVWEQTPI[EWRSXFMKIRSYKLXSHVE[ER]ƤVQGSRGPYWMSR

We are at an early stage of our comparative work, but we may want to investigate the effects of impact assessments 

SRɄXVERWTEVIRG]MRXLIJYPPVEROMRKWXEKI

27&IXXIV6IKYPEXMSR*VEQI[SVO1ERYEP4VEGXMGEP+YMHERGIJSV9/+SZIVRQIRX3JƤGMEPW1EVGLLXXTW[[[KSZYO

KSZIVRQIRXYTPSEHWW]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPIFMW&IXXIVVIKYPEXMSRJVEQI[SVOQERYEPTHJ

286IKYPEXSV]4SPMG]'SQQMXXIIVIGSQQIRHEXMSRWYWIH[LIRWGVYXMRMWMRKMQTEGXEWWIWWQIRXW1E]LXXTW[[[KSZYO

government/publications/how-the-regulatory-policy-committee-scrutinises-impact-assessments/regulatory-policy-committeerecommendations-used-when-scrutinising-impact-assessments 

SECTIONS OF THE FRAMEWORK 2.3 

The Promise of Evidence-Based Policymaking 2041
2.4 Testing and Evaluation

8IWXMRKEPPS[WTSPMG]HIZIPSTQIRXXSFIMRJSVQIHF]VIEP[SVPHI\TIVMIRGIXSKEXLIVMRJSVQEXMSRERHJIIHFEGO

ERHɄXSɄMRGSVTSVEXIYRJSVIWIIRMRƥYIRGIWSRXLITSPMG]ŭWIJJIGXWFIJSVIIQFEVOMRKSRER]QENSVGSWXWERHVISVKERMWEXMSR

)ZEPYEXMSRTVSQSXIWEQSVIW]WXIQEXMGERHSFNIGXMZISVKERMWEXMSRSJMRJSVQEXMSREFSYXXLITSPMG]ŭWIJJIGXW[LMGLEW[IPP

as informing policy development is an essential part of the accountability of government.

8LI8IWXMRKERH)ZEPYEXMSRWIGXMSR[EWEREVIESJ[IEORIWWJSVQER]HSGYQIRXWMXXIRHIHXSFIWSQIXLMRKXLEX[EW

done well or not at all. It was rare for policy documents to include plans to measure the impact of the policy, or to outline 

XLIHITEVXQIRXŭWRI\XWXITWMRHIZIPSTMRKMX-X[EWWPMKLXP]QSVIGSQQSRJSVGSRWYPXEXMSRWXSWIXSYXGPIEVP][LEX[SYPH

be done with the inputs, although still only around half of those in our sample scored 2 or above for this, which is hard 

XSɄYRHIVWXERHEWMXŭWEWXVEMKLXJSV[EVH[IPPIWXEFPMWLIHVIUYMVIQIRX

8LIFIWXHSGYQIRXWLEHGPIEVTPERWXSQIEWYVIWYGGIWWERHXMQIXEFPIWJSVIZEPYEXMSR('17ŭW'YPXYVEP4VSXIGXMSR

Fund was one of the most transparent in this section of the framework. For instance, there was a clear proposition 

JSVɄEɄQSRMXSVMRKERHIZEPYEXMSRWGLIQIJSVXLI*YRHERHEGORS[PIHKIQIRXSJXLIHMJƤGYPX]SJQIEWYVMRKXLMRKWWYGL

EWɄWSGMEPHIZIPSTQIRXERH[IPPFIMRKTTSJXLIGSRWYPXEXMSR
 29

8LMW[SYPHLEZIFIIRMQTVSZIHMJ('17LEHSYXPMRIH[LEXXLIQIEWYVIWEVILS[XLI][SYPHFIHIZIPSTIHERH[LIR

MX[SYPHVITSVXSRTVSKVIWW%KSSHI\EQTPISJHSMRKXLMW[EW(J)ŭWTPERWJSV*YPP%GEHIQMWEXMSRSJXLI7GLSSP7]WXIQ

[LMGL[LMPIRSX[IPPVIJIVIRGIHWIIFIPS[
[IVIXVERWTEVIRXSR8IWXMRKERH)ZEPYEXMSR[MXLGPIEVEGGSYRXEFPITPERW

for measurements and for publication of some of these annually, such as the Parent Portal and new performance 

XEFPIWɄ[IFWMXI

29'YPXYVEP4VSXIGXMSR*YRH+SZIVRQIRX4VSTSWEPWXS4VSXIGX'YPXYVEP,IVMXEKI3ZIVWIEW.ERYEV]TT

SECTIONS OF THE FRAMEWORK 2.4 

2042 The Promise of Evidence-Based Policymaking
%RSXLIVI\EQTPISJKSSHTVEGXMGIMRIZEPYEXMSRMWXLI(VEJX%MV5YEPMX]4PERW[LMGLWLS[IH[LEXXLIHITEVXQIRXI\TIGXIH

the policy to achieve and by what date. 30(IJVEŭWGSRWYPXEXMSRSR7XEXYXSV]4SWX1SZIQIRX8IWXMRKSJ'EXXPIJSV8&EPWS

stood out. There is a stated desire to pilot elements of the policy intervention, along with a set of criteria for doing this 

TSJXLIGSRWYPXEXMSR
 31

('17ŭW7TSVXW+SZIVRERGI'SHI[EWEPMKRIHXSWSQISJMXWOI]TIVJSVQERGIMRHMGEXSVWT
32 which increased 

XLIɄXVERWTEVIRG]WGSVIXLSYKLMX[EWRSXIHXLEXMX[EWRSXTSWWMFPIXSXIPPLS[XLIFEWIPMRIJSVXLIWI[SYPHFIHIXIVQMRIH

(J)ŭW2EXMSREP8IEGLMRK7IVZMGITPERWMRGPYHIHETMPSXERHWGSVIHQSVILMKLP]SR8IWXMRKERH)ZEPYEXMSRXLERSRSXLIV

TEVXWSJɄXLIJVEQI[SVO-X[EWRSXIHXLEXMX[EWRSXIWTIGMEPP]GPIEVEFSYXLS[XLITMPSX[MPPEJJIGXXLITPERWJSVXLIJYPPVSPP

SYXSJɄXLITVSKVEQQI 33

30(VEJXTPERWXSMQTVSZIEMVUYEPMX]MRXLI9/8EGOPMRKRMXVSKIRHMS\MHIMRSYVXS[RWERHGMXMIWũ9/SZIVZMI[HSGYQIRX7ITXIQFIVTT

31-QTVSZMRK8&'EXXPI'SRXVSPWMRGPYHMRKETVSTSWEPJSVWXEXYXSV]TSWXQSZIQIRXXIWXMRK%YKYWX

327TSVXMRK*YXYVI%2I[7XVEXIK]JSVER%GXMZI2EXMSR(IGIQFIVT

33)HYGEXMSREP)\GIPPIRGI)ZIV][LIVI1EVGLT

SECTIONS OF THE FRAMEWORK 2.4 22

The Promise of Evidence-Based Policymaking 2043
0EVKIP]XLSYKL8IWXMRKERH)ZEPYEXMSR[EWRSXWLEVIH[MXLXLITYFPMG[LMGLQE]QIERXLEXXLIVIEVIRSWIVMSYWTPERW

XSHSMXXLEXMXŭWRSXFIIRXLSYKLXEFSYXXLEXHITEVXQIRXWEVIRSXMRGPMRIHXSFISTIREFSYXMXSVXLEXXLI]EVIRSXTYXXMRK

IJJSVXMRXSGSQQYRMGEXMRKMX('0+ŭW4VSQSXMRK7YTTP]SJ7XEVXIV,SQIWLEHWGSVIH[IPPW
SR(MEKRSWMWFYXVIJIVVIH

XSɄ8IWXMRKERH)ZEPYEXMSRMRE[E]XLEXTISTPISYXWMHISJXLITSPMG]TVSGIWW[SYPHPMOIP]VIEHEWHMJƤHIRX

TSJXLIMQTEGXɄEWWIWWQIRX
 34

8LIHITEVXQIRXŭW6MKLXXS&Y]JSV,SYWMRK%WWSGMEXMSR8IRERXWEPWSKEZIPMXXPIMRJSVQEXMSREFSYXRI\XWXITW8LI7XEVXIV

,SQIWWIGXMSRSJXLIWEQIFMPPEGLMIZIHFIXXIVWGSVIWJSVXLMWEQM\SJWERHW


What consultations will do with inputs

;IXSSOMRXSEGGSYRXXLIIEVP]WXEKISJXLITSPMGMIWMRVIPEXMSRXSTPERWJSV8IWXMRKERH)ZEPYEXMSR*SVIEVP]WXEKI

GSRWYPXEXMSRW[IPSSOIHWMQTP]EX[LIXLIVXLI][IVIGPIEVEFSYXXLITPERRIHYWIWSJMRTYXWERHXLIRI\XWXEKI

SJɄTSPMG]ɄHIZIPSTQIRX

8LIVI[EWERMWWYIEGVSWWHITEVXQIRXW[MXLGSRWYPXEXMSRWRSXI\TPEMRMRK[LEX[SYPHFIHSRI[MXLXLIMRJSVQEXMSR

GSPPIGXIH3RII\EQTPIMW(IGGŭWTVSTSWIH'LERKIWXS*MRERGMEP7YTTSVXJSV7SPEV4:?TLSXSZSPXEMGWA[LMGLWGSVIHZIV]

[IPPSR(MEKRSWMW4VSTSWEPERH-QTPIQIRXEXMSRFYXTSSVP]SR8IWXMRKERH)ZEPYEXMSRFIGEYWIMXHMHRSXI\TPEMR[LEX

MXɄ[SYPHHS[MXLXLIɄRI\XVSYRHSJMRJSVQEXMSRKEXLIVMRKMRWYJƤGMIRXHIXEMP(IJVEŭW&ERRMRKXLI&YVRMRKSJ;EWXI3MP

EWɄEɄ*YIPMRɄ,IEXIVW[LMPIWGSVMRKVIEWSREFP][IPPEKEMRWXSXLIVTEVXWSJXLIJVEQI[SVO[EWRSXXVERWTEVIRXEFSYXRI\X

WXITWWGSVMRKEQM\SJɄWERHW

-RGSRXVEWX('0+ŭWXIGLRMGEPGSRWYPXEXMSRSR6IJSVQWXSXLI2I[,SQIW&SRYWXLI,SQI3JƤGIŭWGSRWYPXEXMSRSRXLI

7XEPOMRK4VSXIGXMSR3VHIV(IJVEŭW(VEJX%MV5YEPMX]4PERWERHXLI'EFMRIX3JƤGIŭW)WXEFPMWLMRK'SQQSR1IEWYVIWSJ

7SGMS)GSRSQMG&EGOKVSYRH[IVIEPPGVIHMXIHJSVFIMRKGPIEVEFSYXLS[XLI][SYPHVIWTSRHXSMRTYXW8LI'EFMRIX3JƤGIŭW

)RKPMWL0ERKYEKI6IUYMVIQIRXWJSV4YFPMG7IGXSV;SVOIVW[EWERI\EQTPISJFIMRKEFPIXSKMZIXVERWTEVIRXI\TPEREXMSRW

EFSYXXLMWEXEZIV]IEVP]WXEKISJHIZIPSTQIRXTSJMQTEGXEWWIWWQIRX
35

34,SYWMRKERH4PERRMRK&MPP-QTEGX%WWIWWQIRX3GXSFIVT

35)RKPMWL0ERKYEKI6IUYMVIQIRXJSV4YFPMG7IGXSV;SVOIVWMQTEGXEWWIWWQIRX%YKYWXT

“...The Code of Practice will be prepared once a full understanding of the effect 

SJɅXLIWXEXYXSV]HYX]SRTYFPMGWIVZMGIWLEWFIIRKEMRIHXLVSYKLGSRWYPXEXMSRű

SECTIONS OF THE FRAMEWORK 2.4 

2044 The Promise of Evidence-Based Policymaking
(,ŭWTVSTSWEPJSV)\XIRHMRK'LEVKIWJSV2,77IVZMGIWJSV3ZIVWIEW:MWMXSVWERH1MKVERXW[EWGPIEVEFSYXXLI[E]XLEX

XLIɄGSRWYPXEXMSR[SYPHFIYWIHXSHIZIPSTXLITSPMG]ERH[SYPHLEZIWGSVIHWEGVSWWXLIFSEVHMJMXLEHMRHMGEXIH[LEX

XLIɄFEWMW[EWJSVMXWTPERWJSVMQTPIQIRXEXMSR


(J8ŭW'LERKIWXSXLI(VMZMRK8IWXŬ'EWLFEGOŭTVSTSWEP[EWEXERIEVP]WXEKISJHIZIPSTQIRXERH[EWXVERWTEVIRX

MRɄMXWɄVITSVXEFSYXXLIKETWPIJXF]XLIWXVEXIK]GSRWYPXEXMSR 36

(J)ŭW2EXMSREP*YRHMRK*SVQYPEWGSVIHXSTQEVOWJSVXVERWTEVIRG]EFSYX[LEX[SYPHLETTIRRI\X37

-XMWYRVIEWSREFPIXSI\TIGXTISTPIXSXEOIXLIXMQIERHIJJSVXXSGSRXVMFYXIXSGSRWYPXEXMSRWMJHITEVXQIRXWEVIRSXGPIEV

EFSYX[LEXXLIMVTYVTSWIMW;LIRXLIKSZIVRQIRXŭWGSRWYPXEXMSRTVMRGMTPIW[IVIVIZMWIHMRGEFMRIXWIGVIXEV]

7MVɄ.IVIQ],I][SSHTSMRXIHSYXŰXLMWMWRSXWMQTP]EFSYXSTIRKSZIVRQIRXMXMWEPWSGVYGMEPXSSYVIJJSVXWXSTVSZMHI

XLIɄFIWXTSWWMFPIWIVZMGIXSXLITYFPMGű 38

367EJIWIGYVIWYWXEMREFPI8LI1SXSVMRK7IVZMGIW%KIRGMIW138WJSVEPPQEOIWSJZILMGPIW%TVMPT

372EXMSREPJYRHMRKJSVQYPEIUYEPMX]EREP]WMW1EVGLT

38'SRWYPXEXMSRW[LEXŭWRI[ERH[L]XLI]EVIWSMQTSVXERX7MV.IVIQ],I][SSH.ERYEV]

LXXTWGMZMPWIVZMGIFPSKKSZYOGSRWYPXEXMSRW[LEXWRI[ERH[L]XLI]EVIWSMQTSVXERX

SECTIONS OF THE FRAMEWORK 2.4 24

The Promise of Evidence-Based Policymaking 2045
CROSS-CUTTING ISSUES

%GVSWWXLII\EQTPIW[IPSSOIHEXWSQIQSVIKIRIVEPXLIQIWIQIVKIH[LMGL[IXLMROEPP

HITEVXQIRXW[SYPHFIRIƤXJVSQXEOMRKEGGSYRXSJEWXLI]PSSOXSTVIWIRXRI[TSPMGMIW

;IWIXSYXXLIWIQSVIKIRIVEPƤRHMRKWFIPS[

3.1 Referencing and sharing work done

In many of the less transparent policies, it became clear that departments had looked at evidence or undertaken their 

S[RVIWIEVGLERHEREP]WMWŪMRWSQIGEWIWI\XIRWMZIERHLMKLUYEPMX][SVOŪFYXXLI]LEHRSXWLEVIHXLMWMRXLIMVTSPMG]

HSGYQIRXW8LMWMWETVIVIUYMWMXISJMRJSVQIHHMWGYWWMSRŪJSVXLITYFPMGTEVPMEQIRXERHWTIGMEPMWXWŪERHMXMWIWWIRXMEP

XSɄORS[PIHKIQEREKIQIRX[MXLMRKSZIVRQIRX

-RQER]GEWIWHSGYQIRXWLEHYWIHFYXRSXVIJIVIRGIHIZMHIRGIJVSQXLMVHTEVXMIW

 8LI8VIEWYV]ŭWGSRWYPXEXMSRSR%FSPMWLMRKXLI'EVFSR6IHYGXMSR'SQQMXQIRXWEMHXLIKSZIVRQIRXLEHŰVIZMI[IH

IZMHIRGISRXLISTIVEXMSRSJI\MWXMRKWGLIQIWPSSOIHEXPIWWSRWPIEVRIHJVSQWGLIQIWSZIVWIEWERHWSYKLX

initial views from a range of businesses, academics and other bodies”.

39 It made claims about how energy 

IJƤGMIRG]GERɄWTYVTVSHYGXMZMX]FYXHMHRSXTVSZMHIVIJIVIRGIWSVPMROWJSVXLMW

 8LI(ITEVXQIRXJSV&YWMRIWW-RRSZEXMSRERH7OMPPW&-7
ŭGSRWYPXEXMSRSR1SZMRKXLI3TIVEXMSRWSJXLI0ERH

6IKMWXV]MRXSXLI4VMZEXI7IGXSV[EWEGSQTVILIRWMZITEKIHSGYQIRXFYXHMHRSXVIJIVIRGIWXEXIQIRXWWYGL

EWŰ8LIQSHIPMWSRI[LMGLLEWFVSEHP]WTIEOMRKFIIRMQTPIQIRXIHWYGGIWWJYPP]IPWI[LIVIJSVI\EQTPIMR

Canada”.40%WXLIMQTPIQIRXEXMSRIPWI[LIVI[EWHIWGVMFIHEWEWYGGIWW[I[SYPHI\TIGXXSWIIVIJIVIRGIWXS

assessments. However, even if there had been no easily referenced reports to cite, the department could have 

TVSZMHIHEFVMIJWYQQEV]XSQEOIMXTSWWMFPIXSɄMHIRXMJ]XLII\EQTPIWERHXLIMRHMGEXSVWSJWYGGIWWXLEXEVIFIMRK

referred to.

 *VSQEVIZMI[SJXLIWGSVIWSJXLI(ITEVXQIRXJSV8VERWTSVX(J8
ŭWTSPMG]TVSTSWEP1EOMRK8EGXMPI4EZMRK

7YVJEGIW)EWMIVXS9WIEWɄEɄ2EZMKEXMSREP8SSPERH;EVRMRK7]WXIQ[IFIPMIZIXLEXXLIHITEVXQIRXLEHHSRI

EɄGSRWMHIVEFPIEQSYRXSJɄ[SVOSRXLMWWYFNIGXFYXMXHMHRSXWLEVIMXSVWIXSYXXLI[SVOMRKWJSVXLIWYKKIWXMSR

 8LITVSTSWEPXSQIVKI1ERGLIWXIVŭW0SGEP.YWXMGI%VIEW 41 had an open discussion of alternative approaches, 

particularly on implementation, which would have positioned it for a high transparency score. However it did not 

TVSZMHIQYGLMRJSVQEXMSREFSYXXLIWSYVGIWMX[EWYWMRK8LMW[EWEPWSXVYISJXLI(ITEVXQIRXJSV)RZMVSRQIRX

*SSHERH6YVEP%JJEMVW(IJVE
ŭW7MRKPI%RMQEP)WXEFPMWLQIRX0MGIRGIXLI(ITEVXQIRXSJ,IEPXL(,
ŭW

EWWIWWQIRXSJXLIGEWIJSVEHE]2,7ERH(J)ŭW)HYGEXMSREP)\GIPPIRGI)ZIV][LIVI[LIVI(J)LEHHIZIPSTIH

EQIXLSHSPSK]ERHYRHIVP]MRKHEXEJSVMXWI\TIVMQIRXEPEREP]WMWFYXMXWTVSTSWEPHMHRSXPIEHXLIVIEHIVXSXLMW

These left the basis for the policies open to speculation.

396IJSVQMRKXLIFYWMRIWWIRIVK]IJƤGMIRG]XE\PERHWGETI7ITXIQFIVTTEVE

40'SRWYPXEXMSRSRQSZMRK0ERH6IKMWXV]STIVEXMSRWXSXLITVMZEXIWIGXSV1EVGLTTEVE

418LMWGSRWYPXEXMSR[EWMWWYIHF]1S.SRFILEPJSJ1ERGLIWXIVŭW.YHMGMEP&YWMRIWW+VSYTERH,1'87

-XMWRSXER1S.TSPMG]FYX[ILEZIVIXEMRIHMXFIGEYWIMXMPPYWXVEXIWXLMWMWWYI

CROSS-CUTTING ISSUES 3.1 

2046 The Promise of Evidence-Based Policymaking
(J8ŭWGSRWYPXEXMSRERHMQTEGXEWWIWWQIRXJSVMRGVIEWIHTIREPXMIWJSVYWMRKQSFMPITLSRIW[LMPIHVMZMRKMRGPYHIH

EGGIWWMFPIPMROWXSXLIIZMHIRGIFEWIXLVSYKLSYX43

%WHMHXLI'EFMRIX3JƤGIŭWIRKEKIQIRXHSGYQIRXSR)WXEFPMWLMRK'SQQSR1IEWYVIWSJ7SGMS)GSRSQMG&EGOKVSYRH 44

42'LMPH7EJIX]3RPMRI%KI:IVMƤGEXMSRJSV4SVRSKVETL]*IFVYEV]T

43%GSRWYPXEXMSRSRGLERKIWXSXLI*M\IH4IREPX]2SXMGIERHTIREPX]TSMRXWJSVXLIYWISJELERHLIPHQSFMPI

TLSRI[LMPWXHVMZMRK.ERYEV]T

44)RKEKIQIRX(SGYQIRX(IZIPSTMRKE'SQQSRWIXSJ1IEWYVIWJSV)QTPS]IVWSRXLI7SGMS)GSRSQMG

&EGOKVSYRHWSJXLIMV;SVOJSVGIERH%TTPMGERXW1E]T

8LIVI[IVITPIRX]SJI\EQTPIW[LIVIXLITSPMG]VIWIEVGL[EWXVERWTEVIRXERHMRGPYHIHVIJIVIRGIWERHPMROWXSXLIIZMHIRGI

FEWI*SVI\EQTPIXLI(ITEVXQIRXJSV'YPXYVI1IHMEERH7TSVX('17
ŭTVSTSWEPJSV%KI:IVMƤGEXMSRJSV4SVRSKVETLMG

1EXIVMEP3RPMRI[EWGPIEVEFSYXLS[MXLEHYWIHɄWSYVGIW 42

CROSS-CUTTING ISSUES 3.1 

The Promise of Evidence-Based Policymaking 2047
-RWSQIGEWIWEHITEVXQIRXLEHYRHIVXEOIRSVɄGSQQMWWMSRIHMXWS[RVIWIEVGLFYXHMHRSXQIRXMSRMXSVVIJIVIRGIMX

8LIɄ,SQI3JƤGIŭWTVSTSWEPXS7XVIRKXLIRXLI6SPISJXLI(MWGPSWYVIERH&EVVMRK7IVZMGI(&7
MRMXWGSYRXIVI\XVIQMWQ

WXVEXIK][EWWIXSYXEWJSPPS[W 45

In follow up with the department, it transpired that research had been done prior to this announcement and fed into 

this decision, including a remodelling review46SJXLI(&7ERHE'VMQMREP6IGSVHW6IKMQIVIZMI[47 None of these were 

mentioned or referenced. 

&]GSRXVEWX('17ŭ[LMXITETIVSRXLI*YXYVISJXLI&&'WGSVIH[IPPJSVMRGPYHMRKXLIVIZMI[FILMRHMXWTVSTSWEP

XSɄMRXVSHYGIE9RMXEV]&SEVHJSVXLISVKERMWEXMSRERHJSVWTIGMƤGVIJIVIRGIWXSXLIVIPIZERXTEVXWSJMXXLEXLEHFIIR

YWIHɄXSɄWLETIXLITSPMG]

When a department has undertaken its own research or reviews to inform its policies, these should be included 

SVɄEGGIWWMFPI

45'SYRXIV)\XVIQMWQ7XVEXIK]3GXSFIVT

46:IXXMRKERH&EVVMRK7GLIQIVIQSHIPPMRKVIZMI[(MWGPSWYVIERH&EVVMRK7IVZMGIXL*IFVYEV]

https://www.gov.uk/government/publications/vetting-and-barring-scheme-remodelling-review-reportand-recommendations

47'VMQMREP6IGSVHW6IKMQIVIZMI[TLEWIX[S,SQI3JƤGI2SZIQFIVLXXTW[[[KSZYO

government/publications/criminal-records-regime-review-phase-two

CROSS-CUTTING ISSUES 3.1 

2048 The Promise of Evidence-Based Policymaking
3.2 Gesturing vs referencing

8LITVSZMWMSRSJPMROWERHVIJIVIRGIWHSIWRSXMXWIPJJYPP]WEXMWJ]XVERWTEVIRG]VIUYMVIQIRXW-XQMKLXWIIQSFZMSYW

FYXɄXLIɄVIJIVIRGIWHSRIIHXSTVSZMHIWTIGMƤGIRSYKLPMROWXSIREFPITISTPIXSYRHIVWXERH[LEXTMIGISJIZMHIRGI[EW

used to inform which aspect of the policy proposal.

;IWE[I\EQTPIW[LIVIPMROWPIHXSEGSPPIGXMSRSJTSPMG]HSGYQIRXWSVHITEVXQIRXEPLSQITEKIW[MXLRSGPIEVHMVIGXMSR

XSXLIIZMHIRGIFEWI8LITPERWJSV*YPP%GEHIQMWEXMSRSJXLI7GLSSP7]WXIQPEMHSYXMR(J)ŭWEducational Excellence 

Everywhere white paper provided a footnote for the methodology behind a map showing the performance of schools 

MRɄ)RKPERH 48

&YXMXPIHXLIVIEHIVXSEGSPPIGXMSRSJTYFPMGEXMSRWEFSYXXVERWTEVIRG]SR(J)ŭWKSZYO[IFWMXI 49

%RSXLIVPMROXSEŬJYPPIVWXEXIQIRXSJ(J)ŭWWXVEXIK]ŭPIEHWXS(J)ŭWKSZYOLSQITEKI-RFSXLGEWIWEQIQFIVSJXLITYFPMG

LSTMRKXSPSSOEXXLIIZMHIRGIFEWI[SYPHIRGSYRXIVWYFWXERXMEPHMJƤGYPXMIW

48)HYGEXMSREP)\GIPPIRGI)ZIV][LIVI1EVGLT

49LXXTW[[[KSZYOKSZIVRQIRXTYFPMGEXMSRW#OI][SVHW!
TYFPMGEXMSRCƤPXIVCSTXMSR!XVERWTEVIRG]

HEXE
XSTMGW	&	(!EPP
HITEVXQIRXW	&	(!HITEVXQIRXJSVIHYGEXMSR
SJƤGMEPCHSGYQIRXC

WXEXYW!EPP
[SVPHCPSGEXMSRW	&	(!EPP
JVSQCHEXI!
XSCHEXI!

CROSS-CUTTING ISSUES 3.2 

The Promise of Evidence-Based Policymaking 2049
)ZIRXVERWTEVIRXTSPMG]HSGYQIRXWWSQIXMQIWWGSVIHPIWW[IPPSRXLMWTSMRX8LI,SQI3JƤGIŭW7MRKPI0IKMWPEXMZI

4VSZMWMSRXS4VSZMHIJSV)UYMTQIRX-RXIVJIVIRGIERMQTEGXEWWIWWQIRX
[EWGPIEVEFSYXXLIWSYVGIWJSVMXWFEGOKVSYRH

XSɄXLIMWWYIERH[SYPHLEZIWGSVIHFIXXIVMJMXLEHVIJIVIRGIHXLIWIQSVITVIGMWIP]%W[SYPHXLI(ITEVXQIRXJSV;SVO

ERHɄ4IRWMSRW(;4
ŭW6IHYGMRKXLI&IRIƤX'ETXSeeJSV+VIEXIV0SRHSR
[LMGLMRGPYHIHEKSSHHMWGYWWMSR

SJXLIIZMHIRGIFYX[EWUYMXIMRGSRWMWXIRXMRTVSZMHMRKVIJIVIRGIWJSVMXIK 50

8LI1MRMWXV]SJ.YWXMGI1S.
ŭW'ETSRXLI*IIWXLEX6IKYPEXIH'PEMQW1EREKIQIRX'SQTERMIWGER'LEVKI'SRWYQIVW

referenced work by the Financial Ombudsman and the FCA, but it was hard to tell when referenced material ended and new 

material began.

Transparent can be short and simple

-X[SYPHFIEQMWXEOIXSMQEKMRIXLEXEHSGYQIRXGVS[HIH[MXLVIJIVIRGIWSVI\XIRWMZII\XVEGXWMWFIXXIVKVSYRHIHSVQSVI

transparent. References should be meaningful and useful, to enable the reader to understand how the source is relevant 

ERHXSIREFPIXLIQXSEWWIWWXLEXWSYVGIJSVXLIQWIPZIWMJXLI][MWL[LIRMXMWXLIFEWMWJSVEWMKRMƤGERXGSRGPYWMSRSVXLI

authority of a policy statement.

0IRKXLMWRSXETVIVIUYMWMXISJXVERWTEVIRG]-XMWRSXRIGIWWEV]XSMRGPYHIZSPYQMRSYWEQSYRXWSJQEXIVMEP;LEXQEXXIVW

JSVXVERWTEVIRG]MWGPIEVEREP]WMWSJIZMHIRGIERHHMWGYWWMSRSJMXWVIPEXMSRWLMTXSXLITSPMG];IWE[KSSHI\EQTPIWSJXLMW

FIMRKEGLMIZIHMRWSQISJXLIWLSVXIWXHSGYQIRXW[II\EQMRIH

 ŵ(IJVEŭWGSRWYPXEXMSRSR7XVIRKXLIRIH1IEWYVIWEKEMRWX)TMXVM\[EWNYWXTEKIWPSRK-XGSRXEMRIHGPIEV

VIJIVIRGMRKERYTJVSRXHMWGYWWMSRSJXLIPMQMXEXMSRWSJXLITSPMG]ERHIZMHIRGIERHGPIEVMHIRXMƤGEXMSRSJ[LEX

WYFQMWWMSRWXSXLIGSRWYPXEXMSR[SYPHEGLMIZI8LMW[EWNYHKIHXSFINYWXEWKSSHEWERHF]WSQIWGSVIVW

WPMKLXP]FIXXIVXLER
XLI(ITEVXQIRXSJ)RIVK]ERH'PMQEXI'LERKI(IGG
ŭW*IIHMR8EVMJJWTSPMG]TVSTSWEP

[LMGLLEWSZIVTEKIWSJQEXIVMEP

 ŵ8LI,SQI3JƤGIŭWTEKIGSRWYPXEXMSRSRXLI7XEPOMRK4VSXIGXMSR3VHIV[EWXVERWTEVIRX8LIVIEWSRMRK[EW

clearly laid out and included discussion of the evidence of an ongoing problem that was not being addressed 

F]I\MWXMRKQIEWYVIWERHXLITSXIRXMEPSJXLMWJYVXLIVQIEWYVIXSEHHVIWWMX8LMWEGLMIZIHXLIWEQIPIZIP

SJɄXVERWTEVIRG]EW(,ŭW(IEXL'IVXMƤGEXMSR6IJSVQW[LMGL[EWSZIVTEKIW

(J)ŭW7GLSSPW2EXMSREP*YRHMRK*SVQYPE[LMPIWGSVMRK[IPPSR(MEKRSWMWERH8IWXMRKERH)ZEPYEXMSRGSRXEMRIHWSQYGL

material across four documents that scorers found it bewildering to work out what evidence had actually been used. 

50;IPJEVI6IJSVQERH;SVO&MPP-QTEGX%WWIWWQIRXJSVXLIFIRIƤXGET.YP]T

CROSS-CUTTING ISSUES 3.2 

2050 The Promise of Evidence-Based Policymaking
A note on referencing practice

8LI+SZIVRQIRX(MKMXEP7IVZMGILEWEWX]PIKYMHI 51 The Regulatory Policy Committee guidance on impact 

EWWIWWQIRXWɄEWOW

“Does the IA reference the source of data, research and evidence used and is the robustness of each of these 

GPIEVP]ɄHIQSRWXVEXIH#ű 52

&YXEQSVIWYFWXERXMEPWIXSJTVMRGMTPIWQE]FIYWIJYP8LIVIEVIWSQIXVERWTEVIRG]VIPIZERXTSMRXWJVSQXLI'SQQMXXII

SR4YFPMGEXMSR)XLMGW 53

 • Ű%YXLSVWWLSYPHVITVIWIRXXLI[SVOSJSXLIVWEGGYVEXIP]MRGMXEXMSRWERHUYSXEXMSRWű

 • Ű?ŷA%YXLSVWWLSYPHRSXGST]VIJIVIRGIWJVSQSXLIVTYFPMGEXMSRWMJXLI]LEZIRSXVIEHXLIGMXIH[SVOű

 • “4.3 Relevant previous work and publications, both by other researchers and the authors’ own, should be properly 

EGORS[PIHKIHERHVIJIVIRGIH8LITVMQEV]PMXIVEXYVIWLSYPHFIGMXIH[LIVITSWWMFPIű

To this we would add the points above about being clear about what aspect of the policy document is being referenced 

XSɄ[LEXEWTIGXSJXLIWSYVGI

51LXXTW[[[KSZYOKYMHERGIWX]PIKYMHIEXS^SJKSZYOWX]PIZMI[IH3GXSFIV


52 https://www.gov.uk/government/publications/how-the-regulatory-policy-committee-scrutinisesimpact-assessments/regulatory-policy-committee-recommendations-used-when-scrutinising-impactEWWIWWQIRXWTVSHYGIVIPMEFPIIWXMQEXIWSJGSWXWERHFIRIƤXW-XIQZMI[IH3GXSFIV


53;EKIV)
/PIMRIVX7
6IWTSRWMFPIVIWIEVGLTYFPMGEXMSRMRXIVREXMSREPWXERHEVHWJSVEYXLSVW%

TSWMXMSRWXEXIQIRXHIZIPSTIHEXXLIRH;SVPH'SRJIVIRGISR6IWIEVGL-RXIKVMX]7MRKETSVI.YP]

'LETXIVMR1E]IV8
7XIRIGO2IHW
4VSQSXMRK6IWIEVGL-RXIKVMX]MRE+PSFEP)RZMVSRQIRX

-QTIVMEP'SPPIKI4VIWW;SVPH7GMIRXMƤG4YFPMWLMRK7MRKETSVITT
-7&2


CROSS-CUTTING ISSUES 3.2 

The Promise of Evidence-Based Policymaking 2051
3.3 Absent or weak evidence

In the absence of evidence

+SZIVRQIRXWSJXIRLEZIXSEGX[LIVIXLIIZMHIRGIFEWIMW[IEOSVEFWIRX8LI]QE]RSXLEZIXLIPY\YV]SJ[EMXMRKJSV

XLSWIKETWXSFIƤPPIHFIJSVIXLI]MRXVSHYGITVSTSWEPW

8LIQSWXXVERWTEVIRXTSPMG]HSGYQIRXW[IVIXLSWIXLEXEGORS[PIHKIHXLMWERHI\TPEMRIHLS[XLIHITEVXQIRX[SYPHƤPP

XLIKETSVIZEPYEXIXLITSPMG]EXEPEXIVTSMRX8LIGSRWYPXEXMSRSR6IHYGMRK(MWVYTXMSRSR0SGEPŬ%ŭ6SEHW[EWGPIEVEFSYX

[LEXXLIHITEVXQIRX(J8
HMHRŭXORS[ 54

8LIHITEVXQIRXI\TPEMRIHXLEXMX[SYPHYWIXLIGSRWYPXEXMSRXSƤPPMRKETWMRXLIIZMHIRGIFEWI 55

%RSXLIVI\EQTPIMW1S.ŭWTVSTSWIH4ERIPJSV4YFPMGP]*YRHIH'VMQMREP%HZSGEG]8LIGSRWYPXEXMSRLMKLPMKLXIHEVIEW

[LIVIUYERXMXEXMZIIZMHIRGIMWPMQMXIH 56

8LIGSRWYPXEXMSRHSGYQIRXEPWSEGORS[PIHKIHTSMRXW[LIVIXLIIZMHIRGIMWPEVKIP]ERIGHSXEP 57

546IHYGMRK(MWVYTXMSRSR0SGEPŬ%ŭ6SEHW-QTEGX%WWIWWQIRX
%TVMPT%RRI\%2YQFIVSJ[SVOWMRWGSTISJXLIVIUYMVIQIRX

556IHYGMRK(MWVYTXMSRSR0SGEPŬ%ŭ6SEHW-QTEGX%WWIWWQIRX
%TVMPT

564VIWIVZMRKERH)RLERGMRKXLI5YEPMX]SJ'VMQMREP%HZSGEG]3GXSFIVT

574VIWIVZMRKERH)RLERGMRKXLI5YEPMX]SJ'VMQMREP%HZSGEG]3GXSFIVT

CROSS-CUTTING ISSUES 3.3 

2052 The Promise of Evidence-Based Policymaking
Weakness in the evidence base

8LIQSWXXVERWTEVIRXTSPMG]HSGYQIRXWEGORS[PIHKIH[IEORIWWIWERHHMWGYWWIHXLIQ*SVI\EQTPI(IJVEŭW(VEJX%MV

5YEPMX]4PERWLEHERIEWMP]EGGIWWMFPIWYQQEV]SJEWWYQTXMSRWERHEWWSGMEXIHYRGIVXEMRXMIW8LMW[SYPHWGSVIE 58

8LIWEQIHITEVXQIRXŭWTVSTSWEPWJSV7XEXYXSV]4SWX1SZIQIRX8IWXMRKSJ'EXXPIJSV8&EPWSMRGPYHIHEWIRWMXMZMX]EREP]WMW

XSGETXYVIYRGIVXEMRXMIWMRXLIIZMHIRGIFEWIMRXLMWGEWIMRXLIHITEVXQIRXŭWS[RQSHIPPMRK 59

58(VEJX)ZMHIRGI%RRI\%WWIWWQIRXSJXLITPERWXSMQTVSZIEMVUYEPMX]MRXLI9/7ITXIQFIVTT

593TXMSRWXSMRGVIEWIXLIGLERGISJEGLMIZMRK3JƤGMEPP]8&*VII38*
WXEXYWJSVXLI8&0S[6MWO%VIE-QTEGX%WWIWWQIRX
%YKYWXT

CROSS-CUTTING ISSUES 3.3 

The Promise of Evidence-Based Policymaking 2053
('17ŭGSRWYPXEXMSRSR%KI:IVMƤGEXMSRJSV4SVRSKVETLMG1EXIVMEP3RPMRI[EWXVERWTEVIRXEFSYXXLIPEGOSJEGEHIQMG

GSRWIRWYWMXLEHIRGSYRXIVIH 60

(;4ŭWTSPMG]&ERRMRK1IQFIV&SVRI'SQQMWWMSRMR;SVOTPEGI4IRWMSRWXSTVIZIRXIQTPS]IIWTE]MRKGSQQMWWMSR

fees for occupational pensions, was also transparent about uncertainties caused by the available evidence (as well 

EWɄHIQSRWXVEXMRKLS[MXLEHHVE[RSRTVIZMSYWGSRWYPXEXMSRWXSERW[IVWSQITSMRXW


3RXLISXLIVLERHXLI+SZIVRQIRX)UYEPMXMIW3JƤGIŭW+IRHIV4E]+ETTVSTSWEPWMRGPYHIHEPSXSJEWWIVXMSRWERHWXEXMWXMGW

EFSYXTE]KETWFYXHMHRSXGSRWMHIVXLI[IEORIWWIWERHGSRXVEHMGXMSRWMR[LEXMXLEHMRGPYHIH(IGGŭW'SRXVEGXWJSV

(MJJIVIRGIJSV'EVFSR'ETXYVIERH7XSVEKILEHEWMQMPEVMWWYI8LMWOMRHSJSQMWWMSR[EWGSQQSRERHEVIEWSR[L]

WSQIXVERWTEVIRXTSPMGMIWHMHRSXWGSVIE*SVXLIQSWXTEVXHITEVXQIRXWSRP]VIJIVVIHXSKETWMRORS[PIHKIMRVIPEXMSR

XSɄGSRWYPXEXMSRUYIWXMSRWSVI\TPEMRMRKQMWWMRKMRJSVQEXMSRMRERMQTEGXEWWIWWQIRX8LIWXVIRKXLWERH[IEORIWWIW

SJɄXLIIZMHIRGIFEWI[IVIVEVIP]HMWGYWWIH

60'LMPH7EJIX]3RPMRI%KI:IVMƤGEXMSRJSV4SVRSKVETL]*IFVYEV]TT

CROSS-CUTTING ISSUES 3.3 

2054 The Promise of Evidence-Based Policymaking
3.4 Values-based policies

+SZIVRQIRXWWSQIXMQIWMRXVSHYGITSPMGMIWXLEXEVITVIHSQMRERXP]I\TVIWWMSRWSJZEPYIWŪVEXLIVXLERNYWXMƤIHNYWX

F]ɄXLIMVMRXIRHIHSYXGSQIWERHIJJIGXMZIRIWW-RXLSWIGEWIWXLIVIQE]FIPIWWSJEVSPIJSVIZMHIRGI

4SPMGMIW[IVIGSRWMHIVIHXSEGLMIZIEFEWMGPIZIPSJXVERWTEVIRG]MJXLI][IVIGPIEVEFSYXXLIMVVEXMSREPIFIMRKZEPYIWFEWIH

VEXLIVXLERIZMHIRGIFEWIHERHMJXLI]WYTTSVXIHER]XIWXEFPIGPEMQWEFSYXXLIWMXYEXMSRXLI][IVIEHHVIWWMRK8LI][IVI

GSRWMHIVIHXSFIJYPP]XVERWTEVIRXMJXLI]EPWSHMWGYWWIHXLIIZMHIRGIERHI\TPEMRIHXLIPMQMXWSJMXWVSPIMRHIZIPSTMRK

XLIɄTSPMG]

Values-based policies

8LIVI[IVIMRJEGXZIV]JI[TSPMGMIWXLEX[IVIRSXFEWIHSRWSQIXIWXEFPIGPEMQWSVEWWYQTXMSRW7II%TTIRHM\JSV

I\EQTPIWSJXIWXEFPIWXEXIQIRXWZIVWYWZEPYIWXEXIQIRXW
8LISRP]I\EQTPI[IEWWIWWIHXSFITYVIP]ZEPYIWFEWIH

[EWWYVTVMWMRKP]XLI8VIEWYV]ŭWGSRWYPXEXMSRSR6IJSVQWXSXLI8E\EXMSRSJ2SR(SQMGMPIW8LIƤVWXWXEXIQIRX[LMGL

MWɄXIWXEFPI[EWHMWVIKEVHIHEWXLMW[EWRSXXLIWXEXIHKSEPSJXLITSPMG]-X[EWGPIEVXLEXXLIGLERKIW[IVIFIMRKTVSTSWIH

SRXLITVMRGMTPISJJEMVRIWW 61

8LI(ITEVXQIRXJSV'SQQYRMXMIWERH0SGEP+SZIVRQIRX('0+
ŭW9THEXMRKXLI0SGEP+SZIVRQIRX8VERWTEVIRG]'SHI

WGSVIHQEMRP]WERHWSQIWFIGEYWIMX[EWGPIEVEFSYXXLIɄZEPYIWFEWMWSJMXWVEXMSREPIŪXLEXXLIKSZIVRQIRXFIPMIZIW

transparency is the bedrock of accountability — but also made claims about the use and function of the data that it did not 

provide any references for.

%GORS[PIHKMRKHMJJIVIRXMRƥYIRGIWEPWSLIPTWXLITYFPMGXSJSPPS[XLIGLEMRSJVIEWSRMRKFILMRHXLITSPMG]ERHXSWIILS[

IZMHIRGILEWFIIR[IMKLIHEPSRKWMHISXLIVTVIWWYVIW(J8ŭWGSRWYPXEXMSRERHMQTEGXEWWIWWQIRXJSV'LERKIWXSXLI*M\IH

4IREPX]2SXMGIERH4IREPX]4SMRXWJSVXLI9WISJE,ERH,IPH4LSRI;LMPWX(VMZMRKEGORS[PIHKIHLS[KVS[MRKTVIWWYVI

JVSQXLIQIHMEERHTYFPMGSTMRMSRLEHJIHMRXSXLIHIGMWMSR 62

616IJSVQWXSXLIXE\EXMSRSJRSRHSQMGMPIW7ITXIQFIVWIGXMSR*SVI[SVH

62%GSRWYPXEXMSRSRGLERKIWXSXLI*M\IH4IREPX]2SXMGIERHTIREPX]TSMRXWJSVXLIYWISJELERH

LIPHQSFMPITLSRI[LMPWXHVMZMRK1SZMRK&VMXEMR%LIEH%YKYWXT

CROSS-CUTTING ISSUES 3.4 

The Promise of Evidence-Based Policymaking 2055
Manifesto and political commitments

1ER]TSPMGMIWEVIMRXVSHYGIHEWEVIWYPXSJIPIGXMSRTVSQMWIWMRQERMJIWXSWŪERHXLIVIMWETYFPMGI\TIGXEXMSRXLEX

governments deliver on these commitments. 633JƤGMEPWSJXIRVEMWIHXLMW[MXLYWEWEPMQMXEXMSRSRXVERWTEVIRG]EFSYX

the use of evidence, suggesting there should be different evidence standards for policies that come out of electoral 

competition to those which are developed inside government. However, we found that policies that had originated 

MRɄQERMJIWXSWJIEXYVIHEQSRKXLIFIWXERHXLI[SVWXJSVXVERWTEVIRG];LEXHMHQEXXIV[EW[LIXLIVXLIGSQQMXQIRX

focused on the QIERWŪEGSQQMXQIRXXSHSWSQIXLMRK[LIVIXLIIZMHIRGIFEWI[EWSJXIRRSXXVERWTEVIRXSVXLIends

— the outcome the policy was intended to achieve, which left space for evidence to be considered about the best way to do 

that.

(;4ŭW=SYXL3FPMKEXMSRERH(,ŭW(E]2,7EVSWIJVSQQIERWJSGYWIHQERMJIWXSGSQQMXQIRXWERHKEZIPMQMXIH

MRJSVQEXMSREFSYXXLIWSYVGISJXLITSPMG](J)ŭWTVSTSWEPJSVMQTPIQIRXMRKXLI)RKPMWL&EGGEPEYVIEXI[EWFEWIHSRE

WTIGMƤGQERMJIWXSGSQQMXQIRXXSŰVIUYMVIWIGSRHEV]WGLSSPTYTMPWXSXEOI+'7)WMR)RKPMWLQEXLWWGMIRGIEPERKYEKI

and history or geography”.649RPMOIXLISXLIVWMXLEHEVIEWSREFP]XVERWTEVIRXERH[IPPGMXIHI\TPSVEXMSRSJXLITVSFPIQ

it was trying to address, but less transparency on why this was the best way of addressing the problem and its costs 

ERHɄFIRIƤXW

&-7ŭTVSTSWEPJSV&EPPSX8LVIWLSPHW?JSVWXVMOIEGXMSRAMR-QTSVXERX4YFPMG7IVZMGIWLEHXS[SVO[MXLEZIV]WTIGMƤG

QERMJIWXSGSQQMXQIRXŰ-RHYWXVMEPEGXMSRMRXLIWIIWWIRXMEPWIVZMGIW[SYPHVIUYMVIXLIWYTTSVXSJEXPIEWXTIVGIRXSJEPP

those entitled to take part in strike ballots”.658LIWGSVIVWJIPXXLEXXLIGSRWYPXEXMSR[EWYRGPIEV[L]XLI	XLVIWLSPHLEH

been used rather than any other number.

7TIGMƤGQERMJIWXSGSQQMXQIRXWXSXLIQIERWSJEGLMIZMRKSYXGSQIWQE]LMRHIVXLIKSZIVRQIRXŭWWYFWIUYIRXEFMPMX]

XSGSRWMHIVIZMHIRGISREPXIVREXMZITVSTSWEPW-RGSRXVEWXXLI(J8ŭWTVSTSWEPJSV'LERKIWXSXLI*M\IH4IREPX]2SXMGI

ERH4IREPX]4SMRXWJSVXLI9WISJE,ERH,IPH4LSRI;LMPWX(VMZMRK[EWPMROIHXSERSTIRIRHIHGSQQMXQIRXŰŷVIHYGI

XLIRYQFIVSJG]GPMWXWERHSXLIVVSEHYWIVWOMPPIHERHMRNYVIHSRSYVVSEHWIZIV]]IEVű.

66 The evidence in this was clearly 

GMXIHERHHMWGYWWIH8LI'EFMRIX3JƤGIŭW)RKPMWL0ERKYEKI6IUYMVIQIRXWJSV4YFPMG7IGXSV;SVOIVW[EWFEWIHSRER

SYXGSQIWSVMIRXIHQERMJIWXSGSQQMXQIRXXSŰPIKMWPEXIXSIRWYVIXLEXIZIV]TYFPMGWIGXSV[SVOIVSTIVEXMRKMREGYWXSQIV

JEGMRKVSPIQYWXWTIEOƥYIRX)RKPMWLű.

67;LMPIMXTVSHYGIHQM\IHWGSVIWJSVXVERWTEVIRG]MXHMH[IPPSRERSTIRHMWGYWWMSR

about alternative options to achieve this.

1SVIMQTPIQIRXEXMSRJSGYWIHGSQQMXQIRXWXLEX[IVIWXMPPFVSEHERHRSXXMIHXSWTIGMƤGQIEWYVIWEPWSWGSVIHUYMXI[IPP

&-7ŭTVSTSWEPJSVE7QEPP&YWMRIWW'SQQMWWMSRIVGEQIJVSQEFVSEHQERMJIWXSGSQQMXQIRXXSIWXEFPMWLŰERI[WQEPP

business conciliation service to mediate in disputes, especially over late payment”.

68 It had a transparent diagnosis of the 

TVSFPIQ[MXLVIJIVIRGIWXSXLIWYVZI]WERHVIWIEVGLEFSYXXLIHMWEHZERXEKIWWQEPPFYWMRIWWIWJEGI('17ŭTVSTSWEPJSV

%KI:IVMƤGEXMSRJSV4SVRSKVETLMG1EXIVMEP3RPMRI[EWGPIEVEFSYXIPIQIRXWSJXLITSPMG]]IXXSFIHIGMHIHSRJSVɄMRWXERGI

EGMZMPZIVWYWGVMQMREPIRJSVGIQIRXVIKMQIXLIKSZIVRQIRXŭWTVIJIVVIHSTXMSR
ERHEGORS[PIHKIHYRGIVXEMRXMIWMRXLI

evidence base, which is a marker of transparency.

63 8VYWXMRKSZIVRQIRXMWKVS[MRKFYXMXRIIHWXSHIPMZIV-RWXMXYXIJSV+SZIVRQIRXXL7ITXIQFIV


LXXT[[[MRWXMXYXIJSVKSZIVRQIRXSVKYOWMXIWHIJEYPXƤPIWTYFPMGEXMSRW-J+CTSPPMRKCRSXIC;)&THJ

64'SRWIVZEXMZI4EVX]1ERMJIWXSLXXTWWIY[IWXEQE^SRE[WGSQQERMJIWXS

'SRWIVZEXMZI1ERMJIWXSTHJT

65-FMHT

66-FMHT

67-FMHT

68-FMHT

CROSS-CUTTING ISSUES 3.4 

2056 The Promise of Evidence-Based Policymaking
Budget announcements

7SQISJXLITSPMGMIWVIZMI[IHLEHFIIRERRSYRGIHF]XLI8VIEWYV]EXEFYHKIXSVWTIRHMRKVIZMI[ERHXLIRŬLERHIH

HS[RŭXSSXLIVHITEVXQIRXWJSVJYVXLIVHIZIPSTQIRX;IJSYRHXLEXTVSTSWEPWXLEXSVMKMREXIHMRXLI&YHKIXSV%YXYQR

7XEXIQIRX[IVITEVXMGYPEVP]RSRXVERWTEVIRXEFSYXXLIIZMHIRGIFEWIYWIH;IEVIRSXXLIƤVWXXSƤRHXLMWXLI7SGMEP

7IGYVMX]%HZMWSV]'SQQMXXIIŭWGLEMVQER4EYP+VE]VIGIRXP]GSQTPEMRIHEFSYXXLIUYEPMX]SJ[IPJEVIQIEWYVIWTVIWIRXIH

XSɄLMWGSQQMXXIISVMKMREXMRKMRXLI&YHKIXŰWIGSRHEV]PIKMWPEXMSR?MWAFIMRKTVIWIRXIHXSYW[MXLSYXQIERMRKJYPEREP]WMW

SJɄMQTEGXSVMRXIVEGXMSRW[MXLSXLIVTEVXWSJXLIFIRIƤXW]WXIQű.

69

*SVXLMWEWWIWWQIRX[LIREFYHKIXERRSYRGIHEGSRWYPXEXMSR[IWSYKLXSYXXLSWIHSGYQIRXWXSEWWIWWMRWXIEHSJNYWX

VIP]MRKSRXLIQEXIVMEPTVSHYGIHEXXLIXMQISJXLIERRSYRGIQIRX[LIVIXLIKSZIVRQIRXLEHQSZIHWXVEMKLXXSEGXMSR

without any intervening stages, we looked at the budget or spending announcement.

70

8LI=SYXL3FPMKEXMSR[EWEQERMJIWXSGSQQMXQIRXFVSYKLXJSV[EVHF]XLI7YQQIV&YHKIX71

8LIVI[EWERSYXPMRISJGLERKIWXSWTIRHMRKSRT8LMW[EWXLII\XIRXSJXLIKSZIVRQIRXŭWTVSTSWEPEXXLMWWXEKI

7GSVIVWJSYRHRSVIJIVIRGIXSER]OMRHSJIZMHIRGIEFSYXXLITVSFPIQWXLEXXLI=SYXL3FPMKEXMSRWSYKLXXSXEGOPIWYGL

EWɄLS[QER]]SYRKTISTPIEVISYXSJ[SVO[L]XLITVSTSWEP[EWGLSWIRERHLS[MXGSYPHFIHIPMZIVIH

('17ŭGSRWYPXEXMSRSR&YWMRIWW6EXIW6IPMIJJSV0SGEP2I[WTETIVW[LMGLSVMKMREXIHJVSQEQERMJIWXSGSQQMXQIRX

ERHɄ&YHKIXERRSYRGIQIRXHMHRSXMRHMGEXI[LEXIZMHIRGILEHFIIRYWIHXSGSRGPYHIXLIGYVVIRXW]WXIQMWRSX[SVOMRK

JSVɄPSGEPRI[WTETIVWERHKEZIPMXXPIMRHMGEXMSRSJLS[GSWXW[IVIEWWIWWIH

8LIVIETTIEVWXSFIEW]WXIQMGTVSFPIQEWWSGMEXIH[MXLTSPMGMIWERRSYRGIHMR&YHKIXW[LMGLRIIHWXSFIEHHVIWWIH

3RITEVXMGYPEVJIEXYVI[LMGLGSYPHI\TPEMRXLMWMWXLI[E]MR[LMGL&YHKIXTVSTSWEPWEVIHIZIPSTIHŪ[MXLXLI8VIEWYV]

EGXMRKEWXLITSPMG]TVSQSXIV[LSXLIRLERHWETSPMG]SRXSEHITEVXQIRXXSMQTPIQIRX8LIJEGXXLEXQER]&YHKIX

QIEWYVIWEVIHIZIPSTIHMRWIGVIXERHXLI8VIEWYV]HSIWRSXTPE]MXWRSVQEPVSPIEWTSPMG]GLEPPIRKIVQE]I\TPEMR

FYXɄRSXɄI\GYWI
[L]XLIVIEVIWSQER]XVERWTEVIRG]MWWYIWEVSYRH&YHKIXERRSYRGIQIRXW 72

3RI[E]XSEHHVIWWXLMWQMKLXFIXSIRWYVIXLEXEPP&YHKIXERRSYRGIQIRXWEVIJSPPS[IHF]EJYVXLIVTVIMQTPIQIRXEXMSR

TSPMG]TVSTSWEP;IRSXIHXLEX[LIVIGSRWYPXEXMSRWLEHJSPPS[IHWTIRHMRKERRSYRGIQIRXW[LIXLIVJVSQXLI&YHKIX

SVMRHIIHEWTIRHMRKVIZMI[
XVERWTEVIRG][EWFIXXIV(IGGŭWGSRWYPXEXMSRSRXLI)\IQTXMSRJVSQXLI'SWXWSJXLI

6IRI[EFPIW3FPMKEXMSRERH*IIHMR8EVMJJ[LMGL[EWERRSYRGIHMRXLI7TIRHMRK6IZMI[[EWXVERWTEVIRX8LI'ET

SR)EVP])\MX'LEVKIWJSV1IQFIVWSJ3GGYTEXMSREP4IRWMSR7GLIQIWERRSYRGIHF]XLI8VIEWYV]FYXWIXSYXMRHIXEMP

F](;4GPIEVP]QSHIPPIHGSWXWERHFIRIƤXWI\TPEMRIH[LEX[EW]IXXSFIHIGMHIHERHEGORS[PIHKIHGSRXVEHMGXSV]

information about that. 

697SGMEP7IGYVMX]%HZMWSV]'SQQMXXII%RRYEP6ITSVXT

70;IMRMXMEPP]MRGPYHIHXLI8VIEWYV]ŭW7SJX(VMRO0IZ]ERRSYRGIQIRXFYXXLMWMRGPYHIHTPERWJSVEGSRWYPXEXMSR

which fell outside our time frame. 

717YQQIV&YHKIX,18VIEWYV].YP]T

728LMWMWWYIMWVEMWIHMRXLI-RWXMXYXIJSV+SZIVRQIRXŭWƤRHMRKWSRXE\EXMSRTSPMG]-QTVSZMRKXE\TSPMG]QEOMRK)QIVKMRKƤRHMRKWER

MRZMXEXMSRXSGSQQIRX-RWXMXYXIJSV+SZIVRQIRXXLI'LEVXIVIH-RWXMXYXISJ8E\EXMSRERHXLI-RWXMXYXIJSV*MWGEP7XYHMIWLXXT[[[

MRWXMXYXIJSVKSZIVRQIRXSVKYOWMXIWHIJEYPXƤPIWTYFPMGEXMSRW4SWXVSYRHXEFPI	TETIV	*-2%0	*-2%0THJ

CROSS-CUTTING ISSUES 3.4 

The Promise of Evidence-Based Policymaking 2057
NEXT STEPS

8LVSYKLXLMWEWWIWWQIRX[ILEZIIWXEFPMWLIHETVSGIWWJSVGSPPEXMRKTSPMGMIWWIPIGXMRKEWEQTPIERHGSRHYGXMRK

XLIɄVIZMI[;ILEZIEPWSFIIREFPIXSMHIRXMJ]QSVIWTIGMƤGEPP]XLIJIEXYVIWSJXVERWTEVIRG]EFSYXXLIYWISJIZMHIRGI

MRɄTSPMG]QEOMRK;ILSTIXLEXF]HMWGYWWMRKXLITPERW[MXLHITEVXQIRXWSZIVXLITEWXWM\QSRXLWERHGSRXMRYMRK

XSɄHMWGYWWXLIJIEXYVIWERHƤRHMRKWMX[MPPFITSWWMFPIJSVXLIQXSEHHVIWWQER]SJXLIMWWYIWERHXSWIPJQSRMXSVXLIMV

WXERHEVHSJXVERWTEVIRG]MREHZERGISJRI\X]IEVŭWVEROIHEWWIWWQIRX

-R.YP][I[MPPEWWIWWNYWXSZIVE]IEVSJTSPMG]TVSTSWEPWWXEVXMRKJVSQXLIHEXIXLIRI[EHQMRMWXVEXMSR[EWJSVQIH

to the start of recess. The resulting ranked table of departmental scores will be presented under the four headings of the 

XVERWTEVIRG]JVEQI[SVO(MEKRSWMW4VSTSWEP-QTPIQIRXEXMSRERH8IWXMRKERH)ZEPYEXMSR;II\TIGXXLEXXLITVIWIRXEXMSR

SJXLMW[MPPFI

 a. A ranked table of departments showing average scores for each of the four headings of the framework.

 F (ITEVXQIRXF]HITEVXQIRXXEFPIWWLS[MRKLS[IEGLSJXLITSPMGMIWMRXLIMVWEQTPIGSRXVMFYXIH

XSɄXLIɄEZIVEKIWGSVI

 G %GSQQIRXEV]SRƤRHMRKW

There are several assessment methods issues that we want to resolve. For this we will convene a methods review group 

XSɄLIPTYWHIXIVQMRIXLIƤREPQIXLSHSPSK]-RXLIWTMVMXSJGSRWYPXEXMSRERHIRKEKIQIRXXLEXLEWMRJSVQIHXLMWTVSNIGX

XLMWɄ[MPPMRGPYHIEREP]WXWERHTSPMG]TVSJIWWMSREPWMRSYVHMWGYWWMSRW8LIMWWYIW[I[ERXXSGSRWMHIVJYVXLIVEVI

 ŵ%QSVIIJƤGMIRXTVSGIWWJSVKEXLIVMRKTSPMG]TVSTSWEPW

 • How best to cover the breadth of departmental work.

 ŵ8LIVIUYMVIHJVEQI[SVOEHETXEXMSRRSXIWJSVWGSVMRKHMJJIVIRXOMRHWSJHSGYQIRXW

 • How best to represent the departmental scores through sums and averages. 

 ŵ,S[XSWGSVIZEPYIWFEWIHTSPMGMIW

 ŵ;LIXLIV[IWLSYPHQEOIER]JYVXLIVVIZMWMSRWXSXLIJVEQI[SVOWII%TTIRHM\ERH%TTIRHM\


;IEVIEPWSOIIRXSLEZIJIIHFEGOSRXLIMWWYIW[ILEZIMHIRXMƤIHMRXLMWVITSVX

Timetable

 (IGIQFIV 1IXLSHWVIZMI[

 1EVGL (MWGYWWMSRSJƤRHMRKW[MXLTSPMG]TVSJIWWMSREPWERHEREP]WXW

 .YP] *MREPMWIPMWXSJEPPTSPMGMIW

  7EQTPIWIPIGXMSRWRSXMƤIHXSHITEVXQIRXW

 7ITXIQFIV %WWIWWQIRX

 2SZIQFIV 4YFPMGEXMSRSJVEROIHHITEVXQIRXEPXVERWTEVIRG]WGSVIW

NEXT STEPS 

2058 The Promise of Evidence-Based Policymaking
ACKNOWLEDGEMENTS

8LMWVITSVX[EWHVEJXIHF]8VEGI]&VS[R[MXLEWWMWXERGIJVSQ.MPP6YXXIV-RWXMXYXIJSV+SZIVRQIRX

ERHɄ;MPP0SVH1EXIVMEPERHEREP]WMW[IVITVITEVIHF];MPP0SVHERH7XITLERMI1EXLMWIR-X[EW

TVSHYGIHF]7IRWIEFSYX7GMIRGIMRMRTEVXRIVWLMT[MXLXLI-RWXMXYXIJSV+SZIVRQIRXERHXLI

%PPMERGIJSV9WIJYP)ZMHIRGI6IWIEVGL[EWWYTTSVXIHF]EKVERXJVSQXLI2YJƤIPH*SYRHEXMSR

7IRWIɄEFSYX7GMIRGILEWƤREPVIWTSRWMFMPMX]JSVXLIGSRXIRX

;I[SYPHPMOIXSXLERO7ɳPI0ERI.SREXLER&VIGOSRXLI%PPMERGIJSV9WIJYP)ZMHIRGI.IR+SPH

-J+ERHɄXLIQER]SVKERMWEXMSRWŪERHMRHMZMHYEPWJVSQXLIQŪ[LSKEZIJIIHFEGOSVJSVYQW

JSVɄHMWGYWWMSRSJXLITVSNIGXŭWHIZIPSTQIRXMRGPYHMRK(ITEVXQIRXEP(MVIGXSVWSJ%REP]WMW+SZIVRQIRX

7SGMEP6IWIEVGLIVW'LMIJ7GMIRXMƤG%HZMWSVWXLI(ITEVXQIRXJSVɄ-RXIVREXMSREP(IZIPSTQIRX

XLIɄ6IKYPEXSV]4SPMG]'SQQMXXII4EVPMEQIRXEV]GSQQMXXIIWTIGMEPMWXWXLI)YVSTIER'SQQMWWMSR

6IKYPEXSV]7GVYXMR]&SEVHXLI'EFMRIX3JƤGI;LEX;SVOWXIEQ)QQE+SVHSREXɄ,18VIEWYV]ERH

SXLIVW[LSKEZIIUYEPP]MRXIPPMKIRXLIPTERHMRTYXFYXGMZMPWIVZMGIVYPIWHSRŭXEPPS[YWXSREQIERHXLI

many people who tested the framework and scored policies.

8LI2YJƤIPH*SYRHEXMSRMWERIRHS[IHGLEVMXEFPIXVYWXXLEXEMQWXSMQTVSZIWSGMEP[IPPFIMRKMRXLI[MHIWXWIRWI-XJYRHW

research and innovation in education and social policy and also works to build capacity in education, science and social 

WGMIRGIVIWIEVGL8LI2YJƤIPH*SYRHEXMSRLEWJYRHIHXLMWTVSNIGXFYXXLIZMI[WI\TVIWWIHEVIXLSWISJXLIEYXLSVW

ERHɄRSXRIGIWWEVMP]XLSWISJXLI*SYRHEXMSR1SVIMRJSVQEXMSRMWEZEMPEFPIEX[[[RYJƤIPHJSYRHEXMSRSVK

ACKNOWLEDGEMENTS 

The Promise of Evidence-Based Policymaking 2059
APPENDIX 1: EVALUATING 

TRANSPARENCY OF EVIDENCE

Step 1: defining a policy

;IHIƤRIHXLMWEWEWTIGMƤGMRXIVZIRXMSREMQMRKJSV[LEXMWMRXYMXMZIJSVXLITYFPMGERHXLIYWYEP[E]XLEXTSPMGMIW

EVIɄTVIWIRXIHMRERRSYRGIQIRXWERHJSVHIGMWMSR*SVMRWXERGIGLERKIWXSXLIHVMZMRKXIWXXSVIHYGIEGGMHIRXW[SYPH

FIɄGSRWMHIVIHEWMRKPITSPMG]TVSTSWEPIEGLRI[XIWXUYIWXMSR[SYPHRSXFI

%RYQFIVSJHIGMWMSRWIEVP]SRJYVXLIVWLETIH[LMGLQIEWYVIW[IPSSOIHEXERH[LEXHIƤRIHETSPMG]%HQMRMWXVEXMZI

QIEWYVIWWYGLEWGLERKMRKXLI[E]HITEVXQIRXEPWXEXMWXMGWEVITYFPMWLIH
[IVIPEVKIP]I\GPYHIHEWRSXWYFWXERXMEP

IRSYKLXSUYEPMJ]EWTSPMGMIW-RHMZMHYEPWTIRHMRKHIGMWMSRWERHQSRI]GSQQMXXIHXSTVII\MWXMRKTVSKVEQQIW[IVIEPWS

QSWXP]I\GPYHIHEWHIVMZMRKJVSQTSPMG]ERRSYRGIQIRXWQEHISYXWMHISYVXMQIJVEQI

Step 2: finding policies

3YVMRMXMEPTPER[EWXSWIPIGXXLSWITVSTSWEPW[LMGLƤXXLIHITEVXQIRXEPTVMSVMXMIWEWWIXSYXMRXLIWMRKPIHITEVXQIRXEP

TPERW&YXEWXLI-RWXMXYXIJSV+SZIVRQIRXJSYRHMRMXWEREP]WIWSJXLIWIXLI]QSWXP]GSQTVMWIHSJYRWTIGMƤGMRHMGEXMSRW

SJɄXLIHMVIGXMSRSJEGXMSR 738LI2EXMSREP%YHMX3JƤGIEPWSMHIRXMƤIHXLEXWMRKPIHITEVXQIRXEPTPERWHMHRSXŰTVSZMHIXLI

degree of Parliamentary accountability that was promised, nor represent good practice in transparency.ű 748LIƤVWXEXXIQTX

EXKEXLIVMRKEPMWXSJTSPMGMIWGEQIJVSQHS[RPSEHMRKXLIŬERRSYRGIQIRXWŭWIGXMSRSJKSZYOŪYWMRK%YKYWXERHɄ2SZIQFIV

EWEWXEVXMRKTSMRX;IJSYRHXLMWXSFIERMQTIVJIGX[E]SJKEXLIVMRKTSPMGMIWERHSTXIHJSVEREHHMXMZIETTVSEGL

QERYEPP]KEXLIVMRKEPPTSPMGMIWTVSTSWIHF]KSZIVRQIRXFIX[IIR1E]ERH1E]

Gov.uk was the main platform for this process, but in practice it is not an effective system for the public to engage 

MRɄTSPMG]QEOMRK-XMWQSVIWYMXIHXSMRXIVEGXMSR[MXLTYFPMGWIVZMGIW8LIVIMWRSGSQTVILIRWMZIPMWXSJXLIKSZIVRQIRXŭW

TSPMGMIWJSVTISTPIXSƤRH+SZYOŭWŬTSPMGMIWŭWIGXMSRMWEGSPPIGXMSRSJTVIWWVIPIEWIWTSPMG]TETIVWWTIIGLIWKYMHERGIERH

other documents under very broad headlines.

The homepages of individual departments were of little use, since they all use the gov.uk master list for the policies section 

SJXLIMVTEKIW8LIGPSWIWXXSERI\EQTPISJKSSHTVEGXMGIMWXLI8VIEWYV]ŭWTEKI[LMGLLEWEPMROXSXLIQEMRHSGYQIRXW

[LIVIMXWTSPMGMIWEVIERRSYRGIHMRXLIXSTVMKLXGSVRIVSJXLITEKIWIIFIPS[
 75&YXIZIRLIVIXLITVSFPIQW[ILEZI

SYXPMRIHVIQEMRIH

736IWTSRWIXS7MRKPI(ITEVXQIRXEP4PERW-RWXMXYXIJSV+SZIVRQIRXXL*IFVYEV]LXXT[[[MRWXMXYXIJSVKSZIVRQIRXSVKYORI[W

latest/response-single-departmental-plans

74+SZIVRQIRXŭWQEREKIQIRXSJMXWTIVJSVQERGITVSKVIWW[MXLWMRKPIHITEVXQIRXEPTPERW2EXMSREP%YHMX3JƤGI.YP]LXXTW[[[

RESSVKYO[TGSRXIRXYTPSEHW+SZIVRQIRXWQEREKIQIRXSJMXWTIVJSVQERGITHJ

75 https://www.gov.uk/government/organisations/hm-treasury 

APPENDIX 1 

2060 The Promise of Evidence-Based Policymaking
-X[EWZIV]HMJƤGYPXXSƤRHXLIKSZIVRQIRXŭWWTIGMƤGMRXIVZIRXMSRWMRXLMWJSVQEX;IXLIVIJSVILEHXSWSYVGIXLIQJVSQ

ZEVMSYWTYFPMGHSGYQIRXWJSVXLITYVTSWIWSJXLIEHHMXMZIPMWX4PEGIW[LIVI[IGSYPHƤRHTSPMGMIWSVGPYIWEFSYXXLIMV

I\MWXIRGIMRGPYHIH

 • Consultations

 • Government bills

 • White papers 

 ŵ&YHKIXERHWTIRHMRKVIZMI[ERRSYRGIQIRXW

 • Impact assessments

 • Announcements on gov.uk and press releases

 ŵ'SRWIVZEXMZI4EVX]1ERMJIWXS

 • Single departmental plans

8LIVIEVIWSQIGEWIW[LIVIMX[EWHMJƤGYPXXSMHIRXMJ]XLIETTVSTVMEXIHSGYQIRXXSWGSVI3RIGEWIMWXLI1MRMWXV]

SJɄ.YWXMGI1S.
ŭWTSPMG]Ʉ'LERKIWXSXLI(IƤRMXMSRSJŰ7EQTPIVűJSV(2%8IWXMRKMR4VMZEXI*EQMP]0E['EWIW8LISVMKMREP

consultation was sent to accredited laboratories, since it was considered only relevant to them. It was not published online. 

8LIGSRWYPXEXMSRVIWTSRWI[LMGLHIXEMPIHXLIFEGOKVSYRHXSXLITSPMG]VIWTSRWIWXSXLIGSRWYPXEXMSRERHRI\XWXITW

[EWTYXMRXLITYFPMGHSQEMRSR1S.ŭW'SRWYPXEXMSR,YF;LMPWXMRQSWXGEWIW[IPSSOIHEXXLISVMKMREPGSRWYPXEXMSR

HSGYQIRXLIVI[ILEHXSPSSOEXXLIVIWTSRWI-RXLMWGEWIXLITVSTSWMXMSRWEVIGPIEVP]WSYVGIHERHXLIKSZIVRQIRXŭWYWI

SJXLIIZMHIRGI[EWQSWXP][IPPI\TPEMRIH

Step 3: scope

;IGLSWIXSPSSOEXHSQIWXMGHITEVXQIRXWXLI'EFMRIX3JƤGI&YWMRIWW-RRSZEXMSRERH7OMPPW'SQQYRMXMIWERH0SGEP

+SZIVRQIRX)HYGEXMSR'YPXYVI1IHMEERH7TSVX)RZMVSRQIRX*SSHERH6YVEP%JJEMVW8VERWTSVX;SVOERH4IRWMSRW

,IEPXL)RIVK]ERH'PMQEXI'LERKI,18VIEWYV]1MRMWXV]SJ.YWXMGIERHXLI,SQI3JƤGI76

8LIVI[IVIGIVXEMRSVKERMWEXMSRWERHMRWXMXYXMSRWXLEX[II\GPYHIHJVSQSYVEREP]WMW

 ŵ(YIXSXLIREXYVISJXLIMV[SVOMX[EWRIEVP]MQTSWWMFPIXSKIXEWIPIGXMSRSJŬTSPMGMIWŭEW[ILEHHIƤRIHXLIQ

JSVXLI*SVIMKR3JƤGIERHXLI1MRMWXV]SJ(IJIRGI1S(
8LIVI[IVIRI\XXSRSETTPMGEFPIGSRWYPXEXMSRWERH

MQTEGXEWWIWWQIRXWEWWSGMEXIH[MXLXLIQERHXLIVIPIZERXTSPMG]HSGYQIRXWWYGLEWɄXLIɄ2EXMSREP7IGYVMX]

7XVEXIK]ERH7XVEXIKMG(IJIRGIERH7IGYVMX]6IZMI[
[IVIXSSFVSEH8LI(ITEVXQIRXJSV-RXIVREXMSREP

(IZIPSTQIRX(J-(
[EWEPWSGSRWMHIVIHFYXI\GPYHIH-RXLIEFWIRGISJXLI&MPEXIVEPERH1YPXMPEXIVEP%MH

VIZMI[W[LMGLLEHRSXFIIRTYFPMWLIHEXXLIXMQI[I[IVIGVIEXMRKXLIQEWXIVPMWX[IGSYPHRSXƤRHER]XLMRK

XLEXUYEPMƤIHEWE(J-(TSPMG]1SWXSJMXWEGXMZMXMIW[IVITVSNIGXFEWIH;IRSXI(J-(LEWFIIRYWMRKSYVHVEJX

JVEQI[SVOWMRGIMXWTYFPMGEXMSRMR3GXSFIVXSVIƥIGXSRXLIMRJSVQEXMSRGSRXEMRIHMRɄMXWɄEMHɄVIZMI[W

 ŵ*SVXLMWƤVWXWXEKI[I[IVIPSSOMRKEXTSPMGMIWJVSQ;LMXILEPPHITEVXQIRXW;IHMHRSXKEXLIVTSPMGMIWJVSQ

XLIɄ;IPWL7GSXXMWLSV2SVXLIVR-VMWLKSZIVRQIRXW;IEVIOIIRXSI\TERHXLIYWISJXLIJVEQI[SVOERHɄXLI

IZMHIRGIXVERWTEVIRG]EKIRHEMRXLIJYXYVI;ILEZITVIWIRXIHMXEXWIZIVEPQIIXMRKWERHEPWSEXɄEɄQIIXMRKSJ

XLIRI[6IKYPEXSV]7GVYXMR]&SEVHSJXLI)YVSTIER'SQQMWWMSR[LMGLTPERWXSEHSTXEWTIGXWSJMX

 ŵ;IVIQSZIHTSPMGMIWMQTPIQIRXIHF]I\IGYXMZIEKIRGMIWWYGLEWXLI)RZMVSRQIRX%KIRG]8LMW[EWHYIXS

both time constraints and the fact these bodies are mostly implementing policies set by central departments.

768LIXMQIJVEQI[IPSSOIHEXGEQIFIJSVIXLIGLERKIWXSXLIQEGLMRIV]SJKSZIVRQIRXWYGLEWXLIQIVKIVSJ(IGGERH&-7MRXSXLI

(ITEVXQIRXSJ&YWMRIWW)RIVK]ERH-RHYWXVMEP7XVEXIK]
QEHIMR.YP]

APPENDIX 1 

The Promise of Evidence-Based Policymaking 2061
8LIWXEKI[IPSSOIHEX[EW[LIRETSPMG][EWƤVWXTVSTSWIHF]XLIKSZIVRQIRX8LIHSGYQIRXW[IVIXLIƤVWXWYFWXERXMZI

SRIWXSSYXPMRIXLIKSZIVRQIRXŭWTVSTSWEPW8LISRP]GEWIW[LIR[IPSSOIHEXEƤVWXERRSYRGIQIRXWYGLEWEFYHKIX

announcement or press release) was when the government had moved straight from that to implementation. In some 

GEWIWEFYHKIXSVWTIRHMRKVIZMI[[SYPHGSQQMXEHITEVXQIRXXSLSPHMRKEGSRWYPXEXMSRSVVIZMI[*SVI\EQTPIXLI

7TIRHMRK6IZMI[ERH%YXYQR7XEXIQIRXQEHIXLMWGSQQMXQIRXVIKEVHMRKXLI2I[,SQIW&SRYW 77

-RGEWIWPMOIXLMW[IWGSVIHXLIGSRWYPXEXMSRXLEX[EWQIRXMSRIH;II\GPYHIHGEPPWJSVIZMHIRGIJVSQSYVEWWIWWQIRX

;IHMWGSZIVIHEWQEPPRYQFIVSJTSPMGMIWJSPPS[MRKEWWIWWQIRXXLEXWLSYPHLEZIFIIRI\GPYHIH8LIWIHMHRSXTVSZMHI

EɄTVSTSWEPSVTVIJIVVIHSTXMSRJVSQXLIKSZIVRQIRX

Pre-policies

 There were a few cases where proposals looked like policies but were not. Their introductions may have presented 

EHIZIPSTIHGSRWMHIVEXMSRSJXLIMWWYIWSVWYFNIGXFYXXLIMVTVMQEV]TYVTSWI[EWXSQEVWLEPMRJSVQEXMSRMR

SRITPEGISVƤRHSYX[LEX[EWKSMRKSR8LIWI[IVIXLI(ITEVXQIRXJSV&YWMRIWW-RRSZEXMSRERH7OMPPW&-7
ŭ

'VIEXMRK9/6IWIEVGLERH-RRSZEXMSR[LMGLERRSYRGIHXLEXXLIKSZIVRQIRXMWGSQQMXXMRKXSXLIVIWYPXW

SJɄEɄVIZMI[XLIɄ(ITEVXQIRXJSV)HYGEXMSR(J)
ŭW6IKMSREP%HSTXMSR%KIRGMIW4VSKVEQQIXLIɄ'EFMRIX3JƤGIŭW

%RXM'SVVYTXMSR-RRSZEXMSR,YFXLI(ITEVXQIRXJSV'SQQYRMXMIWERH0SGEP+SZIVRQIRX('0+
ŭW&ERRMRK

ERHɄ&PEGOPMWXMRK6SKYI0ERHPSVHWERHXLI(ITEVXQIRXJSV'YPXYVI1IHMEERH7TSVX('17
ŭ&VSEHFERH9RMZIVWEP

Service Obligation.

Carrying out (court) orders

 -R.YP]XLI1S.MWWYIHERMQTEGXEWWIWWQIRXERHI\TPEREXSV]QIQSVERHYQXS%QIRHXLI'MZMP0IKEP

%MH1IVMXW'VMXIVME8LMWGLERKI[EWXSGSQTP][MXLE,MKL'SYVXVYPMRKXSEZSMHEVMWOSJFIMRKYRPE[JYP[LMPI

XLIKSZIVRQIRXETTIEPIHMX-X[EWRSXETVIJIVVIHWSPYXMSRXSERMHIRXMƤIHTVSFPIQ8LITSPMG]WGSVIH[IPP

for transparency – it was very clear what the government was proposing to do and why. Future assessments 

[MPPMHIRXMJ]WMXYEXMSRWPMOIXLMWERHXIWXXLIQWMQTP]JSVEGPIEVI\TPEREXMSRVEXLIVXLEREKEMRWXXLIɄJYPP

XVERWTEVIRG]ɄJVEQI[SVO

777TIRHMRKVIZMI[ERHEYXYQRWXEXIQIRXYTHEXIH2SZIQFIV7IGXMSR)RKPMWLHIZSPYXMSR0SGEP

KSZIVRQIRXVIJSVQLXXTW[[[KSZYOKSZIVRQIRXTYFPMGEXMSRWWTIRHMRKVIZMI[ERHEYXYQRWXEXIQIRX

HSGYQIRXWWTIRHMRKVIZMI[ERHEYXYQRWXEXIQIRX

APPENDIX 1 

2062 The Promise of Evidence-Based Policymaking
Step 4: assessment

;IIRHIHYT[MXLEPMWXSJMRXIVZIRXMSRWJSVXLI]IEVXS1E]8LIVIEVIQIEWYVIW[IMRGPYHIH[LMGLSRVIƥIGXMSR

QE]RSXUYEPMJ]EWTSPMGMIWERHWSQI[LMGLQE]SZIVPET&YXMXWXMPPVITVIWIRXWXLIZEWXQENSVMX]SJ[LEXXLIKSZIVRQIRX

introduced in our time frame and gives a strong indication of which departments produce the most policy. The number 

SJɄTSPMGMIWJSVIEGLHITEVXQIRX[EWMRƥYIRGIHF]XLIEFWIRGISJQENSVTSPMG]HSGYQIRXWXLEXLEHFIIRERRSYRGIHFYX

[IVI]IXXSGSQISYX*SVMRWXERGIXLI(ITEVXQIRXJSV)RZMVSRQIRX*SSHERH6YVEP%JJEMVW(IJVE
ŭWƥEKWLMT]IEVTPER

JSVXLIIRZMVSRQIRXERHXLI0MJI'LERGIW7XVEXIK][IVI]IXXSFITYFPMWLIH

-R1E][ILIPHE[SVOWLSTEXXLI-RWXMXYXIJSV+SZIVRQIRX[MXLZSPYRXIIVWJVSQEZEVMIX]SJTSPMG]FEGOKVSYRHW

from the civil service fast stream, to academia, to charities. The participants were split into groups and given a policy 

document. They applied the framework to the document and scored it. As a result of these discussions, we made two 

RSXEFPIGLERKIWXSXLIJVEQI[SVOXLIVIZMWIHZIVWMSRGERFIJSYRHMR%TTIRHM\


 ŵ8LI:EPYIJSV1SRI]WIGXMSR[EWQIVKIH[MXL-QTPIQIRXEXMSRERH4VSTSWEP

 ŵ%UYIWXMSREFSYXJYVXLIVTPERWJSVGSRWYPXEXMSR[EWEHHIHXSEGGSYRX

JSVXLIJEGX[IEVIPSSOMRKEXIEVP]WXEKIHSGYQIRXW

There was a lot of discussion about what should be done when a department says that there is no evidence.

We began the process intending to produce a full ranking of departments. The changes to the machinery of government 

MR.YP]GSQTPMGEXIHXLMWXEWO;IEPWS[ERXIHXSXEOIXMQIXSIRKEKI[MXLHITEVXQIRXWJYVXLIVSRWSQI

SJɄXLIɄQIXLSHSPSKMGEPUYIWXMSRWVEMWIHSRTVSHYGMRKEJYPPVEROMRK;IXLIVIJSVIXSSOXLIHIGMWMSRXSTVSHYGIXLMWKSSH

ERHFEHTVEGXMGIGSQQIRXEV]XLMW]IEVERHEJYPPVEROMRKMR7ITXIQFIV

*VSQSYVPMWXSJTSPMGMIW[IWIPIGXIHIMKLXHMWGVIXIMRXIVZIRXMSRWTIVHITEVXQIRXFEWIHSRXLIKSZIVRQIRXŭWS[R

documents about its priorities and ranging, as far as possible, across their policy portfolios. This often meant policies that 

[IVIQERMJIWXSGSQQMXQIRXWSV[IVIQIRXMSRIHMRXLIHITEVXQIRXŭW7MRKPI(ITEVXQIRXEP4PER*SVI\EQTPI[IWIPIGXIH

TSPMGMIWJSV(J)VERKMRKEGVSWWXLIWYFNIGXEVIEWSJWGLSSPWWSGMEP[SVOERHXLIJYRGXMSRWSJXLI+SZIVRQIRX)UYEPMXMIW

3JƤGI;IEPWSXSSOTYFPMGMRXIVIWXMRXSEGGSYRX[LIRHIGMHMRKFIX[IIREREVVS[EHQMRMWXVEXMZIQIEWYVISVSRIXLEX

[SYPHLEZIQSVITYFPMGWMKRMƤGERGI
ERHWSYKLXXSIRWYVIEWMQMPEVTVSƤPISJTSPMG]HSGYQIRXWJSVIEGL

Following feedback from departments on the range of policy areas and the documents we had selected, we narrowed our 

PMWXHS[RXSWM\TSPMGMIW-RERYQFIVSJGEWIW[IGLERKIHXLIHSGYQIRXW[IPSSOIHEXERHEQIRHIHXLIWEQTPIFEWIH

SRɄSYVGSVVIWTSRHIRGI;IFIRIƤXIHJVSQIRKEKIQIRX[MXLIZIV]HITEVXQIRXMRZSPZIHMRXLMWI\IVGMWIXSWSQIHIKVII

We spent some time clarifying that our focus is on evidence transparency as opposed to the quality of evidence or the 

merits of the policy.

%KVSYTSJWGSVIVWYWIHXLIQSHMƤIHJVEQI[SVOXSWGSVIERHGSQQIRXSRXLMWWEQTPI(YVMRKXLIWGSVMRKTVSGIWW

[IɄVIGSKRMWIHXLEXWSQIWIGXMSRWSJXLIJVEQI[SVOTEVXMGYPEVP]8IWXMRKERH)ZEPYEXMSRQE]RSXFIVIPIZERXJSVZIV]

IEVP]WXEKIHSGYQIRXW*SVXLMWVIEWSR[IKEZIWGSVIVWXLISTXMSRSJE[EVHMRKXLIWIWIGXMSRWEŰ2SX%TTPMGEFPIűVEXMRK

ERHɄWYFWIUYIRXP]EVIHIZIPSTMRKXLIUYIWXMSRWEFSYXYWISJGSRWYPXEXMSRQEXIVMEPERHXVERWTEVIRG]EFSYXRI\XWXITW

8LIMRWMKLXWERHWGSVIWXLEX[IGSPPIGXIHJVSQXLMWTVSGIWWWYFNIGXXSVIZMI[JSVQIHXLIFEWMWSJXLMWVITSVX

ERHɄXLIɄGSQTIRHMYQSJKSSHTVEGXMGI

APPENDIX 1 42

The Promise of Evidence-Based Policymaking 2063
LEVEL: 0 1 2 3 Worked Example

Diagnosis

This concerns why something is proposed, 

ie what the issue is that will be addressed.

8LIHSGYQIRXWLSYPHI\TPEMR

• what policymakers know about the issue, 

MXWɄGEYWIWIJJIGXWERHWGEPI

• how policymakers have assessed the strengths 

and weaknesses of that evidence.

So, can you see what 

evidence has been 

used and the role 

MXɄLEWɄTPE]IH#

2SXWYJƤGMIRXP]

JSVPIZIPɄ

)ZMHIRGIMW

mentioned, 

with some 

I\TPEREXMSR

SJɄLS[MXLEW

been used.

As in level 

ERHXLI

supporting 

evidence 

is mostly 

linked to the 

relevant parts 

of the policy, 

properly cited 

ERHƤRHEFPI

and there is 

discussion of 

how it has been 

used.

Supporting 

evidence is 

consistently 

linked to the 

relevant parts 

of the policy, 

properly cited 

ERHƤRHEFPI

and there is 

assessment of 

uncertainties and 

contradictions 

in the evidence 

base.

The government has assessed 

XLII\XIRXSJTVSFPIQHVMROMRK

MRɄXLI9/XLIIGSRSQMG

ERHɄLYQERGSWX

LEVEL: 0 1 2 3 Worked Example

Proposal

;LEXMWXLIKSZIVRQIRXŭWGLSWIRMRXIVZIRXMSR#

8LIHSGYQIRXWLSYPHI\TPEMR

• why the government has chosen this intervention

• what evidence, if any, that choice is based on

• how policymakers have assessed the strengths 

and weaknesses of the evidence base, including 

what has been tried before and whether that 

worked or not

• whether there are other options and why they have 

not been chosen

• what the government plans to do about any 

part of the intervention that has not yet been 

HIGMHIHɄYTSR

ŵ[LEXXLIGSWXWERHFIRIƤXWEVIIWXMQEXIHXSFI

and the assumptions behind those calculations.

So, can you see what 

evidence has been 

used and the role 

MXɄLEWɄTPE]IH#

2SXWYJƤGMIRXP]

JSVPIZIPɄ

)ZMHIRGIMW

mentioned, 

with some 

I\TPEREXMSR

SJɄLS[MXLEW

been used.

As in level 

ERHXLI

supporting 

evidence 

is mostly 

linked to the 

relevant parts 

of the policy, 

properly cited 

ERHƤRHEFPI

and there is 

discussion of 

how it has been 

used.

Supporting 

evidence is 

consistently 

linked to the 

relevant parts 

of the policy, 

properly cited 

ERHƤRHEFPI

and there is 

assessment of 

uncertainties and 

contradictions 

in the evidence 

base.

The government has chosen 

to implement minimum unit 

pricing for alcohol, instead 

SJJSVɄI\EQTPIMRGVIEWMRK

EPGSLSPXE\IWSVWXEVXMRKERI[

educational campaign.

APPENDIX 2: 

THE REVISED TRANSPARENCY FRAMEWORK, JULY 2016 

APPENDIX 2: THE REVISED TRANSPARENCY FRAMEWORK, JULY 2016 

2064 The Promise of Evidence-Based Policymaking
LEVEL: 0 1 2 3 Worked Example

Implementation

How will the chosen intervention be introduced and 

VYR#

8LIHSGYQIRXWLSYPHI\TPEMR

• why this method for delivering the intervention 

LEWɄFIIRGLSWIR

• what evidence, if any, that decision is based on

• whether there are other methods and if so the 

reasons for not choosing them

• if the way to deliver the intervention is still being 

decided, what the method is for deciding

ŵ[LEXXLIGSWXWERHFIRIƤXWEVIIWXMQEXIHXSFI

ERHɄXLIEWWYQTXMSRWFILMRHXLSWIGEPGYPEXMSRW

 

So, can you see what 

evidence has been 

used and the role 

MXɄLEWɄTPE]IH#

2SXWYJƤGMIRXP]

JSVPIZIPɄ

)ZMHIRGIMW

mentioned, 

with some 

I\TPEREXMSR

SJɄLS[MXLEW

been used.

As in level 

ERHXLI

supporting 

evidence 

is mostly 

linked to the 

relevant parts 

of the policy, 

properly cited 

ERHƤRHEFPI

and there is 

discussion of 

how it has been 

used.

Supporting 

evidence is 

consistently 

linked to the 

relevant parts 

of the policy, 

properly cited 

ERHƤRHEFPI

and there is 

assessment of 

uncertainties and 

contradictions 

in the evidence 

base.

The government has decided 

XSɄMQTPIQIRXQMRMQYQYRMX

pricing through a voluntary 

EKVIIQIRX[MXLQENSVVIXEMPIVW

rather than through legislation.

LEVEL: 0 1 2 3 Worked Example

Testing and evaluation

,S[[MPP[IORS[MJXLITSPMG]LEW[SVOIH#

8LIHSGYQIRXWLSYPHI\TPEMR

• any testing that has been or will be done

• plans to measure the impact of the policy 

ERHɄXLIɄSYXGSQIWXLEX[MPPFIQIEWYVIH

• plans to evaluate the effects of the policy, 

MRGPYHMRKɄEXMQIXEFPI

• plans for using further inputs

So, can you see what 

evidence has been 

used and the role 

MXɄLEWɄTPE]IH#

2SXWYJƤGMIRXP]

JSVPIZIPɄ

Some indication 

of success 

measures 

but no plans 

for testing/

evaluation 

SVɄI\TPEREXMSR

of why 

inappropriate).

1SVI

comprehensive 

success 

measures 

SVɄTVSGIWW

for developing 

them outlined). 

Also provides 

details about 

use of testing 

and plans for 

evaluation 

SVI\TPEMRW

why testing 

or evaluation 

would not be 

appropriate.

As in level 2 but 

I\TPEMRWXLI

reasons for the 

use of testing 

and plans for 

evaluation. It is 

also clear what 

will happen to the 

results of testing 

and evaluation, 

including timing 

and plans for 

publication.

The government sets out how 

it plans to measure the results 

of the policy. The government 

sets out plans for piloting, initial 

evaluation of those results and 

timetable for publication and 

then describes decision process 

EVSYRHVSPPSYXMJXLIIZEPYEXMSR

MWɄWEXMWJEGXSV]

APPENDIX 2: THE REVISED TRANSPARENCY FRAMEWORK, JULY 2016 44

The Promise of Evidence-Based Policymaking 2065
LEVEL: 0 1 2 3 Worked Example

Diagnosis

This concerns why something is proposed, 

ie what the issue is that will be addressed.

The document should explain:

• what policymakers know about the issue, 

MXWɄGEYWIWIJJIGXWERHWGEPI

• how policymakers have assessed the strengths 

ERH[IEORIWWIWSJXLEXIZMHIRGI

So, can you see what

evidence has been

used and the role

MXɄLEWɄTPE]IH#

Not clearly

IRSYKLJSV

PIZIPɄ

Evidence is

mentioned,

with some

explanation

SJɄLS[MXLEW

been used.

%WMRPIZIPFYX

the supporting

evidence is

linked to the

relevant parts

SJXLITSPMG]

properly cited

and you could 

ƤRHXLIWSYVGI

As in level 2 but

the evidence

base is also

assessed and

uncertainties and

contradictory

MRJSVQEXMSREVI

acknowledged.

The government has assessed

XLII\XIRXSJTVSFPIQHVMROMRK

MRɄXLI9/XLIIGSRSQMG

ERH LYQERGSWX

LEVEL: 0 1 2 3 Worked Example

Proposal

What is the government’s chosen intervention?

The document should explain:

• why the government has chosen this intervention

• what evidence, if any, that choice is based on

• how policymakers have assessed the strengths 

and weaknesses of the evidence base, including

what has been tried before and whether that 

worked or not

• whether there are other options and why they have

not been chosen

• what the government plans to do about any

part of the intervention that has not yet been 

HIGMHIHɄYTSR

So, can you see what

evidence has been

used and the role

MXɄLEWɄTPE]IH#

Not clearly

enough for 

PIZIPɄ

Evidence is

mentioned,

with some

explanation

SJɄLS[MXLEW

been used.

%WMRPIZIPFYX

the supporting

evidence 

is linked to

relevant parts

of the policy,

properly cited

and you could 

ƤRHXLIWSYVGI

As in level 2 but

the evidence

base is also

assessed and

uncertainties and

contradictory

information are

acknowledged. 

The government has chosen

to implement minimum unit 

pricing for alcohol, instead

SJJSVɄI\EQTPIMRGVIEWMRK

alcohol taxes or starting a new

educational campaign. 

APPENDIX 3:

THE ORIGINAL FRAMEWORK

APPENDIX 3: THE ORIGINAL FRAMEWORK 

2066 The Promise of Evidence-Based Policymaking
LEVEL: 0 1 2 3 Worked Example

Implementation

How will the chosen intervention be rolled out?

The document should explain:

• why this method for delivering the intervention

LEWɄFIIRGLSWIR

• what evidence, if any, that decision is based on

• whether there are other methods and if so the 

reasons for not choosing them

• if the way to deliver the intervention is still being

So, can you see what

evidence has been

used and the role

MXɄLEWɄTPE]IH#

Not clearly

enough for

PIZIPɄ

Evidence is

mentioned,

with some

explanation

SJɄLS[MXLEW

been used.

%WMRPIZIPFYX

the evidence

is linked to

relevant parts

of the policy,

properly cited

and you could 

ƤRHXLIWSYVGI

As in level 2 but

the evidence

base is also

assessed and

uncertainties and

contradictory

information are

acknowledged.

The government has decided

XSɄMQTPIQIRXQMRMQYQYRMX

pricing through a voluntary

agreement with major retailers

rather than through legislation.

LEVEL: 0 1 2 3 Worked Example

Value for money

8LMWGSRWMHIVWXLIGSWXWERHFIRIƤXWSJXLITSPMG]

XSɄWLS[[L]XLIKSZIVRQIRXXLMROWMXMW[SVXLHSMRK

8LIHSGYQIRXWLSYPHI\TPEMR

ŵ [LEXXLIGSWXWERHFIRIƤXWEVIIWXMQEXIHXSFI

ŵ XLIEWWYQTXMSRWFILMRHXLSWIGEPGYPEXMSRW

ŵ [LEXIZMHIRGIMWFIMRKYWIHXSQEOI

XLSWIɄEWWYQTXMSRW

ŵ XLIYRGIVXEMRXMIWEFSYXXLIGSWXWERHFIRIƤXW

ERHɄLS[PMOIP]XLIƤKYVIWEVIXSGLERKI

So, can you see what

evidence has been

used and the role

MXɄLEWɄTPE]IH#

2SXGPIEVP]

IRSYKLJSV

PIZIPɄ

The

EWWYQTXMSRW

(the basis)

JSVɄGSRGPYWMSRW

EFSYXVMWOW

ERHɄFIRIƤXW

EVIɄHIWGVMFIH

%WMRPIZIP

FYXWYTTSVXMRK

IZMHIRGIMWEPWS

TVSTIVP]GMXIH

ERH]SYGSYPH

ƤRHXLIWSYVGI

%WMRPIZIP

FYXMXɄMWEPWS

GPIEVLS[XLI

uncertainties

in these 

EWWYQTXMSRW

LEZIFIIR

GSRWMHIVIH

8LIEWWIWWQIRXWLS[W

XLIɄTSXIRXMEPSJXLITVSTSWEP

XSɄVIHYGITVSFPIQHVMROMRK

FYXɄEPWSXLIMQTEGXWSRFYWMRIWW

IKWYTIVQEVOIXWTYFW


XLIɄTYFPMGWIGXSVIKTSPMGI2,7


ERHXSXLITYFPMGSJɄVEMWMRKɄTVMGIW

APPENDIX 3: THE ORIGINAL FRAMEWORK 

The Promise of Evidence-Based Policymaking 2067
LEVEL: 0 1 2 3 Worked Example

Testing and evaluation

How we will know if the policy has worked?

The document should explain:

• plans to measure the impact of the policy

and the outcomes that will be measured 

ŵ TPERWXSXIWXXLITSPMG] ƤVWXSVVIEWSRW[L]RSX

• plans to evaluate the effects of the policy including

a timetable

So, can you see what

evidence has been

used and the role

MXɄLEWɄTPE]IH#

Not clearly

enough for 

PIZIPɄ

Some indication

of success

measures

but no plans

for testing/

evaluation

SVɄI\TPEREXMSR

of why

inappropriate)

More

comprehensive

success

measures

SVɄTVSGIWW

for developing

them outlined).

Also provides 

details about 

use of testing

and plans for

evaluation

or explains

why testing 

or evaluation

would not be 

appropriate.

As in level 2 but

explains the 

reasons for the

use of testing

and plans for

evaluation. It is

also clear what

will happen to the

results of testing

ERHIZEPYEXMSR

including timing

and plans for

publication.

The government sets out how

it plans to measure the results

of the policy. The government

WIXWSYXTPERWJSVTMPSXMRKMRMXMEP

evaluation of those results

ERHɄXMQIXEFPIJSVTYFPMGEXMSR

ERHɄXLIRHIWGVMFIWHIGMWMSR

process around roll-out if the 

IZEPYEXMSRMWɄWEXMWJEGXSV]

APPENDIX 3: THE ORIGINAL FRAMEWORK 

2068 The Promise of Evidence-Based Policymaking
APPENDIX 4: POLICY 

DOCUMENTS REVIEWED

Cabinet Office

Anti-Corruption Innovation Hub.

9/3TIR+SZIVRQIRX2EXMSREP%GXMSR4PER?TSPMG]

TETIVAXL1E]
LXXTW[[[KSZYOKSZIVRQIRX

TYFPMGEXMSRWYOSTIRKSZIVRQIRXREXMSREPEGXMSR

TPERYOSTIRKSZIVRQIRXREXMSREPEGXMSR

TPERPEWXEGGIWWIHXL2SZIQFIV


Anti-Lobbying Clause in Government Grant Agreements.

+SZIVRQIRXERRSYRGIWRI[GPEYWIXSFIMRWIVXIHMRXS

KVERXEKVIIQIRXW?TVIWWVIPIEWIAXL*IFVYEV]


LXXTW[[[KSZYOKSZIVRQIRXRI[WKSZIVRQIRX

ERRSYRGIWRI[GPEYWIXSFIMRWIVXIHMRXSKVERX

EKVIIQIRXWPEWXEGGIWWIHXL2SZIQFIV


English Language Requirements 

JSV4YFPMG7IGXSVɄ;SVOIVW

'SRWYPXEXMSRVITSVXSRHVEJXPERKYEKIVIUYMVIQIRXWJSV

TYFPMGWIGXSV[SVOIVW'SHISJ4VEGXMGI?GSRWYPXEXMSR

VITSVXA*IFVYEV]
LXXTW[[[KSZYOKSZIVRQIRX

YTPSEHWW]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI

'SRWYPXEXMSRCVITSVXCSRCHVEJXCPERKYEKICVIUYMVIQIRXWC

JSVCTYFPMGCWIGXSVC[SVOIVWC'SHICSJC4VEGXMGITHJ

)RKPMWL0ERKYEKI6IUYMVIQIRXJSV4YFPMG7IGXSV;SVOIVW

?MQTEGXEWWIWWQIRXAXL%YKYWX
LXXTW[[[KSZ

uk/government/uploads/system/uploads/attachment_

HEXEƤPI-QTEGXC%WWIWWQIRXCC)RKPMWLC*PYIRG]C

MRC4YFPMGC7IGXSVC;SVOIVWCCCJSVCTYFPMGEMXSRTHJ

(VEJX'SHISJTVEGXMGISRXLI)RKPMWLPERKYEKIVIUYMVIQIRX

JSVTYFPMGWIGXSV[SVOIVW4EVXSJXLI-QQMKVEXMSR%GX

?A:IVWMSR3GXSFIV
LXXTW[[[KSZYO

government/uploads/system/uploads/attachment_data/

ƤPI(VEJXC'SHICSJC4VEGXMGICSRCXLIC)RKPMWLC

0ERKYEKIC6IUYMVIQIRXCJSVC4YFPMGC7IGXSVC;SVOIVWCTHJ

EPPPEWXEGGIWWIHXL2SZIQFIV


8LMWETTIRHM\GSRXEMRWEPPXLITSPMG]HSGYQIRXW[IVIZMI[IHPMWXIHF]HITEVXQIRX

(SGYQIRXW[IVITYFPMWLIHF]XLIGSVVIWTSRHMRKHITEVXQIRXYRPIWWSXLIV[MWIWXEXIH

Establishing Common Measures 

SJ7SGMS)GSRSQMG&EGOKVSYRH

)RKEKIQIRX(SGYQIRX(IZIPSTMRKE'SQQSRWIX

SJ1IEWYVIWJSV)QTPS]IVWSRXLI7SGMS)GSRSQMG

Backgrounds of their Workforce and Applicants1E]


LXXTW[[[KSZYOKSZIVRQIRXYTPSEHWW]WXIQ

YTPSEHWEXXEGLQIRXCHEXEƤPICCGSC

engagement_document_employer_measures_on_socio_

IGSRSQMGCFEGOKVSYRHTHJPEWXEGGIWWIHXL2SZIQFIV




-RXVSHYGXMSRSJ2I[4S[IVWJSV&SHMIWXS(MWGPSWI

-HIRXMƤIH(EXEJSVXLI4YVTSWISJ-QTVSZMRK4YFPMG

Service Delivery.

&IXXIV9WISJ(EXEMR+SZIVRQIRX'SRWYPXEXMSR4ETIV

*IFVYEV]
LXXTW[[[KSZYOKSZIVRQIRX

uploads/system/uploads/attachment_data/

ƤPIC(EXEC0IKMWPEXMSRC4VSTSWEPWCC'SRC

(SGCCƤREPCCCTHJ

-RXVSHYGXMSRSJRI[TS[IVWJSVFSHMIWXSHMWGPSWIMHIRXMƤIH

HEXEJSVXLITYVTSWISJMQTVSZMRKTYFPMGWIVZMGIHIPMZIV]

?MQTEGXEWWIWWQIRXALXXTW[[[KSZYOKSZIVRQIRX

YTPSEHWW]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI

Consultation_Stage_Impact_Assessment_for_Public_

7IVZMGIC(IPMZIV]CTS[IVTHJFSXLPEWXEGGIWWIHXL

2SZIQFIV


2I[4S[IVWJSV&SHMIWXS(MWGPSWI-HIRXMƤIH(EXEJSV

XLI4YVTSWISJ8EOMRK%GXMSRMR'SRRIGXMSR[MXL(IFX

3[IHXSE7TIGMƤIH4YFPMG%YXLSVMX]

&IXXIV9WISJ(EXEMR+SZIVRQIRX'SRWYPXEXMSR4ETIV

*IFVYEV]
LXXTW[[[KSZYOKSZIVRQIRX

uploads/system/uploads/attachment_data/

ƤPIC(EXEC0IKMWPEXMSRC4VSTSWEPWCC'SRC

(SGCCƤREPCCCTHJ

-RXVSHYGXMSRSJRI[TS[IVWJSVFSHMIWXSHMWGPSWIMHIRXMƤIH

data for the purpose of taking action in connection with debt 

S[IHXSEWTIGMƤIHTYFPMGEYXLSVMX]?MQTEGXEWWIWWQIRXA

WX*IFVYEV]
LXXTW[[[KSZYOKSZIVRQIRX

YTPSEHWW]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI

'SRWYPXEXMSRC7XEKIC-QTEGXC%WWIWWQIRXCJSVC(IFXC

TS[IVTHJFSXLPEWXEGGIWWIHXL2SZIQFIV


APPENDIX 4 

The Promise of Evidence-Based Policymaking 2069
Department for Business, 

Innovation and Skills 

Apprenticeships Levy.

%TTVIRXMGIWLMTW0IZ])QTPS]IVS[RIHETTVIRXMGIWLMTW

training?GSRWYPXEXMSRA%YKYWX
LXXTW[[[KSZYO

government/uploads/system/uploads/attachment_data/

ƤPIFMWETTVIRXMGIWLMTWPIZ]GSRWYPXEXMSR

THJPEWXEGGIWWIHXL2SZIQFIV


&EPPSX8LVIWLSPHW?JSVWXVMOIEGXMSRA

in Important Public Services.

8VEHI9RMSR6IJSVQ'SRWYPXEXMSRSRFEPPSXXLVIWLSPHWMR

MQTSVXERXTYFPMGWIVZMGIW.YP]
LXXTW[[[KSZYO

government/uploads/system/uploads/attachment_data/

ƤPI&-7GSRWYPXEXMSRSRFEPPSXXLVIWLSPHW

MRMQTSVXERXTYFPMGWIVZMGIWTHJ

&EPPSX8LVIWLSPHWMR-QTSVXERX4YFPMG7IVZMGIW

'SRWYPXEXMSR-QTEGX%WWIWWQIRX.YP]
LXXTW

www.gov.uk/government/uploads/system/uploads/

EXXEGLQIRXCHEXEƤPI&-7-%C

XLVIWLSPHTHJFSXLPEWXEGGIWWIHXL2SZIQFIV


'VIEXMRK9/6IWIEVGLERH-RRSZEXMSR

7YGGIWWEWE/RS[PIHKI)GSRSQ]8IEGLMRK)\GIPPIRGI

Social Mobility and Student Choice?[LMXITETIVA1E]


LXXTW[[[KSZYOKSZIVRQIRXYTPSEHW

W]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPIFMW

WYGGIWWEWEORS[PIHKIIGSRSQ][IFTHJPEWX

EGGIWWIHXL2SZIQFIV


1SZMRKXLI3TIVEXMSRWSJXLI0ERH6IKMWXV]

into the Private Sector.

'SRWYPXEXMSRSRQSZMRK0ERH6IKMWXV]STIVEXMSRWXSXLI

private sectorXL1EVGL
LXXTW[[[KSZYO

government/uploads/system/uploads/attachment_data/

ƤPI&-7GSRWYPXEXMSRSRQSZMRKPERH

VIKMWXV]STIVEXMSRWXSXLITVMZEXIWIGXSVTHJPEWX

EGGIWWIHXL2SZIQFIV


2EXMSREP0MZMRK;EKI

7YQQIV&YHKIX,18VIEWYV] .YP]
LXXTW

www.gov.uk/government/uploads/system/uploads/

EXXEGLQIRXCHEXEƤPIC7YQQIVC

&YHKIXCC;IFC%GGIWWMFPITHJ

7YQQIV&YHKIXHEXEWSYVGIW,18VIEWYV].YP]


LXXTW[[[KSZYOKSZIVRQIRXYTPSEHWW]WXIQ

YTPSEHWEXXEGLQIRXCHEXEƤPI(EXECWSYVGIWCC

7YQQIVC&YHKIXCTHJFSXLPEWXEGGIWWIHXL

2SZIQFIV


Small Business Commissioner.

)RXIVTVMWI&MPP%7QEPP&YWMRIWW'SQQMWWMSRIV?GSRWYPXEXMSRA

XL.YP]
LXXTW[[[KSZYOKSZIVRQIRXYTPSEHW

W]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI&-7

EWQEPPFYWMRIWWGSQQMWWMSRIVTHJPEWXEGGIWWIHXL

2SZIQFIV


Department for Communities and Local 

Government (DCLG)

&ERRMRKERH&PEGOPMWXMRK6SKYI0ERHPSVHW

8EGOPMRKVSKYIPERHPSVHWERHMQTVSZMRKXLITVMZEXIVIRXEP

WIGXSV%XIGLRMGEPHMWGYWWMSRTETIV?GSRWYPXEXMSRA%YKYWX


LXXTW[[[KSZYOKSZIVRQIRXYTPSEHWW]WXIQ

YTPSEHWEXXEGLQIRXCHEXEƤPI(MWGYWWMSRCTETIVC

*-2%0THJPEWXEGGIWWIHXL2SZIQFIV


&VSEHIRMRKXLI(IƤRMXMSRSJ%JJSVHEFPI,SYWMRK

Consultation on proposed changes to national 

planning policy(IGIQFIV
LXXTW[[[KSZYO

government/uploads/system/uploads/attachment_data/

ƤPIC'SRWYPXEXMSRCHSGYQIRXTHJ

Consultation on proposed changes to national planning 

TSPMG])UYEPMXMIW7XEXIQIRX(IGIQFIV
LXXTW[[[

gov.uk/government/uploads/system/uploads/attachment_

HEXEƤPI)UYEPMXMIWCWXEXIQIRXC244*CƤRTHJFSXL

PEWXEGGIWWIHXL2SZIQFIV


Promoting Supply of Starter Homes.

,SYWMRKERH4PERRMRK&MPP-QTEGX%WWIWWQIRX

XL3GXSFIV
LXXT[[[TEVPMEQIRXYOHSGYQIRXW

MQTEGXEWWIWWQIRXW-%THJ

,SYWMRKERH4PERRMRK&MPP)\TPEREXSV]2SXIWXL

3GXSFIV
LXXT[[[TYFPMGEXMSRWTEVPMEQIRXYO

TEFMPPWGFMPPIRIRTHJFSXLPEWX

EGGIWWIHXL2SZIQFIV


6IJSVQWXSXLI2I[,SQIW&SRYW

2I[,SQIW&SRYW7LEVTIRMRKXLI-RGIRXMZI8IGLRMGEP

Consultation(IGIQFIV
LXXTW[[[KSZYO

government/uploads/system/uploads/attachment_data/

ƤPICCRLFCHVEJXCGSRHSGCTYFPMWLIHCZIVWMSR

THJPEWXEGGIWWIHXL2SZIQFIV


Right to Buy for Housing Association Tenants.

,SYWMRKERH4PERRMRK&MPP-QTEGX%WWIWWQIRX

XL3GXSFIV
LXXT[[[TEVPMEQIRXYOHSGYQIRXW

MQTEGXEWWIWWQIRXW-%THJ

,SYWMRKERH4PERRMRK&MPP)\TPEREXSV]2SXIWXL

3GXSFIV
LXXT[[[TYFPMGEXMSRWTEVPMEQIRXYO

TEFMPPWGFMPPIRIRTHJFSXLPEWX

EGGIWWIHXL2SZIQFIV


APPENDIX 4 

2070 The Promise of Evidence-Based Policymaking
9THEXMRKXLI0SGEP+SZIVRQIRX8VERWTEVIRG]'SHI

7XVIRKXLIRMRK0SGEP+SZIVRQIRX8VERWTEVIRG]

'SRWYPXEXMSRSRGLERKIWXSXLI0SGEP+SZIVRQIRX

8VERWTEVIRG]'SHI1E]
LXXTW[[[KSZYO

government/uploads/system/uploads/attachment_data/

ƤPI7XVIRKXLIRMRKCXLICPSGEPCKSZIVRQIRXC

XVERWTEVIRG]CGSHICGSRWYPXEXMSRTHJPEWXEGGIWWIHXL

2SZIQFIV


Department for Culture, 

Media and Sport (DCMS)

%KI:IVMƤGEXMSRJSV4SVRSKVETLMG1EXIVMEP3RPMRI

'LMPH7EJIX]3RPMRI%KI:IVMƤGEXMSRJSV4SVRSKVETL]

?GSRWYPXEXMSRA*IFVYEV]
LXXTW[[[KSZYO

government/uploads/system/uploads/attachment_data/

ƤPI%:C'SRWYPXEXMSR('17CC*MREPCCC

THJ

-HIRXMJ]MRKXLI6SYXIWF][LMGL'LMPHVIR:MI[4SVRSKVETL]

3RPMRI-QTPMGEXMSRWJSV*YXYVI4SPMG]QEOIVW7IIOMRKXS

0MQMX:MI[MRKŪ6ITSVXSJ)\TIVX4ERIPJSV('17XL

2SZIQFIV
LXXTW[[[KSZYOKSZIVRQIRX

YTPSEHWW]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI

6ITSVXCSJC('17C)\TIVXC4ERIPCC%YXYQRCCC*-2%0C

THJ

6IKYPEXSV]8VMEKI%WWIWWQIRX%KI:IVMƤGEXMSRJSV

TSVRSKVETLMGQEXIVMEPSRPMRIXL.ERYEV]
LXXTW

www.gov.uk/government/uploads/system/uploads/

EXXEGLQIRXCHEXEƤPI68%C('17C%:C

THJEPPPEWXEGGIWWIHXL2SZIQFIV


&VSEHFERH9RMZIVWEP7IVZMGI3FPMKEXMSR

%2I[&VSEHFERH9RMZIVWEP7IVZMGI3FPMKEXMSR

ConsultationVH1EVGL
LXXTW[[[KSZYO

government/uploads/system/uploads/attachment_data/

ƤPI&VSEHFERHC9RMZIVWEPC7IVZMGIC3FPMKEXMSRTHJ

PEWXEGGIWWIHXL2SZIQFIV


'YPXYVEP4VSXIGXMSR*YRH

'YPXYVEP4VSXIGXMSR*YRH+SZIVRQIRX4VSTSWEPWXS

4VSXIGX'YPXYVEP,IVMXEKI3ZIVWIEW?GSRWYPXEXMSRA.ERYEV]


LXXTW[[[KSZYOKSZIVRQIRXYTPSEHW

W]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI'YPXYVEPC

TVSXIGXMSRCJYRHCGSRWYPXEXMSRCCCTHJPEWXEGGIWWIHXL

2SZIQFIV


6IUYMVMRK(MVIGX1EVOIXMRK'EPPIVW

XS4VSZMHI'EPPMRK0MRI-HIRXMƤGEXMSR

6IUYMVMRKHMVIGXQEVOIXMRKGEPPIVWXSTVSZMHI'EPPMRK0MRI

-HIRXMƤGEXMSR?GSRWYPXEXMSRA.ERYEV]
LXXTW

www.gov.uk/government/uploads/system/uploads/

EXXEGLQIRXCHEXEƤPI'0-CGSRWYPXEXMSRCZIVWMSRCC

4(*CTHJPEWXEGGIWWIHXL2SZIQFIV


7TSVXW+SZIVRERGI'SHI

7TSVXMRK*YXYVI%2I[7XVEXIK]JSVER%GXMZI2EXMSR

'EFMRIX3JƤGI(IGIQFIV
LXXTW[[[KSZYO

government/uploads/system/uploads/attachment_data/

ƤPI7TSVXMRKC*YXYVIC%'')77-&0)THJPEWX

EGGIWWIHXL2SZIQFIV


9RMXEV]&SEVHJSVXLI&&'

%&&'JSVXLIJYXYVIEFVSEHGEWXIVSJHMWXMRGXMSR?[LMXI

TETIVA1E]
LXXTW[[[KSZYOKSZIVRQIRX

YTPSEHWW]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI

('17C%C&&'CJSVCXLICJYXYVICPMROIHCVIZTHJPEWX

EGGIWWIHXL2SZIQFIV


DCMS and DCLG

&YWMRIWW6EXIW6IPMIJJSV0SGEP2I[WTETIVW

8LIGEWIJSVE&YWMRIWW6EXIW6IPMIJJSVPSGEPRI[WTETIVW

Consultation, ('17ERH('0+XL.YP]
LXXTW

www.gov.uk/government/uploads/system/uploads/

EXXEGLQIRXCHEXEƤPI*MREPCGSRWYPXEXMSRC

HSGYQIRXCCFYWMRIWWCVEXIWCVIPMIJCCC.YP]CJSVC

TYFPMGEXMSRTHJPEWXEGGIWWIHXL2SZIQFIV


Department for Education

English Baccalaureate.

'SRWYPXEXMSRSRMQTPIQIRXMRKXLI)RKPMWL&EGGEPEYVIEXI

VH2SZIQFIV
LXXTW[[[KSZYOKSZIVRQIRX

YTPSEHWW]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI

'SRWYPXEXMSRCSRCMQTPIQIRXMRKCXLIC)RKPMWLC

&EGGEPEYVIEXITHJPEWXEGGIWWIHXL2SZIQFIV


*YPP%GEHIQMWEXMSRSJXLI7GLSSP7]WXIQ

Educational Excellence Everywhere?[LMXITETIVA1EVGL


LXXTW[[[KSZYOKSZIVRQIRXYTPSEHWW]WXIQ

YTPSEHWEXXEGLQIRXCHEXEƤPI)HYGEXMSREPC

)\GIPPIRGIC)ZIV][LIVITHJ

)HYGEXMSREP)\GIPPIRGI)ZIV][LIVI%WWIWWQIRXSJ

-QTEGX1EVGL
LXXTW[[[KSZYOKSZIVRQIRX

YTPSEHWW]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI

)HYGEXMSREPCI\GIPPIRGICIZIV][LIVICC-QTEGXC

%WWIWWQIRXTHJFSXLPEWXEGGIWWIHXL2SZIQFIV


+IRHIV4E]+ET+SZIVRQIRX)UYEPMXMIW3JƤGI


'PSWMRKXLI+IRHIV4E]+ET+SZIVRQIRX'SRWYPXEXMSR

(J)ERH+SZIVRQIRX)UYEPMXMIW3JƤGIXL.YP]


LXXTW[[[KSZYOKSZIVRQIRXYTPSEHWW]WXIQ

YTPSEHWEXXEGLQIRXCHEXEƤPI+IRHIVC4E]C+ETC

'SRWYPXEXMSRTHJ

APPENDIX 4 

The Promise of Evidence-Based Policymaking 2071
W-QTEGX%WWIWWQIRX+SZIVRQIRX)UYEPMXMIW3JƤGI

VH.YP]
LXXTW[[[KSZYOKSZIVRQIRX

uploads/system/uploads/attachment_data/

ƤPI7C-QTEGXC%WWIWWQIRXC+6))2HSG\FSXL

PEWXEGGIWWIHXL2SZIQFIV


National Teaching Service.

Educational Excellence Everywhere ?[LMXITETIVA1EVGL


LXXTW[[[KSZYOKSZIVRQIRXYTPSEHWW]WXIQ

YTPSEHWEXXEGLQIRXCHEXEƤPI)HYGEXMSREPC

)\GIPPIRGIC)ZIV][LIVITHJ

)HYGEXMSREP)\GIPPIRGI)ZIV][LIVI%WWIWWQIRXSJ

-QTEGX1EVGL
LXXTW[[[KSZYOKSZIVRQIRX

YTPSEHWW]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI

)HYGEXMSREPCI\GIPPIRGICIZIV][LIVICC-QTEGXC

%WWIWWQIRXTHJFSXLPEWXEGGIWWIHXL2SZIQFIV




2I[7SGMEP;SVO6IKYPEXSV

'LMPHVIRŭWWSGMEPGEVIVIJSVQ%ZMWMSRJSVGLERKI?TSPMG]

TETIVA.ERYEV]
LXXTW[[[KSZYOKSZIVRQIRX

YTPSEHWW]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI

Childrens_social_care_reform_a_vision_for_change.pdf 

PEWXEGGIWWIHXL2SZIQFIV


6IKMSREP%HSTXMSR%KIRGMIW4VSKVEQQI

Regionalising adoption?TSPMG]TETIVA.YRI
LXXTW

www.gov.uk/government/uploads/system/uploads/

EXXEGLQIRXCHEXEƤPI6IKMSREPMWMRKCEHSTXMSRTHJ

)HYGEXMSRERH%HSTXMSR&MPP)\TPEREXSV]2SXIWVH.YRI


LXXT[[[TYFPMGEXMSRWTEVPMEQIRXYOTEFMPPW

GFMPPIRIRTHJFSXLPEWXEGGIWWIH

XL2SZIQFIV


7GLSSPW2EXMSREP*YRHMRK*SVQYPE

7GLSSPWREXMSREPJYRHMRKJSVQYPE+SZIVRQIRXGSRWYPXEXMSR

ŪWXEKISRIXL1EVGL
LXXTWGSRWYPXIHYGEXMSR

KSZYOJYRHMRKTSPMG]YRMXWGLSSPWREXMSREPJYRHMRK

formula/supporting_documents/Schools_NFF_

GSRWYPXEXMSRTHJ

7GLSSPWERHLMKLRIIHWJYRHMRKVIJSVQ8LIGEWIJSV

GLERKIERHGSRWYPXEXMSRWYQQEV]1EVGL
LXXTW

GSRWYPXIHYGEXMSRKSZYOJYRHMRKTSPMG]YRMXWGLSSPW

REXMSREPJYRHMRKJSVQYPEWYTTSVXMRKCHSGYQIRXW

7YQQEV]	ERH	GEWI	JSV	GLERKITHJ

2EXMSREPJYRHMRKJSVQYPEIUYEPMX]EREP]WMW1EVGL


LXXTWGSRWYPXIHYGEXMSRKSZYOJYRHMRKTSPMG]YRMX

WGLSSPWREXMSREPJYRHMRKJSVQYPEWYTTSVXMRKCHSGYQIRXW

)UYEPMX]CEREP]WMWTHJ

7GLSSPVIZIRYIJYRHMRK'YVVIRXJYRHMRKEVVERKIQIRXW

1EVGL
LXXTWGSRWYPXIHYGEXMSRKSZYOJYRHMRK

TSPMG]YRMXWGLSSPWREXMSREPJYRHMRKJSVQYPEWYTTSVXMRKC

documents/Current_funding_system.pdf (all last accessed 

XL2SZIQFIV


Department for Environment, 

Food and Rural Affairs

&ERRMRKXLI&YVRMRKSJ;EWXI3MPEWE*YIPMR,IEXIVW

'SRWYPXEXMSRSRXLIMQTPIQIRXEXMSRSJEQIRHQIRXWXSXLI

IRZMVSRQIRXEPTIVQMXXMRKKYMHERGISR[EWXIMRGMRIVEXMSR

7ITXIQFIV
LXXTWGSRWYPXHIJVEKSZYO

EXQSWTLIVIPSGEPIRZMVSRQIRXXIEQ[EWXIMRGMRIVEXMSR

KYMHERGIWYTTSVXMRKCHSGYQIRXW'SRWYPXEXMSR	

(SGYQIRXTHJ

-QTPIQIRXEXMSRSJEQIRHQIRXWXSIRZMVSRQIRXEP

TIVQMXXMRKKYMHERGISR[EWXIMRGMRIVEXMSR?MQTEGX

EWWIWWQIRXA7ITXIQFIV
LXXTWGSRWYPXHIJVE

KSZYOEXQSWTLIVIPSGEPIRZMVSRQIRXXIEQ[EWXI

MRGMRIVEXMSRKYMHERGIWYTTSVXMRKCHSGYQIRXW%RRI\	

%	-QTEGX	%WWIWWQIRX	THJFSXLPEWXEGGIWWIH

XL2SZIQFIV


'LERKIWXS4PEWXMGERH+PEWW4EGOEKMRK6IG]GPMRK

Business Targets.

Consultation on changes to the plastic packaging recycling 

FYWMRIWWXEVKIXWJSVERHRI[XEVKIXWJSVTPEWXMG

ERHKPEWWJSV2SZIQFIV
LXXTWGSRWYPX

defra.gov.uk/waste/plasticandglasstargets/supporting_

HSGYQIRXW4PEWXMG	8EVKIX	%QIRHQIRX	

GSRWYPXEXMSR	ƤREPTHJ

4PEWXMG
+PEWW4EGOEKMRK6IG]GPMRK&YWMRIWW8EVKIXW

?MQTEGXEWWIWWQIRXAWX2SZIQFIV


LXXTWGSRWYPXHIJVEKSZYO[EWXITPEWXMGERHKPEWWXEVKIXW

WYTTSVXMRKCHSGYQIRXW4PEWXMG	KPEWW	XEVKIXW	

GSRWYPXEXMSR	-%	*-2%0THJFSXLPEWXEGGIWWIHXL

2SZIQFIV


Clean Air Zones, Draft Air Quality Plans.

'SRWYPXEXMSRSRHVEJXTPERWXSMQTVSZIEMVUYEPMX]8EGOPMRK

nitrogen dioxide in our towns and cities7ITXIQFIV


LXXTWGSRWYPXHIJVEKSZYOEMVUYEPMX]HVEJXEUTPERW

WYTTSVXMRKCHSGYQIRXW'SRWYPXEXMSR	HSGYQIRX	

	HVEJX	TPERW	XS	MQTVSZI	EMV	

UYEPMX]	7ITXIQFIV		ƤREP	ZIVWMSR	

JSPHIVTHJ

(VEJXTPERWXSMQTVSZIEMVUYEPMX]MRXLI9/8EGOPMRK

RMXVSKIRHMS\MHIMRSYVXS[RWERHGMXMIWŪ9/SZIVZMI[

HSGYQIRX7ITXIQFIV
LXXTWGSRWYPXHIJVEKSZ

YOEMVUYEPMX]HVEJXEUTPERWWYTTSVXMRKCHSGYQIRXW

(VEJX	TPERW	XS	MQTVSZI	EMV	UYEPMX]	

MR	XLI	9/		3ZIVZMI[	HSGYQIRX	

7ITXIQFIV		ƤREP	ZIVWMSR	JSPHIVTHJ

APPENDIX 4 

2072 The Promise of Evidence-Based Policymaking
(VEJX)ZMHIRGI%RRI\%WWIWWQIRXSJXLITPERWXSMQTVSZI

air quality in the UK7ITXIQFIV
LXXTWGSRWYPX

HIJVEKSZYOEMVUYEPMX]HVEJXEUTPERWWYTTSVXMRKC

HSGYQIRXW(VEJX	)ZMHIRGI	%RRI\			

EWWIWWQIRX	SJ	TPERW	XS	MQTVSZI	EMV	

UYEPMX]	MR	XLI	9/		7ITXIQFIV	

THJEPPPEWXEGGIWWIHXL2SZIQFIV


Single Animal Establishment Licence.

'SRWYPXEXMSRSRXLIVIZMI[SJERMQEPIWXEFPMWLQIRXW

licensing in England(IGIQFIV
LXXTWGSRWYPX

HIJVEKSZYOERMQEPLIEPXLERH[IPJEVIGSRWYPXEXMSRSR

XLIVIZMI[SJERMQEPPMGIRWMRKWYTTSVXMRKCHSGYQIRXW

'SRWYPXEXMSR	(SGYQIRX		6IZMI[	SJ	

%RMQEP	)WXEFPMWLQIRXW	0MGIRWMRK	MR	

)RKPERHTHJPEWXEGGIWWIHXL2SZIQFIV


Statutory Post-Movement Testing of Cattle for TB.

-QTVSZMRK8&GEXXPIGSRXVSPWMRGPYHMRKETVSTSWEPJSV

WXEXYXSV]TSWXQSZIQIRXXIWXMRK%GSRWYPXEXMSRI\IVGMWI

GSRXVMFYXMRKXSXLIHIPMZIV]SJXLI+SZIVRQIRXŭW7XVEXIK]

JSVEGLMIZMRK3JƤGMEPP]&SZMRI8YFIVGYPSWMW*VII38*


status for EnglandXL%YKYWX
LXXTWGSRWYPX

HIJVEKSZYOFSZMRIXFFSZMRIXFGEXXPIGSRXVSPWTSWX

QSZIQIRXXIWXMRKWYTTSVXMRKCHSGYQIRXW'EXXPI	

QIEWYVIW	GSRWYPXEXMSR	HSGYQIRXTHJ

3TXMSRWXSMRGVIEWIXLIGLERGISJEGLMIZMRK3JƤGMEPP]

8&*VII38*
WXEXYWJSVXLI8&0S[6MWO%VIE?MQTEGX

EWWIWWQIRXAXL(IGIQFIV
LXXTWGSRWYPX

HIJVEKSZYOFSZMRIXFFSZMRIXFGEXXPIGSRXVSPWTSWX

QSZIQIRXXIWXMRKWYTTSVXMRKCHSGYQIRXW()*6%	

-%	2S		1IEWYVIW	XS	EGLMIZI	

38*	JSV	XLI	06%THJFSXLPEWXEGGIWWIHXL

2SZIQFIV


7XVIRKXLIRIH1IEWYVIWEKEMRWX)TMXVM\

Consultation on strengthened protection against Epitrix 

3GXSFIV
LXXTW[[[KSZYOKSZIVRQIRXYTPSEHW

W]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPIITMXVM\

GSRWYPXTETIVTHJPEWXEGGIWWIHXL2SZIQFIV




Department for Transport

'LERKIWXSXLI(VMZMRK8IWXŬ'EWLFEGOŭ

1SXSVMRK7IVZMGIW7XVEXIK]'SRWYPXEXMSR1SZMRK&VMXEMR

Ahead 2SZIQFIV
LXXTW[[[KSZYOKSZIVRQIRX

YTPSEHWW]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI

QSXSVMRKWIVZMGIWWXVEXIK]GSRWYPXEXMSRTHJPEWX

EGGIWWIHXL2SZIQFIV


7EJIWIGYVIWYWXEMREFPI8LI1SXSVMRK7IVZMGIW%KIRGMIW

138WJSVEPPQEOIWSJZILMGPIW%TVMP
LXXTW

www.gov.uk/government/uploads/system/uploads/

EXXEGLQIRXCHEXEƤPIQSXSVMRKWIVZMGIWWXVEXIK]

pdf 

'LERKIWXSXLI*M\IH4IREPX]2SXMGIERH4IREPX]4SMRXW

JSVXLI9WISJE,ERH,IPH4LSRI;LMPWX(VMZMRK

%GSRWYPXEXMSRSRGLERKIWXSXLI*M\IH4IREPX]2SXMGI

ERHTIREPX]TSMRXWJSVXLIYWISJELERHLIPHQSFMPI

TLSRI[LMPWXHVMZMRK1SZMRK&VMXEMR%LIEH (January 


LXXTW[[[KSZYOKSZIVRQIRXYTPSEHWW]WXIQ

YTPSEHWEXXEGLQIRXCHEXEƤPILERHLIPHQSFMPI

TLSRIHVMZMRKTHJPEWXEGGIWWIHXL2SZIQFIV


1EOMRK8EGXMPI4EZMRK7YVJEGIW)EWMIVXS9WIEWE

2EZMKEXMSREP8SSPERH;EVRMRK7]WXIQ

-RXIVMQGLERKIWXSXLI+YMHERGISRXLIYWISJ8EGXMPI

4EZMRK7YVJEGIW1SZMRK&VMXEMR%LIEH%YKYWX


LXXTW[[[KSZYOKSZIVRQIRXYTPSEHWW]WXIQ

YTPSEHWEXXEGLQIRXCHEXEƤPIMRXIVMQGLERKIW

XEGXMPITEZMRKKYMHERGITHJPEWXEGGIWWIHXL2SZIQFIV




2I[&YW*VERGLMWMRK4S[IVWJSV0SGEP%YXLSVMXMIW

&YW6IJSVQ;SVOWLSTW&EGOKVSYRH(SGYQIRX1SZMRK

Britain Ahead7ITXIQFIV
LXXTW[[[KSZYO

government/uploads/system/uploads/attachment_data/

ƤPIFYWIWVIJSVQ[SVOWLSTWFEGOKVSYRHTHJ

PEWXEGGIWWIHXL2SZIQFIV


2I[4EVXRIVWLMTJSV6EMP4EWWIRKIV7IVZMGIWMRXLI

South East.

A new approach to rail passenger services in London and 

XLI7SYXL)EWX;SVOMRKMRTEVXRIVWLMTXSMQTVSZIWIVZMGIW

and support growth?TVSWTIGXYWA(J8ERH8VERWTSVXJSV

0SRHSR.ERYEV]
LXXTW[[[KSZYOKSZIVRQIRX

YTPSEHWW]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI

HJXXƥVEMPTVSWTIGXYWTHJPEWXEGGIWWIHXL2SZIQFIV




6IHYGMRK(MWVYTXMSRSR0SGEPŬ%ŭ6SEHW

6SEH;SVOW6IHYGMRK(MWVYTXMSRSR0SGEPŬ%ŭ6SEHWŪ

Moving Britain Ahead%TVMP
LXXTW[[[KSZYO

government/uploads/system/uploads/attachment_data/

ƤPIVIHYGMRKHMWVYTXMSRSRPSGEPEVSEHWTHJ

6IHYGMRK(MWVYTXMSRSR0SGEPŬ%ŭ6SEHW?MQTEGX

EWWIWWQIRXAXL%TVMP
LXXTW[[[KSZYO

government/uploads/system/uploads/attachment_data/

ƤPIVIHYGMRKHMWVYTXMSRSRPSGEPEVSEHWMQTEGX

EWWIWWQIRXTHJFSXLPEWXEGGIWWIHXL2SZIQFIV


APPENDIX 4 

The Promise of Evidence-Based Policymaking 2073
Department for Work and Pensions 

Banning Member-Borne Commission 

MR;SVOTPEGI4IRWMSRW

&IXXIV[SVOTPEGITIRWMSRW&ERRMRKQIQFIVFSVRI

GSQQMWWMSRMRSGGYTEXMSREPTIRWMSRWGLIQIW

?GSRWYPXEXMSRA3GXSFIV
LXXTW[[[KSZYO

government/uploads/system/uploads/attachment_data/

ƤPIFERRMRKQIQFIVFSVRIGSQQMWWMSR

GSRWYPXEXMSRSGXTHJ

&ERRMRKQIQFIVFSVRIGSQQMWWMSRMRSGGYTEXMSREP

TIRWMSRWGLIQIWYWIHJSVEYXSQEXMGIRVSPQIRX?MQTEGX

EWWIWWQIRXAXL(IGIQFIV
LXXTW[[[KSZYO

government/uploads/system/uploads/attachment_data/

ƤPIMQTEGXEWWIWWQIRXFERRMRKCQIQFIV

FSVRICGSQQMWWMSRMREIWGLIQIWTHJFSXLPEWX

EGGIWWIHXL2SZIQFIV


'ETSR)EVP])\MX'LEVKIWJSV1IQFIVW

of Occupational Pension Schemes.

'ETTMRKIEVP]I\MXGLEVKIWJSVQIQFIVWSJSGGYTEXMSREP

TIRWMSRWGLIQIW)RWYVMRKEJEMVERHGSRWMWXIRXETTVSEGL

EGVSWWEPPHIƤRIHGSRXVMFYXMSRTIRWMSRWŪ4YFPMG

consultation1E]
LXXTW[[[KSZYOKSZIVRQIRX

YTPSEHWW]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI

GSRWYPXEXMSRGETTMRKIEVP]I\MXGLEVKIWJSVQIQFIVWSJ

SGGYTEXMSREPTIRWMSRWGLIQIWTHJ

-RXVSHYGMRKEGETSRIEVP]I\MXGLEVKIWMRXVYWXFEWIH

SGGYTEXMSREPTIRWMSRWGLIQIW1E]
LXXTW

www.gov.uk/government/uploads/system/uploads/

EXXEGLQIRXCHEXEƤPIMQTEGXEWWIWWQIRX

GETSRIEVP]I\MXGLEVKIWMRXVYWXFEWIHSGGYTEXMSREP

TIRWMSRWGLIQIWTHJ

6IKYPEXSV]4SPMG]'SQQMXXIISTMRMSRSR-RXVSHYGMRKE

GETSRIEVP]I\MXGLEVKIWMRXVYWXFEWIHSGGYTEXMSREP

TIRWMSRWGLIQIWXL1E]
LXXTW[[[KSZYO

government/uploads/system/uploads/attachment_data/

ƤPIVTGSTMRMSRGETSRIEVP]I\MXGLEVKIWMR

XVYWXFEWIHSGGYTEXMSREPTIRWMSRWGLIQIWTHJEPPPEWX

EGGIWWIHXL2SZIQFIV


0MQMXMRKXLI'LMPH)PIQIRXSJ9RMZIVWEP

'VIHMXERH8E\'VIHMXW

;IPJEVI6IJSVQERH;SVO&MPP-QTEGX%WWIWWQIRXSJ8E\

'VIHMXWERH9RMZIVWEP'VIHMXGLERKIWXS'LMPH)PIQIRXERH

*EQMP])PIQIRX.YP]
LXXT[[[TEVPMEQIRXYO

HSGYQIRXWMQTEGXEWWIWWQIRXW-%)THJ

;IPJEVI6IJSVQERH;SVO&MPP)\TPEREXSV]2SXIWXL.YP]


LXXT[[[TYFPMGEXMSRWTEVPMEQIRXYOTEFMPPW

GFMPPIRIRTHJFSXLPEWXEGGIWWIH

XL2SZIQFIV


6IHYGMRKXLI&IRIƤX'ET

;IPJEVI6IJSVQERH;SVO&MPP-QTEGX%WWIWWQIRXJSV

XLIFIRIƤXGET.YP]
LXXT[[[TEVPMEQIRXYO

HSGYQIRXWMQTEGXEWWIWWQIRXW-%THJ

;IPJEVI6IJSVQERH;SVO&MPP)\TPEREXSV]2SXIWXL.YP]


LXXT[[[TYFPMGEXMSRWTEVPMEQIRXYOTEFMPPW

GFMPPIRIRTHJFSXLPEWXEGGIWWIH

XL2SZIQFIV


Youth Obligation.

7YQQIV&YHKIX,18VIEWYV].YP]
LXXTW

www.gov.uk/government/uploads/system/uploads/

EXXEGLQIRXCHEXEƤPIC7YQQIVC

&YHKIXCC;IFC%GGIWWMFPITHJ

-QTEGXSRLSYWILSPHW(MWXVMFYXMSREPEREP]WMWXS

EGGSQTER]7YQQIV&YHKIX,18VIEWYV].YP]


LXXTW[[[KSZYOKSZIVRQIRXYTPSEHW

W]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI49C

(MWXVMFYXMSREPC%REP]WMWTHJ

7YQQIV&YHKIXTSPMG]GSWXMRKW, ,1+SZIVRQIRX

.YP]
LXXTW[[[KSZYOKSZIVRQIRXYTPSEHW

W]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI4SPMG]C

GSWXMRKWCWYQQIVCFYHKIXCTHJEPPPEWXEGGIWWIHXL

2SZIQFIV


Department of Energy 

and Climate Change

Changes to Financial Support for Solar PV 

?TLSXSZSPXEMGA

'SRWYPXEXMSRSRGLERKIWXSƤRERGMEPWYTTSVXJSVWSPEV

4:'SRXVSPPMRKWTIRHMRKSRWSPEV4:TVSNIGXWSJ1;

and below within the Renewables Obligation (22nd July 


LXXTW[[[KSZYOKSZIVRQIRXYTPSEHWW]WXIQ

YTPSEHWEXXEGLQIRXCHEXEƤPI7SPEVC4:C[MXLMRC

XLIC63CGSRWYPXEXMSRTHJ

'SRWYPXEXMSRSRGLERKIWXSƤRERGMEPWYTTSVXJSVWSPEV

4:TVSNIGXWEX1;ERHFIPS[YRHIVXLI6IRI[EFPIW

Obligation ?MQTEGXEWWIWWQIRXAXL.YP]
LXXTW

www.gov.uk/government/uploads/system/uploads/

EXXEGLQIRXCHEXEƤPI7SPEVC4:C[MXLMRCXLIC

63CGSRWYPXEXMSRCC-QTEGXC%WWIWWQIRXTHJFSXLPEWX

EGGIWWIHXL2SZIQFIV


Contracts for Difference for Carbon Capture 

ERHɄ7XSVEKI

)PIGXVMGMX]1EVOIX6IJSVQ%QIRHQIRXXS'SRXVEGXWJSV

(MJJIVIRGI(IƤRMXMSRSJ)PMKMFPI+IRIVEXSV
6IKYPEXMSRW

Ū'SRWYPXEXMSRSRETVSTSWIHEQIRHQIRXXSIREFPI

VIXVSƤX'EVFSR'ETXYVIERH7XSVEKITVSNIGXWXSFIIPMKMFPI

for a Contract for Difference3GXSFIV
LXXTW

www.gov.uk/government/uploads/system/uploads/

APPENDIX 4 

2074 The Promise of Evidence-Based Policymaking
EXXEGLQIRXCHEXEƤPI)16C''7C'J(CIPMKMFMPMX]C

GSRWYPXEXMSRCCGSRWYPXEXMSRCHSGYQIRXTHJPEWXEGGIWWIH

XL2SZIQFIV


)\IQTXMSRJVSQXLI'SWXWSJXLI6IRI[EFPIW3FPMKEXMSR

ERH*IIHMR8EVMJJ

'SRWYPXEXMSRSRMQTPIQIRXMRKERI\IQTXMSRJSV)RIVK]

-RXIRWMZI-RHYWXVMIWJVSQXLIMRHMVIGXGSWXWSJXLI

6IRI[EFPIW3FPMKEXMSRERH*IIHMR8EVMJJ7GLIQIW

%TVMP
LXXTW[[[KSZYOKSZIVRQIRXYTPSEHW

W]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI)--C

GSRHSGCTHJ

1SZMRKJVSQGSQTIRWEXMSRXSI\IQTXMSRJVSQXLIGSWXW

SJXLI6IRI[EFPIW3FPMKEXMSRERH*IIH-R8EVMJJJSVIRIVK]

intensive industries?MQTEGXEWWIWWQIRXAWX%TVMP


LXXTW[[[KSZYOKSZIVRQIRXYTPSEHWW]WXIQ

YTPSEHWEXXEGLQIRXCHEXEƤPI)--C)\IQTXMSRC

63CERHC*-8C-%C7MKRIHC*SVC4YFPMGEXMSRCC%TVMPCC

%RRI\C9TCCCTHJFSXLPEWXEGGIWWIHXL2SZIQFIV




4VSZMHMRK3JKIQ[MXL4S[IVWXS-QTPIQIRX7[MXGLMRK

ERH7IXXPIQIRX6IJSVQW

(VEJX1IEWYVIW*EWXERHVIPMEFPIW[MXGLMRKERH,EPJLSYVP]

IPIGXVMGMX]WIXXPIQIRXTS[IVW
?MQTEGXEWWIWWQIRXAWX

.ERYEV]
LXXTW[[[KSZYOKSZIVRQIRXYTPSEHW

W]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI-QTEGXC

%WWIWWQIRXCC(VEJXC1IEWYVIWCC*EWXCERHC6IPMEFPIC

7[MXGLMRKCERHC,EPCCCTHJ

Draft Legislation on Energy.ERYEV]
LXXTW

www.gov.uk/government/uploads/system/uploads/

EXXEGLQIRXCHEXEƤPI(VEJXC0IKMWPEXMSRCSRC

)RIVK]THJFSXLPEWXEGGIWWIHXL2SZIQFIV


6IJSVQWXSXLI;EVQ,SQI(MWGSYRX7GLIQI

;EVQ,SQI(MWGSYRX7GLIQI?GSRWYPXEXMSRAXL%TVMP


LXXTW[[[KSZYOKSZIVRQIRXYTPSEHWW]WXIQ

YTPSEHWEXXEGLQIRXCHEXEƤPI*MREPC;EVQC

,SQIC(MWGSYRXCGSRWYPXEXMSRCJSVCTYFPMGEXMSRTHJ

;EVQ,SQI(MWGSYRX)\XIRWMSRXS?MQTEGX

EWWIWWQIRXAXL%TVMP
LXXTW[[[KSZYO

government/uploads/system/uploads/attachment_data/

ƤPI*-2%0C;EVQC,SQIC(MWGSYRXCC

I\XIRWMSRCGSRWYPXEXMSRC-%C'327908%8-32THJFSXLPEWX

EGGIWWIHXL2SZIQFIV


6IZMWMRK*IIHMR8EVMJJW

'SRWYPXEXMSRSREVIZMI[SJXLI*IIHMR8EVMJJWWGLIQI

XL%YKYWX
LXXTW[[[KSZYOKSZIVRQIRX

YTPSEHWW]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI

'SRWYPXEXMSRCSRCEC6IZMI[CSJCJIIHMRCXEVMJJCWGLIQITHJ

4IVMSHMG6IZMI[SJ*-8W?MQTEGXEWWIWWQIRXAXL

%YKYWX
LXXTW[[[KSZYOKSZIVRQIRXYTPSEHW

W]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI-%CJSVC

*-8WCGSRWYPXEXMSRC%YKYWXCCC*-2%0CHSG\CCI

WMKREXYVICMRGPYHIHCCZTHJFSXLPEWXEGGIWWIHXL

2SZIQFIV


Department of Health

HE]2,7

HE]2,7WIVZMGIWEJEGXWLIIX?TSPMG]TETIVAXL.YP]


LXXTW[[[KSZYOKSZIVRQIRXTYFPMGEXMSRW

HE]RLWWIVZMGIWEJEGXWLIIXHE]RLWWIVZMGIWE

JEGXWLIIXPEWXEGGIWWIHXL2SZIQFIV


(IEXL'IVXMƤGEXMSR6IJSVQW

-RXVSHYGXMSRSJ1IHMGEP)\EQMRIVWERH6IJSVQWXS

(IEXL'IVXMƤGEXMSRMR)RKPERHERH;EPIW4SPMG]ERH

(VEJX6IKYPEXMSRWŪ'SRWYPXEXMSRXL1EVGL


LXXTW[[[KSZYOKSZIVRQIRXYTPSEHWW]WXIQ

YTPSEHWEXXEGLQIRXCHEXEƤPI('6C'SRWYPXMSRC

(SGYQIRXTHJ

'SRWYPXEXMSRSR6IJSVQMRKXLI(IEXL'IVXMƤGEXMSR

4VSGIWWMR)RKPERHERH;EPIW?MQTEGXEWWIWWQIRXAXL

.ERYEV]
LXXTW[[[KSZYOKSZIVRQIRXYTPSEHW

W]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI-QTEGXC

EWWIWWQIRXC%THJ

6IJSVQMRKHIEXLGIVXMƤGEXMSR-RXVSHYGMRKWGVYXMR]F]

1IHMGEP)\EQMRIVWŪ0IWWSRWJVSQXLITMPSXWSJXLI

VIJSVQWWIXSYXMRXLI'SVSRIVWERH.YWXMGI%GX1E]


LXXTW[[[KSZYOKSZIVRQIRXYTPSEHWW]WXIQ

YTPSEHWEXXEGLQIRXCHEXEƤPI(IEXLCGIVXMƤGEXIC

VIJSVQWCTMPSXWCCVITSVXC%THJ

(IEXL'IVXMƤGEXMSR6IKYPEXMSRW?WXEXYXSV]

MRWXVYQIRXALXXTW[[[KSZYOKSZIVRQIRXYTPSEHW

W]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI(IEXLC

GIVXMƤGEXMSRCVIKYPEXMSRWC%THJ

JYVXLIVHSGYQIRXWEZEMPEFPIEXLXXTW[[[KSZYO

KSZIVRQIRXGSRWYPXEXMSRWHIEXLGIVXMƤGEXMSRVIJSVQWEPP

TEKIWPEWXEGGIWWIHXL2SZIQFIV


)\XIRHMRK'LEVKIWJSV2,77IVZMGIWJSV3ZIVWIEW

:MWMXSVWERH1MKVERXW

1EOMRKEJEMVGSRXVMFYXMSR%GSRWYPXEXMSRSRXLII\XIRWMSR

SJGLEVKMRKSZIVWIEWZMWMXSVWERHQMKVERXWYWMRKXLI

2,7MR)RKPERH(IGIQFIV
LXXTW[[[KSZYO

government/uploads/system/uploads/attachment_data/

ƤPI2,7CGLEVKMRKCEGGTHJ

:MWMXSVERH1MKVERX'SWX6IGSZIV]Ū)\XIRHMRK'LEVKMRK

?MQTEGXEWWIWWQIRXAXL2SZIQFIV
LXXTW

www.gov.uk/government/uploads/system/uploads/

APPENDIX 4 

The Promise of Evidence-Based Policymaking 2075
EXXEGLQIRXCHEXEƤPI-QTEGXC%WWIWWQIRXTHJ

FSXLPEWXEGGIWWIHXL2SZIQFIV


-RJIGXIH&PSSH6IJSVQSJ7YTTSVX

-RJIGXIHFPSSHVIJSVQSJƤRERGMEPERHSXLIVWYTTSVX

?GSRWYPXEXMSRA.ERYEV]
LXXTW[[[KSZYO

government/uploads/system/uploads/attachment_data/

ƤPI-RJIGXIHCFPSSHCGSRWCHSGTHJ

(IZIPSTMRKERI[W]WXIQSJƤRERGMEPERHSXLIVWYTTSVXJSV

TISTPIMRJIGXIH[MXLLITEXMXMW'ERHSV,-:XLVSYKLFPSSH

and blood products in the UK?MQTEGXEWWIWWQIRXAWX

.ERYEV]
LXXTW[[[KSZYOKSZIVRQIRXYTPSEHW

W]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI-QTEGXC

%WWIWWQIRXTHJFSXLPEWXEGGIWWIHXL2SZIQFIV




2I[%PGSLSP+YMHIPMRIW

,S[XSOIITLIEPXLVMWOWJVSQHVMROMRKEPGSLSPXSEPS[

PIZIP+SZIVRQIRXVIWTSRWIXSXLITYFPMGGSRWYPXEXMSR

%YKYWX
LXXTW[[[KSZYOKSZIVRQIRXYTPSEHW

W]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI'13C

(VMROMRKC+SZC6IWTTHJ

9/'LMIJ1IHMGEP3JƤGIVWŭ%PGSLSP+YMHIPMRIW6IZMI[

7YQQEV]SJXLITVSTSWIHRI[KYMHIPMRIW (January 


LXXTW[[[KSZYOKSZIVRQIRXYTPSEHWW]WXIQ

YTPSEHWEXXEGLQIRXCHEXEƤPIWYQQEV]THJFSXL

PEWXEGGIWWIHXL2SZIQFIV


%PGSLSP+YMHIPMRIW6IZMI[Ū6ITSVXJVSQXLI+YMHIPMRIW

HIZIPSTQIRXKVSYTXSXLI9/'LMIJ1IHMGEP3JƤGIVW

.ERYEV]
LXXTW[[[KSZYOKSZIVRQIRXYTPSEHW

W]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI+(+C

VITSVX.ERTHJEGGIWWIHXL2SZIQFIV


NHS Bursary Reforms.

6IJSVQMRKLIEPXLGEVIIHYGEXMSRJYRHMRKGVIEXMRKE

sustainable future workforce?GSRWYPXEXMSRA1E]


LXXTWGSRWYPXEXMSRWHLKSZYO[SVOJSVGILIEPXLGEVI

IHYGEXMSRJYRHMRKWYTTSVXMRKCHSGYQIRXW,IEPXLGEVI	

IHYGEXMSR	JYRHMRK	VIJSVQTHJ

6IJSVQWXSJYRHMRKERHƤRERGMEPWYTTSVXJSVRYVWMRK

QMH[MJIV]ERH%,4&YVWEV]WXYHIRXW76
?MQTEGX

EWWIWWQIRXAXL%TVMP
LXXTW[[[KSZYO

government/uploads/system/uploads/attachment_data/

ƤPI2,7C&YVWEV]C-%THJ

)UYEPMX]%REP]WMW'SRWYPXEXMSRWXEKI
Ū6IJSVQMRK

LIEPXLGEVIIHYGEXMSRJYRHMRKGVIEXMRKEWYWXEMREFPIJYXYVI

workforce%TVMP
LXXTW[[[KSZYOKSZIVRQIRX

YTPSEHWW]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI

)%2,7CFYVWEV]THJEPPPEWXEGGIWWIHXL2SZIQFIV




HM Treasury

%FSPMWLMRKXLI'EVFSR6IHYGXMSR'SQQMXQIRX

6IJSVQMRKXLIFYWMRIWWIRIVK]IJƤGMIRG]XE\PERHWGETI

?GSRWYPXEXMSRA7ITXIQFIV
LXXTW[[[KSZYO

government/uploads/system/uploads/attachment_data/

ƤPI49CFYWMRIWWCIRIVK]THJPEWXEGGIWWIH

XL2SZIQFIV


Help to Save.

7YQQIV&YHKIX,18VIEWYV].YP]
LXXTW

www.gov.uk/government/uploads/system/uploads/

EXXEGLQIRXCHEXEƤPIC7YQQIVC

&YHKIXCC;IFC%GGIWWMFPITHJ

-QTEGXSRLSYWILSPHW(MWXVMFYXMSREPEREP]WMWXS

EGGSQTER]7YQQIV&YHKIX,18VIEWYV].YP]


LXXTW[[[KSZYOKSZIVRQIRXYTPSEHW

W]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI49C

(MWXVMFYXMSREPC%REP]WMWTHJ

7YQQIV&YHKIXTSPMG]GSWXMRKW,1+SZIVRQIRX

.YP]
LXXTW[[[KSZYOKSZIVRQIRXYTPSEHW

W]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI4SPMG]C

GSWXMRKWCWYQQIVCFYHKIXCTHJEPPPEWXEGGIWWIHXL

2SZIQFIV


2I[(IPMZIV]1SHIPJSV4YFPMG*MRERGMEP+YMHERGI

4YFPMGƤRERGMEPKYMHERGIVIZMI[TVSTSWEPJSVGSRWYPXEXMSR

1EVGL
LXXTW[[[KSZYOKSZIVRQIRXYTPSEHW

W]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI49C

Public_Financial_Guidance_proposal_for_consultation_

*-2%0THJPEWXEGGIWWIHXL2SZIQFIV


6IJSVQWXSXLI8E\EXMSRSJ2SR(SQMGMPIW

6IJSVQWXSXLIXE\EXMSRSJRSRHSQMGMPIW?GSRWYPXEXMSRA

XL7ITXIQFIVYTHEXIHXL%YKYWX
LXXTW

[[[KSZYOKSZIVRQIRXGSRWYPXEXMSRWVIJSVQWXSXLI

XE\EXMSRSJRSRHSQMGMPIWVIJSVQWXSXLIXE\EXMSRSJRSR

HSQMGMPIWPEWXEGGIWWIHXL2SZIQFIV


7SJX(VMROW-RHYWXV]0IZ]

7YQQIV&YHKIX,18VIEWYV].YP]
LXXTW

www.gov.uk/government/uploads/system/uploads/

EXXEGLQIRXCHEXEƤPIC7YQQIVC

&YHKIXCC;IFC%GGIWWMFPITHJ

7YQQIV&YHKIXTSPMG]GSWXMRKW,1+SZIVRQIRX

.YP]
LXXTW[[[KSZYOKSZIVRQIRXYTPSEHW

APPENDIX 4 

2076 The Promise of Evidence-Based Policymaking
W]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI4SPMG]C

GSWXMRKWCWYQQIVCFYHKIXCTHJFSXLPEWXEGGIWWIH

XL2SZIQFIV


7XVIRKXLIRMRKXLI+SZIVRERGI%GGSYRXEFMPMX]ERH

8VERWTEVIRG]SJXLI&EROSJ)RKPERH

&EROSJ)RKPERH&MPPXIGLRMGEPGSRWYPXEXMSR.YP]


LXXTW[[[KSZYOKSZIVRQIRXYTPSEHWW]WXIQ

YTPSEHWEXXEGLQIRXCHEXEƤPIFEROCSJCIRKPERHC

FMPPCZTHJPEWXEGGIWWIHXL2SZIQFIV


Home Office

Ban on Psychoactive Substances.

Creation of a blanket ban on new psychoactive substances 

in the UK ?MQTEGXEWWIWWQIRXAXL1E]
LXXTW

www.gov.uk/government/uploads/system/uploads/

EXXEGLQIRXCHEXEƤPI247+&-QTEGX%WWIWWQIRX

THJPEWXEGGIWWIHXL2SZIQFIV


-RXVSHYGMRKE7XEPOMRK4VSXIGXMSR3VHIV

-RXVSHYGMRKE7XEPOMRK4VSXIGXMSR3VHIVŪEGSRWYPXEXMSR

(IGIQFIV
LXXTW[[[KSZYOKSZIVRQIRX

YTPSEHWW]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI

-RXVSHYGMRKCEC7XEPOMRKC4VSXIGXMSRC3VHIVCCEC

GSRWYPXEXMSRTHJPEWXEGGIWWIHXL2SZIQFIV


2I['VMQMREP3JJIRGISJ(VMZMRK;LMPIER-PPIKEP

Migrant.

-QQMKVEXMSR&MPP)\TPEREXSV]2SXIWXL7ITXIQFIV


LXXT[[[TYFPMGEXMSRWTEVPMEQIRXYOTEFMPPW

GFMPPIRIRTHJ

3ZIVEVGLMRK-QTEGX%WWIWWQIRX-QQMKVEXMSR&MPPXL

7ITXIQFIV
LXXT[[[TEVPMEQIRXYOHSGYQIRXW

MQTEGXEWWIWWQIRXW-%THJFSXLPEWXEGGIWWIH

XL2SZIQFIV


6IJSVQMRKXLI-RHITIRHIRX4SPMGI'SQTPEMRXW

Commission.

6IJSVQMRKXLI-RHITIRHIRX4SPMGI'SQTPEMRXW

'SQQMWWMSRWXVYGXYVIERHKSZIVRERGIũ%TYFPMG

consultation(IGIQFIV
LXXTW[[[KSZYO

government/uploads/system/uploads/attachment_data/

ƤPI-4''C+SZIVRERGIC'SR(SG;)&

ZC9/C3CCCTHJPEWXEGGIWWIHXL2SZIQFIV


7MRKPI0IKMWPEXMZI4VSZMWMSRXS4VSZMHIJSV)UYMTQIRX

Interference.

-RZIWXMKEXSV]4S[IVW&MPP)UYMTQIRX-RXIVJIVIRGI?MQTEGX

EWWIWWQIRXAXL2SZIQFIV
LXXTW[[[KSZYO

government/uploads/system/uploads/attachment_data/

ƤPI-QTEGXC%WWIWWQIRX)UYMTQIRXC-RXIVJIVIRGI

THJ

*EGXWLIIX8EVKIXIH)UYMTQIRX-RXIVJIVIRGIXL3GXSFIV


LXXTW[[[KSZYOKSZIVRQIRXYTPSEHWW]WXIQ

YTPSEHWEXXEGLQIRXCHEXEƤPI*EGXWLIIX

8EVKIXIHC)UYMTQIRXC-RXIVJIVIRGITHJ

(VEJX-RZIWXMKEXSV]4S[IVW&MPP2SZIQFIV
LXXTW

www.gov.uk/government/uploads/system/uploads/

EXXEGLQIRXCHEXEƤPI(VEJXC-RZIWXMKEXSV]C

4S[IVWC&MPPTHJEPPPEWXEGGIWWIHXL2SZIQFIV


7XVIRKXLIRMRKXLI6SPISJXLI(MWGPSWYVIERH&EVVMRK

Service.

'SYRXIV)\XVIQMWQ7XVEXIK]?[LMXITETIVA,1

+SZIVRQIRX3GXSFIV
LXXTW[[[KSZYO

government/uploads/system/uploads/attachment_data/

ƤPIC'QC%GGIWWMFPITHJPEWX

EGGIWWIHXL2SZIQFIV


Ministry of Justice

%QIRHXLI'MZMP0IKEP%MH1IVMXW'VMXIVME

8LI'MZMP0IKEP%MH1IVMXW'VMXIVME
%QIRHQIRX
2S


6IKYPEXMSRW?MQTEGXEWWIWWQIRXAVH.YP]


LXXT[[[PIKMWPEXMSRKSZYOYOMETHJW

YOMECCIRTHJ

)\TPEREXSV]1IQSVERHYQXSXLI'MZMP0IKEP%MH1IVMXW

'VMXIVME
%QIRHQIRX
2S
6IKYPEXMSRWVH.YP]


LXXT[[[PIKMWPEXMSRKSZYOYOWM

THJWYOWMIQCCIRTHJFSXLPEWXEGGIWWIHXL

2SZIQFIV


'ETSRXLI*IIWXLEX6IKYPEXIH'PEMQW1EREKIQIRX

Companies can Charge Consumers.

'PEMQW1EREKIQIRX6IKYPEXMSRŪ'SRWYPXEXMSR

'YXXMRKXLIGSWXWJSVGSRWYQIVWŪ*MRERGMEP'PEMQW

XL*IFVYEV]
LXXTWGSRWYPXNYWXMGIKSZYO

HMKMXEPGSQQYRMGEXMSRWGYXXMRKGSWXWJSVGSRWYQIVW

ƤRERMGEPGPEMQWWYTTSVXMRKCHSGYQIRXW'SRWYPXEXMSR	

	'16		'YXXMRK	XLI	GSWXW	JSV	

GSRWYQIVW		*MRERGMEP	'PEMQW			

*IF	THJ

%RRI\-QTEGX%WWIWWQIRX'PEMQW1EREKIQIRX

6IKYPEXMSRũ'YXXMRKXLIGSWXWJSVGSRWYQIVWŪ*MRERGMEP

'PEMQWRH(IGIQFIV
LXXTWGSRWYPXNYWXMGIKSZ

YOHMKMXEPGSQQYRMGEXMSRWGYXXMRKGSWXWJSVGSRWYQIVW

ƤRERMGEPGPEMQWWYTTSVXMRKCHSGYQIRXW%RRI\	%	

	'16	-QTEGX	%WWIWWQIRX		'YXXMRK	

XLI	GSWXW	JSV	GSRWYQIVW		*MRERGMEP	

'PEMQW		13.			'SZIV	

THJ

%RRI\-QTEGX%WWIWWQIRXLXXTWGSRWYPXNYWXMGIKSZ

YOHMKMXEPGSQQYRMGEXMSRWGYXXMRKGSWXWJSVGSRWYQIVW

ƤRERMGEPGPEMQWWYTTSVXMRKCHSGYQIRXW%RRI\	%	

	'16	-QTEGX	%WWIWWQIRX		'YXXMRK	

APPENDIX 4 

The Promise of Evidence-Based Policymaking 2077
XLI	GSWXW	JSV	GSRWYQIVW		*MRERGMEP	

'PEMQW		13.		THJEPPPEWX

EGGIWWIHXL2SZIQFIV


'LERKIWXSXLI(IƤRMXMSRSJŰ7EQTPIVűJSV(2%8IWXMRK

MR4VMZEXI*EQMP]0E['LMPHVIR
'EWIW

1MRMWXV]SJ.YWXMGI6IWTSRWI(2%8IWXMRKMRTVMZEXIJEQMP]

PE[GLMPHVIR
GEWIW&PSSH8IWXW)ZMHIRGISJ4EXIVRMX]


6IKYPEXMSRW'SRWYPXEXMSRSRTSWWMFPIGLERKIWXS

XLIHIƤRMXMSRSJŰWEQTPIVűERH[LEXEWEQTPIVQYWXHS

?GSRWYPXEXMSRVIWTSRWIAXL3GXSFIV
LXXTW

GSRWYPXNYWXMGIKSZYOHMKMXEPGSQQYRMGEXMSRWHRE

XIWXMRKMRTVMZEXIJEQMP]PE[GLMPHVIRGEWIWVIWYPXW

HREXIWXMRKGSRWYPXEXMSRVITSVXTHJPEWXEGGIWWIHXL

2SZIQFIV


-RXVSHYGMRKE4ERIPJSV4YFPMGP]*YRHIH'VMQMREP

%HZSGEG]

4VIWIVZMRKERH)RLERGMRKXLI5YEPMX]SJ'VMQMREP%HZSGEG]

?GSRWYPXEXMSRAWX3GXSFIV
LXXTWGSRWYPX

NYWXMGIKSZYOHMKMXEPGSQQYRMGEXMSRWIRLERGMRKXLI

UYEPMX]SJGVMQMREPEHZSGEG]WYTTSVXMRKCHSGYQIRXW

GSRWYPXEXMSRTETIVTHJ

4VIWIVZMRKERH)RLERGMRKXLI5YEPMX]SJ'VMQMREP%HZSGEG]

?MQTEGXEWWIWWQIRXAXL7ITXIQFIV
LXXTW

GSRWYPXNYWXMGIKSZYOHMKMXEPGSQQYRMGEXMSRWIRLERGMRK

XLIUYEPMX]SJGVMQMREPEHZSGEG]WYTTSVXMRKCHSGYQIRXW

MQTEGXEWWIWWQIRXTHJ

4VIWIVZMRKERH)RLERGMRKXLI5YEPMX]SJ'VMQMREP%HZSGEG]

ũ)UYEPMX]7XEXIQIRXWX3GXSFIV
LXXTWGSRWYPX

NYWXMGIKSZYOHMKMXEPGSQQYRMGEXMSRWIRLERGMRKXLI

UYEPMX]SJGVMQMREPEHZSGEG]WYTTSVXMRKCHSGYQIRXW

IUYEPMX]WXEXIQIRXTHJEPPPEWXEGGIWWIHXL2SZIQFIV




Merger of Local Justice Areas in Greater Manchester.

'SRWYPXEXMSRSRXLIQIVKIVSJXLI0SGEP.YWXMGI%VIEWMR

+VIEXIV1ERGLIWXIVŪ%GSRWYPXEXMSRYRHIVXEOIRSRFILEPJ

SJXLI+VIEXIV1ERGLIWXIV.YHMGMEP&YWMRIWW+VSYTERH,1

Courts and Tribunals ServiceVH%YKYWX
LXXTW

GSRWYPXNYWXMGIKSZYOHMKMXEPGSQQYRMGEXMSRWQIVKIV

SJPSGEPNYWXMGIEVIEWKVIEXIVQERGLIWXIVWYTTSVXMRKC

HSGYQIRXWGSRWYPXEXMSRHSGTHJPEWXEGGIWWIHXL

2SZIQFIV


6EXMSREPMWMRKXLI'SYVXERH8VMFYREP)WXEXI

4VSTSWEPSRXLITVSZMWMSRSJGSYVXERHXVMFYREPIWXEXIMR

England and Wales?GSRWYPXEXMSRAXL.YP]
LXXTW

GSRWYPXNYWXMGIKSZYOHMKMXEPGSQQYRMGEXMSRWTVSTSWEP

SRXLITVSZMWMSRSJGSYVXERHXVMFYREPIWYWIVCYTPSEHW

VIJSVQIWXEXIWREXMSREPGSRWYPXEXMSRCSJƤGMEPWIRWMXMZIC

ƤREPCTHJ

-QTEGX%WWIWWQIRXSR,IV1ENIWX]ŭW'SYVXWERH8VMFYREPW

Service proposals on the provision of courts services 

in England and Wales, LXXTWGSRWYPXNYWXMGIKSZYO

HMKMXEPGSQQYRMGEXMSRWTVSTSWEPSRXLITVSZMWMSRSJ

GSYVXERHXVMFYREPIWYWIVCYTPSEHWMQTEGXEWWIWWQIRX

LQGXWIWXEXIWZƤREPTHJFSXLPEWXEGGIWWIHXL

2SZIQFIV


APPENDIX 4 

2078 The Promise of Evidence-Based Policymaking
APPENDIX 5: TESTABLE CLAIMS. 

WHEN IS EVIDENCE EXPECTED

4SPMG]QEOMRKMWEFSYXƤRHMRK[E]WXSMRƥYIRGIERHSVKERMWIWSGMIX]WSMXMWFEWIHSREGSQFMREXMSR

of politics, values and pragmatism. Some claims about why a policy is being introduced cannot be 

tested with evidence and some can. 

&YX[LMPI[IGEREPPEKVIIXLEXXLIVIEVIQER]JEGXSVWMREHIGMWMSRMXŭWYWIJYPXSHMWXMRKYMWL[LIVI[I[SYPHI\TIGX

WYTTSVXMRKIZMHIRGIERH[LIVI[I[SYPHRŭX;ILEZIEVMKLXXSORS[[LIRTSPMG]QEOIVWEVIWMQTP]EWWIVXMRKFIPMIJW

ERHɄ[LIRXLI]EVIQEOMRKGPEMQWEFSYXJEGX8LIVIMWE[SVPHSJHMJJIVIRGIFIX[IIRI\TVIWWMRKTSPMXMGEPZEPYIWERHQEOMRK

TVSQMWIWXLEXGSYPHSRIHE]FITVSZIHVMKLXSV[VSRK*SVI\EQTPI

“Hosting the Olympics will be an exciting 

and prestigious thing for our city to do.”

“Marriage is the bedrock of our society and we 

should support and recognise that commitment 

through a married couples’ tax allowance.”

“Supporting children from all backgrounds 

is a priority for this government, so we need 

a national network of early years centres 

to show that commitment.”

“We don’t believe that what people do in their 

personal lives is our business. So we support 

decriminalisation of small amounts of cannabis 

for personal use.”

“It is not fair that some rich people don’t play 

by the same rules as everyone else. ‘Nondoms’ should be taxed.”

“This government is committed to devolving 

power to the regions because everyone has 

a right to local democracy.”

“Hosting the Olympics will encourage more 

people to take up sport and get more exercise, 

and there will be a net financial benefit from 

increased tourism and investment.”

“A married couple’s allowance will increase 

the number of children growing up in married 

households and reduce anti-social behaviour.”

“If we provide early years centres we will 

measurably improve the education of children 

who have access to them.”

“Cannabis has substantial health benefits 

and we should decriminalise it.”

“‘Non-doms’ should be taxed because it will 

bring more money to the Exchequer, even 

if some of them leave as a result.”

“This government is committed to devolving 

power to the regions because it will stimulate 

economic regeneration.”

8LIWII\EQTPIW[IVITYXXSKIXLIV[MXLLIPTJVSQ(V(EZMH6SFIVX+VMQIW APPENDIX 5 

The Promise of Evidence-Based Policymaking 2079
Sense about Science is an independent campaigning charity that 

GLEPPIRKIWXLIQMWVITVIWIRXEXMSRSJWGMIRGIERHWGMIRXMƤGIZMHIRGI

in public life. We advocate openness and honesty about research 

ƤRHMRKWERH[SVOXSIRWYVIXLITYFPMGMRXIVIWXMRWSYRHWGMIRGI

ERHɄIZMHIRGIMWVITVIWIRXIHERHVIGSKRMWIHMRTYFPMGHMWGYWWMSR

ERHɄTSPMG]QEOMRK

4YFPMWLIHF]7IRWIEFSYX7GMIRGIMR2SZIQFIVɄ

(IWMKRIHF]*VERGIWGE8SVXSVE

For more copies or for further information, please contact 

7XITLERMI1EXLMWIRWQEXLMWIR$WIRWIEFSYXWGMIRGISVK

Sense about Science

E'PIVOIR[IPP+VIIR

0SRHSR)'6(4




www.senseaboutscience.org

LIPPS$WIRWIEFSYXWGMIRGISVK

6IKMWXIVIHGLEVMX]2S

'SQTER]2S

2080 The Promise of Evidence-Based Policymaking
MISSING 

EVIDENCE:

An inquiry into the delayed 

publication of governmentcommissioned research

The Right Honourable Sir Stephen Sedley

June 2016

The Promise of Evidence-Based Policymaking 2081
2082 The Promise of Evidence-Based Policymaking
0(9$;14$016+(49(+$0'18(4616+(*18(40/(06

$%176$6+,4'1)(8(4;6+,0*9(($40!+$6>5$./156

6+4((37$46(451)$64,..,102170'5$;($4#(5+17.'

I\TIGX;LMXILEPPXSɄWTIRHMX[MWIP]SRSYVFILEPJ

=IXɄ[LMPIEɄPSXSJɄIRIVK]MWɄWTIRXWGVYXMRMWMRKXLITVMZEXI

PMZIWSJɄMRHMZMHYEPTSPMXMGMERW[IORS[PMXXPIEFSYXLS[

QYGLTYFPMGTSPMG]MWɄFEWIH

SRɄEWWIVXMSRLS[QYGLVIWXW

SRɄHSKQEERHLS[QYGL

MWɄKVSYRHIHSRVMKSVSYW

6(56,0*$0'$0$.;5,5

4(*$4'6+,5$510(1)6+(

/156,/2146$06&+$..(0*(5

)$&,0*17451&,(6;)9('10>6

ORS[[LEX[SVOW[IɄEVIPIJX

[MXLEWWYQTXMSRŪERHɄEWɄ[I

EPPORS[EWWYQTXMSRMWXLI

QSXLIVSJɄEɄPSXSJɄQMWXEOIW

WSQISJɄXLIQZIV]WIVMSYW

;LIXLIVXLIMWWYIMWWEJIX]SJZEGGMRIWSVɄXLIɄVMKLX[E]

XSɄXIEGLGLMPHVIRSVXLIFIWXETTVSEGLXSHVYKEFYWI

SVɄGVMQITSPMGMIWFEWIHSRKYIWW[SVOLEPJXVYXLSVFIPMIJ

EVIRSXNYWXTSXIRXMEPP]ZIV][EWXIJYPFYXɄHERKIVSYW

!+(4($4(69118(44,',0*241%.(/56+$6*11'(8,'(0&(

MWɄSJXIRWGEVGIERHXLEXEHZIVWEVMEPHIFEXIHSIWRSX

$.9$;5(0&174$*(1%-(&6,8,6;76(8(09+(0,/2146$06

(8,'(0&((:,5659('10>6$.9$;5*(6615((,6

!+,5,59+; ,4 6(2+(0 ('.(;+$5'10(57&+$27%.,&

5(48,&(9,6++,54(2146 ,4 6(2+(0,510(1)6+(">5

)14(/156(:2(46510$'/,0,564$6,8(.$99,6+(:2(4,(0&(

MRXLI,MKL'SYVXERH'SYVXSJ%TTIEP;IEWOIH

LMQXSɄVIZMI[LS[QYGLKSZIVRQIRXVIWIEVGLKIXW

TYFPMWLIHLS[QYGLMWɄLMHHIRJVSQYWERHMJWS[L]

,MWVITSVXFVIEOWRI[KVSYRH

-EQTPIEWIHXSWIILIɄMWɄRSXGVMXMGEPSJTSPMXMGMERW

ERHɄRSVHSIWLIUYEVVIP[MXLXLIɄTSPMXMGEPREXYVI

SJɄTSPMG]QEOMRK7YFNIGXMZMX]MWMRIZMXEFPI;IɄEVIɄEPP

XIQTXIHXSɄGLIVV]TMGOIZMHIRGIŪ[LEXTW]GLSPSKMWXW

&$..&1*0,6,8(%,$51%(.9,00(4$0,(.$+0(/$00

GEPPWMXɄŬXLMROMRKJEWXŭVEXLIVXLERKMZMRKIUYEP[IMKLX

61)$&656+$670'(4/,0(17424(&10&(26,10576,6

,5(52(&,$..;+$4')1421.,6,&,$0561)$&(,0&108(0,(06

)$&656,5016-7566+$62(12.($664$&6('612$46;21.,6,&5

XIRHXSFIVSSXIHXSɄEɄREVVEXMZIFYXXLEXIZIRSTIR

QMRHIHTSPMXMGMERWEVII\TIGXIHXSɄJSPPS[ETEVX]

PMRIXSɄLEZITSPMGMIWJSVɄIEGLSGGEWMSRERHɄXVIRGLERX

ZMI[WJSVɄIZIV]5YIWXMSR8MQIETTIEVERGI8LI]ɄEVI

FYJJIXIHF]XLIRI[WG]GPIERHɄF]ɄHMWXVEGXMSRWERH

GVMWIWJEQSYWP]HIWGVMFIHEWŰIZIRXWHIEVFS]

IZIRXWű=IXɄXLI]JEGIVMHMGYPIMJXLI]QEOI9XYVRW

IZIR[LIRXLI]GLERKIXLIMVQMRHWMRɄXLIɄPMKLX

SJɄFIXXIVɄIZMHIRGI

7S[IEPPTPE]ETEVXMRTIVWYEHMRKTSPMG]QEOIVWXSɄREMP

6+(,4&1.1745616+(/$56,66.(910'(46+$651/(

21.,6,&$.$'8,5(45$4(5&(26,&$.$%1764(5($4&+5,0&(

XLI]JIEVMXɄQMKLXGSQIYT[MXLXLI[VSRKERW[IVW

3RɄXLIɄSXLIVLERHWSQI[LSJEZSYVVIWIEVGL

XLMROYRFVMHPIHTYFPMGEXMSRMWEWXITXSSJEVWMRGI

MJɄIQFEVVEWWMRKERW[IVWQYWXKSSRXLITYFPMGVIGSVH

XLIVIMWEVMWOXLEXPIWWVIWIEVGL[MPPFIGSQQMWWMSRIH

 ,4 6(2+(04(8($.5+196+,0*5*1$'4,)6%76$.51+19

WSQITSPMG]QEOIVWVIWSPZIXLIWIXIRWMSRWQYGL

%(66(46+$016+(45

2SSRIORS[WLS[QYGLXLIKSZIVRQIRXWTIRHW

104(5($4&+0)$&610(1)6+(241%.(/5 ,4 6(2+(0

$''4(55(5,5+19.,66.(&114',0$6,106+(4(,5$&4155

TSPMG]ɄVIWIEVGL&YXIWXMQEXIWWYKKIWXEFSYX

eɄFMPPMSREɄ]IEVKSIWXSTVSZMHMRKERIZMHIRGI

FEWIJSVTYFPMGTSPMG]%RHɄJSVSRIVIEWSRSVERSXLIV

EWɄLMWɄVITSVXQEOIWGPIEVWSQISJXLMWJEMPWXSWII

XLIɄPMKLXSJHE]

&&$5,10$..;6+,5,5%(&$75(1)'(.,%(4$6(57224(55,10

FYX7MV7XITLIRTSMRXWXSHIITIVW]WXIQMGTVSFPIQW

!+("*18(40/(06+$5$&4(',6$%.(4(276$6,10

JSVɄSTIRRIWWŪMX[EWVIGIRXP]VEROIHEWSRI

SJɄXLIɄQSWXɄXVERWTEVIRXMRXLI[SVPH1MWWMRK

SVɄFYVMIHɄVIWIEVGLVITVIWIRXWEHIQSGVEXMGHIƤGMX

-XɄJIIHWG]RMGMWQERHLEWRSTPEGIMRER]LSRIWX

ERHɄ[IPPGSRHYGXIHɄEHQMRMWXVEXMSR

7SLMWVITSVXMWE[EOIYTGEPP7MV7XITLIR7IHPI]

215(5,/2146$0637(56,105$66+(+($461)9+,&+

MWɄXLMWLS[GER[IGVIEXIEPIZIPTPE]MRKƤIPH[LIVI

TSPMG]QEOIVWGERJSPPS[JEGXWQSVIIEWMP]XLERWOI[

XLIQXSƤXTVIGSRGIMZIHTSWMXMSRWERH[LIVIVIPMEFPI

(8,'(0&(&17065)14/14(6+$02(457$5,8(5170'

%,6(5(+$5%(*70$&108(45$6,106+$65+17.'019

XEOITPEGI[MHIP]XLSYKLXJYPP][MXLSYXTSMRXWGSVMRK

ERHɄ[MXLSYXɄHIJIRWMZIRIWW






DGHPRFUDWLFGHȴFLW

	





DQGbZHOOFRQGXFWHG

DGPLQLVWUDWLRQ

FOREWORD

by Nick Ross

	
MWENSYVREPMWXFVSEHGEWXIV

ERHɄXVYWXIIɄSJɄ7IRWIEFSYX7GMIRGI

The Promise of Evidence-Based Policymaking 2083
2084 The Promise of Evidence-Based Policymaking
CONTENTS

1 Introduction

2 Summary

3 Section 1:

 Is there a problem?

5 Section 2:

 The system of conducting, commissioning and publishing government research

11 Section 3:

 Reasons for the delayed publication or withholding of government research

30 Section 4:

 Remedies and recommendations

37 Afterword Tracey Brown, director of Sense about Science

39 Appendix I:

 Inquiry methods and call for evidence

41 Appendix II:

 List of contributors

45 Appendix III:

 Freedom of Information requests

47 Appendix IV:

 Bibliography

The Promise of Evidence-Based Policymaking 2085
!+("0,6(',0*'1/*18(40/(0652(0'5$%176<
%,..,10

E]IEVSRVIWIEVGLMRXIRHIHXSKYMHI

HIZIPSTQSHMJ]ERHQSRMXSVTSPMG]SRE[MHIZEVMIX]SJMWWYIW7SQISJɄXLMWɄTSPMG]VIWIEVGL

MWɄHSRIMRLSYWI7SQISJɄMXɄMWɄGSRHYGXIHF]SYXWMHIMRHMZMHYEPWERHɄFSHMIW[LSEVITEMHXSHSMX

65$,/59(4(61

@ 9RHIVWXERH[LEXLEWLETTIRIHMRORS[RGEWIW

SJɄHIPE]MRVIWIEVGLTYFPMGEXMSR

@  ((+19241%.(/$6,&4(5($4&+&$0%(+$0'.('9(..

@ '(06,);211424$&6,&(,06+(&1//,55,10,0*

ERHɄHMWWIQMREXMSRSJKSZIVRQIRXVIWIEVGL

@ 4$9$66(06,1061*11'24$&6,&(6+$6&17.'

ERHɄWLSYPHFIVITPMGEXIH

!+,5,5$012214670(/1/(0661&105,'(46+(8,5,%,.,6;

1)6+((8,'(0&(%$5()1427%.,&21.,&;18(40/(06

$0'&,8,.51&,(6;$4('(8(.12,0*6+(">56+,4'$6,10$.

%GXMSR4PERSR3TIR+SZIVRQIRX[LMGL[MPPTVSTSWI

[E]WXSQEOIKSZIVRQIRXQSVISTIRERHEGGSYRXEFPI

8LI-RHITIRHIRX'SQQMWWMSRSRɄ*VIIHSQ

SJɄ-RJSVQEXMSRLEWVIMRJSVGIHXLITVMRGMTPISJSTIRRIWW

-RXVSHYGMRKMXWVITSVXXSTEVPMEQIRXMR1EVGL

0SVHɄ&VMHKIWWEMHŰ;IEVIGSQQMXXIHXSFIMRKXLIQSWX

XVERWTEVIRXKSZIVRQIRXMRXLI[SVPHű-RXLIɄ;SVPH

;MHI;IF*SYRHEXMSRVEROIHXLI9/EWXLIQSWX

XVERWTEVIRXKSZIVRQIRXMRXLI[SVPH=IXEWXLMWVITSVX

HIQSRWXVEXIWXLI[MXLLSPHMRKSJI\XIVREPVIWIEVGLJVSQ

27%.,&5&476,0;4(/$,05$241%.(/

%XEXMQISJTYFPMGWTIRHMRKGSRWXVEMRXWMXMWEPWS

XSɄFIɄI\TIGXIHXLEXKSZIVRQIRX[MPPWIIOSTTSVXYRMXMIW

XSɄMRGVIEWIXLIZEPYIHIVMZIHJVSQVIWIEVGLVIHYGI[EWXI

ERHMHIRXMJ]MRIJJIGXMZISVQMWXEOIRTSPMG]ɄHIGMWMSRW

8LMWMRUYMV][EWMRMXMEXIHMR7ITXIQFIV-XWWGSTI

[EWHIXIVQMRIHF]MRMXMEPGSRWYPXEXMSR[MXLVIWIEVGLIVW

GMZMPWIVZERXWERHTSPMXMGMERW8LIɄMRHMZMHYEPW

MRɄXLIɄWGMIRXMƤGTSPMG]ERHTSPMXMGEPWIGXSVW[LS

TVSZMHIH[VMXXIRWYFQMWWMSRWERHXLSWI[MXL[LSQ

[IɄGSRHYGXIHMRXIVZMI[WEVIPMWXIHMR%TTIRHM\--

7SQIɄEVIUYSXIHFIPS[

FYXXLMWVITSVXLEWFIRIƤXIH

KVIEXP]JVSQGSRXEGX[MXLEPPSJXLIQ-XWGSRXIRX

XLSYKLVIQEMRWXLIEYXLSVŭWVIWTSRWMFMPMX]

!+(,037,4;$0'6+(241'7&6,101)6+,54(21469(4(

5722146('%;$*4$06)41/6+( !+$4,6$%.(

8VYWX6IKMWXIVIH'LEVMX]2S
ERHF]2MGO

155(6$,.51)6+(,037,4;>5/(6+1'5$0'6+(&$..

JSVɄIZMHIRGIEVIMR%TTIRHM\-

INTRODUCTION

!+,54(2146,5&10&(40('9,6+6+(27%.,&$6,10

SJɄVIWIEVGL[LMGLKSZIVRQIRXLEW)41/

(:6(40$.4(5($4&+(45!+(4($5105)146+,5$4(%16+

TVEGXMGEPERHPIKEP8LIZSPYQISJɄVIWIEVGLHSRI[MXLMR

HITEVXQIRXWERHXLIZEVMIX]SJɄJSVQWMXɄXEOIWEVILYKI

ERHETEVXJVSQXLIIWXMQEXIHKPSFEPGSWX
YRUYERXMƤIH

8LIVSYXMRIERRSYRGIQIRXERHɄTYFPMGEXMSRSJEPP

WYGLVIWIEVGL[SYPHRSXSRP]GVIEXIEɄGSRWMHIVEFPI

$'/,0,564$6,8(%74'(0,6917.'4(37,4(6+(5(*4(*$6,10

SJVIWIEVGLƤRHMRKWJVSQTSPMG]EHZMGIXSQMRMWXIVW

[LMGLEXTVIWIRX[LIXLIVVMKLXP]SV[VSRKP]IRNS]W

,//70,6;)41/',5&.1574((8(0,0.,6,*$6,10!+(&$5(

)1464$052$4(0&;$%176,064$'(2$46/(06$.,02765

,06121.,&;)14/$6,10,56+(4()14(',))(4(06)41/

XLIGEWI[LMGLXLMWVITSVXI\EQMRIWJSVMRZEVMEFPI

TYFPMGEXMSRSJɄI\TIVXVIWIEVGLERHƤRHMRKWI\XIVREPP]

&1//,55,10('%;*18(40/(06

+SZIVRQIRX[LMGLVIEHMP]VIGSKRMWIWXLEXXLITYFPMG

,5(06,6.('615((6+(241'7&61)(:6(40$.4(5($4&+2$,'

JSV[MXLMXWQSRI]LEWWIXKYMHIPMRIWERHTVSXSGSPWXLEX

VIUYMVITVSQTXTYFPMGEXMSR%PPHITEVXQIRXWVIƥIGXXLMW

4(37,4(/(06,06+(,44(5($4&+564$6(*,(5

10(6+(.(55*18(40/(064(5($4&+,5016$.9$;5

TYFPMWLIHMRWYJƤGMIRXXMQIJSVMRJSVQIHTYFPMG

',5&755,10!+(4(+$8(%((0,056$0&(51)27%.,&$6,10

FIMRKLIPHFEGO[LIRXLIƤRHMRKW[IVITSPMXMGEPP]

E[O[EVH7YGLGEWIWLEZIMRGPYHIHVIWIEVGLSRJSSH

FEROWMQQMKVEXMSRGLSMGISJ+4ERHHVYKWTSPMG]

!+(5(,056$0&(5+$8(.('61&10&(40$%176

@ *EMPYVIXSQEOITYFPMGP]JYRHIHVIWIEVGLEZEMPEFPI

XSɄXLITYFPMG

@ %PEGOSJXVERWTEVIRG]EFSYXXLIFEWMW

SJɄKSZIVRQIRXɄHIGMWMSRWERHXLIVSPIXLEX

IZMHIRGIɄTPE]IHMRɄVIEGLMRKXLIQ

@ 4('7&('9,..,0*0(551)4(5($4&+(4561$55,56

MRɄTSPMG]QEOMRK

-RVIWTSRWIXSXLIWITYFPMGGSRGIVRW7IRWIEFSYX

 &,(0&(,0,6,$6('6+,5,037,4;,0616+('(.$;('27%.,&$6,10

SJKSZIVRQIRXVIWIEVGLERHMRXSTSWWMFPIVIQIHMIW

(6$,.51)6+($76+145+,21)6+(4(2146$4(5(6176

MRɄ%TTIRHM\-

 INTRODUCTION

2086 The Promise of Evidence-Based Policymaking
2SGSQTVILIRWMZIEGGSYRXI\MWXWSJLS[QYGLKSZIVRQIRXVIWIEVGLMWGSQQMWWMSRIH

RSVɄSJɄLS[QYGLSJMXMWTYFPMWLIHERHɄ[LIR8LMWMRMXWIPJMWEWMKRMƤGERXTVSFPIQ

ERHɄMXɄMWɄ[L]ɄXLMWVITSVXVIGSQQIRHWXLEXXLIVIWLSYPHFIEWMRKPITYFPMGP]WIEVGLEFPI

HEXEFEWIɄSJɄEPPɄI\XIVREPKSZIVRQIRXVIWIEVGL

LEWEPVIEH]FIIRWYFQMXXIHJSVYWIF]ɄKSZIVRQIRX

7MQMPEVP]ENSYVREPŭWTVMRXWGLIHYPIWLSYPHRSXEJJIGX

XLIXMQIPMRIWWSJɄVIWIEVGLJSVɄTYFPMGHMWGYWWMSR

SJɄTSPMG]VIZMI[IHTVITVMRXWWLSYPHFIɄYWIH

[LIVIɄRIGIWWEV]

%REP]WMWSJXLIVIEWSRWJSVHIPE]IHTYFPMGEXMSR

ERHɄSJɄTVEGXMGIWXLEXEPPS[HMJƤGYPXVIWIEVGL

XSFILERHPIH[IPPTSMRXWXSXLIJSPPS[MRK

4(&1//(0'$6,105

 $56$0'$4',5('&(064$.4(*,56(41)$..(:6(40$..;

&1//,55,10('*18(40/(064(5($4&+

 &.$4,6;109+$6&1056,676(5(:6(40$..;

&1//,55,10('*18(40/(064(5($4&+

 $&.($4&1//,6/(0661241/2627%.,&$6,10

MRɄVIWIEVGLGSRXVEGXW

 4176,0(27%.,&$6,101)4(5($4&+*18(40/(06

LEWGSRWMHIVIHMRTSPMG]JSVQYPEXMSR[MXL

MJɄETTVSTVMEXIVIEWSRWJSVVINIGXMRKMX

 $&.($456$6(/(061)6+(&744(064(37,4(/(065

JSVɄTVSQTXTYFPMGEXMSRERHEHLIVIRGIXSXLIQ

 64$,0,0*,04(5($4&+)1421.,&;&1//70,&$6145

!+(5(4(&1//(0'$6,105$4(5(6176,0/14('(6$,.

EXɄXLIIRHSJXLIVITSVX

SUMMARY

8LIQSWXTVSQMRIRXMWWYIMWHIPE]MRXLIɄTYFPMGEXMSR

SJɄVIWIEVGLƤRHMRKWXSEGGSQQSHEXITSPMXMGEPGSRGIVRW

EFSYXTSPMG]ERRSYRGIQIRXW-JɄKSZIVRQIRXTSPMG]

MWɄJSVQYPEXIHERHERRSYRGIHFIJSVIXLIɄQIHME

ERHɄXLIɄTYFPMGORS[[LEXI\TIVXIZMHIRGIQMRMWXIVW

LEZIVIGIMZIHTYFPMGHMWGYWWMSRMRGPYHMRKTEVPMEQIRXEV]

WGVYXMR]
QE]FILERHMGETTIHSVɄWXMƥIH(IPE]HIWMKRIH

WMQTP]XSEZSMHTSPMXMGEPIQFEVVEWWQIRXMWɄRSXIXLMGEPP]

EGGITXEFPIERHɄXLMWVITSVXTYXWJSV[EVHTVSTSWEPW

XSIRWYVIXMQIP]TYFPMGEGGIWWXSɄVIWIEVGLIZIRŪ

MRHIIHIWTIGMEPP]Ū[LIVIMXɄGSRƥMGXW[MXLTSPMG]

-XɄ[MPPFIWYKKIWXIHXLEXVIWIEVGLWLSYPHSVHMREVMP]

FIɄTYFPMWLIHEWWSSREWɄMXɄMWɄVIEH]ERHELIEHSJTSPMG]

$00170&(/(06570.(556+,5,5*(07,0(.;,/24$&6,&$%.(

()(44$.%(;10'6+(6,/(1)21.,&;241/7.*$6,105+17.'

VIUYMVIZIV]WXVSRKNYWXMƤGEXMSR

;27%.,&$6,109(/($010.,0($&&(55616+(4(2146

SJɄXLIVIWIEVGL-XETTIEVWGYVVIRXP]XLEXYRGIVXEMRXMIW

18(4)14/$6$0'$0;4(8,(9241&(55&$0'(.$;

TYFPMGEXMSRERHWSQI[MXRIWWIWLEZIHIWGVMFIH

GSRGIVRWXLEXXLMWYRGIVXEMRX]GERFIɄYWIHMRXIRXMSREPP]

!+(24$&6,&(5+17.'%(61$*4((6+(6;2($0')14/$6

SJXLIVIWIEVGLVITSVXEWɄTEVXSJɄXLIɄGSQQMWWMSR

0;4(8,(9241&(55$0'6,/,0*1)6+$65+17.'$.51

FIɄEKVIIHMREHZERGI4PERWJSVENSYVREPTYFPMGEXMSR

EVIRSXEVIEWSRJSVɄHIPE]MRKTYFPMGEXMSR[LIVIEVITSVX



8LIQSWXVIGIRX]IEVJSV[LMGL3JƤGIJSV2EXMSREP7XEXMWXMGW327
ƤKYVIWEVI

EZEMPEFPIMWɅ8LI327FYPPIXMRSR9/+SZIVRQIRX)\TIRHMXYVISR7GMIRGI

)RKMRIIVMRKERH8IGLRSPSK]EPWSWLS[WXLEXKSZIVRQIRXWTIRXeFMPPMSRSRMRLSYWI

6
(TYVGLEWIH6
(ERHSRJYRHMRKI\XIVREPSVKERMWEXMSRWXSVYR6
(FSXLMRXLI

9/ERHɅSZIVWIEW-XɅMWɅHMJƤGYPXXSXIPPLS[QYGLSJXLMW[EWWTIRXSRGSQQMWWMSRIH

VIWIEVGL+SZIVRQIRXWITEVEXIP]WTIRXeFMPPMSRSRJYRHMRKVIWIEVGLGSYRGMPW

8LIɅVIWIEVGLHSRIF]XLIWIMRWXMXYXMSREPFSHMIWMWRSX[MXLMRXLIWGSTISJXLMWMRUYMV]



5YSXEXMSRWYRPIWWSXLIV[MWIWXEXIHMRXLIXI\X

SVɅJSSXRSXIWEVIJVSQWYFQMWWMSRWXSɅXLIɅMRUYMV]

SUMMARY 

The Promise of Evidence-Based Policymaking 2087
IS THERE 

A PROBLEM?

Section 1

 SECTION 1: IS THERE A PROBLEM?

2088 The Promise of Evidence-Based Policymaking
 6/,*+6+$8(%((06+17*+66+$6$0;241%.(/1)'(.$;('27%.,&$6,101)*18(40/(064(5($4&+&17.'

FIVIEHMP]MHIRXMƤIHERHUYERXMƤIH)EGLHITEVXQIRXSJWXEXIERHI\IGYXMZIEKIRG]QYWXORS[

[LEXVIWIEVGLGSRXVEGXWMXLEWMWWYIHMRWE]XLIPEWXXLVII]IEVWERHFIEFPIXSWE][LIXLIV

XLIɄVIWIEVGLIVWLEZIVITSVXIHERHMJXLI]LEZI[LIRXLIMVVITSVX[EWQEHIɄTYFPMG

 -XLEWFIIRHMWXYVFMRKXSƤRHXLEXXLMWMWRSXLS[XLMRKWEVI8LII\XIRXSJVITSVXMRKERHɅVIGSVH

OIITMRKZEVMIWIRSVQSYWP]FIX[IIRKSZIVRQIRXHITEVXQIRXW9WMRK*VIIHSQSJɄ-RJSVQEXMSR

*S-
VIUYIWXWMXLEWIQIVKIHXLEXWSQIHITEVXQIRXWŪIKXLIɄ(ITEVXQIRXJSV)RZMVSRQIRX

*SSHɄERHɄ6YVEP%JJEMVWXLI(ITEVXQIRXJSV8VERWTSVXERHXLI(ITEVXQIRXSJ,IEPXLŪKMZIɄSRPMRI

EGGIWWXSɄHEXEFEWIWXLEXWIXSYX[LEXVIWIEVGLLEWFIIRGSQQMWWMSRIHERH[LEXWXEKI

MXɄLEWVIEGLIH3XLIVWŪIKɄXLIɄ(ITEVXQIRXJSV'SQQYRMXMIWERH0SGEP+SZIVRQIRX('0+
Ū

SRP]ɄPMWXTYFPMWLIHWXYHMIW[MXLSYXMRHMGEXMRK[LIXLIVXLIɄPMWXWGSRXEMREPPVIWIEVGLXLEXLEWFIIR

GSQQMWWMSRIH7IIɄ%TTIRHM\---JSVJYPPHIXEMPWSJɄXLIɄ*S-VIUYIWXW

 7SQIHITEVXQIRXWVIWTSRHIHXS*S-VIUYIWXWF]WYTTPIQIRXMRKXLIMVTYFPMGPMWXW[MXLQSVIGSQTPIXI

MRJSVQEXMSR('0+TVSZMHIHEGSQTVILIRWMZIPMWXSJWXYHMIWGSQQMWWMSRIHGSQTPIXIHERHTYFPMWLIH

FIX[IIRERH&]GSRXVEWXXLI(ITEVXQIRXSJ)RIVK]ERH'PMQEXI'LERKIERHXLI*SVIMKR

3JƤGIHMHRSXORS[[LEXLEHFIIRGSQQMWWMSRIHFIGEYWIXLEXMRJSVQEXMSR[EWHMWTIVWIH[MXLMR

XLIɄHITEVXQIRXERHRSXLIPHGIRXVEPP]%HHMRKXSXLIHMJƤGYPX]SJƤRHMRKSYXEFSYXKSZIVRQIRX

VIWIEVGLMXLEWFIIRWYKKIWXIHXSYWXLEXMRGSQMRKEHQMRMWXVEXMSRWQE]WSQIXMQIWGSRWMKR

HMVIGXP]XSɅXLIɅ2EXMSREP%VGLMZIWVIWIEVGLFIWTSOIRF]XLIMVTVIHIGIWWSVW

 -RXLIEFWIRGISJXLIGSQTVILIRWMZIMRJSVQEXMSRXLEXSYKLXXSFIEZEMPEFPIMXLEWFIIRRIGIWWEV]

61(:$/,0(9+$6+$5+$22(0(',0&$5(59+(4(6+()$,.74(6127%.,5+*18(40/(064(5($4&++$'%((0

4(2146(',06+(/(',$14(.5(9+(4(

 ;IORS[JVSQE7YTVIQI'SYVXNYHKQIRXXLEXMRXLILSQIWIGVIXEV][MXLLIPHTYFPMGEXMSR

SJɄVIWIEVGLXLEXJEMPIHXSWYTTSVXETVSTSWIHTSPMG]SJHIR]MRKZMWEWXSɄWTSYWIW[LIVISRI

SJɄXLIɄGSYTPI[EWYRHIV&YXGSRGIVRWSZIVXLIMRHITIRHIRGISJɄKSZIVRQIRXVIWIEVGLVIEGL

JYVXLIVFEGO4VSJIWWSV6SKIV8EVPMRKJSVQIVLIEHSJɄXLIɄ,SQI3JƤGI6IWIEVGLERH4PERRMRK9RMX

[VSXIMREFSYXWXVEMRIHVIPEXMSRWFIX[IIRKSZIVRQIRXERHWSGMEPVIWIEVGLIVWHYVMRKXLIW

ERHWERHMRXLIɄ%WWSGMEXMSRSJɄ7SGMEP6IWIEVGL3VKERMWEXMSRWVIZMI[IHGSRHMXMSRWEFSYX

TYFPMGEXMSRMRɄVIWIEVGLGSRXVEGXWFIGEYWIXLIMVQIQFIVWZMI[IHXLIQEWFIGSQMRKŰMRGVIEWMRKP]

10(4175$0'4(564,&6,8(?

	 8LIVIMWRSVIGIRXIZMHIRGISJXLIMRHIƤRMXIWYTTVIWWMSRSJVIWIEVGL,S[IZIVXLIVIMWɄKSSH

IZMHIRGIXLEXVIWIEVGLTYFPMGEXMSRHSIWKIXHIPE]IH8LIVIEWSRWMREHHMXMSRXSɄTSPMXMGEPGSRGIVRW

EFSYXXLIɄXMQMRKSJTYFPMGEXMSRQE]MRGPYHIYRGIVXEMRX]EFSYXTIIVVIZMI[EFSYX[LEXGSYRXW

EWɄKSZIVRQIRXVIWIEVGLERHEFSYX[LEXWLSYPHFITYFPMWLIHMRɄVIPEXMSRXSTSPMG]ERRSYRGIQIRXW

=IXHIPE]IHTYFPMGEXMSRGERFIEWHEQEKMRKEWMRHIƤRMXIWYTTVIWWMSR%(&$75(,6'(24,8(5

TEVPMEQIRXEVMERWXLIQIHME2+3WERHSXLIVWSJɄXLIɄXMQIP]EGGIWWXLI]RIIHMRSVHIVXSFIEFPI

XSɄIRKEKI[MXLTSPMG]JSVQEXMSRMRXLIPMKLXSJGSRXIQTSVERISYWIZMHIRGI

 8LIVI[MPPEP[E]WFIGEWIWMR[LMGLKSZIVRQIRXMWHSYFXJYPEFSYXSVHMWWEXMWƤIH[MXLXLIUYEPMX]

SJɄXLIɄVIWIEVGL8LMWHSIWRSXSVHMREVMP]NYWXMJ]HIPE]MRTYFPMGEXMSRXLIQSVIETTVSTVMEXIGSYVWI

MWɄJSVɄSTIRP]WXEXIHKVSYRHWSJHSYFXSVHMWEKVIIQIRXXSEGGSQTER]TYFPMGEXMSR

 54VSJIWWSV7MVɅ2MKIP7LEHFSPX61.'75

 Ȋ$VLJQLȴFDQWUHDVRQWRSXEOLVKRSHQO\LVWKDWȆPDQ\H\HVȇRQWKHbPDWHULDO

LVbRQHbZD\WRTXLFNO\KLJKOLJKWVKRUWFRPLQJVLQWKHZRUNLWVHOIȃ

RSHQbSXEOLFDWLRQbWHQGVWRLPSURYHWKHbTXDOLW\RIGDWDDQGbDQDO\VLVȋ

 !+(,/2155,%,.,6;1)$55(/%.,0*$)7..2,&674(1)*18(40/(064(5($4&++$5,0(.7&6$%.;%(&1/(

ERɄEWTIGXSJXLMWMRUYMV]8LMWVITSVXMRGPYHIWWSQIVIQIHMIW*MVWXLS[IZIVMXɄMWRIGIWWEV]

XSɄHIWGVMFI[LEXGSYRXWEWKSZIVRQIRXVIWIEVGLERHXSWOIXGLXLIVYPIWTVIWIRXP]KSZIVRMRK

MXɄ7IGXMSR
7IGXMSRHIWGVMFIWJIEXYVIWSJXLITVSFPIQSJHIPE]IHTYFPMGEXMSRJVSQIZMHIRGI

MRɄXLITYFPMGHSQEMRSVSFXEMRIHMRXLIGSYVWISJXLIMRUYMV]7IGXMSRHIWGVMFIWWSQIIRGSYVEKMRK

(:$/2.(51)*18(40/(06%1',(527%.,5+,0*4(5($4&+'(52,6(&+$..(0*(514&1064$',&6,1059,6+&744(06

TSPMG]8LIVITSVXGSRGPYHIW[MXLVIGSQQIRHEXMSRWJSVQEOMRKHIGMWMSRWEFSYXTYFPMGEXMSRQSVI

GSRWMWXIRXJEMVERHSTIRFIEVMRKMRQMRHXLEXTVSXSGSPERHTVEGXMGIWLSYPHRSXFIWYGLEWXSMRLMFMX

QMRMWXIVWERHGMZMPWIVZERXWJVSQGSQQMWWMSRMRKVIWIEVGLSRTSPMXMGEPP]HMJƤGYPXUYIWXMSRW

SECTION 1: IS THERE A PROBLEM? 	

The Promise of Evidence-Based Policymaking 2089
Section 2

THE SYSTEM 

OF CONDUCTING, 

COMMISSIONING 

AND PUBLISHING 

GOVERNMENT RESEARCH


 SECTION 2: THE SYSTEM OF CONDUCTING, COMMISSIONING AND PUBLISHING GOVERNMENT RESEARCH

2090 The Promise of Evidence-Based Policymaking
What constitutes government research?

 6IWIEVGLMRMXWFVSEHIWXWIRWIWMKRMƤIWER]EXXIQTXXSSFXEMRSVKEXLIVMRJSVQEXMSR%GMZMPWIVZERX

SVɄTSPMG]EHZMWIV[LSMWEWOIHXSGERZEWWSTMRMSREQSRKGSPPIEKYIWXSYRHIVXEOIELSVM^SRWGER

SVɄWMQTP]XS+SSKPIWSQIMRJSVQEXMSRMWHSMRKKSZIVRQIRXEPVIWIEVGL&YXVIWIEVGLSJXLMWOMRH

MWɄRSXɄXLIɄWYFNIGXSJXLMWVITSVX8LMWVITSVXMWEFSYXHMWGVIXILMKLPIZIPWXYHMIWHIWMKRIHXSIPMGMX

SVɅIWXEFPMWLMRJSVQEXMSRF]QIERWSJQIEWYVIQIRXXIWXMRKIZEPYEXMSRERHEREP]WMW!+,5,06740

MQTPMIWXLEXFSXLQIXLSHWERHVIWYPXWWLSYPHFISFNIGXMZIERHRIYXVEPTIVQMXXMRKMRXIVVSKEXMSRSJGPEMQW

ERHɄTSPMGMIWXS[LMGLXLI]VIPEXI;LMPIMRHMZMHYEPTVSHYGXWSJVIWIEVGLQE]JEPPWLSVXSJXLIWIWXERHEVHW

XLIVIMWɄEɄKIRIVEPI\TIGXEXMSRXLEXVIWIEVGL[MPPMRGVIEWIXLIWYQSJORS[PIHKI

 +SZIVRQIRXGSQQMWWMSRWWYGLVIWIEVGLMREZEVMIX]SJ[E]W&VSEHP]WTIEOMRKMREHHMXMSRXSɄVIWIEVGL

YRHIVXEOIR[MXLMRHITEVXQIRXWSJWXEXIERHŬEVQŭWPIRKXLŭEKIRGMIWVIWIEVGLQE]FIGEVVMIHSYXF]ɄTVMZEXI

EKIRGMIWSJEUYEWMKSZIVRQIRXEPGLEVEGXIVSVMRHITIRHIRXI\XIVREPFSHMIWWYGLEWɄYRMZIVWMXMIW-XɄMW[MXL

XLITYFPMGEXMSRSJVIWIEVGLYRHIVXEOIRF]ɄXLIWII\XIVREPFSHMIWXLEXXLMWVITSVXMWɄGSRGIVRIH

 8LIJSVIKSMRKMWJEVJVSQI\LEYWXMZIJSVMRWXERGII\XIVREPEKIRGMIWSVGSRWYPXERXWSVMRHMZMHYEP

$&$'(/,&5/$;%('4$)6(',061$55,569,6+'(2$46/(06$.4(5($4&+

 54EXVMGME,I[MXXJSVQIVWIGVIXEV]SJWXEXIJSVLIEPXLXSPHYW

Ȋ)RUPDQ\\HDUVQRZJRYHUQPHQWGHSDUWPHQWVKDYHEHHQRXWVRXUFLQJDbFRQVLGHUDEOH

DPRXQWRIZKDWXVHGWREHFRUH:KLWHKDOOIXQFWLRQV6RbDbFRQVXOWDQF\PD\EHbDVNHG

WRbUHYLHZWKHOLWHUDWXUHDQGRULQWHUQDWLRQDOH[SHULHQFHRQDSDUWLFXODULVVXH

WRbXQGHUWDNHVWDWLVWLFDODQDO\VLVRUHFRQRPLFPRGHOOLQJRISROLF\RSWLRQVWRUXQ

DbSXEOLFFRQVXOWDWLRQWRDQDO\VHWKHUHVXOWVRIbDbSXEOLFFRQVXOWDWLRQRUWRSURYLGH

SROLF\DGYLFH6RPHWLPHVWKHH[WHUQDOFRQVXOWDQF\XVHVSURSULHWDU\H[SHUWLVHDQGΖ3ȋ

 %PPSJXLMWYRHSYFXIHP]GVIEXIWHMJƤGYPXMIWSJɄXE\SRSQ]JSVɄEɄGSQTVILIRWMZIVIKMWXIVSJKSZIVRQIRX

VIWIEVGLERHMXMW[L]XLMWVITSVXJSGYWIWSRMRHITIRHIRXI\XIVREPVIWIEVGL[LIVIXLIFSYRHEVMIW

EVIVIEWSREFP]GPIEVERHXLIGEWIJSVVSYXMRIIEVP]HMWGPSWYVIMWɄWXVSRK&YXQYGLSJXLIVITSVXŭW

VIEWSRMRKERHQYGLSJ[LEXMXVIGSQQIRHWMRVIPEXMSRXSɄMRHITIRHIRXVIWIEVGL[MPPFIVIPIZERX

XSɄLS[EɄKSZIVRQIRXGSQQMXXIHXSSTIRRIWWHIGMHIWXSɄHIEP[MXLXLIHMWGPSWYVISJMRXIVREPVIWIEVGL

%WOMRKɄXLIɄTYFPMGXSVIP]SRMRJSVQEXMSRXLI]GERRSXWIIMWɄSFNIGXMSREFPIMRTVMRGMTPI[LEXIZIVXLIWSYVGI

SVɄREXYVISJXLIMRJSVQEXMSR

 +MZIRXLIGSQTPI\MX]SJXLIW]WXIQERHXLIPEGOSJVSFYWXHEXEIQIVKMRKJVSQXLIMRMXMEPWGSTMRKI\IVGMWI

XLIJSPPS[MRKVIQMX[EWEHSTXIHMRXLIGEPPJSVIZMHIRGIŰ8LIMRUYMV][MPPGSRWMHIVXLIɄTYFPMGEXMSR

SJɄVIWIEVGLMRGPYHMRKEREP]WIWSJSJƤGMEPWXEXMWXMGWMRMXMEXIHɄF]Ʉ;LMXILEPPHITEVXQIRXWERHXLIEVQŭW

PIRKXLFSHMIWXLEXVITSVXXSXLIQ;IɄEPWS[IPGSQIɄWYFQMWWMSRWVIPEXMRKXSHIZSPZIHEHQMRMWXVEXMSRW

FYXɄRSXXSɄPSGEPɄEYXLSVMXMIWű

 %PXLSYKLXLI3JƤGIJSV2EXMSREP7XEXMWXMGWŭ7GMIRGI)RKMRIIVMRKERHɅ8IGLRSPSK]ƤKYVIWXIPP

YWɅXLEXɅETTVS\MQEXIP]eFRMWWTIRXERRYEPP]SRɅKSZIVRQIRXVIWIEVGL

XLIVIMWRSVIPMEFPI

FVIEOHS[RSJ[LEXXLEXQSRI]MWWTIRXSRSV[LEXLETTIRWXSXLIWXYHMIW18(40/(06'1(5

RSXɄGSPPIGXXLMWMRJSVQEXMSRGIRXVEPP]8SHSWSMX[SYPHLEZIXSVIP]SRHITEVXQIRXEPHEXEFYXQER]

HITEVXQIRXWERHɄEVQŭWPIRKXLFSHMIWHSRSXGSPPIGXMRJSVQEXMSRGIRXVEPP]EFSYX[LEXVIWIEVGLXLI]LEZI

&1//,55,10('!($/59,6+,0'(2$46/(065$0'$4/>5.(0*6+%1',(5&$0&10'7&614&1//,55,106+(,4

S[RVIWIEVGL[MXLSYXXLIVIWXSJɄXLISVKERMWEXMSRORS[MRKEFSYXMX



(ITEVXQIRXJSV&YWMRIWW-RRSZEXMSR
7OMPPW


$VNLQJWKHSXEOLFWRUHO\

RQbLQIRUPDWLRQWKH\FDQQRW




ZKDWHYHUWKHVRXUFHRUQDWXUH

RIWKHLQIRUPDWLRQ



SECTION 2: THE SYSTEM OF CONDUCTING, COMMISSIONING AND PUBLISHING GOVERNMENT RESEARCH 

The Promise of Evidence-Based Policymaking 2091
	 %XXLIWXEVXSJXLMWMRUYMV]MX[EWEWWYQIHXLEXHITEVXQIRXEPGLMIJWGMIRXMƤGEHZMWIVW'7%W
[SYPH

&10&(406+(/5(.8(59,6+$..4(5($4&+,06+(,4'(2$46/(06$0'+$8($018(48,(91)9+$6+$22(0561,6

-XETTIEVWXLMWMWɄRSXEP[E]WXLIGEWI8LI+SZIVRQIRX3JƤGIJSV7GMIRGIŭWHSGYQIRX'LMIJɄ7GMIRXMƤG

%HZMWIVWERHɄXLIMVSJƤGMEPWERMRXVSHYGXMSR
WIXWSYXLS['7%WTVSZMHIWGMIRXMƤGEHZMGI

XSɄQMRMWXIVWERHLIPTTVITEVIXLIMVHITEVXQIRXŭWVIWIEVGLWXVEXIKMIWXLEXWIXFVSEHSFNIGXMZIW

8LI]ɄEPWSɄSZIVWIIVIWIEVGLMRXLIREXYVEPWGMIRGIW&YX[IɄJSYRHXLEXXLIɄ[E]'7%WGEVV]SYXXLIMVVSPI

XLIMVWIRMSVMX]ERHɄXLIMVEGGIWWXSQMRMWXIVWERHTIVQERIRXWIGVIXEVMIWZEVMIWFIX[IIRHITEVXQIRXW

'7%WLEZIEɄQMRMQEPVSPIMRSZIVWIIMRKWSGMEPWGMIRXMƤGVIWIEVGL[LMGLJSVQWEWMKRMƤGERXTVSTSVXMSR

SJVIWIEVGLMRɄQER]HITEVXQIRXWERHɄMWɄKSZIVRIHF]ɄXLIWITEVEXIKSZIVRQIRXWSGMEPVIWIEVGL

+76
ɄTVSJIWWMSR

Government research

 8LIW]WXIQSJKSZIVRQIRXVIWIEVGLMWZEWXGSQTPMGEXIHERHJVEKQIRXIH!+(	/,0,56(4,$.

HITEVXQIRXWMR;LMXILEPPERHXLILYRHVIHWSJTYFPMGFSHMIWERHEKIRGMIWXLEXVITSVXXSɄXLIQ

XEOIɄZIV]ɄHMJJIVIRXETTVSEGLIWXSGSRHYGXMRKGSQQMWWMSRMRKERHTYFPMWLMRKVIWIEVGL7SHS

XLIɄHIZSPZIHEHQMRMWXVEXMSRWSJ7GSXPERH2SVXLIVR-VIPERHERH;EPIW8LMWɄWIGXMSRWYQQEVMWIW

XLIɄW]WXIQXLIɄVIGSKRMXMSRMRI\MWXMRKKYMHERGIERHTVSXSGSPWXLEXɄƤRHMRKWWLSYPHFITYFPMWLIH

TVSQTXP]ERHɄ[LIXLIVXLMWVIUYMVIQIRXMW[IPPYRHIVWXSSHERHJSPPS[IH

 18(40/(06)70'54(5($4&+,06919$;56+417*+6+(5&,(0&(%7'*(6$'/,0,56(4('%;6+((2$46/(06

JSV&YWMRIWW-RRSZEXMSRERH7OMPPWERHQEMRP]HMWXVMFYXIHXLVSYKLVIWIEVGLGSYRGMPWERHYRMZIVWMXMIW


ERHXLVSYKLVIWIEVGLGSRHYGXIHSVGSQQMWWMSRIHF]ɄHITEVXQIRXWSJWXEXII\IGYXMZIEKIRGMIW

ERHɄEVQŭWPIRKXLFSHMIWMRXLIMVS[RTSPMG]ɄEVIEW

 %VERKISJVIWIEVGLIVWGEVVMIWSYXKSZIVRQIRXVIWIEVGLGMZMPWIVZERXW[SVOMRK[MXLMR;LMXILEPP

'(2$46/(0656+15(,0$4/>5.(0*6+%1',(5$0'(:(&76,8($*(0&,(5,0&.7',0*27%.,&5(&6144(5($4&+

FSHMIWMRHITIRHIRXVIWIEVGLIVWMRYRMZIVWMXMIWERHVIWIEVGLMRWXMXYXIWERHɄGSRWYPXERGMIW

 18(40/(06&10'7&654(5($4&+)141%-(&6,8(54700,0*)41/$55(55,0*6+((:6(061)0$674$.$0'51&,$.

241%.(/561,0)14/,0*6+('(8(.12/(061)21.,&;$0'(8$.7$6,0*,65,/2$&6

 8LI+SZIVRQIRX3JƤGIJSV7GMIRGII\TPEMRIHXSYWŰ%W[MXLVIWIEVGLMRER]SXLIVGSRXI\XXLIVI

MWɄEɄWTIGXVYQSJMRGVIEWMRKJSVQEPMX]VERKMRKJVSQFEWMGŬHIWOVIWIEVGLŭXLVSYKLQSVIJSVQEPP]

QEREKIHTVSNIGXWXSVIWIEVGL[LMGLMWWYFNIGXXSJSVQEPTIIVVIZMI[ű

 6IWIEVGLGSRHYGXIHJSVKSZIVRQIRXMWKSZIVRIHF]EZEVMIX]SJVYPIWTVMRGMTPIWKYMHERGIERHɄTVSXSGSPW

[LMGLYWIXLIMVS[RHIƤRMXMSRWSJ[LEXGSYRXWEWVIWIEVGLERH[LIRERHLS[MXWLSYPHFITYFPMWLIH

8LIWIEVIHMWGYWWIHMRTEVEKVETLW7IGXMSREHHVIWWIW[LIXLIVXLIWIHIƤRMXMSRWGVIEXI

YRGIVXEMRX]SZIV[LEXWLSYPHFITYFPMWLIHERHɄ[LIRERHQEOIWVIGSQQIRHEXMSRWJSVKVIEXIVGPEVMX]

Reasons for publishing government research

 4YFPMWLMRKKSZIVRQIRXVIWIEVGLJEGMPMXEXIWWGVYXMR]SJ[LEXMWFIMRKHSRISVTVSTSWIHSRɄFILEPJ

SJɄXLIɄTYFPMG8LMWQEXXIVWQSVIMRERIVE[LIRKSZIVRQIRXJVIUYIRXP]TSMRXWXSɄIZMHIRGI

XSɄNYWXMJ]ɄTSPMG]

 8LIVYPIWERHWXEXIQIRXWXLEXGSZIVKSZIVRQIRXVIWIEVGLEGORS[PIHKIXLEXVIWIEVGLJYRHIH

F]ɄXE\TE]IVWWLSYPHFIQEHITYFPMGP]EZEMPEFPI

 4YFPMGEXMSRSJKSZIVRQIRXGSQQMWWMSRIHWXYHMIWMWEQSRKSXLIVXLMRKWEQIGLERMWQJSVɄIRWYVMRK

XLEXɄHITEVXQIRXWORS[[LEXVIWIEVGLXLI]LEZIHSRIXLIQWIPZIWMRXLITEWXERHIREFPIWXLIQXSɄEZSMH

HYTPMGEXMRKXLIMVS[RVIWIEVGL%PXLSYKLSRIGERRSXIWXMQEXIXLIɄI\XIRXSJWYGLHYTPMGEXMSRFIGEYWI

SJɄXLIPEGOSJHEXEEFSYXVIWIEVGLGMZMPWIVZERXWWE]XLEXHITEVXQIRXWWTIRHWMKRMƤGERXXMQIXV]MRK

XSƤRHTEWXWXYHMIWXLEXXLI]GSRHYGXIHSVTEMHJSV7%.,5+,0**18(40/(064(5($4&+$.51(0$%.(5

HITEVXQIRXWXSPIEVRJVSQIEGLSXLIVŭWƤRHMRKW

 SECTION 2: THE SYSTEM OF CONDUCTING, COMMISSIONING AND PUBLISHING GOVERNMENT RESEARCH

2092 The Promise of Evidence-Based Policymaking
	 %RMQTSVXERXZEPYISJI\XIVREPVIWIEVGLMWXLEXMXVIHYGIWXLIVMWOXLEXKSZIVRQIRXVIWIEVGL[MPP

FIɄWPERXIHSVTIVGIMZIHXSFIWS


 *YPPTYFPMGEXMSRGEREPWSLIPTXSIRWYVIXLEXXLIIZMHIRGIJVSQKSZIVRQIRXVIWIEVGLMRJSVQW[MHIV

WGMIRXMƤGORS[PIHKIF]KMZMRKVIWIEVGLIVWEGGIWWXSEPEVKIVFSH]SJƤRHMRKW

 6IWIEVGLIVW[LSLEZII\TIVMIRGIHHIPE]WMRXLIMVVIWIEVGLFIMRKTYFPMWLIHXSPHXLMWMRUYMV]XLEX

MXɅLEWEJJIGXIHXLIMV[MPPMRKRIWWXSTYXXLIQWIPZIWJSV[EVHXSVYRKSZIVRQIRXGSQQMWWMSRIH

VIWIEVGLMRɅXLIɅJYXYVI!+(%$'27%.,&,6;$551&,$6('9,6+6+(&$5(5'(5&4,%(',0 (&6,10/$;+$8(

MRƥYIRGIHSXLIVWMRXLMWVIKEVH-XMWGPIEVP]RSXMRKSZIVRQIRXŭWMRXIVIWXSVMRXLITYFPMGMRXIVIWX

MJɄXLIVIɄMWɄEɄREVVS[MRKTSSPSJTISTPI[LSEVI[MPPMRKXS[SVOSRKSZIVRQIRXGSRXVEGXW

 ,66.(1401',55(06)41/6+(5(%41$'241215,6,1059$5(0&1706(4(',06+(&1745(1)6+,5,037,4;

8LIVIɄMWɄEKIRIVEPI\TIGXEXMSRXLEXXLIVIWYPXWSJKSZIVRQIRXVIWIEVGLWLSYPHFIɄQEHITYFPMGP]

ERHɄTVSQTXP]EZEMPEFPI

 *SVI\EQTPI%RHVI[1MPPIVJSVQIV14ERHGLEMVSJXLI,SYWISJ'SQQSRWWGMIRGIERHɅXIGLRSPSK]

WIPIGXGSQQMXXIIWEMH

Ȋ7KHJRDOVKRXOGEHDFXOWXUHZKHUHJRYHUQPHQWDLPVWRȆVXFNLQȇDVbPXFKJRRG

TXDOLW\DGYLFHDQGGDWD>DV@LWbFDQSXWWLQJDVPXFKDVbSRVVLEOHLQbWKHbSXEOLFGRPDLQ

ȃZLWKWKHGHIDXOWTXHVWLRQEHLQJȆZK\bVKRXOGQȇWZHSXEOLVKDOORIWKLV"ȋ

7MV:IROM6EQEOVMWLRERTVIWMHIRXSJɅXLI6S]EP7SGMIX] 5$,'

ȊΖWLVLPSRUWDQWWKDWJRYHUQPHQWLVWUDQVSDUHQWDERXWKRZSROLF\GHFLVLRQVKDYH

EHHQPDGH7RHQDEOHSROLF\PDNLQJWREHSURSHUO\VFUXWLQLVHGWKHbHYLGHQFH

RQbZKLFKLWLVPDGHVKRXOGEHDYDLODEOH7KLVbLQFOXGHVGDWDKHOGE\WKHJRYHUQPHQW

DQGbWKHbȴQGLQJVRIbFRPPLVVLRQHGUHVHDUFKȋ

(V7EVEL;SPPEWXSR14GLEMVSJXLI,SYWISJ'SQQSRWLIEPXLWIPIGXGSQQMXXIIWEMH

ȊΖIbUHVHDUFKLVZLWKKHOGKRZFDQWKHSXEOLFMXGJHZKHWKHULWUHȵHFWVVFLHQWLȴF

FRQVHQVXVZKHWKHULWȇVFRUUHFWDQGKRZLWOLQNVLIDWDOOWRbSROLF\GHFLVLRQV"

2IbFRXUVHbWKRVHGHFLVLRQVWDNHDFFRXQWRIRWKHUFRQVLGHUDWLRQVEXWWKHHYLGHQFH

EDVHIRUWKHPVKRXOGEHRSHQWRbVFUXWLQ\ȋ

Rules require the prompt and full publication of government research

 +SZIVRQIRXVIGSKRMWIWXLITYFPMGI\TIGXEXMSRSJTYFPMGEXMSRGSHMJ]MRKMXMRVYPIWVIUYMVMRKTVSQTX

ERHɄGSQTPIXITYFPMGEXMSRSJVIWIEVGLGSRHYGXIHSVGSQQMWWMSRIHF];LMXILEPPHITEVXQIRXW,S[IZIV

XLIWIVYPIWEVIRSXGPIEVP]HIƤRIHMRWSQIEVIEWXLI]EVIWXVSRKP]WXEXIHERHMRSXLIVW[IEOP]MQTPMIH

 18(40/(064(5($4&+,06+(0$674$.5&,(0&(5,5*18(40('%;6+(+,*+.(8(.24,0&,2.(55(6176

MRɄ4VMRGMTPIWɄSJWGMIRXMƤGEHZMGIXSKSZIVRQIRX+SZIVRQIRX3JƤGIJSV7GMIRGIE
*YVXLIV

KYMHIPMRIWWIXSYXLS[XLIWITVMRGMTPIWWLSYPHFIɄETTPMIHF]WGMIRXMƤGEHZMWSV]GSQQMXXIIW

ERHɄHITEVXQIRXW8LIWIɄMRGPYHI8LI+SZIVRQIRX'LMIJ7GMIRXMƤG%HZMWIVŭW+YMHIPMRIWSRXLI9WI

SJɄ7GMIRXMƤGERH)RKMRIIVMRK%HZMGIMR4SPMG]1EOMRK+SZIVRQIRX3JƤGIJSV7GMIRGIF


[LMGLɄWIXɄSYXŰLS[WGMIRXMƤGERHɄIRKMRIIVMRKEHZMGIWLSYPHFIWSYKLXERHETTPMIHűERH'LMIJ7GMIRXMƤG

%HZMWIVWERHɄXLIMVSJƤGMEPWERMRXVSHYGXMSR[LMGLŰWIXWSYXFVSEHKYMHIPMRIWJSVXLI[SVOSJGLMIJ

WGMIRXMƤGEHZMWIVWMRHITEVXQIRXWű-REHHMXMSRIEGLHITEVXQIRXWIXWEVIWIEVGLWXVEXIK]VIEJƤVQMRK

XLIɄGSQQMXQIRXXSTYFPMWLMRKMXWVIWIEVGLMRJYPP

 !+(27%.,&$6,101)*18(40/(0651&,$.5&,(0&(4(5($4&+,5&18(4('%;6+( 241)(55,10>527%.,&$6,10

TVSXSGSP+SZIVRQIRX)GSRSQMG
7SGMEP6IWIEVGL8IEQ


SECTION 2: THE SYSTEM OF CONDUCTING, COMMISSIONING AND PUBLISHING GOVERNMENT RESEARCH 

The Promise of Evidence-Based Policymaking 2093
Are rules requiring prompt publication well-understood and followed?

 8(46+(2$56)(9;($456+,5*41721)47.(5+$5(81.8(')41/*7,'$0&(,061&1'(51)24$&6,&(

ERHɄYPXMQEXIP]MRXSGMZMPWIVZMGIERHQMRMWXIVMEPTVSXSGSPWQEOMRKXLIQTVSKVIWWMZIP]QSVIW]WXIQEXMG

$.6+17*+6+(;4(/$,070(8(0.;$22.,('$&4155',))(4(06$4($51)4(5($4&+6,570'(45611'6+$66+( 

241)(55,1072*4$'(',6527%.,&$6,10*7,'$0&(61&,8,.5(48,&(24161&1.9+,&+/($056+$6,6&$0%(/14(

JSVQEPP]IRJSVGIH
JSPPS[MRKGSRGIVRWXLEXXLII\TIGXEXMSRSJSTIRTYFPMGEXMSR[EWFVIEOMRKHS[R

MRɄWSQIHITEVXQIRXW

 %WXLIWIVYPIWYPXMQEXIP]EGORS[PIHKI[LEXQEXXIVWJSVHIQSGVEXMGEGGSYRXEFMPMX]MWXMQIP]EGGIWW

XSɅVIWIEVGLXSJEGMPMXEXITYFPMGMRXIVEGXMSR[MXLTSPMG]-JXLIVYPIW[IVIJSPPS[IHMRIZIV]GEWI[SYPHXLI]

24(8(066+('(.$;5$0'9,6++1.',0*6+$6$4(',5&755(',06+()1..19,0*5(&6,105

 )ZMHIRGIWYFQMXXIHXSXLMWMRUYMV]F]VIWIEVGLIVWGMZMPWIVZERXWERHTEVPMEQIRXEVMERWWYKKIWXW

XLEXɄXLIɄW]WXIQSJKYMHIPMRIWERHVYPIWPIEZIWVSSQJSVYRGIVXEMRXMIWSZIV

 $9+(6+(4$2$46,&7.$44(5($4&+241-(&6,557%-(&66127%.,&$6,10

 F
MJWS[LIRERH

 G
[LMGLVYPIWXEOITVIGIHIRGI

 8LITVIWWYVISRTSPMG]QEOIVWXSEPMKRXLITYFPMGEXMSRSJIZMHIRGI[MXLTSPMG]EKIRHEWWLEVTIRW

XLIɄUYIWXMSRSJ[LIRWXYHMIWWLSYPHFITYFPMWLIHMRVIPEXMSRXSXLITSPMG]XLI]EVIMRXIRHIHXSMRJSVQ

(MJJIVIRXZMI[WSRLS[ERH[L]XLMWQEXXIVWERHVIGSQQIRHEXMSRWXSVIWSPZIXLIXIRWMSREVIWIXSYX

MRɄXLIJSPPS[MRKWIGXMSRW

	 *MVWXLS[IZIVMXQE]FILIPTJYPXSPSSOEXWSQITVSQMRIRXGEWIWMR[LMGLTYFPMGEXMSRSJVIWIEVGL

LEWɄFIIRLIPHFEGOMREREXXIQTXXSKEYKIXLIWMKRMƤGERGISJɄXLIɄTVSFPIQ


 SECTION 2: THE SYSTEM OF CONDUCTING, COMMISSIONING AND PUBLISHING GOVERNMENT RESEARCH

2094 The Promise of Evidence-Based Policymaking
Rules governing the publication of government research

6FLHQWLȴFDGYLVHUVDUHIUHHWRFRPPXQLFDWHSXEOLFO\WKHLUDGYLFHWRbJRYHUQPHQW

VXEMHFWWRbQRUPDOFRQȴGHQWLDOLW\UHVWULFWLRQVLQFOXGLQJZKHQLWDSSHDUV

WRbEHLQFRQVLVWHQWZLWKJRYHUQPHQWSROLF\VFLHQWLȴFbDGYLVHUVKDYHWKHULJKW

WRbHQJDJHZLWKWKHPHGLDDQGSXEOLFLQGHSHQGHQWO\RIWKHbJRYHUQPHQW

DQGbVKRXOGVHHNLQGHSHQGHQWPHGLDDGYLFHRQVXEVWDQWLYHSLHFHVRIZRUN

VFLHQWLȴFDGYLFHWRJRYHUQPHQWVKRXOGEHbPDGHSXEOLFO\DYDLODEOHXQOHVVWKHUH

DUHRYHUULGLQJUHDVRQVVXFKbDVbQDWLRQDOVHFXULW\RUbWKHbIDFLOLWDWLRQRIDFULPH

IRUQRWGRLQJVRJRYHUQPHQWVKRXOGSXEOLFO\H[SODLQWKHUHDVRQVIRUbSROLF\

GHFLVLRQVSDUWLFXODUO\ZKHQWKHGHFLVLRQLVQRWFRQVLVWHQWZLWKVFLHQWLȴFDGYLFH

DQGbLQbGRLQJVRVKRXOGbDFFXUDWHO\UHSUHVHQWWKHHYLGHQFHȋ

“

7KHUHZLOOEHSURPSWUHOHDVHRIDOOJRYHUQPHQWUHVHDUFKDQGDQDO\VLV

*RYHUQPHQWVRFLDOUHVHDUFKDQGbDQDO\VLVPXVWEHUHOHDVHGLQDZD\WKDW

SURPRWHVSXEOLFbWUXVWȋ

“

4VMRGMTPIWSJWGMIRXMέGEHZMGIXSKSZIVRQIRX


'HSDUWPHQWVDQGSROLF\PDNHUVVKRXOGȐDGRSWDQRSHQDQGWUDQVSDUHQW

DSSURDFKWRWKHVFLHQWLȴFDGYLVRU\ SURFHVVDQGSXEOLVKWKHHYLGHQFH

DQGbDQDO\VLVDVVRRQDVSRVVLEOHȋ

“

'LMIJ7GMIRXMέG%HZMWIVWERHXLIMVSJέGMEPWERMRXVSHYGXMSR


4YFPMWLMRKVIWIEVGLERHEREP]WMWMRKSZIVRQIRX+764YFPMGEXMSR4VSXSGSP


DGRSWDQRSHQDQGWUDQVSDUHQWDSSURDFKWRWKHVFLHQWLȴFDGYLVRU\ SURFHVV

DQGSXEOLVKWKHbHYLGHQFHDQGbDQDO\VLVDVVRRQDVSRVVLEOHDQGH[SODLQ

SXEOLFO\WKHUHDVRQVIRUSROLF\GHFLVLRQVSDUWLFXODUO\ZKHQbWKHGHFLVLRQ

DSSHDUVWRbEHLQFRQVLVWHQWZLWKVFLHQWLȴFDGYLFHȋ

“

8LI+SZIVRQIRX'LMIJ7GMIRXMέG%HZMWIVͱW+YMHIPMRIWSRXLI9WISJ7GMIRXMέGERH)RKMRIIVMRK%HZMGI

MRc4SPMG]1EOMRK
(ITEVXQIRXWERHTSPMG]QEOIVW[MXLMRXLIQWLSYPH

SECTION 2: THE SYSTEM OF CONDUCTING, COMMISSIONING AND PUBLISHING GOVERNMENT RESEARCH 

The Promise of Evidence-Based Policymaking 2095
Section 3

REASONS FOR THE 

DELAYED PUBLICATION 

OR WITHHOLDING OF 

GOVERNMENT RESEARCH 

 SECTION HEADING SECTION 3: REASONS FOR THE DELAYED PUBLICATION OR WITHHOLDING OF GOVERNMENT RESEARCH

2096 The Promise of Evidence-Based Policymaking
What are the scale and significance of the problem?

 1ER]KSZIVRQIRXHITEVXQIRXWHSRSXORS[ERHGSYPHRSXXIPPYWLS[QER]VIWIEVGLWXYHMIW

XLI]ɅLEZIGSQQMWWMSRIHSV[LMGLSJXLIQLEZIFIIRTYFPMWLIH(.19$4(6+(4(57.651)$66(/265

XSɄƤPPXLIKETW

 *SYVSYXSJHITEVXQIRXWERHXLIX[SI\IGYXMZIEKIRGMIWXLEX[IVIEWOIHXLI)RZMVSRQIRX

%KIRG]ERH4YFPMG,IEPXL)RKPERH
TVSZMHIHJYPPPMWXWSJWXYHMIWXLI]LEHGEVVMIHSYXXLIQWIPZIW

SVɄGSQQMWWMSRIHJVSQSYXWMHII\TIVXW8IRHITEVXQIRXWVITPMIHXLEXMX[SYPHFIXSSXMQIGSRWYQMRK

ERHɄGSWXP]XSƤRHXLMWMRJSVQEXMSRWSXLEXYRHIV7IGXMSRSJXLI*VIIHSQSJ-RJSVQEXMSR%GX

XLI]Ʉ[IVIɄRSXVIUYMVIHXSVITP]7II8EFPISRTEKIJSVHIXEMPWSJXLIHITEVXQIRXW

 8LI(IJIRGI7GMIRGIERH8IGLRSPSK]0EFSVEXSV]XLI1MRMWXV]SJ(IJIRGIŭWVIWIEVGLEVQVIWTSRHIH

XLEXMXGSYPHXEOIYTXS[SVOMRKHE]WXSƤRHXLIVIPIZERXHIXEMPWWMRGIMXLEWVSYKLP]VIWIEVGL

WXYHMIWVYRRMRKMRER]SRIƤRERGMEP]IEV7IZIVEPSXLIVHITEVXQIRXWHMVIGXIHXLIMRUYMV]XSTYFPMGP]

EGGIWWMFPIPMWXWSVHEXEFEWIWSJVIWIEVGLWXYHMIWFYX[IVIRSXTVITEVIHEKEMRF]ZMVXYISJ7IGXMSR


XSɄWE][LIXLIVXLIWIMRGPYHIHEPPWXYHMIWXLI]LEHGSQQMWWMSRIH

 8LEXQER]KSZIVRQIRXHITEVXQIRXWHSRSXORS[SVGERRSXXIPPYW[LEXVIWIEVGLXLI]LEZI

&1//,55,10(',5)47564$6,0*)146+(27%.,&$0')142$4.,$/(06$4,$05 7%/,55,105616+,5,037,4;$.51

VEMWIHGSRGIVRWEFSYXIJJIGXWSRXLIIJƤGMIRG]SJKSZIVRQIRX*SVI\EQTPIWSQIHITEVXQIRXWYRWYVI

EFSYX[SVOXLEXLEWFIIRHSRIMRXLITEWXLEZIXSWTIRHWMKRMƤGERXXMQIERHQSRI]MRZIWXMKEXMRK

XLIMVS[RVIWIEVGLLMWXSV]FIJSVIYRHIVXEOMRKRI[TVSNIGXW!+(4(,5$.51.,66.(12214670,6;

JSVɄHITEVXQIRXWXSɄPIEVRJVSQIEGLSXLIVŭWVIWIEVGLFIGEYWIXLIVIEVIRSWXERHEVHMWIHVIGSVHW

SJɄ[LEXɄLEWXEOIRTPEGI

The hunt for research

-REHHMXMSRXSXLIPEGOSJWXERHEVHMWIHERHGSQTVILIRWMZIVIGSVHOIITMRKMXWIIQWXLEXKSZIVRQIRXIQTPS]IIWIRGSYRXIV

WMKRMƤGERXTVSFPIQW[MXLƤRHMRKWYGLQEXIVMEPEWQE]FIEZEMPEFPIXLVSYKLKSZIVRQIRXŭWS[R[IFWMXIERHIPIGXVSRMG

VIGSVHW8LMWMRUYMV]HMHRSXƤRHER]TSWMXMZIVIJIVIRGIXSXLIMRJSVQEXMSREZEMPEFPISRKSZYO;IEWOIHGYVVIRXERHJSVQIV

GMZMPWIVZERXWEW[IPPEWVIWIEVGLIVW[LSGEVVMIHSYXKSZIVRQIRXGSQQMWWMSRIHVIWIEVGLLS[IEW]MXMWXSƤRHVIWIEVGL

KSZIVRQIRXLEWGSQQMWWMSRIHMRXLITEWX



7IGXMSRSJXLI*VIIHSQSJ-RJSVQEXMSR%GXI\IQTXWEYXLSVMXMIWJVSQTVSZMHMRKMRJSVQEXMSRMJXLIGSWXSJHSMRK

WSɅI\GIIHWERETTVSTVMEXIPMQMX*SVQMRMWXIVMEPHITEVXQIRXWXLMWPMQMXMWKIRIVEPP]WIXEXe

2ɝFLDOVȴQGLWKDUGWRȴQG

ZRUNWKHLURZQGHSDUWPHQWGLG

SUHYLRXVO\DVKDSSHQHGZLWK

UHVHDUFKRQbELRIXHOV”

7REHKRQHVWLWZRXOGQHYHURFFXUWRPHWRWU\>WRȴQG

UHVHDUFKRQJRYXN@ȃΖZRXOGDOZD\VDVNVRPHRQHΖNQHZLQ

WKHUHOHYDQWDUHDȋ $QGLI\RXGLGQȇWNQRZVRPHRQHLQbWKHDUHD"

ȊΖQP\FDVHΖZRXOGHLWKHUULQJVRPHRQHLQWKDWGHSDUWPHQW

ZKRΖNQRZRUbDVVXPHLWGRHVQRWH[LVW”

5HSRUWVFDQEHKLGGHQRQREVFXUHVHUYHUVPDNLQJ

ȴQGLQJVPRUHUHDGLO\DYDLODEOHRQIRUbLQVWDQFH

5$1'bZHEVLWHWKDQRQJRYXN7KLVOHDYHVFLYLO

VHUYDQWVXQDEOHWRȴQGWKHLURZQZRUN”

ΖI \RXZDQWWRȴQG

RXWZKDWJRYHUQPHQW

LVbGRLQJ*RRJOHLW”

“

“

“

“

8LIWIGSQQIRXW[IVIVIGSVHIHRSREXXVMFYXEFP]MRXLIGSYVWISJXLIMRUYMV]

SECTION 3: REASONS FOR THE DELAYED PUBLICATION OR WITHHOLDING OF GOVERNMENT RESEARCH 

The Promise of Evidence-Based Policymaking 2097
 The creation of ghost research

 -RXLSWIHITEVXQIRXW[LIVIXLIVIMWRSQIGLERMWQXSƤRHSVXVEGOVIWIEVGLXLIVIMWEVMWOXLEXSJƤGMEPW

[MPPRSXFIEFPIXSƤRHVIWIEVGLEXEPP-XQE]FIXLEXSRP]XLISJƤGMEPW[LSGSQQMWWMSRIHXLIVIWIEVGL

ORS[EFSYXMX3RGISJƤGMEPWPIEZIXLIMVTSWXXLIORS[PIHKIMWPSWXERHMJXLIVIWIEVGLLEWRSXFIIR

TYFPMWLIHMREREGGIWWMFPIFSH]SJPMXIVEXYVIMXPIEZIWRSVIGSVHERHMXWVIWYPXW[MPPRIZIVFIYWIH

-XɄFIGSQIWKLSWXVIWIEVGL

	 8LIPEGOSJEJYPPEGGSYRXSJ[LEXLETTIRWXSKSZIVRQIRXVIWIEVGLQEOIWMXHMJƤGYPXXSEWWIWW

JVSQɄXLIɄORS[RGEWIWSJHIPE]LS[QYGLXLIW]WXIQERHTVSGIHYVIGSRXVMFYXI

 5(EZMH;EPOIVLIEHɅSJɅTSPMG]EXXLI%GEHIQ]SJ7SGMEP7GMIRGIWWEMH

Ȋ(YLGHQFHKDVWREHPRUHWKDQDQHFGRWDOIRUbWKHbSUREOHP>RIUHVHDUFKEHLQJ

ZLWKKHOG@WREHFRQVLGHUHGV\VWHPLFȋ

 !+(0((')1447.(5$0'*7,'(.,0(524,14,6,5,0*($4.;27%.,&$6,10/$;,0)$&6,0',&$6(4(&1*0,6,10

SJɄTVIWWYVIW[MXLMRXLIW]WXIQXSHSXLISTTSWMXI;IRSXIXLEXTVSGIHYVIWJSVXLITYFPMGEXMSR

SJɄKSZIVRQIRXWSGMEPVIWIEVGL[IVIVIGIRXP]YTKVEHIHJVSQKYMHERGIXSTVSXSGSP

 4VSJIWWSV(EQI7EPP](EZMIWGLMIJQIHMGEPSJƤGIVJSV)RKPERHXSPHXLIMRUYMV]XLEXXLIW]WXIQW

MRɄTPEGIRS[WYTTSVXɄTYFPMGEXMSR

Ȋ$OWKRXJKDGHFDGHRUbPRUHDJRWKHUHPD\KDYHEHHQPRUHRIDSUREOHPZLWK

UHVHDUFKEHLQJGHOD\HGFOHDUHUJXLGDQFHDQGbSXEOLFDWLRQIUDPHZRUNVLQSODFHWRGD\

PHDQWKHUHLVQȇWDPDMRUSUREOHPDQ\PRUHȋ

 8LMWVITSVXPSSOWEXWSQITVSQMRIRXI\EQTPIWSJHIPE]IHTYFPMGEXMSRXLEXLEZIGSQIXSPMKLXGLMIƥ]

XLVSYKLQIHMEVITSVXW&YXIZMHIRGIWYFQMXXIHXSXLIMRUYMV]MRGPYHMRKXLEXSRXLITYFPMGEXMSRSJTSPMG]

XVMEPWERHWXEXMWXMGEPEREP]WMWWYKKIWXWXLEXHIPE]WQE]FIQSVIJVIUYIRX

)H,YQTLIVWSRLIEHSJEWWIWWQIRX9/7XEXMWXMGW%YXLSVMX]WEMH

Ȋ8.6$ȴQGLWXQDFFHSWDEOHIRU>SXEOLF@FODLPVWREHPDGHZLWKRXWHYHU\RQHKDYLQJ

DFFHVVWRWKHDQDO\VLVEHKLQGWKHPVRZHSUHVVIRUSXEOLFDWLRQ7KDWZHKDYH

WRbGRbWKLVUHSHDWHGO\PHDQVWKDWWKHSULQFLSOHVRISURPSWSXEOLFDWLRQIRXQGLQFRGHV

JRYHUQLQJUHVHDUFKPLJKWQRWEHWKDWVWURQJO\HPEHGGHGȋ

 !+(.(8(.1)',514*$0,5$6,10'(5&4,%('$%18(/$;9(..%(&1064,%76,0*6152(&7.$6,106+$627%.,&$6,10

SJVIWIEVGLMWVSYXMRIP]HIPE]IHJSVTSPMXMGEPVIEWSRW-JWSXLMWMWFSXLYRRIGIWWEV]ERHLEVQJYP

XSɄKSZIVRQIRXWMRGIMXMWPMOIP]XLEXXLIQENSVMX]SJI\XIVREPVIWIEVGLWXYHMIWEVITYFPMWLIH[MXLSYX

70'7('(.$;

 %PXLSYKLW]WXIQEXMGETTVEMWEPSJTYFPMGEXMSRTEXXIVRWMWGYVVIRXP]MQTSWWMFPIMXMWTSWWMFPIXSEWWIWW

XLIGMVGYQWXERGIWMR[LMGLORS[RHIPE]WLEZISGGYVVIH'EWIWSJHIPE]IHTYFPMGEXMSRERHIZMHIRGI

WYFQMXXIHXSXLMWMRUYMV]WYKKIWXXLEXKSZIVRQIRXVIWIEVGLQE]FILIPHFEGOJSVVIEWSRWMRGPYHMRK

@XLIHIWMVIXSEPMKRTYFPMGEXMSR[MXLXLIERRSYRGIQIRXSJTSPMG]VIWYPXMRKMRHIPE]IHTYFPMGEXMSR

SJɄƤRHMRKWXLEXEVITSPMXMGEPP]MRGSRZIRMIRX

@70&(46$,06;18(4+192((44(8,(9$0'',5&755,105$%1764(5($4&+37$.,6;$0'8$.,',6;5+17.'

FIɄLERHPIHERH

@YRGIVXEMRX]SZIV[LEXGSYRXWEWKSZIVRQIRXVIWIEVGLERHHMJJIVIRGIWMRXLIMRXIVTVIXEXMSR

SJɄXLIɄVYPIW

 !+(6+140,(56241%.(/,56+($'-756/(061)6+(6,/,0*1)27%.,&$6,10)1421.,6,&$.4($5105

 SECTION 3: REASONS FOR THE DELAYED PUBLICATION OR WITHHOLDING OF GOVERNMENT RESEARCH

2098 The Promise of Evidence-Based Policymaking
Attorney General's Office 1(5016+1.'6+(,0)14/$6,104(37(56('

Cabinet Office 8SSGSWXP]XSTVSZMHIXLIMRJSVQEXMSR

EWMXMWLIPHMRɄQER]ɄHMJJIVIRXƤPIWERHPSGEXMSRW

Department for Business, Innovation and Skills 8SSGSWXP]XSTVSZMHIXLIMRJSVQEXMSR

EWMXMWLIPHMRɄQER]ɄHMJJIVIRXƤPIWERHPSGEXMSRW

Department for Communities and Local Government 418,'('$.,561)$..&1//,55,10('4(5($4&+

Department for Culture, Media and Sport 418,'('$.,561)$..&1//,55,10('4(5($4&+

Department for Education 0MROIHXSERI\MWXMRKPMWXSJEPPTYFPMWLIHVITSVXW

Department of Energy and Climate Change 8SSGSWXP]XSTVSZMHIXLIMRJSVQEXMSR

EWMXMWLIPHMRɄQER]ɄHMJJIVIRXƤPIWERHPSGEXMSRW

Department for International Development 0MROIHXSTYFPMGP]EGGIWWMFPIHEXEFEWI

1)$..&1//,55,10('4(5($4&+

Department for Transport $5$27%.,&.;$&&(55,%.('$6$%$5(1)$..&1//,55,10('

4(5($4&+611&156.;612418,'(,0)14/$6,10$%176

GSQTPIXIRIWWEWMXMWLIPHMRQER]HMJJIVIRXƤPIW

ERHɄPSGEXMSRW

Department for Work and Pensions 0MROIHXSERI\MWXMRKPMWXSJEPPTYFPMWLIHVITSVXW

XSSɄGSWXP]XSTVSZMHIMRJSVQEXMSREFSYXGSQTPIXIRIWW

EWɄMXɄMWLIPHMRQER]HMJJIVIRXƤPIWERHPSGEXMSRW

Department for Environment, Food and Rural Affairs 0MROIHXSTYFPMGP]EGGIWWMFPIHEXEFEWI

1)$..&1//,55,10('4(5($4&+

Department of Health 0MROIHXSTYFPMGP]EGGIWWMFPIHEXEFEWI

1)$..&1//,55,10('4(5($4&+

Environment Agency 418,'('$.,561)$..&1//,55,10('4(5($4&+

Foreign Office 8SSGSWXP]XSTVSZMHIXLIMRJSVQEXMSR

EWMXMWLIPHMRQER]HMJJIVIRXƤPIWERHPSGEXMSRW

Home Office 8SSGSWXP]XSTVSZMHIXLIMRJSVQEXMSR

EWMXMWLIPHMRQER]HMJJIVIRXƤPIWERHPSGEXMSRW

Ministry of Defence 8SSGSWXP]XSTVSZMHIXLIMRJSVQEXMSR

EWMXMWLIPHMRQER]HMJJIVIRXƤPIWERHPSGEXMSRW

Ministry of Justice 418,'('$.,561)$..&1//,55,10('4(5($4&+

Northern Ireland Office 1(5016+1.'6+(,0)14/$6,104(37(56('

Office of the Advocate General for Scotland ,'016&$44;17614&1//,55,104(5($4&+

,06+(2(4,1',037,4('$%176

Office of the Leader of the House of Commons 1(5016+1.'6+(,0)14/$6,104(37(56('

Office of the Leader of the House of Lords 1(5016+1.'6+(,0)14/$6,104(37(56('

Public Health England 418,'('$.,561)$..&1//,55,10('4(5($4&+

Scotland Office ,'016&$44;17614&1//,55,104(5($4&+

,06+(2(4,1',037,4('$%176

Treasury 8SSGSWXP]XSTVSZMHIXLIMRJSVQEXMSR

EWMXMWLIPHMRQER]HMJJIVIRXƤPIWERHPSGEXMSRW

UK Export Finance 418,'('$.,561)$..&1//,55,10('4(5($4&+

Wales Office ,'016&$44;17614&1//,55,104(5($4&+

,06+(2(4,1',037,4('$%176

Table 1: Details of responses to Freedom of Information requests

SECTION 3: REASONS FOR THE DELAYED PUBLICATION OR WITHHOLDING OF GOVERNMENT RESEARCH 

The Promise of Evidence-Based Policymaking 2099
The politics of publication

The desire to align publication with the announcement of policy

 !+(4(,5*11'(8,'(0&(1)24(5574(561$.,*027%.,&$6,101)4(5($4&+%16+9,6+*18(40/(06>5

%41$'21.,&;$*(0'$5$0'9,6+2$46,&7.$421.,&,(54(.$6,0*616+((8,'(0&(,65(.)

 *MSRE*S\GLMIJI\IGYXMZISJɅXLI7GMIRGI1IHME'IRXVI WTSOISJXLIVSPISJ8LI+VMHEW]WXIQ

[LIVIF]WTIGMEPEHZMWIVWMR2YQFIVGERQEREKI[IIOP]KSZIVRQIRXERRSYRGIQIRXWEGVSWW

$..2$4651)*18(40/(06

Ȋ1RbUHSRUWbRUbSXEOLFDWLRQFDQEHUHOHDVHGZLWKRXWDȆVORWȇRQWKH*ULGȐ

7KHUHbDUHPXOWLSOHGHOD\VDQGbWKHbWLPLQJLVRIWHQSROLWLFLVHGZHZHUHWROG

WKDWRQHUHSRUWRQDQLPDOUHVHDUFKZDVGHOD\HGEHFDXVH1XPEHUbVSHFLDO

DGYLVHUVIHOWLWZDVXQIRUWXQDWHWLPLQJEHFDXVHLWFODVKHGZLWKWKHSULPH

PLQLVWHUȇVȆ$O]KHLPHUȇV:HHNȇ$QRWKHUSDJHUHSRUWRQPHQWDOKHDOWK

ZDVGHOD\HGEHFDXVHLWFRQWDLQHGWZRbSDJHVRQOLQNVEHWZHHQWKHUHFHVVLRQ

DQGbGHSUHVVLRQZKLFKFODVKHGZLWKDPDMRUVSHHFKRQbWKHbHFRQRP\ȋ

 !+(4($4($6.($566918,(9510+196+(6,/,0*1)27%.,&$6,105+17.'4(.$6(616+($00170&(/(06

SJɄTEVXMGYPEVKSZIVRQIRXTSPMGMIW3RIZMI[MWXLEXVIWIEVGLWLSYPHFITYFPMWLIH[LIR

MXɄMWɄGSQTPIXIERHMREHZERGISJXLIVIPEXIHTSPMG]WSXLEX4EVPMEQIRXI\TIVXWERHXLITYFPMG

GERɄGSRWMHIVXLIIZMHIRGIFIJSVIXLITSPMG]MWERRSYRGIH

4VSJIWWSV(EQI7EPP](EZMIWWEMH

Ȋ7KHUHȇVDbGL΍HUHQFHEHWZHHQH[WHUQDOO\FRPPLVVLRQHGUHVHDUFKDQGWKDW

GRQHGLUHFWO\E\FLYLOVHUYDQWV:LWKWKHbODWWHULWȇVULJKWWKDWGHSDUWPHQWVFDQ

DOLJQWKHbWLPLQJRISXEOLFDWLRQWRZLWKLQZHHNVRIDbSROLF\DQQRXQFHPHQW

EXWbLIbWKHbSROLF\DQQRXQFHPHQWLVWRWDNHDORWORQJHUWKH\VKRXOGSXEOLVK

UHVHDUFKDVVRRQDVSRVVLEOHHYHQLIWKDWȇVLQDGYDQFHRIWKHSROLF\

>$VbIDUbDVbWKH'HSDUWPHQWRIb+HDOWKLVFRQFHUQHGWKH@WLPLQJRIH[WHUQDOO\

FRPPLVVLRQHGUHVHDUFKLVDOPRVWZKROO\DbPDWWHUIRUWKHUHVHDUFKHUVȋ

 %REPXIVREXMZIZMI[MWXLEXMXMWNYWXMƤEFPIXSLSPHFEGOTYFPMGEXMSRSJVIWIEVGLERHVIPIEWI

MXɄXSKIXLIV[MXLXLITSPMG]

 2MGSPE&PEGO[SSH14GLEMVSJXLI,SYWISJ'SQQSRWWGMIRGIERHɅXIGLRSPSK]WIPIGX

	

XSPHYW

Ȋ2IWHQWKHUHȇVYDOXHWRUHOHDVLQJUHVHDUFKDWbWKHbVDPHbWLPHDVSROLF\5HOHDVLQJ

ȴQGLQJVWKDWFULWLFLVHJRYHUQPHQWSROLF\ZLWKRXWGHWDLOVRIbKRZJRYHUQPHQW

LVbGHDOLQJZLWKWKHFULWLFLVPFRXOGGDPDJHFRQȴGHQFHZKHUHWKHbUHVHDUFK

KDVLPPHGLDWHLPSOLFDWLRQVIRUSXEOLFVDIHW\7KLVPDNHVLWPRUHLPSRUWDQW

WKDWGHOD\VDUHbH[SODLQHGDQGFODULȴHGLVJRYHUQPHQWSUHSDULQJLWVUHVSRQVH

WRbFRPSOH[ȴQGLQJVZHUHbWKHȴQGLQJVWKHPVHOYHVGLVOLNHGDQGZKHUHGRHV

WKHUHVHDUFKVLWLQUHODWLRQWRRWKHUSROLF\DQQRXQFHPHQWVLQbWKHJRYHUQPHQWȇV

FRPPXQLFDWLRQVȆJULGȇ"ȋ

 -XWIIQWVIEWSREFPIXLEXYRPIWWTYFPMGWEJIX]MWIRHERKIVIHVIWIEVGLWLSYPHFITYFPMWLIH[LIR

MXMWGSQTPIXI;LIVIHIPE]WSGGYVXLI]WLSYPHFIWEXMWJEGXSVMP]I\TPEMRIH8LMWMWRSQSVIXLER

6+(47.(5*18(40,0*'(2$46/(06$.4(5($4&+4(37,4(!+(,/2146$0&(1)6,/(.;$&&(5561(8,'(0&(

MWEGORS[PIHKIHF]XLIJVIUYIRXQIRXMSRSJTVSQTXTYFPMGEXMSRMRXLIWIVYPIW



:KHUHGHOD\V

RFFXUWKH\VKRXOG

	

H[SODLQHG

 SECTION 3: REASONS FOR THE DELAYED PUBLICATION OR WITHHOLDING OF GOVERNMENT RESEARCH

2100 The Promise of Evidence-Based Policymaking
	 -QTPMGMXMREXXIQTXWXSGSRXVSPXLIXMQMRKSJVIWIEVGLTYFPMGEXMSRMWXLIMHIEXLEXXLMWMWXSXLIFIRIƤX

SJɄXLIɄTSPMG],S[IZIVMRWXERGIWSJHIPE]IHTYFPMGEXMSRMRGPYHMRKVIWIEVGLMRXSQMRMQYQEPGSLSPɄTVMGMRK

'EWIɄ7XYH]Ʉ
ERH[E]WXSXEGOPIWYKEVGSRWYQTXMSR'EWI7XYH]
MRɄJEGXɄMPPYWXVEXIXLILEVQ

XLEXɄGERɄF]HSRIF]ɄHIPE]MRKTYFPMGEXMSRXSEPMKR[MXLXLIERRSYRGIQIRXSJTSPMG]


 -RFSXLXLIWIGEWIWTYFPMGEXMSRSJVIWIEVGL[EWHIPE]IHXSGSMRGMHI[MXLXLIERRSYRGIQIRX

SJɄKSZIVRQIRXTSPMG]

 (V7EVEL;SPPEWXSR1461.'75

Ȋ'HOD\>LQSXEOLVKLQJUHVHDUFKLQWRVXJDUFRQVXPSWLRQ@LVFOHDUO\LQH΍HFWDVEDG

DVbQRQSXEOLFDWLRQDVWKHVFLHQWLȴFFRPPXQLW\GRQȇWJHWWRbDQDO\VHWKHHYLGHQFH

EDVHbRQZKLFKJRYHUQPHQWLVEDVLQJLWVVWUDWHJ\>WRUHGXFHFKLOGKRRGREHVLW\@ȋ

 !+(4($4($07/%(41)16+(44($51059+;27%.,&$6,101)4(5($4&+/$;%('(.$;('4(2146,061+19

6+()11'5722.;%(&$/(&106$/,0$6('9,6++145(/($6$5( 67';$22($4561+$8(%((0'(.$;('

TEVXP]FIGEYWISJJIEVSJLS[XLITYFPMGQMKLXVIEGXXSXLIƤRHMRKWERHTEVXP]MRSVHIVXSEGGSQQSHEXI

&+$0*(5,0&$%,0(69+,&+4(37,4('/14(6,/()14/,0,56(456124(2$4($4(52105(

Delayed publication of findings that are politically inconvenient 

 8LI(ITEVXQIRXJSV)RZMVSRQIRX*SSHERH6YVEP%JJEMVW(IJVE
I\TPEMRIHHIPE]MRXLITYFPMGEXMSR

SJɄJSSHFEROWVIWIEVGLMR'EWI7XYH]
EWVIWYPXMRKJVSQXLITIIVVIZMI[TVSGIWW

,S[IZIVɄXLIɄEYXLSVWSJXLIWXYH]WEMHXLEXXLIMRMXMEPTIIVVIZMI[[EWTSWMXMZIERHXLEXGSRGIVRW

[IVIVEMWIHWYFWIUYIRXP]EFSYXLS[XLIƤRHMRKW[SYPHMQTEGXSRTSPMG]-RXLIGEWISJVIWIEVGL

MRXSɄMQQMKVEXMSRERHXLIPEFSYVQEVOIX'EWI7XYH]
MXETTIEVWXLEXKSZIVRQIRX[EWLETT]

XSɄTYFPMWLTVIZMSYWVIWIEVGLXLEXWYTTSVXIH[LEXQMRMWXIVWLEHFIIRWE]MRKFYXLIPHFEGOEREP]WMW

XLEXGLEPPIRKIHMX-RXLIGEWISJEWXYH]MRXSHVYKWTSPMG]MRSXLIVGSYRXVMIW'EWI7XYH]
XLMWMRUYMV]

[EWɄYREFPIXSƤRHER]VIEWSRWSXLIVXLERTSPMXMGEPSRIWJSV[L]TYFPMGEXMSR[EWHIPE]IH

 -REPPSJXLIWIMRWXERGIWXLIHIPE]WMRTYFPMGEXMSRETTIEVXSLEZIFIIRGEYWIHXSZEV]MRKHIKVIIW

F]ɄTSPMXMGEPGSRGIVRWEFSYXXLIMQTPMGEXMSRWSJXLIVIWIEVGL8LMWMWRSXGSQQIRWYVEXI[MXL

24,0&,2.(5(,6+(41),0'(2(0'(064(5($4&+141)'(/1&4$6,&'(%$6(65+17.'$.51%(016('6+$6

RSRIɄSJɄXLIɄKYMHERGISRTYFPMGEXMSRSJVIWIEVGLVIGSKRMWIWXLMWEWEZEPMHVIEWSRJSVHIPE]

 8LIVIGPIEVP]EVIERHEP[E]W[MPPFITSPMXMGEPTVIWWYVIWXSGSRXVSPXMQMRKERHMRJSVQEXMSR

7SQIJIEVXLEXɄEXXIQTXWXSVIHYGIXLMWTVIWWYVIGSYPHPIEHXSPIWWVIWIEVGLFIMRKGSQQMWWMSRIH

 *SVMRWXERGI.SREXLER4SVXIW2EXMSREP-RWXMXYXISJ)GSRSQMGERH7SGMEP6IWIEVGLWEMH

 Ȋ7KHUHLVDQDEVROXWHO\LQVXSHUDEOHGLOHPPDZLWKWKHSUHVHQWV\VWHP7KHPRUH

WKHUXOHVDUHFKDQJHGWRDYRLGWKHREYLRXVFXUUHQWSUREOHPVSROLWLFDOO\PRWLYDWHG

GHOD\VHQGOHVVUHGUDIWLQJHWFWKHPRUHWKHLQFHQWLYHVRQPLQLVWHUVZLOOEHVLPSO\

QRWbWRFRPPLVVLRQLQGHSHQGHQWUHVHDUFKȋ

 3XLIVWVEMWIHWMQMPEVUYIWXMSRW8LIɄTSWWMFMPMX]SJXLMWYRMRXIRHIHGSRWIUYIRGIMWGSRWMHIVIH

MRɄHMWGYWWMSRSJɄXLIɄVIGSQQIRHEXMSRWMR7IGXMSR[LIVIMXMWRSXIHXLEXHITEVXQIRXW[MXL

VIWIEVGLɄVIKMWXIVWEVIEQSRKXLIQSWXVIWIEVGLɄEGXMZI

 ;LEXIZIVXLIGEWIWYFNIGXMRKVIWIEVGLTYFPMGEXMSRXSTSPMXMGEPTVIWWYVIWMWRSXEWYMXEFPIVIWTSRWI

-XɄRSXSRP]GLEPPIRKIWTVMRGMTPIWERHTVSXSGSPWFYXJVIUYIRXP]GEYWIWTSPMXMGEPHMJƤGYPXMIWSJMXWS[R

;LMPIɄQEXXIVWSJXMQMRKQYWXSGGEWMSREPP]FIRIKSXMEXIHŪMRGPYHMRK

)1451/(24$&6,&$.4($5105',5&755(')746+(410=,6917.'%(%(66(4

JSVɄKSZIVRQIRXXSMQTVSZIMXWGSQQYRMGEXMSRSJGSQTPI\SVɄGLEPPIRKMRK

ƤRHMRKWXSXLITYFPMGERH[LIVIRIGIWWEV]TYFPMWLRSXIWI\TPEMRMRK

[L]ɄTSPMG]ERHɄVIWIEVGLLEZIRSXVIEGLIHXLIWEQIGSRGPYWMSRW

7KHUHFOHDUO\DUH

DQGDOZD\VZLOOEH






DQGLQIRUPDWLRQ



SECTION 3: REASONS FOR THE DELAYED PUBLICATION OR WITHHOLDING OF GOVERNMENT RESEARCH 

The Promise of Evidence-Based Policymaking 2101
Uncertainty over how peer review and discussions about research quality 

and validity should be handled 

 !+,54(2146016(56+4((&$5(59+(4(6+(2((44(8,(9241&(559$5&,6('$52$461)6+(4($510

JSVɄHIPE]IHTYFPMGEXMSR;LIRTYFPMGEXMSRSJVIWIEVGLSRXLIVMWMRKYWISJJSSHFEROW'EWI7XYH]Ʉ


ERHSRɄIZEPYEXMRKGLERKIWXS+4TVEGXMGIFSYRHEVMIW'EWI7XYH]
[EWHIPE]IHKSZIVRQIRX

WTSOIWTIVWSRWWEMHXLEXXLIVITSVXW[IVIE[EMXMRKTIIVVIZMI[

 'EWI7XYH]WLS[WXLIHMJƤGYPXMIWXLEXGEREVMWIMJKSZIVRQIRX[MXLLSPHWVIWIEVGL

IZIRMJXLMWMWɄFIGEYWISJHSYFXWSZIVXLIUYEPMX]SJVIWIEVGL[MXLSYXI\TPEMRMRKMXWVIEWSRW

Case Studies

 !+(())(&651)/,0,/7/$.&1+1.24,&,0* $*(

 ('7&,0*57*$4&1057/26,10 $*(	

 !+(+145(/($65&$0'$. $*(


 8LIIJJIGXSJMQQMKVEXMSRSRYRIQTPS]QIRX 4EKI

 06(40$6,10$.&1/2$4,5101)'47*.$95 $*(

	 8LIMRGVIEWMRKYWISJJSSHFEROW 4EKI


 +115,0*$$9$;)41/9+(4(;17.,8( $*(


 !+(/,0,/7/$*()14$/$44,$*((064;8,5$ 4EKI

 8LIIJJIGXSJJVEGOMRKSRLSYWITVMGIW 4EKI

 SECTION 3: REASONS FOR THE DELAYED PUBLICATION OR WITHHOLDING OF GOVERNMENT RESEARCH

2102 The Promise of Evidence-Based Policymaking
THE EFFECTS OF MINIMUM ALCOHOL PRICING

Case study 1

+SZIVRQIRXTYFPMWLIHMXWEPGSLSPWXVEXIK]YRHIVXEOMRKXSMRXVSHYGIEQMRMQYQTVMGITIVYRMX

SJɄEPGSLSP	

!+(*18(40/(06.$70&+('$&1057.6$6,1010'(.,8(4,0*6+,5564$6(*; 
ERHEWOIHVIWIEVGLIVW

EXɄ9RMZIVWMX]SJ7LIJƤIPHXSQSHIPXLIMQTEGXSJEQMRMQYQYRMXTVMGI

6IWIEVGLIVWTVSZMHIHEHVEJXVITSVXXSXLI,SQI3JƤGI(,ERHXLI8VIEWYV]%WOIHXSQSHIP

XLIIJJIGXSJETQMRMQYQTVMGIXLI]IWXMQEXIHMX[SYPHŰVIHYGIEPGSLSPGSRWYQTXMSRF]	

PIEHMRKXSɄETTVS\MQEXIP]JI[IVHIEXLWTIV]IEVHYIXSEPGSLSPJI[IVLSWTMXEPEHQMWWMSRW

ERHɄJI[IVGVMQIWűEJXIVXIR]IEVW8LI]GSQTEVIHXLMW[MXLXLIIJJIGXWSJFERRMRKXLIWEPI

SJɄEPGSLSPFIPS[GSWXTVMGI[LMGLXLIMVQSHIPWLS[W[SYPHFIJEVPIWWIJJIGXMZIEXVIHYGMRKTVSFPIQ

HVMROMRKMX[EWIWXMQEXIHXSŰVIHYGIGSRWYQTXMSRF]NYWX	PIEHMRKXSEVSYRHJI[IVHIEXLW

JI[IVLSWTMXEPEHQMWWMSRWERHJI[IVGVMQIWVIPEXIHXSEPGSLSPTIV]IEVű

3RXLIWEQIHE]EWXLI7LIJƤIPHWXYH][EWVIPIEWIHXLIKSZIVRQIRXVINIGXIHEQMRMQYQTVMGI

8LIɄGVMQIɄTVIZIRXMSRQMRMWXIVERRSYRGIHXLEXKSZIVRQIRXSTXIHMRWXIEHJSVEFERSRɄFIPS[GSWXɄWEPIW

8LIEYXLSVWSJXLIVIWIEVGLVIPIEWIHEWXEXIQIRXWE]MRKŰ8LIKSZIVRQIRXHMHRSXFEV?YWAJVSQ

4(.($5,0*6+(4(21465=9(+$'$*4((')41/6+(1765(66+$69(917.'$.,*017427%.,&$6,109,6+

XLIɄKSZIVRQIRXŭWVIWTSRWIXSXLIEPGSLSPWXVEXIK]GSRWYPXEXMSRXSIRWYVI[ITVSZMHIHXLIMVMQTEGX

$55(55/(061)/,0,/7/70,624,&,0*9,6+6+(/1567261'$6((8,'(0&(2155,%.($8,0*241'7&('

EHHMXMSREPEREP]WIWSRFIPS[GSWXWIPPMRKMR.YRIEJYVXLIVGSRWMHIVEXMSR[EWXLEXMJXLIWI[IVI

TYFPMWLIHHMVIGXP]FIJSVIXLIKSZIVRQIRXŭWERRSYRGIQIRXXLEXGSYPHFIɄGSRWMHIVIHEHIJEGXS

$00170&(/(061)*18(40/(0621.,&;?

7LIJƤIPHVIWIEVGLIVWTYFPMWLIHXLIMVGSQTEVEXMZIWXYH]MRXLI&1.



According to Dr Sarah Wollaston MP, the Department of Health (DH) delayed the publication 

of a study of the effects on alcohol abuse of a minimum price compared with a ban on below 

cost sales, until the day of a government policy announcement.

8LIKSZIVRQIRXŭWWXVEXIK]WEMHXLEXEQMRMQYQTVMGISJTTIVYRMXŰGSYPHQIERJI[IVGVMQIWIEGL]IEV

ERHɄJI[IVEPGSLSPVIPEXIHHIEXLWE]IEVF]XLIIRHSJXLIHIGEHIű-RXLIMRXIVMQMXGLERKIHMXWɄQMRH6IWTSRHMRK

XSXLIɄ7LIJƤIPHWXYH]ERHXSSXLIVIZMHIRGIWYFQMXXIHXSXLIGSRWYPXEXMSRSRXLEXWXVEXIK]XLIKSZIVRQIRXWEMH

XLIɄGSRWYPXEXMSRLEHŰRSXTVSZMHIHIZMHIRGIXLEXGSRGPYWMZIP]HIQSRWXVEXIWXLEXQMRMQYQYRMXTVMGMRK[MPPEGXYEPP]

HSɄ[LEXMXMWɄQIERXXSVIHYGITVSFPIQHVMROMRK[MXLSYXTIREPMWMRKEPPXLSWI[LSHVMROVIWTSRWMFP]ű'MXMRKERŰEFWIRGI

SJɄXLEXIQTMVMGEPIZMHIRGIűXLI]WEMHEQMRMQYQTVMGI[EWFIMRKHIPE]IHŰYRXMP[ILEZIGSRGPYWMZIIZMHIRGIXLEXMXɄ[MPP

FIɄIJJIGXMZIű


*EMPMRKXSTYFPMWLXLI7LIJƤIPHWXYH]ELIEHSJERRSYRGMRKEɄGLERKISJKSZIVRQIRXTSPMG]TVIZIRXIH

XLIɄTYFPMGJVSQNYHKMRK[LIXLIVXLMWGLERKI[EWNYWXMƤEFPI

!+($..(*$6,10

199(4(*18(40/(0621.,&;$0'27%.,&'(%$6($))(&6('

!,/(.,0(

1EV

2SZ

.YP

1EV

.YP

.ER

7IT



;SPPEWXSR7


	

,1+SZIVRQIRX





,SQI3JƤGI




,SPQIW.




&VIRRER%IXEP





,SQI3JƤGI


SECTION 3: REASONS FOR THE DELAYED PUBLICATION OR WITHHOLDING OF GOVERNMENT RESEARCH 

The Promise of Evidence-Based Policymaking 2103
REDUCING SUGAR CONSUMPTION

Case study 2

4YFPMG,IEPXL1MRMWXIV.ERI)PPMWSR14[VSXIXS4,)EWOMRKXLIQXSGSQTPIXIVIWIEVGLXLEX

[SYPHŰTVSZMHIHVEJXVIGSQQIRHEXMSRWXSMRJSVQXLIKSZIVRQIRXŭWJYXYVIXLMROMRKSRWYKEV

MRXLIHMIXű4,)LEHQEHIEGSQQMXQIRXXSVIZMI[MRKXLMWIZMHIRGIMRMXWVITSVX7YKEV

VIHYGXMSRVIWTSRHMRKXSXLIGLEPPIRKI



,SYWISJ'SQQSRWLIEPXLWIPIGXGSQQMXXIIGLEMV(V;SPPEWXSREWOIH(,XSTYFPMWLXLIɄVITSVX

(,VIJYWIHSRXLIKVSYRHXLEXXLIIZMHIRGIMWŰGYVVIRXP]MRJSVQMRKJYXYVIKSZIVRQIRX

TSPMG]űWMG
(V;SPPEWXSRXLIRGEPPIHSR4,)XSTYFPMWLMXMRHITIRHIRXP]SVEXPIEWXQEOI

MXɄEZEMPEFPIXSLIVGSQQMXXIIŭWMRUYMV]SRXEGOPMRKGLMPHLSSHSFIWMX]4,)HIGPMRIHWE]MRKXLEX

ŰMXɄMWɄETTVSTVMEXIXLEXXLIKSZIVRQIRXLEWXMQIXSGSRWMHIVXLIVITSVXMRXLIYWYEP[E]űWSMX[MPP

FIɄTYFPMWLIHEPSRKWMHIXLIKSZIVRQIRXŭWSFIWMX]WXVEXIK]

4,)TYFPMWLIHMXWVITSVX

175(1)1//105+($.6+5(.(&6&1//,66((27%.,5+('6+(4(214610,65,037,4;,061$&6,10

SRɄGLMPHLSSHSFIWMX]

According to Dr Sarah Wollaston MP, the Department of Health (DH) delayed the publication 

of an evidence review by Public Health England (PHE) looking at ways to reduce sugar consumption.

%LMKLTVSƤPIGEQTEMKRPIHF]8:GLIJ.EQMI3PMZIVLEHGEPPIHJSVEXE\SRWYKEV]HVMROWXSVIHYGISFIWMX]

8LIɄLIEPXLWIPIGXGSQQMXXIIMRUYMV]MRXSXEGOPMRKSFIWMX]ERHXLITYFPMGHIFEXISRXLIQIVMXWSJEWYKEV

XE\[IVIɄFIXXIVMRJSVQIHSRGIXLI4,)VIWIEVGL[EWTYFPMWLIH&]XLIRXLIHIPE]LEHGEYWIHEZIV]TYFPMG

',5$*4((/(0618(4+194(5($4&+5+17.',0)14/21.,&;$0'27%.,&5&476,0;

!+($..(*$6,10

199(4(*18(40/(0621.,&;$0'27%.,&'(%$6($))(&6('

!,/(.,0(

1EV

7IT3GX

0'3GX

2SZ

%XXLIXMQISJ[VMXMRKXLITVIWIRXVITSVXXLIKSZIVRQIRXMW]IXXSTYFPMWLMXWSFIWMX]WXVEXIK]

;SPPEWXSR7


4YFPMG,IEPXL)RKPERH


,SYWISJ'SQQSRWLIEPXLWIPIGXGSQQMXXIIE


4YFPMG,IEPXL)RKPERH


,SYWISJ'SQQSRWLIEPXLWIPIGXGSQQMXXIIF


 SECTION 3: REASONS FOR THE DELAYED PUBLICATION OR WITHHOLDING OF GOVERNMENT RESEARCH

2104 The Promise of Evidence-Based Policymaking
THE HORSEMEAT SCANDAL

Case study 3

*VS^IRQIEXTVSGIWWIHEXTPERXWMR-VIPERHERH)RKPERHXIWXIHTSWMXMZIJSVLSVWI(2%

7YTIVQEVOIXW[MXLHVI[HS^IRWSJTVSHYGXW


(IJVEEWOIH'LVMW)PPMSXXTVSJIWWSVSJJSSHWEJIX]EX5YIIRŭW9RMZIVWMX]&IPJEWXXSGSRHYGX

ERɄMRUYMV]MRXSXLIMRXIKVMX]SJXLIJSSHWYTTP]RIX[SVO

!+(+YEVHMERVITSVXIHXLEXTYFPMGEXMSRSJXLIMRUYMV]ŭWƤREPVITSVX[EWHIPE]IH

8LIMRUYMV]ŭWƤREPVITSVX[EWTYFPMWLIH

According to the Guardian, the Department for Environment, Food and Rural Affairs (Defra) delayed 

the publication of an independent report into the contamination of supermarket meat products 

with horsemeat.	

!+(,037,4;>5,06(4,/4(2146+$'4$,5('$&10&(406+$6&76561.1&$.$76+14,6;5(48,&(5+$'4('7&('6+(07/%(4

SJWXEJJ[MXLJSSHPE[IRJSVGIQIRXWOMPPWERHGSYPHYRHIVQMRIIRJSVGIQIRXEKEMRWXGVMQIMRXLIQIEX

MRHYWXV](IPE]IHTYFPMGEXMSRSJXLIƤREPVITSVXTVIZIRXIHWGVYXMR]SJXLIWIGPEMQWERHKEZIVMWIXSWTIGYPEXMSR

XLEXXLIɄVITSVX[EWPMOIP]XSFIIQFEVVEWWMRKJSVXLIKSZIVRQIRX-XTVIZIRXIHXLITYFPMGJVSQKIXXMRKEɄJYPP

TMGXYVISJɄXLIɄGEYWIWSJEWIVMSYWGSRXEQMREXMSRIZIRXERH[LMPIMXMWYRHIVWXERHEFPIXLEXQMRMWXIVW[ERXIH

XSɄFIɄ[IPPɄSVKERMWIHXSVIWTSRHXLIHIPE]IHTYFPMGEXMSRIRHIHYTGVIEXMRKEQSVIGLEPPIRKMRKIRZMVSRQIRX

MRɄ[LMGLXSɄQEOIXLEXVIWTSRWI

!+($..(*$6,10

199(4(*18(40/(0621.,&;$0'27%.,&'(%$6($))(&6('

!,/(.,0(

.ER

.YR

%YK

7IT

	0E[VIRGI*



&&'2I[W


0E[VIRGI*


(ITEVXQIRXJSV)RZMVSRQIRX*SSHERH6YVEP

%JJEMVWERH*SSH7XERHEVHW%KIRG]


SECTION 3: REASONS FOR THE DELAYED PUBLICATION OR WITHHOLDING OF GOVERNMENT RESEARCH 

The Promise of Evidence-Based Policymaking 2105
THE EFFECT OF IMMIGRATION ON UNEMPLOYMENT

Case study 4

,SQIWIGVIXEV]8LIVIWE1E]HIPMZIVIHEWTIIGLGPEMQMRKXLEXŰJSVIZIV]EHHMXMSREPSRILYRHVIH

MQQMKVERXWXLI]?XLIMRHITIRHIRX1MKVEXMSR%HZMWSV]'SQQMXXIIAIWXMQEXIHXLEX&VMXMWL[SVOIVW

917.'016%((/2.1;('?  8LIVITSVX1E]GMXIH 9$5&4,6,&,5('%;10$6+$0146(51)6+($6,10$.

-RWXMXYXISJ)GSRSQMGERH7SGMEP6IWIEVGLJSVGLIVV]TMGOMRKSRIVIWYPXJVSQQER]XLEXSZIVEPPWLS[IH

RSWXEXMWXMGEPP]WMKRMƤGERXIJJIGX

(950,*+6,08(56,*$6,10$..(*('6+$6&,8,.5(48,&($0$.;5,51)6+(5$/('$6$9$524(8(06(')41/%(,0*

TYFPMWLIHF]XLITVMQIQMRMWXIVŭWSJƤGIEPXLSYKLŰ(S[RMRK7XVIIXWSYVGIWXSPHXLI&&'XLIVITSVXLEH

016;(6%((0&1/2.(6('$0'9$50164($';61%(4(.($5('?




8LI,SQI3JƤGIŭWEREP]WMWMWTYFPMWLIH-XRSXIWXLEXXLI1MKVEXMSR%HZMWSV]'SQQMXXIIVITSVXJSYRH

ŰXLIVIMWVIPEXMZIP]PMXXPIIZMHIRGIXLEXQMKVEXMSRLEWGEYWIHWXEXMWXMGEPP]WMKRMƤGERXHMWTPEGIQIRX

SJɄ9/REXMZIWŷ[LIRXLIIGSRSQ]LEWFIIRWXVSRKűERHXLEXHMWTPEGIQIRXŰHMWWMTEXIWSZIVXMQIű 

-XɄEPWSVITSVXIHSXLIVVIWIEVGLŰJEMPIHXSMHIRXMJ]ER]WXEXMWXMGEPP]WMKRMƤGERXMQTEGXWSJRIXQMKVEXMSR

SRɄGPEMQERXGSYRXVEXIWű

According to a BBC Newsnight investigation, Downing Street withheld research into the number 

of UK workers unemployed as a consequence of immigration.


('7&,0*0(6/,*4$6,10616+(6(051)6+175$0'52(4;($4,5$.10*56$0',0**18(40/(0621.,&;6$22($456+$6/,0,56(45

TSMRXIHWIPIGXMZIP]XSWYTTSVXMZIVIWIEVGLXSNYWXMJ]XLMWTSPMG]ERHWYTTVIWWIHXLITYFPMGEXMSRSJEQSVIGSQTVILIRWMZI

EREP]WMWXLEXWYKKIWXIHXLEXXLIŰHMWTPEGIQIRXIJJIGXűSJMQQMKVEXMSRMWRSXEWMKRMƤGERXTVSFPIQ)GSRSQMWXW

ERHɄNSYVREPMWXW[IVIPIJXTSSVP]TPEGIHXSWGVYXMRMWIXLIMQTEGXSJEQMKVEXMSRGETFIGEYWIXLMWWXYH][EWRSXTYFPMWLIH

SRɄXMQIERHQMRMWXIVWERHWTSOIWTIVWSRWGSRXMRYIHXSYWIHYFMSYWWXEXMWXMGWMRWYTTSVXSJXLIMVTSPMG]

!+($..(*$6,10

199(4(*18(40/(0621.,&;$0'27%.,&'(%$6($))(&6('

!,/(.,0(

(IG

	6+1EV

6+1EV


'SSO'


1E]8


1MKVEXMSR%HZMWSV]'SQQMXXII


4SVXIW.


(IZPMR'IXEP


 SECTION 3: REASONS FOR THE DELAYED PUBLICATION OR WITHHOLDING OF GOVERNMENT RESEARCH

2106 The Promise of Evidence-Based Policymaking
INTERNATIONAL COMPARISON OF DRUG LAWS

Case study 5

'SEPMXMSRKSZIVRQIRXTYFPMWLIHMXWHVYKWXVEXIK]MRGPYHMRKEGSQQMXQIRXXSŰVIZMI[RI[

IZMHIRGISR[LEX[SVOWMRSXLIVGSYRXVMIWERH[LEX[IGERPIEVRJVSQMXű	

,SQIWIGVIXEV]8LIVIWE1E]SVHIVIHERMRXIVREXMSREPWXYH]SJHVYKWPE[WMR1EVGL




-RɄ1E] QMRMWXIVWFIKEREVIZMI[SJHVYKPE[WMRXLMVXIIRGSYRXVMIWMRGPYHMRK4SVXYKEP

(IRQEVOERHZEVMSYWWXEXIWMRXLI97%XLEXLEZIHIGVMQMREPMWIHTSWWIWWMSRSJWSQIHVYKW



,SQI3JƤGIQMRMWXIV2SVQER&EOIVEGGYWIHXLIKSZIVRQIRXSJŰWYTTVIWWMRKűXLIVITSVX

&.$,/,0*,6+$'%((04($'; )1427%.,&$6,105,0&(7.;



,SQI3JƤGITYFPMWLIHMXWVIZMI[GSRGPYHMRKXLIVIMWRSGPIEVGSVVIPEXMSRFIX[IIRXLIWIZIVMX]

SJɄTYRMWLQIRXYRHIVHVYKPE[WERHXLITVIZEPIRGISJHVYKYWI


2SVQER&EOIVVIWMKRWEW,SQI3JƤGIQMRMWXIV

According to Liberal Democrats, Conservative members of the coalition government delayed 

the publication of a Home Office review of drug laws in thirteen countries.

8LIKSZIVRQIRXQEHIEGSQQMXQIRXXSVIZMI[MRKHVYKPE[WMRSXLIVGSYRXVMIWERHETTIEVWXSLEZIHIPE]IH

XLIɄTYFPMGEXMSRSJXLEXVIZMI[FIGEYWIXLIƤRHMRKW[IVITSPMXMGEPP]GLEPPIRKMRKJSVXLIHSQMRERXGSEPMXMSRTEVXRIV

-XɄMWYRPMOIP]XLEXXMQIP]TYFPMGEXMSR[SYPHLEZITVSQTXIHEGLERKIMRPIKMWPEXMSRSVIZIREQSVIJSVQEPVIZMI[

SJ9/HVYKWTSPMG]FSXLLEHEPVIEH]FIIRVYPIHSYXF]XLI'SRWIVZEXMZIW
,S[IZIVMX[SYPHLEZIEPPS[IHJYPPIV

TYFPMGWGVYXMR]SJXLILSQIWIGVIXEV]ŭWGPEMQWXLEXXLI9/ŭWETTVSEGLXSHVYKVIKYPEXMSRMW[SVOMRKERHEFIXXIV

,0)14/('27%.,&'(%$6(10+196+("&$0.($40)41/6+(9$;'47*5$4(4(*7.$6('(.5(9+(4(

!+($..(*$6,10

199(4(*18(40/(0621.,&;$0'27%.,&'(%$6($))(&6('

!,/(.,0(

(IG

1EV

WYQQIV

3GX

3GX

2SZ

;MRXSYV4ERH8VEZMW%


	,1+SZIVRQIRX



8VEZMW%E


8VEZMW%F


1SVVMW2



,SQI3JƤGI


;EXX2


SECTION 3: REASONS FOR THE DELAYED PUBLICATION OR WITHHOLDING OF GOVERNMENT RESEARCH 

The Promise of Evidence-Based Policymaking 2107
THE INCREASING USE OF FOOD BANKS

Case study 6

(IJVEEWOIHEXIEQSJVIWIEVGLIVWPIHF]XLI9RMZIVWMX]SJ;EV[MGOXSGSRHYGXE6ETMH)ZMHIRGI

%WWIWWQIRXSJXLIVIWIEVGLPMXIVEXYVISRIQIVKIRG] JSSHEMH8LI]PSSOIHEXTYFPMWLIH

PMXIVEXYVIERHMWWYIHEGEPPJSVIZMHIRGIXLEXTVSQTXIHTVSZMHIVWSJ JSSHEMHERHSXLIVEKIRGMIW

61&1064,%76(6+(,4(:2(4,(0&(

%WXIIVMRKGSQQMXXIIMRGPYHMRK(IJVEXLI(ITEVXQIRXJSV;SVOERH4IRWMSRWERHXLI

(2$46/(061)($.6+$22418('6+(4(2146

8LIQIHMEVITSVXIHEPPIKEXMSRWXLEXXLIWXYH][EWFIMRKŰLMHHIRű ERHŰWYTTVIWWIHű 

%WOIHɄF]ɄE+YEVHMER NSYVREPMWXXSI\TPEMRXLIHIPE]E(IJVEWTSOIWTIVWSRWEMHŰ+SZIVRQIRX

)70'('4(5($4&+241-(&65$4(4(37,4('61*16+417*+6+(0(&(55$4;4(8,(9$0'37$.,6;$5574$0&(

TVSGIWWTVMSVXSTYFPMGEXMSR3RGIXLMWTVSGIWWMWGSQTPIXIXLIVITSVX[MPPFITYFPMWLIH

SRɄXLIɄKSZIVRQIRXŭW[IFWMXIű

(IJVETYFPMWLIHƤREPVITSVX

567';27%.,5+(',06+(&1. JSYRHXLEXXLIVMWIMRXLIYWISJ JSSHFEROWŰMWEWWSGMEXIH[MXL

GYXWXSPSGEPEYXLSVMX]WTIRHMRKERHGIRXVEP[IPJEVIWTIRHMRKűERHXLEXXLIŰLMKLIWXPIZIPW

SJ JSSHFEROYWILEZISGGYVVIH[LIVIXLIVILEZIFIIRXLILMKLIWXVEXIWSJWERGXMSRMRK

YRIQTPS]QIRXERHGYXWMRGIRXVEP[IPJEVIWTIRHMRKű 	

According to its authors, the Department for Environment, Food and Rural Affairs (Defra) delayed 

the publication of a study looking at the growing use of emergency food aid for eight months.

;IPJEVI1MRMWXIV0SVH*VIYHLEHEVKYIHMR.YP]XLEXQSVITISTPI[IVIYWMRKJSSHFEROWFIGEYWIQSVI

SJɄXLIQI\MWXIHERHE[EVIRIWWSJXLIQLEHMRGVIEWIHŪMRSXLIV[SVHWXLEXMRGVIEWIHWYTTP]LEHGEYWIHKVIEXIV

YWIRSXMRGVIEWIHHIQERH,IVINIGXIHEPMROFIX[IIRXLIKSZIVRQIRXŭWFIRIƤXVIJSVQWERHERMRGVIEWIHYWI

SJJSSHFEROW
!+(/,0,56(4>5&.$,/&17.'016%((8$.7$6(',06+($%5(0&(1)(8,'(0&()41/6+('(.$;('4(2146

8LIVITSVXGSRXVEHMGXIHXLIQMRMWXIVMXGMXIHMQQIHMEXIƤRERGMEPGVMWMWSJXIRVIPEXIHXSGLERKIWXSFIRIƤXW

EWɄERɄMQTSVXERXHVMZIVSJJSSHFEROYWI

!+("*18(40/(06'1(5016&1..(&6'$6$10)11'$,'06+($%5(0&(1)5;56(/$6,&(8,'(0&(106+(75(1))11'

FEROWMRXLI9/XLIVIWIEVGLIVWVIPMIHSRWXYHMIWJVSQXLI97%'EREHEERHSXLIVGSYRXVMIWERHSRGEWIWXYHMIW

JVSQTVSZMHIVWSJJSSHFEROW8MQIP]TYFPMGEXMSR[SYPHLEZILMKLPMKLXIHXLIKETWMRXLIIZMHIRGIFEWI-X[SYPH

$.51+$8($..19('%(66(4,0)14/('5&476,0;1)/,0,56(45>&.$,/5$0'+$8(4(37,4('*18(40/(0661$''4(556+('$6$

!+($..(*$6,10

199(4(*18(40/(0621.,&;$0'27%.,&'(%$6($))(&6('

!,/(.,0(

.YR

2SZ

*IF

1EV

(YKER)


3EOIWLSXX-


&YXXPIV4


0EQFMI1YQJSVH,IXEP


	0SSTWXVE6IXEP



1SVVMW2


*IF1EV

 SECTION 3: REASONS FOR THE DELAYED PUBLICATION OR WITHHOLDING OF GOVERNMENT RESEARCH

2108 The Promise of Evidence-Based Policymaking
CHOOSING A GP AWAY FROM WHERE YOU LIVE

Case study 7

18(40/(06$00170&('$2,.165&+(/(61$..192$6,(065614(*,56(49,6+$1765,'(6+(.1&$.,6;

MRɄ[LMGLXLI] PMZI

6IWIEVGLIVWEXXLI0SRHSR7GLSSPSJ,]KMIRIERH8VSTMGEP1IHMGMRITVSZMHIHERMRXIVMQVITSVX

SRɄXLIɄTMPSXWGLIQIEXXLIVIUYIWXSJXLI(ITEVXQIRXSJ,IEPXL8LIPIEHVIWIEVGLIVWEMHXLEX

XLIɄVITSVX[EWMRGSQTPIXIFIGEYWIXLI][IVIWXMPPEREP]WMRKXLIHEXE

*MREPVITSVXTYFPMWLIHMRXLIWEQI[IIOEWRI[+4GSRXVEGX[LMGLMRGPYHIHERIRXMXPIQIRXXSGLSSWI

$',))(4(06




According to the general practitioners’ (GP) magazine Pulse, the government failed to publish 

an evaluation of a pilot scheme allowing people to register with GPs outside their local area until 

after a decision was taken to roll out the scheme nationwide.

6IWIEVGLIVWIZEPYEXMRKXLI+4GLSMGITMPSXWGLIQIWE]XLI][IVIEWOIHXSTVSZMHIERMRXIVMQVITSVXXSXLI(ITEVXQIRX

SJɄ,IEPXLMRSVHIVXSMRJSVQEVIRIKSXMEXMSRSJXLI+4GSRXVEGXŪFIJSVIVSFYWXIGSRSQMGHEXEGSYPHFIGSPPIGXIH8LIƤREP

VITSVX[EWXLIRVIPIEWIHMRXLIWEQI[IIOEWERI[+4GSRXVEGX[LMGLMRGPYHIHXLIIRXMXPIQIRXSJTEXMIRXWXSGLSSWI

SYXSJEVIE+4W8LIƤREPVITSVXWLS[IHPS[XEOIYTSJXLIWGLIQISRP]SYXSJEXSXEPSJIPMKMFPITVEGXMGIWXSSO

TEVXMRXLITMPSXERHSJXLIWIHMHRSXVIKMWXIVER]TEXMIRXWJVSQSYXWMHIXLIMVFSYRHEV]8LIGSRXVEGXRIKSXMEXMSRWQMKLX

[IPPLEZIXEOIREHMJJIVIRXGSYVWIMJXLITMPSXVIWYPXWLEHFIIREZEMPEFPI%XXLIXMQI2MKIP4VEMXMIWHITYX]IHMXSVSJ4YPWI

QEKE^MRIWEMHŰ+4TVEGXMGIFSYRHEVMIW[MPPFIEFSPMWLIHJVSQRI\XQSRXL?%TVMPAMR)RKPERHERHXLIKSZIVRQIRX

56,..+$5;(66127%.,5+$0(8$.7$6,101),652,.165,0616+,55&+(/(!+,5,5$+7*(0$6,10$.21.,&;&+$0*($0'6+((8,'(0&(

JSVɄMXɄLEWRSX]IXFIIRTYFPMWLIHű

-XVIQEMRWYRGPIEV[LIXLIVTYFPMGEXMSRSJXLIƤREPVITSVX[EWGEYWIHF]SZIVPSRKTIIVVIZMI[GSRWIUYIRXVIZMWMSR

QMRMWXIVMEPTVIWWYVIXSQSHMJ]ƤRHMRKWSVHITEVXQIRXEPMRIVXME

!+($..(*$6,10

199(4(*18(40/(0621.,&;$0'27%.,&'(%$6($))(&6('

!,/(.,0(

%TV

.YP

1EV

0MRH7


(ITEVXQIRXSJ,IEPXL



4VMGI'


SECTION 3: REASONS FOR THE DELAYED PUBLICATION OR WITHHOLDING OF GOVERNMENT RESEARCH 	

The Promise of Evidence-Based Policymaking 2109
THE MINIMUM AGE FOR A MARRIAGE ENTRY VISA

Case study 8

8LILSQIWIGVIXEV]GSQQMWWMSRIH4VSJIWWSV1EVMERRI,IWXIVSJXLI9RMZIVWMX]SJ&VMWXSP

XSɄVITSVXSRɄXLIHIWMVEFMPMX]SJVEMWMRKXLIQMRMQYQEKIJSVEQEVVMEKIZMWEJVSQXS

MRɄSVHIVXSHIXIVJSVGIHQEVVMEKIW

8LI&VMWXSPVITSVXWYKKIWXIHXLEXER]WYGLGLERKI[SYPHFIYRNYWXMƤEFP]HIXVMQIRXEP

XSɄZSPYRXEV]QEVVMEKIWERH HMWGVMQMREXSV]SRVEGMEPERHIXLRMGKVSYRHWMRVIPEXMSRXSEVVERKIH

QEVVMEKIW8LIVITSVX[EWRSXTYFPMWLIHF]XLI,SQI3JƤGI8LILSQIWIGVIXEV]PEXIV

I\TPEMRIHRSRTYFPMGEXMSRSRXLIKVSYRHXLEXWLIERHX[SI\XIVREPTIIVVIZMI[IVW[LMPI

YRGVMXMGEPSJXLIQIXLSHSPSK][IVIHMWWEXMWƤIH[MXLXLIYWISJYRWYFWXERXMEXIHWXEXIQIRXW

YRGPIEVXIVQMRSPSK]ERHWEQTPMRKFMEWMRXLIƤVWXHVEJXSJXLIVITSVX8LIVIWIEVGLIVWVIZMWIH

6+(4(2146,04(52105(616+(&1//(065)41/2((44(8,(9(45

!+(4,561.4(21469$527%.,5+(',0'(2(0'(06.;

8LILSQIWIGVIXEV]MWWYIHEGSRWYPXEXMSRTETIVEWOMRK[LIXLIVVEMWMRKXLIQEVVMEKIZMWEEKI

XSɄ[MPPLIPTXSVIHYGIJSVGIHQEVVMEKIW

8LI,SYWISJ'SQQSRWLSQIEJJEMVWWIPIGXGSQQMXXIIJSYRHMRWYJƤGMIRXIZMHIRGIXLEX

MXɄ[SYPHHSWS-RZMI[SJXLIVMWOWMXYVKIHJYVXLIVVIWIEVGLERHEWOIHJSVGSRGPYWMZIIZMHIRGI

6+$6&+$0*(5917.'016,0$'8(46(06.; ',5&4,/,0$6(

!+(+1/(5(&4(6$4;$55(46('6+$66+(&10&.75,8((8,'(0&(517*+6%;6+(5(.(&6&1//,66((019

I\MWXIHMRXLIJSVQSJ ƤKYVIWWLS[MRKXLEXJSVGIHQEVVMEKIWTIEOFIX[IIREKIWERHɄ

8LIɄ7YTVIQI'SYVXPEXIVTSMRXIHSYXXLEXXLIIZMHIRGIXLIWIPIGXGSQQMXXIIEWOIHJSV[EW

0166+(6;2,&$.$*(1)8,&6,/51) )14&('/$44,$*(%769+(6+(44$,5,0*6+(8,5$$*(917.''(6(4

XLIɄTVEGXMGI

8LILSQIWIGVIXEV]EQIRHIHXLIMQQMKVEXMSRVYPIWXSVEMWIXLIEKIJSVɄEɄQEVVMEKIZMWE

JVSQɄɄXS



8LI7YTVIQI'SYVXLIPHXLEXMREQIRHMRKXLIVYPIW[MXLSYXŰVSFYWXIZMHIRGISJER]WYFWXERXMEP

HIXIVVIRXIJJIGXŷYTSRJSVGIH QEVVMEKIWűXLILSQIWIGVIXEV]LEHɄJEMPIHɄXSɄIWXEFPMWL

EɄTVSTSVXMSREXIVIWTSRWIXSETVIWWMRKWSGMEPRIIHXLIVIF]MRZEPMHEXMRKLIVɄVYPIGLERKI

The home secretary withheld publication of research that failed to support her proposed policy 

of denying entry to spouses where one of the couple was under 21. The history emerged in the judgment 

of the Supreme Court in R (Quila) v Secretary of State for the Home Department.

!+($..(*$6,10

!,/(.,0(

*IF

%YK

(IG

6+.YR

6+2SZ

6+3GX

.YP



&]KSMRKELIEH[MXLEVYPIGLERKIXLEXLEHEWIVMSYWMQTEGXSRXLILYQERVMKLXWSJ]SYRKTISTPIERHHSMRK

WSɄ[MXLSYXEREHIUYEXIIZMHIRGIFEWIKSZIVRQIRXGSRXVEZIRIHXLIPE[ERH[EWSFPMKIHXSVIXVIEX4YFPMGEXMSR

SJXLI&VMWXSPVITSVXMRF]XLI,SQI3JƤGIXSKIXLIV[MXLMXWS[RGVMXMUYI[SYPHLEZIFIIRERSTIR

ERHɄTVSHYGXMZI[E]SJHIZIPSTMRKTSPMG]

199(4(*18(40/(0621.,&;$0'27%.,&'(%$6($))(&6('

?A9/7'

,IWXIV1IXEP


,SYWISJ'SQQSRWLSQIEJJEMVWWIPIGXGSQQMXXII


9/&SVHIV%KIRG]



 SECTION 3: REASONS FOR THE DELAYED PUBLICATION OR WITHHOLDING OF GOVERNMENT RESEARCH

2110 The Promise of Evidence-Based Policymaking
THE EFFECT OF FRACKING ON HOUSE PRICES

Case study 9

(IJVEVIGIMZIHEVIUYIWXXSVIPIEWIMRJSVQEXMSRSRXLIIGSRSQMGWSJWLEPIKEWYRHIV)RZMVSRQIRXEP

-RJSVQEXMSR6IKYPEXMSRW	

(37(569$54(57%/,66('

(IJVEVIWTSRHIHXSXLIVIUYIWXF]TYFPMWLMRKŬ7LEPI+EW6YVEP)GSRSQ]-QTEGXWVITSVXŭ[MXLPEVKI

5(&6,1054('$&6('

!+(+YEVHMER

 (EMP]1EMP	$0' 8IPIKVETL
4$05614,(5&4,6,&,5,0*6+(4('$&6,10

*SPPS[MRKEGSRXIWXIH*VIIHSQSJ-RJSVQEXMSRVIUYIWXXLI-RJSVQEXMSR'SQQMWWMSRIVVIUYMVIH(IJVE

6127%.,5+$0704('$&6('8(45,101)6+(4(2146

()4$27%.,5+('6+(704('$&6('4(2146



According to MPs Barbara Keeley (Labour) and Caroline Lucas (the Green party), and anti-fracking 

campaigners, the Department for Environment, Food and Rural Affairs (Defra) censored a study 

examining the effect of shale gas drilling on the rural economy, to remove evidence suggesting 

that house prices would fall.

8LITVIWIRXKSZIVRQIRXMWGSQQMXXIHXSI\TERHMRKJVEGOMRKJSVWLEPIKEW-XMWYRPMOIP]XLEXJYPPTYFPMGEXMSR[SYPHLEZI

EPXIVIHTSPMG]FYXMX[SYPHLEZIEPPS[IHKVIEXIVWGVYXMR]SJXLITSXIRXMEPMQTEGXWSJXLMWTSPMG]

8LITYFPMGHIFEXIEFSYXJVEGOMRKLEWJSGYWIHSRXLIVMWOWXSLIEPXLERHXLIIRZMVSRQIRXJVSQXLIXIGLRMUYI

ERHɄSRɄXLIɄTSXIRXMEPIGSRSQMGMQTEGXW1YGLSJXLIEZEMPEFPIIZMHIRGISRVMWOWXSLIEPXLERHXLIIRZMVSRQIRXMWJVSQ

XLIɄ97%[LMGLHMJJIVWWMKRMƤGERXP]JVSQXLI9/KISPSKMGEPP]ERHMRLS[JVEGOMRKMWVIKYPEXIH2SRIXLIPIWWEVIZMI[

SJɄXLMWɄIZMHIRGIF]ɄXLI6S]EP7SGMIX]ERHXLI6S]EP%GEHIQ]SJ)RKMRIIVMRKGSRGPYHIHGSRGPYHIHXLEXŰXLILIEPXL

WEJIX]ɄERHIRZMVSRQIRXEPVMWOWGERFIQEREKIHIJJIGXMZIP]MRXLIɄ9/űERHXLEXŰWIMWQMGVMWOWEVIPS[ű 
(52,6(6+,5

IZMHIRGIFIMRKTYFPMGP]EZEMPEFPIQIHMEVITSVXMRKSJJVEGOMRKJVIUYIRXP]SZIVWXEXIWXLIORS[RVMWOWERHPIEZIW

XLITYFPMGɄMPPMRJSVQIH	

-RTYFPMWLMRKEVIHEGXIHZIVWMSRSJXLMWWXYH](IJVEEGORS[PIHKIHXLITYFPMGMRXIVIWXMRXLIMRJSVQEXMSRFIMRKTYFPMWLIH

&YXMXEPWSWXEXIHŰXLIVIMWEWXVSRKTYFPMGMRXIVIWXMR[MXLLSPHMRKXLI?VIHEGXIHAMRJSVQEXMSRFIGEYWIMXMWMQTSVXERXXLEX

SJƤGMEPWGERGSRWMHIVMQTPMGEXMSRWSJTSXIRXMEPMQTEGXWERHWGIREVMSWEVSYRHXLIHIZIPSTQIRXSJXLIWLEPIKEWMRHYWXV]

ERHXSHIZIPSTSTXMSRW[MXLSYXXLIVMWOXLEXHMWGPSWYVISJIEVP]XLMROMRKGSYPHGPSWIHS[RHMWGYWWMSRű 	

(IJVEJEMPIHXSEGORS[PIHKIXLEXXLIVIHEGXMSRGVIEXIHEZSMH[LMGL[EWƤPPIHF]WTIGYPEXMSRERHERIGHSXIMRTPEGI

SJɄIZMHIRGIERH[LMGLQEHIMXLEVHIVJSVXLITYFPMGXSNYHKIXLIGPEMQWXLEXJVEGOMRK[MPPFIRIƤXXLIGSYRXV]IGSRSQMGEPP]

!+($..(*$6,10

199(4(*18(40/(0621.,&;$0'27%.,&'(%$6($))(&6('

!,/(.,0(

(IG

.YR

.YP

%YK

1EWSR6


	7TIRGIV&



0IER+


-RJSVQEXMSR'SQQMWWMSRIVŭW3JƤGI


(ITEVXQIRXJSV)RZMVSRQIRX*SSHERH6YVEP%JJEMVW



8LI6S]EP7SGMIX]ERHXLI6S]EP%GEHIQ]SJɅ)RKMRIIVMRK


	7IRWIEFSYX7GMIRGI


	(ITEVXQIRXJSV)RZMVSRQIRX*SSHERH6YVEP%JJEMVWF


6+.YR

56.YP

SECTION 3: REASONS FOR THE DELAYED PUBLICATION OR WITHHOLDING OF GOVERNMENT RESEARCH 

The Promise of Evidence-Based Policymaking 2111
Waiting for publication in peer-reviewed journals

 8LIHIWMVIXSTYFPMWLVIWIEVGLMRTIIVVIZMI[IHNSYVREPWGEREPWSFIEWSYVGISJHMJƤGYPX]-XLEWFIIR

WYKKIWXIHXLEXVIWIEVGLIVWQE]LSPHVITSVXWFEGOJSVJIEVXLEXTYFPMGEXMSRF]KSZIVRQIRXGSYPH

&1/241/,5()7674(27%.,&$6,10%;-1740$.5

 *SVMRWXERGI4VSJIWWSV(EZMH'SKKSRJVSQXLI9RMZIVWMX]SJɅ7SYXLEQTXSR[LSLEWGLEMVIH

SVɄGSRXVMFYXIHXSWGMIRXMƤGGSQQMXXIIWEHZMWMRKRYQIVSYWHITEVXQIRXWWEMH

Ȋ6XFKVLWXDWLRQVKDYHEHHQKDQGOHGRQDFDVHE\FDVHEDVLV)LQGLQJVZHUHUHYLHZHG

LQFRQȴGHQFHE\LQGHSHQGHQWVFLHQWLVWVZKRDGYLVHGRQZKHWKHUWKH\LQGLFDWHG

DbQHHGIRUbXUJHQWFKDQJHVWRSROLF\RUbSUDFWLFH:KHUHLWKDVEHHQQHFHVVDU\WRGHIHU

SXEOLFDWLRQLQWKLVZD\DQbDVVHVVPHQWKDVEHHQPDGHWRbHQVXUHWKDWWKHUHVXOWV

GRbQRWLQGLFDWHDQHHGIRULPPHGLDWHDFWLRQHJWRSURWHFWWKHKHDOWKRIbWKHbSXEOLF

ΖIbWKHUHZHUHVXFKDQHHGVWHSVZRXOGEHWDNHQWRHQVXUHUDSLGSXEOLFDWLRQ

XVXDOO\bMRXUQDOVZLOOJLYHSULRULW\LQWKLVVLWXDWLRQΖIWKLVGLGQRWKDSSHQWKHEHQHȴW

RIbH[WHUQDOSHHUUHYLHZZRXOGEHORVWDQGbPDQ\UHVHDUFKHUVZRXOGEHGHWHUUHGIURP

XQGHUWDNLQJWKHbUHVHDUFKȋ

-XɄMWɄRSXIHXLEXSZIVXLITEWXHIGEHIQSWXVIWIEVGLTYFPMWLIVWLEZIGPEVMƤIHXLEXXLI]ɄEGGITX

QERYWGVMTXW[LIVIXLIVILEWFIIRWSQITVMSVTYFPMGEXMSRSJƤRHMRKWHYIXSVIKYPEXSV]

SVɄSXLIVɄVIUYMVIQIRXW

	 ((44(8,(9,51)6(0)70'$/(06$..;,/2146$06616+(&4(',%,.,6;$0'8$.,',6;1)6+(4(5($4&+

 (V.SREXLER1IRHIPPIGXYVIVEXXLI9RMZIVWMX]SJ(YRHIIWEMH

Ȋ&XUUHQWO\WULDOVFDQKDYHPDMRUSROLF\LPSDFWDQGbUHFHLYHDORWRIPHGLDSXEOLFLW\

ZLWKRXWDQ\DGHTXDWHSXEOLFDWLRQ7KLVLQFUHDVHVWKHULVNRIWULDOVZLWKXQUHDOLVHG

ȵDZVKDYLQJSROLF\LPSDFWDQGPHDQVWKDWWKHJRYHUQPHQWFDQFODLPWKDWWULDOV

LQGLFDWHSROLF\VXFFHVVDQGbHQMR\PHGLDUHSRUWVVWDWLQJDVPXFKZKLOHWKHUHKDYH

EHHQLQDGHTXDWHRSSRUWXQLWLHVWRbFULWLFDOO\DVVHVVWKHVHWULDOVȋ


 -XMWRSXEGLMIZEFPISVGSQTEXMFPI[MXLTVSQTXTYFPMGEXMSRSJMRHITIRHIRXVIWIEVGLXLEXɄEPPɄGVMXMGEP

HMWGYWWMSRFIEHHVIWWIHTVMZEXIP]ƤVWX,S[IZIVMX[SYPHGPIEVP]RSXWIVZIXLITYFPMGMRXIVIWXMJ

YRVIPMEFPIVIWIEVGLVIWYPXW[IVIVYWLIHSYXFIJSVIXLI][IVIGSQTPIXIHERHGLIGOIH8LMWVIUYMVIQIRX

ERHXLIEHHMXMSREPXMQIMXQE]XEOIWLSYPHFIGSRWMHIVIHMRXLIGSQQMWWMSRMRKSJɄVIWIEVGLERHMR

ER]ɄTSPMG]TPERRMRKVIPEXIHXSMX8LMWLEWRSXFIIRLS[XLSWIHMWGYWWMSRWEFSYXUYEPMX]ERHɄVIPMEFMPMX]

LEZIFIIRLERHPIHMRWSQISJXLIGEWIWVITSVXIHXSXLMWMRUYMV]-JXLIVIMWɄERɄYREZSMHEFPIHIPE]GEYWIH

F]ɄXLIɄTIIVVIZMI[TVSGIWWSVF]HMWGYWWMSREFSYXVIWIEVGLUYEPMX]XLIRɄHITEVXQIRXW[SYPHHSFIXXIV

XSɄI\TPEMRXLMWERHXSXEOIGEVIRSXXSEWOXLITYFPMGXSVIP]SRER]VIWYPXWXLI]GERRSXWII

 07/%(41)4(5210'(06557**(56('+7/$04($5105)14'(.$;5,027%.,&$6,101)*18(40/(06

VIWIEVGLVERKMRKJVSQXYVRSZIVSJWXEJJXSPEGOSJGSRƤHIRGI8LIGMZMPWIVZERXWGLEVKIH[MXL

VIGIMZMRKERHɄTYFPMWLMRKVIWIEVGLEVIRSXRIGIWWEVMP]XLSWI[LSEVIGSRƤHIRXERHWYJƤGMIRXP]ZIVWIH

MRɄXLIɄGSQQMWWMSRSJXLIVIWIEVGLXSFIEFPIXSGSQQIRXSRMXWZEPMHMX]SVXS[LEXI\XIRXMXEHHVIWWIW

6+(21.,&;37(56,10

 ,8,.5(48$06518(45((,0*4(5($4&+1)6(0/18(10)41/6+(,421565'74,0*241-(&65!+(,44(2.$&(/(065

EVIREXYVEPP]PIWWGSRƤHIRXMRGSQQYRMGEXMRKXLIɄƤRHMRKWSJVIWIEVGL[LMGL

XLI][IVIRSXMRZSPZIHMRɄGSQQMWWMSRMRK'SRXMRYMX]FIX[IIRWXEJJQMKLXFI

IEWMIVMJXLIVI[EWEGIRXVEPHMVIGXSV]SJɄVIWIEVGL[LMGL[SYPHEXXLIPIEWX

2418,'(,0)14/$6,10$%1769+$6+$'%((0&1//,55,10('$0'$6%(5656,/7.$6(

EɄTVSTIVLERHMRKSZIVSJXLIVIWIEVGLGSRXVEGXERHVIPEXMSRWLMT

&LYLOVHUYDQWV

RYHUVHHLQJ

	

PRYHRQIURP




SURMHFWV



SECTION 3: REASONS FOR THE DELAYED PUBLICATION OR WITHHOLDING OF GOVERNMENT RESEARCH

2112 The Promise of Evidence-Based Policymaking
 -XLEWEPWSFIIRWYKKIWXIHXLEX[LIVIGMZMPWIVZERXWEVIYRJEQMPMEV[MXLVIWIEVGLQIXLSHWXLMWPIEHW

XSɄJVYWXVEXIHGSQQYRMGEXMSRWERHXIRWMSRW[MXLVIWIEVGLIVWERHYPXMQEXIP]XSHIPE]MRTYFPMWLMRK

4(57.65!+(4(,5$*4($6'($.1)4(5($4&+(:2(46,5($&4155*18(40/(0651/(1)6+,59$5(:+,%,6('

MRɄXLIMRWMKLXJYPVIWTSRWIWXSXLMWMRUYMV]&YXVIWIEVGLMWGSQQMWWMSRIHSZIVWIIRVIGIMZIH

ERHɄGSQQYRMGEXIHF]ɄEɄ[MHIVERKISJTIVWSRRIP

 8LIKSZIVRQIRXWSGMEPVIWIEVGLGSQTIXIRG]JVEQI[SVOEREHHIRHYQXSXLIGMZMPWIVZMGIGSHI


HIƤRIWXLITVSJIWWMSREPGSQTIXIRG]JSVQIQFIVWSJXLIWSGMEPVIWIEVGLTVSJIWWMSR[MXLMRKSZIVRQIRX

ERHɄHIZSPZIHKSZIVRQIRXW3RVIEGLMRKKVEHIGLMIJVIWIEVGLSJƤGIVPIZIPEGMZMPWIVZERXMWI\TIGXIH

XSLEZIEXLSVSYKLERHHIXEMPIHORS[PIHKISJVIWIEVGLQIXLSHSPSK]ERHXLIIZMHIRGIFEWIMRXLIƤIPH

SJTSPMG]XLI][SVOMRERHXSXEOIVIWTSRWMFMPMX]JSVVIWIEVGLTYFPMGEXMSR

 18(40/(0651/(6,/(5&+$0*(59+$6,69$065)41/6+(4(5($4&+,06+(&1745(1)6+(241-(&6

*SVɄMRWXERGI(V'LVMWXMERZER7XSPOSJ6%2(I\TPEMRIHXLEXMRIZEPYEXMRKETSPMG]MQTEGXHITEVXQIRXW

QE]GLERKIXLIɄSYXGSQIQIEWYVIWFIGEYWISJGLERKIWMRKSZIVRQIRXTSPMG]QSZMRKSVWXEJJ

XYVRSZIVQEMRP]MRɄXLIɄGMZMPWIVZMGIXIEQHMVIGXMRKXLIVIWIEVGL
[LMGLGERPIEHXSWLMJXMRKTVMSVMXMIW

6&$0$.51+$22(06+$6$66+(21,061)27%.,&$6,10*18(40/(06&+$0*(56+()14/$6,69$0656127%.,5+

VIWYPXWMREPXLSYKLEGGSVHMRKXSZER7XSPOMXMWQSVIYWYEPP]XLIGEWIXLEXXLIVILEWFIIRPMXXPIXLSYKLX

SVɄHMWGYWWMSREFSYXXLEXJSVQEXYRXMPTYFPMGEXMSRMWMQQMRIRX7YGLGLERKIWXSXLIVIWIEVGLTVSXSGSPGER

GEYWIXIRWMSRERHHIPE][LMGLGSYPHFIEZSMHIHMJGSQQMWWMSRIVWSJVIWIEVGLERHVIWIEVGLIVW[SVOIH

GPSWIP]EXXLISYXWIXRSXSRP]XSWIXXLIɄVIWIEVGLUYIWXMSRFYXXSEKVIITYFPMGEXMSRTPERW

Uncertainty over what counts as research 

and differences in the interpretation of rules

 8LMWMRUYMV]LEWGSRWMHIVIHLS[VIWIEVGLHIƤRIHEWXLIGSPPIGXMSRERHIZEPYEXMSRSJHEXEMWɄTYFPMWLIH

-XMWRIGIWWEV]XSFIEVMRQMRHXLIHMWXMRGXMSRFIX[IIRWXVEXIKMGEHZMGI[LMGLGEVVMIWRSI\TIGXEXMSR

SJɄTYFPMGEXMSRERHXLIHEXEERHEREP]WMWXLEXYRHIVTMRMXERHSYKLXSVHMREVMP]XSɄGSQIJVSQSVɄFIɄTYX

MRɄXLITYFPMGHSQEMR


 5$,'

Ȋ>1Ζ(65GRHV@UHVHDUFKRQWKHH[SHFWDWLRQLWZLOOEHbSXEOLVKHG7KHUHPLJKW

EHDQbH[HPSWLRQIRUVWUDWHJLFDGYLFHEXWDOOSROLF\UHVHDUFKDQGbSURJUDPPH

HYDOXDWLRQVVKRXOGEHSXEOLVKHGȋ

(V7EVEL;SPPEWXSR14 5$,'

 Ȋ0LQLVWHUVQHHGWRDVNIRUbVWUDWHJLFDGYLFHEXWbWKHbSULQFLSOHLVWKDWSXEOLFIXQGV

SD\bIRUUHVHDUFKWKDWLQIRUPVWKDWDGYLFHVRVKRXOGEHbSXEOLFO\bDYDLODEOHȋ

 6,5(8,'(06)41/9+$66+(,037,4;9$561.'%16+%;/(/%(451)6+(4(5($4&+&1//70,6;$0'%;&,8,.

5(48$0656+$66+(4(,570&(46$,06;$%176+19(8,'(0&(5722.,('%;24,8$6(&1057.6$0&,(5,561%(+$0'.('

1057.6$065/$;2418,'(564$6(*,&$'8,&(%76/$;$.51&10'7&64(5($4&+!+(4(,5014($5109+;6+,5

WLSYPHGVIEXIGSRJYWMSRFIX[IIRVIWIEVGLERHGSRWYPXERG]IZIRMJXLIVIMWWSQIVSSQJSVɄSZIVPET

;LIVIXLMWQMKLXFIETTPMGEFPIGMZMPWIVZERXWWLSYPHIWXEFPMWLMREHZERGIXLEXVIWIEVGLƤRHMRKW

ERHɄWXVEXIKMGEHZMGI[MPPFITVSZMHIHMRWITEVEXIVITSVXW

 8LIVIMWJYVXLIVYRGIVXEMRX]SZIVLS[SXLIVX]TIWSJVIWIEVGLWYGLEWMRXIVREPJEGXƤRHMRKSVIZEPYEXMSR

WXYHMIWƤXMRXSXLITYFPMGEXMSRVYPIW'EWI7XYH]WLS[WXLEXVIWIEVGLMRXSXLIIJJIGXSJJVEGOMRK

SRɄLSYWITVMGIW[EWRSXTYFPMWLIHFIGEYWI(IJVEGSRWMHIVIHMXɄXSFIERMRXIVREPVITSVX;LIRMX[EW

VIPIEWIHMRVIWTSRWIXSɄERɄ)RZMVSRQIRXEP-RJSVQEXMSR6IUYIWXMX

9$5+($8,.;4('$&6('





KRZHYLGHQFHVXSSOLHG

E\bSULYDWHFRQVXOWDQFLHV

LVbWREHKDQGOHG

SECTION 3: REASONS FOR THE DELAYED PUBLICATION OR WITHHOLDING OF GOVERNMENT RESEARCH 

The Promise of Evidence-Based Policymaking 2113
	 (2$46/(061)($.6+10.;&105,'(45567',(56+$6/((6,06(40$6,10$..;4(&1*0,5('56$0'$4'5

EWɄVIWIEVGL8LMWQIERWXLEXGSRWYPXERG]ERHSXLIVJSVQWSJI\XIVREP[SVOEVII\GPYHIHJVSQ

27%.,&$6,1047.(5


 8LMWHMGLSXSQ]MWVIƥIGXIHMRVIWIEVGLGSRXVEGXW-XETTIEVWXLEXVIWXVMGXMSRWSR[LEXGERFITYFPMWLIH

KIRIVEPP]SRP]ETTP]XSGSQQIVGMEPP]WIRWMXMZISVTIVWSREPMRJSVQEXMSR[LMGLGERRSXMRɄTVMRGMTPI

FIɄSFNIGXMSREFPI;MXRIWWIWWXEXIHXLEXXLIYWYEPJSVQSJɄ[SVHWMWɄXLEXTIVQMWWMSRXSTYFPMWL

ŰWLEPPɄRSXɄFIYRVIEWSREFP][MXLLIPHű

 7SQIGSRXVEGXWEPWSVIWXVMGXXLIXMQMRKSJVIWIEVGLIVWŭEFMPMX]XSTYFPMWLJSVMRWXERGIF]EWOMRK

JSVEɄQSRXLŭWTVMSVRSXMGI-JEVIEWSREFP]WLSVXXMQIXSVIGIMZIERHVIEHGSQQMWWMSRIHVIWIEVGL

MWɄWTIGMƤIHMREHZERGIERHHSIWRSXVIWYPXMRXLITYFPMGFIMRKEWOIHXSXVYWXETSPMG]FEWIH

SRɄVIWYPXWMXGERRSXWIIXLMW[SYPHRSXERHWLSYPHRSXYWYEPP]FIGSRWMHIVIHEHIPE]MRTYFPMGEXMSR

8LIɄEGGITXEFMPMX]SJXLMWWLSVXVIEHMRKXMQI[MXLMRXLIHIƤRMXMSRSJTVSQTXTYFPMGEXMSRQMKLXEHHVIWW

51/(1)6+(&10&(405,057%/,55,1054$,5('+(4($0',0 (&6,10	$%17664$052$4(0&;',5&174$*,0*

XLIɄGSQQMWWMSRMRKSJVIWIEVGLMRXLIƤVWXTPEGI

 %XXIRXMSRLEWFIIRHVE[RXSGSRXVEGXWWYGLEWXLISRIMRTPEGIJSVXLIJSSHFEROWVIWIEVGL

MRɄ'EWIɄ7XYH][LMGLWYKKIWXXLEXHITEVXQIRXWS[RXLIVIWYPXWERHGERTVIZIRXTYFPMGEXMSR[MXLSYX

KMZMRKVIEWSRW-XMWYRGPIEV[LIXLIVXLMWMW[MHIWTVIEHTVEGXMGIERHMXRIIHWXSFIGPEVMƤIHXLEXVIWIEVGL

GSRXVEGXWEVIɄWYFNIGXXSSZIVVMHMRKTVMRGMTPIWERHTVSXSGSPWVIKEVHMRKTYFPMGEXMSR

 ;LMPITMPSXWERHVERHSQMWIHGSRXVSPPIHXVMEPWEVIMRGVIEWMRKP]FIMRKYWIHXSXIWXTSPMGMIW

WSQIɄHITEVXQIRXWERHEKIRGMIWETTIEVVIPYGXERXXSTYFPMWLXLIQIXLSHWERHVIWYPXW

8LMWɄLEWɄGVIEXIHɄXIRWMSR[LIVIXLSWIMRWMHIERHSYXWMHIKSZIVRQIRX[LSEVIGSQQMXXIHXSɄWXERHEVHW

SJɄJYPPERHɄTVSQTXTYFPMGEXMSRJEGIEGLSMGISJGSRHYGXMRKXVMEPWORS[MRKXLEXXLI]QE]RSXKIXTYFPMWLIH

SVɄMRWMWXMRKSRTYFPMGEXMSR[MXLXLIVMWOXLEXHITEVXQIRXW[MXLHVE[WYTTSVXJSVXLIɄXVMEPW6IWIEVGLIVW

[MWLMRKXSɄWGVYXMRMWIXLIXVMEPWWSQIXMQIWVIUYIWXXLIɄHEXEYRHIVXLIɄ*VIIHSQSJɄ-RJSVQEXMSR%GX

EWɄEɄPEWXVIWSVXFYXXLIWIHSRSXEP[E]WVIWYPXMRJYPPHMWGPSWYVIERHɄEVIRSXEɄWYFWXMXYXIJSVTYFPMGEXMSR

,S[IZIV[MXRIWWIWMRXLMWMRUYMV]EPWSWEMHXLEXXLIɄŬ;LEXɄ;SVOWŭGSPPEFSVEXMSRWWYGLɄEWɄMRɄIHYGEXMSR

LEZIWLS[RXLEXXVMEPVIWYPXWGERFITYFPMWLIHERHɄLERHPIH[IPP


 SECTION 3: REASONS FOR THE DELAYED PUBLICATION OR WITHHOLDING OF GOVERNMENT RESEARCH SECTION : R ASONS FOR THE DELAY D PUB ICATION OR WITHHOLD NG OF GO ERNM NT RE EARCH

2114 The Promise of Evidence-Based Policymaking
Section 4

REMEDIES AND 

RECOMMENDATIONS

SECTION 4: REMEDIES AND RECOMMENDATIONS 

The Promise of Evidence-Based Policymaking 2115
Awkward research does get published

 'SRXVEV]XSXLIMQTVIWWMSRGVIEXIHF]XLIFILEZMSYVMRGEWIW[LIVITYFPMGEXMSRLEWFIIRHIPE]IH

QER]ɄVIWIEVGLIVWERHGMZMPWIVZERXWEVIJEQMPMEV[MXLWXVEXIKMIWXLEXGERLIPTKSZIVRQIRX

XSɄGSQQMWWMSRERHTYFPMWLGSRXVSZIVWMEPVIWIEVGL[MXLSYXJIEVSJLS[XLITYFPMGSVXLIQIHME

QMKLXVIWTSRH7SQIɄI\EQTPIWEVIHIWGVMFIHFIPS[8LIWIMRGPYHITYFPMWLMRKQSVIMRJSVQEXMSR

EFSYXHITEVXQIRXEPVIWIEVGLETTSMRXMRKEHLSGGSQQMXXIIWXSSZIVWIIVIWIEVGLGSQQMWWMSRIVW

ERHɄVIWIEVGLIVW[SVOMRKGPSWIP]XSKIXLIVXSWIXUYIWXMSRWERHEKVIISRTYFPMGEXMSRXMQMRKERHɄTVSGIWW

ERHɄEPPS[MRKXLIɄVIEREP]WMWSJWIRWMXMZIHEXE8LIWII\EQTPIWMRJSVQXLIVIGSQQIRHEXMSRW

EXɄXLIɄIRHɄSJXLMWWIGXMSRXSɄMQTVSZIXLIZMWMFMPMX]ERHYXMPMX]SJKSZIVRQIRXVIWIEVGL

Publishing more information about government research

 8LI(ITEVXQIRXJSV)RZMVSRQIRX*SSHERH6YVEP%JJEMVW(IJVE
(ITEVXQIRXJSV-RXIVREXMSREP

(IZIPSTQIRX(ITEVXQIRXJSV8VERWTSVX(J8
ERH(ITEVXQIRXSJ,IEPXLLEZITYFPMGHEXEFEWIW

[LIVIVIWIEVGLMWVIKMWXIVIHERHTVSKVIWWXSTYFPMGEXMSRMWXVEGOIH8LIVIWLSYPHFIEWMRKPITYFPMGP]

WIEVGLEFPIHEXEFEWIYWIHF]EPPHITEVXQIRXW6+$6%7,.'5106+(/15675()7.)($674(51)6+(5(

'(2$46/(06$.5;56(/5(&1//(0'$6,10!+,5917.'(0$%.(2(12.(611%6$,0,0)14/$6,10$%176

TPERRIHVIWIEVGLEW[IPPEWXSPSGEXIERHYWIXLIVITSVXWSJGSQTPIXIH[SVO

 (ITEVXQIRXWQMKLXEPWSGSRWMHIVEVVERKIQIRXWWYGLEWXLSWIYWIHF]XLI;IPWLKSZIVRQIRX

MRɄMXWɄ)QIVKMRK*MRHMRKW6ITSVXWXSQEOIIEVP]VIWIEVGLƤRHMRKWSVMRXIVMQƤRHMRKWEZEMPEFPI

[MXLɄXLIɄRIGIWWEV]GEZIEXWEFSYXTIIVVIZMI[MRGIVXEMRGEWIW[LIVIXLIVIMWETVIWWMRKRIIH

JSVɄVIWIEVGLXSɄGSRXVMFYXIXSGYVVIRXTSPMG]HMWGYWWMSRW

 Publicly searchable research databases

 ()4$>5 &,(0&( ($4&+9(%5,6(+$5$27%.,&.;5($4&+$%.('$6$%$5(1)$..4(5($4&+&1//,55,10('

JVSQɄXLMVHTEVXMIW8LMWMWFEWIHSR[LEX(IJVEWXEJJ[LSEVIGMZMPWIVZERXWIRXIVMRXS3QRMGSQ

XLIMVɄMRXIVREPHEXEFEWIJSVQEREKMRKEPPMRJSVQEXMSRVIPEXIHXSVIWIEVGL-RXIVREPUYEPMX]EWWYVERGI

ERHEYHMXMWYRHIVXEOIRF]XLI3QRMGSQWYTTSVXXIEQXSIRWYVIXLEX(IJVEVIWIEVGLQEREKIVW

ERHɄTVSGYVIQIRXWXEJJEVIYWMRKXLIHEXEFEWIERHGSQTPIXMRKMRJSVQEXMSRETTVSTVMEXIP]3QRMGSQ

MWEPWSYWIHEWɄEɄHSGYQIRXVITSWMXSV]XSLIPTXLIHITEVXQIRXXVEGOTVITVSNIGXMHIEWTVSGYVIQIRX

HSGYQIRXWWYGLɄEWGSRXVEGXWQIIXMRKRSXIWHYVMRKXLIPMZIWXEKISJXLITVSNIGXERHVITSVXW

 3QRMGSQEPPS[WXLIHITEVXQIRXXSVIZMI[MXW[LSPIVIWIEVGLPERHWGETI-RXIRHIHTYFPMGEXMSRHEXIMW

LS[IZIVRSXEƤIPHMRXLIW]WXIQEPXLSYKLWXEVXERHƤRMWLHEXIWEVIŪEPPHIPMZIVEFPIWEVIXSFIƤREPMWIH

F]ƤRMWLHEXIEJXIV[LMGLTIIVVIZMI[ERHSXLIVUYEPMX]EWWYVERGIEGXMZMXMIWWXEVX

	 Mapping research on to government policy

 69$5)14/(4.;6+(24$&6,&(6+$6*18(40/(06'(2$46/(065917.'27%.,5+$241-(&6,106+$65+19('9+$6

VIWIEVGL[EWTPERRIHSZIVXLIGSQMRK]IEVEPSRKXLIPMRIWSJXLI,SQI3JƤGIŭWVIWIEVGLTVSKVEQQI

!+,524$&6,&($22($4561+$8(.$4*(.;(0'(')1..19,0*70&(46$,06;18(4'(2$46/(06$.4(5($4&+

FYHKIXWMRɄ4YFPMWLMRKEJSV[EVHPSSO[SYPHIREFPISXLIVWIGXSVWERHHITEVXQIRXWXLIVIWIEVGL

GSQQYRMX]TEVPMEQIRXERHXLITYFPMGXSWIITPERRIHVIWIEVGLEPSRKWMHIXLIKSZIVRQIRXŭWTSPMG]

TVMSVMXMIW8LI;IPWLKSZIVRQIRXŭW*SVXLGSQMRK4YFPMGEXMSRW[IFTEKIGSRXMRYIWXSJYPƤPPXLMWJYRGXMSR

 %PXLSYKLTSPMG]SJXIRQSZIWJEWXIVXLERVIWIEVGLXLMWHSIWRSXQIERXLEXXLITYFPMGGERRSX

WGVYXMRMWIXLII\XERXIZMHIRGIFEWI8LI;IPWLKSZIVRQIRXLEWHIZIPSTIH)QIVKMRK*MRHMRKW6ITSVXW

[LMGLXLI]EWOVIWIEVGLIVWERHHITEVXQIRXWXSTVITEVI[LIRERYVKIRXTSPMG]ERRSYRGIQIRXRIIHW

XSɄFIQEHIFIJSVIJYPPVIWIEVGLƤRHMRKWEVIEZEMPEFPI

 SECTION 4: REMEDIES AND RECOMMENDATIONS

2116 The Promise of Evidence-Based Policymaking
Ensuring the independent commissioning and publication of government research

 %RYQFIVSJMRHMZMHYEPWKMZMRKIZMHIRGIXSXLMWMRUYMV][EVRIHXLEXEFPEROIXGEPPJSVEPPKSZIVRQIRX

GSQQMWWMSRIHVIWIEVGLXSFISTIRP]TYFPMWLIHQMKLXGLMPPTSPMG]QEOIVWŭ[MPPMRKRIWWXSGSQQMWWMSR

VIWIEVGL8LMWHSIWRSXETTIEVXSLEZIFIIRXLIGEWI[LIVIEVIWIEVGLVIKMWXIVMWMRSTIVEXMSR

IKɄEXɄ(IJVEERHEPPVIWIEVGLMWXVEGOIHMRTYFPMGZMI[

 6+$5%((057**(56('6+$610(9$;61$..(8,$6(6+(24(5574(5106+(&1//,55,10,0*$0'27%.,&$6,10

SJKSZIVRQIRXVIWIEVGL[SYPHFIXSWITEVEXIXLIEREP]XMGEPERHTSPMG]QEOMRKJYRGXMSRWSJIZIV]

'(2$46/(06

.SREXLER4SVXIW2EXMSREP-RWXMXYXISJ)GSRSQMGERH7SGMEP6IWIEVGLWEMH

ȊΖI\RXWHOOPLQLVWHUVWKDWUHVHDUFKZLOODXWRPDWLFDOO\EHSXEOLVKHGRQDȴ[HG

WLPHWDEOHZLWKQRVFRSHIRUGHOD\VRUbLQWHUIHUHQFHWKHQZLWKKRQRXUDEOH

H[FHSWLRQVWKH\ZLOOFRPPLVVLRQPXFKOHVVRIbLWbRUbQRQH$QGbLQbFXUUHQWEXGJHWDU\

FLUFXPVWDQFHVWKHUHZLOOEHQRSDUWLFXODUSHQDOW\IRUWKLV7KLVWUDGHR΍VLPSO\

FDQQRWEHUHVROYHGH[FHSWE\PDNLQJWKHFRPPLVVLRQLQJRIUHVHDUFKLQGHSHQGHQWȐ

LWȇVbJHQHUDOO\UHFRJQLVHGLQWHUQDWLRQDOO\DQGLVWKHUHDVRQWKDWWKHΖ0)DQG:RUOG

%DQNVD\KDYHbVWURQJFRQVWLWXWLRQDOO\LQGHSHQGHQWHYDOXDWLRQRɝFHV7KH8.

QHHGVWKHVDPHIRUHDFKPDMRUGHSDUWPHQWWKHUHFRXOGEHVRPHJURXSLQJZLWK

LQGHSHQGHQWKHDGVQRWIURPJRYHUQPHQWVRPHSURWHFWLRQRIbEXGJHWVDWOHDVW

WKUHHRUȴYH\HDUEXGJHWVWKHIUHHGRPWRHYDOXDWHZKDWWKH\VHHȴWDQGSXEOLVK

WKHUHVXOWVDQGbDbGLUHFWUHSRUWLQJOLQHWR&RPPRQVGHSDUWPHQWDOFRPPLWWHHV

¢bODb1$2>1DWLRQDO$XGLW2ɝFH@7KHUHLVQRWKLQJSDUWLFXODUO\UDGLFDOFRPSOLFDWHG

RUbGLɝFXOWbDERXWDQ\RIWKLVH[FHSWWKDWPLQLVWHUVZRXOGQȇWOLNHLWȋ

 8LIVIMWWSQIWYTTSVXJSVXLITVMRGMTPISJWITEVEXMRKVIWIEVGLJVSQTSPMG]QEOMRKEQSRKWXQIQFIVW

SJXLIVIWIEVGLGSQQYRMX]GMZMPWIVZERXWERHTSPMXMGMERWFYXXLIWITVSTSWEPWVEMWIXLIUYIWXMSRSJ[LS

917.'6+(0&1//,55,106+(4(5($4&+ 7&+$5(2$4$6,10/,*+6$.51%4,0*,06137(56,106+(4(.$6,105+,2

FIX[IIRTSPMG]QEOIVW[LSGSQQMWWMSRVIWIEVGLERHEGEHIQMGWGEVV]MRKMXSYXERHMRHIIHMXQMKLX

,65(.)+$8(6+(())(&61)4('7&,0*21.,6,&,$05>,06(4(56,0&1//,55,10,0*$0'2$;,0*)144(5($4&+

 016+(457**(56,10/$'(616+,5,037,4;9$56+$64(5($4&+(45&17.'4(2146',4(&6.;612$4.,$/(06

-XɄMWɄYRHIVWXSSHXLEXXLIJSVQIVLIEHSJ[LEX[EWXLIRXLI,SQI3JƤGI6IWIEVGL9RMXGSRGPYHIH

MRɄXLIɄWXLEXXLMWWLSYPHFIXLIRSVQWSEWXSTVIWIVZIXLIMRHITIRHIRGISJVIWIEVGL;LMPIFSXL

SJXLIWYKKIWXMSRWEFSZIEVI[SVXL]SJGSRWMHIVEXMSRXLI]MQTP]ERYRPMOIP]QENSVSZIVLEYPSJEW]WXIQ

6+$6&17.'/((627%.,&$6,104(37,4(/(065/7&+/14(5,/2.;

Appointing ad hoc committees 

 %TTSMRXMRKEHLSGGSQQMXXIIWXSSZIVWIITSXIRXMEPP]HMJƤGYPXVIWIEVGLGERFIIWTIGMEPP]YWIJYPFIGEYWI

,6$..195*18(40/(066121,0661,0'(2(0'(06(8,'(0&($0'615(6176,6521.,&;4(52105(5(2$4$6(.;

,IVIɄEVIWSQII\EQTPIW

 Biofuels and food crops

 -R%TVMPXLIKSZIVRQIRXŭW6IRI[EFPI8VERWTSVX*YIP3FPMKEXMSRGSQQMXXIHXLI9/XSIRWYVMRK

XLEXFMSJYIPWQEOIYT	F]ZSPYQISJVSEHXVERWTSVXJYIPWEPIWMRGVIEWMRKF]	E]IEV

XSɄ	ɄF]Ʉ8LMW[EWEVIWYPXSJER)9(MVIGXMZIMRTVSQSXMRKXLIYWISJFMSJYIPW

SECTION 4: REMEDIES AND RECOMMENDATIONS 

The Promise of Evidence-Based Policymaking 2117
 )ZMHIRGILEHIQIVKIHXLEXVEMWIHŰGSRGIVREFSYXXLIVSPISJFMSJYIPWMRVMWMRKJSSHTVMGIWEGGIPIVEXMRK

HIJSVIWXEXMSRERHHSYFXWEFSYXXLIGPMQEXIFIRIƤXWű-R*IFVYEV]ETETIVF]7IEVGLMRKIVIXEPMR

7GMIRGIWYKKIWXIHXLEXMRXLI97FMSJYIPWGEYWIHPERHYWIGLERKIWXLEXPIHXSMRGVIEWIHRIXIQMWWMSRW

1)*4((0+175(*$5(5	!+,5241/26('6+(&+,()5&,(06,5651)()4$$0')!61&1//,55,10$4(8,(9

SJXLIIZMHIRGIFILMRHXLITSPMG]SJMRGVIEWMRKFMSJYIPYWIERHMXWMRHMVIGXIJJIGXWERH[LIXLIVXLIVI

5+17.'%($/14$614,7/

 8LIVIZMI[[EWGLEMVIHF]4VSJIWWSV)H+EPPEKLIVGLEMVSJXLI6IRI[EFPI*YIPW%KIRG]

XLIɄMRHITIRHIRXEKIRG]GVIEXIHXSMQTPIQIRXXLITSPMG]
-XGSRGPYHIHXLEX[LMPIXLIVI[EWRSRIIH

JSVɄEQSVEXSVMYQERHXLEXŰXLIVIMWEJYXYVIJSVEWYWXEMREFPIFMSJYIPWMRHYWXV]űXLIVEXISJMRGVIEWI

MRɄXLIɄYWISJFMSJYIPWWLSYPHFIVIHYGIHXSEZSMHEKVMGYPXYVEPPERHFIMRKHMWTPEGIH

	 &IGEYWIXLI9/[EWGSQQMXXIHXSEVETMHMRGVIEWIMRXLIYWISJFMSJYIPWERHFIGEYWIXLIMQTEGX

SRɄJSSHGVSTW[EWGSRXVSZIVWMEPXLIVIWIEVGLERHVIGSQQIRHEXMSRWGSYPHLEZIFIIRE[O[EVH

JSVXLIɄKSZIVRQIRX-XETTIEVWXLEXFIGEYWISJEGSPPEFSVEXMZIIJJSVXFIX[IIRVIWIEVGLIVWERHGMZMP

WIVZERXWXSIRWYVIXLEXGSYRXIVIZMHIRGIERHYRGIVXEMRX][IVITVSTIVP]EHHVIWWIHHYVMRKXLIVIWIEVGL

MX[EWTSWWMFPIXSTYFPMWLXLIƤREPVITSVX[MXLMRWM\QSRXLWSJXLIVIWIEVGLFIMRKGSQQMWWMSRIH


 The effect of mobile phones on health

 1IHMEWXSVMIWERHGEQTEMKRKVSYTWGPEMQIHXLEXQSFMPITLSRIWERHXVERWQMXXIVQEWXW[IVIGEYWMRK

GERGIVHIWTMXIEFEPERGISJWGMIRXMƤGIZMHIRGIXSXLIGSRXVEV]%RMRHITIRHIRXI\TIVXKVSYTSRQSFMPI

TLSRIWLEHGSRGPYHIHMRXLEXEPXLSYKLQSFMPITLSRIWERHXVERWQMXXIVWEVIYRPMOIP]XSGEYWI

HMWIEWIJYVXLIVVIWIEVGL[EWVIUYMVIHXSEHHVIWWTSWWMFPIFMSPSKMGEPIJJIGXWSJPS[PIZIPVEHMSJVIUYIRG]

VEHMEXMSR8LI1SFMPI8IPIGSQQYRMGEXMSRWERH,IEPXL6IWIEVGL18,6
TVSKVEQQI[EWWIXYT

MRɄXSGSQQMWWMSRERHSZIVWIIXLMWVIWIEVGLNSMRXP]JYRHIHF]KSZIVRQIRXERHXLIQSFMPI

TLSRIɄMRHYWXV]

 8LI18,6ŭWVIWIEVGLWTERRIH]IEVWERHGSRGPYHIHXLEXXLIVI[EWRSEWWSGMEXMSRFIX[IIR

QSFMPITLSRIYWIERHGERGIV8LI18,6ŭWGSQQMXXII[EWIJJIGXMZIMRGSQQMWWMSRMRKGSRXVSZIVWMEP

ERHɄGSQTPI\VIWIEVGLERHGSQQYRMGEXMRKMXXSXLITYFPMG8LIMRXIVTSWMRKSJERMRHITIRHIRX

GSQQMXXIILIPTIHKSZIVRQIRXXSTYFPMWLVIPMEFPIVIWIEVGLMREJVEQI[SVOXLEX[EWVIGSKRMWIH

F]ɄGSQQIRXEXSVWEWMRHITIRHIRX

 The Depleted Uranium Oversight Board

 8LI1MRMWXV]SJ(IJIRGIWIXYTXLMWMRHITIRHIRXFSEVHXSSZIVWIIXLIHIZIPSTQIRXSJEPEFSVEXSV]XIWX

XSGLIGOJSVI\TSWYVIXSHITPIXIHYVERMYQ(9
EQSRKWXZIXIVERWSJXLI+YPJ;EVERHSXLIVGSRƥMGXW

MRɄXLIIEVP]W8LIFSEVHMRGPYHIHIWXEFPMWLIHWGMIRXMWXWZIXIVERWŭKVSYTWERHGEQTEMKRIVW

XSɄIRWYVIXLEXEFVSEHVERKISJZMI[W[EWGSRWMHIVIH-XGSRGPYHIHXLEXXLIVI[EWRSHIXIGXEFPI

I\TSWYVIXS(98LIVIWIEVGL[EWTYFPMWLIH[MXLSYXHIPE]EPSRKWMHIEQMRSVMX]VIZMI[XLEXUYIWXMSRIH

6+(8$.,',6;1)6+(6(56

Research commissioners and researchers working together to set questions 

and agree on publication

 Organisational preparedness for unconventional terrorist attacks

 6IWIEVGLMRXSLS[TVITEVIHTYFPMGERHTVMZEXISVKERMWEXMSRWEVIXSVIWTSRHXSGLIQMGEPFMSPSKMGEP

VEHMSPSKMGEPSVɄRYGPIEVXIVVSVMWXMRGMHIRXWLEWFIIRTYFPMWLIHTVSQTXP]ERHMRJYPPRSX[MXLWXERHMRK

GSRGIVRWXLEXXLIɄƤRHMRKWSVIZIRXLIUYIWXMSRWGSYPHFIWIIREWMHIRXMJ]MRK[IEORIWWIWMRREXMSREP

WIGYVMX]6IWIEVGLIVWɄGMZMPWIVZERXWERHɄSXLIVWXEOILSPHIVWQIXXSHMWGYWWXLIUYIWXMSRWXSFIEWOIH

VIWIEVGLQIXLSHWERH[E]WXSɄGSQQYRMGEXIƤRHMRKWXSXLITYFPMG7YGLIZIRXWETTIEVXSLIPTFYMPH

XVYWXFIX[IIREGEHIQMGWERHXLSWIGSQQMWWMSRMRKɄVIWIEVGL

	7IEVGLMRKIV8IXEP


 SECTION 4: REMEDIES AND RECOMMENDATIONS

2118 The Promise of Evidence-Based Policymaking
Allowing reanalysis of sensitive data

 ,16IZIRYI
'YWXSQWŭ(EXE0EFERHXLI(ITEVXQIRXJSV)HYGEXMSRŭW2EXMSREP4YTMP(EXEFEWIKMZI

ETTVSZIHVIWIEVGLIVWEGGIWWXSWIRWMXMZIHEXE[LMGLXLIVIWIEVGLIVWGERXLIREREP]WIERHTYFPMWL

8LIɄ)HYGEXMSR)RHS[QIRX*SYRHEXMSRTYFPMWLIWMRE[E]XLEXEPPS[WVIWIEVGLIVWXSVIEREP]WI

XLIɄVIWYPXWJVSQXLIMVXVMEPW7YGLɄW]WXIQWIRWYVIXLEXTIVWSREPP]WIRWMXMZIHEXEGERFIYWIH

Principles and recommendations

 -RXLIPMKLXSJXLIQEXIVMEPWYQQEVMWIHMRXLMWVITSVXXLIJSPPS[MRKFVSEHTVMRGMTPIWEVIGSQQIRHIH

  4VSQTXERHJYPPTYFPMGEXMSRSJKSZIVRQIRXVIWIEVGLMWEQEXXIVRSXSJGSRXVEGXFYXSJɄTYFPMG

HYX];LMPIVIWIEVGLGSRXVEGXW[MPPRIGIWWEVMP]ZEV]MRXLIMVTVSZMWMSRWEPPGSRXVEGXWWLSYPH

WTIPPSYXXLMWSFPMKEXMSRSJTVMRGMTPI[LMGLVIƥIGXWXLIHITEVXQIRXEPVYPIWKSZIVRMRK

I\XIVREPɄVIWIEVGL

  2SVIHEGXMSRWWLSYPHFIQEHIMRTYFPMWLIHVIWIEVGLI\GITXJSVZIVMƤEFPIPIKEPSVWIGYVMX]

4($510557&+$5'$6$2416(&6,10140$6,10$.5(&74,6;

  7EZIMR[LSPP]I\GITXMSREPGMVGYQWXERGIWTYFPMGEXMSRSJI\XIVREPVIWIEVGLWLSYPHTVIGIHISV

EXɄPEXIWXEGGSQTER]TVSQYPKEXMSRSJER]TSPMG]MRMXMEXMZI[LMGLMWTVIWIRXIHEWHITIRHIRXSRMX

149+,&+&.($4.;,5,06(0'('61%(

  'SRƥMGX[MXLGYVVIRXSVMQTIRHMRKTSPMG]MRMXMEXMZIWMWRSXEREGGITXEFPIVIEWSRJSVHIPE]MRK

SVɄ[MXLLSPHMRKTYFPMGEXMSRSJI\XIVREPVIWIEVGL-RWYGLWMXYEXMSRWKSZIVRQIRXWLSYPH

FIɄTVITEVIHXSTYFPMWLMXWVIEWSRWJSVHMWEKVIIQIRXERHPIXER]HIFEXIFIEMVIHMRTYFPMG

 
 18(40/(06$0'4(5($4&+(455+17.'$'126/14(1)6+(24$&6,&(5'(5&4,%(',06+(24(&(',0*

TEVEKVETLWMRGSQQMWWMSRMRKGSRHYGXMRKERHGSQQYRMGEXMRKTVSFPIQEXMGEPVIWIEVGL

 8LIVIGSQQIRHEXMSRWXLEXJSPPS[WIIOXSEZSMHER]QIEWYVIWXLEX[SYPHMRLMFMXQMRMWXIVWERHGMZMP

WIVZERXWJVSQGSQQMWWMSRMRKMRXLIƤVWXTPEGIVIWIEVGLMRXSTSPMXMGEPP]WIRWMXMZIMWWYIWSVVIWIEVGLXLEX

QMKLXKIRIVEXIE[O[EVHƤRHMRKW)ZIRWSMXMWVIGSKRMWIHXLEXXLIVI[MPPFISGGEWMSREPVIWIEVGLXLEX

GERRSXFITYFPMWLIHSVGERRSXFITYFPMWLIHMRJYPP
JSVI\EQTPIJSVVIEWSRWSJREXMSREPWIGYVMX]

Recommendation I A standardised central register of all externally 

commissioned government research

 &YMPHMRKSRI\MWXMRKHITEVXQIRXEPVIWIEVGLHEXEFEWIWXLIKSZIVRQIRXWLSYPHMRWXMXYXI

EɄW]WXIQɄMRɄ[LMGL

 E
 %WIEVGLEFPIPMWXSJGSQQMWWMSRIHI\XIVREPVIWIEVGLMWTYFPMWLIHERHQEMRXEMRIHMRGPYHMRK

[LSMWGSQQMWWMSRMRKERHGSRHYGXMRKXLIVIWIEVGL[LEXMWFIMRKQIEWYVIH[LEXQIXLSHW

EVIFIMRKYWIHERH[LEXSYXGSQIWEVII\TIGXIHERHEXMQIXEFPIJSVGSQTPIXMSRTIIVVIZMI[

ERHɄTYFPMGEXMSRSJXLIVIWIEVGL

 F
 )EGLWXYH]LEWEYRMUYIMHIRXMƤIVXLEXGSRJSVQWXSSTIRHEXEWXERHEVHWWSXLEXER]

EQIRHQIRXWXSXLIWIƤIPHWHYVMRKXLIGSYVWISJXLIVIWIEVGLGERFIXVEGOIHERH[MPP

FIɄEGGSQTERMIHF]ERI\TPEREXMSRMWPMROIHXSEWWSGMEXIHHEXEWIXWERHMWEWWSGMEXIH

[MXLɄGSQTVILIRWMZIQIXEHEXE'SRWMHIVEXMSRWLSYPHFI

*,8(0616+(5(,557(5,0)7674(',5&755,105$%176'(6$,.1)

$4(*,56(41)4(5($4&+

Recommendation I A standardised central register of all externally

commissioned government research



3URPSWDQGIXOOSXEOLFDWLRQ

RIbJRYHUQPHQWUHVHDUFK

LVbDPDWWHUQRWRIbFRQWUDFW

EXWbRIbSXEOLFbGXW\

SECTION 4: REMEDIES AND RECOMMENDATIONS 	

The Promise of Evidence-Based Policymaking 2119
 0$'',6,10

 &  7&+$4(*,56(4917.'(0$%.('(2$46/(06561$00170&(9+$66+(;2.$06127%.,5+

MRɄXLIɄGSQMRKɄQSRXLW

 H
 1MRMWXIVWWLSYPHEVVERKIJSVTEVPMEQIRXXSFIXSPHJSVI\EQTPIF]ɄTPEGMRKXLIMRJSVQEXMSR

MRɄXLIɄPMFVEVMIWSJXLI,SYWISJ'SQQSRWERHXLIɄ,SYWISJ0SVHW[LEXVIWIEVGLLEWFIIR

GSQQMWWMSRIHMRɄIEGLUYEVXIVERH[LEXVIWIEVGLLEWFIIRVIGIMZIHERHɄTYFPMWLIH

'SRWYPXEXMSRWYKKIWXWXLEXXLIJYRGXMSRSJQEMRXEMRMRKEGIRXVEPVIKMWXIV[SYPHFIQSWXETTVSTVMEXIP]

YRHIVXEOIRF]SVYRHIVXLIEYWTMGIWSJXLIɅ'EFMRIX3JƤGI

 *VSQXLITIVWTIGXMZISJXLMWMRUYMV]XLIFIRIƤXWSJEVIKMWXIV[SYPHFI

@7%.,&&.$4,6;1021.,&;,557(5$0'$&&1706$%,.,6;)1427%.,&)70'5

@-QTVSZIQIRXWMRXLIXVEGOMRKERHTVSQTXTYFPMGEXMSRSJVIWIEVGL

@8$,.$%,.,6;1)4(5($4&+616+(18(4$..4(5($4&+%$5($0'

@'SRƤHIRGIMRXLIW]WXIQSJYRHIVXEOMRKVIWIEVGLJSVKSZIVRQIRXEQSRKMRHITIRHIRX

VIWIEVGLɄSVKERMWEXMSRW

 %VIKMWXIVWLSYPHXLIVIJSVIMRGPYHIEPPVIWIEVGLGSRHYGXIHF]MRHITIRHIRXSVKERMWEXMSRW

ERHɄER]ɄVIWIEVGLXLEXMWɄGSRHYGXIHF]EVQŭWPIRKXLFSHMIWERHSXLIVHMVIGXP]IQTPS]IHEKIRGMIW

[LMGLɄKSZIVRQIRXQMKLX[MWLXLITYFPMGXSZMI[EWSFNIGXMZIERHMRHITIRHIRX

 8LIWYFQMWWMSRWXSXLMWMRUYMV]ERHMRHIIHXLIJVYWXVEXMSRI\TVIWWIHF]TISTPIMRKSZIVRQIRX

'(2$46/(0659+15647**.('612418,'(759,6+,0)14/$6,10$%1766+((:6(06619+,&+4(5($4&+

MWɄTYFPMWLIHTVSQTXP]WLS[XLEXEVIKMWXIVSJGSQQMWWMSRIHVIWIEVGL[SYPHTVSZMHIEVERKI

SJɄFIRIƤXWɄXSɄKSZIVRQIRXMXWIPJERHXSXLSWI[LS[SVO[MXLKSZIVRQIRXMRGPYHMRK

@1565$8,0*%;$81,',0*4(2(6,6,10

@'SWXWEZMRKXLVSYKLMRGVIEWIHYWISJI\MWXMRKVIWSYVGIWJSVTSPMG]QEOMRK

@'SWXIJJIGXMZIRIWWXLVSYKLMQTVSZIHERHGSQFMRIHYWISJVIWIEVGLƤRHMRKW

@4*$0,5$6,10$./(/14;

@-QTVSZMRKXLIQIXLSHWERHHEXESJXLII\MWXMRKORS[PIHKIFEWI[LIRYRHIVXEOMRKRI[VIWIEVGL

@4155'(2$46/(06$.&114',0$6,10$0'5+$4,0*1)4(5174&(5

 %QSVIGSQTVILIRWMZIVIKMWXIV[SYPHIRWYVIXLEXKSZIVRQIRXSFXEMRWXLIWIFIRIƤXWXSS

Recommendation II Clarity on what constitutes externally commissioned 

government research

 8SKMZITVEGXMGEPIJJIGXXSXLIHMWXMRGXMSRWTIPPIHSYXMRTEVEKVETLNSMRXKYMHERGIWLSYPHFIɄMWWYIH

F]ɄKSZIVRQIRXŭWLIEHWSJEREP]WMW8LMW[SYPHI\TPEMRJSVXLIFIRIƤXSJEPPHITEVXQIRXWLS[ɄXSɄWITEVEXI

VIWIEVGLJVSQEHZMGIIWTIGMEPP][LIRFSXLJIEXYVIMRXLIWEQIHSGYQIRXMRSVHIVXSɄIRWYVIXLITVSQTX

27%.,&$6,101)6+()14/(4

Recommendation III A clear commitment to prompt publication 

in research contracts

 ;LMPIGSRXVEGXW[MPPRIGIWWEVMP]ZEV]MRXLIMVTVSZMWMSRWEPPVIWIEVGLGSRXVEGXWRIIHXSEGORS[PIHKIXLEX

EWVYPIWERHTVSXSGSPWVIGSKRMWITVSQTXERHJYPPTYFPMGEXMSRSJKSZIVRQIRXVIWIEVGLMWXLIRSVQ

Recommendation II Clarity on what constitutes externally commissioned

government research

Recommendation III A clear commitment to prompt publication

in research contracts


 SECTION 4: REMEDIES AND RECOMMENDATIONS

2120 The Promise of Evidence-Based Policymaking
Recommendation IV Routine publication of research the government has 

considered in policy formulation, with, if appropriate, 

reasons for rejecting it

 ;LIRTYFPMWLMRKTSPMG]KSZIVRQIRXHITEVXQIRXWWLSYPHTVSZMHIEVIJIVIRGIERHVIPIZERXPMROW

XSɄVIWIEVGLIZMHIRGIXLEXLEWFIIRGSRWMHIVIHERHMJXLITSPMG]GSRƥMGXW[MXLXLIIZMHIRGIERSXI

(:2.$,0,0*9+;6+((8,'(0&(9$54(-(&6('

 ;LIVIVIWIEVGLMWGSRWMHIVIHXSFIWYFWXERHEVHTYFPMGEXMSRQE]FIHIPE]IHJSVPSRKIRSYKLXSLEZI

MXɄMRHITIRHIRXP]EWWIWWIH-JTSSVUYEPMX]MWGSRƤVQIHXLIHIJEYPXTSWMXMSRWLSYPHFITYFPMGEXMSR

[MXLɄEɄGVMXMUYI

 ;LIVITSPMG]VYRWELIEHSJVIWIEVGLMRXIVMQVIWIEVGLƤRHMRKWWLSYPH[LIVITVEGXMGEFPIFITYFPMWLIH

 )\XIVREPVIWIEVGL[LIRWYFQMXXIHWLSYPHFITVSQTXP]QEHIEGGIWWMFPISRPMRIF]XLIHITEVXQIRX

SVEKIRG]GSQQMWWMSRMRKMX8LMWWLSYPHFIRSXIHSRER]VIKMWXIV*YXYVITYFPMGEXMSRMRJSVI\EQTPI

WTIGMEPMWXNSYVREPWWLSYPHRSXFIVIKEVHIHEWEWYFWXMXYXIJSVTVSQTXSRPMRIEGGIWW[LIVIEƤREPVITSVX

+$5%((0/$'(616+(*18(40/(06

Recommendation V A clear statement of the current requirements for prompt 

publication and adherence to them

 (MWGVITERGMIWFIX[IIRXLIYRHIVWXERHMRKERHTVEGXMGIWSJHMJJIVIRXHITEVXQIRXWERH[MXLMR

HITEVXQIRXWLEZIVIGYVVIHXLVSYKLSYXXLIIZMHIRGILIEVHERHWYFQMXXIHXSXLMWMRUYMV]1MRMWXIVW

WLSYPHFIEWOIHXSGSRƤVQXLEXXLIMVHITEVXQIRXWYRHIVWXERHERHEHLIVIXSXLIGYVVIRXVYPIWSYXPMRIH

MR7IGXMSRVIKEVHMRKTVSQTXTYFPMGEXMSRSJXLMVHTEVX]VIWIEVGL-HIEPP]SRIWIXSJVYPIWERHTVMRGMTPIW

WLSYPHFIETTPMGEFPIEGVSWWEPPSJKSZIVRQIRXFYXEXXLIZIV]PIEWXHITEVXQIRXWWLSYPHFIHMVIGXIH

XSɄWYFQMXER]EHHMXMSREPKYMHIPMRIWXLI]LEZITVSHYGIHXSMRXIVREPERHI\XIVREPWGVYXMR]ERHETTVSZEP

XSɄIRWYVIEHLIVIRGIXSXLIWIVYPIWERHTVMRGMTPIW

Recommendation VI Training in research for policy communicators

 %TVSKVEQQISJXVEMRMRKERHFVMIƤRKWLSYPHFIIWXEFPMWLIHEWTEVXSJXLIMRHYGXMSRSJQMRMWXIVW

WTIGMEPɄEHZMWIVWERHGSQQYRMGEXMSRWSJƤGMEPWXSGSZIVXLIVYPIWVIKEVHMRKTYFPMGEXMSRSJGSQQMWWMSRIH

VIWIEVGLERHMRWSQIGEWIWXVEMRMRKMRLS[XSLERHPIVIWIEVGLGSQQYRMGEXMSRGSRƤHIRXP]ERHSTIRP]

,0$21.,6,&$.&106(:6

 ;LMPIXLIVIMWGSRWMHIVEFPIVIWIEVGLI\TIVMIRGIMRKSZIVRQIRXXLMWMWRSXRIGIWWEVMP]HITPS]IH

MRɄXLIɄGSQQYRMGEXMSRSJVIWIEVGL[LMGLMWSJXIRYRHIVXEOIRF]XLSWI[MXLEQSVITSPMXMGEPVSPISV[LS

EVIQSVIGSRGIVRIH[MXLGSQQYRMGEXMSRERHRIKEXMZITYFPMGMX]*YVXLIVQSVIXLIVIWIIQWXSFIPMXXPI

5+$4,0*14$224(&,$6,10$&4155*18(40/(061)6+15(&$5(59+(4('(2$46/(065+$8(+$0'.('4(5($4&+

TYFPMGEXMSR[IPPERHIRKEKIHMREVIWTIGXJYPERHSTIRHMWGYWWMSR[MXLXLITYFPMGERHLS[XLMWLEW

FIIRɄEGLMIZIH

 4($6(4&4155'(2$46/(06$.5+$4,0*1)4(5($4&+&1//70,&$6,10(:2(4,(0&(55+17.'%((56$%.,5+('

!+,5/,*+6%($&+,(8('6+417*+6+((:,56,0*'(2$46/(06$.+($'51)$0$.;5,514$016+(4%1';

Recommendation V A clear statement of the current requirements for prompt

publication and adherence to them

Recommendation VI Training in research for policy communicators

SECTION 4: REMEDIES AND RECOMMENDATIONS 

The Promise of Evidence-Based Policymaking 2121
8LIƤRHMRKWSJ7MV7XITLIRŭWMRUYMV]QE]VIEWWYVIWSQI

2(12.(,65((/56+(4(,50>6(8,'(0&(6+$64(5($4&+

MWɄJVIUYIRXP][MXLLIPHJVSQTYFPMGHMWGYWWMSR8LIVIEVI

47.(56+$656,27.$6(241/2627%.,&$6,10$0'6+(4($4(

XMQIW[LIRHMJƤGYPXVIEHTSPMXMGEPP]E[O[EVH
VIWIEVGL

,527%.,5+('$0'+$0'.('9(.. 1/(215,6,8(21,065

6+$69(&$0+$0*1061$.10*

XLIɄGLEPPIRKMRKVSEHELIEH

&YX[IWLSYPHRSXFIɄVIEWWYVIH

[IWLSYPHFIɄEPEVQIH7MV7XITLIR

5(617661,08(56,*$6(57224(55,10

FYXJSYRH[IEOVYPIWERHGLESXMG

W]WXIQW-XɄXYVRWSYXXLEX[IHSRŭX

ORS[[LEXLEWFIGSQISJɄQMPPMSRW

1)2170'51)*18(40/(06

&1//,55,10('4(5($4&+

+SZIVRQIRXMXWIPJHSIWRŭXORS[WSQIHITEVXQIRXWLEZI

RSMHIELS[QYGLVIWIEVGLXLI]ɄLEZIGSQQMWWMSRIH

[LIXLIVMX[EWTYFPMWLIHSV[LIVIMXɄEPPɄMWɄRS[

756$5$.$4/,0*,56+($.$&4,6;9,6+9+,&+51/(2$465

SJɄKSZIVRQIRXTYXYTETTEVIRXP]MRWYVQSYRXEFPISFWXEGPIW

61'1,0*6+,0*56+$66+(,42((45,016+(45/$0$*(9,6+176

ETVSFPIQ8LIWO]LEWRSXJEPPIRMRɄEXHITEVXQIRXWXLEX

/$,06$,0$4(5($4&+4(*,56(4!+(;&106,07(61&1//,55,10

VIWIEVGL[MXLXLIEHZERXEKISJɄXLIRORS[MRK[LIXLIV

MXŭWɄFIIRTYFPMWLIHŪERH[LIVIMXMW

7IRWIEFSYX7GMIRGIWTSOI[MXLXLI'EFMRIX3JƤGI

ERHɄSXLIVWEFSYXMRGPYHMRKEVIKMWXIVMRXVERWTEVIRG]

TPERW[LIR7MV7XITLIRƤVWXI\TPSVIHMXIEVP]MRXLI

MRUYMV];IɄ[MPPVIWTSRHXSLMWVIGSQQIRHEXMSRW

F]ɄTVIWWMRKƤVQP]ERHTYFPMGP]JSVMX

%VIKMWXIVEPSRK[MXLXLIVIGSQQIRHIHGPIEVIV

GSRXVEGXWERHVYPIWEFSYXTYFPMGEXMSR[MPPKMZI

XLIɄTYFPMGɄGSRƤHIRGIXLEXMRJSVQEXMSRMWRŭXFIMRKOITX

JVSQXLIQERHKMZIVIWIEVGLIVWGSRƤHIRGIMR[SVOMRK

9,6+*18(40/(06

#+$61)21.,6,&$./$01(784,0* ,4 6(2+(05$;5

SJɄGSYVWIMXMRƥYIRGIWTYFPMGEXMSRSJVIWIEVGL

FYXɄLIɄLEWɄGEWXMXɄMRERI[PMKLX;LIRVIWIEVGLMWRŭX

TYFPMWLIHMXɄMWɄRSXXLIGSRƤHIRGISJXLIWTIGMEPEHZMWIVW

ERHTSPMXMGEPGSQQYRMGEXSVW[MRRMRKXLVSYKLFYXɄVEXLIV

XLIMVPEGOSJɄGSRƤHIRGI7SXLIVITSVXEHZSGEXIW

$5/$46(4$2241$&+61&1//70,&$6,0*4(5($4&+

MRɄEɄTSPMXMGEPGSRXI\X-ŭQɄWSɄKPEHEFSYXXLMW7IRWIEFSYX

7GMIRGI[SVOW[MXLXLI[MHIWXVERKISJTYFPMGKVSYTW

SRɄXLIQSWXHMJƤGYPXMWWYIW;IƤRHMXTY^^PMRKXLEX

TSPMG]QEOIVWKSɄMRɄWIEVGLSJTYFPMGXVYWXERHGSRƤHIRGI

Ū[LMGLMWɄSJXIR[L]XLI]GSQQMWWMSRMRHITIRHIRX

VIWIEVGLMRXLIƤVWXTPEGIŪERH]IX[SRŭXXVYWXXLITYFPMG

[MXLEɄHMWGYWWMSREFSYXXLIƤRHMRKW

-ŭQWYVIXLI]HSRŭX[ERXXSFIMRXLEXTSWMXMSRERHɄER]LS[

XVERWTEVIRG]MWLIVIXSWXE]8LMWɄMWɄERɄSTTSVXYRMX]

RSXɄSRP]XSVIGSZIVQMPPMSRWSJɄTSYRHWSJɄ[EWXIH

VIWIEVGLFYXXSKMZITSPMXMGMERWXLIɄSYXXLI]ɄRIIHŪ

TYFPMWLɄSVɄFIɄHEQRIH

6LU6WHSKHQVHW

RXWWRLQYHVWLJDWH



EXWIRXQGZHDN



V\VWHPV

AFTERWORD

by Tracey Brown

8VEGI]&VS[RMWXLIHMVIGXSVSJ7IRWIEFSYX7GMIRGI

 AFTERWORD

2122 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 2123
Appendix I

INQUIRY METHODS 

AND CALL FOR EVIDENCE


 APPENDIX I: INQUIRY METHODS AND CALL FOR EVIDENCE

2124 The Promise of Evidence-Based Policymaking
Inquiry methods

8LI6X,SR7MV7XITLIR7IHPI][LSMWPIEHMRKXLIMRUYMV]MWETVMZ]GSYRWIPPSVERHJSVQIV0SVH.YWXMGISJ%TTIEP

,ILEWWIVZIHEWEREHLSGNYHKISJXLI)YVSTIER'SYVXSJ,YQER6MKLXWERHLEWFIIREZMWMXMRKTVSJIWWSV

EXɄXLIɄ9RMZIVWMX]SJ3\JSVH,IMWEXVYWXIISJ7IRWIEFSYX7GMIRGI

8LIVITSVXLEWFIIRVIWIEVGLIHERHGSEYXLSVIHF](V4VEXIIO&YGLTSPMG]EWWSGMEXIERH(V7ɳPI0ERI

HMVIGXSVɄSJɄGEQTEMKRWERHTSPMG]EX7IRWIEFSYX7GMIRGI

-RMXMEPGSRWYPXEXMSRW[MXLQIQFIVWSJXLIVIWIEVGLGSQQYRMX]GMZMPWIVZERXWERHTSPMXMGMERWLIPTIHYWXSJVEQI

XLIMRUYMV]ŭWGEPPJSVIZMHIRGIWIIFIPS[
[LMGL[IMWWYIHMR2SZIQFIV;IMRZMXIH[VMXXIRWYFQMWWMSRW

ERHɄMRXIVZMI[IHMRXIVIWXIHTEVXMIWXSKEXLIVIZMHIRGI

;ILEZIFIRIƤXIHKVIEXP]JVSQXLI[VMXXIRWYFQMWWMSRWSVEPGSRXVMFYXMSRWERHEHZMGIJVSQXLITISTPIPMWXIH

MRɄ%TTIRHM\--FYXXLIGSRXIRXSJXLMWVITSVXMWXLIVIWTSRWMFMPMX]SJMXWEYXLSVW

!+(,037,4;$0'6+(241'7&6,101)6+,54(21469(4(5722146('%;$*4$06)41/6+( !+$4,6$%.(!4756

6IKMWXIVIH'LEVMX]2S
ERHF]2MGO6SWW

Call for evidence

 Tuesday 24th November 2015

7IRWIEFSYX7GMIRGIMWWIIOMRKMRJSVQEXMSRERHZMI[WEFSYXLS[KSZIVRQIRXGSQQMWWMSRWERHTYFPMWLIWVIWIEVGL

JVSQXLIEGEHIQMGGSQQYRMX]TVSJIWWMSREPFSHMIWXLIGMZMPWIVZMGIGLEVMXMIWXLIQIHMEERHSXLIVMRXIVIWXIHTEVXMIW

6IWIEVGLGSRHYGXIHSVGSQQMWWMSRIHF]GIRXVEPKSZIVRQIRXMWERMQTSVXERXTEVXSJKYMHMRKHIZIPSTMRK

/1',);,0*$0'/10,614,0*21.,&;.6+17*+'(2$46/(06$.*7,'(.,0(54(37,4(241/26$0'&1/2.(6(27%.,&$6,10

SJɄWYGLVIWIEVGLXLIVILEZIFIIRVITIEXIHEPPIKEXMSRWMRVIGIRX]IEVWSJTYFPMGEXMSRFIMRKLIPHFEGO

!+,5+$54$,5('27%.,&&10&(405)14$07/%(41)4($5105

@*EMPYVIXSQEOITYFPMGP]JYRHIHVIWIEVGLEZEMPEFPIXSXLITYFPMG

@%PEGOSJXVERWTEVIRG]SRXLIFEWMWSJKSZIVRQIRXHIGMWMSRWERHXLIVSPISJIZMHIRGIMRVIEGLMRKXLIQ

@4SXIRXMEPIJJIGXWSRXLI[MPPMRKRIWWSJVIWIEVGLIVWXSEWWMWXMRTSPMG]QEOMRK

-XMWRSXORS[RLS[WMKRMƤGERXSV[MHIWTVIEHXLIHIPE]IHTYFPMGEXMSRSV[MXLLSPHMRKSJKSZIVRQIRXVIWIEVGL

MWERHɄXLIVIMWPMXXPIGSQTEVEXMZIIZMHIRGISJLS[HMJJIVIRXHITEVXQIRXWSVEKIRGMIWFILEZI8LIVI[MPPEPWS

FIɄI\EQTPIWSJKSSHTVEGXMGIJSVMRWXERGI[LIVITSXIRXMEPP]E[O[EVHVIWIEVGLLEWFIIRTYFPMWLIHTVSQTXP]

7IRWIEFSYX7GMIRGIMWGSRHYGXMRKERMRUYMV]XSFIPIHF]SRISJMXWXVYWXIIWXLI6X,SR7MV7XITLIR7IHPI]

MRXSɄXLIWGEPIERHWSYVGIWSJXLITVSFPIQERHMRXSTSWWMFPIVIQIHMIW%RMRMXMEPWGSTMRKI\IVGMWI[EWGSRHYGXIH

FIX[IIRXL7ITXIQFIVERHXL3GXSFIVXSIWXEFPMWLXLMWGEPPJSVIZMHIRGI

8LIMRUYMV][MPPGSRWMHIVXLITYFPMGEXMSRSJVIWIEVGLMRGPYHMRKEREP]WIWSJSJƤGMEPWXEXMWXMGWMRMXMEXIHF];LMXILEPP

'(2$46/(065$0'6+($4/>5.(0*6+%1',(56+$64(2146616+(/#($.519(.&1/(57%/,55,1054(.$6,0*61'(81.8('

EHQMRMWXVEXMSRWFYXRSXXSPSGEPEYXLSVMXMIW8LIMRUYMV][MPPGSRWMHIV[LEXLEWLETTIRIHMRIWXEFPMWLIHMRWXERGIW

SJVIWIEVGLFIMRKLIPHFEGOERH[MPPWYKKIWXLS[GSRXVSZIVWMEPSVMRGSRGPYWMZIVIWIEVGLGERFILERHPIH

;IMRZMXIWYFQMWWMSRWEFSYX]SYVI\TIVMIRGISJGSQQMWWMSRMRKGSRHYGXMRKTYFPMWLMRKSVEGGIWWMRKKSZIVRQIRX

VIWIEVGLTEVXMGYPEVP]MRXLIJSPPS[MRKGSRXI\XW

'SRXVEGXWERHVYPIWHSVIWIEVGLGSRXVEGXWQEOITYFPMGEXMSRTVMRGMTPIWGPIEV#%VIXLIWITVMRGMTPIWIXLMGEPP]

ERHɄPIKEPP]ETTVSTVMEXI#;LSMWVIWTSRWMFPI[MXLMRHITEVXQIRXWJSVIRWYVMRKXLEXVIWIEVGLMWTYFPMWLIH

TVSQTXP]ɄERHJYPP]#

)\TIGXEXMSRWSJTYFPMGEXMSR XLIWIZEV]IWTIGMEPP]EWVIKEVHWXLIXMQMRKSJXLIERRSYRGIQIRXSJTSPMG]

ERHɄMRɄVIPEXMSRXSGSRXVSZIVWMEPXSTMGW;LEXNYWXMƤGEXMSRWQE]XLIVIFIJSVHIPE]MRKSV[MXLLSPHMRKTYFPMGEXMSR

SJɄKSZIVRQIRXGSQQMWWMSRIHVIWIEVGL#

 4SXIRXMEPMQTVSZIQIRXW[LEXGLERKIW[SYPHMQTVSZIXLI[E]KSZIVRQIRXGSQQMWWMSRWGSRHYGXW

ERHɄTYFPMWLIWVIWIEVGL#,S[GERKSSHTVEGXMGIFIWTVIEH#

APPENDIX I: INQUIRY METHODS AND CALL FOR EVIDENCE 

The Promise of Evidence-Based Policymaking 2125
Appendix II

LIST OF CONTRIBUTORS

 APPENDIX II: LIST OF CONTRIBUTORS

2126 The Promise of Evidence-Based Policymaking
;ILEZILEHXLIFIRIƤXSJ[VMXXIRWYFQMWWMSRWSVEPHMWGYWWMSRWERHEHZMGIJVSQXLIJSPPS[MRK

TISTPI;MXLXLIEYXLSVWŭTIVQMWWMSR[VMXXIRWYFQMWWMSRWEVITYFPMWLIHMRJYPPEX



Richard Bartholomew

Professor Sir John Beddington

Professor Virginia Berridge

Nicola Blackwood MP

Sir Iain Chalmers

Professor David Coggon

Tim Colbourne

Professor Brian S Collins

Professor Dame Sally Davies

Fiona Fox

Dr Emma Gordon

Dr Michael Hallsworth

Deborah Hargreaves

Rod Harrison

Professor Marianne Hester

*SVQIVNSMRXLIEHSJKSZIVRQIRXWSGMEPVIWIEVGL

4VSJIWWSVSJETTPMIHTSTYPEXMSRFMSPSK]-QTIVMEP'SPPIKI0SRHSR

WIRMSVɄEHZMWIV3\JSVH1EVXMR7GLSSP9RMZIVWMX]SJ3\JSVH

JSVQIVɄKSZIVRQIRXGLMIJWGMIRXMƤGEHZMWIV

4VSJIWWSVSJLMWXSV]ERHHMVIGXSV'IRXVIJSV,MWXSV]MR4YFPMG,IEPXL

0SRHSR7GLSSPSJ,]KMIRIERH8VSTMGEP1IHMGMRI[VMXXIRWYFQMWWMSR


'LEMV,SYWISJ'SQQSRWWGMIRGIERHXIGLRSPSK]WIPIGXGSQQMXXII

'SSVHMREXSV.EQIW0MRH-RMXMEXMZIJSYRHMRKQIQFIVSJ'SGLVERI

1..$%14$6,10$0'$/2%(..1..$%14$6,10

4VSJIWWSVSJSGGYTEXMSREPERHIRZMVSRQIRXEPQIHMGMRI

"0,8(45,6;1) 176+$/2610

(MVIGXSVSJTSPMG]3TIR6IEWSRJSVQIVWTIGMEPEHZMWIV

ERHHITYX]GLMIJSJWXEJJXSHITYX]TVMQIQMRMWXIV2MGO'PIKK14

4VSJIWWSVSJIRKMRIIVMRKTSPMG]ERHHMVIGXSV-RXIVREXMSREP'IRXVI

JSVɄ-RJVEWXVYGXYVI*YXYVIW9'0JSVQIVGLMIJWGMIRXMƤGEHZMWIV

(ITEVXQIRXɄSJɄ&YWMRIWW-RRSZEXMSRERH7OMPPWERH(ITEVXQIRXJSV8VERWTSVX

'LMIJQIHMGEPSJƤGIVJSV)RKPERH(ITEVXQIRXSJ,IEPXLIQIVMXYWTVSJIWWSV

/2(4,$.1..(*(10'10

'LMIJI\IGYXMZI7GMIRGI1IHME'IRXVI[VMXXIRWYFQMWWMSR


,IEHSJKSZIVRQIRXIGSRSQMGERHWSGMEPVIWIEVGL8VIEWYV]

(MVIGXSVLIEPXL
XE\&ILEZMSYVEP-RWMKLXW8IEQ

(MVIGXSV,MKL4E]'IRXVIGLEMVIH,MKL4E]'SQQMWWMSR

7IRMSVTVMRGMTEPVIWIEVGLSJƤGIV7GSXXMWLKSZIVRQIRX

4VSJIWWSVSJKIRHIVZMSPIRGI
MRXIVREXMSREPTSPMG]9RMZIVWMX]SJ&VMWXSP

Rt Hon Patricia Hewitt *SVQIV14ERHWIGVIXEV]SJWXEXIJSVLIEPXL

Ed Humpherson ,IEHSJEWWIWWQIRX9/7XEXMWXMGW%YXLSVMX]

Dr Julian Huppert *SVQIV14

Dr Eric Jensen %WWSGMEXITVSJIWWSV9RMZIVWMX]SJ;EV[MGO

Professor Lord Krebs )QIVMXYWTVSJIWWSVSJ^SSPSK]ERHJSVQIVTVMRGMTEP

.IWYW'SPPIKI9RMZIVWMX]SJ3\JSVH

Dr Daniel Lambauer (MVIGXSVJSVZEPYIJSVQSRI]HIZIPSTQIRX2EXMSREP%YHMX3JƤGI

Rt Hon David Laws *SVQIV14QMRMWXIVSJWXEXIJSVWGLSSPWQMRMWXIVSJWXEXIJSV'EFMRIX3JƤGI

ERHGLMIJWIGVIXEV]XSXLI8VIEWYV]I\IGYXMZIGLEMVQER'IRXVI*SVYQ

APPENDIX II: LIST OF CONTRIBUTORS 	

The Promise of Evidence-Based Policymaking 2127
'LMIJWGMIRXMƤGEHZMWIV(ITEVXQIRXJSV)HYGEXMSR

7IRMSVNSYVREPMWX4YPWI

,IEHSJMRXIVREXMSREP%GEHIQ]SJ1IHMGEP7GMIRGIW

(ITEVXQIRXJSV)RZMVSRQIRX*SSHERH6YVEP%JJEMVW

'LMIJVIWIEVGLIVERHLIEHSJ)HYGEXMSR%REP]XMGEP7IVZMGIW(MZMWMSR

 &166,5+*18(40/(06

'LMIJWSGMEPVIWIEVGLSJƤGIV;IPWLKSZIVRQIRX

0IGXYVIV9RMZIVWMX]SJ(YRHII[VMXXIRWYFQMWWMSR


*SVQIV14ERHGLEMVSJXLI,SYWISJ'SQQSRWWGMIRGI

$0'6(&+01.1*;5(.(&6&1//,66((

7SGMEPWGMIRXMWXERHVIWIEVGLTVSKVEQQIQEREKIV

(ITEVXQIRXJSV)RZMVSRQIRX*SSHERH6YVEP%JJEMVW

(2$46/(06$..(&674(4,0(8,'(0&(%$5('51&,$.,06(48(06,10$0'21.,&;

IZEPYEXMSR9RMZIVWMX]SJ3\JSVH

,IEHSJTSPMG];IPPGSQI8VYWX

,IEHSJVIWIEVGL(ITEVXQIRXJSV)HYGEXMSR

4VMRGMTEPVIWIEVGLJIPPS[JSVQIVHMVIGXSV
2EXMSREP-RWXMXYXISJ)GSRSQMG

ERH7SGMEP6IWIEVGLJSVQIVGLMIJIGSRSQMWX'EFMRIX3JƤGI
2YQFIV

190,0* 64((694,66(057%/,55,10

)HMXSV4YPWI

,IEHSJTYFPMGEJJEMVW6S]EP7SGMIX]

4(5,'(061)6+(1;$. 1&,(6;94,66(057%/,55,10

6IEHIVMRVMWOERHXIVVSV/MRKW'SPPIKI0SRHSR

7IRMSVTSPMG]EREP]WX8VERWJSVQ(VYK4SPMG]*SYRHEXMSR

4VSKVEQQIHMVIGXSV-RWXMXYXIJSV+SZIVRQIRX

'LMIJWGMIRXMWXERHLIEHSJVIWIEVGLERHIZEPYEXMSR&ILEZMSYVEP

-RWMKLXW8IEQTSWXHSGXSVEPVIWIEVGLJIPPS[,EVZEVH/IRRIH]7GLSSP

SJɄ+SZIVRQIRXVIWIEVGLEWWSGMEXI'IRXVIJSV1EVOIXERH4YFPMG

3VKERMWEXMSR9RMZIVWMX]SJ&VMWXSP

4VMRGMTEPSJ.IWYW'SPPIKI3\JSVHTVSJIWWSVSJGSQTYXIVWGMIRGI9RMZIVWMX]

1):)14'&+$,4/$01)6+(2(0$6$056,676(

'SJSYRHIV7GMIRGISKVEQTSWXHSGXSVEPVIWIEVGLIV*VERGMW'VMGO-RWXMXYXI

(MVIGXSV6%2()YVSTI

Dr Tim Leunig

Sofia Lind

Catherine Luckin

Simon Luker 

"Pudrey MacDougall

Dr Steven Marshall 

Dr Jonathan Mendel

Andrew Miller

Rachel Muckle

Dr Kathryn Oliver

Nicola Perrin

Vicky Petrie

Jonathan Portes

Nigel Praities

Becky Purvis

Sir Venki Ramakrishnan

Dr Brooke Rogers

Steve Rolles

Jill Rutter

Dr Michael Sanders

Professor Sir Nigel Shadbolt

Dr Andrew Steele

Dr Christian van Stolk

Professor Roger Tarling 9RMZIVWMX]SJ7YVVI]JSVQIVLIEHSJXLI,SQI3JƤGI6IWIEVGL

ERHɄ4PERRMRKɄ9RMX

	 APPENDIX II: LIST OF CONTRIBUTORS

2128 The Promise of Evidence-Based Policymaking
Dr Suzanne Taylor 6IWIEVGLJIPPS[0SRHSR7GLSSPSJ,]KMIRIERH8VSTMGEP1IHMGMRI

94,66(057%/,55,10

Jane Tinkler ,IEHSJWSGMEPWGMIRGI4EVPMEQIRXEV]3JƤGISJ7GMIRGIERH8IGLRSPSK]

Dr Arnaud Vaganay 6IWIEVGLEWWSGMEXI0SRHSR7GLSSPSJ)GSRSQMGWERH4SPMXMGEP7GMIRGI

David Walker *SVQIVLIEHSJTSPMG]%GEHIQ]SJ7SGMEP7GMIRGIW[VMXXIRWYFQMWWMSR


Professor Sir Mark Walport +SZIVRQIRXGLMIJWGMIRXMƤGEHZMWIV+SZIVRQIRX3JƤGIJSV7GMIRGI

[VMXXIRWYFQMWWMSR
LSRSVEV]HMWXMRKYMWLIHTVSJIWWSVSJQIHMGMRI

/2(4,$.1..(*(10'10

Professor Jeremy Watson 4VSJIWWSVSJIRKMRIIVMRKW]WXIQWERHZMGIHIER*EGYPX]SJ)RKMRIIVMRK

7GMIRGIW9'0GLMIJWGMIRXMWXERHIRKMRIIV&YMPHMRK6IWIEVGL

)WXEFPMWLQIRXJSVQIVGLMIJWGMIRXMƤGEHZMWIV(ITEVXQIRXJSV'SQQYRMXMIW

ERH0SGEPɄ+SZIVRQIRX

Naomi Weir %WWMWXERXHMVIGXSV'EQTEMKRJSV7GMIRGIERH)RKMRIIVMRK

Professor James Wilsdon 4VSJIWWSVSJVIWIEVGLTSPMG](ITEVXQIRXSJ4SPMXMGWERHHMVIGXSVSJMQTEGX

ERHIRKEKIQIRX*EGYPX]SJ7SGMEP7GMIRGIW9RMZIVWMX]SJ7LIJƤIPH

Dr Sarah Wollaston MP +$,41)6+(175(1)1//105+($.6+5(.(&6&1//,66((

APPENDIX II: LIST OF CONTRIBUTORS 		

The Promise of Evidence-Based Policymaking 2129
Appendix III

FREEDOM 

OF INFORMATION 

REQUESTS

	
 APPENDIX III: FREEDOM OF INFORMATION REQUESTS

2130 The Promise of Evidence-Based Policymaking
#(4(37(56('6+()1..19,0*,0)14/$6,10)41/	"/,0,56(4,$.'(2$46/(065$0')41/691(:(&76,8(

EKIRGMIW4YFPMGɄ,IEPXL)RKPERHERHXLI)RZMVSRQIRX%KIRG]
YRHIVXLI*VIIHSQSJ-RJSVQEXMSR

%GXɄ

 (IXEMPWSJEPPVIWIEVGLWXYHMIWGSQQMWWMSRIHF]?XLIHITEVXQIRXEKIRG]AFIX[IIR.YP]ERH.YP]

MRGPYHMRKXMXPIHEXIXLIWXYH][EWGSQQMWWMSRIH[LS[EWGSQQMWWMSRIHXSGEVV]SYXXLIWXYH]

$0'9+(0$0'9+(4(6+(4(57.659(4(27%.,5+('

 %WYQQEV]SJLS[QER]VIWIEVGLWXYHMIW[IVIGEVVMIHSYXSVGSQQMWWMSRIHF]?XLIHITEVXQIRX

EKIRG]AFIX[IIR.YP]ERH.YP]ERHLS[QER]VIQEMRYRTYFPMWLIHEWSJ.YP]

 %WXEXIQIRXSJLS[GSQTPIXI?XLIHITEVXQIRXEKIRG]AŭWTYFPMGP]EZEMPEFPIHEXEFEWIWSJVIWIEVGL

WXYHMIWEVIF]WYQQEVM^MRKXLIHIXEMPWSJEPPVIWIEVGLWXYHMIWXLEXLEZIFIIRGSQQMWWMSRIH

FYXɄEVIɄRSXɄVIKMWXIVIHSRXLIHEXEFEWIW

*SVHIXEMPWSJHITEVXQIRXEPVIWTSRWIWWII8EFPIT

APPENDIX III: FREEDOM OF INFORMATION REQUESTS 

The Promise of Evidence-Based Policymaking 2131
Appendix IV

BIBLIOGRAPHY

	 APPENDIX IV: BIBLIOGRAPHY

2132 The Promise of Evidence-Based Policymaking
%PPMERGIJSV9WIJYP)ZMHIRGI
9WMRKIZMHIRGI

;LEXɅ[SVOW#LXXT[[[EPPMERGIYWIJYPIZMHIRGI

SVKEWWIXW%PPMERGI4SPMG]9WMRKIZMHIRGIZTHJ

%GGIWWIHɄXL1E]


&&'2I[W
,SVWIQIEXWGERHEP;MXLHVE[R

TVSHYGXWERHXIWXVIWYPXW &&'2I[W?SRPMRIA

RHɄ1EVGLLXXT[[[FFGGSYORI[W

[SVPH%GGIWWIHXL1E]


&VIRRER%IXEP
 4SXIRXMEPFIRIƤXWSJQMRMQYQ

YRMXTVMGMRKJSVEPGSLSPZIVWYWEFERSRFIPS[GSWX

WIPPMRKMR)RKPERHQSHIPPMRKWXYH] &1.

KLXXT[[[FQNGSQGSRXIRXFQN

KJYPPTHJLXQP%GGIWWIHXL1E]


&YXPIV4
*SSHFEROW[LIVIMWXLIŬWYTTVIWWIHŭ

KSZIVRQIRXVITSVXSR9/JSSHTSZIVX]# 8LI+YEVHMER

4EXVMGO&YXPIVŭWGYXWFPSK?FPSKAXL2SZIQFIV

LXXT[[[XLIKYEVHMERGSQWSGMIX]TEXVMGOFYXPIVGYXW

FPSKRSZJSSHFEROWKSZIVRQIRXWYTTVIWWIH

HIJVEVITSVXSRJSSHEMH%GGIWWIHXL1E]


'MZMP7IVZMGI0IEVRMRKRSHEXI
+SZIVRQIRX7SGMEP

6IWIEVGL'SQTIXIRG]*VEQI[SVO

LXXTWGMZMPWIVZMGIPIEVRMRKGMZMPWIVZMGIKSZYO

WMXIWHIJEYPXƤPIWKWVCTVSJIWWMSREPCGSQTIXIRG]C

JVEQI[SVOCCƤREPHSG\

'SPPMRW&
)RKMRIIVMRKTSPMG]IZMHIRGIEHZMGI

ERHI\IGYXMSR-R(SYFPIHE]ERH;MPWHSRIHW
*YXYVI

HMVIGXMSRWJSVWGMIRXMƤGEHZMGIMR;LMXILEPP

LXXT[[[GWETGEQEGYOQIHMEYTPSEHWƤPIW

JHWE[THJ%GGIWWIHXL1E]


'SSO'
-QQMKVEXMSRMQTEGXVITSVX[MXLLIPH

F]Ʌ(S[RMRK7XVIIX &&'2I[W?SRPMRIAXL1EVGL

LXXT[[[FFGGSYORI[WYOTSPMXMGW

%GGIWWIHXL1E]


(ITEVXQIRXJSV&YWMRIWW-RRSZEXMSR
7OMPPW


7GMIRGIIRKMRIIVMRKERHXIGLRSPSK]WXEXMWXMGW

LXXTW[[[KSZYOKSZIVRQIRXWXEXMWXMGWWGMIRGI

IRKMRIIVMRKERHXIGLRSPSK]WXEXMWXMGW

%GGIWWIHXL1E]


(ITEVXQIRXJSV)HYGEXMSR
8IEGLIVWŭ[SVOPSEH

HMEV]WYVZI]6IWIEVGLVITSVXLXXTW[[[KSZYO

KSZIVRQIRXYTPSEHWW]WXIQYTPSEHWEXXEGLQIRXCHEXE

ƤPI(*)66THJ%GGIWWIHXL1E]


(ITEVXQIRXJSV)RZMVSRQIRX*SSHERH6YVEP%JJEMVW

E
7LEPI+EW6YVEP)GSRSQ]-QTEGXW

?(VEJXVITSVXA1EVGLLXXTW[[[KSZYOKSZIVRQIRX

YTPSEHWW]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI

'4$)65+$.(*$5474$.(&101/;,/2$&64(21462')

%GGIWWIHXL1E]


(ITEVXQIRXJSV)RZMVSRQIRX*SSHERH6YVEP%JJEMVW

F
6IUYIWXJSVMRJSVQEXMSRGSVVIWTSRHIRGI

GSZIVMRKXLIIGSRSQMGWSJWLEPIKEW?0IXXIVAXL

.YP]LXXTW[[[KSZYOKSZIVRQIRXYTPSEHWW]WXIQ

YTPSEHWEXXEGLQIRXCHEXEƤPI6*-C

)GSRSQMGWCSJC7LEPIC+EWC6IHEGXIHTHJ

%GGIWWIHXL1E]


(ITEVXQIRXJSV)RZMVSRQIRX*SSHERH6YVEP%JJEMVW


(VEJX7LEPI+EW6YVEP)GSRSQ]-QTEGXWTETIV

LXXTW[[[KSZYOKSZIVRQIRXYTPSEHWW]WXIQ

YTPSEHWEXXEGLQIRXCHEXEƤPIHVEJXWLEPIKEW

474$.(&101/;,/2$&64(21462')

%GGIWWIHXL1E]Ʉ


(ITEVXQIRXJSV)RZMVSRQIRX*SSHERH6YVEP%JJEMVW

ERH*SSH7XERHEVHW%KIRG]
)PPMSXX6IZMI[MRXS

XLIMRXIKVMX]ERHEWWYVERGISJJSSHWYTTP]RIX[SVOW

ƤREPVITSVXLXXTW[[[KSZYOKSZIVRQIRXYTPSEHW

W]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPIIPPMSX

VIZMI[ƤREPVITSVXNYP]THJ

%GGIWWIHXL1E]


(ITEVXQIRXSJ,IEPXL
4MPSXWGLIQIXSEPPS[

TEXMIRXWXSGLSSWIGSRZIRMIRX+4TVEGXMGI?2I[WWXSV]

SRPMRIAXL%TVMPLXXTW[[[KSZYOKSZIVRQIRXRI[W

2,.165&+(/(61$..192$6,(06561&+115(&108(0,(06

KTTVEGXMGI%GGIWWIHXL1E]


(ITPIXIH9VERMYQ3ZIVWMKLX&SEVH
*MREPVITSVX

SJXLI(ITPIXIH9VERMYQ3ZIVWMKLX&SEVH

LXXT[[[FERHITPIXIHYVERMYQSVKIRHSGWTHJ

%GGIWWIHXL1E]


(IZPMR'IXEP
-QTEGXWSJQMKVEXMSRSR9/REXMZI

IQTPS]QIRXEREREP]XMGEPVIZMI[SJXLIIZMHIRGI

,SQI3JƤGIERH(ITEVXQIRXSJ&YWMRIWW-RRSZEXMSR

ERHɄ7OMPPWLXXTW[[[KSZYOKSZIVRQIRXYTPSEHW

W]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPISGG

THJ%GGIWWIHXL1E]


(YKER)
+SZIVRQIRXEGGYWIHSJWYTTVIWWMRK

XLIHEQRMRKVITSVXXLEXWYKKIWXWMXWƥEKWLMT[IPJEVI

VIJSVQWEVIJSVGMRKIZIVQSVITISTPIXSVIWSVXXSJSSH

FEROW8LI-RHITIRHIRX?SRPMRIAXL*IFVYEV]

LXXT[[[MRHITIRHIRXGSYORI[WYOTSPMXMGW

*18(40/(06$&&75('1)57224(55,0*6+('$/0,0*

VITSVXXLEXWYKKIWXWMXWƥEKWLMT[IPJEVIVIJSVQW

EVILXQP%GGIWWIHXL1E]


)RKMRIIVMRKERH4L]WMGEP7GMIRGIW6IWIEVGL'SYRGMP

RSHEXI
7ERHTMXW?3RPMRIALXXTW[[[ITWVG

EGYOJYRHMRKLS[XSETTP]VSYXIWRIX[SVOMHIEW

[LEXMWEWERHTMX%GGIWWIHXL1E]


*S\*
%FEXXPIXSSJEV#7GMIRGI1IHME'IRXVI

*MSRE*S\ŭWFPSK?FPSKARH7ITXIQFIV

LXXT[[[WGMIRGIQIHMEGIRXVISVKEFEXXPIXSSJEV

%GGIWWIHXL1E]


*S\*
*MSRE*S\Ŭ8MQIXSIRHXLIGYPXYVI

SJɅWMPIRGIMRKSZIVRQIRXWGMIRGIŭ6IWIEVGL

4VSJIWWMSREP?SRPMRIAXL%TVMPLXXT[[[

VIWIEVGLVIWIEVGLGSQRI[WEVXMGPI#EVXMGPI-H!

%GGIWWIHXL1E]


APPENDIX IV: BIBLIOGRAPHY 	

The Promise of Evidence-Based Policymaking 2133
+EPPEKLIV)
8LI+EPPEKLIV6IZMI[SJXLI

MRHMVIGXIJJIGXWSJFMSJYIPWTVSHYGXMSR!+((0(9$%.(

*YIPW%KIRG]LXXT[IFEVGLMZIREXMSREPEVGLMZIWKSZ

YOLXXTVIRI[EFPIJYIPWEKIRG]

KSZYOWMXIWVIRI[EFPIJYIPWEKIRG]KSZYOƤPIWC

HSGYQIRXW6ITSVXCSJCXLIC+EPPEKLIVCVIZMI[THJ

%GGIWWIHXL1E]


+SZIVRQIRX)GSRSQMG
7SGMEP6IWIEVGL8IEQ


4YFPMWLMRKVIWIEVGLERHEREP]WMWMRKSZIVRQIRX+76

4YFPMGEXMSR4VSXSGSPLXXTW[[[KSZYOKSZIVRQIRX

YTPSEHWW]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI

+76CTYFPMGEXMSRCTVSXSGSPCC*-2%0THJ

%GGIWWIHXL1E]


+SZIVRQIRX3JƤGIJSV7GMIRGIE


4VMRGMTPIWSJWGMIRXMƤGEHZMGIXSKSZIVRQIRX

LXXTW[[[KSZYOKSZIVRQIRXTYFPMGEXMSRWWGMIRXMƤG

EHZMGIXSKSZIVRQIRXTVMRGMTPIWTVMRGMTPIWSJWGMIRXMƤG

EHZMGIXSKSZIVRQIRX%GGIWWIHXL1E]


+SZIVRQIRX3JƤGIJSV7GMIRGIF
8LI

+SZIVRQIRX'LMIJ7GMIRXMƤG%HZMWIVŭW+YMHIPMRIWSR

XLI9WISJɅ7GMIRXMƤGERH)RKMRIIVMRK%HZMGIMR4SPMG]

1EOMRKLXXTW[[[KSZYOKSZIVRQIRXYTPSEHW

W]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI

KGWEKYMHIPMRIWWGMIRXMƤGIRKMRIIVMRKEHZMGITSPMG]

QEOMRKTHJ%GGIWWIHXL1E]


+SZIVRQIRX3JƤGIJSV7GMIRGI
'LMIJ7GMIRXMƤG

%HZMWIVWERHXLIMVSJƤGMEPWERMRXVSHYGXMSR

LXXTW[[[KSZYOKSZIVRQIRXYTPSEHWW]WXIQ

YTPSEHWEXXEGLQIRXCHEXEƤPIGLMIJ

WGMIRXMƤGEHZMWIVWERHSJƤGMEPWMRXVSHYGXMSRTHJ

%GGIWWIHXL1E]


,E]RIW0IXEP
8IWXPIEVREHETX(IZIPSTMRK

TYFPMGTSPMG][MXLVERHSQMWIHGSRXVSPPIHXVMEPW

'EFMRIX3JƤGILXXTW[[[KSZYOKSZIVRQIRX

YTPSEHWW]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI

80%THJ%GGIWWIHXL1E]


,IWXIV1IXEP
*SVGIHQEVVMEKIXLIVMWO

JEGXSVWERHXLIIJJIGXSJVEMWMRKXLIQMRMQYQEKIJSV

EWTSRWSVERHSJPIEZIXSIRXIVXLI9/EWEWTSYWISV

ƤERGɰI
LXXT[[[FVMWXSPEGYOQIHMEPMFVEV]WMXIW

WTWQMKVEXIHHSGYQIRXWVOƤREPVITSVXTHJ

%GGIWWIHXL1E]


,1+SZIVRQIRX
(VYKWXVEXIK]6IHYGMRK

(IQERH6IWXVMGXMRK7YTTP]&YMPHMRK6IGSZIV]

7YTTSVXMRK4ISTPIXS0MZIE(VYK*VII0MJI

LXXTW[[[KSZYOKSZIVRQIRXYTPSEHWW]WXIQ

YTPSEHWEXXEGLQIRXCHEXEƤPIHVYK

WXVEXIK]THJ%GGIWWIHXL1E]


,1+SZIVRQIRX
8LI+SZIVRQIRXŭW%PGSLSP

7XVEXIK]LXXTW[[[KSZYOKSZIVRQIRXYTPSEHW

W]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPIEPGSLSP

WXVEXIK]THJ%GGIWWIHXL1E]


,SPQIW.
&ERSRFIPS[GSWXEPGSLSPWEPIW

[SYPHFIXMQIWPIWWIJJIGXMZIXLERQMRMQYQTVMGMRK

8LIɄ'SRZIVWEXMSR?SRPMRIAXL.ERYEV]

LXXTWXLIGSRZIVWEXMSRGSQFERSRFIPS[GSWXEPGSLSP

WEPIW[SYPHFIXMQIWPIWWIJJIGXMZIXLERQMRMQYQ

TVMGMRK%GGIWWIHXL1E]


,SQI3JƤGI
%PGSLSPWXVEXIK]GSRWYPXEXMSR

LXXTW[[[KSZYOKSZIVRQIRXGSRWYPXEXMSRWEPGSLSP

WXVEXIK]GSRWYPXEXMSR%GGIWWIHXL1E]


,SQI3JƤGI
2I\XWXITWJSPPS[MRKXLI

GSRWYPXEXMSRSRHIPMZIVMRKXLI+SZIVRQIRXŭWEPGSLSP

WXVEXIK]LXXTW[[[KSZYOKSZIVRQIRXYTPSEHW

W]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI%PGSLSPC

GSRWYPXEXMSRCVIWTSRWICVITSVXCZTHJ

%GGIWWIHXL1E]


,SQI3JƤGI
(VYKW-RXIVREXMSREPGSQTEVEXSVW

LXXTW[[[KSZYOKSZIVRQIRXYTPSEHW

W]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI

47*506(40$6,10$.1/2$4$61452')

%GGIWWIHXL1E]


,SYWISJ'SQQSRWLIEPXLWIPIGXGSQQMXXIIE


'SVVIWTSRHIRGISRGLMPHLSSHSFIWMX]WXVEXIK]

LXXT[[[TEVPMEQIRXYOFYWMRIWWGSQQMXXIIW

GSQQMXXIIWE^GSQQSRWWIPIGXLIEPXLGSQQMXXIITYF

PMGEXMSRW#X]TI!
WIWWMSR!
WSVX!JEPWI
MRUYMV]!

%GGIWWIHXL1E]


,SYWISJ'SQQSRWLIEPXLWIPIGXGSQQMXXIIF


'LMPHLSSHSFIWMX]FVEZIERHFSPHEGXMSR

LXXT[[[TYFPMGEXMSRWTEVPMEQIRXYOTEGQ

GQWIPIGXGQLIEPXLTHJ

%GGIWWIHXL1E]


,SYWISJ'SQQSRWLSQIEJJEMVWWIPIGXGSQQMXXII


(SQIWXMG:MSPIRGI*SVGIH1EVVMEKIERH

Ŭ,SRSYVŭFEWIHZMSPIRGILXXT[[[TYFPMGEXMSRW

TEVPMEQIRXYOTEGQGQWIPIGXGQLEJJM

THJERHLXXT[[[TYFPMGEXMSRWTEVPMEQIRXYOTE

GQGQWIPIGXGQLEJJMMTHJ

%GGIWWIHXL1E]


-RJSVQEXMSR'SQQMWWMSRIVŭW3JƤGI
)RZMVSRQIRXEP

-RJSVQEXMSR6IKYPEXMSRW(IGMWMSRRSXMGIVIJIVIRGI

*)6
LXXTWMGSSVKYOQIHMEEGXMSR[IZI

XEOIRHIGMWMSRRSXMGIWJIVCTHJ

%GGIWWIHXL1E]


-RXIVREXMSREP'SQQMXXIISJ1IHMGEP.SYVREP)HMXSVW


'PMRMGEP8VMEP6IKMWXVEXMSR%7XEXIQIRXJVSQ

XLIɅ-RXIVREXMSREP'SQQMXXIISJ1IHMGEP.SYVREP)HMXSVW

?)HMXSVMEPSRPMRIALXXT[[[MGQNISVKRI[WERH

IHMXSVMEPWGPMRCXVMEPCWITTHJ

%GGIWWIHXL1E]


/IRR]'
8LIMQTEGXSJEGEHIQMESRɅ4EVPMEQIRX

TIVGIRXSJ4EVPMEQIRXJSGYWIHMQTEGXGEWI

	
 APPENDIX IV: BIBLIOGRAPHY

2134 The Promise of Evidence-Based Policymaking
WXYHMIW[IVIJVSQWSGMEPWGMIRGIW!+(10'10

7GLSSPSJ)GSRSQMGWERH4SPMXMGEP7GMIRGI-QTEGX

&PSK?FPSKAXL3GXSFIVLXXTFPSKWPWIEGYO

MQTEGXSJWSGMEPWGMIRGIWXLIMQTEGXSJYO

EGEHIQMESRTEVPMEQIRX%GGIWWIHXL1E]


/MRKŭW'SPPIKI0SRHSRERH(MKMXEP7GMIRGI


8LIɅREXYVIWGEPIERHFIRIƤGMEVMIWSJVIWIEVGLMQTEGX

%RɅMRMXMEPEREP]WMWSJ6IWIEVGL)\GIPPIRGI*VEQI[SVO

6)*
MQTEGXGEWIWXYHMIW,MKLIV)HYGEXMSR

*YRHMRK'SYRGMPJSV)RKPERHLXXT[[[OGPEGYOWWTT

TSPMG]MRWXMXYXITYFPMGEXMSRW%REP]WMWSJ6)*MQTEGXTHJ

%GGIWWIHXL1E]


0EQFMI1YQJSVH,IXEP
,SYWILSPH*SSH

7IGYVMX]MRXLI9/%6IZMI[SJ*SSH%MH

LXXTW[[[KSZYOKSZIVRQIRXYTPSEHWW]WXIQ

YTPSEHWEXXEGLQIRXCHEXEƤPILSYWILSPHJSSH

WIGYVMX]YOTHJ%GGIWWIHXL1E]


0E[VIRGI*
6S[EWLSVWIQIEXƤPIWLIPZIH

8LI+YEVHMER?SRPMRIAXL%YKYWXLXXT[[[

XLIKYEVHMERGSQYORI[WEYKSJƤGMEPVITSVX

+145(/($65&$0'$.'(.$;('0(9)11'5$)(6;)($45

%GGIWWIHXL1E]


0IER+
;ERXXSORS[LS[JVEGOMRK[MPPEJJIGX

]SY#7SVV]XLEXŭWEWXEXIWIGVIX8LI(EMP]8IPIKVETL

?SRPMRIAXL%YKYWX2SPSRKIVEZEMPEFPI

0MRH7
(IGMWMSRSRI\XIRHMRKTVEGXMGIFSYRHEV]

TMPSXWXSFIQEHI[MXLSYXŬVSFYWXŭIGSRSQMGHEXE

4YPWI?SRPMRIAXL7ITXIQFIVLXXT[[[TYPWIXSHE]

GSYO]SYVTVEGXMGITVEGXMGIXSTMGWTVEGXMGIMRGSQI

'(&,5,1010(:6(0',0*24$&6,&(%170'$4;2,.16561%(

QEHI[MXLSYXVSFYWXIGSRSQMGHEXEEVXMGPI

%GGIWWIHXL1E]


0SSTWXVE6IXEP
%YWXIVMX]WERGXMSRWERHXLI

VMWISJJSSHFEROWMRXLI9/&1.L

LXXT[[[FQNGSQGSRXIRXFQNL

%GGIWWIHXL1E]


0SVH6SWI
&IXXIVPIEHIVWLMTJSVXSQSVVS[

2,7ɅPIEHIVWLMTVIZMI[LXXTW[[[KSZYO

KSZIVRQIRXYTPSEHWW]WXIQYTPSEHWEXXEGLQIRXCHEXE

ƤPI0SVHC6SWIC2,7C6ITSVXCEGGTHJ

%GGIWWIHXL1E]


1EWSR6
*VEGOMRKGEQTEMKRIVWGVMXMGMWI

ŬGIRWSVIHŭVITSVXSRLSYWITVMGIW8LI+YEVHMER

?SRPMRIAXL%YKYWXLXXT[[[XLIKYEVHMERGSQ

IRZMVSRQIRXEYKJVEGOMRKGIRWSVIHLSYWI

TVMGIVITSVX%GGIWWIHXL1E]


1E]8
%RMQQMKVEXMSRW]WXIQXLEX[SVOW

MRɅXLIɅREXMSREPMRXIVIWX?7TIIGLAXL(IGIQFIV

LXXTW[[[KSZYOKSZIVRQIRXWTIIGLIWLSQI

5(&4(6$4;52((&+10$0,//,*4$6,105;56(/6+$6

[SVOWMRXLIREXMSREPMRXIVIWX%GGIWWIHXL1E]


1IRHIP.WYFQMXXIH
9RTYFPMWLIHTSPMG]XVMEPWXLI

VMWOSJJEPWITSWMXMZIWERHXLITIVWMWXIRGISJEYXLSVMX]

FEWIHTSPMG])ZMHIRGI
4SPMG]

1IRK=IXEP
1SHIPPIHMRGSQIKVSYTWTIGMƤG

MQTEGXWSJEPGSLSPQMRMQYQYRMXTVMGMRKMR)RKPERH

4SPMG]ETTVEMWEPWYWMRKRI[HIZIPSTQIRXW

XSɅXLI7LIJƤIPH%PGSLSP4SPMG]1SHIP8


8LIɄ9RMZIVWMX]SJ7LIJƤIPHLXXT[[[WLIJEGYO

TSPSTSP]CJWƤPINYP]VITSVXTHJ

%GGIWWIHXL1E]


1MKVEXMSR%HZMWSV]'SQQMXXII
%REP]WMWSJXLI

-QTEGXWSJ1MKVEXMSRLXXTW[[[KSZYOKSZIVRQIRX

YTPSEHWW]WXIQYTPSEHWEXXEGLQIRXCHEXEƤPI

EREP]WMWSJXLIMQTEGXWTHJ%GGIWWIHXL1E]


1SFMPI8IPIGSQQYRMGEXMSRWERH,IEPXL6IWIEVGL

4VSKVEQQIQEREKIQIRXGSQQMXXII


1SFMPIɅ8IPIGSQQYRMGEXMSRWERH,IEPXL6IWIEVGL

4VSKVEQQI6ITSVX($.6+416(&6,10*(0&;

LXXT[[[QXLVSVKYOHSGYQIRXW18,6CVITSVXC

THJ%GGIWWIHXL1E]


1SVVMW2
(IQERHJSVJSSHFEROWLEWRSXLMRK

XSHS[MXLFIRIƤXWWUYII^IWE]W;SVOQMRMWXIV0SVH

*VIYH 8LI-RHITIRHIRX?SRPMRIARH.YP]LXXT[[[

MRHITIRHIRXGSYORI[WYOTSPMXMGWHIQERHJSVJSSH

FEROWLEWRSXLMRKXSHS[MXLFIRIƤXWWUYII^IWE]W

[SVOQMRMWXIVPSVHJVIYHLXQP

%GGIWWIHXL1E]


1SVVMW2
'SRWIVZEXMZIWWYTTVIWWIHZMXEP

VITSVXWSRHVYKEFYWIWE]W,SQI3JƤGIQMRMWXIV

8LIɄ-RHITIRHIRX?SRPMRIAVH3GXSFIV

LXXT[[[MRHITIRHIRXGSYORI[WYOTSPMXMGWXSVMIW

57224(55('8,6$.4(2146510'47*$%75(5$;5+1/(

SJƤGIQMRMWXIVLXQP%GGIWWIHXL1E]


3EOIWLSXX-
1MRMWXIVWLMHIWYVKIMRYWISJɅJSSH

FEROW 8LI7YRHE]8MQIW?SRPMRIAVH2SZIQFIV

LXXT[[[XLIWYRHE]XMQIWGSYOWXSRI[W4SPMXMGW

EVXMGPIIGI%GGIWWIHXL1E]


3PMZIV/IXEP
%W]WXIQEXMGVIZMI[

SJɅFEVVMIVWXSERHJEGMPMXEXSVWSJXLIYWISJIZMHIRGI

F]ɅTSPMG]QEOIVW&1',IEPXL7IVZMGIW6IWIEVGL


LXXTFQGLIEPXLWIVZVIWFMSQIHGIRXVEPGSQ

EVXMGPIW

%GGIWWIHXL1E]


4EKI)
8LIVIMWWYJƤGMIRXIZMHIRGI

XSɅWYKKIWX;LMXILEPPMWPIERMRKSRVIWIEVGLIVW

XSɅTVSHYGITSPMXMGEPP]YWIJYPVIWIEVGL!+(10'10

7GLSSPSJ)GSRSQMGWERHɄ4SPMXMGEP7GMIRGI-QTEGX

&PSK?FPSKAWX3GXSFIVLXXTFPSKWPWIEGYO

MQTEGXSJWSGMEPWGMIRGIWKSZIVRQIRXWPIER

SRVIWIEVGLIVW%GGIWWIHXL1E]


APPENDIX IV: BIBLIOGRAPHY 

The Promise of Evidence-Based Policymaking 2135
4SVXIW.
&VMXMWLNSFWERHJSVIMKR[SVOIVW

XSHE]ŭWVITSVXWSRMQQMKVEXMSRERHYRIQTPS]QIRX

2EXMSREP-RWXMXYXISJ)GSRSQMGERH7SGMEP6IWIEVGLFPSK

?FPSKAXL.ERYEV]LXXT[[[RMIWVEGYOFPSKFVMXMWL

NSFWERHJSVIMKR[SVOIVWXSHE]WVITSVXWMQQMKVEXMSR

ERHYRIQTPS]QIRX:^W:UXV0I8%GGIWWIHXL1E]




4SVXIW.
0IXWXEXMWXMGMERWGV]JSYP[LIR

TSPMXMGMERWFIRHXLIXVYXL *MRERGMEP8MQIW?SRPMRIAXL

.ERYEV]LXXT[[[JXGSQGQWWHGJEEE

IGJIEFHILXQPE\^^V.5+YG<

%GGIWWIHXL1E]


4VMGI'
3RIXLMVHSJTVEGXMGIFSYRHEV]TMPSX

WYVKIVMIWJEMPIHXSVIKMWXIVEWMRKPITEXMIRX4YPWI

?SRPMRIAXL1EVGLLXXT[[[TYPWIXSHE]GSYO

TSPMXMGEPTVEGXMGIFSYRHEVMIWSRIXLMVHSJTVEGXMGI

%170'$4;2,.16574*(4,(5)$,.('614(*,56(4$5,0*.(

TEXMIRXEVXMGPI9^/WƥCX)

%GGIWWIHXL1E]


4YFPMG,IEPXL)RKPERH
7YKEVVIHYGXMSR

VIWTSRHMRKXSXLIGLEPPIRKILXXTW[[[KSZYO

KSZIVRQIRXYTPSEHWW]WXIQYTPSEHWEXXEGLQIRXCHEXE

ƤPI7YKEVC6IHYGXMSRC6IWTSRHMRKCXSCXLIC

'LEPPIRKICC.YRITHJ%GGIWWIHXL1E]


4YFPMG,IEPXL)RKPERH
7YKEVVIHYGXMSR

XLIɅIZMHIRGIJSVEGXMSRLXXTW[[[KSZYO

KSZIVRQIRXYTPSEHWW]WXIQYTPSEHWEXXEGLQIRXCHEXE

ƤPI7YKEVCVIHYGXMSRC8LICIZMHIRGICJSVCEGXMSR

THJ%GGIWWIHXL1E]


6IWIEVGL)\GIPPIRGI*VEQI[SVO6)*
MQTEGX

GEWIWXYHMIW?(EXEFEWIA(MKMXEP7GMIRGILXXTMQTEGX

VIJEGYOGEWIWXYHMIW%GGIWWIHXL1E]


8LI6S]EP7SGMIX]ERHXLI6S]EP%GEHIQ]SJ

)RKMRIIVMRK
7LEPIKEWI\XVEGXMSRMRXLI9/

EVIZMI[SJɅL]HVEYPMGJVEGXYVMRKLXXTWVS]EPWSGMIX]

SVKbQIHMETSPMG]TVSNIGXWWLEPIKEWI\XVEGXMSR

WLEPIKEWTHJ%GGIWWIHXL1E]


8LI6S]EP7XEXMWXMGEP7SGMIX]
4YXXMRKIZMHIRGI

EXXLILIEVXSJTSPMG]HIFEXILXXT[[[VWWSVKYO

-QEKIW4(*MRƥYIRGMRKGLERKIHEXEQERMJIWXS

2766,0*(8,'(0&($66+(+($461)6+(21.,&;'(%$6(2')

%GGIWWIHXL1E]


6YXXIV.
)ZMHIRGIERHIZEPYEXMSRMRTSPMG]

QEOMRKETVSFPIQSJWYTTP]SVHIQERH#056,676(

JSVɄ+SZIVRQIRXLXXT[[[MRWXMXYXIJSVKSZIVRQIRXSVK

YOWMXIWHIJEYPXƤPIWTYFPMGEXMSRWIZMHIRGI	ERH	

IZEPYEXMSR	MR	XIQTPEXICƤREPCTHJ

%GGIWWIHXL1E]


6YXXIV.ERH+SPH.
7LS[]SYV[SVOMRKW

EWWIWWMRKLS[KSZIVRQIRXYWIWIZMHIRGI

XSɅQEOIɅTSPMG]056,676()1418(40/(06

LXXT[[[MRWXMXYXIJSVKSZIVRQIRXSVKYOTYFPMGEXMSRW

WLS[]SYV[SVOMRKW%GGIWWIHXL1E]


7IEVGLMRKIV8IXEP
9WISJ97'VSTPERHW

JSV&MSJYIPW-RGVIEWIW+VIIRLSYWI+EWIW8LVSYKL

)QMWWMSRWJVSQ0ERH9WI'LERKI7GMIRGI


TTLXXTWGMIRGIWGMIRGIQEKSVK

GSRXIRX%GGIWWIHXL1E]


7IRWIEFSYX7GMIRGI
-HSRŭXORS[[LEX

XSɅFIPMIZIQEOMRKWIRWISJWGMIRGIWXSVMIW

LXXT[[[WIRWIEFSYXWGMIRGISVKHEXEƤPIW

VIWSYVGIW-(SRX/RS[;LEX8S&IPMIZIC[IFTHJ

%GGIWWIHXL1E]


7IRWIEFSYX7GMIRGI
4IIVVIZMI[XLIRYXW

ERHɅFSPXWLXXT[[[WIRWIEFSYXWGMIRGISVKHEXE

ƤPIWVIWSYVGIW4IIVVIZMI[C8LIRYXWERHFSPXWTHJ

%GGIWWIHXL1E]


7IRWIEFSYX7GMIRGI
1EOMRKWIRWI

SJɅYRGIVXEMRX]LXXT[[[WIRWIEFSYXWGMIRGISVKHEXE

ƤPIWVIWSYVGIW7%7C179CVITVMRXCGSQTVIWWIH

THJ%GGIWWIHXL1E]


7IRWIEFSYX7GMIRGI
Ű*VEGOMRKVMWOGSQTEVIH

XSɅXLEPMHSQMHIERHEWFIWXSWMR;EPTSVXVITSVXű#

7IRWIɄEFSYX7GMIRGI*SV8LI6IGSVH?SRPMRIA

XLɄ2SZIQFIVYTHEXIHXL(IGIQFIV


LXXT[[[WIRWIEFSYXWGMIRGISVKVIWSYVGIWTLT

UYSXJVEGOMRKVMWOGSQTEVIHXSXLEPMHSQMHIERHEWFIWXSW

MR[EPTSVXVITSVXUYSX%GGIWWIHXL1E]


7TIRGIV&
8LIJVEGOMRKGSZIVYT(IJVEGIRWSVW

OI]VITSVXXMQIWMRTEKIWMRQSZIHIWGVMFIH

EWɅŬGSQMGEPŭF]GEQTEMKRIVW1EMP3RPMRI?SRPMRIAXL

%YKYWXYTHEXIHXL%YKYWX
LXXT[[[HEMP]QEMP

GSYORI[WEVXMGPI8LIJVEGOMRKGSZIV(IJVE

GIRWSVWOI]VITSVXXMQIWTEKIWHIWGVMFIHGSQMGEP

GEQTEMKRIVWLXQP%GGIWWIHXL1E]


8LIWYTVIQIGSYVX
	
	
	

		
	
	

JSVɅXLI,SQI(ITEVXQIRX%TTIPPERX
?A9/7'

LXXTW[[[WYTVIQIGSYVXYOGEWIWYOWG

LXQP%GGIWWIHXL1E]


!EVPMRK6
6IPEXMSRWFIX[IIR+SZIVRQIRX

6IWIEVGLIVWERH%GEHIQMGW 8LI,S[EVH.SYVREP

SJɄ'VMQMREP.YWXMGI
TTLXXTSRPMRIPMFVEV]

[MPI]GSQHSMN\EFWXVEGX

%GGIWWIHXL1E]


8VERWJSVQ(VYK4SPMG]*SYRHEXMSR
)TMGXLVII

]IEV*3-FEXXPIXSVIPIEWIWYTTVIWWIH,SQI3JƤGIHVYK

WXVEXIK]IZEPYEXMSRVIEGLIWMXWHIRSYIQIRX!4$05)14/

(VYK4SPMG]*SYRHEXMSR8VERWJSVQ1IHME&PSK?FPSKAXL

.ERYEV]PEWXYTHEXIHXL.YRI
LXXTXVERWJSVQHVYKW

FPSKWTSXGSYOITMGXLVII]IEVJSMUYIWXXS

VIPIEWILXQP%GGIWWIHXL1E]


 APPENDIX IV: BIBLIOGRAPHY

2136 The Promise of Evidence-Based Policymaking
8VEZMW%E
8LIVIWE1E]SVHIVWWXYH]MRXS[LMGL

HVYKPE[W[SVOMRSXLIVGSYRXVMIW8LI+YEVHMER?SRPMRIA

XL1EVGLLXXT[[[XLIKYEVHMERGSQTSPMXMGW

QEVXLIVIWEQE]WXYH]HVYKPE[W

%GGIWWIHXL1E]


8VEZMW%F
,SQI3JƤGIXSYVXSWXYH]HVYK

TSPMGMIWMRGSYRXVMIW 8LI+YEVHMER?SRPMRIAXL1E]

LXXT[[[XLIKYEVHMERGSQTSPMXMGWQE]

LSQISJƤGIHVYKTSPMGMIWMRXIVREXMSREP

%GGIWWIHXL1E]


9/&SVHIV%KIRG]
1EVVMEKIZMWEWXLI[E]

JSV[EVH1.10*(4$8$,.$%.(

;EXX2
2SVQER&EOIVVIWMKRWEW,SQI3JƤGI

QMRMWXIV[MXLTEVXMRKWLSXEX1E]8LI+YEVHMER

?SRPMRIAVH2SZIQFIVLXXT[[[XLIKYEVHMERGSQ

TSPMXMGWRSZRSVQERFEOIVVIWMKRWLSQI

SJƤGIQMRMWXIV%GGIWWIHXL1E]


;MRXSYV4ERH8VEZMW%
9/KSZIVRQIRXŭW

HVYKPE[WWYVZI][EWWYTTVIWWIH0MF(IQQMRMWXIV

WE]W8LIɄ+YEVHMER?SRPMRIAXL3GXSFIVLXXT[[[

XLIKYEVHMERGSQTSPMXMGWSGXKSZIVRQIRX

'47*.$955748(;57224(55('.,%'(//,0,56(4014/$0

FEOIV%GGIWWIHXL1E]


;SPPEWXSR7
8LI+SZIVRQIRXLEWJEMPIHXSPIEH

F]I\EQTPISRSTIRHEXE8LI(EMP]8IPIKVETL?SRPMRIA

6+$07$4;1.10*(4$8$,.$%.(

;SPPEWXSR7
0IXXIVJVSQXLI'SQQMXXII'LEMV

XS(YRGER7IPFMI'LMIJ)\IGYXMZI4YFPMG,IEPXL

)RKPERHSRIZMHIRGISRVIHYGMRKWYKEVGSRWYQTXMSR

?0IXXIVAXL3GXSFIVLXXT[[[TEVPMEQIRXYO

HSGYQIRXWGSQQSRWGSQQMXXIIW,IEPXL

'SVVIWTSRHIRGI(YRGER7IPFMI4,)

VIPIZERXTEVEWTHJ%GGIWWIHXL1E]


APPENDIX IV: BIBLIOGRAPHY 


The Promise of Evidence-Based Policymaking 2137
 (05($%176 &,(0&(,5$0,0'(2(0'(06&$/2$,*0,0*&+$4,6;6+$6

GLEPPIRKIWXLIQMWVITVIWIRXEXMSRSJWGMIRGIERHWGMIRXMƤGIZMHIRGI

MRɄTYFPMGPMJI;IɄEHZSGEXISTIRRIWWERHLSRIWX]EFSYXVIWIEVGL

ƤRHMRKWERH[SVOXSIRWYVIXLITYFPMGMRXIVIWXMRWSYRHWGMIRGI

ERHɄIZMHIRGIMWVITVIWIRXIHERHVIGSKRMWIHMRTYFPMGHMWGYWWMSR

ERHɄTSPMG]QEOMRK

7IRWIEFSYX7GMIRGIMWEWQEPPXIEQ[SVOMRK[MXLXLSYWERHW

SJɄWYTTSVXIVWJVSQɄ[SVPHPIEHMRKVIWIEVGLIVWXSGSQQYRMX]KVSYTW

8LMWVITSVX[EWTYFPMWLIHMRɄF]7IRWIEFSYX7GMIRGI

[LMGLɄLEWƤREPVIWTSRWMFMPMX]JSVɄXLIGSRXIRX

(IWMKRIHF]*VERGIWGE8SVXSVE

*SVQSVIGSTMIWSVJSVJYVXLIVMRJSVQEXMSRTPIEWIGSRXEGX



E'PIVOIR[IPP+VIIR

0SRHSR)'6(4




9995(05($%1765&,(0&(14*

(037,4,(55(05($%1765&,(0&(14*

6IKMWXIVIHGLEVMX]2S

'SQTER]2S

8LMW[SVOMWPMGIRWIHYRHIVE'VIEXMZI'SQQSRW%XXVMFYXMSR2SR'SQQIVGMEP2S(IVMZEXMZIW-RXIVREXMSREP0MGIRWI

2138 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 2139


E'PIVOIR[IPP+VIIR

0SRHSR)'6(4




8LIMRUYMV]ERHXLITVSHYGXMSRSJXLMWVITSVX[IVIWYTTSVXIHF]EɄKVERXJVSQ

XLI.6778'LEVMXEFPI8VYWX6IKMWXIVIH'LEVMX]2S
ERHɄF]Ʉ2MGO6SWW

2140 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl0-r7t7

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0266

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

It is absolutely not the business of the federal government to compile records on its

citizens, cradle to grave, or even higher education to employer. The security risks are far

too great and protection from further privacy infringement must be maintained.

The Promise of Evidence-Based Policymaking 2141
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl0-2g5i

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0267

Comment on FR Doc # 2016-22002

Submitter Information

Name: Ann Ebberts

Address:

Alexandria, 22301

Email: aebberts@agacgfm.org

General Comment

Please see the attached document with AGA's comments in response to the Request For

Comments for the Commission on Evidence-Based Policymaking -- Docket -

160907825-6825-01. We would be pleased to respond to any questions you have

regarding our submission. Thank You, Ann Ebberts, CEO, Association of Government

Accountants (AGA)

Attachments

AGA_Response_CEP_Dec14

2142 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 2143
2144 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 2145
2146 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl0-obxe

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0268

Comment on FR Doc # 2016-22002

Submitter Information

Name: Toni Becker

General Comment

Stop amassing data and dossier building on Americans! We need LESS government

programs, not more. Few people trust that the government will protect Americans'

privacy or that the data will be secure. Many Americans understand that BIG data is the

new gold rush and we're sick of the public sector ballooning off the backs of hardworking tax payers. For instance, the massive amount of data being extracted from

school children currently is unconscionable! Obama gutting FERPA to make that data

available to "researchers" without knowledge or consent of parents is the creepy norm,

not the exception. Corporations reaping windfall profits off the backs of small, innocent

children's data is the norm, not the exception. STOP! Few Americans believe that our

personal business is any business of the government and even fewer believe that the

government has the personal best for individual Americans at heart. The public sector

today serves and protects ITSELF and abuses the hand that feeds it. Government needs

to shrink and its workforce must be gutted. NO to data amassing on Americans. NO to

needing a "commission" on "evidence-based policymaking. NO to the notion of

"evidence-based policymaking." NO to giving government more power and the means to

abuse the hands that feed it!!!

The Promise of Evidence-Based Policymaking 2147
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl1-zmg5

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0269

Comment on FR Doc # 2016-22002

Submitter Information

Name: Atheendar Venkataramani

Address:

Boston, MA, 02114

Email: avenkataramani@partners.org

General Comment

I hold an MD and PhD (Health Economics) and work as a physician-researcher at

Massachusetts General Hospital. I have broad experience in evidence-based

policymaking, both as a consumer of evidence and as a researcher who generates

evidence. I will focus my reply on questions 16-19.

In the last 20 years, there have been revolutionary developments in statistical techniques,

data availability, and computing power, all of which have enabled detailed, valid, and

(statistically and politically) unbiased assessments of the impacts of public sector

programs. The Commission on Evidence-Based Policymaking can play an important role

in leveraging these trends towards the improvement of public policy by (1) making large

federal and state databases more readily available (including identification of local

geographies), (2) working closely with government, private sector, and academic

partners with expertise in theoretical and statistical techniques that leverage "natural

experiment" approaches, (3) and advocating for the rollout of programs in a manner that

supports rigorous evaluation.

2148 The Promise of Evidence-Based Policymaking
With specific regards to points (2) and (3), ideally policy evaluation should be build into

program design whenever possible. This could be done in a number of ways. A likely

scenario is where program budgets do not support a initial full rollout. In this case, the

initial roll-out could be randomized across beneficiaries or locations, who would go on

to receive benefits as the program grows in size. This allows for the ability to assess

program impacts across individuals or locations who are otherwise similar, but some of

whom randomly received the program on the account of initial constraints. Such a study

design has far greater validity than simple pre-post program comparisons (as other

factors could have changed over time, biasing the results). Another likely scenario is

where a program has a sharp eligibility threshold. Dedicated data collection of people

who are just on either side of the threshold - who are otherwise similar but of whom

some received program benefits - allows for a robust natural experiment that is typically

free from bias. These wait-list and threshold based designs have been used successfully

in the evaluation of Medicaid expansions in Oregon and charter school policies in

Massachusetts, among other examples. 

Randomized controlled trials and quasi-experimental designs (such as regression

discontinuity or differences-in-differences) should serve as the sole bedrock of program

evaluation. A scientific revolution in statistics and econometrics has repeatedly

demonstrated the superiority of such methods over anecdotes, pre-post comparisons, or

typical observational study designs like cross-sectional regression. It is possible create an

infrastructure where program design is informed in real time by ongoing policy

experiments, with iterative experimentation. Specifically, small randomized trials can

inform elements of program design, with the design building in opportunities for

additional trials or quasi-experiments. This "real-time" research has the benefit of

generating immediately useful short run policy information and, since it is part of the

initial program rollout, can be done with minimal additional cost. Underlying this edifice

would be the incorporation of a structure that allows for a longer-term evaluation of the

program. Such a priori long-term designs can be creatively leveraged by members of the

Commission and their associated teams, and partner agencies and researchers. 

The Promise of Evidence-Based Policymaking 2149
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl1-nm76

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0270

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

I oppose non-consensual data mining. Informed consent is key to our liberty. Please stop

this assault on our freedom.

2150 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl2-84m6

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0271

Comment on FR Doc # 2016-22002

Submitter Information

Name: Dana Fortson

General Comment

Repeal Common Core Standards and data mining! Stop using our children as a way for

big business to make money. The Government has no right to my child's information.

The Promise of Evidence-Based Policymaking 2151
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl2-1qz7

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0272

Comment on FR Doc # 2016-22002

Submitter Information

Name: Rep. Justin Fawson Utah House District 7

General Comment

The de-identification of student information should be treated identically to HIPAA.

Requiring parents to consent to student information tracking in order for their student to

participate in the curriculum flies in the face of the fundamental purpose of our

education system. 

2152 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl2-87p5

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0273

Comment on FR Doc # 2016-22002

Submitter Information

Name: Missy T

General Comment

I believe in informed consent and my God given civil rights, signed by our forefathers

that this is still a free country, The United States of America. I oppose non-consensual

data mining and if I choose that I don't want to share my families information in any

way, shape or form that you halt immediately, otherwise, suffer the consequences

accordingly!

The Promise of Evidence-Based Policymaking 2153
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl2-ky3p

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0274

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

I oppose data mining without my consent. I oppose data mining. Period. These are MY

children, not yours. They don't get tested and gathered like a lab rat. Enough is enough.

Give me my freedom back and get out of my children's lives and my family's life.

2154 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl2-e523

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0275

Comment on FR Doc # 2016-22002

Submitter Information

Name: Coretha Rozendaal

Address:

Monroe, IA, 50170

Email: Corkjoy@juno.com

General Comment

I believe in informed consent. I oppose non-consensual data mining. Stop this madness!

The Promise of Evidence-Based Policymaking 2155
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl2-avzk

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0276

Comment on FR Doc # 2016-22002

Submitter Information

Name: Hank Hank

General Comment

This issue should be a mute point. We have an amendment that completely disallows

non-consensual probing of our children known as the Hatch Amendment.

2156 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl3-shw4

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0277

Comment on FR Doc # 2016-22002

Submitter Information

Name: Wendy Hart

General Comment

I am opposed to allowing greater data access. Children's data needs to be properly

overseen and managed by their parents. Even in medicine, these children's data would

only be allowed to be used in research with the informed consent of their parent or

guardian. But in education, why do we feel that it is any less invasive? In fact, it has the

potential to pigeon-hole children from an early age and limit their future possibilities.

We live in an age where, for the first time in history, people who have never seen,

spoken with, or cared about a particular child, will be able to know vast amounts of

information about them and make decisions about them without the benefit of their

parents as gate-keepers. In the past, parents were the only source of the vast amounts of

information on their children. They could choose what information and to whom this

information should be shared. Now, they don't even know that it is being shared. How

can we possibly say that 'the best interests' of a child are paramount when this

information is being proffered without any sort of knowledge of the individual or a

motivation to maintain their privacy by those who know and care about them, their

parents and their teachers?

The Promise of Evidence-Based Policymaking 2157
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl3-sduh

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0278

Comment on FR Doc # 2016-22002

Submitter Information

Name: Pamela Sanford

General Comment

I am against mining data of our children. At the very least it must require parental

permission.

2158 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl3-y2c7

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0279

Comment on FR Doc # 2016-22002

Submitter Information

Name: Laura Carlson

General Comment

I support parent control and parent informing of every data bit gathered by government

institutions.

The Promise of Evidence-Based Policymaking 2159
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl3-uhug

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0280

Comment on FR Doc # 2016-22002

Submitter Information

Name: Ruth Rose

General Comment

I believe in informed consent. I oppose non-consensual data mining of my and other

children in schools. Further, the whole of the Federal Department of Education should be

dissolved and the money/control returned to the States.

2160 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl3-8vmr

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0281

Comment on FR Doc # 2016-22002

Submitter Information

Name: Carolyn McLarty

General Comment

Data collection beyond basic information is not the role of education. The right to

privacy is

more important than collecting data

for companies to crunch and manipulate and sell. Collecting data without consent is

egregiously taking advantage of a vulnerable population to control them, their future,

and use them for commercial and political gain.

The Promise of Evidence-Based Policymaking 2161
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl3-jrdt

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0282

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

Regarding the data-mining of our children under Common Core...this is why Donald

Trump won in a landslide. The frustration with unelected bureaucrats, their Big-Brother

mentality and the overall over-reach of government is why the American people are

demanding that the Feds get out of education. Hands off our children. Hands off our

education in general.

2162 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl3-8wc0

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0283

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

NO MORE DATA SHARING! STOP SHARING MY CHILDRENS' DATA! STOP

COLLECTING MY CHILDRENS' DATA!

In Article 1 Section 8 of The Constitution of the United States of America, also known

as the enumerated powers clause, this sort of power is not granted to the federal

government, and thus the exercise of such powers is totally unconstitutional!

The Promise of Evidence-Based Policymaking 2163
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl3-ylj2

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0284

Comment on FR Doc # 2016-22002

Submitter Information

Name: Donna B.

General Comment

Stop the Data Mining without the informed consent of parents or guardians. We are not a

communist/socialist country. Thank you.

2164 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl3-jvj0

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0285

Comment on FR Doc # 2016-22002

Submitter Information

Name: s s

General Comment

I believe in informed consent. I oppose non-consensual data mining. Stop this madness.

The Promise of Evidence-Based Policymaking 2165
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl3-svaw

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0286

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

My children are not commodities. Stop sharing their information.

2166 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl4-siok

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0287

Comment on FR Doc # 2016-22002

Submitter Information

Name: R. Crawford

General Comment

"I'm against elected or unelected officials using our children's data. I can't trust elected

officials to run their own lives, let a loan their campaign honorably. So how the heck can

I trust a unelected officials. It's time we start producing good citizens. Stop testing our

children for data. I do not consent to letting my children or grandchildren be used for

collecting data.

The Promise of Evidence-Based Policymaking 2167
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl6-rc7w

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0288

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

If the Commission is looking for unique and very promising examples of government

programs encouraging Evidence-Based Policymaking, it should learn from and highlight

Development Innovation Ventures (DIV) at USAID. DIV is very focused on maximizing

results while minimizing costs to the taxpayers. It encourages companies, academics,

NGOs and others to compete to help people better, faster and cheaper by focusing on

solutions that are the most cost-effective, backed by the most evidence, and have the

strongest pathways to scale and sustainability. DIV is a program that should be amplified

at USAID and a model that should be copied by many other parts of government. 

here are some links worth reading:

Lean Startup Goes to Washington: SSIR

https://ssir.org/articles/entry/the_lean_startup_goes_to_washington

OMB Memo M-13-17

https://www.whitehouse.gov/sites/default/files/omb/memoranda/2013/m-13-17.pdf

support from the Coalition For Evidence Based Policy

http://coalition4evidence.org/wp-content/uploads/2013/07/Coalition-Board-of-Advisors2168 The Promise of Evidence-Based Policymaking
Update-07-11-13.pdf

Can Silicon Valley Save the World: Foreign Policy

http://foreignpolicy.com/2013/06/24/can-silicon-valley-save-the-world/

and more at:

http://divatusaid.tumblr.com/tagged/What%20others%20are%20saying%20about%20us

and www.usaid.gov/div 

The Promise of Evidence-Based Policymaking 2169
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl4-6cx7

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0289

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

This madness must be stopped. Our Constitution guarantees United States citizens the

right to life, liberty, and the pursuit of happiness. This is MY right to my own life and

my children's right to their own lives. This does not give the government a right to OUR

lives. The data mining of our children in the name of education is a violation of our basic

God given rights which are protected by the Constitution.

2170 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl4-htbz

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0290

Comment on FR Doc # 2016-22002

Submitter Information

Name: Marianne Dwyer

General Comment

Our children are not to be used for research and marketing. Stop using our children's

information for your gain. Their privacy is too important

The Promise of Evidence-Based Policymaking 2171
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl4-ovtp

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0291

Comment on FR Doc # 2016-22002

Submitter Information

Name: Wendy Ruf

General Comment

I am opposed to allowing greater data access. Children's data needs to be managed by

their parents. Allowing greater access

has the potential to pigeon-hole children from an early age and limit their opportunities

in the future. People who have never 

seen, spoken with, or cared about a particular child, will be able to know every detail

about them and make decisions about 

them without their parents knowledge or consent. Parents have always been the only

source of the of ALL information on 

their children. They could choose what information and to whom this information could

and should be shared. Now, they don't

even know what information it is being shared. How can we possibly say that 'the best

interests' of a child are paramount when

this information is being proffered without the individual, parents or teachers who know

and care about them?

2172 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl4-ulu2

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0292

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

Stop data mining Americans without their informed consent... INFORMED meaning you

inform us what you are gathering, what it will be used for and then you are not allowed

to data mine us without our signed consent. Where in the Constitution does it say that the

federal government has the right to subject citizens to this?

The Promise of Evidence-Based Policymaking 2173
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl5-37qm

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0293

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

We must stop the data mining and we must push for restoration of FERPA and COPA

laws.There must be INFORMED consent with OPT-INs, not opt-OUTS! No "surveys",

either, without opt-ins. FERPA was absolutely GUTTED in 2011 (by regulation) and

now anyone claiming to have an "educational interest" has access to our kids' PII.

Personally indentifiable information is PRIVATE PROPERTY and this private property

is being stolen from our kids & used & abused on a daily basis. Most tragically, it is

being used to push the progressive ideology on our kids & to plug them into one of 16

"career clusters" -- as young as 6th grade. This WFD (workforce development) model is

supported by Dems and Republicans alike and our kids are being USED by business and

publishers and others. This process MIGHT be somewhat acceptable if it was directed by

the child and parent, but it is NOT. (And yes, kids can be identified by very few data

points, especially kids in small schools). Parents will never quit fighting this data theft

and abuse. We will never quit fighting to protect our kids.

2174 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl5-oso7

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0294

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

I believe in informed consent. I oppose non-consensual data mining. Stop this madness.

The Promise of Evidence-Based Policymaking 2175
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl5-94jk

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0295

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

I believe in informed consent and oppose non-consensual data mining. Data privacy is at

the core of our liberty and for the government to unilaterally use my data without my

consent is unconstituional.

2176 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl5-ca4z

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0296

Comment on FR Doc # 2016-22002

Submitter Information

Name: Paul Blair

General Comment

The purpose of education is to provide our children a classic liberal arts education. This

is a necessity for a free people where the government is accountable to "we the people."

There is no need for data mining in a free society. Data mining is only used by those

societies that wish to exercise control over the people by the government.

The Promise of Evidence-Based Policymaking 2177
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl5-lqxz

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0297

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

Although intentions may be good, each step taken in these types of things makes life in

general more difficult. These things highly affect "trust". Unfortunately policies lije

these create wedges, and do not unite community/parents. Children need to be allowed to

be children without being spied on. Parents need to be comfortable in trusting that the

true sole purpose of a public education for their children is to educate children. These

things have nothing to do with educating children and more to do with turning our

schools into scientific testing labs whether for economic science or otherwise.

2178 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl5-b0pz

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0298

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Reed

General Comment

I believe in informed consent. I oppose non-consensual data mining.

The Promise of Evidence-Based Policymaking 2179
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl5-e7ei

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0299

Comment on FR Doc # 2016-22002

Submitter Information

Name: Jan Mayfield

General Comment

Stop data mining, need consent

2180 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl5-fvyb

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0300

Comment on FR Doc # 2016-22002

Submitter Information

Name: Kelly Eggertsen

Address:

Salt Lake City, UT, 84105

Email: KELLYATHOMPSON@MSN.COM

General Comment

Please stop the madness of data mining our students and, especially, doing so without

parental consent. Student information should be private and respected. Moreover the

companies and corporations that benefit so much from this data-mining should have to

ask permission for it, pay the subject for the information and abide by respectable

standards of research and development. FERPA needs to be amended to actually do what

most parents think that it does. It needs to have muscles amended back into it to protect

our students from exploitation. What's going on right now is not right. Please have the

courage to shut it down. Also, this is unfair to the taxpayer who fronts this bill while

corporations benefit. 

The Promise of Evidence-Based Policymaking 2181
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl6-pz7s

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0301

Comment on FR Doc # 2016-22002

Submitter Information

Name: Tina Anonymous

Address:

UT,

General Comment

I am opposed to allowing greater data access. Children's data needs to be properly

managed by their parents. In medicine, children's data would only be allowed to be used

in research with the informed consent of their parent or guardian. But in education this is

lacking. Why would we want it any less invasive? We live in an era where people who

have never met my child, will be able to know vast amounts of information about them

and make decisions about them without me even being able to say yes or no to the

release of their information. In the past, parents were the only source of the information

on their children. They could choose what information and to whom this information

should be shared with.

2182 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl6-iprn

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0302

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

Explicit informed consent of ANY data gleaning of an individual is THEFT of individual

intellectual property. Compulsory, mandated, legislated, un-transparent participation in

technology environments is THEFT of individual intellectual property and an invasion of

privacy and robs the individual of the ability to protect himself.

Those who contrived to construct and force these mandates and systems on the

individual through legislative/government and corporate means are cowards and

inhumane. They are not for choice. They are not leaders.

They are cowards and weak-minded individuals who are afraid to allow human beings

the ability to make their own decisions and compete on the same playing field.

They have to create protectionist environments that allow them to be successful and

control others. They are cowards and weak.

Worst of all, those of you supporting and working to data gleaning initiatives are

inhumane.

The Promise of Evidence-Based Policymaking 2183
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl6-620s

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0303

Comment on FR Doc # 2016-22002

Submitter Information

Name: karen price

General Comment

I believe in informed consent.

2184 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl6-hzma

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0304

Comment on FR Doc # 2016-22002

Submitter Information

Name: Charlotte O'Hara

General Comment

Data mining our children is an unconscionable act in and of itself. Then to add insult to

injury, allowing data collection on a child without direct parental knowledge and consent

is abhorrent to every tenant of a free society. Absolutely no data mining without full

disclosure to the parents and consent from the parents.

The Promise of Evidence-Based Policymaking 2185
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl6-a58d

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0305

Comment on FR Doc # 2016-22002

Submitter Information

Name: Ivan Richeson

General Comment

Just more unneeded, unwanted and inappropriate intrusion by the government

2186 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl6-1rp3

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0306

Comment on FR Doc # 2016-22002

Submitter Information

Name: Patty Burke

General Comment

Would any of you like your personal information tracked from birth to death? I think

not. It is wrong. We are not a communist country yet. I'll be fighting every day to make

sure that we don't become a communist country, including supporting political

candidates that uphold the U.S. constitution.

The Promise of Evidence-Based Policymaking 2187
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl6-86om

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0307

Comment on FR Doc # 2016-22002

Submitter Information

Name: Tina LeBaron

General Comment

Please no. Information should always be private and not shared with anyone.

2188 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl7-mkyn

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0308

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

The existing set up of the Federal Statistical Research Data System and existing calls for

research from the IRS's SOI program allow for important research which can inform

policy. However, the IRS system limits access to a small number of researchers. Better

evidence comes when many researchers have access to these type of data. Existing

efforts like the QCEW program and synthetic SIPP data allow researchers to use linked

administrative data without risking privacy. Other OECD countries have ways of

protecting privacy while allowing widespread access to administrative data. We should

copy these. It would be good to require that states share data on federally funded

programs and link these to federal data maintained by IRS and SSA to get a full picture

of program use and income. these data should be made more broadly available to

academics at universities and think tanks much as the FSRDC system does. Using these

data there requires that individuals promise to never reveal the data.

The Promise of Evidence-Based Policymaking 2189
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl7-xzro

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0309

Comment on FR Doc # 2016-22002

Submitter Information

Name: Gregory Goode

General Comment

Stop...just stop...that means NO! You may not data mine our children....WTH?????

2190 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl7-sbi0

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0310

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

I am 100% opposed to the government tracking me or my children in any way or form.

You have no business collecting this data.

The Promise of Evidence-Based Policymaking 2191
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl7-drzt

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0311

Comment on FR Doc # 2016-22002

Submitter Information

Name: Jenny Baker

General Comment

Personal Identifiable data must not be mined without personal consent.

2192 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl7-1j1k

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0312

Comment on FR Doc # 2016-22002

Submitter Information

Name: Sallie Henry

General Comment

I oppose non consensual data mining. It is an infringement on the liberty of the

American people.

The Promise of Evidence-Based Policymaking 2193
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl7-w5tb

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0313

Comment on FR Doc # 2016-22002

Submitter Information

Name: Debra Miller

General Comment

I believe in parental rights. I oppose non-consensual and hidden data mining. Stop this

madness."

2194 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl7-a9t2

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0314

Comment on FR Doc # 2016-22002

Submitter Information

Name: Irma Holmes

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all public school students,

commonly referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008 the reasons against creating it

have only become more persuasive.

In the past few years, data held by federal agencies has been hacked, including the

personal information more than 22 million individuals, not only federal employees and

contractors but also their families and friends, from the records of the Office of

Personnel Management. The US Department of Education has especially weak security

standards in its collection and storage of student data, and recently received a grade of

"D" for its security protections.

The Promise of Evidence-Based Policymaking 2195
In addition, preK-12 student data currently collected by State Education Departments

that would potentially be shared with the federal government include upwards of 700

highly sensitive elements, including students' immigrant status, disabilities, disciplinary

records, and homelessness.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from preK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban.

Yours,

Irma Holmes Elk Point, South Dakota

2196 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl8-nr1t

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0315

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

I believe in informed consent. I oppose non-consensual data mining. Stop this madness.

The Promise of Evidence-Based Policymaking 2197
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl8-rijc

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0316

Comment on FR Doc # 2016-22002

Submitter Information

Name: Karen Jenkins

General Comment

I believe in informed consent. Stop this madness!

2198 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl8-vumc

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0317

Comment on FR Doc # 2016-22002

Submitter Information

Name: Shane Trahan

Address:

Durham, 27709

Email: srt@rti.org

General Comment

On behalf of RTI International thank you for the opportunity to provide comments for

the Commission on Evidence-Based Policymaking. We welcome the opportunity to

contribute more research and assistance in development of this commissions objectives

now and in the future. RTI International was established in 1958 and is one of the

nation's largest nonprofit research organizations, employing 4,150 employees who work

in several U.S. locations and nearly 85 countries. Contract research is our core business;

we provide unparalleled study and analysis in health and pharmaceuticals, education and

training, surveys and statistics, advanced technology, international development,

economic and social policy, energy and the environment, and laboratory and chemistry

services.

Attachments

EvidenceBasedPolicyComments

The Promise of Evidence-Based Policymaking 2199
ZĞƋƵĞƐƚĨŽƌŽŵŵĞŶƚǀŝĚĞŶĐĞĂƐĞĚWŽůŝĐLJ

ĞƉĂƌƚŵĞŶƚŽĨŽŵŵĞƌĐĞŽĐŬĞƚϭςϬεϬϳΘϮρͲςΘϮρͲϬϭ

KŶďĞŚĂůĨŽĨZd//ŶƚĞƌŶĂƚŝŽŶĂůƚŚĂŶŬLJŽƵĨŽƌƚŚĞŽƉƉŽƌƚƵŶŝƚLJƚŽƉƌŽǀŝĚĞĐŽŵŵĞŶƚƐĨŽƌƚŚĞŽŵŵŝƐƐŝŽŶ

ŽŶǀŝĚĞŶĐĞͲĂƐĞĚWŽůŝĐLJŵĂŬŝŶŐ͘tĞǁĞůĐŽŵĞƚŚĞŽƉƉŽƌƚƵŶŝƚLJƚŽĐŽŶƚƌŝďƵƚĞŵŽƌĞƌĞƐĞĂƌĐŚĂŶĚ

ĂƐƐŝƐƚĂŶĐĞŝŶĚĞǀĞůŽƉŵĞŶƚŽĨƚŚŝƐĐŽŵŵŝƐƐŝŽŶƐŽďũĞĐƚŝǀĞƐ͘Zd//ŶƚĞƌŶĂƚŝŽŶĂůǁĂƐĞƐƚĂďůŝƐŚĞĚŝŶϭερΘĂŶĚ

ŝƐŽŶĞŽĨƚŚĞŶĂƚŝŽŶ͛ƐůĂƌŐĞƐƚŶŽŶƉƌŽĨŝƚƌĞƐĞĂƌĐŚŽƌŐĂŶŝnjĂƚŝŽŶƐ͕ĞŵƉůŽLJŝŶŐκ͕ϭρϬĞŵƉůŽLJĞĞƐǁŚŽǁŽƌŬŝŶ

ƐĞǀĞƌĂůh͘^͘ůŽĐĂƚŝŽŶƐĂŶĚŶĞĂƌůLJΘρĐŽƵŶƚƌŝĞƐ͘ŽŶƚƌĂĐƚƌĞƐĞĂƌĐŚŝƐŽƵƌĐŽƌĞďƵƐŝŶĞƐƐ͖ǁĞƉƌŽǀŝĚĞ

ƵŶƉĂƌĂůůĞůĞĚƐƚƵĚLJĂŶĚĂŶĂůLJƐŝƐŝŶŚĞĂůƚŚĂŶĚƉŚĂƌŵĂĐĞƵƚŝĐĂůƐ͕ĞĚƵĐĂƚŝŽŶĂŶĚƚƌĂŝŶŝŶŐ͕ƐƵƌǀĞLJƐĂŶĚ

ƐƚĂƚŝƐƚŝĐƐ͕ĂĚǀĂŶĐĞĚƚĞĐŚŶŽůŽŐLJ͕ŝŶƚĞƌŶĂƚŝŽŶĂůĚĞǀĞůŽƉŵĞŶƚ͕ĞĐŽŶŽŵŝĐĂŶĚƐŽĐŝĂůƉŽůŝĐLJ͕ĞŶĞƌŐLJĂŶĚƚŚĞ

ĞŶǀŝƌŽŶŵĞŶƚ͕ĂŶĚůĂďŽƌĂƚŽƌLJĂŶĚĐŚĞŵŝƐƚƌLJƐĞƌǀŝĐĞƐ͘

ĂƐĞĚŽŶŝĚĞŶƚŝĨŝĞĚďĞƐƚƉƌĂĐƚŝĐĞƐĂŶĚĞdžŝƐƚŝŶŐĞdžĂŵƉůĞƐ͕ǁŚĂƚĨĂĐƚŽƌƐƐŚŽƵůĚďĞĐŽŶƐŝĚĞƌĞĚŝŶ

ƌĞĂƐŽŶĂďůLJĞŶƐƵƌŝŶŐƚŚĞƐĞĐƵƌŝƚLJĂŶĚƉƌŝǀĂĐLJŽĨĂĚŵŝŶŝƐƚƌĂƚŝǀĞĂŶĚƐƵƌǀĞLJĚĂƚĂ͍

ŶƐƵƌŝŶŐƚŚĞƐĞĐƵƌŝƚLJŽĨĚĂƚĂŝƐĐƌŝƚŝĐĂůƚŽŝŶƐƚŝůůŝŶŐƚƌƵƐƚŝŶĂŶLJƐLJƐƚĞŵ͘dŽƐƵƉƉŽƌƚĞǀŝĚĞŶĐĞͲ

ďĂƐĞĚƉŽůŝĐLJ͕ĞdžŝƐƚŝŶŐĚĂƚĂǁŝůůďĞĐŽŵďŝŶĞĚĂŶĚůŝŶŬĞĚǁŝƚŚŽƚŚĞƌĚĂƚĂĐŽůůĞĐƚĞĚƚŚƌŽƵŐŚŶĞǁ

ƉƌŽŐƌĂŵƐǀŝĂƐƵƌǀĞLJŽƌŽƚŚĞƌŵĞĂŶƐ͘dŽĞŶƐƵƌĞƚŚĞŝŶƚĞŐƌŝƚLJŽĨƚŚĞĚĂƚĂ͕ǁĞƌĞĐŽŵŵĞŶĚƐƚŽƌŝŶŐ

ŝŶĨŽƌŵĂƚŝŽŶĂŶĚĚĂƚĂǁŝƚŚŝŶĂƚŝĞƌĞĚƐLJƐƚĞŵďĂƐĞĚŽŶĚĂƚĂĐŚĂƌĂĐƚĞƌŝƐƚŝĐƐ͘dŚŝƐƚLJƉĞŽĨŵŽĚĞů

ĐĂŶŝŶĐůƵĚĞĚĂƚĂƚŚĂƚǁŽƵůĚĨŝƚǁŝƚŚŝŶ&ĞĚĞƌĂů/ŶĨŽƌŵĂƚŝŽŶWƌŽĐĞƐƐŝŶŐ^ƚĂŶĚĂƌĚƐ;&/W^ͿůŽǁŽƌ

ŵŽĚĞƌĂƚĞĐŽŵƉƵƚŝŶŐĞŶǀŝƌŽŶŵĞŶƚƐ͘ĂĐŚĞŶǀŝƌŽŶŵĞŶƚŝƐĞdžƉůĂŝŶĞĚďĞůŽǁ͗

&/W^>ŽǁͲ&/W^ůŽǁŝƐŝŶĐůƵƐŝǀĞŽĨƉƵďůŝĐůLJĂǀĂŝůĂďůĞĚĂƚĂƚŚĂƚŚĂƐďĞĞŶƚŚŽƌŽƵŐŚůLJƐĐƌƵďďĞĚ͕ĚĞͲ

ŝĚĞŶƚŝĨŝĞĚ͕ŽƌĂŐŐƌĞŐĂƚĞĚƐŽƚŚĂƚŶŽƉĞƌƐŽŶĂůůLJŝĚĞŶƚŝĨŝĂďůĞŝŶĨŽƌŵĂƚŝŽŶĐĂŶďĞŽďƚĂŝŶĞĚƚŚƌŽƵŐŚ

ĂŶĂůLJƐŝƐ͘ĂƚĂĐŽŶƐƵŵĞƌƐĐĂŶĚŽǁŶůŽĂĚĂŶĚĂŶĂůLJnjĞ&/W^ůŽǁĚĂƚĂƚŚƌŽƵŐŚĂŶĞĂƐLJͲƚŽͲƵƐĞ

ŝŶƚĞƌĨĂĐĞƌĞƋƵŝƌŝŶŐůŝŵŝƚĞĚĂƵƚŚĞŶƚŝĐĂƚŝŽŶ͘džĂŵƉůĞƐŝŶĐůƵĚĞƚŚĞŵĞƌŝĐĂŶŽŵŵƵŶŝƚLJ^ƵƌǀĞLJ

;^ͿĂŶĚĞŶƐƵƐƌĞůĂƚĞĚĚĂƚĂ͘dŚĞƚĂƌŐĞƚĂƵĚŝĞŶĐĞĨŽƌƚŚŝƐƚLJƉĞŽĨĚĂƚĂŝƐďƌŽĂĚĂŶĚĨĂƌͲ

ƌĞĂĐŚŝŶŐ͘

&/W^DŽĚĞƌĂƚĞͲ&/W^ŵŽĚĞƌĂƚĞĞŶǀŝƌŽŶŵĞŶƚƐĂƌĞŝŶĐůƵƐŝǀĞŽĨƉůĂƚĨŽƌŵƐƚŚĂƚĂƌĞŚŝŐŚůLJ

ƌĞƐƚƌŝĐƚŝǀĞ͘^ƵĐŚƉůĂƚĨŽƌŵƐƐŚŽƵůĚŝŶĐůƵĚĞĂƌŝŐŽƌŽƵƐǀĞƚƚŝŶŐƉƌŽĐĞƐƐƚŽĂůůŽǁĂĐĐĞƐƐƚŽŵŝĐƌŽͲ

ĚĂƚĂƚŚĂƚŵĂLJŝŶĐůƵĚĞƉĞƌƐŽŶĂůůLJŝĚĞŶƚŝĨŝĂďůĞŝŶĨŽƌŵĂƚŝŽŶ͕ŚĞĂůƚŚͲƌĞůĂƚĞĚĚĂƚĂ͕ƚĂdžƌĞůĂƚĞĚĚĂƚĂ͕

ŬĞLJŝŶĚƵƐƚƌŝĂůƉĞƌĨŽƌŵĂŶĐĞĚĂƚĂ͕ĂŶĚŽƚŚĞƌƚLJƉĞƐŽĨƉĞƌƐŽŶĂůŽƌĐŽŶĨŝĚĞŶƚŝĂůĚĂƚĂ͘&/W^

ŵŽĚĞƌĂƚĞƉůĂƚĨŽƌŵƐĂůůŽǁƌĞƐĞĂƌĐŚĞƌƐƚŽĂĐĐĞƐƐĚĂƚĂǁŝƚŚŝŶĂƐĞĐƵƌĞĚĞŶǀŝƌŽŶŵĞŶƚĨŽƌůŝŵŝƚĞĚ

ĂŵŽƵŶƚŽĨƚŝŵĞĂŶĚƵŶĚĞƌƚŚĞǁĂƚĐŚĨƵůĞLJĞŽĨƚŚĞĚĂƚĂƉƌŽǀŝĚĞƌ͘ZĞƐĞĂƌĐŚĚĞƌŝǀĞĚĨƌŽŵƐƵĐŚ

ĚĂƚĂŝƐƌĞǀŝĞǁĞĚďLJĂƐƉŽŶƐŽƌďĞĨŽƌĞƌĞƐƵůƚƐĂƌĞƐŚĂƌĞĚ͕ĞŶƐƵƌŝŶŐƚŚĂƚĚĂƚĂĂŶĚŽƚŚĞƌĞůĞŵĞŶƚƐ

ŽĨƚŚĞƌĞƐĞĂƌĐŚŵĂŝŶƚĂŝŶĐŽŶĨŝĚĞŶƚŝĂůŝƚLJŽĨƚŚĞŝŶĨŽƌŵĂƚŝŽŶĐŽŶƚĂŝŶĞĚǁŝƚŚŝŶƚŚĞĞŶǀŝƌŽŶŵĞŶƚ͘

hƐŝŶŐƚŚŝƐƚǁŽͲƚŝĞƌĞĚĂƉƉƌŽĂĐŚǁŽƵůĚĂůůŽǁďĂƐŝĐƌĞƐĞĂƌĐŚƚŽďĞƐƚŽƌĞĚĂŶĚĂĐĐĞƐƐĞĚǁŝƚŚŝŶĂ

&/W^ůŽǁĞŶǀŝƌŽŶŵĞŶƚĂŶĚŵŽƌĞƚĂƌŐĞƚĞĚƌĞƐĞĂƌĐŚƚŽďĞƉĞƌĨŽƌŵĞĚǁŝƚŚŝŶƚŚĞ&/W^ŵŽĚĞƌĂƚĞ

ĞŶǀŝƌŽŶŵĞŶƚ͘ƵƌƌĞŶƚůLJ͕ƐĞǀĞƌĂůĞŶƐƵƐƵƌĞĂƵZĞƐĞĂƌĐŚĂƚĂĞŶƚĞƌƐ;Z͛ƐͿůŽĐĂƚĞĚĂƌŽƵŶĚ

ƚŚĞĐŽƵŶƚƌLJĞŵƉůŽLJ&/W^ŵŽĚĞƌĂƚĞĞŶǀŝƌŽŶŵĞŶƚƐƚŽĂůůŽǁƌĞƐĞĂƌĐŚĞƌƐƚŽĂĐĐĞƐƐĞŶƐƵƐ

ŵŝĐƌŽĚĂƚĂĨŽƌŚŝŐŚůLJĐŽŵƉƵƚĂƚŝŽŶĂůƌĞƐĞĂƌĐŚ͘dŚĞƐĞZĞƐĞĂƌĐŚĂƚĂĞŶƚĞƌƐĂƌĞůŽĐĂƚĞĚǁŝƚŚŝŶ

ƌĞŐŝŽŶĂůƌĞƐĞĂƌĐŚĐĞŶƚĞƌƐǁŚĞƌĞŚŝŐŚůLJƐŬŝůůĞĚƐƚĂĨĨĂƌĞĂďůĞƚŽŵĂŝŶƚĂŝŶĚĂƚĂƐĞĐƵƌŝƚLJǁŚŝůĞ

ŚĞůƉŝŶŐƌĞƐĞĂƌĐŚĞƌƐĂĐĐĞƐƐĂŶĚƵƐĞƚŚĞĚĂƚĂ͘hƐĞƌƐŽĨƚŚĞŵŝĐƌŽĚĂƚĂĂƌĞƌĞƋƵŝƌĞĚƚŽƐƵďŵŝƚ

ĂďƐƚƌĂĐƚƐĚĞƐĐƌŝďŝŶŐƚŚĞƌĞƐĞĂƌĐŚƚŚĞLJĂƌĞĐĂƌƌLJŝŶŐŽƵƚĂŶĚŝŶĚŝĐĂƚŝŶŐŚŽǁůŽŶŐƚŚĞLJƌĞƋƵŝƌĞ

2200 The Promise of Evidence-Based Policymaking
ĂĐĐĞƐƐƚŽƚŚĞĚĂƚĂ͘dŚĞ&/W^ŵŽĚĞƌĂƚĞĞŶǀŝƌŽŶŵĞŶƚƐŝŶĐůƵĚĞƐŽĨƚǁĂƌĞƚŽŽůƐƐƵĐŚĂƐ^^͕

^hE͕ĂŶĚZƐƚĂƚŝƐƚŝĐĂůƉĂĐŬĂŐĞƐ͖ĚĂƚĂďĂƐĞƉůĂƚĨŽƌŵƐƐƵĐŚĂƐKƌĂĐůĞ͕^Y>^ĞƌǀĞƌ͕ĂŶĚDLJ^Y>͖

ĂŶĚĐŽŵŵŽŶĂƉƉůŝĐĂƚŝŽŶƐƐƵĐŚĂƐDŝĐƌŽƐŽĨƚtŽƌĚĂŶĚdžĐĞů͘dŚĞƐĞƚŽŽůƐĂůůŽǁƌĞƐĞĂƌĐŚĞƌƐƚŽ

ƉĞƌĨŽƌŵƌĞƐĞĂƌĐŚǁŝƚŚŝŶƚŚĞƐĞĐƵƌĞĚĞŶǀŝƌŽŶŵĞŶƚ͘dŽĞŶƐƵƌĞĐŽŶĨŝĚĞŶƚŝĂůŝƚLJ͕ŝŶĨŽƌŵĂƚŝŽŶ

ĐĂŶŶŽƚůĞĂǀĞƚŚŝƐƐĞĐƵƌĞĚĞŶǀŝƌŽŶŵĞŶƚƵŶƚŝůŝƚŚĂƐďĞĞŶƌĞǀŝĞǁĞĚďLJZĞƐĞĂƌĐŚĂƚĂĞŶƚĞƌƐƚĂĨĨ͘

WƵďůŝƐŚĞĚƌĞƐĞĂƌĐŚƌĞƐƵůƚŝŶŐĨƌŽŵǁŽƌŬĂƚƚŚĞƐĞZĞƐĞĂƌĐŚĂƚĂĞŶƚĞƌƐŝŶĐůƵĚĞƐďƵƚŝƐŶŽƚ

ůŝŵŝƚĞĚƚŽƉĂƉĞƌƐĨŽƌƉƵďůŝĐĂƚŝŽŶƐŽƌŶĞǁůLJĐƌĞĂƚĞĚĚĂƚĂƐĞƚƐ͘ƐŝŵŝůĂƌŵŽĚĞůŝƐĐƵƌƌĞŶƚůLJďĞŝŶŐ

ƵƐĞĚďLJƚŚĞ^ƵďƐƚĂŶĐĞďƵƐĞĂŶĚDĞŶƚĂů,ĞĂůƚŚ^ĞƌǀŝĐĞƐĚŵŝŶŝƐƚƌĂƚŝŽŶƐ;^D,^Ϳ^ƵďƐƚĂŶĐĞ

ďƵƐĞĂŶĚDĞŶƚĂů,ĞĂůƚŚĂƚĂƌĐŚŝǀĞ;^D,Ϳ͘ŶĂůLJƐŝƐŽŶƉƵďůŝĐĚĂƚĂŝƐĚŽŶĞƚŚƌŽƵŐŚĂ

ŚŝŐŚůLJŝŶƚĞƌĂĐƚŝǀĞǁĞďͲďĂƐĞĚƉůĂƚĨŽƌŵ͘ZĞƐĞĂƌĐŚĞƌƐĐĂŶĂƉƉůLJĨŽƌǀŝƌƚƵĂůĂĐĐĞƐƐƚŽŵŽƌĞĚĞƚĂŝůĞĚ

ŵŝĐƌŽĚĂƚĂĐŽůůĞĐƚĞĚƚŚƌŽƵŐŚŽƚŚĞƌƐƵƌǀĞLJƐ͕ŝŶĐůƵĚŝŶŐƚŚĞEĂƚŝŽŶĂů^ƵƌǀĞLJŽĨƌƵŐhƐĞĂŶĚ

,ĞĂůƚŚ͘

,ŽǁƐŚŽƵůĚƚŚĞŽŵŵŝƐƐŝŽŶĚĞĨŝŶĞ͞ƋƵĂůŝĨŝĞĚƌĞƐĞĂƌĐŚĞƌƐĂŶĚŝŶƐƚŝƚƵƚŝŽŶƐ͍͟dŽǁŚĂƚĞdžƚĞŶƚƐŚŽƵůĚ

ĂĚŵŝŶŝƐƚƌĂƚŝǀĞĂŶĚƐƵƌǀĞLJĚĂƚĂŚĞůĚďLJŐŽǀĞƌŶŵĞŶƚĂŐĞŶĐŝĞƐďĞŵĂĚĞĂǀĂŝůĂďůĞƚŽ͞ƋƵĂůŝĨŝĞĚ

ƌĞƐĞĂƌĐŚĞƌƐĂŶĚŝŶƐƚŝƚƵƚŝŽŶƐ͍͟

͞YƵĂůŝĨŝĞĚƌĞƐĞĂƌĐŚĞƌƐĂŶĚŝŶƐƚŝƚƵƚŝŽŶƐ͟ƐŚŽƵůĚŝŶĐůƵĚĞŚŝŐŚůLJƐŬŝůůĞĚƐƚĂƚŝƐƚŝĐŝĂŶƐĂŶĚƌĞůĞǀĂŶƚ

ƐƵďũĞĐƚŵĂƚƚĞƌĞdžƉĞƌƚƐŝŶĂƉƉƌŽƉƌŝĂƚĞĨŝĞůĚƐĂŶĚŽƌŐĂŶŝnjĂƚŝŽŶƐƚŚĂƚƵŶĚĞƌƐƚĂŶĚƚŚĞŝŵƉŽƌƚĂŶĐĞ

ŽĨĚĂƚĂƐĞĐƵƌŝƚLJĨŝƌƐƚĂŶĚĨŽƌĞŵŽƐƚĂŶĚŚĂǀĞĚĞŵŽŶƐƚƌĂƚĞĚĂŶĂďŝůŝƚLJƚŽƉƌŽǀŝĚĞƚŚĞŶĞĐĞƐƐĂƌLJ

ƐŬŝůůƐĞƚƐƚŽĨƵůůLJƵŶĚĞƌƐƚĂŶĚĂŶĚƵƚŝůŝnjĞĚĂƚĂĂŶĂůLJƐŝƐ͘dŚĞŐŽĂůƐŚŽƵůĚŶŽƚďĞƚŽůŝŵŝƚƚŚĞƵƐĞŽĨ

ƚŚĞĚĂƚĂďƵƚƐŚŽƵůĚďĞũƵƐƚƚŚĞŽƉƉŽƐŝƚĞ͗ŝƚƐŚŽƵůĚďĞƚŚĞĚĞƐŝƌĞŽĨƚŚĞŽŵŵŝƐƐŝŽŶĂŶĚŽƚŚĞƌƐ

ǁŝƚŚŝŶƚŚĞŐŽǀĞƌŶŵĞŶƚƚŽŵĂŬĞƚŚŝƐĚĂƚĂĂǀĂŝůĂďůĞĨŽƌĞǀŝĚĞŶĐĞͲďĂƐĞĚƉŽůŝĐLJƌĞƐĞĂƌĐŚďLJƉŽůŝĐLJ

ŵĂŬĞƌƐ͕ĞĚƵĐĂƚŝŽŶĂůŝŶƐƚŝƚƵƚŝŽŶƐ͕ĨŽƵŶĚĂƚŝŽŶƐ͕ĂŶĚĐŽŵŵĞƌĐŝĂůŝŶƚĞƌĞƐƚƐĂƐǁĞůů͘dŚĞ

ŽŵŵŝƐƐŝŽŶƐŚŽƵůĚĞŶĐŽƵƌĂŐĞƉĂƌƚŶĞƌƐŚŝƉƐĂŵŽŶŐĂůůĨĂĐĞƚƐŽĨŐŽǀĞƌŶŵĞŶƚ͕ĂĐĂĚĞŵŝĂ͕ĂŶĚ

ŝŶĚƵƐƚƌLJƚŚĂƚŚĂǀĞĂǀĞƐƚĞĚŝŶƚĞƌĞƐƚŝŶƐƵĐĐĞƐƐĨƵůůLJŝŵƉůĞŵĞŶƚŝŶŐƉŽůŝĐŝĞƐƚŚĂƚďĞŶĞĨŝƚh^

ŝŶƚĞƌĞƐƚƐĂĐƌŽƐƐƚŚĞŶĂƚŝŽŶĂŶĚĂƌŽƵŶĚƚŚĞǁŽƌůĚ͘ĂƚĂƚŚĂƚŝŶĐůƵĚĞƐďƵƚŝƐŶŽƚůŝŵŝƚĞĚƚŽ

ƉĞƌƐŽŶĂůůLJŝĚĞŶƚŝĨŝĂďůĞŝŶĨŽƌŵĂƚŝŽŶ͕ŚĞĂůƚŚͲƌĞůĂƚĞĚĚĂƚĂ͕ƚĂdžͲƌĞůĂƚĞĚĚĂƚĂ͕ĂŶĚŬĞLJŝŶĚƵƐƚƌŝĂů

ƉĞƌĨŽƌŵĂŶĐĞĚĂƚĂƐŚŽƵůĚďĞŬĞƉƚŝŶĂŚŝŐŚůLJƐĞĐƵƌĞĐŽŵƉƵƚŝŶŐĞŶǀŝƌŽŶŵĞŶƚƚŚĂƚŝƐĂǀĂŝůĂďůĞƚŽ

ƋƵĂůŝĨŝĞĚƌĞƐĞĂƌĐŚĞƌƐĂŶĚŝŶƐƚŝƚƵƚŝŽŶƐƉƌŽǀŝĚĞĚƚŚĞLJĂŐƌĞĞƚŽĚĂƚĂĐŽŶĨŝĚĞŶƚŝĂůŝƚLJƌĞƐƚƌŝĐƚŝŽŶƐ͘dŽ

ĨŽƐƚĞƌŽƉĞŶŶĞƐƐĂŶĚďĂƐŝĐƌĞƐĞĂƌĐŚ͕ĚĂƚĂƚŚĂƚŚĂƐďĞĞŶƐĐƌƵďďĞĚƐŚŽƵůĚďĞŵĂĚĞǁŝĚĞůLJ

ĂǀĂŝůĂďůĞĂŶĚĞĂƐLJƚŽĂĐĐĞƐƐ͘

tŚĂƚƚĞĐŚŶŽůŽŐŝĐĂůƐŽůƵƚŝŽŶƐĨƌŽŵŐŽǀĞƌŶŵĞŶƚŽƌƚŚĞƉƌŝǀĂƚĞƐĞĐƚŽƌĂƌĞƌĞůĞǀĂŶƚĨŽƌĨĂĐŝůŝƚĂƚŝŶŐĚĂƚĂ

ƐŚĂƌŝŶŐĂŶĚŵĂŶĂŐĞŵĞŶƚ͍

ZĂƚŚĞƌƚŚĂŶĚŝĐƚĂƚŝŶŐƌĞůĞǀĂŶƚƐŽůƵƚŝŽŶƐ͕ƚŚĞŽŵŵŝƐƐŝŽŶƐŚŽƵůĚĨŽĐƵƐŽŶďƵŝůĚŝŶŐĂĨƌĂŵĞǁŽƌŬ

ĨŽƌĚĂƚĂƐŚĂƌŝŶŐĂŶĚŵĂŶĂŐĞŵĞŶƚďĂƐĞĚŽŶĚĞĨŝŶĞĚƐƚĂŶĚĂƌĚƐŽĨĚĂƚĂƐŚĂƌŝŶŐ͘dŚĞƐĞƐƚĂŶĚĂƌĚƐ

ĐŽƵůĚŝŶĐůƵĚĞĂŐƌĞĞĚͲƵƉŽŶĚĂƚĂĨŽƌŵĂƚƐĂŶĚƐƚĂŶĚĂƌĚƐĞƐƚĂďůŝƐŚĞĚďLJǁŽƌŬŝŶŐŐƌŽƵƉƐ

ĐŽŵƉƌŝƐĞĚŽĨĂĐĂĚĞŵŝĐƐĂŶĚŽƚŚĞƌƐƵďũĞĐƚŵĂƚƚĞƌĞdžƉĞƌƚƐ͘ĞĐĂƵƐĞƚĞĐŚŶŽůŽŐLJĐŚĂŶŐĞƐƌĂƉŝĚůLJ͕

ƚŽĨŽĐƵƐŽŶƚĞĐŚŶŽůŽŐLJŝƚƐĞůĨĐŽƵůĚƐƚŝĨůĞŝŶŶŽǀĂƚŝŽŶĂŶĚĐƌĞĂƚĞƐŝůŽƐŽĨĚĂƚĂƚƌĂƉƉĞĚǁŝƚŚŝŶ

ŽƵƚĚĂƚĞĚƐŽůƵƚŝŽŶƐ͘dŚĞǁŽƌŬŝŶŐŐƌŽƵƉƐǁŽƵůĚďĞĂďůĞƚŽƉƌŽŵŽƚĞĚĂƚĂĂĐĐĞƐƐĂŶĚ

ƐƚĂŶĚĂƌĚŝnjĂƚŝŽŶŽĨĚĂƚĂĂŶĚĨĂĐŝůŝƚĂƚĞƚŚĞĂŶĂůLJƐŝƐŽĨĞdžŝƐƚŝŶŐĚĂƚĂĂŶĚƚŚĞĐŽůůĞĐƚŝŽŶŽĨŶĞǁĚĂƚĂ

The Promise of Evidence-Based Policymaking 2201
ƚŚƌŽƵŐŚƐƵƌǀĞLJƐĂŶĚŽƚŚĞƌŵĞĂŶƐ͕ĚĞƉĞŶĚŝŶŐŽŶƚŚĞƉŽƉƵůĂƚŝŽŶďĞŝŶŐƐƚƵĚŝĞĚ͘ƌŽďƵƐƚ

ĨƌĂŵĞǁŽƌŬǁŝůůƐƵƉƉŽƌƚĚĂƚĂŵĂŶĂŐĞŵĞŶƚƚĞĐŚŶŽůŽŐLJĂƐŝƚĞǀŽůǀĞƐ͕ĂŶĚŝƚǁŝůůŵĞĞƚƚŚĞŶĞĞĚƐŽĨ

ƚŚĞĐŽŵŵƵŶŝƚLJŽǀĞƌƚŝŵĞŝŶĐůƵĚŝŶŐŝŶŶŽǀĂƚŝǀĞǁĂLJƐƚŚĂƚĨĂĐŝůŝƚĂƚĞƐĂĐĐĞƐƐ͕ĂŶĂůLJƐŝƐ͕ĂŶĚƐŚĂƌŝŶŐ

ĂĐƌŽƐƐǀĂƌŝŽƵƐĐŽŶƐƚŝƚƵĞŶĐŝĞƐ͘

tŚĂƚďĂƌƌŝĞƌƐĐƵƌƌĞŶƚůLJĞdžŝƐƚĨŽƌƵƐŝŶŐƐƵƌǀĞLJĂŶĚĂĚŵŝŶŝƐƚƌĂƚŝǀĞĚĂƚĂƚŽƐƵƉƉŽƌƚƉƌŽŐƌĂŵ

ŵĂŶĂŐĞŵĞŶƚĂŶĚͬŽƌĞǀĂůƵĂƚŝŽŶĂĐƚŝǀŝƚŝĞƐ͍

dŚĞĐƵƌƌĞŶƚĚĂƚĂĐŽůůĞĐƚĞĚďLJŵĂŶLJƉƌŽŐƌĂŵƐĂŶĚƐƵƌǀĞLJƐŝƐŚŝŐŚůLJǀĂůƵĂďůĞ͘,ŽǁĞǀĞƌ͕ǁŚŝůĞĂŶLJ

ŐŝǀĞŶĚĂƚĂĐŽůůĞĐƚŝŽŶĞĨĨŽƌƚƐĞƌǀĞƐĂƐƉĞĐŝĨŝĐƉƵƌƉŽƐĞ͕ĂůĂĐŬŽĨƐƚĂŶĚĂƌĚĚĞĨŝŶŝƚŝŽŶƐŵĂŬĞƐŝƚ

ĚŝĨĨŝĐƵůƚƚŽŝŶƚĞŐƌĂƚĞƚŚŝƐĚĂƚĂĂĐƌŽƐƐĚŽŵĂŝŶƐ͘ĂƚĂƌĞƵƐĞŝƐŝŶĐƌĞĂƐŝŶŐůLJƌĞĐŽŐŶŝnjĞĚĂƐĂĐƌŝƚŝĐĂů

ĐŽŶƚƌŝďƵƚŽƌƚŽƌĞƐĞĂƌĐŚ͕ďĞĐĂƵƐĞƚŚĞĐŽůůĞĐƚŝŽŶŽĨĚĂƚĂĐĂŶďĞĐŽƐƚůLJĂŶĚƚŝŵĞĐŽŶƐƵŵŝŶŐ͘ĂƚĂ

ƐŚŽƵůĚďĞƚŚŽƵŐŚƚŽĨĂƐĂĐƌƵĐŝĂůƌĞƐŽƵƌĐĞƚŚĂƚŚĂƐǀĂůƵĞĂŶĚ͕ŝĨĐŽŵďŝŶĞĚǁŝƚŚŽƚŚĞƌĚĂƚĂ͕ĐĂŶ

ƉŽƚĞŶƚŝĂůůLJŝŶĐƌĞĂƐĞŝŶǀĂůƵĞĂƐŵŽƌĞŝŶĨŽƌŵĂƚŝŽŶŝƐŐůĞĂŶĞĚĨƌŽŵĂŶĂůLJƐŝƐ͘/ĨƚŚĞŽŵŵŝƐƐŝŽŶ

ĐĂŶĞƐƚĂďůŝƐŚĂŶĚĨŽƐƚĞƌĨƌĂŵĞǁŽƌŬƐƚŚĂƚƐƚƌŝǀĞƚŽĚĞǀĞůŽƉĚĂƚĂƐƚĂŶĚĂƌĚƐĂŶĚǁŽƌŬǁŝƚŚŬĞLJ

ĚĂƚĂĂŶĚŝŶĨŽƌŵĂƚŝŽŶƐƚĂŬĞŚŽůĚĞƌƐ͕ƚŚĞƐĞƐƚĂŶĚĂƌĚƐǁŝůůƉƌŽŵŽƚĞƌĞƵƐĞĂŶĚŝŶĐƌĞĂƐĞƚŚĞ

ĂĐĐĞƐƐŝďŝůŝƚLJŽĨĚĂƚĂƚŽƌĞƐĞĂƌĐŚĞƌƐ͘ĞǀĞůŽƉŝŶŐĚĂƚĂƐƚĂŶĚĂƌĚƐŝƐďLJŶŽŵĞĂŶƐĂƐŵĂůůƚĂƐŬ͕ďƵƚŝƚ

ŝƐĐƌŝƚŝĐĂůƚŽŵĂŬĞƉŽƐƐŝďůĞĞĂƐLJĂĐĐĞƐƐ͕ĚŝƐƐĞŵŝŶĂƚŝŽŶ͕ĂŶĚĂŶĂůLJƐŝƐĂĐƌŽƐƐĚŽŵĂŝŶƐ͘

,ŽǁĐĂŶĚĂƚĂ͕ƐƚĂƚŝƐƚŝĐƐ͕ƌĞƐƵůƚƐŽĨƌĞƐĞĂƌĐŚĂŶĚĨŝŶĚŝŶŐƐĨƌŽŵĞǀĂůƵĂƚŝŽŶ͕ďĞďĞƐƚƵƐĞĚƚŽŝŵƉƌŽǀĞ

ƉŽůŝĐŝĞƐĂŶĚƉƌŽŐƌĂŵƐ͍

dĂŬŝŶŐŝŶƚŽĂĐĐŽƵŶƚůŝŵŝƚĞĚƌĞƐŽƵƌĐĞƐ͕ǁĞŵƵƐƚůŽŽŬĂƚǁĂLJƐƚŽŵĂdžŝŵŝnjĞƌĞƚƵƌŶŽŶŝŶǀĞƐƚŵĞŶƚ͘

tŚĞƚŚĞƌƚŚĞƌĞƐŽƵƌĐĞďĞƚŝŵĞ͕ůĂďŽƌ͕ŽƌŵŽŶĞLJ͕ƉŽůŝĐŝĞƐĂŶĚƉƌŽŐƌĂŵƐƐŚŽƵůĚďĞĨŽĐƵƐĞĚŽŶ

ƌĞƚƵƌŶŝŶŐǀĂůƵĞƚŽĐŽŶƐƚŝƚƵĞŶƚƐĂŶĚŝŵƉƌŽǀŝŶŐƚŚĞůŝǀĞƐŽĨƚŚĞŵĞƌŝĐĂŶƉĞŽƉůĞ͘dŽĨĂĐŝůŝƚĂƚĞ

ďĞƐƚƉƌĂĐƚŝĐĞƐĂŶĚŝŵƉƌŽǀĞĚŽƵƚĐŽŵĞƐ͕ĚĂƚĂĐĂŶďĞĐŽůůĞĐƚĞĚĂŶĚĂŶĂůLJnjĞĚ͕ĂŶĚŵŽĚĞůƐĐĂŶďĞ

ĚĞǀĞůŽƉĞĚƚŽƉƌŽǀŝĚĞƉƌŽďĂďŝůŝƐƚŝĐŽƵƚĐŽŵĞƐĨŽƌŐŝǀĞŶƉŽůŝĐLJͬƉƌŽŐƌĂŵŝŵƉůĞŵĞŶƚĂƚŝŽŶƐ͘dŚĞƐĞ

ŵŽĚĞůƐĐĂŶĂĐƚĂƐĂŐƵŝĚĞĂŶĚĐĂŶĨĂĐŝůŝƚĂƚĞĚŝƐĐƵƐƐŝŽŶƚŽǁĂƌĚƐŽůƵƚŝŽŶƐ͘tŚŝůĞƚŚĞƐĞŵŽĚĞůƐ

ŚĂǀĞůŝŵŝƚĂƚŝŽŶƐ͕ŝŶĐůƵĚŝŶŐƵŶŬŶŽǁŶǀĂƌŝĂďůĞƐĂŶĚŽƚŚĞƌĞdžƚĞƌŶĂůŝƚŝĞƐ͕ƚŚĞLJĐĂŶŽĨĨĞƌƌĞƐĞĂƌĐŚĞƌƐ

ƵƐĞĨƵůƐƚĂƌƚŝŶŐƉŽŝŶƚƐƚŽĚƌŝǀĞƚŚĞŝƌĂŶĂůLJƐŝƐĂŶĚƌĞƐĞĂƌĐŚ͘DŽĚĞůƐĂƌĞĂůƐŽƐƵďũĞĐƚƚŽ

ĐŽŶĨŝƌŵĂƚŝŽŶďŝĂƐ͕ǁŚŝĐŚŝƐƚŚĞƚĞŶĚĞŶĐLJƚŽŝŶƚĞƌƉƌĞƚƌĞƐĞĂƌĐŚŝŶĂǁĂLJƚŚĂƚĐŽŶĨŽƌŵƐƚŽŽŶĞ͛Ɛ

ƉƌĞĞdžŝƐƚŝŶŐďĞůŝĞĨƐŽƌŚLJƉŽƚŚĞƐĞƐ͘dŚĞŽŵŵŝƐƐŝŽŶŵƵƐƚĂĐŬŶŽǁůĞĚŐĞƚŚŝƐďŝĂƐĂŶĚŝŶƚƌŽĚƵĐĞ

ĨƌĂŵĞǁŽƌŬƐƚŽŵĂŶĂŐĞŝƚĂƉƉƌŽƉƌŝĂƚĞůLJ͘tŽƌŬŝŶŐƚŚƌŽƵŐŚĂĐĂĚĞŵŝĐĐŚĂŶŶĞůƐĂƐǁĞůůĂƐŝŶǀŽůǀŝŶŐ

ƐƚĂŬĞŚŽůĚĞƌƐŽŶŵĂŶLJƐŝĚĞƐŽĨĂƉĂƌƚŝĐƵůĂƌŝƐƐƵĞ͕ƚŚĞŽŵŵŝƐƐŝŽŶƐŚŽƵůĚǁŽƌŬƚŽǁĂƌĚ

ƐƚĂŶĚĂƌĚŝnjŝŶŐĚĂƚĂĂŶĚƵŶďŝĂƐĞĚŵŽĚĞůĚĞǀĞůŽƉŵĞŶƚ͘KǀĞƌƚŝŵĞ͕ǁŝƚŚƐƵŝƚĂďůĞĨƌĂŵĞǁŽƌŬƐŝŶ

ƉůĂĐĞ͕ĚĂƚĂǁŝůůďĞĐŽŵĞďĞƚƚĞƌ͕ŵŽĚĞůƐǁŝůůĂĚĂƉƚƚŽĐŚĂŶŐŝŶŐŶŽƌŵƐ͕ĂŶĚŶĞǁŵĞƚƌŝĐƐǁŝůůďĞ

ĞƐƚĂďůŝƐŚĞĚŝŶƐƉĞĐŝĨŝĐĂƌĞĂƐƚŚĂƚǁŝůůĨƵƌƚŚĞƌŝŵƉƌŽǀĞŽƵƚĐŽŵĞƐĂŶĚŵŽĚĞůĂĐĐƵƌĂĐLJ͘

^ŚĂŶĞdƌĂŚĂŶ

Zd//ŶƚĞƌŶĂƚŝŽŶĂů

ϯϬκϬŽƌŶǁĂůůŝƐZĚ

ƵƌŚĂŵEϮϳϳϬε

ƐƌƚΛƌƚŝ͘ŽƌŐ

2202 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl8-j7fc

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0318

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

I OPPOSE SB2098. There should not be any additional data tracking of our children.

No. No. No. 

Mom of 3 in central Arkansas.

The Promise of Evidence-Based Policymaking 2203
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl8-74w7

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0319

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

The Federal Government has no need and no authority to obtain information on my

children's education for any reason. Nor do they have any reason or any authority to

collect sensitive psychological/social-emotional information. You cannot keep it safe

and you have no right to it.

2204 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl9-gii8

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0320

Comment on FR Doc # 2016-22002

Submitter Information

Name: Tom Thelen

General Comment

I oppose data mining my personal information. You should have consent first.

The Promise of Evidence-Based Policymaking 2205
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl9-hy3l

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0321

Comment on FR Doc # 2016-22002

Submitter Information

Name: Kim Dancy

Address:

Washington, DC, 20001

Email: dancy@newamerica.org

General Comment

See attached file(s).

Attachments

New America CEP Comments

2206 The Promise of Evidence-Based Policymaking
December 14, 2016 

	 $" #
#

%$  ! 

"%#

 
%)'(+%'''*

Dear Dr. Abraham and Dr. Haskins, 

Thank you for the opportunity to comment on the Commission on Evidence-based Policymaking efforts to 

improve data availability and infrastructure in order to answer critical policy questions. As experts in 

federal postsecondary policy, New America’s Education Policy Program is committed to using data and 

evidence to drive our policy recommendations whenever possible, while ensuring student privacy and data 

security remain firmly in place. In our work, we have encountered many data limitations, many of which 

are driven by a 2008 ban on combining federally held administrative data on students and their outcomes 

for policy evaluation and research purposes. 

Q3. Based on identified best practices and existing examples, how should existing government data 

infrastructure be modified to best facilitate use of and access to administrative and survey data? 

In the years since a federal student unit-record system was first proposed by the Bush administration in 

2005, and banned by the Higher Education Act in 2008, the need to use data to drive postsecondary 

education forward has only become more critical. Since then, college has become increasingly more 

expensive,1

 and colleges now serve a larger and more diverse student body than ever before, requiring 

schools to accommodate students with a much wider array of abilities, experiences, and needs.2

 In this 

environment, better information could help institutions better serve their own students, help policymakers 

support institutions and students alike, and allow students to easily identify the programs and schools that 

offer them the best chance of success.3

 

 

1

“Average Net Price over Time for Full-Time Students, by Sector,” College Board Trends in Higher Education: 

2

 Rachel Fishman and Kim Dancy, “More Than Tuition: Today's Students and Institutional Incentives for Setting 

Yearly Budgets,” EdCentral, May 5, 2016: https://www.newamerica.org/education-policy/edcentral/more-than-tuition3/

3

 Kim Dancy, “Transparency, Accountability, and Trump: Gainful Employment Data are Worth Keeping Around,” 

EdCentral, November 18, 2016: https://www.newamerica.org/education-policy/edcentral/transparency-accountabilityand-trump/

The Promise of Evidence-Based Policymaking 2207


$!"
&	
"
&%#$	&%# #&#	
	

Even in light of soaring costs, better-than-ever labor market returns to a college degree4 have driven 

expansion in postsecondary enrollment. The answer to the question “Is College worth it?” is an 

unequivocal “yes” - on average. But the real question is: in which programs, at which colleges, at which 

price, and for which students is it worth it? Students, families, and taxpayers are spending unprecedented 

amounts on higher education, but remain largely in the dark about how to spend these precious dollars. 

Students and families may know a lot about an institution’s campus culture or online flexibility, but they 

know little about whether students from particular institutions graduate and get good jobs that allow them 

to pay down their debts. 

Why can’t we answer these questions? The federal government either doesn’t have—or can’t use—the 

right data. That’s true, not because it is technically impossible, but because it is illegal. In 2008, Congress 

passed a law that banned the creation of a federal student unit record system to enable existing data 

systems to speak with one another and answer critical questions. Much of the data needed to answer key 

questions are already collected, but can’t legally be connected. Without the ban, the Department of 

Education could gather student-level data already collected and stored by schools, states, and the federal 

government, safeguard it, and link it across schools and to other data sources. The data points of such a 

structure, called a student unit record system, could be a powerful tool to better understand the 

trajectories, struggles, and successes of an increasingly diverse student body. If only they were 

connected. 

There are a number of state and federal datasets that can and should be leveraged, but which cannot 

currently be linked. These include: (1) the Department of Health and Human Services National Directory of 

New Hires (NDNH) and the U.S. Census Bureau’s Longitudinal Employer Household Dynamics (LEHD) 

program, which both include Unemployment Insurance (UI) wage records submitted by states to answer 

workforce and outcomes related questions; (2) SSA and the Internal Revenue Service (IRS), which have 

individual tax records that, in a handful of instances, agencies have found ways to use to report 

employment and earnings outcomes at the program level to better understand student outcomes in the 

workforce; (3) ED, which houses the National Student Loan Data System (NSLDS) and Integrated 

Postsecondary Education Data System (IPEDS), two databases whose information on financial aid and 

student access and success can be disaggregated to see progress for student subpopulations; and (4) the 

DoD and VA, which house data on students using any iteration of the G.I. Bill to fund their education. 

Linking these data sets requires overturning the existing ban on a student-unit record system. 

Q2. Based on identified best practices and existing examples, what factors should be considered in 

reasonable ensuring the security and privacy of administrative and survey data? 

Privacy understandably tops many Americans’ lists of concerns about a student unit record system. 

Recent revelations about widespread surveillance by the National Security Agency and massive data 

breaches at private retailers like Target have brought privacy and data security issues to the forefront of 

public conversations—as well they should. Rapid advances in technology provide tremendous 

opportunities to collect and utilize data for the public good, but have also brought about the potential to do 

 

4

 Anthony P. Carnevale, Stephen J. Rose, and Ban Cheah, The College Payoff: Education, Occupations, Lifetime 

Earnings, (Washington, DC: Georgetown University Center on Education and the Workforce: 2011), 

https://www2.ed.gov/policy/highered/reg/hearulemaking/2011/collegepayoff.pdf .

2208 The Promise of Evidence-Based Policymaking
 

$!"
&	
"
&%#$	&%# #&#	
	

tremendous harm. Any conversation about a federal student unit record system must engage intentionally, 

honestly, and carefully with issues of security and privacy. 

Since the federal government spends hundreds of billions of taxpayer dollars annually to help students go 

to college, it already collects and is responsible for protecting significant amounts of student data across 

various agencies. No matter what protections are put in place, the fact is that any data collection—

including what is already done by colleges and universities for admissions and by the Department for 

administrative purposes such as disbursing and collecting federal student loans— reduces privacy to 

some extent. The overarching question then becomes whether a reduction in privacy is worth the broader 

individual and societal rewards, and how to minimize those risks where they do occur. Students already 

exchange some anonymity for valuable services or information. Every year, millions of students submit 

highly personal information, including their families’ income information, through the Free Application for 

Federal Student Aid (FAFSA). Additional value could be derived if these existing data sources were 

connected to enable policymakers, taxpayers, institutions and students to better assess questions like: 

● How many low-income, Pell Grant recipient, first-generation, veteran, adult, transfer, and part-time 

students, attend each college? At which schools do these students have the best odds of making it 

to graduation? 

● How long does it take students, particularly students who enter with less academic preparation or 

fewer financial resources, to complete college? 

● Do the students who don’t graduate transfer, or do they drop out? 

● How much and how often do students borrow, and how long does it take the typical graduate to 

repay these loans? 

 

While there may be compelling reasons to accept some of the trade-offs that come with a federal student 

unit record system—not least the hundreds of billions of dollars students and taxpayers spend each year 

on higher education—thoughtful and strong privacy and security policies must be put in place. Student 

privacy can be addressed using a risk-based framework, where the risks of disclosures or violations are 

evaluated and weighed against the benefits of using data to address key policy questions. Every attempt 

to mitigate potential risk must be taken, and individuals are held accountable for any violations that occur. 

Unfortunately, student privacy has often been used as a cover for institutional privacy, with higher 

education institutions blocking data collection and connection to prevent closer scrutiny of their programs 

and student outcomes--to the detriment of students themselves.5

 Indeed, in the higher education sphere, 

privacy interests have frequently been leveraged to block data collection and shield universities from 

efforts at transparency. Moving forward, we must absolutely address student privacy concerns, but should 

not allow institutional privacy to get in the way of transparency and accountability. 

As it stands, institutions hold extensive student-level records. But even in the states and schools most 

committed to transparency, they are still forced to work with imperfect fixes that could be easily addressed 

at the federal level. For example, the University of Texas system has spent years working to connect 

institutionally held data on student enrollment and persistence with earnings data held by federal 

 

5

 Amy Laitinen and Clare McCann, The College Blackout (Washington, DC: New America, March, 2014), 

http://www.collegeblackout.org/

The Promise of Evidence-Based Policymaking 2209
!

$!"
&	
"
&%#$	&%# #&#	
	

agencies. This year, they succeeded in working out a contract with the U.S. Census Bureau enabling 

these data connections.6

 This makes the UT system one of the only schools in the country that will be able 

to understand earnings data at the program level for all of its students at all of its campuses, in addition to 

identifying migration patterns, industries of employment, and more. But the UT schools won’t be able to 

compare themselves to others: Just down the road, at Baylor University in Waco, Texas, a private 

university, student earnings data will remain unavailable or incomplete. This limits both UT’s ability to 

benchmark itself against its peer institutions and a student’s confidence in deciding whether and where to 

enroll. In other states, the problems are far more severe: schools may be able to connect to earnings and 

employment data for students who remain in-state, but the data trail ends if those former students work or 

reside across state lines.7

Lessons Learned from State Longitudinal Data Systems 

While the federal government is prohibited from linking or sharing any of its student data, the absence of a 

federal student unit record system of the type described here does not mean that student records are not 

currently collected or shared outside of individual colleges and universities. For one thing, in response to 

the 2008 ban on the creation of a federal student unit record system, the Department created a 

competitive grant program to assist states in setting up and administering their own student record 

systems, known as State Longitudinal Data System (SLDS).8

 As a result, today nearly every state has 

such a system, and these systems vary widely with respect to which data are collected and how they are 

stored, data governance structures, and privacy and security regulations and protocols. This variation in 

privacy structures lends itself to an exploration of best practices, and also illustrates lessons to be learned 

for the use of data held at the federal level. 

ED's Office of the Inspector General (OIG) has conducted audits of two State Longitudinal Data Systems 

(Oregon’s9 and Virginia's10) - a third, Indiana's, is still in-progress. The states were "judgmentally selected" 

on the basis of their SLDS grant funding, the status of their funded projects, and the number of reported 

data system breaches.11 Oregon did not consider its own system to constitute an SLDS, but OIG 

disagreed and concluded that the state’s K-12 database did constitute an SLDS. Two Virginia systems 

were audited: a statewide data query interface which administrators use to search for data, and the K-12 

exposure database from which data is retrieved. Neither state’s audited system has had any documented 

breaches.

 

6

“UT System partners with U.S. Census Bureau to provide salary and jobs data of UT graduates across the nation,” 

University of Texas System, September 22, 2016: https://www.utsystem.edu/news/2016/09/22/ut-system-partnersus-census-bureau-provide-salary-and-jobs-data-ut-graduates-across

7

 For example, the Western Interstate Commission for Higher Education’s “Multistate Longitudinal Data Exchange”: 

http://www.wiche.edu/longitudinaldataexchange

8

 See http://nces.ed.gov/programs/slds/

9

 United States Department of Education Office of Inspector General, “Protection of Personally Identifiable 

Information in Oregon’s Statewide Longitudinal Data System,” September 27, 2016: 

http://www2.ed.gov/about/offices/list/oig/auditreports/fy2016/a02p0007.pdf

10 United States Department of Education Office of Inspector General, “Protection of Personally Identifiable 

Information in the Commonwealth of Virginia’s Longitudinal Data System,” July 12, 2016: 

“http://www2.ed.gov/about/offices/list/oig/auditreports/fy2016/a02p0006.pdf

11 From 2007 to present, Oregon received about $19 mil in three packages of IES funding for their SLDS; Virginia received 

about $24 mil in two packages. 

2210 The Promise of Evidence-Based Policymaking
"

$!"
&	
"
&%#$	&%# #&#	
	

Oregon started developing their Consolidated Collection System (CCS) in 2003-4, and today it is 

comprised of 81 data stores that incorporate student academic performance, graduation rates, and 

discipline incidents. The database is accessed from central login according to a "need-to-know" hierarchy 

of differentiated access--that is, a school administrator can view their own district's data, but not that of 

other districts. External research requests are processed individually and the requested data is collected 

by the Oregon Department of Education (ODE) then transmitted to the requester in anonymized form.

The Virginia Longitudinal Data System (VLDS), deployed in 2013, queries five state agencies - including 

the Virginia Department of Education’s Single Sign-on Web System (SSWS), the state’s K-12 exposure 

database, which was audited along with VLDS. VLDS reads hashed data from the queried database, 

which it hashes again (assigning a new identifier) before returning the request. VLDS does not store data, 

but processes individual requests as needed, completely anonymizing any output produced. Research 

queries are submitted through VLDS, and VDOE returns the requested data from SSWS. These results 

remain available for 10 days and are then deleted. Again, permissions are assigned according to a system 

of differentiated access. 

In both cases, the OIG found the SLDS operations did not conform to the state’s own data security 

policies. Lacking documentation of security plans or of security operations such as maintenance or breach 

reports was cited as a failing of both systems. The audit found that ODE failed to implement a security 

plan, conduct annual risk assessments, and classify their database’s contents according to state 

standards. OIG also noted that administrators in Virginia’s SSWS did not update its risk assessment, did 

not remedy vulnerabilities detected by an internal scan, and did not conform to the state's password 

policies. However, the report also noted that the VLDS system, which combines data held by different 

state agencies in Virginia, but does not store or hold any such data, presented no increase in risk of 

disclosure or inappropriate use. Thus, while data security protocols for existing federal data should be 

rigorous and thorough, there are models available for combining these data sources in ways that help 

address critical policy questions without increasing the risk to student privacy. 

The reports also noted several areas of best practice that should be adopted and implemented for any 

federally held student data, including: 

● Having a security plan that includes:

○ Risk assessments/identification,

○ Detection protocols,

○ Monitoring and testing protocols (including automated vulnerability scans every 90 days), 

and

○ Data security classifications and training protocols

● Ensuring sufficient staffing to carry out the above tasks

● Storing readable data in encrypted format 

● Reducing linkages to minimum necessary to reduce transmission vulnerabilities

● Requiring new passwords after determined length of time or number of data generations (e.g., 

20 data pulls), and disabling inactive accounts.

The Promise of Evidence-Based Policymaking 2211
#

$!"
&	
"
&%#$	&%# #&#	
	

Protecting Privacy at the Federal Level Remains Critical 

Many of these policies are already enacted in the work of the Department of Education’s National Center 

for Education Statistics (NCES), classified as a statistical agency and therefore subject to stringent 

requirements under the Privacy Act of 1974, the Education Sciences Reform Act of 2002 (ESRA), and the 

E-Government Act of 2002. Under the ESRA, for example, an individual who discloses personally 

identifiable information is subject to a fine of up to $250,000, five years in prison, or both—a harsh 

consequence that ensures data security is not taken lightly.12 In fact, the Department has withheld 

confidential information from its student-level cross-sectional and longitudinal surveys, while still making 

this information available to qualified researchers for policy analysis and research. At the SSA, whose data 

would be used to match education and earnings records, employees face similar punishments for 

violations of privacy law. 

 

Moreover, both the Department of Education and the Social Security Administration are experienced in 

data matching and de-identification practices. The Department of Education’s Disclosure Review Board 

reviews data releases and restricts those that could contain re-identifiable student information. The 

Department of Education’s National Forum on Education Statistics has also published recommended best 

practices for distributing data to external researchers, which ensure that taxpayers and the public benefit 

from federal data collections without violating the privacy of the students in question. SSA employees are 

also very experienced in the data practices in question, and currently conduct data-matching projects with 

the National Technical Institute for the Deaf and for gainful employment regulations, among others.13

 

Other agencies, such as the Census Bureau, securely maintain extremely personal and sensitive 

information on a large scale. The Department of Education should work with the Census Bureau to 

establish procedures that would limit the potential for hacking, theft, or inadvertent release of private data, 

including limiting data access to only a few people in a few locations, building up information technology 

infrastructure to ensure that the agency is better equipped to detect and deter hackers, and conducting 

regular audits and updates of security standards and practices. 

 

In developing a student unit record system, privacy advocates and experts should sit together with 

consumer information and student advocates, legal experts, data analysts, and higher education policy 

researchers to create security and privacy protocols and protections for such a system. A good place to 

begin could be with an existing federal model for privacy and security, which lists its key elements as: 

transparency; avenues for individuals to correct their data; a specific and narrow purpose of the data; data 

collection and retention that does not exceed the scope of the purpose; use of the data that is strictly 

limited to the program’s purpose; “accurate, relevant, timely, and complete” data; data security; and 

accountability and regular auditing of the program.14

 

12 US Social Security Administration, Program Operations Manual System, GN 03301.045, “Penalties for Violation of 

the Information Laws.” TN 3 (11-98) (19 July 2012): https://secure.ssa.gov/poms.nsf/lnx/0203301045 . 13 Laitinen, Amy and Clare McCann, The College Blackout (Washington, DC: New America, March, 2014), 

http://www.collegeblackout.org/

14 “Federal Enterprise Architecture,” White House Office of Management and Budget, May 2, 2012: 

http://www.whitehouse.gov/omb/e-gov/fea ; and “Best Practices: Elements of a Federal Privacy Program: Version 

1.0,” Federal CIO Council Privacy Committee, June 2010: 

2212 The Promise of Evidence-Based Policymaking
$

$!"
&	
"
&%#$	&%# #&#	
	

To improve postsecondary education for students and their families, help policymakers make informed 

decisions about where to invest resources, and allow institutions to understand their strengths and 

weaknesses relative to peer institutions, better use of existing data at the federal level is necessary. We 

recommend that the Commission on Evidence Based Policymaking encourage Congress to overturn this 

ban, so that data can be used to deliver better education outcomes to students, more efficiently and 

effectively. Doing so allows policymakers to explore the most appropriate and secure ways in which to 

combine data, either on a case-by-case basis using a federated model, such as that used by Virginia and 

other states, or through the development of a stand alone student-unit-record system. This allows for the 

evaluation of the privacy considerations particular to the proposed system, which will matter tremendously 

for its impacts on students, and enables the utilization of lessons learned from other statistical agencies, 

state education data systems, and more. 

Sincerely, 

Amy Laitinen, 

Director, Higher Education Initiative, Education Policy Program 

Kim Dancy, 

Policy Analyst, Higher Education Initiative, Education Policy Program 

 

http://energy.gov/sites/prod/files/Elements%20of%20a%20Federal%20Privacy%20Program%20v1.0_June2010%20

Final.pdf 

 

 

 

The Promise of Evidence-Based Policymaking 2213
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl9-quzb

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0322

Comment on FR Doc # 2016-22002

Submitter Information

Name: Kim Barden

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all students, commonly

referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008, the reasons against creating it

have only become more compelling.

In the past few years, much highly personal data held by federal agencies has been

hacked, including the release of the records of the Office of Personnel Management

involving more than 22 million individuals, not only federal employees and contractors

but also their families and friends.

The US Department of Education has been found to have especially weak security

2214 The Promise of Evidence-Based Policymaking
standards in its collection and storage of student data, and received a grade of D for its

security protections.

In addition, preK-12 student data currently collected by state departments of education

that would potentially be shared with the federal government include upwards of 700

highly sensitive personal data elements, including students' immigrant status, disabilities,

disciplinary records, and homelessness data.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from PreK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban in

the report you provide to Congress.

Thank you,

Kim Barden

Pennsylvania

The Promise of Evidence-Based Policymaking 2215
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl9-bmwn

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0323

Comment on FR Doc # 2016-22002

Submitter Information

Name: Christine B

General Comment

I am opposed to data mining without consent of the subjects. I believe in informed

consent.

2216 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl9-k1sk

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0324

Comment on FR Doc # 2016-22002

Submitter Information

Name: Paul Decker

Address:

Washington, DC, 20002

Email: pdecker@mathematica-mpr.com

General Comment

See attached file(s)

Attachments

Mathematica Letter to EBPM Commission 12-13-2016

The Promise of Evidence-Based Policymaking 2217
An Affirmative Action/Equal Opportunity Employer

1100 1st Street, NE, 12th Floor 

Washington, DC 20002-4221 

Telephone (202) 484-9220 

mathematica-mpr.com 

December 14, 2016 

Dear Members of the Commission on Evidence-Based Policymaking: 

Thank you for giving the research community an opportunity to contribute to the important work of 

the Commission. For nearly 50 years, Mathematica has been dedicated to delivering evidence-based 

research, data, and objective analysis of the highest quality. We believe that data used for evidence-based 

policymaking should be viewed as a treasured resource, one that must be carefully safeguarded. It is clear 

that the Commission takes safeguarding this resource very seriously, as we do at Mathematica. However, 

good stewardship of any national treasure must include employing it for the benefit of the nation. We fall 

woefully short of meeting our stewardship obligations for these data when we fail to realize that the 

difficulties in accessing them or the restrictions on their use keep much of their value locked away. At 

Mathematica, we are honored to support the Commission in addressing this difficult challenge. 

In response to question numbers 1, 6, 9, 10, 16, 17, 18, and 19 listed on the “Commission on EvidenceBased Policymaking Comments,” docket number: USBC-2016-0003, I—along with my colleagues Tamara 

Barnes, Peter Schochet, Irma Perez-Johnson, and Alexandra Resch—have drafted the following comments. 

We expect these insights will help inform the work of the Commission and potentially provide discussion 

topics for future Commission meetings. 

Mathematica Response to the Commission’s Request for Comments

Question # 1. Are there successful frameworks, policies, practices, and methods to overcome challenges 

related to evidence-building from state, local, and/or international governments the Commission 

should consider when developing findings and recommendations regarding Federal evidence-based 

policymaking? If so, please describe. 

Researcher-practitioner partnerships have shown promise as a framework that facilitates continuous 

program improvement and evidence-building at all levels of government. Numerous partnerships focused 

on using behavioral insights to improve programs have developed in recent years. The White House Social 

and Behavioral Sciences Team partnered with several federal agencies to improve programs by simplifying 

communication to program participants and employing behavioral nudges. Mathematica is working with 

three agencies in the U. S. Department of Labor (DOL) on Behavioral Interventions for Labor-Related 

Programs, a project in which we apply behavioral insights to DOL programs. In these three examples, 

program changes were evaluated rigorously, often using randomized controlled trials (RCTs). We provide 

examples of similar partnerships with state and local Temporary Assistance for Needy Families (TANF) 

agencies in our response to question #16. 

 

These partnership approaches combine evaluation and evidence building with a cycle of activities to 

support better programs and policies that are based on evidence. Activities are adapted for each specific 

program. Most effective partnerships recognize that different policy questions require different levels of 

2218 The Promise of Evidence-Based Policymaking
LETTER TO: Commission on Evidence-Based Policymaking 

FROM: Mathematica Policy Research 

DATE: December 14, 2016 

PAGE: 2

evidence and can support different levels of program evaluation, allowing us to right-size the evidencebuilding to the problem at hand. 

To achieve an ideal framework: 

1. It is important to begin with a clear and well-articulated definition of the problem. 

2. Next, efforts to improve policies and programs should be grounded in the available knowledge 

base about why the problem exists, including what has been done to address it and in what 

contexts. If available, this knowledge base provides solid grounding and context for further 

innovation and experimentation. 

3. Then we identify untested, promising strategies and decide which to pursue or test given 

operational, contextual, or other constraints. 

4. Once promising strategies are selected, we can move into prototyping, refining the selected 

strategies with input from relevant users or other stakeholders. We attempt to pilot-test on a 

large enough scale to assess feasibility and uncover important implementation challenges. 

Then we continue refining additional iterations until we achieve a practical model. 

5. We develop a proof of concept test to yield evidence on whether the promising strategy leads 

to result(s). If yes, we may refine further based on implementation experiences or move on to 

more rigorous tests. If not, we may go back to Step 3 to explore other promising approaches. 

6. Once we have a promising idea on a pilot scale, we can shift to testing, using the most rigorous 

method(s) possible. 

7. If the program or policy concept is proven at scale, we can move to broader adoption or 

adaptation in other contexts while maintaining effectiveness. 

Evidence-building and testing activities permeate each step and must be right-sized to fit our goals and 

answer the appropriate questions at each stage of the process. This is the framework that underlies our 

evidence-based technical assistance (EBTA) and other support activities. We find that when program 

administrators participate in these activities and the evaluation efforts are focused on program needs, 

program administrators quickly see the value of evaluation and evidence and are enthusiastic about 

participating in further evidence-building. 

Currently, this approach is rarely implemented sequentially or fully in the way we have described. 

Rather, evidence-building activities happen at any point in the sequence, with or without the benefit of the 

evidence from the preceding steps. The end result is a spotty mix of evidence on the wide range of 

government programs and policies in existence, with evidence for some stages of continuous program 

improvement, but not all. However, there are promising signs (including the work of this Commission) of 

a growing interest in and recognition of the value of an evidence-based approach to policy and program 

development and ongoing improvement. 

Question #6: Should a single or multiple clearinghouse(s) for administrative and survey data be established 

to improve evidence-based policymaking? What benefits or limitations are likely to be encountered in 

either approach? 

The idea of establishing a single clearinghouse for administrative and survey data is very attractive, 

but is likely to take a long time to achieve, and during the wait for that goal to be achieved, its worth may 

not be evident to all. An alternative framework that may be more easily implemented would require each 

agency that houses administrative and/or survey data to establish its own data clearinghouse center that 

The Promise of Evidence-Based Policymaking 2219
LETTER TO: Commission on Evidence-Based Policymaking 

FROM: Mathematica Policy Research 

DATE: December 14, 2016 

PAGE: 3

would coordinate with a centralized body acting as a “virtual” data clearinghouse. This virtual data 

clearinghouse would be the body through which applications for access to data would be submitted and 

approved, and would serve as the coordinator between each agency’s data clearinghouse center, facilitating 

access to and linking of data across agencies. 

Question #9: What specific administrative or legal barriers exist for accessing survey and administrative 

data? 

In its white papers provided to the Commission, entitled Using Administrative and Survey Data for 

Evidence-Building and Barriers to Using Administrative Data for Evidence-Building, the Office of 

Management and Budget did an exemplary job of laying out virtually all of the administrative and legal 

barriers that we face daily at Mathematica in our work with administrative and survey data. 

What is required above all else is fostering a mindset that administrative and survey data are among 

our national treasures and should be put to their best and highest use. At Mathematica, we have observed 

that some federal agencies are slowly shifting their mindset in this direction. Federal legislation that 

embodies this principle and mandates updating laws and agency regulations in light of this principle would 

be invaluable in achieving this goal more broadly among all stakeholders, including federal agencies, state 

and local governments, and federal program grantees. 

Gaining access to the administrative data of state and local governments can be particularly 

challenging, and this is an area where Mathematica confronts the reality of federalism and our republican 

form of government almost daily. In laying a foundation for cooperation in this area, it would be beneficial 

to establish a model state act that incorporates the principle of stewardship of administrative and survey 

data as a state treasure. This model state act could, for example, establish a mandate and processes for 

providing access to administrative data for purposes that benefit both the state and the nation when 

appropriate. Such a model state act would be similar in nature to other model state acts, like the model 

administrative procedures act or model public health act. 

With respect to federal program grantees, particularly those that are not state or local government 

entities, a condition of the grant should be the provision of administrative or survey data by the grantee. 

Standardized language to include in all grant agreements could be developed as a model for sponsoring 

agencies. In cases where individual consent is required for the provision of such data, the sponsoring 

agencies can and should plan to provide standardized consent forms and other tools to help ease the burden 

of obtaining individual consents. 

Finally, with agencies holding the attitude that administrative and survey data are a national treasure, 

one action that can be initiated right away is the use of System of Record Notices (SORNs), which are 

mandated by the Privacy Act. SORNs are the instruments agencies use to inform the public of what the 

agency’s permitted uses and disclosure standards are for data under their control. A properly drafted SORN 

would be particularly helpful with respect to survey data that were collected with individual consent. A 

properly drafted SORN coupled with an individual consent form that properly references the sponsoring 

agency’s SORN, or any amendments to the SORN, would govern the future use, protection, and disclosure 

of the collected survey data. 

2220 The Promise of Evidence-Based Policymaking
LETTER TO: Commission on Evidence-Based Policymaking 

FROM: Mathematica Policy Research 

DATE: December 14, 2016 

PAGE: 4

Question #10: How should the Commission define “qualified researchers and institutions?” To what extent 

should administrative and survey data held by government agencies be made available to “qualified 

researchers and institutions?”

We understand this definition is the basis on which decisions will be made as to who can gain access 

to administrative and survey data in the data clearinghouse(s). Defining who is “in” and who is “out” is 

always an important task. Mathematica urges the Commission to view this task as possessing the inherent 

possibility of shaping the future of evidence-based research by defining who can and cannot conduct such 

activities with data from the clearinghouse(s). We do not have a crystal ball in which to see the innovations 

or changes in industry and occupations that will produce “qualified researchers and institutions” decades 

from now. What we can see is that if you limit this definition to classifications of individuals or institutions 

as they exist today you may, ab initio, limit the future of the development of public policy research, and we 

believe such a limit to be a bad thing. We strongly recommend that the Commission use objective standards 

that are not based on the classification of individuals or entities. Some such objective standards could be, 

for example, that the proposed use of the data is permitted by law, that relevant ethical and conflict of 

interest disclosure standards are met, and that the individual or entity can meet data confidentiality and 

security standards. 

Question #16: How can data, stats, results of research, and evaluation findings best be used to improve 

programs and policies? 

Research can be translated into actionable policy changes and program improvements at all levels of 

government. The examples below highlight just a few of the ways that Mathematica’s research results and 

evaluation findings are being used to improve programs and policies at the federal and state level. 

x Our evaluation of Adolescent Pregnancy Prevention Approaches (PPA) is a major federal effort 

to expand available evidence on effective ways to prevent pregnancy and reduce related sexual 

risk behaviors among teens in the United States. The eight-year (2008–2016) evaluation 

documented and tested new and innovative approaches to teen pregnancy prevention in seven 

sites across the United States. Six of the seven sites received federal funding from the Office of 

Adolescent Health or Administration on Children, Youth and Families in the U.S. Department of 

Health and Human Services as part of the federal government’s broader effort to invest in 

evidence-based approaches to teen pregnancy prevention. 

x For DOL, the Behavioral Interventions for Labor-Related Programs project involves working 

with three DOL agencies to examine how behavioral science principles can improve the 

performance and outcomes of selected programs. For example, Mathematica worked with the 

Occupational Safety and Health Administration (OSHA) to test whether changes to its citation 

process could increase employer responsiveness and reduce referrals to the OSHA National 

Office. Behavioral solutions were found to improve program outcomes and performance, and 

they can inform the broad adoption of such principles within DOL and many other related 

programs nationwide. Insights gleaned from this work are contributing to the growing body of 

evidence on the effectiveness of behavioral strategies. 

The Promise of Evidence-Based Policymaking 2221
LETTER TO: Commission on Evidence-Based Policymaking 

FROM: Mathematica Policy Research 

DATE: December 14, 2016 

PAGE: 5

x In work for the Millennium Challenge Corporation, Mathematica’s evaluation of “girl-friendly” 

schools helped identify program changes that could positively impact girls’ education in Niger. 

Valuable data from Niger provided compelling evidence for governments and donors in thinking 

about how to develop effective programs to provide access to quality education for children across 

the developing world, especially girls. Mathematica researchers conducted the evaluation in 178 

villages across Niger. They assessed efforts to improve educational outcomes for children in 

targeted communities and took a concerted look at environmental factors related to increasing 

girls’ access to schooling.

x As part of an “employment-focused programming” initiative, sponsored by the Colorado 

Department of Human Services (CDHS), in 2015–2016, Mathematica worked intensively with 

three local welfare agencies to assess their readiness to implement employment-focused 

programming and identify opportunities for innovation and improvement. We identified one of 

these sites, Larimer County, as the top candidate and formalized a researcher/practitioner 

partnership with its TANF program administrators. Over the past two years, we have designed and 

tested changes to communications and other nudges to help TANF recipients better understand 

program rules, complete required activities, avoid sanctions, and achieve self-sufficiency. As a 

result of this partnership, Larimer County staff have learned that evaluation can be timely and can 

directly inform day-to-day policy decisions. 

x Another researcher-practitioner partnership with Ramsey County, Minnesota’s TANF agency, 

identified a new service delivery approach—the Lifelong Learning Initiative (LLI). In 2015, 

Mathematica was contracted to perform three years of technical assistance on the design, refining, 

and ultimately the full-scale implementation of this concept. During Phase 1 of the project, we 

facilitated a collaborative learning and design process in which staff at all levels of the organization 

(administrators, supervisors, frontline staff, and even clients) contributed to an in-depth program 

assessment and provided input on the intervention’s early design. In Phase 2, we created 

intervention materials based on Phase 1 recommendations and launched three six-week learning 

cycles in which intervention training was provided to a cohort of staff. After the three learning 

cycles, we revamped and polished all of the intervention materials and delivered training for all 

agency staff in August 2016. Now our focus is on enhancing and strengthening the LLI intervention 

by incorporating rapid cycle testing and other formative evaluation techniques into implementation. 

Question #17: To what extent can and should program and policy evaluation be addressed in program 

designs? 

The continuous program improvement framework presented in response to question #1 indicates that 

evidence can and should be integrated into every stage of program and policy design and development. 

Research and evaluation activities should be aligned with the goals at any given stage and can be rightsized to meet important constraints or other considerations. Our experience working with local and state 

agencies suggests that many agencies do not understand their role in program evaluation. In fact, many see 

research as something that is done to them and often does not closely relate to their day-to-day program 

concerns. Agencies developing or implementing new programs are eager to learn whether they are effective, 

but they struggle to find the resources, including staff time and expertise, to conduct evaluations themselves. 

2222 The Promise of Evidence-Based Policymaking
LETTER TO: Commission on Evidence-Based Policymaking 

FROM: Mathematica Policy Research 

DATE: December 14, 2016 

PAGE: 6

To support this work, it would be ideal to set aside a fraction of program funding to support a full 

range of evidence-based, continuous quality improvement and evaluation, not just one-time program 

evaluations. Agencies beginning this work also benefit from technical assistance that helps them use 

evidence and evaluation to answer their own day-to-day policy questions, demonstrates that evaluation can 

be useful and timely, and develops their capacity to interpret and generate evidence. 

Question #18: How can or should program evaluation be incorporated into program designs? What 

specific examples demonstrate where evaluation has been successfully incorporated in program 

designs? 

It is important that policymakers use evidence-based research to improve their interventions, programs, 

and policies. Evidence-based research must be rigorous, with proper research questions and study designs 

to answer the questions a study was meant to answer. Too often, policy is made using subjective judgement 

or poorly designed research. 

To make research most effective, it is critical for it to be ongoing and allow continuous program 

improvement, instead of being conducted at interim points only. This means that staff at government 

agencies must be included in the scientific process so they are invested in the research from the start. It is 

also critical to recognize that research that will be useful to program managers will include a wide range of 

research methods, not only RCTs. Program managers seeking to refine a current program may use 

descriptive or non-experimental analyses that help generate hypotheses about what is working and what is 

not. They may also use machine-learning methods and big data to predict which services might be most 

effective for each program participant based on his or her characteristics and needs. These managers could 

then propose changes to their program and evaluate these changes using an RCT or a quasi-experimental 

design (QED) before deciding to roll out the change widely. 

For this to be successful, the nation needs to expand the research community beyond the relatively 

small number of academics and research organizations that are currently conducting social policy research 

to investigate the effectiveness of policies and programs. Government agency staff are natural participants 

to include in research efforts, because (1) they are best informed about the proper interventions to test and 

how they should be implemented, (2) they can build “rapid-cycle” evaluations into their daily service 

activities to test incremental changes in their programs, and (3) they may have access to administrative data 

that can provide outcomes for the evaluations. Perhaps most important, these staff could weigh in from the 

start about how the evaluation results can be used in practice and be involved in plans for improving the 

tested interventions. 

The examples of researcher-practitioner collaborations provided in response to question #16 

demonstrate that agency staff can participate in this work and develop internal capacity for further 

evaluation and evidence-building. Additional support to agency staff comes in the form of evaluation 

technical assistance and toolkits to support evaluation. Mathematica has supported these efforts through 

several contracts, including one with the Administration for Children and Families in the U.S. Department 

of Health and Human Services and another with the U.S. Department of Education (ED). Through these 

contracts we provide technical assistance to grantees and local areas conducting impact evaluations, and 

we produce user-friendly materials (such as how-to briefs, software, and webinars) to support their 

evaluations. Our recent experience developing an online toolkit for rapid cycle evaluations of educational 

The Promise of Evidence-Based Policymaking 2223
LETTER TO: Commission on Evidence-Based Policymaking 

FROM: Mathematica Policy Research 

DATE: December 14, 2016 

PAGE: 7

technologies for the Office of Educational Technologies at ED suggests that there is high demand for these 

products. The beta version of our toolkit was released in late October, and we have received inquiries from 

over 70 school districts that are interested in using the tools and receiving technical assistance. These efforts 

should be expanded. 

The development of evidence at the local level may complement large federal evaluations, but the 

evaluation of federal initiatives requires evaluations with a broader focus. However, to be most useful to 

program administrators, large federal evaluations should include careful evaluation of specific program 

components and variation in program effects. Much federal research has been designed to answer “big” 

questions such as “What are the average effects of a federal grant program on participants’ outcomes?” 

Although a rigorous RCT for such an evaluation can help us assess whether individuals receiving grant 

services have better outcomes, on average, than a control group does, it is not always clear how such results 

can foster continuous program improvement. This is because a complex grant program can have multiple 

program components and mediating pathways associated with the overall program impacts, and a typical 

“thumbs-up/thumbs-down” evaluation of a grant program examines the effects of each mediating factor in 

isolation using non-experimental methods that are prone to sample selection biases. Thus, it may be useful 

for those who develop research agendas to consider smaller research questions about specific mediators 

that are varied as part of the research design (for example, different dosages of case management services). 

These types of results may be more useful to program managers, but would require a research agenda 

that could build over time to help us understand which interventions are most effective and for whom, and 

whether they can be replicated in different settings. 

Question #19: To what extent should evaluations specifically with either experimental (sometimes referred 

to as “randomized control trials”) or quasi-experimental designs be institutionalized in programs? 

What specific examples demonstrate where such institutionalization has been successful and what best 

practices exist for doing so? 

RCTs and QEDs answer questions of impact, which are at the core of assessments of the effectiveness 

and efficiency of public investments. As such, they should be institutionalized into every public program 

and policy where appropriate. However, it is most critical to develop evidence of the impact of previously 

unproven programs. With limited public resources, it seems prudent to focus the effort and cost of rigorous 

evaluations on the programs that are not yet proven. Tiered evidence grants are an example of 

institutionalizing evaluation requirements while recognizing that different stages of program development 

require different types of evidence. These grants support innovation by providing some funding to 

promising programs that are not yet fully developed, but appropriately require rigorous evaluation using 

RCTs or QEDs when the program is ready to be scaled up to widespread use. It is important to recognize 

that other methods may be most appropriate for early program stages and to support program development 

and refinement before investing in more rigorous, and typically more costly, evaluations. 

Focusing on the rigorous evaluation of unproven programs, some laws authorizing funding of public 

programs require RCT or QED evaluations of the programs (for example, the Workforce Innovation & 

Opportunity Act). In addition, some RFPs for federal grants to implement programs require that grantees 

participate in or implement their own evaluations. For example, grant programs including Race to the Top 

and School Improvement Grants required state cooperation with evaluations and facilitated the research on 

these programs. These regulations have been an important impetus for rigorous program evaluations that 

are much less common in Europe and elsewhere outside the United States. Nonetheless, there could be 

2224 The Promise of Evidence-Based Policymaking
LETTER TO: Commission on Evidence-Based Policymaking 

FROM: Mathematica Policy Research 

DATE: December 14, 2016 

PAGE: 8

better enforcement by federal agencies or strengthening of these regulations by making participation in or 

cooperation with evaluation activities a condition for receipt of federal grant funds. 

Mandated participation could improve the efficiency of federal program evaluation efforts by reducing 

recruiting costs. Recruiting sites for program evaluations when participation is voluntary is often difficult 

and expensive, because we must overcome sites’ resistance to participate in RCT or QED evaluations and 

also gain cooperation against competing site priorities. Site recruitment can be especially challenging when 

grants are disbursed at the state level but programs are run locally (for example, education programs). 

Stronger enforcement and mandates for site participation in evaluation at all the necessary levels can help 

spur research, reduce evaluation costs, and improve the generalizability of study results by facilitating the 

recruitment of a more representative set of sites than are typically included in program evaluations. 

Thank you again for this opportunity to answer these important questions. Should you have any 

questions about the contents of this submission, please contact me at PDecker@mathematica-mpr.com, or 

contact my colleague Carmen Ferro, public affairs manager at Mathematica Policy Research, at (202) 552-

6405, or at cferro@mathematica-mpr.com. 

Sincerely, 

Paul Decker 

President and Chief Executive Officer 

Mathematica Policy Research 

cc: 

Peter Schochet, Senior Fellow, Mathematica Policy Research 

Tamara Barnes, General Counsel, Mathematica Policy Research 

Irma Perez-Johnson, Senior Researcher, Mathematica Policy Research 

Alexandra Resch, Senior Researcher, Mathematica Policy Research 

Jennifer de Vallance, Director of Public Affairs, Mathematica Policy Research 

Shelly Martinez, Executive Director, Commission on Evidence-Based Policymaking 

Nick Hart, Policy and Research Director, Commission on Evidence-Based Policymaking 

The Promise of Evidence-Based Policymaking 2225
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl9-lcu8

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0325

Comment on FR Doc # 2016-22002

Submitter Information

Name: April Clyburne-Sherin

General Comment

In response to Question 13: What technological solutions from government or the private

sector are relevant for facilitating data sharing and management?

The creation of clinicaltrials.gov was an important milestone for improving access to

important clinical trial information that can be used to improve treatment and policy.

Investing in the usability and interoperability of clinicaltrials.gov will promote more

complete data and higher quality data to be shared. 

2226 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl9-gr0h

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0326

Comment on FR Doc # 2016-22002

Submitter Information

Name: Christianna BUrt

General Comment

I am a college sophomore. I am opposed to non-consensual data collection and the

housing of personally identifiable information at the federal level. I watched the public

hearing this week on YouTube. Please respect individual data rights. We are watching

what you do

The Promise of Evidence-Based Policymaking 2227
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tl9-8izj

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0327

Comment on FR Doc # 2016-22002

Submitter Information

Name: Leonie Haimson

Address:

New York, NY, 10011

Email: leonie@classsizematters.org

General Comment

To the Commission:

Class Size Matters is a non-profit advocacy organization dedicated to providing

information on the benefits of smaller classes in increasing student learning and

narrowing the achievement gap. 

Given that class size reduction is one of the few measurable education reforms proven to

enhance student success, we believe it very important that federal data reporting of

aggregate student data be improved. This data is especially unreliable, and is often years

overdue when it is released by the Department. Class sizes should also be collected for

individual classrooms because there is much variability within schools. The size of a

class in which a child spends his or her time determines the quality of the environment

and the instruction that the student receives. For some purposes, these can be aggregated

to school averages, but these are not as clearly connected to learning or other student

outcomes. (Note that these are simple and easy to collect, since every teacher has a class

roster with this information.) Special education class sizes should be reported separately.

2228 The Promise of Evidence-Based Policymaking
In addition, many of the independent evaluations of programs funded by the US

Department of Education through the Investing in Innovation Fund (i3) grant, are being

suppressed because federal officials are waiting for its grantees to release them, even

though the evaluations were also paid for with federal dollars. The US ED only lists

links to evaluations of 33 of the 49 I3 grantees from 2010, financed with $646 million,

and some of the links provided are to summaries rather than the actual evaluations. Only

four evaluations of 23 grantees in 2011 funded with $148 million are listed. See

http://innovation.ed.gov/files/2016/09/i3evaluations-160913.xls

When i3 officials were asked why, the response was "I 3 provides links to evaluations

that are currently publicly available. The i3 team continues to work with grantees ... to

make their evaluation findings publicly available." We urge you to ensure that all

independent evaluations paid for with federal taxpayer funds be immediately released.

Finally, the FOIA log of the US Department of Education shows there are hundreds of

public records requests going back as far as 2010 that have not been fulfilled. 

In conclusion, we strongly urge the CEP to recommend that the U.S. Department of

Education improve their monitoring, record-keeping and reporting of aggregate data

when it comes to class sizes, accelerate their responses to public records requests, and

release all independent evaluations of educational programs they have funded - all of

which can be accomplished without the collection of any personal student information. 

Yours,

Leonie Haimson

Executive Director, Class Size Matters 

Attachments

CSM comments to CEP 12.14.16

The Promise of Evidence-Based Policymaking 2229
The Commission on Evidence-Based Policymaking (CEP) 

Input@cep.gov 

December 14, 2016 

To the Commission: 

Class Size Matters is a non-profit advocacy organization dedicated to providing information on the 

benefits of smaller classes in increasing student learning and narrowing the achievement gap. 

Given that class size reduction is one of the few measurable education reforms proven to enhance 

student success, we believe it very important that federal data reporting of aggregate student data be 

improved. This data is especially unreliable, and is often years overdue when it is released by the 

Department. Class sizes should also be collected for individual classrooms because there is much 

variability within schools. The size of a class in which a child spends his or her time determines the 

quality of the environment and the instruction that the student receives. For some purposes, these can 

be aggregated to school averages, but these are not as clearly connected to learning or other student 

outcomes. (Note that these are simple and easy to collect, since every teacher has a class roster with 

this information.) Special education class sizes should be reported separately. 

In addition, many of the independent evaluations of programs funded by the US Department of 

Education through the Investing in Innovation Fund (i3) grant, are being suppressed because federal 

officials are waiting for its grantees to release them, even though the evaluations were also paid for with 

federal dollars.1

 We urge you to ensure that all independent evaluations paid for with federal taxpayer 

funds be immediately released. 

Finally, the FOIA log of the US Department of Education shows there are hundreds of public records 

requests going back as far as 2010 that have not been fulfilled.

2

 

 

1

 The US ED only lists links to evaluations of 33 of the 49 I3 grantees from 2010, financed with $646 million, and 

some of the links provided are to summaries rather than the actual evaluations. Only four evaluations of 23 

grantees in 2011 funded with $148 million are listed. http://innovation.ed.gov/files/2016/09/i3evaluations160913.xls When i3 officials were asked why, the response was “I 3 provides links to evaluations that are currently 

publicly available. The i3 team continues to work with grantees ... to make their evaluation findings publicly 

available.” ( i3@ed.gov email to Leonie Haimson, August 19, 2016.) 

2

 http://www2.ed.gov/policy/gen/leg/foia/request-status-log.pdf 

2230 The Promise of Evidence-Based Policymaking
In conclusion, we strongly urge the CEP to recommend that the U.S. Department of Education improve 

their monitoring, record-keeping and reporting of aggregate data when it comes to class sizes, 

accelerate their responses to public records requests, and release all independent evaluations of 

educational programs they have funded – all of which can be accomplished without the collection of any 

personal student information. 

Yours, 

Leonie Haimson 

Executive Director 

The Promise of Evidence-Based Policymaking 2231
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tla-flhg

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0328

Comment on FR Doc # 2016-22002

Submitter Information

Name: Lynne Taylor

General Comment

We do not need more data mining which combines information from WIOA and ESSA

in the guise of a better educated workforce. We do not need or support the LADDER Act

which is in the Senate HELP committee and would use this private information in ways

to implement the planned National Workforce Plan Council, as laid out in S3174.

Our nation is only shifting the focus of education from academic to workforce based to

satisfy the US/UN 2015 Sustainable Development Goals agreement which calls for a

21st century globally prepared citizen worker.

We MUST stop the UN efforts to strip the US of its education/its heritage/its people's

private information!

I am including a chart I have created after researching not only S3174, but the other

related bills which will help create education/workforce based files of information on our

students, of all ages!

Attachments

2232 The Promise of Evidence-Based Policymaking
fededoverreachchart

The Promise of Evidence-Based Policymaking 2233
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 14, 2016

Tracking No. 1k0-8tla-nqbx

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0329

Comment on FR Doc # 2016-22002

Submitter Information

Name: Abigail Potter

General Comment

See attached file(s)

Attachments

American Trucking Associations Comments on Evidence Based Policymaking Nov 2016

2234 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 2235
2236 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 2237
2238 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tla-kmrk

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0330

Comment on FR Doc # 2016-22002

Submitter Information

Name: Robin Devey

Address:

Orem, UT, 84058

Email: robindevey@gmail.com

General Comment

Our nation must tighten data privacy rules in every respect. Big corporations stand to

gain a lot of money off of our children's data are in favor the loosening of data privacy.

Please leave address information, and other such sensitive information, at the local level

where it belongs.

The Promise of Evidence-Based Policymaking 2239
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tla-y1mh

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0331

Comment on FR Doc # 2016-22002

Submitter Information

Name: Caroline Whistler

General Comment

See attached file(s)

Attachments

Third Sector Response to CEP Request

2240 The Promise of Evidence-Based Policymaking
 

 

Boston & San Francisco | (617) 252-2920 | info@thirdsectorcap.org | www.thirdsectorcap.org

December 14, 2016 

Third Sector Capital Partners, Inc. Comments to the Commission on Evidence-Based Policymaking 

Docket ID USBC–2016–0003 

Third Sector Capital Partners, Inc. (“Third Sector”) is pleased to provide comments on the work of the 

Commission of Evidence-Based Policymaking. Third Sector applauds the Commission’s efforts to 

thoughtfully develop best practices to increase the availability and use of government data in support of 

evidence-building activities related to government programs and policies, while protecting the privacy and 

confidentiality of such data. 

Third Sector is a nonprofit advisory services firm that leads governments, high-performing nonprofits, and 

private funders in building evidence-based initiatives that address society’s most persistent challenges. Our 

ultimate goal is to accelerate America’s transition to a performance-based social sector. Through our work 

with communities across the country that are exploring and launching Pay for Success projects, as well as 

our newest efforts to improve the data infrastructure that supports these projects, Third Sector has a deep 

understanding of the complexities that exist in creating and implementing evidence-based policy. 

The belief that decisions about social programs must be made deliberately and based on evidence is at the 

core of our work. That being said, we must be equally deliberate and evidence-based when deciding to 

reduce or eliminate existing funding for programs; programs should no more be cut because leaders are 

skeptical that they are effective than they should be funded based on optimism alone. As more public 

policies are developed with outcomes-based requirements, we must recognize that those who are actually 

delivering critical services in the field still face significant challenges in delivering evidence-based, datadriven services. We cannot impose a higher bar on these critical service-providers without assuring that 

they have the technical capacity, data access, and resources to reach that bar. 

Second, how existing federal funding is appropriated will be critical to the continued growth of evidencebased practices. At one time, models like Pay for Success (PFS) required new appropriations in order for 

state and local governments to promise to make success payments. Existing appropriations were tied to feefor-service and other legacy paradigms. Recently, however, existing funding streams are being “PFSenabled.” For instance, through the Workforce Innovation & Opportunity Act (WIOA), existing federal 

dollars are being made available to states to re-allocate to PFS. This is a critical development. 

An obstacle to taking advantage of PFS-enabled funding streams, however, is that states essentially must 

ask service-providers to make part of their existing allocations contingent on outcomes with little “upside.” 

Practical implementation for models like PFS is a complicated and challenging process for organizations 

still needing to actively serve people in need. So in addition to making funding “PFS-enabled,” Congress 

should consider including mandated PFS social programming with any new initiative. For example, if there 

is a new infrastructure program, it should include evidence-based PFS programming around workforce 

development and should consider offering additional funding to initially make bonus payments to those 

providers who reach outcomes and/or provide technical assistance to those providers willing to enter into 

contingent contracts. Mandatory use of PFS contracting could then be feathered in over time. 

The Promise of Evidence-Based Policymaking 2241
 

 

Boston & San Francisco | (617) 252-2920 | info@thirdsectorcap.org | www.thirdsectorcap.org

Turning to evaluation, we also know that caution must be taken about how we define success. This work 

must be approached with a certain amount of flexibility, while assuring that we create a feedback loop that 

allows for programs to continuously improve their work delivering outcomes to those they serve. 

A well-crafted randomized control trial (RCT) is the best way to definitely show that a program works -- in 

a certain place, at a certain time with a certain population. We must also, acknowledge, however, what it 

does not show: if that program will be effective in another place or time or with a different or more 

narrowly defined population. There is a real danger that we will be too quick to determine that a program 

does not work and deprive people in real need of an effective intervention. 

RCTs also far too often do not provide useful information to providers. They should be paired with 

program evaluation and enhanced access to data that provides an opportunity to refine services to make 

them more effective. The accountability that is inherent in RCT’s must be balanced with a bias to 

continuous learning, openness, and improvement. It is only when evaluation is not used solely to judge 

programmatic quality that providers and those who they serve will reap the full value of an outcomesoriented mindset for government. 

Finally, in order to best measure the impact social programs are having, existing government data 

infrastructure must be modified to better use and access administrative and survey data. The Commission 

should encourage better access and linking of disparate federal administrative data. Through Third Sector’s 

preliminary work in creating better data infrastructure, we believe that developing this area will allow for 

high-quality program evaluation which in return can assure that evidence-based programs are being 

accurately measured and evolving to meet the needs of those they are serving. 

Third Sector is encouraged by the Commission’s work and the questions it has put forward. We strongly 

believe that a connection between evidence-based policy and an outcomes-oriented mindset is needed in 

government in order to develop effective and impactful policy. As evidence-based outcomes are integrated 

into public policy, we must assure that social services are still available to those in need. However, we 

hope that data, research, and evaluation are used in a thoughtful manner to improve public programs and 

policies. 

Thank you for allowing us the opportunity to comment. 

Sincerely, 

Caroline Whistler 

President & Co-Founder 

Third Sector Capital Partners, Inc. 

2242 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tla-kems

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0332

Comment on FR Doc # 2016-22002

Submitter Information

Name: Alice Linahan

Address:

Argyle, TX, 76226

Email: alinahan@verizon.net

General Comment

As a mom of 3 students, I do not give consent for their private data to be collected by the

state or federal government. I did not have my children to be "human capital" for the

government.

As stated by Emmett McGroarty, director of education at the American Principles

Project... "The government has no constitutional, statutory, or moral right to collect data,

especially highly personal and sensitive socioemotional data on our children." 

Parents are waking up and will stand up against all elected officials and government

officials who violate the privacy of our children and their teachers.

The Promise of Evidence-Based Policymaking 2243
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tla-csgi

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0333

Comment on FR Doc # 2016-22002

Submitter Information

Name: Jeff Davis

General Comment

I believe in informed consent. I oppose non-consensual data mining of children in

schools.

2244 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tla-7qgb

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0334

Comment on FR Doc # 2016-22002

Submitter Information

Name: anonymous anonymous

General Comment

I oppose collecting data without informed consent.

The Promise of Evidence-Based Policymaking 2245
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tla-15la

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0335

Comment on FR Doc # 2016-22002

Submitter Information

Name: GINGER RUSSELL

General Comment

I am appalled that our government has forsaken the Constitution with the Department of

Education and are now creating dossiers on our children without our consent. This needs

to stop immediately. 

Children are no longer taught to succeed on their own accord but are being indoctrinated

and molded into what the Government them to become. PLEASE STOP THIS!!!

2246 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tla-7eol

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0336

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

This country was founded on some amazing principles. The most amazing is that

individuals could be free to do as they please. Government did not control the people,

document excessively or prohibit such ideas as religion, personal character etc. Currently

we see government trying to gain more access to children and adults alike. Which is

creating a national database. Who has access to all of this government documentation in

the digital format? Is it easily traceable? What is the purpose for this data? Will it

identify certain people, ethnicity, gender, etc? Can it tell who the child is, where they

live, what their family structure is like? If it does what purpose does it serve?

Let me tell you how I view this...DANGEROUS IDEA. It is unacceptable and

unamerican. We are a free market, free ideas, freedom should abound everywhere. This

idea breeds control, corruption, manipulation etc. There are no safe guards when it

comes to digital content. It does not exist. To place our children at the front and parade

them in front of everyone from the day they are born is WRONG! Children and adults

alike deserve privacy and NO data collection. Our children should have a future that they

decide. Not a future dictated by data, data, and more data that proves nothing except the

past. You can not measure the human soul and it's worth. 

Data collection has the potential to turn against the children who are our future. And

mark my words, bills like this will be turned against our children. The children will

suffer with poverty, unemployment etc if they are graded, data collected and data mined.

The Promise of Evidence-Based Policymaking 2247
We are seeing that right now...many employers are requesting access to facebook,

snapchat, twitter etc to see if a person is a deemable candidate for hire. It shows the past

not the potential of a person. WHICH IS WRONG TO DO. 

America is suppose to be the land of second, third , fourth chances to make a better life.

This prevents and destroys liberty for the upcoming generations. They deserve a chance

to success, failure and success again without being destroyed by data.

2248 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tlb-7q5z

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0337

Comment on FR Doc # 2016-22002

Submitter Information

Name: James Coleman

Address:

Smyrna, TN, 37167

Email: lillip@comcast.net

General Comment

No data mining. No tracking. No private entities with any access to educational data of

any kind. Not even with informed consent. Sharing of data with the private sector will

never end with any outcome other than to allow private concerns to monetarily benefit

from data, gathered by public funds, through selling it to other private entities. Children

are malleable and often startlingly changeable over time and deserve the right to develop

in a personal, private sphere.

The Promise of Evidence-Based Policymaking 2249
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tlb-aa4x

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0338

Comment on FR Doc # 2016-22002

Submitter Information

Name: Ann Marie Banfield

General Comment

I believe in informed consent. I oppose non-consensual data mining.

2250 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tlb-df94

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0339

Comment on FR Doc # 2016-22002

Submitter Information

Name: Jan Bast

General Comment

"I believe in informed consent. I oppose non-consensual data mining. Please consider

what this will lead to.

The Promise of Evidence-Based Policymaking 2251
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tlc-4is9

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0340

Comment on FR Doc # 2016-22002

Submitter Information

Name: Tamara Scott

General Comment

Every parent deserves the right of informed consent before any data or information is

collected concerning their minor children. It is ethically wrong to miss-use the education

system as a vehicle to confiscate non-consensual data mining. To amass information

without one's knowledge strikes at the heart of America's foundations and our liberties as

a private citizenry. This is government run amok and exactly why the states

overwhelmingly sent a mandate to drain the swamp in DC.

2252 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tlc-xp0j

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0341

Comment on FR Doc # 2016-22002

Submitter Information

Name: Heather Stancil

General Comment

I oppose non-consensual data mining of either adults or our children. We are not

commodities. Stop this madness.

The Promise of Evidence-Based Policymaking 2253
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tld-8ea5

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0342

Comment on FR Doc # 2016-22002

Submitter Information

Name: Heather Hicks

Address:

Forked River, 08731

Email: heather.hicks@comcast.net

General Comment

Docket ID USBC-2016-0003-0001 "Commission on Evidence-Based Policymaking

Comments"

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all students, commonly

referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008, the reasons against creating it

have only become more compelling.

In the past few years, much highly personal data held by federal agencies has been

hacked, including the release of the records of the Office of Personnel Management

involving more than 22 million individuals, not only federal employees and contractors

2254 The Promise of Evidence-Based Policymaking
but also their families and friends.

The US Department of Education has been found to have especially weak security

standards in its collection and storage of student data, and received a grade of D for its

security protections.

In addition, preK-12 student data currently collected by state departments of education

that would potentially be shared with the federal government include upwards of 700

highly sensitive personal data elements, including students' immigrant status, disabilities,

disciplinary records, and homelessness data.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from PreK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban in

the report you provide to Congress.

Sincerely,

Heather Hicks

Special Education Teacher, NJ

Parents Across America- Ocean County NJ

The Promise of Evidence-Based Policymaking 2255
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tld-3r4p

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0343

Comment on FR Doc # 2016-22002

Submitter Information

Name: Felice Levine

Address:

Washington, DC, 20005

Email: flevine@aera.net

General Comment

These comments submitted by the American Educational Research Association (AERA)

replace an earlier comment that AERA submitted on November 14, 2016.

Attachments

AERA Comments to CEP_FINAL SUBMISSION_12-14-16

2256 The Promise of Evidence-Based Policymaking
1430 K Street, NWxWashington, DC 20005x(202) 238Ͳ3200

Facsimile (202) 238Ͳ3250xhttp://www.aera.net

December 14, 2016 

Mr. Nick Hart

Policy and Research Director

Commission on Evidence-Based Policymaking 

Nicholas.r.hart@census.gov 

Docket ID: 160907825-6825-01 Commission on Evidence-Based Policymaking Comments 

Dear Mr. Hart, 

On behalf of the American Educational Research Association (AERA), I am pleased to have the 

opportunity to offer comments in response to the request from the Commission on EvidenceBased Policy to inform the Commission’s work and provide feedback on core questions. The 

AERA is the major national scientific association of more than 25,000 faculty, researchers, 

graduate students, and other distinguished science professionals dedicated to advancing 

knowledge about education, encouraging scholarly inquiry related to education, and promoting 

the use of research to improve education and serve the public good. Founded in 1916, the 

association is committed to the highest standards of research rigor, integrity, and responsibility 

for research scientists, as reflected in such policy documents as AERA’s research standards 

and code of ethics as well as longstanding programs to encourage data sharing and access 

consonant with responsible use. 

Our members use and analyze federal statistics and data in their research and rely on the 

objectivity and trustworthiness of this information. AERA members are interested in increasing 

access to and usefulness of impartial and accurate information to best improve policy and 

practice decision making. Our division of Education Policy counts over 2,500 members and our 

Measurement and Research Methods division more than 3,000 research scientists. Also, AERA 

members participate in 25 established Special Interest Groups focused on various aspects of 

evaluation and statistics. 

AERA applauds the Commission on Evidence-Based Policymaking for the comprehensive 

examination, careful consideration, and inclusive approach. Given the considerable expertise on 

the panel and the extensive feedback from the community both to these questions and in 

meetings and public hearings, we are responding selectively to questions only where AERA has 

specific examples in education to offer or a unique observation based on our own work in this 

area. We have tried to highlight some of the most useful examples from the decades of 

guidance and practice addressing these important questions. The Commission can play a 

critical leadership role in establishing how to expand access to administrative, survey, and 

linked data consonant with appropriate concerns for privacy and protecting confidentiality. 

The Promise of Evidence-Based Policymaking 2257
ʹ

The time is especially ripe for a fresh look under the current circumstances of expanded 

electronic access to wide-ranging information, growing capacity to examine this information 

efficiently, and rising costs in time and money in implementing major surveys. In just this past 

decade, we have seen promising uses of rapidly collected digital information, the transformation 

of administrative information largely in paper form to digitized administrative data systems, and 

a deeper appreciation that quality administrative data, well and widely utilized, can make for 

more robust research that can speak to policy and program development and implementation. 

Fortunately, federal agencies have considerable experience using administrative data systems, 

linking data, and devising strategies for secure access and use. Furthermore, federal agencies 

have provided guidance about how best to expand use of administrative data systems with 

appropriate mechanisms for data protection and access commensurate with the level of risk. 

For over 30 years, the National Academies, most typically at the request of federal agencies, 

has examined and provided guidance on access to federal data assets aligned with privacy 

protection, confidentiality, and data security including useful examples, observations, and 

recommendations that can inform the Commission’s work. See Appendix A. 

In responding to your questions, we believe that many of the obstacles and challenges to 

evidence-based policymaking reside with linking data available in one state/jurisdiction, 

department, or survey to data in another domain or venue. Federal statistical agencies and state 

longitudinal data systems have made tremendous strides not only in creating secure and 

workable data infrastructure, but also in using those data to improve student outcomes. For 

example, since 2009, under the leadership of the National Center for Education Statistics, there 

has been considerable progress in the development of Common Education Data Standards 

(CEDS) for use in creating and maintaining statewide longitudinal data systems (SLDS). There 

also, however, needs to be attention to a data infrastructure that could enable the next level of 

using data to determine what works and where to invest our money. The most effective solution 

would be to develop and link to student indicators that could effectively allow policy analysts and 

researchers to follow students through their education experience and into the workforce. 

Before turning to the specific questions, AERA would like to raise two very important issues: 

The first relates to the necessity of protecting the vitality, autonomy, and independence of our 

statistical agencies. Consistent with recommendations offered in the National Academies of 

Sciences report, Principles and Practices for a Federal Statistical Agency, careful consideration 

must be given to the authority to protect agency independence from political or other undue 

outside influence. 

In particular, “a strong position of independence” is one of its four key principles. The 

ensuing guidance for this principle includes the following: 

Authority to release statistical information and accompanying materials (including 

press releases) without prior clearance for the statistical content by department 

policy officials is essential so that there is no opportunity for or perception of political 

manipulation of any of the information. (p. 41) 

Such organizational aspects as direct access to the secretary of the agency’s 

department and separate budgetary authority are neither necessary nor sufficient for 

a strong position of independence that protects a statistical agency from undue 

political influence, but they facilitate such independence. (p. 37)

2258 The Promise of Evidence-Based Policymaking
͵

We urge you to consider the importance of the leadership of our federal statistical agencies. 

Not only must leadership reflect technical expertise and understanding of data use, but 

leaders should also have the autonomy and status to allow them to meaningfully advise 

agency heads and to insulate them from political influence. The Presidential appointment of 

directors of statistical agencies helps to ensure the accountability of these statistical 

agencies to Congress and to the public. Furthermore, these appointments provide for direct 

reporting on statistical indicators to the Office of the Secretary.

The second relates to the continued need for a student unit record system that can track 

students and provide legitimate access aligned with privacy protection. Simply put, students 

and families, policy makers, and the general public have a deep and legitimate need for 

high quality information about postsecondary education to inform everything from lifechanging individual choices to public policies affecting a $500 billion sector of the economy. 

Unfortunately, in the United States, our postsecondary education data systems represent a 

complex patchwork of data collections and work-arounds that, even taken together, do not 

support these information needs. 

The nation needs a student unit record system that captures the full experience of 

postsecondary education at the individual student level and allows for aggregation to the 

program, institution, and system levels to move our understanding of this vital component of 

our education and human capital development system to the modern era. We urge the 

Commission to consider this issue in a bi-partisan and scientifically sound way as a service 

to the nation and to the publics we serve. 

Overarching Questions 

1. Are there successful frameworks, policies, practices, and methods to overcome challenges 

related to evidence-building from state, local, and/or international governments the Commission 

should consider when developing findings and recommendations regarding Federal evidencebased policymaking? If so, please describe. 

There is a history of practice and guidance from the federal statistical agencies and from the 

Office of Management and Budget (OMB) Office of Information and Regulatory Affairs (OIRA) 

directed to promoting the quality of federal data and statistical information to enhance evidencebased policies and programs. The leadership of OIRA’s Statistical and Policy Office has been 

instrumental in the federal and international arenas that could be usefully extrapolated in 

considering how best to strengthen data quality, access, and use with state and local data. 

In the domain of education programs and policy, the State Longitudinal Data Systems (SLDS) 

Grant Program, authorized by the Educational Technical Assistance Act of 2002, Title II of the 

statute that created the Institute of Education Sciences (IES), is arguably, a successful 

framework for state educational evidence-building. 

Thanks to federal support, the program has enabled the successful design, development, 

implementation, and expansion of K12 and P-20W (early learning through the workforce) 

longitudinal data systems in nearly every state. In fact, 47 states have legislated state funds to 

continue the operations after using federal funds to build the infrastructure. As articulated by 

Robert Swiggum in his testimony before the House Education and Workforce Committee, SLDS 

The Promise of Evidence-Based Policymaking 2259
Ͷ

enable states to efficiently and accurately manage, analyze, and use education data. In 

Georgia, SLDS have led to data-informed decisions to improve student learning and outcomes. 

As part of a joint conference directed to developing model guidelines for use of longitudinal 

administration systems convened by AERA and the Organisation for Economic Co-operation 

and Development (OECD) in December 2015, representatives from administrative data systems 

described how linkages in data sets can provide for robust findings with policy implications. One 

model for the U.S. is the United Kingdom’s Administrative Data Research Network, a repository 

of administrative linked and de-identified data sets made available to social science researchers 

under secure conditions. Data linked among multiple sources for approved research projects 

have provided relevant information for policy decisions with the goal of benefiting society. For 

example, the National Pupil Database—which connects data sources for exam results, 

attendance records, name of the school a child is attending with a student identifier—allows for 

decisions on how much money from the national education budget is given to particular local 

authorities and schools. Another project in process is linking data on unemployment benefits 

and successive sanctions with Scottish data on school attendance to determine whether there 

are unintended consequences to children when parents’ unemployment benefits are stopped 

with the aim to encourage them to return to work. 

2. Based on identified best practices and existing examples, what factors should be considered 

in reasonably ensuring the security and privacy of administrative and survey data? 

Federal agencies and research organizations are successfully ensuring security and privacy of 

data by considering the nature of the data, having procedures for determining who has access 

to data and for what purpose as well as controlling the nature of the access, including location. 

Assuming that the risk is the ability to drill down and identify a particular student, the National 

Center for Education Statistics has systems in place to protect individual data, strip direct 

identifiers, and mask indirect indicators. 

In fact, NCES has pioneered making available data sets with personally identifiable information 

(PII) to researchers, through restricted-use data licenses. Authorized users are subject to the 

laws, regulations, and penalties that apply to the NCES use of confidential data of up to 

$250,000 and six months in jail. The NCES Statistical Standards Program monitors the licensing 

process and inspections. IES reviews analysis of NCES data. The NCES website has extensive 

materials on data access to public use and restricted-use data, including a Restricted-Use Data 

Procedure Manual (NCES 2007 at http://nces.ed.gov/pubs96/96860rev.pdf) 

The Interuniversity Consortium for Political and Social Research (ICPSR) has established a 

protocol for preserving respondent confidentiality that starts with the depositors of data, 

requiring documentation for information that could identify respondents, which could establish 

restricted use to the data. ICPSR offers four levels of restricted use: Traditional Restricted Data, 

Physical Data Enclave, Restricted Online Analysis, and Delayed Dissemination. 

Data Infrastructure and Access

3. Based on identified best practices and existing examples, how should existing government 

data infrastructure be modified to best facilitate use of and access to administrative and survey 

data?

4. What data-sharing infrastructure should be used to facilitate data merging, linking, and 

access for research, evaluation, and analysis purposes? 

2260 The Promise of Evidence-Based Policymaking
ͷ

When considering best practices regarding data infrastructure, we encourage you to consult the 

OMB Guidance for Providing and using Administrative Data for Statistical Purposes (February

14), 2014. This document provides tools and detailed guidance on the interaction of the use of 

administrative data for statistical purposes with the Privacy Act requirement. In addition, Sharing 

Data While Protecting Privacy (M-11-02 of November 3, 2010), Open Data Policy-Managing 

Information as an Asset (M-13-13 of May 9, 2013), and Next Steps in the Evidence and 

Innovation Agenda (M-13-17 of July 26, 2013) are three useful OMB memoranda. 

5. What challenges currently exist in linking state and local data to federal data? Are there 

successful instances where these challenges have been addressed? 

We addressed above the critical need for a student unit record system. Here we address your 

specific question. When students move to another state, either during PreK-12 or to attend 

college, a state has no way to follow these students. For many, the ultimate goal of education is 

to prepare individuals for a transition to the workforce and an independent and productive life. 

The only way to capture information about students who move from state to state would be to, in 

some non-identifying or masked way, have records that can be linked based on the student. 

The Integrated Postsecondary Education System (IPEDS) is an extremely useful widely used 

resource. Unfortunately, IPEDS data collection is limited to full-time students who begin their 

studies in the fall. 

Looking specifically at NCES, efforts to access data in the state longitudinal data systems must 

be negotiated with each individual state. Even for states inclined to make every effort to share 

state-level data, laws and regulations regarding protections of student data privacy are 

sufficiently ambiguous leading states to err on the side of caution deciding against sharing data. 

6. Should a single or multiple clearinghouse(s) for administrative and survey data be established 

to improve evidence-based policymaking? What benefits or limitations are likely to be 

encountered in either approach? 

Subject to security concerns, a central repository for collected data is a good idea as it will 

facilitate administrative access and research. To maximize the value, data should be made 

accessible to responsible researchers. Support of a central clearinghouse does not mean that 

all federal agency data need to be physically stored in a single unit (or its cloud). It does, 

however, suggest that a single entity should be responsible for setting standards, tracking data 

assets, assessing risk, and ensuring appropriate levels of data access and management. 

A significant benefit of a clearinghouse would be the opportunity to provide long-term safe 

storage of data. A fundamental part of the scientific process is verifying findings and testing new 

hypotheses using the same dataset, and premature destruction of the data can waste valuable 

resources and limit building cumulative and reproducible knowledge. Having the benefit of 

understanding the complete data picture, the management of the clearinghouse could provide 

some latitude and direction to educational agencies and research organizations to determine 

when data are no longer needed for the agreed upon scientific purposes and to retain 

identifiable datasets where necessary under strictly-controlled, secure conditions (as is done 

with other federal and state statistical and record-keeping systems). 

8. What factors or strategies should the Commission consider for how a clearinghouse(s) could 

be self-funded? What successful examples exist for self-financing related to similar purposes? 

The Promise of Evidence-Based Policymaking 2261
͸

While still too early to tell if effective, the recently passed bipartisan reauthorization of the 

Elementary and Secondary Education Act (ESEA) includes Sec. 8601. Evaluations. This section 

requires IES to do evaluations of each program authorized in the Act. The legislation allows the 

Secretary to reserve .5 percent of the program account for the evaluation costs and 

dissemination of findings. In addition the section provides guidance on the design of 

evaluations. 

(a) RESERVATION OF FUNDS. – Except as provided in subsection (b) and (e), the 

Secretary, in consultation with the Director of the Institute of Education Sciences, may 

reserve not more than 0.5 percent of the amount appropriated for each program 

authorized under this Act to carry out activities under this section. (ESSA) 

In addition to the cost of conducting evaluations, AERA encourages the Commission to think 

about the workforce capacity to most instructively conduct and interpret evaluations. IES has 

developed tremendous technical capacity in this regard, thanks to hiring flexibility permitted by 

their accepted service hiring authority. 

9. What specific administrative or legal barriers currently exist for accessing survey and 

administrative data? 

As mentioned in previous responses, numerous legal, regulatory, and operational barriers 

prevent federal agencies from linking to state data. Census and NCES have been required to 

approach each state individually to initiate data linkages. Even when states are motivated to 

share data, they are often advised against doing so due to legal concerns. 

10. How should the Commission define “qualified researchers and institutions?” To what extent 

should administrative and survey data held by government agencies be made available to 

“qualified researchers and institutions?” 

Administrative and survey data held by government agencies should be maximally available to 

qualified researchers and institutions. Users could be required to provide data use and 

management plans that would explicate the researcher's intentions for their data both during 

and after the research project. Such procedures are common among agencies and institutions 

that provide access under restricted conditions. 

In thinking about safeguarding student data, the question of who is qualified to have access 

should include consideration of whether those who already collect and manage the data would 

benefit from additional training and capacity building. Meaningful training and support of 

government employees or other users would help build a culture of trust in schools and school 

systems and implement best practices in data privacy and security. 

The Privacy Technical Assistance Center (PTAC) is a strong example of effective federal 

government tools to support local infrastructure and capacity building. PTAC provides great 

value to the field through its hotline and its guidance on such important issues as data breach 

response and model terms of service. This important role could be expanded to provide 

additional tools and resources to develop policies and best practices in transparency, 

governance, and privacy and security. 

14. What incentives may best facilitate interagency sharing of information to improve 

programmatic effectiveness and enhance data accuracy and comprehensiveness? 

2262 The Promise of Evidence-Based Policymaking
͹

Interagency information sharing is presumably beneficial to all involved agencies. Removing 

obstacles might be akin to providing incentives. In addition to alleviating concerns about the 

legality of sharing information, agencies engaged in high-level analysis of data would benefit 

from the flexibility in hiring highly qualified staff to maximize the benefit of increasing access to 

data and effective data management plans. 

Data Use in Program Design, Management, Research, Evaluation, and Analysis 

15. What barriers currently exist for using survey and administrative data to support program 

management and/or evaluation activities? 

Currently, administrative and legal barriers too often prevent the use of administrative data. For 

example, researchers looking to determine how certain P-12 programs may affect wages later in 

life are unable to access wage information readily available with IRS or the Social Security 

Administration. Even in cases where two federal agencies are able to reach an agreement to 

share wage data for a particular study, records are limited to individuals with a listed social 

security number, possibly biasing the findings. In some cases, soft matches, using name and 

birthdate may accurately identify individual administrative data without social security numbers. 

Unfortunately, soft matches are time consuming—preventing federal agencies from committing 

the time and resources to pursuing soft matches. 

Users of survey data face different challenges. It is very expensive to collect survey data. Social 

and behavioral scientists have made great strides to improve sampling to enable survey data 

collectors to gather information from a more limited population that can adequately represent a 

broader population. 

Another barrier to using survey and administrative data is the variation in definitions. A 

successful effort to standardize terms, noted above, was the Common Education Data 

Standards. Led by NCES, CEDS are voluntarily developed common data standards for a key 

set of education data elements to streamline the exchange, comparison, and understanding of 

data within and across early learning through postsecondary and workforce settings. 

19. To what extent should evaluations specifically with either experimental (sometimes referred 

to as “randomized control trials”) or quasi-experimental designs be institutionalized in 

programs? What specific examples demonstrate where such institutionalization has been 

successful and what best practices exist for doing so? 

AERA supports the use of the most rigorous methods consonant with the research issues and 

contexts of study and program evaluation. We appreciate the importance of randomized control 

trials and experimental and quasi-experimental designs. Rather than institutionalizing any single 

method, however, we encourage embracing the importance of using methods appropriate to the 

research or the program being evaluated. RCT designs are valuable methodologies in isolating 

effects when appropriate; nevertheless, many interventions and programs cannot be introduced 

under conditions that would differentially provide known benefits or potentially introduce risks 

that would negatively affect individuals. 

Instead of emphasizing only randomized control trials, AERA offers the following definition of 

scientifically based research (SBR), which provides a broader definition grounded in scientific 

standards and principals. It was developed by an expert working group convened by the 

American Educational Research Association (AERA) in June 2008. 

The Promise of Evidence-Based Policymaking 2263
 ͅ

 Alternate Definition of Scientifically Based Research (SBR) 

Supported by AERA Council, July 11, 2008

I. The term “principles of scientific research” means the use of rigorous, systematic, and 

objective methodologies to obtain reliable and valid knowledge. Specifically, such 

research requires: 

A. development of a logical, evidence-based chain of reasoning; 

B. methods appropriate to the questions posed; 

C. observational or experimental designs and instruments that provide reliable 

and generalizable findings; 

D. data and analysis adequate to support findings; 

E. explication of procedures and results clearly and in detail, including specification 

of the population to which the findings can be generalized; 

F. adherence to professional norms of peer review; 

G. dissemination of findings to contribute to scientific knowledge; and 

H. access to data for reanalysis, replication, and the opportunity to build on findings. 

II. The examination of causal questions requires experimental designs using random 

assignment or quasi-experimental or other designs that substantially reduce plausible 

competing explanations for the obtained results. These include, but are not limited to, 

longitudinal designs, case control methods, statistical matching, or time series analyses. 

This standard applies especially to studies evaluating the impacts of policies and 

programs on educational outcomes. 

III. The term “scientifically based research” includes basic research, applied research, and 

evaluation research in which the rationale, design, and interpretation are developed in 

accordance with the scientific principles laid out above. The term applies to all 

mechanisms of federal research support, whether field-initiated or directed. 

This definition is compatible with the language in the recently reauthorized ESSA that 

commends “impact evaluations that use experimental or quasi-experimental designs, where 

practicable and appropriate, and other rigorous methodologies that permit the strongest 

possible causal inference.” 

We wish to conclude with four guiding principles for the Commission’s consideration as you 

proceed with your essential work. 

1. Define ‘evidence’ and ‘effectiveness’ broadly to account for the spectrum of outcomes 

significant to assessing program and policy goals. 

2. Ensure a robust understanding of the methodologies essential to studying effectiveness, 

short- and long-term consequences, and unintended effects. These would include but not be 

limited to experimental and quasi-experimental methods, longitudinal designs, statistical 

matching, and so forth. 

3. Examine and invest in making accessible federal data assets, including administrative 

information, under institutional arrangements and data use agreements that maximize the 

2264 The Promise of Evidence-Based Policymaking
ͻ

capacity to examine policies and programs consonant with privacy provisions and confidentiality 

protections. Review current data use agreements and data management plans to maximize 

access under conditions of data security. 

4. Evaluate the leadership of statistical agencies, maximizing autonomy to allow for expert 

advice based on sound evidence and to safeguard statistical agencies from political influence. 

Leadership should reflect technical expertise and understanding of data use. 

AERA very much supports the Commission’s hard work and open approach to this complex and 

ambitious task. We appreciate the enormous progress that you are making in promoting sound 

policy and programs through greater secure and responsible use of data systems. We welcome 

helping and supporting you in that effort. 

Sincerely,

Felice J. Levine, PhD 

Executive Director 

flevine@aera.net

202.238.3201 

American Educational Research Association 

1430 K Street, NW

Suite 1200

Washington, DC 20005 

The Promise of Evidence-Based Policymaking 2265
ͳͲ

Appendix A

Reports on Protecting Participants and Facilitating Research

from the National Academies

With Additions from document initially prepared in October 2011 

by Dr. Connie Citro, Committee on National Statistics

The challenge of protecting human subjects in biomedical and social and behavioral sciences 

research while facilitating responsible research and access to research data has engaged the 

attention of federal agencies, the National Academies, and the scientific community for 

decades. Below is a chronological list of major reports from the National Research Council and 

the Institute of Medicine. 

National Research Council. (1979). Privacy and confidentiality as factors in survey response.

Washington, DC: National Academy of Sciences. 

National Research Council. (1985). Sharing research data. Washington, DC: National Academy 

Press. 

National Research Council. (1993). Private lives and public policies: Confidentiality and 

accessibility of government statistics. Washington, DC: National Academy Press 

National Research Council. (2000). Improving access to and confidentiality of research data: 

Report of a workshop. Washington, D.C.: National Academy Press. 

Institute of Medicine. (2000). Protecting data privacy in health services research. Washington, 

DC: National Academy Press. 

Institute of Medicine. (2001). Preserving public trust: accreditation and human research 

participant protection programs. Washington, DC: National Academy Press.

Institute of Medicine. (2002). Responsible research: a systems approach to protecting research 

participants. Washington, DC: The National Academies Press. 

National Research Council. (2003). Protecting participants and facilitating social and behavioral 

sciences research. Washington, DC: The National Academies Press. 

Institute of Medicine. (2004). The ethical conduct of clinical research involving children.

Washington, DC: The National Academies press. 

National Research Council and Institute of Medicine. (2005). Ethical considerations for research 

on housing-related health hazards involving children. Washington, DC: The National 

Academies Press. 

National Research Council. (2005). Expanding access to research data: Reconciling risks and 

opportunities. Washington, DC: The National Academies Press. 

Institute of Medicine. (2006). Effect of the HIPAA privacy rule on health research: Proceedings 

of a workshop presented to the National Cancer Policy Forum. Washington, DC: The 

National Academies Press. 

2266 The Promise of Evidence-Based Policymaking
ͳͳ

National Research Council. (2006). Improving business statistics through interagency data 

sharing: Summary of a workshop. Washington, DC: The National Academies Press. 

National Research Council. (2007). Putting people on the map: Protecting confidentiality with 

linked social-spatial data. Washington, DC: The National Academies Press. 

National Research Council. (2007). Engaging privacy and information technology in a digital 

age. Washington, DC: The National Academies Press. 

Institute of Medicine. (2007). Ethical considerations for research involving prisoners. 

Washington, DC: The National Academies Press. 

National Research Council. (2007). Understanding business dynamics: An integrated data 

system for America’s future. Washington, DC: The National Academies Press. 

Institute of Medicine. (2009). Beyond the HIPAA privacy rule: enhancing privacy, improving 

health through research. Washington, DC: The National Academies Press. 

National Research Council. (2009). Ensuring the integrity, accessibility and stewardship of 

research data in the digital age. Washington, DC: The National Academies Press. 

National Research Council. (2009). Principles and practices for a federal statistical agency.

Washington, DC: The National Academies Press. 

National Research Council. (2009). Protecting student records and facilitating education 

research: A workshop summary. Washington, DC: The National Academies Press. 

National Research Council. (2010). Conducting biosocial surveys: Collecting, storing, 

accessing, and protecting biospecimens and biodata. Washington, DC: The National 

Academies Press. 

National Research Council. (2010). Protecting and accessing research data from the Survey of 

Earned Doctorates: a research summary. Washington, DC: The National Academies Press. 

National Research Council (2014). Proposed revisions to the common rule for the protection of 

human subjects in the behavioral and social sciences. Washington, DC: The National 

Academies Press. 



The Promise of Evidence-Based Policymaking 2267
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tld-790g

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0344

Comment on FR Doc # 2016-22002

Submitter Information

Name: James Ruairi Macdonald

Address:

Washington, DC, 20002

Email: jrmacdonald@law.gwu.edu

General Comment

RE: Federal Register Notice #2016-22002 

Submission to Commission on Evidence-Based Policymaking (CEP)

Please consider the following suggestions in response to your questions 3 and 4 in the

Federal Register Notice referenced above. My suggestions focus on the bipartisan

Digital Accountability and Transparency Act of 2014 (DATA Act) (PL 113-101). I

would be delighted to expand upon these suggestions upon request. My e-mail is

jrmacdonald@law.gwu.edu.

My suggestion in response to Question 3: DATA Act implementation is emerging as a

best practice for the development of data standards and new data systems. The DATA

Act is really about describing federal dollars using the same words and definitions (or

taxonomy) about the "who, what, when, where, and how" federal funds are being spent.

The taxonomy has been made public. Federal spending data is soon to be made public

using the new taxonomy. The idea is that if the same words are used by all the people,

organizations, and systems in the money chain, then it becomes easier to follow the

2268 The Promise of Evidence-Based Policymaking
money. The theory is that if citizens, federal staff, and policy-makers can "follow the

money," there will be less waste and better management of federal funds. The process of

developing the taxonomy was open and collaborative. I suggest a similar collaborative

process be used to develop a taxonomy for program datasets and especially for program

evaluation reports. This new taxonomy should describe "who, what, when, where, and

how" the evidence was developed. The taxonomy does not necessarily need to describe

the conclusions or quality of the evaluation reports and datasets. My suggestion is that if

the same words are used by all the people, organizations, and systems in the program

implementation chain, then it becomes easier to "follow the evidence." In theory if

citizens, federal staff, and policy-makers can "follow the evidence," they will focus more

on promising practices and better implementation of federal programs. 

My suggestion in response to Question 4: DATA Act implementation is showing

positive signs for improving the exchange of data between different federal systems. The

DATA Act approach is to use open data standards rather than an expensive new super

system. The DATA Act Information Model Schema (DAIMS) helps the U.S. Treasury to

take information from multiple agency systems for publication on a new website called

USAspending.gov. This data is also to be made available as an open dataset on data.gov.

As mentioned in my previous suggestion, a process similar to DATA Act process could

be used to develop a taxonomy for program evaluation reports and datasets used for

evidence-based policy making. I suggest that an "evidence-based policy information

model schema" or similar would make it easier and less expensive to develop new public

or private systems or processes to manage evidence. Additionally, the words (or

taxonomies or standards) used to describe both DATA Act information and for program

data and documents, it may become easier to link spending information with program

implementation and performance information. I suggest that the benefits of this linking

may include more cost-savvy program decisions and improved decision-making by

congress on where to allocate resources. 

Thank you for your kind attention to these suggestions. 

[End.]

Attachments

CEP_JRMacdonald_12-14-16

The Promise of Evidence-Based Policymaking 2269
J. Ruairi Macdonald, LLM 

Student, Ph.D. (Program Evaluation) 

George Washington University 

Trachtenberg School of Public Policy and Public Administration 

805 21st St. NW, Suite 601 

Washington, DC 20052 

December 14, 2016 

RE: Federal Register Notice #2016-22002 

Submission to Commission on Evidence-Based Policymaking (CEP) 

Please consider the following suggestions in response to your questions 3 and 4 in the Federal Register 

Notice referenced above. My suggestions focus on the bipartisan Digital Accountability and Transparency 

Act of 2014 (DATA Act) (PL 113-101). I would be delighted to expand upon these suggestions upon request. 

My e-mail is jrmacdonald@law.gwu.edu. 

My suggestion in response to Question 3: DATA Act implementation is emerging as a best practice for 

the development of data standards and new data systems. The DATA Act is really about describing federal 

dollars using the same words and definitions (or taxonomy) about the “who, what, when, where, and how” 

federal funds are being spent. The taxonomy has been made public. Federal spending data is soon to be 

made public using the new taxonomy. The idea is that if the same words are used by all the people, 

organizations, and systems in the money chain, then it becomes easier to follow the money. The theory is 

that if citizens, federal staff, and policy-makers can “follow the money,” there will be less waste and better 

management of federal funds. The process of developing the taxonomy was open and collaborative. I 

suggest a similar collaborative process be used to develop a taxonomy for program datasets and especially 

for program evaluation reports. This new taxonomy should describe “who, what, when, where, and how” the 

evidence was developed. The taxonomy does not necessarily need to describe the conclusions or quality of 

the evaluation reports and datasets. My suggestion is that if the same words are used by all the people, 

organizations, and systems in the program implementation chain, then it becomes easier to “follow the 

evidence.” In theory if citizens, federal staff, and policy-makers can “follow the evidence,” they will focus 

more on promising practices and better implementation of federal programs. 

My suggestion in response to Question 4: DATA Act implementation is showing positive signs for 

improving the exchange of data between different federal systems. The DATA Act approach is to use open 

data standards rather than an expensive new super system. The DATA Act Information Model Schema 

(DAIMS) helps the U.S. Treasury to take information from multiple agency systems for publication on a new 

website called USAspending.gov. This data is also to be made available as an open dataset on data.gov. As 

mentioned in my previous suggestion, a process similar to DATA Act process could be used to develop a 

taxonomy for program evaluation reports and datasets used for evidence-based policy making. I suggest 

that an “evidence-based policy information model schema” or similar would make it easier and less 

expensive to develop new public or private systems or processes to manage evidence. Additionally, the 

words (or taxonomies or standards) used to describe both DATA Act information and for program data and 

documents, it may become easier to link spending information with program implementation and 

performance information. I suggest that the benefits of this linking may include more cost-savvy program 

decisions and improved decision-making by congress on where to allocate resources. 

Thank you for your kind attention to these suggestions. 

[End.] 

J. Ruairi Macdonald jrmacdonald@law.gwu.edu 12-14-16 

2270 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tld-wr4y

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0345

Comment on FR Doc # 2016-22002

Submitter Information

Name: Andrew Barnett

General Comment

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all students, commonly

referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008, the reasons against creating it

have only become more compelling.

In the past few years, much highly personal data held by federal agencies has been

hacked, including the release of the records of the Office of Personnel Management

involving more than 22 million individuals, not only federal employees and contractors

but also their families and friends.

The US Department of Education has been found to have especially weak security

standards in its collection and storage of student data, and received a grade of D for its

security protections.

In addition, preK-12 student data currently collected by state departments of education

that would potentially be shared with the federal government include upwards of 700

highly sensitive personal data elements, including students' immigrant status, disabilities,

disciplinary records, and homelessness data.

The Promise of Evidence-Based Policymaking 2271
I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from PreK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban in

the report you provide to Congress.

Yours, [name, state, and organization affiliation if any]

2272 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tld-ct7o

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0346

Comment on FR Doc # 2016-22002

Submitter Information

Name: K Connor

General Comment

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all students, commonly

referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008, the reasons against creating it

have only become more compelling.

In the past few years, much highly personal data held by federal agencies has been

hacked, including the release of the records of the Office of Personnel Management

involving more than 22 million individuals, not only federal employees and contractors

but also their families and friends.

The US Department of Education has been found to have especially weak security

standards in its collection and storage of student data, and received a grade of D for its

security protections.

In addition, preK-12 student data currently collected by state departments of education

that would potentially be shared with the federal government include upwards of 700

highly sensitive personal data elements, including students' immigrant status, disabilities,

disciplinary records, and homelessness data.

The Promise of Evidence-Based Policymaking 2273
I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from PreK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban in

the report you provide to Congress.

2274 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tle-i564

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0347

Comment on FR Doc # 2016-22002

Submitter Information

Name: Hilary Shager

Address:

Madison, WI, 53706

Email: hshager@lafollette.wisc.edu

General Comment

This semester, in a new course at the University of Wisconsin-Madison, Robert M. La

Follette School of Public Affairs, "Evidence-Based Policymaking," 15 master of public

affairs (MPA) students studied the following topics:

1) What does "evidence-based" mean? What are different kinds of evidence, and how are

they useful? How does one judge the rigor of research evidence?

2) What is the difference between an education-based approach to working with

policymakers and an advocacy-based approach?

3) What are good strategies for communicating research to policymakers (including

written and oral presentations, data visualization)?

4) What are examples of successful evidence-based policymaking efforts?

5) What are the limits of using research in policymaking?

We have followed the activities of the Commission and offer our public comments in the

attached document.

The Promise of Evidence-Based Policymaking 2275
Attachments

Commission Public Comments

2276 The Promise of Evidence-Based Policymaking
1 

To: The Commission on Evidence-Based Policymaking

From: Public Affairs 974 Class: Evidence-Based Policymaking at the University of Wisconsin 

Madison, Robert M. La Follette School of Public Affairs

Overarching Questions:

• We have some general questions that we hope the Commission will address: 

o What constitutes quality in evidence? 

o What makes the results from a study valid?

o What are the Commission’s standards for “good evidence”?

Data Infrastructure and Access:

• The Commission should help promote and facilitate partnerships between government 

entities and universities, to encourage the use of administrative data and data preparation, 

management, and analysis. The University of Wisconsin-Madison has several great 

examples of these types of partnerships; for example, the Institute for Research on 

Poverty has been merging administrative data sets and developing protocols for data 

quality and confidentiality for years. These types of partnerships not only lead to 

effective data cleaning and processing, but also provide a strong pool of researchers 

access to the data, to be used to address some of society’s thorniest problems.

• A single database, including both administrative and survey-based data could improve 

evidence-based policymaking. Right now, many useful data sources are difficult to 

access (e.g., only through universities or specialized subscriptions) and expensive. If

such data were made public, one would have to consider the consequences for these types 

of organizations that rely on such data fees as revenue sources. A single database would 

likely encourage more widespread use of the resource among policymakers at all levels, 

as well as the general public. Privacy is certainly also a concern; however, there are 

already many government data warehouses with sound privacy practices that could serve 

as models.

• We had a number of discussions about data privacy and concerns about data being used 

in ways that could potentially harm research subjects (intentionally or unintentionally). 

However, it is also important to weigh concerns about privacy against the consequences 

of not using data to assist in the development and evaluation of policy and programs, or 

not leveraging data that is already collected, which could result in significant cost-savings 

to universities, states, and the federal government (thus, taxpayers). Some suggestions 

for building in privacy protections include de-identifying individual data as much as 

possible, allowing broader access to aggregated-only data.

Data Use in Program Design, Management, Research, Evaluation, and Analysis:

• It is important to discuss with and provide clarity to stakeholders about how data will be 

used. Collection of data can be burdensome and expensive, and is primarily done at the 

ground (program) level. If there is no “grand purpose” for collecting the data (i.e., how 

The Promise of Evidence-Based Policymaking 2277
2 

will it be used; how is it helpful to programs), one can expect lower data quality, 

practitioner burnout, etc. We must work to build better partnerships between researchers 

and program implementers, and incentives for effective data collection and utilization.

• If a program or policy is found to be “ineffective” via evaluation or some other 

“evidence-based” context, it is important that the implementation of the program be 

closely examined. The conceptual effectiveness of a program may be sound, and the 

problem may be one of delivery. Modifying how that policy or program is delivered may 

be the route to take vs. discontinuing the program.

• It is important to build in program evaluation into program design and implementation, 

and programs should continue to be evaluated over time to provide a “feedback loop” for 

quality control and learning. However, some questions must also be addressed: 

o Who should oversee and review evaluation, and ensures ethical evaluation 

practices? Universities often have Institutional Review Boards, but what does this 

look like within programs, small organizations, or different levels of government?

o How will the quality of evaluation be guaranteed? 

o How can privacy for participants be ensured?

o Who will have access to the data?

o How often is data updated (and, related, how valid are results given the age of 

data)?

o How are “opt-out” processes developed and monitored, and how do they impact 

evaluation results?

o Who is responsible for de-identifying data, and how are protocols established?

• Logic models help clarify program components and mechanisms and provide a clear, 

agreed-upon evaluation roadmap.

• One concern is that the most rigorous types of evidence-based standards may not be 

applicable to all situations. For example, RCT’s may not be viable in victim services, 

and some populations (e.g., tribal populations) may be hesitant to comply with such 

protocols. Will services for these situations and groups be considered “lesser”? Not as 

strongly funded?

• Another issue is complexity of programming. Consider economic development—it may 

be very difficult to assess the myriad components in a community economic development 

strategy or to assess of the possible outcomes that could be affected. Sharing 

administrative data across agencies and programs seems key to truly understanding, 

defining, and replicating “what works” in this type of situation.

Instructor: Hilary Shager, PhD

Class Members:

Richelle Andrae

Danny Benson

2278 The Promise of Evidence-Based Policymaking
3 

David Boardman

Scott Coleman

Chad Farley

Andrew Fisher

Falon French

Angela Guarin

Kelly McDowell

Beth Miller

Sam Rebenstorf

Mitchell Running

Sara Sanders

Ken Smith

Karina Virrueta

The Promise of Evidence-Based Policymaking 2279
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tle-2k3k

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0348

Comment on FR Doc # 2016-22002

Submitter Information

Name: Marcia Hora

General Comment

I believe in informed consent. I oppose non-consensual data mining. Stop this madness.

2280 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tle-ug8j

Comments Due: December 14, 2016

Submission Type: API

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0349

Comment on FR Doc # 2016-22002

Submitter Information

Name: Matthew D Shapiro

Address:

Ann Arbor, MI, 48109

Email: matthewshapiroecon@gmail.com

General Comment

See attached file(s)

Attachments

Comment on Data Synchronization for the Commission on Evidence

The Promise of Evidence-Based Policymaking 2281
Comment on Data Synchronization for the Commission on Evidence-Based Policymaking 

I call the Commission’s attention to the Federal Economic Statistics Advisory Committee 

(FESAC) Statement on Data Synchronization. Improved data synchronization among the 

Federal economic statistics agencies would improve the quality of data available for 

policymaking while protecting confidentiality and privacy.

The FESAC statement is inserted below and can be found at http://www.census.gov/fesac/.

Respectfully submitted, 

Matthew D. Shapiro 

Ann Arbor, Michigan 

FESAC STATEMENT ON DATA SYNCHRONIZATION 

December 12, 2014 

The Federal Economics Statistics Advisory Committee (FESAC) recommends 

that legal and practical barriers to synchronizing business data among the Bureau 

of Economic Analysis (BEA), the Bureau of Labor Statistics (BLS) and the 

Bureau of the Census be removed. Removing barriers to synchronizing business 

data will improve the measurement of key national indicators on the health of the 

American economy such as output, productivity, earnings, job growth and 

inflation as well as improve the efficiency and effectiveness of the agencies. 

Accordingly, measures should be taken to enable the sharing of Federal Tax 

Information among these statistical agencies. The legislation required to enable 

data synchronization should be acted on immediately. Such legislation should 

encompass BEA and BLS, in addition to Census, in order to promote maximal 

quality improvements and efficiency gains. Additionally, the statistical agencies 

should work under the Confidential Information Protection and Statistical 

Efficiency Act (CIPSEA) to achieve the full benefits of data synchronization that 

is permitted under current law. All improvements in data synchronization must be 

achieved in a manner that is consistent with maintaining the confidentiality and 

integrity of the data. 

2282 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tle-duep

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0350

Comment on FR Doc # 2016-22002

Submitter Information

Name: James Elliott

Address:

Fairfield, IA, 52556

Email: james.lee.elliott1@gmail.com

General Comment

In 2009 the U.S. Sec.of Ed. said, "Hopefully, someday, we can track children from

preschool to highschool, highschool to college, and college to career. Furthermore in

January 2012, the U.S. Dept. of Ed., without congressional approval (imagine that),

changed the Family Education and Privacy Act to allow the transmission of student's

personally identifiable information to any governmental or private entity designated by

the department. This info includes not only academic performance, but disciplinary

history, family income, religious affiliation, health history, up to 400 data points without

parental consent! This should send a chill down the spine of every parent, grandparent,

guardian, church and Christian in this nation. This nation does not need the Fed. Dept. of

Ed., Common Core or out of control government meddling in places never intended to

be, as was written in this nation's Constitution, for this nation's prosperity and posterity.

The Promise of Evidence-Based Policymaking 2283
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tlf-54f8

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0351

Comment on FR Doc # 2016-22002

Submitter Information

Name: United States Parents in Education

General Comment

Commission on Evidence-Based Policymaking

Bureau of the Census, Department of Commerce

Washington D.C. 

December 14, 2016 

Re: Docket ID: USBC-2016-003

To Whom It May Concern: 

United States Parents Involved in Education (USPIE) is a nationwide grassroots

organization of education advocates and activists. We are formally reaching out to you in

opposition of efforts to integrate student data to make it easier to access. This letter is to

provide public comment representing thousands of grassroots activists across the country

regarding Student Data Privacy. 

Student Personally Identifiable Information (PII) as defined in federal FERPA includes

an overwhelming level "information, behavioral data, DNA samples, nicknames, bus

stop times, family history, academic history, fingerprints, blood samples, religious and

2284 The Promise of Evidence-Based Policymaking
political affiliation and more. 

We oppose the collection of this information without specific written consent of parents.

We oppose the organization of this data into easily sharable formats. And we oppose

efforts to make the data easily accessible. We are asking that this policy be dropped from

consideration. We are also asking that FERPA be strengthened to reduce the amount of

Student PII permitted to be collected and that all data collection regarding students be

authorized with written, signed permission of parents.

Please consider this from the Identity Theft Resource Center 2015 Report: "While still

only a small portion of the calls to the ITRCat around ten percent of the call volume in

Novemberinvolve child identity theft, 2014 was particularly alarming due to the 300%

increase in inquiries and reports over the course of the year. In fact, child identity theft

reports nearly doubled from January to February, and almost tripled from January to

November."

Moreover, during a recent hearing sponsored by CEP, a participant asked, "how could

there be more of a coordination or maybe a virtual one statistical agency where census is

playing a coordinating role?" 

"The CEP member's response was: "here's data with PII on it that's collected from SSA,

here's data with PII on it that's collected from the IRS: here's data with PII on it that's

collected from a state; versus from a statistical agency - if data with PII on it was treated

the same, you know I think that would permit, you know, organizations that were

collecting PII-laden data for different purposes to make those data available more

easily."

The inability to adequately protect computerized data cannot be argued. This past

election cycle clearly demonstrated the ease of unauthorized access to even secured data.

Our children's identity is at risk of being stolen before they even earn a driver's license.

Proposals to make Student PII even more easily accessible are irresponsible. 

USPIE is the nation's largest collaboration of grassroots, education advocates and

activists. We are committed to the goal of truly improving education for all of America's

children.

Sincerely,

U.S. Parents Involved in Education

www.uspie.org

Attachments

Student PII Comment

The Promise of Evidence-Based Policymaking 2285
Commission on Evidence-Based Policymaking 

Bureau of the Census, Department of Commerce 

Washington D.C. 

December 14, 2016 

Re: Docket ID: USBC-2016-003 

To Whom It May Concern: 

United States Parents Involved in Education (USPIE) is a nationwide grassroots organization of education 

advocates and activists. We are formally reaching out to you in opposition of efforts to integrate student data 

to make it easier to access. This letter is to provide public comment representing thousands of grassroots 

activists across the country regarding Student Data Privacy. 

Student Personally Identifiable Information (PII) as defined in federal FERPA includes an overwhelming level 

“information, behavioral data, DNA samples, nicknames, bus stop times, family history, academic history, 

fingerprints, blood samples, religious and political affiliation and more. 

We oppose the collection of this information without specific written consent of parents. We oppose the 

organization of this data into easily sharable formats. And we oppose efforts to make the data easily 

accessible. We are asking that this policy be dropped from consideration. We are also asking that FERPA be 

strengthened to reduce the amount of Student PII permitted to be collected and that all data collection 

regarding students be authorized with written, signed permission of parents. 

Please consider this from the Identity Theft Resource Center 2015 Report: “While still only a small portion of 

the calls to the ITRC—at around ten percent of the call volume in November—involve child identity theft, 2014 was 

particularly alarming due to the 300% increase in inquiries and reports over the course of the year. In fact, child 

identity theft reports nearly doubled from January to February, and almost tripled from January to November.”

Moreover, during a recent hearing sponsored by CEP, a participant asked, "how could there be more of a 

coordination or maybe a virtual one statistical agency where census is playing a coordinating role?” 

“The CEP member's response was: “here’s data with PII on it that’s collected from SSA, here’s data with PII on 

it that’s collected from the IRS: here’s data with PII on it that’s collected from a state; versus from a statistical 

agency – if data with PII on it was treated the same, you know I think that would permit, you know, organizations 

that were collecting PII-laden data for different purposes to make those data available more easily.”

The inability to adequately protect computerized data cannot be argued. This past election cycle clearly 

demonstrated the ease of unauthorized access to even secured data. Our children’s identity is at risk of being 

stolen before they even earn a driver’s license. Proposals to make Student PII even more easily accessible are 

irresponsible. 

USPIE is the nation’s largest collaboration of grassroots, education advocates and activists. We are committed 

to the goal of truly improving education for all of America's children. 

Sincerely, 

U.S. Parents Involved in Education 

www.uspie.org 

2286 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tlf-ztle

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0352

Comment on FR Doc # 2016-22002

Submitter Information

Name: Margaret Levenstein

Address:

Ann Arbor, 48109

Email: maggiel@umich.edu

General Comment

Please see attached file for comments. 

Submission from: Institute for Social Research, University of Michigan, P.O. Box 1248,

Ann Arbor, MI 48106-1248. Contact: Margaret Levenstein, Director, Inter-university

Consortium for Political and Social Research and Research Professor, Institute for Social

Research, MaggieL@umich.edu.

Attachments

Insitute for Social Research CEP RFC Submission December 14, 2016

The Promise of Evidence-Based Policymaking 2287
Commission on Evidence-Based Policymaking Comments

Docket Number: USBC-2016-0003

Response from: Institute for Social Research, University of Michigan, P.O. Box 1248, Ann 

Arbor, MI 48106-1248. Contact: Margaret Levenstein, Director, Inter-university Consortium for 

Political and Social Research and Research Professor, Institute for Social Research, 

MaggieL@umich.edu.

Introduction

The Commission on Evidence-Based Policymaking is examining strategies to expand access to 

and the use of government data for the purposes of building evidence-based evaluation of 

program and policy outcomes while concurrently protecting the privacy and confidentiality of the 

citizens and organizations studied. 

The University of Michigan’s Institute for Social Research (ISR) is pleased to present the 

comments below for consideration as the Commission deliberates these strategies. There are 

several general principles that we believe should guide the Commission as it examines 

opportunities to build and evaluate evidence-based programs and policies with administrative 

and survey data. These principles are woven throughout our comments to the Commission’s 

questions below and are summarized as follows:

Ɣ Confidentiality of individual data and the independence of the federal statistical system 

must remain paramount. Participation in our federal data programs, whether they collect 

survey or administrative data, is premised on the promise that individual data will remain 

confidential and will be used for statistical purposes only. It is never to be used for 

enforcement purposes or for the benefit of particular commercial or political interests. 

Confidence in the estimates produced by our federal statistical system requires 

adherence to these principles at all times. As articulated in the Office of Management 

and Budget’s Statistical Policy Directive No. 1 (2014), it is critical to “Protect the trust of 

information providers by ensuring the confidentiality and exclusive statistical use of their 

responses. Maintaining and enhancing the public’s trust in a Federal statistical agency’s 

or recognized statistical unit’s ability to protect the integrity of the information provided 

under a pledge of confidentiality is essential for the completeness and accuracy of 

statistical information as well as the efficiency and burden of its production.” This is just 

as true when administrative data is re-purposed for statistics. Undermining this trust 

undermines statistical measurement as well as the effectiveness of the programs upon 

which the statistics are based.

Ɣ Two important steps to uphold these principles and assure the independence and 

reliability of the estimates produced by our federal statistical system include the 

following:

a. Data originally generated outside the federal system, either from state and local, 

commercial, non-profit, social media, web-based, or other programs, should be 

aggregated outside the federal system. These data can be cleaned and 

documented, and secure, confidentiality-protecting crosswalks to PII can be 

2288 The Promise of Evidence-Based Policymaking
2

created. Data can then be transferred to the federal statistical system for 

matching to federal data resources. This will preserve respondent and data 

provider confidentiality and increase confidence in the security of the system. It 

also provides a mechanism for state and local civil servants and third-party (e.g., 

academic researchers) to access the non-federal data in a secure environment 

without burdening the federal research data system and its supporting security 

clearance mechanisms. This is the model that has been adopted by the Institute 

for Research on Innovation in Science (IRIS) in its collaboration with the Census 

Bureau’s Innovation Measurement Initiative. This model leverages the expertise 

of those outside the federal statistical system to improve, harmonize, and 

document the non-federal data. This kind of expertise is often lost or reduced 

when data are moved exclusively inside the federal statistical system. 

b. Data generated by federal agencies and programs will almost surely be legally 

required to stay within the federal firewall. These data should be made available 

systematically (promptly, with transparent access procedures, and where there is 

no or limited documentation, with a mechanism for researchers to contribute to 

the data and documentation). This will harness the energies and expertise of 

researchers to improve the data resources of the federal statistical system as 

well as state-of-the-art analyses of policies in order to assure that the inference 

drawn from evidence is scientifically sound. This also assures that there is 

competition in program analysis, so that multiple approaches can contribute to 

the analysis and program evaluation. The Federal Statistical Research Data 

Centers are an important mechanism for providing researcher access to these 

data, but, given the significant ongoing hurdles to their use, they should not be 

the exclusive mode for researcher access. 

Overarching Questions

1. Are there successful frameworks, policies, practices, and methods to overcome challenges 

related to evidence-building from state, local, and/or international governments the Commission 

should consider when developing findings and recommendations regarding Federal evidencebased policymaking? If so, please describe.

Yes, there are several. For example, the Institute for Research on Innovation and Science 

(IRIS), a collaboration of approximately 50 universities based at the Institute for Social Research 

(ISR) at the University of Michigan, provides an excellent model in which universities (mostly 

state, but also private) have voluntarily chosen to share confidential, proprietary data, including 

individual identifiers, with a federal statistical agency for the production of new estimates of the 

impact of national investments in research and development. IRIS has developed the capacity 

to ingest, harmonize, and de-identify transaction-level data from its member institutions. It uses 

these data to produce reports back to its members, restricted datasets available to the research 

community, and datasets that it transmits to the U.S. Census Bureau for linkage to Census data 

assets. The Census Bureau produces additional estimates and reports from these linked 

datasets, and makes the linked data available to qualified researchers with approved projects in 

The Promise of Evidence-Based Policymaking 2289
3

the network of Federal Statistical Research Data Centers (FSRDCs). This model has been able 

to achieve the participation of a large number of institutions, systematic access for the research 

community, and a more valuable research dataset than was the case for an earlier initiative 

strictly within the federal government. These data provide the basis for an evidence-based 

evaluation of a wide range of federal and private programs investing in science and academic 

R&D. IRIS’s model leverages the interest and abilities of the research community to analyze 

these data as well as leveraging the existing data resources of Census Bureau and the 

computing resources of the FSRDCs.

In another initiative, in this case between the Institute for Employment Research (IAB) of the 

German Federal Employment Agency (BA) and the University of Michigan’s ISR, now expanded 

to five other U.S. universities and locations in the UK and on the Continent, hundreds of 

researchers have contributed to the analysis of German labor market reforms through their 

access to restricted, linked survey and administrative data. These initiatives demonstrate both 

the feasibility and the value of academic-government collaborations in overcoming the 

challenges to creating appropriate data infrastructure and harnessing scientific expertise to 

analyze those data for evidence-based policy evaluation.

State and local governments produce large amounts of administrative data on programs that 

they implement, whether funded locally or by the federal government. State and local civil 

servants have important expertise and knowledge about the operations of these programs and, 

therefore, the meaning of the administrative data generated by them. They often lack the data or 

statistical scientific expertise or computing environments in which to analyze or link these 

resources. Partnerships between state and local governments, federal governments, and 

academic institutions can provide the relevant training while developing data resources of value 

to all parties for evidence-based program evaluation.

2. Based on identified best practices and existing examples, what factors should be considered 

in reasonably ensuring the security and privacy of administrative and survey data?

We recommend two complementary approaches to providing security and assuring privacy. 

One is to take steps to assure that the data are safe. This can be achieved through traditional 

anonymization methods (aggregation, suppression, swapping, and noise infusion), data 

encryption, and the creation of fully or partially synthetic data. The other is to take steps to to 

assure that the researcher who analyzes the data is safe and is working in a safe computing 

environment. This is achieved through researcher training and credentialing, scientific peer 

review and pre-registration of research proposals, and the use of secure computing 

environments, such as virtual or physical research data enclaves. These two approaches are 

complementary. Many tasks associated with the work of turning administrative datasets into 

useful analytical datasets, including data cleaning, the production of metadata, and dataset 

linkages, can only be accomplished with access to identifiable data. This then requires secure 

computing and a system for training and vetting researchers. The UK’s Administrative Data 

Research Service has made steps in the direction of researcher training and credentialing. The 

European Union’s Data Without Boundaries project envisioned a researcher “passport” to 

2290 The Promise of Evidence-Based Policymaking
4

facilitate credentialed access across the European statistical agencies. The Sloan Foundation 

has recently supported the Inter-university Consortium for Social and Political Research 

(ICPSR) to build on these earlier projects to establish durable researcher credentials for access 

to confidential data.

Data Infrastructure and Access

3. Based on identified best practices and existing examples, how should existing government 

data infrastructure be modified to best facilitate use of and access to administrative and survey 

data?

Exploiting the potential power of administrative and survey data for evidence-based policy and 

program evaluation requires that both government and non-government analysts are able to 

discover appropriate data resources and gain access necessary to analyze these data 

effectively. To address these needs, an infrastructure characterized by rich metadata about 

administrative and survey data sources, a secure platform for researchers to analyze datasets 

held in other locations, and a standardized and broadly accepted system of researcher 

credentialing must be developed. The existing network of Federal Statistical Research Data 

Centers provides an important mechanism for non-governmental researchers to contribute their 

expertise to the challenges faced by the federal statistical system and to the evaluation of 

programs. 

Ɣ The adoption of standardized researcher credentialing, accepted by multiple federal 

statistical agencies, similar agencies in other countries (such as the German IAB and UK 

Data), and non-governmental providers of confidential data to the research community 

(ISR, NORC, RTI) can reduce barriers to accessing data by enabling qualified 

researchers to analyze data through a modality that is appropriate to their level of datasecurity training and experience. It also assures that access is obtained for legitimate 

research purposes on an equitable basis.

Ɣ In order to make data useful, and the research arising from it replicable, investments 

should be made in data documentation via well-defined metadata fields, and 

infrastructure should be built that enable researchers to locate and analyze datasets 

held in multiple, distinct, secure locations. Community curation, provided by researchers 

who are invested in understanding the data and enabled with appropriate software, can 

assist in building this documentation.

4. What data-sharing infrastructure should be used to facilitate data merging, linking, and 

access for research, evaluation, and analysis purposes?

The characteristics of a data-sharing infrastructure designed to increase the availability and use 

of government data for evidenced-based program evaluation include:

Ɣ Robust search and browse capabilities that leverage standardized metadata, permitting 

researchers to discover data and learn about data in depth

The Promise of Evidence-Based Policymaking 2291
5

Ɣ Capacity to facilitate crowdsourcing (active curation) and improvement of metadata to 

capture and leverage newly acquired knowledge about the data

Ɣ Capacity to recognize varying levels of credentials assigned to a researcher ID

Ɣ Functionality that enables researchers to analyze datasets held in multiple, distinct, 

secure locations, that is, a computing backbone that can support secure, multi-party 

computing.

5. What challenges currently exist in linking state and local data to federal data? Are there 

successful instances where these challenges have been addressed?

Two white papers prepared for the Commission on Evidence-Based Policymaking by the Office 

of Management and Budget, “Using Administrative and Survey Data to Build Evidence” and 

“Barriers to Using Administrative Data for Evidence Building,” explicitly point to the key 

challenges. These include statutory prohibitions that hinder access to the data; policy and legal 

interpretations, which can vary across agencies and federal, state, and local governments; and 

resource and capacity constraints, specifically the lack of appropriate and reliable infrastructure 

to address data sharing and access, management and curation of data, and security and 

privacy concerns.

The Longitudinal Employer-Household Dynamics/Local Employment Dynamics data program is 

our best example of such a collaboration. LEHD highlights both the enormous potential and the 

enormous challenges to creating and making use of linked state and federal data. This 

collaboration has made possible very important new data assets which have revolutionized our 

understanding of local and internal labor markets, job creation and destruction, and job mobility 

for workers in different industries, cohorts, and demographic groups. These valuable data 

remain underutilized because of limitations to access. Providing resources to strengthen state 

and local government statistical capacity would allow those agencies and their civil servants to 

participate more effectively in research using these data. Increased capacity within state and 

local governments would allow these agencies and civil servants to benefit from collaboration 

with external researchers and reduce their incentives to impede research.

6. Should a single or multiple clearinghouse(s) for administrative and survey data be established 

to improve evidence-based policymaking? What benefits or limitations are likely to be 

encountered in either approach?

There are benefits and limitations to both the single- and multiple-clearinghouse approaches.

Overall we endorse a principle of union catalogs, so that data can be discovered and compared. 

A single clearinghouse would facilitate the process of finding and gaining access to the data and 

potentially linking multiple datasets. A clearinghouse would also act as a single point of entry for 

an analyst searching for appropriate data with which to address his/her question, and one might 

expect that a single catalog would have the benefit of consistent metadata to assist the 

researcher in evaluating the options and identifying the most useful source of data. Having a 

single clearinghouse to more efficient linking of datasets, for example if the clearinghouse 

2292 The Promise of Evidence-Based Policymaking
6

functioned as a trusted third party and provided de-identified, linked data to researchers. A 

single clearinghouse, using appropriate software to track dataset versions would also increase 

reproducibility of analyses by make it easier for researchers and policymakers to identify a 

specific instance of a dataset. Given that administrative datasets are updated regularly as new 

data are generated versioning of data is particularly important for rigorous and reproducible 

analysis. The most important benefit to a single clearinghouse is that it would reduce the 

bureaucratic hurdles to analyses that required to access multiple datasets; on the other hand, 

these hurdles creates checks and balances and privacy protection that can be undermined by 

centralization.

Multiple clearinghouses, however, would allow for specialization and expertise around particular

data sources and/or types (which lends itself to strong user support as well) and the flexibility to 

respond more efficiently to changes in formats or uses of data in a particular domain. One of the 

challenges to using administrative data for research and analysis is the lack of accompanying 

documentation about the fields in the dataset. A series of specialized clearinghouses could 

begin to address this because domain-speficic staff expertise could, over time, be used to 

create such documentation -- for example, noting when the underlying meaning of a particular 

field has changed or even simply pointing out that distributions on key variables changed at a 

specific point in time so that the researcher could do the detective work necessary to figure out 

why. Having multiple clearinghouses also spreads and develops the capacity necessary in both 

person-power (tagging, data checking, user support) and hardware/software for storing and 

disseminating the data across multiple organizations. This decentralization provides robustness 

to the infrastructure while increasing privacy protections. 

An efficient and privacy-protecting solution would be to have integrated data catalogs and 

multiple clearinghouses, but secure, multi-party computing across clearinghouses and common 

standards to gain access to data, including:

Ɣ common researcher credentials 

Ɣ peer review and pre-registration of research project proposals

Ɣ data use agreements

Ɣ required metadata fields

7. What data should be included in a potential U.S. government data clearinghouse(s)? What 

are the current legal or administrative barriers to including such data in a clearinghouse or 

linking the data?

8. What factors or strategies should the Commission consider for how a clearinghouse(s) could 

be self-funded? What successful examples exist for self-financing related to similar purposes?

High quality data requires investment in curation. High quality analyses require investment in 

training researchers and civil servants and providing them with up-to-date computing facilities. 

Democracy requires that the data be well-protected. All of these require resources that have to 

be provided by someone. 

The Promise of Evidence-Based Policymaking 2293
7

One model for self-sustaining data access is the consortium model. Most relevant to the types 

of data discussed here is the Institute for Research on Innovation and Science (IRIS), a project 

based at the University of Michigan’s Institute for Social Research. Started with funding from 

external sources, IRIS’s model was to become self-funded by charging institutions (colleges and 

universities) an annual membership fee. This fee provides the member institution with campuslevel reports based on their data, a seat at the table to help prioritize and design future IRIS

products and initiatives, and access to de-identified and aggregate IRIS data for researchers on 

their campus. One benefit to universities is that, although they are required to deposit data 

about their campuses with IRIS annually, the IRIS staff has automated the ability to produce 

charts and reports based upon those data (possible because the same information fields are 

collected from each institution). An organization or government agency that is required to share 

data and/or provide reports based on those data can find that it is in their interest to pay those 

with the skills and resources to properly support the data sharing efforts to carry out those tasks 

rather than reinventing the wheel and building that capacity within each agency. The data center

providing a service such as creating reports or demonstrating use of the data within the 

research community is seen by the data producer as an added benefit. Similarly, the Interuniversity Consortium for Political and Social Research (ICPSR) began as a consortium of 22 

institutions in 1962 and continues the model with over 760 member institutions today. These 

institutions pay an annual membership fee in exchange for access to data curated (and tools 

created) using member funding as well as reduced tuition for students enrolling in the ICPSR 

Summer Program in Quantitative Methods of Social Research. 

It is important to remember, however, that there is no free lunch; if we want better data and 

better analysis than currently exists, resources will have to be obtained to support this. The 

value of this research may well provide the basis for self-financing, but it is more likely that such 

research creates positive externalities without the ability to generate much revenue to support it.

9. What specific administrative or legal barriers currently exist for accessing survey and 

administrative data?

10. How should the Commission define “qualified researchers and institutions?” To what extent 

should administrative and survey data held by government agencies be made available to 

“qualified researchers and institutions?”

It is critical that administrative and survey data held by government agencies be made available 

to qualified researchers and institutions for scientific research and evidence-based analysis. 

Such access provides the policy community with much greater resources for informing policy 

decisions than if we rely exclusively on government analysis. It also increases the likelihood 

that there is diversity in the analytical approaches brought to bear on important policy questions.

“Qualified researchers and institutions” should be established through external scientific peer 

review of proposals and a system of researcher credentialing that creates an incentive for 

researchers to be good data stewards. Because of the current lack of consistency across 

agencies in defining these terms, ICPSR is undertaking a project to research, propose, and test 

2294 The Promise of Evidence-Based Policymaking
8

recommendations for researcher credentialing, the result of which will be a tiered set of 

characteristics that describe “qualified researchers and institutions.” These characteristics will 

stem from those currently employed/accepted by providers of restricted data, in so far as those 

requirements are related to protecting against disclosure risk (i.e., not requirements put into 

place to add bureaucracy or additional “hoops” that must be jumped for data access). We 

anticipate using factors such as whether one has completed requisite training in ethical data 

use, is employed at an accredited academic institution, has secured federal funding, and 

proposes a project that is scientifically sound and that requires access to the data in question.

The ability to disseminate data using a variety of modes (providing metadata only, synthetic 

data, use restricted to a physical or virtual enclave, or encrypted download) also allows for 

flexibility in determining access. That is, if a researcher does not have accepted credentials, or 

is not affiliated with an institution with appropriate technical and legal protections for data, a

researcher might still be allowed access to de-identified data. More sensitive data can be 

restricted to access in a virtual or physical data enclave. In other words, the same data may be 

made available to different researchers under different access modalities based on the 

characteristics of the researcher and the sponsoring institution. 

11. How might integration of administrative and survey data in a clearinghouse affect the risk of 

unintentional or unauthorized access or release of personally-identifiable information, 

confidential business information, or other identifiable records? How can identifiable information 

be best protected to ensure the privacy and confidentiality of individual or business data in a 

clearinghouse?

The integration (linking) of administrative and/or survey data in a clearinghouse without question 

increases the risk of disclosure of entities within the data; however, the federal statistical 

community and the research data community have a long history and reputation for protecting 

confidentiality. This reputation must be maintained and protected by adhering to the Office of 

Management and Budget’s Directive 1 (2014). Policies to maintain these protections and the 

reputation of and confidence in the statistical agencies of the United States include: 

Ɣ Providing access only to credentialed analysts with well-articulated research plans and 

objectives

Ɣ Provision by the clearinghouse of disclosure review of output, notes and other materials 

that are to be taken out of the clearinghouse (secure environment) to prevent unintended 

disclosure of subjects within the dataset(s)

Ɣ Developing and implementing privacy preserving analytical techniques as well as 

disclosure avoidance techniques such as creating synthetic populations that preserves 

statistical information

12. If a clearinghouse were created, what types of restrictions should be placed on the uses of 

data in the clearinghouse by “qualified researchers and institutions?”

The Promise of Evidence-Based Policymaking 2295
9

Clearinghouses can and should require researchers or analysts to submit a detailed proposal of 

the project for which the data are to be used, specifically addressing why the dataset in question 

is necessary for addressing the research question. Once these are vetted, by scientific peer 

review, clearinghouse staff and perhaps an external review board representing the data 

producer and the study population, a conclusion can be drawn on whether the benefits of the 

research project outweigh potential risks. Other restrictions should be consistent with the factors 

listed above -- explicitly agreeing to use the data in an ethical manner (and potentially 

demonstrating completion of training in doing so), restrictions on the computing environment in 

which the data can be analyzed, agreeing to terms of use, and the like. 

13. What technological solutions from government or the private sector are relevant for 

facilitating data sharing and management?

There are a number of private and governmental organizations that offer technological options 

for data sharing and management. Colectica, a Minneapolis-based firm, is an example of a 

research and development firm specializing in data management, integration services, and 

Internet technologies for government, academic, and commercial computing; it offers a range of 

highly specific products and services useful for supporting data sharing and management. They 

offer tools for working with metadata using a variety of documenting standards (e.g., the Data 

Documentation Initiative, DDI,). Colectica also has a portal that offers search, browse, 

visualization, and data management capabilities. 

Other projects exist that could offer either the technology or the functionality considerations that 

would be helpful. One such project is the Sustainable Environment/Actionable Data (SEAD) 

project, funded by the National Science Foundation and based at the University of Michigan. 

SEAD provides a collaborative platform for researchers to curate their data as they undertake 

analyses, so that the documentation is created and captured and can be harvested when the 

data are shared (i.e., in a clearinghouse). A number of organizations, such as ICPSR, NORC, 

and the Michigan Center for the Demography of Aging, use technology to create virtual spaces

in which researchers can analyze data that have significant disclosure issues. Generally, these 

spaces require the researcher to log in to a server housed at the data provider, conduct their 

analyses, and have output vetted before it is released to them. These virtual data enclaves often 

disable connections to the internet, print functionality, email, and other programs/features that 

might compromise data security. Lastly, software (e.g., Fedora) exists for creating and 

managing digital repositories and could be employed by a clearinghouse. 

14. What incentives may best facilitate interagency sharing of information to improve 

programmatic effectiveness and enhance data accuracy and comprehensiveness?

Resources to train civil servants in state, local, and federal agencies to evaluate their own data 

will also increase their capacity to learn from and absorb the analyses done by others. The 

implementation of multiple randomized control trials could also reduce the inclination to limit 

data sharing, as analyses can examine the question of which policies or interventions should be 

supported at scale (not simply whether an individual policy or program should be continued). 

2296 The Promise of Evidence-Based Policymaking
10

Building continuous evaluation and improvement, based on progress toward measurable 

objectives for the relevant population, into policy design provides programs with incentives to 

collect and analyze data in order to identify potential improvements.

We should also work to develop a culture that highlights the intrinsic benefit that most civil

servants, researchers, and the general population receive from having better answer questions 

about program effectiveness and other social issues. Researchers and civil servants will then be 

more likely to suggest improvements to data collection (methods and/or content) that would 

provide more effective analytic data to use in program evaluation. Researchers who are 

analyzing data are also likely to catch anomalies or potential inaccuracies that might be missed 

without researcher engagement. Having multiple researchers with multiple perspectives 

working with the same data will support models that might be more robust than if a single party 

were solely responsible for producing the evaluations. The research community’s embrace of 

data transparency and replicability may provide reinforcement to governmental agencies to 

adhere to similar principles. Sharing data among agencies and with researchers increases the 

return on investments in data creation. It is more efficient use of government resources. It is

rarely the case that a single researcher or organization can study everything that can be 

examined using a given data source. Differences in disciplinary perspectives mean that data 

collected for one purpose might be seen by another investigator as having value for his/her 

project that is completely different. Our statistical agencies employ dedicated civil servants who 

value improvements in the quality of measurement that they produce for our country. 

Recognition and respect for these values and these individuals will enable them to be more 

effective and take the steps necessary to continuously improve our data infrastructure.

Data Use in Program Design, Management, Research, Evaluation, and Analysis

15. What barriers currently exist for using survey and administrative data to support program 

management and/or evaluation activities?

There are currently significant barriers in accessing and using such data, including challenges in 

discovering the existence and location of appropriate data, uncertainty about legal infrastructure 

and processes for providing access to data, lack of documentation of file contents or data 

provenance. There are also limited resources for analyzing data (e.g., appropriate training for 

government employees and non-governmental researchers, appropriate computing 

infrastructure). 

16. How can data, statistics, results of research, and findings from evaluation, be best used to 

improve policies and programs?

Simply using such information is the first step. Organizations may collect data from program 

participants but not use it in evaluating the effectiveness of the program, may create summary 

statistics based on the data but not move further into more sophisticated statistical models, 

and/or may not be aware of existing research that could inform program/policy decisions. The 

ability to link data sources provides an opportunity to put data about program participation into 

The Promise of Evidence-Based Policymaking 2297
11

context in ways that have not been possible before. For example, having information about 

students’ performance for a given school by itself is helpful, but having the ability to link the 

information to such things as parental earning records and teacher characteristics allows an 

educational policy analyst to determine which shifts in student performance are likely a result of 

new policies implemented at the school, characteristics of the school or teachers themselves, or 

other issues related to outside influences such as food insecurity. Comparing data across 

similar contexts or programs is helpful in that the similarities and differences between the 

contexts create quasi-experimental designs, allowing researchers to identify the parts of the 

program that are most effective and those where improvement might be needed. Making data 

available to researchers also provides an avenue for dialogue between academics and 

policymakers that otherwise might not exist. 

17. To what extent can or should program and policy evaluation be addressed in program 

designs?

Program and policy evaluation should be included in program design so that evaluation is based 

on evidence that is available and analyzable by multiple, even competing, research teams, held 

to standards of reproducibility so that all parties can learn from evidence as it accrues in the 

process of program implementation. 

2298 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tlf-nmf8

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0353

Comment on FR Doc # 2016-22002

Submitter Information

Name: Amelia Vance

Address:

Washington, DC, 20005

Email: avance@fpf.org

General Comment

Dear Chairperson Abraham, Co-Chair Hastings, and Commissioners:

On behalf of the Future of Privacy Forum (FPF), thank you for the opportunity to

provide comments in response to the Commission on Evidence-Based Policymaking's

request for comments. FPF is a non-profit organization that serves as a catalyst for

privacy leadership and scholarship, advancing principled data practices in support of

emerging technologies.

The attached comments address question two as it relates to federal education data in the

request for comments in the Federal Register: "Based on identified best practices and

existing examples, what factors should be considered in reasonably ensuring the security

and privacy of administrative and survey data?"

Thank you for your time. Please let us know if we can answer any questions.

Amelia Vance

The Promise of Evidence-Based Policymaking 2299
Policy Counsel, Education

Future of Privacy Forum

Attachments

FPF Comments to Evidence-Based Policymaking Commission

2300 The Promise of Evidence-Based Policymaking
1 

Comments from 

THE FUTURE OF PRIVACY FORUM 

to 

U.S. Department of Commerce, Bureau of the Census 

Commission on Evidence-Based Policymaking 

 

Docket No. USBC-2016-0003

Request for Comment Regarding the “Commission on Evidence-Based Policymaking”

Amelia Vance, Policy Counsel, Education 

John Verdi, Vice President of Policy 

THE FUTURE OF PRIVACY FORUM December 14, 2016 

1400 I St. NW Ste. 450 

Washington, DC 20005 

www.fpf.org 

The Promise of Evidence-Based Policymaking 2301
2 

Dear Chairperson Abraham, Co-Chair Hastings, and Commissioners: 

On behalf of the Future of Privacy Forum (FPF), thank you for the opportunity to provide 

comments in response to the Commission on Evidence-Based Policymaking’s request for 

comments. FPF is a non-profit organization that serves as a catalyst for privacy leadership and 

scholarship, advancing principled data practices in support of emerging technologies. 

The information below addresses question two as it relates to federal education data in the 

request for comments in the Federal Register: “Based on identified best practices and existing 

examples, what factors should be considered in reasonably ensuring the security and privacy of 

administrative and survey data?”

FPF has been working on student privacy for almost three years. In that time, among other 

accomplishments, we jointly created the Student Privacy Pledge with the Software and 

Information Industry Association (a voluntary and legally binding promise by more than 300 ed 

tech companies to date regarding the handling of student data that was endorsed by President 

Obama); have read or provided comments on the more than 400 student privacy bills introduced 

in 49 states since 2014 and the eight federal bills introduced in 2015; written numerous reports 

on student privacy, including a guide for deidentifying student information under FERPA; and 

created FERPA|Sherpa, a website compiling education privacy resources and tools with sections 

aimed at parents, schools, service providers, and policymakers. In particular, over the past year 

we have provided higher education privacy guidance and input to the PostSecondary Data 

Collaborative, a group working on, among other priorities, efforts to improve the data available 

to researchers seeking to better understand how to improve higher education. 

As many other organizations have addressed in their comments, current federal higher education 

administrative data is generally not being used efficiently or effectively. Despite the fact that the 

federal government collects student-level data on demographics, income, assets, and educational 

attainment as well as federal aid amounts and history through the FSA National Student Loan 

Data System and Central Processing System, students are not given the critical information they 

need to know about how students with similar characteristics perform at certain institutions and 

the expected employment value relative to the educational cost. In addition to the FSA data 

systems, the National Center for Education Statistics collects data on an institutional level with 

information about graduation rates and pricing, but this information does not include transfer and 

part-time students, who comprise a significant percentage of the student population. In addition 

to helping to use currently collected information more efficiently, a more comprehensive 

student-level system would enable a range of additional benefits for students, institutions, and 

policymakers.1

Of course, any such database or additional collection of data would have to be paired with stateof-the-art security and address many of the risks and concerns that have been raised in previous 

debates about SURDS in 2008 and in other comments filed before the Commission. Below, we 

discuss a few of the primary concerns. 

 

1 We highly recommend Building a Student-Level Data System, a recent report by Ben Miller from the Center for American 

Progress, for a highly detailed overview of the benefits of a Student Unit Record Data System. 

2302 The Promise of Evidence-Based Policymaking
3 

Concern: The Federal Government Should Not Collect Such Sensitive 

Data About Students On An Individual Level 

Many of the comments to the Commission expressed that the federal government should not 

collect sensitive preK-12 student data such as student disciplinary records, disabilities, socialemotional information, and immigration status. FPF agrees that collecting such sensitive and 

potentially prejudicial information at the federal level is inadvisable and would raise major 

privacy concerns. The data that should be included in a student-level system should be primarily 

focused on postsecondary students. The data sets that would be useful and appropriate for a more 

effectively linked system of student records are presented below, as described in a recent paper 

published by IHEP.2

The privacy “best practice” of data minimization requires that a system only collect data that is 

needed to address pre-determined goals or answer specific questions. The Commission could 

recommend that any new federal student-level data system expressly define what data elements 

may be collected, and propose including a governance process, such as regulatory rulemaking 

with a public comment period, before new elements are added. In order to assuage fears that this 

database would include pre-K-12 sensitive data or even that the system would include such data 

on postsecondary students, the Commission could also recommend that there be a statutory 

provision that enumerates certain data that should never be shared with the federal student-level 

data system and/or limits what data what can be collected on pre-K-12 students. 

Concern: A New Federal Student-Level System Would Create New 

Security Risks For Breaches or Malicious Attacks 

Breaches or unintentional disclosures of data are always a concern, as the publicized breaches of 

sensitive information both inside government (such as the OPM breach) or outside government 

(such as the Target or Sony breaches) have shown. However, there are many ways to manage 

and lower the risk. While our comments below are focused on the privacy guidance necessary to 

avoid inadvertent data loss, we also assume that others will ensure the appropriate security 

safeguards are in place, including both the necessary technical standards and accompanying 

policy protocols. 

First, as discussed above, limitation of what data is collected is essential: data that is not 

collected cannot be breached. Data should only be collected when there is a direct need for that 

data. In addition, each decision about whether to collect a data element should weigh the 

importance of collecting that data element against the damage a breach of that data element 

could cause. For example, many state educational systems have stopped collecting social security 

numbers because of their low value to data use and research, but relatively higher value to 

hackers or other bad actors. 

 

2 Ben Miller, Building a Student-Level Data System, Envisioning the National PostSecondary Data Infrastructure in the 21st

Century, May 2016, page 17, http://www.ihep.org/sites/default/files/uploads/postsecdata/docs/resources/building_a_studentlevel_data_system.pdf. 

The Promise of Evidence-Based Policymaking 2303
4 

Second, limiting who has access to the data, ensuring that those who have access to the data are 

trained in how to protect it, and that they are incentivized to keep it safe are practices that can 

significantly curtail the likelihood of unintended data disclosures. A table from the recent IHEP 

paper, shown below, suggests a role-based access framework that would help protect the privacy 

and security of that information.3

The Commission could recommend that there be strict privacy and security protocols for those 

with access to the student-level data, whether government employees, or researchers. One 

possible model is that of the Internal Revenue Service (IRS), which requires, among other 

protections, that researchers submit formal proposals; physically go to IRS offices to access any 

data; and obtain approval before publishing reports discussing the data. 

Concern: Creating a Student-Level System for Valid Educational Uses 

Could Allow for Future Negative Uses Not Initially Intended

One of the most prevalent concerns in other comments addresses the possibility of this data 

being used for purposes not initially intended, such as for homeland security or law enforcement. 

This is a valid concern; once information is gathered, if adequate legal and procedural safeguards 

are not put in place, that information, such as whether a student is in default on their loans or 

which credits a student attempted but did not receive, could be “repurposed” inappropriately.

Therefore, it is essential to directly limit the purposes for which the information could be used 

and who can access the data. As mentioned above, best practices for data privacy include use 

limitation and role-based access. The Commission could strongly recommend that any federal 

collection of education data have specific use limitations attached in law or regulation that detail 

how a student's personal information may be used, as well as clear restrictions on what is not 

 

3 Miller, page 20. 

2304 The Promise of Evidence-Based Policymaking
5 

allowed, particularly focusing on limits for any data use that extends beyond clearly specified 

educational purposes. In addition, the Commission could recommend that information that is 

made available to external researchers be reliably de-identified. 

 

As discussed above, any proposals to more comprehensively link higher ed student data should 

include limitations on the type of data that could be collected to make it less likely that it could 

be mis-used. If the system does not include, for example, sensitive data such as disciplinary or 

immigration information, law enforcement would likely not be incentivized to seek access to the 

student-level system. 

It is also noteworthy that much of the information that would be in a student-level data systems 

is currently collected and held by the federal government, and the restrictions on sharing that 

data have been sufficient to prevent inappropriate repurposing of the information. If there are 

data elements currently collected that may be inappropriate to continue collecting or to collect in 

the first place, proposals can be crafted to address those specific concerns. However, the only 

effect of this data not being easily linked for access by researchers is that beneficial uses of the 

data on behalf of students is hampered. It would be much better if the existing data the 

government holds was being used to more proactively help students. 

These suggestions only scratch the surface on the potential privacy and security protections that 

could be suggested by the Commission in their recommendations. FPF staff would welcome the 

opportunity to further discuss these issues and provide additional details or action steps on any of 

these recommendations. 

Sincerely, 

Amelia Vance John Verdi

Policy Counsel, Education Vice President of Policy

Future of Privacy Forum Future of Privacy Forum

The Promise of Evidence-Based Policymaking 2305
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tlf-9kww

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0354

Comment on FR Doc # 2016-22002

Submitter Information

Name: Daniel Castro

Address:

Washington, DC, 20005

Email: dcastro@datainnovation.org

General Comment

See attached comments submitted by the Center for Data Innovation, a non-profit, nonpartisan think tank studying the intersection of data, technology, and public policy.

Attachments

2016-evidence-based-policymaking

2306 The Promise of Evidence-Based Policymaking
 

 datainnovation.org 

December 14, 2016 

The Commission on Evidence-Based Policymaking 

Docket ID USBC-2016-000 

Re: Comments to the Commission on Evidence-Based Policymaking 

To Whom It May Concern: 

On behalf of the Center for Data Innovation (datainnovation.org), we are pleased to submit these 

comments in response to a request for comments from the Commission on Evidence-Based 

Policymaking, a bipartisan commission created by Congress to examine how to increase the 

availability and use of government data to build evidence and inform program design while 

protecting the confidentiality of this data.1

 

The Center for Data Innovation is the leading think tank studying the intersection of data, 

technology, and public policy. With staff in Washington, DC, and Brussels, the Center formulates 

and promotes pragmatic public policies designed to maximize the benefits of data-driven 

innovation in the public and private sectors. It educates policymakers and the public about the 

opportunities and challenges associated with data, as well as technology trends such as 

predictive analytics, open data, cloud computing, and the Internet of Things. The Center is a 

nonprofit, nonpartisan research institute affiliated with the Information Technology and 

Innovation Foundation. 

Over the past several years, the federal government has made substantial progress on making its 

data freely available to the public in open, usable formats, and this data has proven to be an 

invaluable resource for informed decision-making. Policymakers should expand on these efforts, 

as well as address the obstacles that remain to effective data sharing and use. 

 

 

1

 U.S. Census Bureau, “Request for Comments for the Commission on Evidence-Based Policymaking” 

September 14, 2016, https://www.regulations.gov/document?D=USBC-2016-0003-0001. 

The Promise of Evidence-Based Policymaking 2307
 

 datainnovation.org 

Please find our responses to the relevant questions in the attached document. 

Sincerely, 

Daniel Castro 

Director 

Center for Data Innovation 

dcastro@datainnovation.org 

Joshua New 

Policy Analyst 

Center for Data Innovation 

jnew@datainnovation.org 

2308 The Promise of Evidence-Based Policymaking
 

 datainnovation.org 

DATA INFRASTRUCTURE AND ACCESS 

BASED ON IDENTIFIED BEST PRACTICES AND EXISTING EXAMPLES, HOW SHOULD EXISTING 

GOVERNMENT DATA INFRASTRUCTURE BE MODIFIED TO BEST FACILITATE USE OF AND ACCESS 

TO ADMINISTRATIVE AND SURVEY DATA? 

Government data should be open and machine-readable by default, in accordance with official 

administration policy issued in May 2013.2

 This means that, unless otherwise legally prohibited, 

government data should use nonproprietary, machine-readable formats, and be licensed to 

maximize reuse, meaning the data is free for anyone to access, modify, and use for any purpose. 

Though agencies have made substantial progress towards publishing data as open and machinereadable by default, the requirements to do so are by executive order, rather than an act of 

Congress, meaning that they could be subject to change under a new presidential 

administration.3

 Regardless of whether these rules remain on the books, federal agencies should 

recognize the importance of open data to mission delivery and evidence-based decision-making 

and continue to treat their data as open by default indefinitely. 

Moreover, all federal agencies should produce and publish enterprise data inventories. Enterprise 

data inventories provide a list of all data assets managed by an agency—both public and nonpublic. By creating enterprise data inventories, federal agencies, policymakers, and other 

stakeholders are better able to discover important government data assets. 

WHAT DATA-SHARING INFRASTRUCTURE SHOULD BE USED TO FACILITATE DATA MERGING, 

LINKING, AND ACCESS FOR RESEARCH, EVALUATION, AND ANALYSIS PURPOSES? 

The federal open data portal Data.gov is a widely-used and effective tool for making open data 

from all levels of government easily accessible to members of the public and private sectors 

alike. Federal agencies should continue to publish their data on Data.gov, ensuring that they 

publish their data in open and machine-readable formats. 

In November 2016, the Department of Commerce partnered with public benefit corporation 

data.world to use data.world’s data science collaboration platform to increase the accessibility, 

 

2 Sylvia M. Burwell et al., “M-13-13,” Office of Management and Budget, May 9, 2013, 

http://www.whitehouse.gov/sites/default/files/omb/memoranda/2013/m-13-13.pdf. 3 Joshua New, “Will Obama be the Last Open Data President?” Center for Data Innovation, November 11, 

2014, https://www.datainnovation.org/2014/11/will-obama-be-the-last-open-data-president/. 

The Promise of Evidence-Based Policymaking 2309
 

 datainnovation.org 

linkability, and usability of its data.4 Federal agencies should pursue innovative partnerships 

such as these that can increase the utility of government data. 

WHAT CHALLENGES CURRENTLY EXIST IN LINKING STATE AND LOCAL DATA TO FEDERAL DATA? 

ARE THERE SUCCESSFUL INSTANCES WHERE THESE CHALLENGES HAVE BEEN ADDRESSED? 

Government data published in proprietary formats, data with restrictive licenses, non-machine 

readable data, data that is not made available free of charge, and data that governments fail to 

publish are all major challenges for linking data between different levels of government. Unless 

all levels of government are publishing open data, it can be very difficult for researchers, for 

example, to know what data is even available to them. And unless data is made available in a 

nonproprietary and machine-readable format and licensed to maximize reuse, these researchers 

would be technically and legally unable to link these datasets. In addition, many datasets are not 

available via application programming interfaces, or APIs. APIs allow developers to access 

datasets programmatically and expose datasets to new uses, especially on websites or mobile 

apps. 

As noted previously, another important challenge is that most federal agencies have failed to 

publish enterprise data inventories—a catalog of all of their datasets, both public and 

nonpublic.5

 Without publicly available enterprise data inventories, members of the public and 

private sector alike can be unable to locate relevant datasets or determine if a certain dataset 

even exists. 

SHOULD A SINGLE OR MULTIPLE CLEARINGHOUSE(S) FOR ADMINISTRATIVE AND SURVEY DATA 

BE ESTABLISHED TO IMPROVE EVIDENCE-BASED POLICYMAKING? WHAT BENEFITS OR 

LIMITATIONS ARE LIKELY TO BE ENCOUNTERED IN EITHER APPROACH? 

Data.gov is already a widely-used tool for discovering public administrative and survey data, so 

there is no need to establish similar tools for public government data. Creating multiple hubs for 

government data would raise costs for federal agencies, if they must submit data to multiple 

sites, and for users, if they must search multiple sites. Data.gov could also serve as an index for 

non-public datasets, but even if they were indexed, these datasets would still only be accessible 

 

4 Justin Antonipillai and Brett Hurt, “data.world to Bring Valuable Commerce Datasets to ‘Social Network 

for Data People,’” U.S. Department of Commerce, November 8, 2016, http://www.esa.doc.gov/undersecretary-blog/dataworld-bring-valuable-commerce-datasets-social-network-data-people. 5 Joshua New, “Congress is Stepping Up to Protect Open Data,” center for Data Innovation, April 19, 

2014, https://www.datainnovation.org/2016/04/congress-is-stepping-up-to-protect-open-data/. 

2310 The Promise of Evidence-Based Policymaking
 

 datainnovation.org 

via the security protocols in place to protect the data. Data.gov is not a repository for datasets, 

and creating a central repository for government data would have limited value because of the 

costs involved of transferring data from multiple systems. A better approach would be to focus on 

improving online identification and authentication technologies, so that federal agencies can 

more easily and securely make non-public datasets available to authorized users. So rather than 

creating a clearinghouse that is a repository of non-public government data, federal agencies 

should create a clearinghouse that is an index of available non-public data sources. 

WHAT FACTORS OR STRATEGIES SHOULD THE COMMISSION CONSIDER FOR HOW A 

CLEARINGHOUSE(S) COULD BE SELF-FUNDED? WHAT SUCCESSFUL EXAMPLES EXIST FOR SELFFINANCING RELATED TO SIMILAR PURPOSES? 

In general, the government should refrain from considering its data as an opportunity to generate 

revenue. Since the government collects data at taxpayers’ expense, taxpayers should be able to 

freely access this information. However, developing the infrastructure to provide large amounts of 

data to the public can be expensive, so in certain cases, a federal agency may consider charging 

access fees for certain data, akin to an entrance fee for a national park. However, such fees 

should solely support the development and upkeep of data infrastructure. 

One successful example of a government agency taking advantage of an innovative financing 

model to build data infrastructure is the National Oceanic and Atmospheric Administration’s 

(NOAA’s) Big Data Partnership. Established through a series of Cooperative Research and 

Development Agreements (CRADAs), the Big Data Partnership is designed to help NOAA publish 

the large amounts of data it collects—20 terabytes per day, on average—but cannot afford to 

make publicly available.6

 The Big Data Partnership partnered with major cloud providers, 

including Amazon Web Services and IBM, to build out NOAA’s data infrastructure at no cost to 

the government, so these partners could access more of NOAA’s data to build useful products 

and services.7

 Importantly, the partner companies do not get prioritized access to this data, as 

the partnership stipulates all of it must be open.8

 The partners are willing to finance the 

 

6 U.S. Department of Commerce, “U.S. Secretary of Commerce Penny Pritzker Announces New 

Collaboration to Unleash the Power of NOAA's Data,” News release, April 21, 2015, 

https://www.commerce.gov/news/press-releases/2015/04/us-secretary-commerce-penny-pritzkerannounces-new-collaboration-unleash. 7 Alexander Kostura and Daniel Castro, Three Types of Public-Private Partnerships That Enable Data 

innovation,” Center for Data Innovation, August 1, 2016, https://www.datainnovation.org/2016/08/threetypes-of-public-private-partnerships-that-enable-data-innovation/. 8 Ibid. 

The Promise of Evidence-Based Policymaking 2311
 

 datainnovation.org 

development of this infrastructure because the products they can build with this data, such as 

improved weather prediction services, will create a substantial enough return on investment. 

WHAT SPECIFIC ADMINISTRATIVE OR LEGAL BARRIERS CURRENTLY EXIST FOR ACCESSING 

SURVEY AND ADMINISTRATIVE DATA? 

There are two notable examples of administrative and legal barriers to accessing and using survey 

and administrative data. 

The federal government has multiple agencies collecting the same types of data for economic 

metrics. The Bureau of Labor Statistics (BLS), Census Bureau, and Bureau of Economic Analysis 

(BEA) all collect and analyze employment, occupation, income, and other labor market and 

economic data to produce valuable statistics like productivity growth, state and national GDP, 

and employment rates that help make policymakers make informed economic policy decisions.9

This leads to considerable statistical discrepancies between agencies. The issue is that neither of 

these metrics are necessarily wrong, but the differences in methods used and factors considered 

by each agency means the same question receives two largely different answers. The root cause 

of this is that Census can use data that BEA and BLS cannot because some of it is commingled 

with tax data, which cannot be shared under Title 26 of the Internal Revenue Code.10 As a result, 

these statistical agencies cannot rely on an internally consistent dataset. 

Another example of a specific barrier is that the Departments of Health and Human Services 

(HHS), Defense, Education and Justice oversee at least 10 different efforts to collect data about 

sexual violence—producing widely varying statistics that, on the surface, appear to measure the 

same thing.11 As a result, policymakers and the public simply do not have reliable, easy-tounderstand data about sexual assault, which can have serious consequences for the effectiveness 

and accountability of the criminal justice system and hinder efforts to combat sexual assault.12

The wide variation in reporting can make research about sexual violence unnecessarily difficult, 

 

9

 Luke Stewart, “We Have a Sharing Problem,” The Information technology and Innovation Foundation, 

June 13, 2012, http://www.innovationfiles.org/we-have-a-sharing-problem/. 10 National Research Council et al., Improving Business Statistics Through Interagency Data Sharing

(National Academies Press, 2006), 15. 

11 Government Accountability Office, Sexual Violence Data: Actions Needed to Improve Clarity and Address 

Differences Across Federal Data Collection Efforts (Washington, DC: 2016), 

http://www.gao.gov/assets/680/678511.pdf. 12 Joshua New, “How Common is Sexual Assault in the United States? The Answer Depends On Who You 

Ask,” Center for Data Innovation, September 1, 2016, https://www.datainnovation.org/2016/09/howcommon-is-sexual-assault-in-the-united-states-the-answer-depends-on-who-you-ask/. 

2312 The Promise of Evidence-Based Policymaking
 

 datainnovation.org 

as comparisons between different groups are all but impossible when methodologies differ so 

greatly, and the lack of clarity about what these different statistics really mean provides fodder 

for toxic arguments that dismiss the severity of the problem of sexual assault, such as those that 

imply that false reporting of rape is commonplace, which is false.13

HOW CAN IDENTIFIABLE INFORMATION BE BEST PROTECTED TO ENSURE THE PRIVACY AND 

CONFIDENTIALITY OF INDIVIDUAL OR BUSINESS DATA IN A CLEARINGHOUSE? 

There are a variety of tools and strategies useful for preserving the security and privacy of data. 

One of the most effective of these methods—de-identification—has been frequently criticized for 

being unreliable due to the perceived risk that bad actors could easily re-identify a data set.14

However, while no information security method, including de-identification, is perfect, the risk of 

reidentification when data is properly de-identified has been greatly exaggerated by some 

commentators, and it would be a mistake to reverse existing federal policy based on these 

inaccuracies.15 The notion that de-identification is an unreliable tool is commonly promulgated 

by commentators misinterpreting primary literature or failing to recognize that many instances of 

reidentification were only possible because the data was improperly de-identified in the first 

place.16 For example, if a data set is properly de-identified in accordance with the Safe Harbor 

Standard defined by the Health Insurance Portability and Accountability Act (HIPAA) which 

requires the removal or modification of 17 specific data elements, only 0.04 percent of 

individuals are “uniquely identifiable.”17 It is important to note the significant difference 

between uniquely identifying an individual, which means recognizing a specific, discrete set of 

characteristics within a data set, and re-identifying an individual, which means gleaning a 

person’s actual identity—his or her name, birth date, and so on—from a data set. 

 

13 Ibid; Ashe Schow, “No, 1 in 5 Women Have Not Been Raped on College Campuses,” Washington 

Examiner, August 13, 2014, http://www.washingtonexaminer.com/no-1-in-5-women-have-not-been-rapedon-college-campuses/article/2551980; Carmen Rios, “Rebel Girls: Our ‘False Rape’ Hysteria is Bullsh*t,” 

Autostraddle, December 31, 2014, https://www.autostraddle.com/rebel-girls-our-false-rape-hysteria-isbullshit-270299/; “False Reporting,” National Sexual Violence Resource Center, 2012, 

http://www.nsvrc.org/sites/default/files/Publications_NSVRC_Overview_False-Reporting.pdf. 14 Ann Cavoukian and Daniel Castro, “Big Data and Innovation, Setting the Record Straight: 

Deidentification Does Work,” Information Technology and Innovation Foundation, June 16, 2014, 

http://www2.itif.org/2014-big-data-deidentification.pdf. 15 Ibid. 

16 Ibid. 

17 Simon Cohn, “Ltr to Sec'y 2007 Dec 21 Secondary Uses,” National Committee on Vital and Health 

Statistics, December 21, 2007, http://www.ncvhs.hhs.gov/wp-content/uploads/2014/05/071221lt.pdf. 

The Promise of Evidence-Based Policymaking 2313
 

 datainnovation.org 

IF A CLEARINGHOUSE WERE CREATED, WHAT TYPES OF RESTRICTIONS SHOULD BE PLACED ON 

THE USES OF DATA IN THE CLEARINGHOUSE BY “QUALIFIED RESEARCHERS AND 

INSTITUTIONS?” 

In cases where data is inherently sensitive, even after de-identification, other methods should be 

used to protect data while also making it available to qualified researchers. For example, the 

Census Bureau operates Federal Statistical Research Data Centers (RDCs), which serve as a 

secure environment for sharing sensitive data with qualified researchers working on important, 

pre-approved projects.18 The Census Bureau collaborates with several statistical agencies to 

operate the RDCs, such as the Agency for Healthcare Research and Quality and BLS.19 While the 

focus of RDCs is statistical agencies, this model could be applied to other government agencies 

and government datasets with similar effectiveness. In addition, government agencies can 

require researchers to abide by licensing agreements for non-public data that imposes 

restrictions on how researchers can use the data. 

CONCLUSION 

Easy access to useable data plays a crucial role in informed-policymaking, and it is encouraging 

to see the Commission for Evidence-Based Policymaking working to increase the accessibility 

and usability of government data for this purpose. Overall, adhering to the principle that 

government data should be open and machine-readable by default would substantially improve 

the quantity and quality of data available for evidence-based policymaking. 

 

18 “Federal Statistical Research Data Centers,” U.S. Census Bureau, accessed December 14, 2016, 

http://www.census.gov/about/adrm/fsrdc/about/available_data.html. 19 Ibid. 

2314 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tlf-krql

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0355

Comment on FR Doc # 2016-22002

Submitter Information

Name: Amy Gines

General Comment

I echo these concerns:

I am opposed to allowing greater data access. Children's data needs to be properly

overseen and managed by their parents. Even in medicine, these children's data would

only be allowed to be used in research with the informed consent of their parent or

guardian. But in education, why do we feel that it is any less invasive? In fact, it has the

potential to pigeon-hole children from an early age and limit their future possibilities.

We live in an age where, for the first time in history, people who have never seen,

spoken with, or cared about a particular child, will be able to know vast amounts of

information about them and make decisions about them without the benefit of their

parents as gate-keepers. In the past, parents were the only source of the vast amounts of

information on their children. They could choose what information and to whom this

information should be shared. Now, they don't even know that it is being shared. How

can we possibly say that 'the best interests' of a child are paramount when this

information is being proffered without any sort of knowledge of the individual or a

motivation to maintain their privacy by those who know and care about them, their

parents and their teachers?

The Promise of Evidence-Based Policymaking 2315
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tlg-rswt

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0356

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

Gathering personal information for a workforce data system or anything else flies in the

face of our constitution. No sharing of personal data without informed consent.

2316 The Promise of Evidence-Based Policymaking
PUBLIC

SUBMISSION

As of: August 04, 2017

Received: December 14, 2016

Status: Posted

Posted: December 15, 2016

Tracking No. 1k0-8tlg-f53w

Comments Due: December 14, 2016

Submission Type: Web

Docket: USBC-2016-0003

Request for Comments for the Commission on Evidence-Based Policymaking

Comment On: USBC-2016-0003-0001

Request for Comments for the Commission on Evidence-Based Policymaking

Document: USBC-2016-0003-0357

Comment on FR Doc # 2016-22002

Submitter Information

Name: Anonymous Anonymous

General Comment

I strongly oppose any proposal that would lead to the creation of a centralized, federal

clearinghouse of the personally identifiable information of all students, commonly

referred to as a student unit-record system or national database.

The risk that such a federal database would pose to student privacy is immense;

including the very real possibility of breach, malicious attack, or the use of this

information for purposes not initially intended. Ever since a federal student unit-record

system was banned by the Higher Education Act in 2008, the reasons against creating it

have only become more compelling.

In the past few years, much highly personal data held by federal agencies has been

hacked, including the release of the records of the Office of Personnel Management

involving more than 22 million individuals, not only federal employees and contractors

but also their families and friends.

The US Department of Education has been found to have especially weak security

standards in its collection and storage of student data, and received a grade of D for its

security protections.

In addition, preK-12 student data currently collected by state departments of education

that would potentially be shared with the federal government include upwards of 700

highly sensitive personal data elements, including students' immigrant status, disabilities,

The Promise of Evidence-Based Policymaking 2317
disciplinary records, and homelessness data.

I am also very concerned about recent revelations of the widespread surveillance on

ordinary citizens by the federal government, and the way in which a national student data

system would be used to expand the tracking of students from PreK into high school,

college, the workforce and beyond. A federal data clearinghouse of student information

could effectively create life-long dossiers on nearly every individual in the nation.

I urge you to strongly oppose the creation of any centralized federal data system holding

students' personally identifiable information and to support the continuation of the ban in

the report you provide to Congress.

Sincerely, Dawn Tapp, NJ, concerned parent

2318 The Promise of Evidence-Based Policymaking
The Promise of Evidence-Based Policymaking 2319

During its fact-finding phase, the Commission on Evidence-Based Policymaking reviewed the findings and recommendations of 

several previous commissions, committees, and 

projects relevant to Federal evidence building. 

These commissions recommended ways to improve the country’s evidence-building system, including suggestions for greater coordination and 

enhanced privacy protection. The Commission 

noted that many of the issues identified by these 

past commissions are still challenges today. The 

commissions reviewed for this report include:

1. Commission Appointed by the Secretary of 

Treasury, 1903. 

In 1903, the Secretary of the Treasury created one of the first commissions to address 

consolidation of statistical functions in the 

Federal government. The commission intended to merge the Treasury’s Statistics of Income 

Division into the Department of Commerce, 

which already housed the Bureau of the Census 

and the Bureau of Labor Statistics. In her book, 

Organizing to Count, Janet Norwood states the 

reason this recommendation wasn’t implemented was because “concentration of the 

power to conduct statistical inquiries in one 

place in the government, it was feared, would 

diminish the effectiveness of other agencies 

and create a statistical bureaucracy that might 

become more powerful than several other existing agencies.”1

2. Report of the Task Force on the Storage of and 

Access to Government Statistics (the Kaysen 

Committee), 1966.2

1. Janet Norwood, Organizing to Count: Change in the Federal Statistical System (Washington, D.C.: Urban Institute Press, 1995): 9–10.

2. Carl Kaysen, Charles C. Holt, Richard Holton, George 

Kozmetsky, H. Russell Morrison, and Richard Ruggles, "Report of 

the Task Force on the Storage of and Access to Government Statistics," The American Statistician 23, no. 3 (1969): 11–19. http://www.

jstor.org/stable/2682574.

In Organizing to Count, Janet Norwood describes the Kaysen Committee and summarizes its goals and outcomes. She states, “In 1966 

the Bureau of the Budget established an independent commission to consider measures 

which should be taken to improve the storage 

of and access to U.S. Government statistics.”3

The Committee focused on the organization 

and operation of the Federal Statistical System. 

In their final report, Carl Kaysen and other 

members of the committee addressed how the 

decentralization and insufficient coordination 

of the Federal statistical system hindered its 

effective operation. The committee recognized 

the effort it would take to create a Central Statistical Agency, so it instead recommended two 

main ways to achieve the benefits of centralization and coordination, the creation of a new 

position, the Director of the Federal Statistical 

System, and of a National Data Center.4

3. Privacy and the National Data Bank Concept 

(House Committee on Government Operations), 1968.5

After the Kaysen committee was convened, 

Congressman Cornelius E. Gallagher, the Chairman of the Special Subcommittee on Invasion 

of Privacy of the House Committee on Government Operations, organized hearings seeking 

information on recent recommendations about 

a national data bank (or center). The subcommittee’s concerns stemmed not only from the 

Kaysen committee, but also an earlier 1965 report from the Social Science Research Council 

(Ruggles Report) and a report by Edgar Dunn, 

3. Janet Norwood, Organizing to Count: Change in the Federal Statistical System (Washington, D.C.: Urban Institute Press, 1995): 12.

4. Carl Kaysen, Charles C. Holt, Richard Holton, George 

Kozmetsky, H. Russell Morrison, and Richard Ruggles, "Report of 

the Task Force on the Storage of and Access to Government Statistics," The American Statistician 23, no. 3 (1969): 11–19. http://www.

jstor.org/stable/2682574.

5. U.S. House Committee on Government Operations, Privacy and 

the National Data Bank Concept, 90 H. Rpt. 35.

Appendix H: Prior Commissions 

Related to Evidence Building
2320 The Promise of Evidence-Based Policymaking

Jr, of the nonprofit think tank Resources for the 

Future. After the hearings, the subcommittee 

produced a study, Privacy and the National Data 

Bank Concept. The study concluded that the 

creation of such a data bank was feasible and 

advisable only if privacy were a top priority. 

The House Committee on Government Operations recommended that no work on the data 

bank be done until privacy protections were 

more fully explored and guaranteed.6

In 1969, the Kaysen committee published 

an addendum to their report in The American 

Statistician responding to the House report. 

Kaysen concluded that the threat to privacy 

be addressed by Congress, such as by creating 

a statute governing the disclosure of information.7

4. Federal Statistics: Report of the President’s 

Commission (the Wallis Commission), 1971.8

In August 1970 , the President charged the 

Wallis Commission with conducting a comprehensive review of the Federal statistical 

system. The President’s directive posed three 

questions: what are the requirements for quantitative information about society; how to minimize respondent burden; and how to organize 

Federal statistical activities for the most effective utilization of statistics. The commission 

recommended extending the coordinating and 

auditing responsibilities of the Office of Management and Budget’s (OMB) Statistical Policy 

Division (SPD). The commission also recommended that SPD work to increase coordination among the statistical agencies, including 

the promotion of legal safeguards. These points 

were combined in the recommendation to 

6. U.S. House Committee on Government Operations, Privacy and 

the National Data Bank Concept, 90 H. Rpt. 35.

7. Carl Kaysen, Charles C. Holt, Richard Holton, George 

Kozmetsky, H. Russell Morrison, and Richard Ruggles, "Report of 

the Task Force on the Storage of and Access to Government Statistics," The American Statistician 23, no. 3 (1969): 11–19. http://www.

jstor.org/stable/2682574.

8. Neil S. Weiner, "The Report of the President's Commission on 

Federal Statistics: A Summary of Recommendations, Views and 

Counterviews," The American Statistician 28, no. 2 (1974): 42–46. 

doi:10.2307/2683597.

establish an independent Statistics Advisory 

Board to review and report on the activities of 

Federal statistical agencies.9

The commission also emphasized that funds 

should be set aside specifically for evaluating 

new statistical programs. The recommendation 

specified that OMB’s role would be to review 

the data needed to evaluate six to eight domestic statistical programs. The recommendations 

called for finding a balance between the data 

needs of the public and the right to privacy of 

individuals. The Wallis Commission’s recommendations led to the creation of the Committee on National Statistics (CNSTAT) at the 

National Academies of Science, Engineering, 

and Medicine, which was envisioned by the 

commission as an advisor to OMB on evaluating statistical programs.10

5. Privacy Protection Study Commission, 1977.11

The Privacy Act of 1974 called for the creation of a Privacy Protection Study Commission (PPSC), which issued its final report in 

July of 1977. The PPSC’s report conveyed two 

central tenets for evidence building. First, research and statistical uses of data about individuals must exclude any result that would 

directly affect an individual’s rights, privileges, 

or benefits. Second, government statistical and 

nonstatistical uses of data should be separated 

by a bright line, a principle referred to as “functional separation.” These two core concepts 

have been carried forward into today’s Federal 

privacy protections and statistical laws.

6. Commission on Federal Paperwork, 1977.12

The Commission on Federal Paperwork was 

charged by Congress and the President with 

9. Janet Norwood, Organizing to Count: Change in the Federal Statistical System (Washington, D.C.: Urban Institute Press, 1995): 

12–14.

10. Ibid.

11. Privacy Protection Study Commission, Personal Privacy in an 

Information Society (Washington, D.C.: 1977), https://epic.org/privacy/ppsc1977report.

12. Statistics: A Report of the Commission on Federal Paperwork, 

(Washington, D.C.: 1977).
The Promise of Evidence-Based Policymaking 2321

the task of creating recommendations to eliminate needless paperwork while ensuring that 

the Government could still have the information necessary to meet the mandate of law 

and operate effectively. The commission developed about 800 recommendations addressing 

a broad range of government activities. The 

Commission concluded that paperwork reform 

required greater support of Congress and proposed that Congress provide for paperwork assessment to be included in legislative reports.13

In 1977, the commission released a report 

focused on the Federal statistical system. This 

was largely in response to President Carter’s 

Reorganization Plan #1, which would reduce 

the number of Office of Statistical Policy (OSP) 

employees in OMB by 28 percent. In 1977, 

there were 33 employees in the office, compared with an average of 70 employees in 1947. 

The commission recommended strengthening 

the staffing in OSP to ensure that OMB could 

be a coordinating unit for the Federal statistical 

system.14

7. Improving the Federal Statistical System: Report of the President’s Reorganization Project 

for the Federal Statistical System (the Bonnen 

Project), 1979.15

In 1979, the President commissioned the 

Bonnen Project to comprehensively review the 

way the government collects, evaluates, and 

disseminates statistical data for the Federal 

government. Its report recommended that OSP 

be established in the Executive Office of the 

President. The Bonnen recommendations came 

at the end of the Carter Administration, which 

had moved most of OSP to the Commerce Department. During the same period, Congress 

passed the Paperwork Reduction Act that 

13. Commission on Federal Paperwork, Final Summary Report (Washington, DC: 1977). https://hdl.handle.net/2027/umn.

31951d00818930g.

14. Statistics(1977): 29.

15. James T. Bonnen, and others, "Improving the Federal Statistical System: Report of the President’s Reorganization Project for 

the Federal Statistical System," reprinted in The American Statistician 35, no. 4 (November 1981).

moved the Office of Statistical Policy back to 

OMB and housed it in the newly created Office 

of Information and Regulatory Affairs. James 

Bonnen, the director of the project, saw this 

organizational change as a step backwards because it did not address the Project’s findings. 

Specifically, he said, “statistical policy was now 

expected to function in a regulatory environment and compete against more important political issues.”16

8. Economic Report of the President, Appendix B, 

“Improving the Quality of Economic Statistics” 

(The Boskin Working Group), 1990.17

In her book, Organizing to Count, Janet Norwood describes the Boskin Working Group’s 

contribution to the 1990 Economic Report of 

the President. The working group initially focused on programmatic issues relating to economic statistical measurement and on budget 

issues. The working group eventually expanded its scope to address the lack of organization 

in the Federal statistical system because it directly impacted the programmatic issues they 

discussed. The working group explored duplication among Federal statistical programs and 

considered if there would be any cost savings 

from consolidation. In the end, however, they 

decided that any reorganization or consolidation would be too costly and perhaps ineffective. The working group instead recommended targeted budget investments and that the 

President develop a data sharing legislative 

proposal that would improve upon some of the 

practical limitations of having multiple statistical agencies contribute to the suite of Federal 

economic statistics.18

16. James T. Bonnen, “[Improving the Federal Statistical System: 

Report of the Presidents Reorganization Project for the Federal 

Statistical System]: Comment.” The American Statistician 35, no. 4 

(November 1981): 209.

17. Economic Report of the President, Appendix B, “Improving the 

Quality of Economic Statistics” (Washington, D.C.: 1990), http://

www.presidency.ucsb.edu/economic_reports/1990.pdf

18. Janet Norwood, Organizing to Count: Change in the Federal Statistical System (Washington, D.C.: Urban Institute Press, 1995).
2322 The Promise of Evidence-Based Policymaking

This is a complimentary volume to the printed report,

"The Promise of Evidence-Based Policymaking."

THE PROMISE OF

EVIDENCE-BASED POLICYMAKING

Report of the Commission on Evidence-Based Policymaking
This page intentionally left blank
The Promise of Evidence-Based Policymaking

Report of the Commission on Evidence-Based Policymaking

Appendices E-H

September 2017