Building anEvidence Basis for
S&T Policy

http://readidata.nitrd.gov/star3

4

A LOT HAS BEEN LEARNED: EXAMPLE – 5
UNDERSTANDING SCIENCE INVESTMENTS

Capturing Scientific Outcomes:
Topic modelling

Automatically learned topics (e.g.):…

t6. conflict violence war international military …
t7. model method data estimation variables …
t8. parameter method point local estimates …
t9. optimization uncertainty optimal stochastic …

NSF
proposals

t11. speech sound acoustic recognition human …

Topic Model:- Use words from(all) text
- Learn T topics

t12. museum public exhibit center informal outreacht13. particles particle colloidal granular material …
t14. ocean marine scientist oceanography …
…

t49
t18
t114
t305

David Newman - UC Irvinet10. surface surfaces interfaces interface …

Topic tags foreach and every
6
proposal

Automated capture of economic
outcomes
GraphML

IngestCondense

CompareInstitutional Support:
NSF SciSIP: 0830287, 0965259
HBS: Department of Research• Data Analysis• Regressions

Train

MatchConceptual frameworks
developed•

Ideas Arise••••are tested,•••Review by peers (scientific journals)
Within firmsmature,••

Funding, infrastructure, regulation shocks? => exogenous variation
Individual activity? => mobilitySerendipity?

Adoption

and…make a significant impact•

Economic? Social? Scientific?

Regulatory Shock• Careful construction of patent database• Knowledge of automobile technologies• Initial description• Analytical drivers

Infrastructure Shock

Of mice and academics: Murray, Aghion, Dewatripont, Kolev and Stern
Natural Experiment
Careful difference in difference
Careful hypothesis structuringIdea transmission as human activityThe Diffusion Of Scientific Knowledge Across Time And Space: Evidence From
Professional Transitions For The Superstars Of Medicine (Azoulay, Graff-Zivin; Sampat)• Focus on 10,450 elite life scientists• Link individuals with their output• Careful Difference in Difference estimatesIdentifying impact• “When and how did we become certain that smoking causes
cancer, coffee does not, and human activity is producing global
climate change?”• Shwed and Bearman: analyzed substantive cases that are now
considered facts, such as the carcinogenicity of smoking and thenon-carcinogenicity of coffee, and then employed that same
analysis to two currently contested cases: the suspectedcarcinogenicity of cellular phones and the relationship between
vaccines and autism.
• http://understandingautism.columbia.edu/papers/the-temporalstructure-of-scientific-consensus-formation.pdf

And..Building a better interagency
system to answer agency questions
STAR METRICS
Science and Technology in America’s Reinvestment – Measuring the EffecTs of
Research on Innovation, Competitiveness and Science13

What is STAR METRICS• Six federal agencies; 85 research institutions
Goal:• Provide a better empirical basis for science policy.
• By providing an open and automated data infrastructure that canbe used by federal agencies, research institutions, andresearchers• By documenting federal investments in science and• By analyzing the resulting relationship between inputs, outputs,and outcomes.

Approach: automatically capture data about the conduct of science –inputs, outputs and the connections between the two

Scientific Workforce

Source: Ian Foster University ofChicago

Level II: Building an Evidence Basis:
Going Beyond the Workforce• A data platform that can link inputs andoutputs/outcomes using automated approaches
leveraging existing data
• Collaborative development of data infrastructure on
broad categories of impact:• knowledge (e.g. publication, citations…)• economic(patents, spin off companies…)• workforce(employment, student mobility…)• social(e.g. health, environment, energy…)

16

Portfolio Characterization• For agencies• Gap analysis: What is being funded in which areas?
• Expertise Locator: Who is doing research in which topics?
• For Researchers• Funding information: What programs are funding researchlike mine?
• Expertise Locator: Who else is doing research like mine?
• For VPs for Research and their Institutions• Gap analysis: Where are my institutional researchstrengths?
• Expertise Locator: How can I connect researchers?
Automatically generated from research proposals

1. Broad Lessons Learned
How can Science of Science Policy inform future
government responses?
Practical• Identifying and harnessing the right expertise
Building data infrastructure (STAR METRICS) to help
identify researchers

19

20

The pitfalls• Need to paint full picture of scientific outcomes• => engagement of scientific community critical• => Open and transparent process• Data misuse• => careful presentation of results• Data quality• => full collaboration• => extensive use of pilots• Confidentiality• => researcher, institution and agency controls
21Ultimate Goals• Fully fledged academic field• Fully fledged analytical tool set across government
agencies• Science policy in same analytical tier as tax policy• Common empirical infrastructure available to alluniversities and science agencies to quickly respond toState, Congressional and OMB requests• Common scientific infrastructure for researchers todevelop and study science policy

Thank you• Comments and questions?