2 T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« ÝÙ« R ÊÃÖ

About The National Science And Technology Council

The NaƟ onal Science and Technology Council (NSTC) was established by ExecuƟ ve Order on November 23,

1993. This cabinet-level council is the principal means by which the President coordinates science, space, and

technology policies across the Federal government. NSTC coordinates diverse paths of the Federal research

and development enterprise.

An important objecƟ ve of the NSTC is the establishment of the clear naƟ onal goals for Federal service and

technology investments in areas ranging from informaƟ on technologies and health research to improving

transportaƟ on systems and strengthening fundamental research. The Council prepares research and

development strategies that are coordinated across the Federal agencies to form a comprehensive investments

package aimed at accomplishing mulƟ ple naƟ onal goals.

For more informaƟ on visit hƩ p://www.ostp.gov/nstc/html/NSTC_Home.html/.

About The Office Of Science And Technology Policy

The Offi ce of Science and Technology Policy (OSTP) was established by the NaƟ onal Science and Technology

Policy, OrganizaƟ on and PrioriƟ es Act of 1976. OSTP’s responsibiliƟ es include advising the President in policy

formulaƟ on and budget development on all quesƟ ons concerning science and technology (S&T); arƟ culaƟ ng

the President’s S&T policies and programs; and fostering strong partnerships among Federal, state and local

governments and the scienƟfi c communiƟ es in industry and academe.

Every fi scal year, OSTP and the Offi ce of Management and Budget (OMB) issue a memorandum enƟ tled

“AdministraƟ on Research and Development Budget PrioriƟ es.” The memorandum highlights the

AdministraƟ on’s research and development prioriƟ es and emphasizes improving management and

performance to maintain excellence and leadership in science and technology. The FY 2008 memorandum is

available at hƩ p://www.ostp.gov/html/budget/2008/m06-17.pdf.

For more informaƟ on visit hƩ p://www.ostp.gov .
T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« ÝÙ« R ÊÃÖ 3

DOE/SC-106
4 T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« ÝÙ« R ÊÃÖ
T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« ÝÙ« R ÊÃÖ 5

DEPARTMENT AND AGENCY CONTRIBUTORS

CommiƩ ee Chair

Bill Valdez, Department of Energy 

Julia Lane, NaƟ onal Science FoundaƟ on

Department of Energy

Shannon Dunphy Lazo

Kohl Gill

Debbie Mayer 

JeaneƩ e Yarrington

Yolanda White

NaƟ onal Science FoundaƟ on

Lynda Carlson 

Mark Boroush

Kaye Husbands-Fealing, Former Co-Chair

Melanie Roberts

Centers for Disease Control and PrevenƟ on

Vilma Carande-Kulis

Central Intelligence Agency

H. ChrisanƟ

Department of Commerce

Patricia Buckley

Department of Defense

Mark Herbst

Spiro Lekoudis

Lewis Sloter

Environmental ProtecƟ on Agency

Phillip Juengst

Dale Pahl

NaƟ onal AeronauƟ cs & Space AdministraƟ on

Jennifer Wiseman

NaƟ onal InsƟ tutes of Health

Deborah Duran

NaƟ onal InsƟ tute for Standards & Technology

Kathleen McTigue

Gregory Tassey

NaƟ onal Oceanic & Atmospheric AdministraƟ on 

Sandra Knight

Avery Sen

Linda SuƩ ora

Offi ce of Management and Budget

Michael Holland

Joydip Kundu

Offi ce of Science and Technology Policy

Irma Arispe

Diane DiEuliis

Ted Wackler

U.S. Department of Agriculture

John King

Robert MacDonald

Sara Mazie

U.S. Geological Survey

Carl Shapiro

U.S. Department of Veterans Aff airs

Cheryl Oros
6 T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« ÝÙ« R ÊÃÖ
T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« ÝÙ« R ÊÃÖ 7

EXECUTIVE OFFICE OF THE PRESIDENT

NATIONAL SCIENCE AND TECHNOLOGY COUNCIL

WASHINGTON, DC 20502

Dear Colleague,

I am pleased to forward this document, “The Science of Science Policy: A Federal Research Roadmap.” It was 

developed in response to my challenge for a new “science of science policy” that will begin to systematically 

address the need for better scientific theories and analytical tools for improving our understanding of the 

efficacy and impact of science and technology policy decisions. It was prepared by an Interagency Task 

Group (ITG) commissioned by the National Science and Technology Council’s (NSTC) Subcommittee on Social, 

Behavioral and Economic Sciences (SBE). 

This Roadmap represents the first organized description of the emergent field of the Science of Science Policy, 

outlining scientific theories and defining terms that encompass efforts in the field thus far. It highlights the 

potential to greatly increase the knowledge base and provide needed insights to improve the data, tools and 

methods that would enable a more rigorous and quantitative basis for science and technology policy. In doing 

so, it identifies ten major science questions grouped into three theoretical themes: Understanding Science 

and Innovation; Investing in Science and Innovation; and Using the Science of Science Policy to Address 

National Priorities. 

Agencies and departments across the Federal Government face similar challenges when setting scientific 

priorities and assessing the effectiveness of current and planned investments. By working together to address 

these themes and questions, share best practices and collaborate on fundamental principles, we will greatly 

enhance our ability to maximize our critical investments in science and technology. 

 John H. Marburger, III
8 T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« ÝÙ« R ÊÃÖ
T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« ÝÙ« R ÊÃÖ 9

CONTENTS

ExecuƟ ve Summary 1

IntroducƟ on 3

 The NaƟ onal ImperaƟ ve 3

 The ScienƟfi c Challenge 4

 The Interagency Task Group (ITG) Charge and Findings 5

The Science of Science Policy Roadmap 9

 Theme 1: Understanding Science and InnovaƟ on 9

 Theme 2: InvesƟ ng in Science and InnovaƟ on 13

 Theme 3: Using the Science of Science Policy to Address NaƟ onal PrioriƟ es 18

Summary of RecommendaƟ ons 22

Current and PotenƟ al Toolkit For Science and InnovaƟ on Policy 25

References 29

Acronyms 33

Appendix A - Charter 35

Apppendix B – Science Magazine Editorial 39
10 T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« ÝÙ« R ÊÃÖ
T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ 1

EXECUTIVE SUMMARY

What is the Science of Science Policy?

The science of science policy (SoSP) is an emerging fi eld of interdisciplinary research, the goal of which is to 

provide a scienƟfi cally rigorous, quanƟ taƟ ve basis from which policy makers and researchers can assess the 

impacts of the NaƟ on’s scienƟfi c and engineering enterprise, improve their understanding of its dynamics, 

and assess the likely outcomes. Research in SoSP could be uƟ lized by the Federal Government, and the wider 

society in general, to make beƩ er R&D management decisions.

The term “science of science policy” was fi rst used by Dr. John H. 

Marburger III, the President’s Science Advisor and Director of the Offi ce 

of Science and Technology Policy (OSTP) in his keynote address to the 

American AssociaƟ on for the Advancement of Science (AAAS) Science 

and Technology Policy Forum in April, 20051

. This was expanded in 

a May 2005 Science magazine editorial (see Appendix B). In that 

editorial, Dr. Marburger called for the creaƟ on of a community of 

pracƟ ce that would create the data sets, tools, and methodologies 

needed to assist science policy decision makers as they invest in Federal 

research and development and make science policy decisions. A 

NaƟ onal Science and Technology Council (NSTC) Interagency Task Group 

(ITG) was created in 2006 to develop a coordinated Federal approach to 

the science of science policy to meet these challenges.

A NaƟ onal ImperaƟ ve for a Science of Science Policy

Development of the science of science policy is criƟ cally important to 

our NaƟ on’s ability to benefi t most eff ecƟ vely from R&D investments. 

In 2007, the U.S. Federal government R&D budget totaled $139 billion1

. 

Although the importance of public investments in science, technology, 

and innovaƟ on is understood, the raƟ onale for specifi c scienƟfi c investment decisions lacks a strong theoreƟ cal 

and empirical basis. Accordingly, given the magnitude of the Federal investment and the importance of that 

investment to our NaƟ on, science policy decision makers must have at their disposal the most rigorous tools, 

methods and data that will enable them to develop sound and cost-eff ecƟ ve investment strategies.

The ITG undertook a literature review to determine the state of the science to date. A quesƟ onnaire was also 

circulated to Federal agencies to ascertain what methods are currently in use for programmaƟ c investment 

decision making, as well as to ask what tools and resources are needed by Federal agencies that are currently 

unavailable. The ITG found that:

• There is a well developed body of social science knowledge that could be readily applied to the 

study of science and innovaƟ on.

• Although many Federal agencies have their own communiƟ es of pracƟ ce, the collecƟ on and 

analysis of data about the science and scienƟfi c communiƟ es they support is heterogeneous and 

unsystemaƟ c.

• Agencies are using very diff erent models, data and tools to understand their investments in science 

and technology.

• The data infrastructure is inadequate for decision-making.

Primary Conclusion 

of the Interagency 

Working Group: 

“Expert judgment” remains 

the best available decision 

support tool for science 

policy makers, but a nascent 

community of practice is 

emerging in the science 

policy arena that holds 

enormous potential to provide 

rigorous and quantitative 

decision support tools in the 

near future. Support and 

development of this emerging 

community of practice 

can provide the Federal 

government with these muchneeded decision tools.
2 T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ

• New tools and data sets could be developed and used to quanƟ fy the impact that the scienƟfi c 

enterprise has had on innovaƟ on and compeƟƟ veness.

This Federal Research Roadmap idenƟfi es three broad theoreƟ cal themes, with 10 underlying scienƟfi c quesƟ ons within those themes, and makes several recommendaƟ ons on how to address those quesƟ ons. More 

generally, the following next steps are recommended for overall implementaƟ on of the Roadmap.

Next Steps

• Federal government agencies should work in concert to establish a theoreƟ cal and empirical 

framework to understand the science and engineering enterprise within the context of the science 

of science policy. TentaƟ vely described in this report as a “Federal InnovaƟ on Framework,” analyses 

could be performed on how Federal investments and policy decisions aff ect the NaƟ on’s system of 

innovaƟ on.

• Establish interagency research prioriƟ es to address the scienƟfi c challenges confronƟ ng the unique 

science policy analysis needs of Federal science and technology agencies. 

• Encourage investment in the development and use of emerging tools, methods, data, and data 

infrastructure to enable science policy decision makers to base investment decisions on more 

rigorous and quanƟ taƟ ve analyses. 

• Hold a public, internaƟ onal workshop to discuss the scienƟfi c basis of the Roadmap and its 

implementaƟ on in the policy arena. The workshop will serve to further inform the U.S. Federal 

approach to the science of science policy in a broader, internaƟ onal context. 
T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ 3

INTRODUCTION

The National Imperative

Federal investments in science and technology have had an enormous impact on innovaƟ on, economic growth, 

and social health and well-being. In addiƟ on to furthering these impacts, future investments by the Federal 

government will be criƟ cal in many arenas, such as miƟ gaƟ ng the consequences of global climate change, 

exploring new energy sources, defending against external threats, and maintaining internaƟ onal compeƟƟ veness. Given the importance of those investments, it is imperaƟ ve that science policy decision makers have at 

their disposal the most rigorous tools, methods and data that will enable them to develop sound investment 

strategies.

And yet the science policy analysis community does not have the best tools, methods and data that would 

allow decision makers to make and manage such future investments opƟ mally. As a result, science policy 

discussions are frequently dominated by advocates for individual scienƟfi c fi elds who argue for their parƟ cular 

interests, but leave policy makers with liƩ le ability to objecƟ vely discriminate between investment opƟ ons. 

Policy decisions may be based upon past pracƟ ces or data trends that may not always accurately refl ect current condiƟ ons. 

Across the Federal government, there is an urgent need for rigorous analysis that can inform Federal research 

and development decision-making. Investments in many strategically important fronƟ ers of science and 

engineering and the allocaƟ on of Federal resources across a complex and decentralized naƟ onal research and 

development (R&D) porƞ olio must be guided by the best data and analysis available. A “science of science 

policy”, fi rst named by the President’s Science Advisor, Dr. John Marburger, must be developed.

The importance of this challenge derives from the magnitude and centrality of the contribuƟ on that science 

and technology make to the U.S. economy. In 2007, the U.S. Federal government R&D budget totaled $139 billion2, affi rming the importance of Federal investments in science and technology.3 It is imperaƟ ve to advance 

the scienƟfi c basis of science policy so that limited Federal resources are invested wisely. ScienƟfi c models 

must be developed, along with methods of collecƟ ng real-Ɵ me quanƟ taƟ ve and qualitaƟ ve data so that future 

policy decisions are based on sound science and informed by meaningful metrics. RetrospecƟ ve analysis is also 

needed, to analyze the impact of Federal investments on scienƟfi c discovery and innovaƟ on, the economy, 

and society. In this way, past investments may help inform future decisions, refi ne the accuracy of models, and 

Climate change is one of the major science challenges that would benefi t from 

science policy analysis and decision making.

Linear trend of annual temperatures, NaƟ onal ClimacƟ c Data Center. 

NaƟ onal Oceanic and Atmospheric AdministraƟ on, 

hƩ p://www.ncdc.noaa.gov/oa/climate/globalwarming.html#q4.
4 T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ

maintain the naƟ on’s dominance in the scienƟfi c arena. 

While federal decision-makers grapple with compeƟ ng 

prioriƟ es to make the best decisions for their own agencies, the rest of the world is vastly increasing its scienƟfi c 

investments, and there is increased foreign compeƟƟ on 

for scienƟfi c ideas and talent. The United Kingdom has 

established a Department for InnovaƟ on, UniversiƟ es 

and Skills; Saudi Arabia has invested $6 billion to establish a University of Science and Technology; and almost 

every developed European and Asian country is aggressively invesƟ ng in and compeƟ ng for scienƟfi c talent. 

Thus, science and technology policy and the choice of 

scienƟfi c investments remain at the forefront of the 

naƟ onal debate. 

The Scientific Challenge

Developing a science of science policy will require the implementaƟ on of a 

rigorous and systemaƟ c research program that is focused on addressing a 

number of scienƟfi c and data challenges, ensuring that the needs of Federal 

R&D agencies and senior decision makers are met. Its creaƟ on as a scienƟfi c 

fi eld of study requires the development of analyƟ cal tools, data bases, and 

management processes capable of providing reliable informaƟ on and the 

best possible basis for the allocaƟ on of public funds available to support 

Federal R&D. It could be built upon the emerging community of science 

in this area, and exisƟ ng experƟ se in policy analysis that is resident in the 

Federal science agencies and elsewhere in the Federal government. A 

community of pracƟ ce in academia and in the private sector could also be 

developed that would provide support to Federal policy makers. 

Current science and technology investment decisions are based on analyses 

that lack a strong theoreƟ cal and empirical basis. Increasingly, economic 

value is based on generaƟ ng and selling ideas, rather than physical goods 

and agricultural products, but the current social scienƟfi c and staƟ sƟ cal 

infrastructure has not kept pace with this change in the nature of economic acƟ vity. Indeed, while it is a 

common belief that innovaƟ on is closely related to investments in science and technology, there is actually a 

limited theoreƟ cal and scienƟfi c foundaƟ on underlying such beliefs. 

But there is an emerging view that many facets of the U.S. innovaƟ on ecosystem have become too complex 

for expert judgment alone to be an eff ecƟ ve decision support tool. Science conƟ nues to accelerate, and 

mulƟ disciplinary collaboraƟ ons are becoming more common; as a result, the complexity of the scienƟfi c 

endeavor is surpassing the ability of experts within parƟ cular scienƟfi c disciplines to understand its totality. 

The tenuous nature of the scienƟfi c links between investments in science and desired outcomes is evident 

from the variety of ways in which outcomes are characterized: someƟ mes by the enƟ ty funding or conducƟ ng 

R&D (e.g., universiƟ es, governments, or businesses), someƟ mes by the phase of discovery (e.g., basic research, 

applied research, or development), and someƟ mes in terms of the end products (e.g., products, processes, 

organizaƟ ons, or knowledge).

Scope of the Science of Science Policy:

The science of science policy includes basic 

and applied research, as well as technology 

development, demonstration, and deployment. 

It includes operational science and technology, 

from early to later stages in operational 

life cycles. It is thus comprehensive of the 

entire spectrum of innovation in science 

and technology activities, from high-risk, 

undirected activities to low risk, applied 

activities. 

TuffCell bi-polar plates 

Argonne National 

Laboratory

http://www.anl.gov/ 
T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ 5

This complexity creates a number of challenges for science policy, and is the 

subject of a vigorous scienƟfi c debate. These challenges include determining 

the appropriate roles of various Federal agencies in diff erent disciplines, 

addressing a host of agency policy and investment quesƟ ons about the 

appropriateness of disciplinary or insƟ tuƟ onal porƞ olios, and understanding 

the relaƟ ve value added of diff erent fi scal policy sƟ muli. In order to address 

this need, the Offi ce of Management and Budget and the Offi ce of Science and 

Technology Policy have encouraged Federal science and technology agencies to 

work cooperaƟ vely to bring the emerging science of science policy community 

of pracƟ ce into a mature state, via the annual “AdministraƟ on Research and 

Development Budget PrioriƟ es” memoranda. 4,5 

The Interagency Task Group (ITG) Charge and Findings

In recogniƟ on of the importance of developing a science of science policy, NSTC established the Science of 

Science Policy ITG, under the purview of the SubcommiƩ ee on Social, Behavioral, and Economic Sciences (SBE). 

The Co-Chairs of the ITG are representaƟ ves of the U.S. Department of Energy (DOE) and the NaƟ onal Science 

FoundaƟ on (NSF). Other parƟ cipaƟ ng agencies include the Departments of Agriculture (USDA), Commerce 

(DOC), Defense (DOD), EducaƟ on (DOEd), Health and Human Services (DHHS), Homeland Security (DHS), 

Interior (DOI), State, TransportaƟ on (DOT), and Veterans Aff airs (VA), as well as the NaƟ onal AeronauƟ cs & 

Space AdministraƟ on (NASA), the NaƟ onal InsƟ tutes of Health (NIH), the US Geological Survey (USGS), the 

Centers for Disease Control and PrevenƟ on (CDC), the NaƟ onal InsƟ tute of Standards and Technology (NIST), 

the Environmental ProtecƟ on Agency (EPA), the NaƟ onal Oceanic and Atmospheric AdministraƟ on (NOAA), 

OSTP, and OMB. The Interagency Task Group had a number of charges as arƟ culated in its Charter (see 

Appendix A). 

First, the ITG developed the following defi niƟ on of the science of science policy: 

 “The science of science policy is an emerging interdisciplinary research area that seeks to develop theoreƟ cal and empirical models of the scienƟfi c enterprise. This scienƟfi c basis can be 

used to help government, and society in general, make beƩ er R&D management decisions by 

establishing a scienƟfi cally rigorous, quanƟ taƟ ve basis from which policy makers and researchers may assess the impacts of the NaƟ on’s scienƟfi c and engineering enterprise, improve their 

understanding of its dynamics, and assess the likely outcomes. Examples of research in the 

science of science policy include models to understand the producƟ on of science, qualitaƟ ve 

and quanƟ taƟ ve methods to esƟ mate the impact of science, and processes for choosing from 

alternaƟ ve science porƞ olios.”

Next, the ITG completed four tasks to address its charge: 1) A review of current Federal eff orts related to 

the science of science policy; 2) ExaminaƟ on of the data that are available for analysis; 3) Development 

of a literature synthesis that brings together academic research from many diff erent disciplines; and 4) 

Development of a roadmap that would chart a path forward for the Federal government to build a community 

of pracƟ ce as well as tools in the science of science policy. This work was assisted by groundwork laid by the 

new grant program in NSF’s Directorate for Social, Behavioral, and Economic Sciences: the Science of Science 

and InnovaƟ on Policy (SciSIP). The program, although independent of the ITG, provided many insights, 

parƟ cularly with regard to the literature synthesis.

Graduate student

Lawrence Berkeley National 

Laboratory

http://www.lbl.gov/.
6 T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ

The fi rst task, a review of current Federal agency eff orts that could be viewed as supporƟ ve of the science of 

science policy, found the following:

• A number of Federal agencies have developed substanƟ al capabiliƟ es for R&D program planning 

and evaluaƟ on to develop and manage their R&D porƞ olios, measure program progress toward 

strategic goals, and to address other operaƟ onal and policy issues. The vast majority of those 

capabiliƟ es relies upon “expert judgment” and does not use quanƟ taƟ ve models or decision 

analysis tools. This is consistent with fi ndings from several reports done by NAS in the 1990s that 

idenƟfi ed expert judgment as the primary tool available to science policy makers.6

 The NAS went 

further in a 2008 report which stated that “the most eff ecƟ ve mechanism for evaluaƟ ng investment 

effi ciency of R&D programs is an expert-review panel.” 7 It may be noted that these reports are 

largely silent on evaluaƟ ons across programs or the establishment of investment prioriƟ es among 

programs in diff erent fi elds.

• Science and technology agencies expended considerable eff ort in response to the requirements for 

strategic planning and evaluaƟ on imposed by the 1993 Government Performance and Results Act 

(GPRA). This aƩ enƟ on to management, planning, and evaluaƟ on was amplifi ed in 2001, as science 

and technology programs increasingly became the subject of OMB’s Program Assessment RaƟ ng 

Tool (PART) assessments. During this period, many agencies found creaƟ ve ways to uƟ lize expert 

judgment and other types of quanƟ taƟ ve and qualitaƟ ve decision analysis tools to meet GPRA and 

PART requirements.

The ITG next examined data needs and found the following: 

• The agencies responsible for collecƟ ng naƟ onal science and 

economic staƟ sƟ cs, such as NSF and DOC, are increasingly 

focused on beƩ er ways to gauge the eff ects of R&D 

investment on the producƟ on of new knowledge and 

technology, on innovaƟ on, and their ripple eff ects throughout 

the economy. 

• Dr. Marburger addressed the OrganizaƟ on for Economic 

CooperaƟ on and Development (OECD) regarding Science 

of Science Policy needs in July 2006.8

 Since that Ɵ me, the 

staƟ sƟ cal and analyƟ c commiƩ ees of the OECD - to which the 

United States has been a major contributor and parƟ cipant 

– have devoted much eff ort to measuring the strength and 

“Complexity economics 

is part of the larger area 

of complex adaptive 

systems that incorporates 

methods from the study of 

such systems in physics, 

biology, computer science, 

and other fi elds.”

- Science of Science Policy: An 

Exploration of Literature and 

Practice, created on behalf 

of the Science of Science 

Policy ITG by the Science and 

Technology Policy Institute

The NAS has published four reports since 1993 that examine how to assess the benefi ts 

and effectiveness of Federal investment in science and technology.
T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ 7

eff ecƟ veness of the innovaƟ on infrastructures of member countries and their contribuƟ ons to 

naƟ onal economic growth. 

• The research data infrastructure suited to examining science and innovaƟ on policy is insuffi cient to 

support today’s requirements. Resources are wasted because researchers are oŌ en forced to use 

data that is not collected for research purposes or within the appropriate scienƟfi c context. As a 

result, their analyƟ cal work frequently cannot be generalized. Limited access to data oŌ en means 

that analyƟ cal work cannot be replicated. 

Finally, the ITG commissioned a literature synthesis designed to 

review academic research in science policy and related fi elds. 

This synthesis examined economic theory, social and behavioral 

sciences, the physical and biological sciences, and a host of other 

disciplines that could provide insights into analyƟ cal methods, 

tools, and data sets useful to the science policy community, 

such as the emerging theories of complexity in mathemaƟ cs and 

systems biology. The ITG also reviewed other work, most notably 

a report by the NAS’s CommiƩ ee on Science, Engineering and 

Public Policy (COSEPUP)9

 and ongoing work by the OECD.10 This 

synthesis led to the conclusion that systems level analysis from 

biology, complexity theory, social network analysis, industrial 

dynamics and other disparate fi elds have rich potenƟ al for the 

future of the science of science policy. A primary conclusion of 

the ITG is that pracƟƟ oners from these related fi elds should begin 

working to mine that potenƟ al for SoSP.

Based upon these three undertakings and their subsequent 

fi ndings, the ITG formed the primary scienƟfi c themes and 

quesƟ ons which comprise the current state of the Science of 

Science Policy. These were incorporated into the Roadmap, which 

sets forth a series of recommendaƟ ons to advance the fi eld of 

SoSP. 

Nobel Laureate 

Daniel 

Kahneman:

 The 2002 Nobel 

Prize was awarded 

to psychologist 

Daniel Kahneman 

for demonstrating 

that in situations 

with uncertainty, 

human judgment often exploits 

rules of thumb which systematically 

contradict fundamental propositions in 

probability theory.

An important fi nding in the science 

policy context is that individuals are 

much more sensitive to the way an 

outcome deviates from a reference 

level (often the status quo) than to 

the absolute outcome. When faced 

with a sequence of decisions under 

risk, individuals thus appear to 

base each decision on its gains and 

losses in isolation rather than on the 

consequences of a decision for their 

wealth as a whole.

2002 Nobel Prize Press Release
8 T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« ÝÙ« R ÊÃÖ
T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ 9

THE SCIENCE OF SCIENCE POLICY ROADMAP

The ITG idenƟfi ed three scienƟfi c themes and 10 major quesƟ ons faced by scienƟfi c agencies if they are to help 

support the development of the SoSP community. They are listed under broad conceptual categories below; 

the associated recommendaƟ ons are idenƟfi ed in the next secƟ on.

Theme 1: Understanding Science and InnovaƟ on

Establishing a scienƟfi c framework for understanding 

science and innovaƟ on is fundamental if policy 

makers are to understand the ways in which their 

decisions are likely to play out. Because innovaƟ on 

in scienƟfi c disciplines relies on human achievement, 

interacƟ on and behavior, research should be grounded 

in the core social and behavioral science disciplines 

of economics, sociology, psychology, and poliƟ cal 

science. TheoreƟ cal models of innovaƟ on must be 

developed, in the same way that economists have 

developed theoreƟ cal models of economic acƟ vity. 

This will provide a context for data collecƟ on. Just 

as the Federal Reserve Board’s econometric model is 

based on a strong theoreƟ cal foundaƟ on describing 

economic behavior, a theoreƟ cal model could be used 

to develop the empirical framework upon which to 

base investment decisions in science. 

The following list idenƟfi es the key quesƟ ons that must be addressed in order to develop such a scienƟfi c 

framework.

QçÝã®ÊÄ 1: W«ã AÙ T« B«ò®ÊÙ1⁄2 FÊçÄã®ÊÄÝ O¥ IÄÄÊòã®ÊÄ? 

InnovaƟ on, the act of creaƟ ng or invenƟ ng new ideas 

or methods, has a basis in social and behavioral acƟ vity. 

The examinaƟ on of the foundaƟ ons of our social and 

economic systems by social scienƟ sts has led to a 

deeper understanding of economic, social, and cogniƟ ve 

principles, resulƟ ng in a number of Nobel prizes, yet 

there has been liƩ le focus on the social and behavioral 

foundaƟ ons of innovaƟ on in science and technology. IniƟ al 

fi ndings may provide a basis for the development of more 

dynamic research programs in this area. For example, 

psychologists funded by NSF have developed models 

of cogniƟ ve processes across individuals and groups 

intended to promote innovaƟ on. Psychologists have also 

studied trans-disciplinary research teams. Examples of 

these include partnership teams between biomedical and 

nanotechnology industries, the collaboraƟ on between 

academic and non-academic scienƟ sts (in fi elds such as 

hydrology, soil and water science), and the development 

The Federal Reserve uses its econometric model to 

set monetary policy.

http://www.econedlink.org/lessons/index.

cfm?lesson=EM712&page=teacher

Nobel Laureate 

Ronald Coase:

The 1991 Nobel Prize was 

awarded to economist 

Ronald Coase for fi nding 

that the institutional 

structure of the economy 

may be explained by 

the relative costs of 

different institutional 

arrangements, combined 

with the parties’ efforts to 

keep total costs at a minimum. 

Alongside price formation, the formation 

of the institutional structure is regarded as 

an integral step in the process of resource 

distribution. 

1991 Nobel Prize Press Release
10 T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ

of virtual social networks (Internet based). Economists and sociologists have begun to extend models of the 

impact of diff erent incenƟ ve and organizaƟ onal structures on human behavior to the scienƟfi c enterprise. 

Some Federal agencies, such as NIH and DOE, have begun to use social network analysis techniques to 

understand the process of innovaƟ on. 

Finding: The ITG’s review of the literature suggests that while there is a well-developed understanding 

of human and social behavior in mulƟ ple disciplines such as economics, psychology, and sociology, this 

understanding has yet to be applied to the study of innovaƟ on within the scienƟfi c enterprise, leaving 

enormous gaps in scienƟfi c knowledge. For example: how does the discovery process work at the individual 

and team level? How could creaƟ ve insights be sƟ mulated? Which insƟ tuƟ onal structures facilitate the 

discovery to innovaƟ on cycle? 

QçÝã®ÊÄ 2: W«ã EøÖ1⁄2®ÄÝ T«ÄÊ1⁄2Ê¦ù Dò1⁄2ÊÖÃÄã, AÊÖã®ÊÄ AÄ D®¥¥çÝ®ÊÄ?

Technology is the applicaƟ on of science, especially to commercial and industrial objecƟ ves. Of key interest, 

then, is advancing understanding of how technology is adopted, who adopts technology and how it spreads 

through society. The current plethora of theories and models that could inform Federal decision-makers in 

moving R&D to deployment, operaƟ ons, and private sector applicaƟ ons, needs to be advanced and applied 

to the current context. This would help inform decisions as to whether funding increases, tax changes, 

regulatory changes, or other policy intervenƟ ons are more or less likely to sƟ mulate technology development, 

adopƟ on and diff usion. 

It is clear that technology adopƟ on and diff usion is a complex process; there is not a linear succession from 

basic research to the market, via applied research, technical development, producƟ on and markeƟ ng. 

Businesses and government rely upon various decision-making tools that promote technology adopƟ on – 

including consideraƟ ons about the return on investment (ROI), intellectual property valuaƟ on, consumer 

preferences, and many other factors that enable technology adopƟ on – but how those decision rules interact 

and lead from the maturaƟ on of a scienƟfi c concept into a market-based technology is currently unclear. It is 

also important to understand how leaders (those likely to take the greatest risk and adopt a technology early) 

behave within a specifi c social network. In some instances, less scienƟfi c consideraƟ ons such as the reputaƟ on 

of the early adopter play a key role in whether a technology gains access and diff uses successfully.

There are indicaƟ ons that answers can be found. The role of economic incenƟ ves in technology adopƟ on 

has been clear since Griliches’s analysis of the adopƟ on of hybrid corn in developing countries.11 Human 

capital is also a criƟ cal element: researchers have found that technology diff uses through social networks, 

since these reduce learning costs, enhance usability and sustainability, and create a social incenƟ ve structure. 

Researchers at NIST, for example, have learned that collecƟ ng data on businesses and collaboraƟ ng with 

academic researchers enables government to do the analyses that promote economic growth by encouraging 

the development of “disrupƟ ve innovaƟ ons”. Depending enƟ rely on high-tech startups to develop disrupƟ ve 

innovaƟ ons and introduce them to market—waiƟ ng for small technology fi rms to mature into, or merge with, 

larger fi rms, thereby transforming industries from the boƩ om up—is a strategy that is both slow and uncertain, 

but NIST has learned that technology adopƟ on can be accelerated through certain targeted investments. 

Current research also suggests that large fi rms increasingly have a need for external partners to help them 

overcome internal, as well as external, barriers to the development of disrupƟ ve innovaƟ ons.12
T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ 11

Finding: The invesƟ gaƟ on of technology adopƟ on and diff usion has been largely confi ned to the academic 

realm. Some government agencies, such as NIST, have collected data in a scienƟfi c fashion, and have 

supported basic research. However, challenges remain, such as the development of technology adopƟ on 

models, as well as research on full systems approaches to mapping science, technology, and innovaƟ on. This 

research could be signifi cantly advanced by developing stronger links between the academic and pracƟƟ oner 

communiƟ es.

QçÝã®ÊÄ 3: HÊó AÄ W«ù DÊ CÊÃÃçÄ®ã®Ý O¥ S®Ä AÄ IÄÄÊòã®ÊÄ FÊÙÃ AÄ EòÊ1⁄2ò?

CommuniƟ es of science provide the backdrop for 

promoƟ ng scienƟfi c discovery and innovaƟ on. 

Building on the exisƟ ng understanding of how such 

communiƟ es evolve would have clear implicaƟ ons for 

investment decisions. Research funding, for example, 

could be structured to encourage the formaƟ on of 

new communiƟ es, as is currently occurring through 

the large Federal investment in the nanoscience 

and syntheƟ c biology communiƟ es. Studying the 

behavior and formaƟ on of communiƟ es could 

avoid unnecessary duplicaƟ on, since a feature of 

communiƟ es is the joint study, documentaƟ on, and 

communicaƟ on of scienƟfi c advances. Finally, using 

communiƟ es as the unit of analysis is another way 

of tracking the scienƟfi c impact of investments. It is 

apparent that the increasing complexity of science 

means that insƟ tuƟ onal and disciplinary boundaries 

are no longer the organizing principle for the 

development of communiƟ es. Social scienƟ sts have 

been able to use new tools to advance understanding 

of social network theory, although the data 

requirements are complex and evolving. The fi eld of 

study of virtual organizaƟ ons is also emerging, with a 

fi rst solicitaƟ on by NSF’s Offi ce of Cyberinfrastructure. 

A number of research agencies have developed the 

infrastructure to study communiƟ es. For example, 

USDA science agencies are using its Current Research 

InformaƟ on System (CRIS) to keep track of performers. 

DEFINITION: DISRUPTIVE INNOVATION:

The term “disruptive technology” was fi rst described by Christensen as the creation 

of cheaper, simpler- to-use versions of existing products (in contrast to “sustaining 

technologies”, which provide incremental improvements on existing products). Now the 

term “disruptive innovation” is commonly used to describe any technological innovation, 

product, or service that uses a “disruptive” strategy, rather than a “sustaining” strategy, 

to impact existing dominant technologies or status quo products in a market. Disruptive 

innovations may create entirely new markets, or new customers within existing 

markets. 

Clayton Christensen, The Innovator’s Dilemma (Harvard Business School Press), 1997.

SCIENCE POLICY IN ACTION: 

BIOTECHNOLOGY RISK ASSESSMENT 

RESEARCH 

USDA is funding work on the multidimensional issues regarding the adoption 

and diffusion of biotechnology. These issues 

revolve around agriculture and food systems, 

markets and consumers, businesses, 

institutions, and social issues.

Ag Engineer and Corn, Agricultural 

Research Service, U.S. Department 

of Agriculture

The changing agriculture enterprise in the 

21st Century includes increasing use of 

transgenic crops. While these crops have 

been widely adopted in the United States 

to great benefi t, there remain a number 

of concerns about their adoption that have 

created barriers to their sale and export to a 

number of countries.
12 T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ

RelaƟ onal databases are being developed by academics, and 

NSF has funded the study of complex distributed project teams 

developed under individual solicitaƟ ons, such as its InformaƟ on 

Technology Research program. DOE has funded three separate social 

network studies targeted at the high-performance compuƟ ng and 

nanoscience communiƟ es.

Finding: Although each Federal agency has its own community of 

pracƟ ce, the collecƟ on and analysis of data about the scienƟ sts 

and the communiƟ es supported by those Federal agencies is 

heterogeneous and unsystemaƟ c. There is liƩ le analysis of the way 

in which the pracƟ ce of science has become distributed across space, 

Ɵ me, and research areas as a result of computaƟ onal advances. As 

a result, there is liƩ le understanding of how scienƟfi c communiƟ es 

respond to changes in funding within research areas and across 

naƟ onal boundaries, or to changes in program foci. For example, 

how important are naƟ onal and internaƟ onal human capital fl ows? 

What is the role of the Internet and cybertools in communicaƟ ng 

scienƟfi c ideas within and across communiƟ es? How do diff erent 

mixes of research performers (industry, Federal laboratories and 

universiƟ es) infl uence the development of science communiƟ es?

T«Ã 1 RÊÃÃÄã®ÊÄÝ

The ITG recommends that Federal government agencies work in concert to establish a theoreƟ cal and 

empirical framework to understand the science and engineering enterprise within the context of the Science of 

Science Policy. 

• An NSTC Working Group should regularly perform 

porƞ olio analyses of the full spectrum of SoSP across the 

Federal government and provide the President’s Science 

Advisor with their results. These analyses are tentaƟ vely 

described in this report as a “Federal InnovaƟ on 

Framework,” which analyses how Federal investments 

and policy decisions aff ect the NaƟ on’s system of 

innovaƟ on. 

• NSF and other agencies should conƟ nue to support 

the development of a theoreƟ cal foundaƟ on through 

exisƟ ng programs of invesƟ gator-iniƟ ated research. 

Workshops and informaƟ onal websites can facilitate that 

dialog.

“Emerging economies learned 

a key lesson: investment in 

innovation capacity is the key to 

higher productivity, higher wages 

and higher economic growth. 

Emerging economies are investing 

in research and virtual, physical 

and educational infrastructure. 

Global companies are establishing 

additional innovation capabilities 

in the emerging world as they 

increasingly collocate R&D with 

new market opportunities. While 

the United States is the world’s 

strongest innovator nation 

today, a wide range of surveys 

shows that many companies 

plan to establish high value and 

knowledge-intensive operations 

offshore, including R&D, and 

that emerging economies are 

now among the most attractive 

destinations for that investment.

Five for the Future. Council on 

Competitiveness, 2007.

“Policymakers must often decide 

whether to make a choice on a 

current assessment of the costs 

and benefi ts of taking action 

based on imperfect information or 

to await additional scientifi c and 

technical information. Moreover, 

while scientifi c knowledge and 

technological development is 

changing constantly, the same is 

not always true of public policy. 

As a result, policies developed 

a number of years ago may not 

refl ect the latest scientifi c and 

technological knowledge.”

Stine, D. D. Science and Technology 

Policymaking: A Primer. Congressional 

Research Service, April 2008.
T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ 13

• Individual agencies should work together to idenƟ fy a core suite of ways to measure and describe 

technology adopƟ on and diff usion. They should also develop ways in which the many scienƟfi c 

communiƟ es of pracƟ ce for each agency could be described and analyzed. Working subgroups such 

as these would be responsible generaƟ ng a report to the larger Working Group.

• Federal agencies could work with internaƟ onal counterparts to develop a consistent approach to 

the Science of Science Policy that transcends naƟ onal boundaries, potenƟ ally through the OECD or 

internaƟ onal meeƟ ng symposia.

Theme 2: InvesƟ ng in Science and InnovaƟ on 

The pragmaƟ c reality facing Federal agencies is that the resources available for invesƟ ng in research are 

limited. Each agency, and each program within an agency, either explicitly or implicitly, makes decisions 

about the allocaƟ on of those resources on an ongoing basis. The ITG’s review revealed that agencies use 

diff erent methods and tools to make those allocaƟ on decisions. Federal agencies have developed budget 

support processes that enable them to make investment decisions on an intra-agency basis, but the increasing 

complexity of the science conducted by those agencies is becoming daunƟ ng. NIH, for example, must manage 

a budget process that supports 27 separate insƟ tutes, while the DOE manages a budget process that includes 

nine major R&D offi ces that invest in a varied and complex range of sciences and technology. The problem is 

compounded when determining the benefi ts of investments across agencies or naƟ onal boundaries. 

QçÝã®ÊÄ 4: W«ã IÝ T« V1⁄2ç O¥ T« Nã®ÊÄ’Ý Pç1⁄2® IÄòÝãÃÄã IÄ S®Ä? 

There is a wide body of scienƟfi c and technical knowledge created solely through Federal research and 

development investment, which would not typically be sought through private investment (i.e. generaƟ on of 

this knowledge does not have an immediate or obvious “payoff ”). Value, in the Federal context context, is thus 

twofold, and refers to both the value of the knowledge produced by governmental eff orts (how much is the 

knowledge worth?), as well as the value of developing that knowledge through governmental eff orts (what is 

the value of learning this with public funds?). Frequently, these values are not known unƟ l viewed in historical 

or anecdotal context. In order to make more informed and prioriƟ zed research investments, Federal agencies 

have a need to beƩ er understand the value of the knowledge likely to be produced from their research 

investments in real Ɵ me. Failure to do so has very real consequences. For example, the famous observaƟ on of 

Nobel Laureate Bob Solow: “You can see computers everywhere except in the producƟ vity staƟ sƟ cs”, turned 

out to be due to a misinterpretaƟ on of data. This not only led to an understatement of U.S. economic growth, 

but also to a fundamental misunderstanding of the importance of informaƟ on technology in contribuƟ ng to 

growth.13 

DEFINITION – FEDERAL INNOVATION FRAMEWORK:

Analyses of the scientifi c theories on science and technology policy decision making that 

explain how federal investments and policy decisions affect the nation’s system of innovation, 

and provide an empirical framework to understand the Nation’s science and engineering 

enterprise. It includes ongoing evaluation of the relevant tools and metrics utilized by 

different Federal Agencies. The Federal Innovation Framework group, led by OSTP, will 

include feedback loops that analyze the impact of various policy instruments. This will 

require cooperation with a wide variety of Federal agencies, including the Department of 

Treasury, DOC and the major Federal statistical agencies to understand the effects of tax 

policy, labor policy and other Federal efforts that impact science policy.
14 T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ

There are many methods of valuing publicly funded knowledge, but most of them rely solely upon qualitaƟ ve 

assessments, such as commiƩ ee reviews and case studies. Various agencies, however, have begun eff orts 

to develop complementary quanƟ taƟ ve methods designed to address the value quesƟ on within their own 

context. For example, DOE has begun to develop risk assessment and modeling tools that could provide 

insights into the value of a mixed porƞ olio of energy effi ciency investments.14 Other eff orts developed by 

agencies include measuring patents, citaƟ ons, prototype products and processes, business fi nance measures, 

as well as collaboraƟ ve relaƟ onships formed and publicaƟ ons.15, 16 NIH counts the reducƟ ons in the direct and 

indirect costs of illness, as well as reducƟ ons in intangible costs due to increases in longevity in beƩ er quality 

of life. In addiƟ on, NSF’s SciSIP program has funded researchers who are aƩ empƟ ng to document the benefi ts 

from publicly funded internaƟ onal collaboraƟ on in bio-fuels, as well as the contribuƟ ons of foreign graduate 

students and post-doctoral students to knowledge creaƟ on and diff usion. 

Finding: Although determining the value of publicly funded knowledge is a criƟ cal outcome measure for Federal science agencies, the analysis is largely agency specifi c. Many of the tools are of uneven quality and the 

broader discourse is oŌ en anecdotal. Many open research quesƟ ons remain. For example, is it possible to develop a full systems approach to mapping science, technology, and innovaƟ on? Is it possible to put in place a 

complete accounƟ ng of intangible assets and their contribuƟ ons to science and technology outcomes to create 

an overall scienƟfi c measure of return on investment? How can the community develop more inclusive measures to capture the spillover eff ects between scienƟfi c discovery and technological innovaƟ on, parƟ cularly 

among universiƟ es, companies, and government laboratories? 

The creaƟ on of a Federal InnovaƟ on Framework, as proposed here, could provide a forum within which 

varied community pracƟ ces can be shared. This Framework group would sƟ mulate dialog promoƟ ng a beƩ er 

understanding of which decision support tools could be used by diff erent agencies, allow for joint data 

collecƟ on eff orts, and sƟ mulate more rigorous methods of analyzing the scienƟfi c process among diff erent 

agencies.

QçÝã®ÊÄ 5: IÝ Iã PÊÝÝ®1⁄2 TÊ “PÙ®ã D®ÝÊòÙù”?

It is extremely unlikely that any single model could predict parƟ cular discoveries and any aƩ empt to build 

such a model should be looked upon with great skepƟ cism. New advances in agent based modeling and an 

increased capacity to simulate diff erent scenarios, however, hold the promise that a series of possible future 

scenarios could be developed. With improved models of the processes that lead to the diff usion of knowledge 

and the evoluƟ on of the communiƟ es of science, new and emerging areas of discovery could be idenƟfi ed 

and targeted for accelerated assistance. For example, researchers have begun to look at gaps in the “Idea 

InnovaƟ on Network”. The Idea InnovaƟ on Network divides research acƟ viƟ es into six diff erent innovaƟ on 

arenas and their relatedness within a parƟ cular industrial sector (see box below). Gaps in our understanding 

of this include the roles of low- versus high-risk science, as well as small versus large-scale science.17 

A similar eff ort has gone into developing a complex systems model of technological evoluƟ on, focusing on 

low-carbon energy technologies. Many agencies, such as DOE, NIH, and NSF are funding new tools that capture 

data on citaƟ ons and patents which can reveal areas of emerging innovaƟ on acƟ vity and the movement 

to market, off ering tantalizing hints of real-Ɵ me predicƟ on of near-term discoveries. USDA CooperaƟ ve 

State Research, EducaƟ on, and Extension Research (CSREES) and Forest Service (FS) are using logic models, 

which describe work inputs and outputs within an organizaƟ on, to plan and assess their R&D porƞ olios 

and programs. Further, CSREES is requiring that logic models be included in proposals submiƩ ed to many 

of its programs. USDA Economic Research Service (ERS) is using environment scanning, commonly used by 

companies to gain factual and subjecƟ ve informaƟ on on business environments they are considering entering, 

to poll external stakeholders on future program direcƟ ons. NASA uses decadal surveys and strategic roadmaps 
T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ 15

to plan, relying heavily on independent assessments, thereby idenƟ fying and prioriƟ zing promising scienƟfi c 

goals, missions, and programs.

Finding: Agencies are using very diff erent approaches and tools designed to develop scenarios that anƟ cipate 

the eff ects of discovery and innovaƟ on. Many agencies are not doing this at all. There is very liƩ le communicaƟ on across agencies, and liƩ le evaluaƟ on of the strengths or benefi ts of diff erent approaches. No agency 

reported using approaches similar to those developed by the Federal Reserve, which expended considerable 

eff ort to create a complex econometric model based on the best available models, data, and tools to understand the impact of diff erent intervenƟ ons on a complex set of outcomes.18 In addiƟ on, there is liƩ le transparency in the analyƟ cal process.

QçÝã®ÊÄ 6: IÝ Iã PÊÝÝ®1⁄2 TÊ DÝÙ® T« IÃÖã O¥ D®ÝÊòÙù OÄ IÄÄÊòã®ÊÄ?

The current state of the art in describing the impact of discovery, typically cast as R&D or technology, rests on 

the results of econometric studies, surveys, case studies, and retrospecƟ ve analyses. 

Econometric studies include the macroeconomic growth models pioneered by Robert Solow19, which show 

the equilibrium growth path for an economy with an assumed endowment of technology, but pay liƩ le 

aƩ enƟ on to the development of technology or how it gets used (innovaƟ on). The Solow models have been 

embellished and expanded by others20, 21, however, these models are limited by their treatment of discovery 

as an exogenous “black box” rather than as part of a larger ecosystem of innovaƟ on. This kind of aggregated 

approach, which has been implemented by the European Union,22 is not fl exible enough to capture the 

complexity of the feedback mechanisms which the nascent literature indicates lies at the core of innovaƟ on.

Surveys are heavily used by a number of agencies. EPA relies on a combinaƟ on of partner surveys, research 

citaƟ ons in regulatory and other documents, and bibliometric analyses to inform their broader program 

reviews that assess research impact. Other agencies also track patents and papers. USDA-CSREES tracks 

publicaƟ ons of papers that result from its investments, as well as patents.23 DOE has been exploring the use of 

a number of stochasƟ c and linear models to understand the impact of scienƟfi c discovery.

Some agencies, such as NIST, prospecƟ vely esƟ mate their impacts through standard benefi t-cost analysis. 

Others, such as the CDC, conduct case studies summarizing selected research projects.24 The CDC has also 

used retrospecƟ ve bibliometric analysis in the past, and is now moving to an internet-based informaƟ on 

tracking system. Most agencies (NIH, DOE, NASA, EPA, USDA) track milestones and use peer-review, advisory 

commiƩ ees, and survey instruments to assess impacts. Other agencies such as USGS have explored the 

importance and value of improved scienƟfi c informaƟ on in land use decision making. A few agencies, such as 

DEFINITION: IDEA INNOVATION NETWORKS

These networks exist at the level of an industrial sector and market sector, and each network 

has six different functional arenas in which various types of innovative processes occur. The six 

research arenas are basic research, applied research, research about product development, 

research on manufacturing processes, research on quality control, and research about the 

commercialization and marketing of products. Each of these functional arenas has its own 

highly trained workers, dedicated research funds, and specifi c outputs. An idea innovation 

network is defi ned as the research activities in each of the six arenas and the connectedness 

within and among these arenas in a particular industrial sector.

Hage, Jerald and J. Roger Hollingsworth. 2000. “A Strategy for the Analyses of Idea 

Innovation Networks and Institutions” Organization Studies (Special Issue: The 

Institutional Dynamics of Innovation Systems) 21(5): 971-1004. 
16 T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ

DOE and NIH, have begun to experiment with dynamic modeling and opƟ ons modeling as ways to describe the 

impact of discovery. 

Finding: Agencies are using a wide variety of approaches to describe the impact of discovery. However, new 

approaches are being developed by the academic community that uƟ lize new tools such as science mapping 

(correlaƟ ng funding with research outputs), and new datasets such as the OECD internaƟ onal database of 

inter-organizaƟ onal collaboraƟ ve agreements. The Federal community sƟ ll lacks a theoreƟ cal framework that 

it can use to assess the impact of science and technology policies on discovery and resultant social welfare 

outcmes. 

QçÝã®ÊÄ 7: W«ã AÙ T« DãÙÃ®ÄÄãÝ O¥ IÄòÝãÃÄã E¥¥ã®òÄÝÝ? 

Federal agencies are accountable for the eff ecƟ veness of their R&D investments. This requirement has been 

made explicit by OMB through the use of R&D Investment Criteria and the PART. As a result, some agencies 

have developed a variety of tools to assess the eff ecƟ veness of their investments, while others sƟ ll lack such 

tools. In terms of assessing the success of individual projects, agencies have made tremendous strides.

Different agencies use different tools to make policy and investment decisions – the method depends 

on the scope and size of the science involved. Types of Evaluation Methods for Assessing R&D 

Benefi ts and Effectiveness for Various Scales of R&D Investment, U.S. Department of Energy.

DEFINITION – SCIENTIFIC DISCOVERY:

Scientifi c discovery may be described as the observation of new phenomena, new actions, 

or new events and providing new reasoning to explain the knowledge gathered through 

such observations with previously acquired knowledge from abstract thought and everyday 

experience. In scientifi c research, exploration is one of three purposes of research, the 

other two being description and explanation. Discovery is made by providing observational 

evidence and attempts to develop an initial, rough understanding of some phenomenon.

http://en.wikipedia.org/wiki/Discovery_%28observation%29 
T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ 17

Finding: Techniques used by Federal agencies to determine program eff ecƟ veness span the spectrum from 

those in the pilot stages to those that are mature. A list of these approaches as revealed by the SOSP ITG 

quesƟ onnaire include:

• Growth AccounƟ ng— used to beƩ er generate esƟ mates of the NaƟ on’s producƟ vity performance in 

terms of contribuƟ ng factors and outputs. 

• Knowledge Economy— these composite knowledge indicators are used to improve investment 

decisions for R&D, educaƟ on, and capital resources.

• Financial ReporƟ ng—these reports are used to provide a balanced scorecard of physical as well as 

intangible assets. 

• ValuaƟ on of InnovaƟ on—business execuƟ ves and fi nancial markets (to beƩ er value R&D acƟ vity 

and related intangibles) esƟ mate fi nancial results, improve long term stock market valuaƟ ons, and 

predict outcomes. 

• System Dynamics—expand the range of “real-Ɵ me” innovaƟ on metrics to help build more robust 

system dynamics models and policy simulaƟ ons. 

• General Purpose Technology (GPT) — improved analysis of the strategic contribuƟ on of GPT’s may 

set the stage for incremental innovaƟ on and have the inherent potenƟ al for pervasive applicaƟ on in 

a wide variety of industries

• Tech-led Regional Development and Clusters—used to shiŌ the emphasis from strengthening inputs 

to the innovaƟ on infrastructures toward improving the effi ciency, rate, and output of innovaƟ on.25

While these tools are extremely useful for assessing program eff ecƟ veness, the evaluaƟ on of complex 

porƞ olios (such as those managed by NSF, DOE or NIH) remains diffi cult. Agencies are keenly aware of the 

defi ciencies in their approaches. One agency, when responding to the ITG’s quesƟ onnaire, noted: “WaiƟ ng to 

count publicaƟ ons, for example, is too late to aff ect the real-Ɵ me assessment of a parƟ cular grant.” Another 

noted: “We need a great deal more outcome data. The data currently available are inadequate for program 

evaluaƟ on purposes. Resources are needed to improve Federal and other data collecƟ ons systems”. 

Of course, evaluaƟ on approaches vary even within an agency, depending on the nature of the needs, as the 

Figure demonstrates. For details, see the “Current and PotenƟ al Toolkit For Science and InnovaƟ on Policy” 

secƟ on. 

T«Ã 2 RÊÃÃÄã®ÊÄÝ

The ITG recommends that a Federal InnovaƟ on Framework, as described in Theme 1, be created to 

accelerate the development of the nascent community of science policy researchers and pracƟƟ oners 

through interagency coordinaƟ on and targeted investments. Led by OSTP, this working group could assist 

each agencies’ specifi c science policy analysis capabiliƟ es, while coordinaƟ ng eff orts to idenƟ fy and promote 

best pracƟ ces and informaƟ on sharing around science policy and R&D investment management. Specifi c 

recommendaƟ ons include:
18 T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ

• Agencies (such as NIH, NSF, DHS, CDC, VA, NASA, USGS, and DOE) should work together to develop 

a pilot data infrastructure that captures key data about their respecƟ ve scienƟfi c communiƟ es. The 

development would include an assessment of the validity of the new visualizaƟ on techniques to 

describe the changing structure of science.

• Agencies should work together to develop pilot standards for idenƟ fying ways of measuring the 

value of knowledge, which could then be adapted to the missions of individual agencies. 

• Agencies should work together to develop standard approaches for using bibliometrics to assess 

science impact.

The academic research community should conƟ nue to be supported to perform the best research to develop 

new analyƟ cal tools, methods, and metrics to support the emerging science of science policy.

Theme 3: Using the Science of Science Policy to Address NaƟ onal PrioriƟ es

By developing the core data, models, and tools that will be necessary to answer the 10 ScienƟfi c Challenges 

addressed in this Roadmap, signifi cant advances in economics, 

sociology, psychology and poliƟ cal science could be made that 

would benefi t all of society.

Advances in the social and economic sciences provide the 

foundaƟ on for the science of science policy and should be 

leveraged to develop decision support tools that policy makers 

could use when grappling with extraordinarily complex naƟ onal 

challenges. Decisions to invest in science are, by necessity, made 

in the context of other investment decisions (such as investments 

in defense and transportaƟ on infrastructure), and the current 

absence of appropriate analyƟ cal tools limit the careful analysis of 

the relaƟ ve costs and benefi ts of such investments. The absence 

of tools stands in sharp contrast to the intricate econometric 

models used by the fi nancial community to understand trade and 

industrial phenomena. These econometric tools have developed 

over decades and are just now reaching maturity as the fi eld 

of economics has matured; in similar manner, it is expected that tools and models for science policy should 

develop, as the fi eld matures.

The development of a Federal InnovaƟ on Framework could be used not only to assess the state of the art 

in SoSP and its relevant tools as described, it could also be uƟ lized to perform analyses relevant to naƟ onal 

prioriƟ es. Examples include addressing climate change technology opƟ ons, agricultural policy that is impacted 

by the decision to pursue biofuels, and understanding the implicaƟ ons of improved health care in the U.S. on 

social support networks such as Medicare and Medicaid. 

The Human Genome Project was a 

concerted Federal effort in science that 

resulted in interagency cooperation to 

solve a national challenge.

U.S. Department of Energy,

http://genomics.energy.gov/

DEFINITION – INNOVATION INFRASTRUCTURE

Innovation infrastructure is the physical and policy infrastructure that supports innovators. 

Innovation infrastructure includes information networks, intellectual property protections, 

business regulations, and structures for collaboration among innovation stakeholders. These 

supporting infrastructures should be adaptive as the needs for scientifi c innovation evolve, 

requiring informed policy that evolves apace.

Innovate America, National Innovation Initiative Report, Council on Competitiveness, 2005, 

http://www.compete.org/images/uploads/File/PDF%20Files/NII_Innovate_America.pdf.
T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ 19

QçÝã®ÊÄ 8: W«ã IÃÖã DÊÝ S®Ä Hò OÄ IÄÄÊòã®ÊÄ AÄ CÊÃÖã®ã®òÄÝÝ?

Vannevar Bush, in his seminal 1945 treaƟ se, “Science, 

The Endless FronƟ er,” wrote: “New products and new 

processes do not appear full-grown. They are founded on 

new principles and new concepƟ ons, which in turn are 

painstakingly developed by research in the purest realms 

of science.” Bush’s treaƟ se helped shape U.S. Federal 

government policy on science and technology and gave 

great support for the noƟ on that innovaƟ on in science 

and technology is a key driver of U.S. naƟ onal economic 

prosperity and naƟ onal security. 

In the last half century, economists and social scienƟ sts 

have aƩ empted to document the impact of the scienƟfi c 

enterprise on U.S. innovaƟ on and compeƟƟ veness, with 

mixed results. At the macro level, economists such as 

Robert Solow (who won the Nobel Prize in Economics 

for his work on the impact of innovaƟ on on economic 

growth) have developed widely accepted theories that 

explain how investments in science and technology 

have a posiƟ ve impact on economic growth. There are 

disputes about the rate of growth and other factors, 

but there is no dispute that the impact is posiƟ ve. At 

the micro level, economists such as Erik Brynjolfsson 

have documented the posiƟ ve impact of informaƟ on 

technology on organizaƟ ons and fi rm performance1. This 

level of analysis is helpful because it provides comfort to Federal policy makers that their investment strategy is 

sound, but macro level econometric models cannot answer the basic quesƟ on, how much funding is enough?

For example, policy makers do not have rigorous and quanƟ taƟ ve tools that explain how various technology 

streams could reduce U.S. energy consumpƟ on, or whether or not investments in rare diseases that affl ict 

small populaƟ ons are more important to society than investments in common diseases, such as HIV or cancer, 

that require long-term investments with uncertain prospects for success.

There is a considerable literature on impact evaluaƟ on in the social sciences that is related to this problem 

set confronted by the science of science policy community. There are also new developments in the ways 

of collecƟ ng data on complex inputs, such as the generaƟ on of ideas through tracking citaƟ ons, as well as 

collecƟ ng informaƟ on on the transmission of ideas through social networks and scienƟfi c communiƟ es. The 

increasing availability of administraƟ ve data, combined with advances in privacy protecƟ on, could allow for the 

examinaƟ on of the impact of diff erent types of tax credits on businesses, or illuminate hiring paƩ erns.

Finding: The ITG fi nds that there is a real opportunity to develop new tools and data sets that could be used 

to quanƟ fy the impact of the scienƟfi c enterprise thus far on innovaƟ on and compeƟƟ veness. These impacts 

could include the generaƟ on of knowledge, the health of the universiƟ es receiving funding, the growth of the 

STEM workforce, or the growth and survival of those businesses and their workforces most closely linked to 

the scienƟfi c enterprise. 

1 See, for example, Aral, S., Brynjolfsson, E. and Wu, D.J., Which Came First, IT or Productivity? The Virtuous Cycle of Investment and 

Use in Enterprise Systems , MIT Center for Digital Business Working Paper, (October, 2006).

Vannevar Bush, President Science Advisor, 

helped establish the science and technology 

policy that enabled the U.S. Federal 

government to play a central role in the 

creation of the world’s leading innovation 

economy.

http://www.carnegieinstitution.org Yearbook 
20 T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ

QçÝã®ÊÄ 9: HÊó CÊÃÖã®ã®ò IÝ T« U.S. S®Äã®¥® WÊÙ»¥ÊÙ? 

The dominance of the U.S. scienƟfi c enterprise is dependent on the quality of its scienƟfi c workforce. Our 

naƟ onal compeƟƟ veness in science has been enhanced by the many foreign students who come to U.S. 

universiƟ es, as well as by the many skilled STEM workers who move to the U.S. from other countries. 

However, U.S. scienƟfi c dominance may be threatened by the ability of other countries to train their own 

technical workforce, as well as the ability of businesses to tap into the innovaƟ ve capacity of a global scienƟfi c 

community and encourage technical workers from the U.S. to relocate to naƟ ons such as China and India.

There has been considerable policy debate about the quality of the STEM workforce but arguments supporƟ ng 

either side of the issue are hindered by a lack of longitudinal data on the post secondary STEM workforce. 

QuesƟ ons about this workforce include: what happens 

to them when they graduate; where do they get 

jobs; and what are their labor market trajectories? 

Industries accustomed to the free fl ow of technical 

workers someƟ mes fi nd that naƟ onal security 

concerns impede their ability to recruit and retain 

internaƟ onal workers. UniversiƟ es are now training 

large numbers of foreign scienƟ sts who return to their 

home countries.

In addiƟ on, few direct measures of the global science 

and engineering (S&E) labor force exist. Recent 

analyƟ cal work has been done in the United States 

and Europe on “network analysis,” which could be 

used to measure the impacts of and reasons for the 

“high-skilled diaspora” that has been discussed in the 

science and engineering literature. Developing and extending the results of those network analysis studies 

could go a long way toward answering quesƟ ons about labor market formaƟ on in S&E.

Finding: Many criƟ cal quesƟ ons about the quality and global nature of the STEM workforce cannot be answered due to a lack of data. While the models and tools exist to study fl ows of workers within and across 

disciplines and naƟ ons, lack of data means that the science policy community cannot answer important quesƟ ons about the scienƟfi c enterprise. 

QçÝã®ÊÄ 10: W«ã IÝ T« R1⁄2ã®ò IÃÖÊÙãÄ O¥ D®¥¥ÙÄã PÊ1⁄2®ù IÄÝãÙçÃÄãÝ IÄ S®Ä PÊ1⁄2®ù?

The primary policy instrument that the U.S Federal government wields for science policy is broad Federal 

investment in key areas of science and technology. As a result, the U.S. has invested trillions of dollars over 

six decades in university research, scienƟfi c workforce development, naƟ onal laboratory infrastructure, major 

scienƟfi c instruments and other areas. This has resulted in an unrivaled, world class scienƟfi c infrastructure 

that will be extremely diffi cult for any other naƟ on to duplicate. This massive investment in science and 

technology by the Federal government is perhaps the key U.S. compeƟƟ ve advantage in the emerging 

internaƟ onal innovaƟ on compeƟƟ on. This compeƟƟ ve advantage has not gone unnoƟ ced, which helps 

explain the drive by China, Korea and other Asian countries to make major naƟ onal investments in science and 

technology and the decision by the European Union to triple its rate of investment in science and technology. 

Investment in science and technology, however, is only one of the policy instruments available to science policy 

makers; others include fostering the role of compeƟƟ on and openness in the promoƟ on of discovery, the 

construcƟ on of intellectual property systems, tax policy, and investment in a STEM workforce. However, the 

Students in the lab - workforce development, NSF.
T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ 21

probable impact of these various policies and intervenƟ ons 

is largely unknown. This lack of knowledge can lead to 

serious and unintended consequences. For example, 

Federal eff orts to increase industrial innovaƟ on through 

university technology transfer programs has led to concerns 

that universiƟ es are now focusing too much on near-term 

research and too liƩ le on their tradiƟ onal strength, which 

is long-term and basic research. The recent doubling of 

investment at NIH has created concerns within the medical 

community about the impact these investments have 

had on the producƟ on of PhDs and tenured faculty at 

universiƟ es.

Finding: There has been very liƩ le investment in the study 

of alternaƟ ve science policy instruments either in the 

United States or in other countries. While the models and 

tools exist to examine the eff ecƟ veness of diff erent approaches, there are gaps in the analyƟ cal structure, the 

data infrastructure, and in the ways in which informaƟ on 

can be conveyed to policy makers.

T«Ã 3 RÊÃÃÄã®ÊNS

The ITG recommends invesƟ ng in data collecƟ on, analyƟ cal 

tools, and ways to present complex informaƟ on. 

• Several core datasets should be established and made available to both the research and policy 

community. 

• The fi rst of these is a longitudinal dataset on businesses, oversampling those businesses 

which are criƟ cal to American compeƟƟ veness, such as high tech, biotech, and 

mulƟ naƟ onal fi rms. ExisƟ ng administraƟ ve data should be used to keep the costs 

manageable and the sample scienƟfi c. 

• The second is a longitudinal dataset on the STEM workforce. Although the original 

dataset could have a survey basis, every eff ort should be made to exploit longitudinal 

administraƟ ve records, and to partner with other countries in order to capture the long 

term dynamic adjustments of workers. 

• The link between workers and fi rms must be tracked through administraƟ ve records and 

other modes of data collecƟ on so that the relaƟ onship between the humans, who are the 

sources of innovaƟ ve ideas, and the fi rms, who bring the ideas to market, can be analyzed.

• The Federal InnovaƟ on Framework must include feedback loops that analyze the impact of various 

policy instruments. This will require cooperaƟ on with a wide variety of other Federal agencies, 

including the Department of Treasury, the Census Bureau and the major Federal staƟ sƟ cal agencies 

to understand the eff ects of tax policy, labor policy and other Federal eff orts that impact science 

policy.

OMB No. 1545-0619 6765 Credit for Increasing Research Activities Form

Department of the Treasury

Internal Revenue Service

Attachment

Sequence No. 81

Attach to your tax return.

Name(s) shown on return Identifying number

Basic research payments to qualified organizations (see instructions) 2

Qualified organization base period amount

2

3

Subtract line 3 from line 2. If zero or less, enter -0-

3

4

Wages for qualified services (do not include wages used in figuring

the work opportunity credit)

4

5

Cost of supplies

5

6

Rental or lease costs of computers (see instructions)

6

7

Enter the applicable percentage of contract research expenses (see

instructions) 

7

8

Total qualified research expenses. Add lines 5 through 8

8

%

9 9

10

Enter average annual gross receipts (see instructions)

10

11

Multiply line 11 by the percentage on line 10

11

12

Subtract line 12 from line 9. If zero or less, enter -0-

12

13

Multiply line 9 by 50% (.50)

13

14

Enter the smaller of line 13 or line 14

14

15

Add lines 1, 4, and 15

15

17 Are you electing the reduced credit under Section 280C? 

16

For Paperwork Reduction Act Notice, see instructions. Cat. No. 13700H Form 6765 (2007)

Enter fixed-base percentage, but not more than 16% (see instructions)

Section A—Regular Credit. Skip this section and go to Section B or C if you are electing or previously elected (and are not revoking)

the alternative incremental credit or the alternative simplified credit, respectively.

Section B—Alternative Incremental Credit. Skip this section if you are completing Section A or C.

Multiply line 29 by 1% (.01)

Subtract line 30 from line 28. If zero or less, enter -0-

Multiply line 29 by 1.5% (.015)

Subtract line 32 from line 28. If zero or less, enter -0-

Subtract line 33 from line 31

Subtract line 35 from line 28. If zero or less, enter -0-

Subtract line 36 from line 33

Basic research payments to qualified organizations (see the line 2

instructions) 

Qualified organization base period amount (see the line 3 instructions)

Subtract line 20 from line 19. If zero or less, enter -0-

19

Multiply line 22 by 20% (.20)

20

Wages for qualified services (do not include wages used in figuring

the work opportunity credit)

21

Add lines 18 and 21

23

Rental or lease costs of computers (see the line 7 instructions)

24

Enter the applicable percentage of contract research expenses (see

the line 8 instructions)

22

Total qualified research expenses. Add lines 24 through 27

26

Enter average annual gross receipts (see the line 11 instructions)

27

19

20

21

23

24

22

26

27

28

29

30

31

32

33

34

35

36

37

28

29

30

31

32

33

34

35

36

37

1 Certain amounts paid or incurred to energy consortia (see instructions) 1

17

16

18 Certain amounts paid or incurred to energy consortia (see the line 1 instructions) 18

If “Yes,” multiply line 16 by 13% (.13). If “No,” multiply line 16 by 20% (.20) and see the instructions 

for the schedule that must be attached. Members of controlled groups or businesses under

common control: see instructions for the schedule that must be attached

Yes No

25 Cost of supplies 25

Multiply line 29 by 2% (.02)

2007

R&D tax credits constitute one type of policy 

instrument. Form 6765, Credit for Increasing 

Research Activities, Internal Revenue Service
22

T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ

T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ 22

SUMMARY OF RECOMMENDATIONS

Theme 1: Understanding Science & InnovaƟ onS®Ä QçÝã®ÊÄ W«ù ®ã ÃããÙÝ RÊÃÃÄã®ÊÄÝ Ä Aã®ÊÄÝ

What are the behavioral foundaƟ ons of innovaƟ on?

Understanding the behavior of individuals and organizaƟ ons is fundamental to developing esƟ mates of the importance of the 

key factors in the innovaƟ on process.

An NSTC Working Group should regularly perform porƞ olio analy- • 

ses of the full spectrum of SoSP across the Federal government and provide advice to the President’s Science Advisor about the results of those analyses. These analyses are tentaƟ vely described in this 

report as a “Federal InnovaƟ on Framework,” which analyses how 

Federal investments and policy decisions aff ect the NaƟ on’s system 

of innovaƟ on. 

NSF and other agencies should conƟ nue to support the develop- • 

ment of a theoreƟ cal foundaƟ on through exisƟ ng programs of 

invesƟ gator-iniƟ ated research. Workshops and informaƟ onal websites can facilitate that dialog.

Individual agencies should work together to idenƟ fy a core suite of • 

ways to measure and describe technology adopƟ on and diff usion. They should also develop ways in which the many scienƟfi c communiƟ es of pracƟ ce for each agency could be described and analyzed. 

Working subgroups such as these would be responsible generaƟ ng 

a report to the larger Working Group.Federal agencies could work with internaƟ onal counterparts to • 

develop a consistent approach to the Science of Science Policy that transcends naƟ onal boundaries, potenƟ ally through the OECD or 

interna onal mee Ɵ Ɵ ng symposia.

What explains technology adopƟ on and diff usion?

IdenƟ fying the drivers of technology adopƟ on is at 

center of the discovery to innovaƟ on cycle.

How and why do communiƟ es 

of science and innovaƟ on form 

and evolve?

Understanding scienƟfi c communiƟ es and their 

associated social networks are a key to understanding the value of knowledge producƟ on. Involving 

scienƟ sts is necessary to guide the development of 

the research quesƟ ons.
T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ 23 T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ 23 

Theme 2: InvesƟ ng in Science and InnovaƟ onS®Ä QçÝã®ÊÄÝ W«ù ®ã ÃããÙÝ RÊÃÃÄã®ÊÄÝ

What is the value of the NaƟ on’s public investment in science?

Developing a way to describe, or think about describing, outcome measures is criƟ cal to moving beyond anecdotal 

evidence.

Agencies (such as NIH, NSF, DHS, CDC, VA, NASA, USGS, and DOE) • should work together to develop a pilot data infrastructure that captures key data about their respecƟ ve scienƟfi c communiƟ es. 

The development would include an assessment of the validity of the new visualizaƟ on techniques to describe the changing structure of science.

Agencies should work together to develop pilot standards for • idenƟ fying ways of measuring the value of knowledge, which 

could then be adapted to the missions of individual agencies. Agencies should work together to develop standard approaches • for using bibliometrics to assess science impact.The academic research community should conƟ nue to be sup- • 

ported to perform the best research to develop new analyƟ cal 

tools, methods, and metrics to support the emerging science of science policy.

Is it possible to “predict discovery”?

A common framework must be adopted so that the quesƟ on can be 

discussed (if not answered).

Is it possible to describe the impact of discovery on innovaƟ on?

The taxpayers, Congress and OMB will hold the scienƟfi c community either 

implicitly or explicitly accountable for Federal research expenditures.

What are the determinants of investment eff ecƟ veness?

Federal scienƟfi c agencies must be 

able to answer this quesƟ on in making 

resource allocaƟ on decisions
24

T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ

T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ 24

 Theme 3: Using the Science of Science Policy to Address NaƟ onal PrioriƟ esS®Ä QçÝã®ÊÄ W«ù ®ã ÃããÙÝ RÊÃÃÄã®ÊÄÝ Ä Aã®ÊÄÝ

What impact does science have on innovaƟ on 

and compeƟƟ veness?

Understanding the ways in which the scienƟfi c enterprise aff ects fi rm compeƟƟ veness, as well as the wages and 

jobs of Americans as described by the America COMPETES Act.

Several core datasets should be established and made available to both • the research and policy community. 

The fi rst of these is a longitudinal dataset on businesses, over- • 

sampling those businesses which are criƟ cal to American compeƟƟ veness, such as high tech, biotech, and mulƟ naƟ onal fi rms. 

ExisƟ ng administraƟ ve data should be used to keep the costs 

manageable and the sample scienƟfi c. 

The second is a longitudinal dataset on the STEM workforce. • Although the original dataset could have a survey basis, every eff ort should be made to exploit longitudinal administraƟ ve 

records, and to partner with other countries in order to capture the long term dynamic adjustments of workers. The link between workers and fi rms must be tracked through • 

administraƟ ve records and other modes of data collecƟ on so 

that the relaƟ onship between the humans, who are the sources 

of innovaƟ ve ideas, and the fi rms, who bring the ideas to market, can be analyzed.

The Federal InnovaƟ on Framework must include feedback loops that • 

analyze the impact of various policy instruments. This will require cooperaƟ on with a wide variety of other Federal agencies, including 

the Department of Treasury, the Census Bureau and the major Federal staƟ sƟ cal agencies to understand the eff ects of tax policy, labor policy 

and other Federal eff orts that impact science policy.

How compeƟƟ ve is the 

U.S. scienƟfi c workforce?

NaƟ onal compeƟƟ veness in science is 

fundamentally aff ected by the ability 

of STEM workers to move across countries, as well as the ability of businesses to tap into the innovaƟ ve capacity 

of a global scienƟfi c community.

What is the relaƟ ve importance of diff erent 

policy instruments in science policy?

At a Ɵ me of limited resources, Federal 

resources must be Ɵ ghtly targeted. As a result, policy makers need to understand the eff ecƟ veness of diff erent 

policy instruments, such as the role of compeƟƟ on and openness in the 

promoƟ on of discovery; the role of 

policy instruments, such as earmarks, peer review, and intellectual property systems; the importance of tax policy and the ways to investment in the appropriate human resources. 
T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ 25

CURRENT AND POTENTIAL TOOLKIT FOR

SCIENCE AND INNOVATION POLICY

An obvious quesƟ on is: “What toolkit of models, tools and metrics is currently available for policy makers 

when making science policy decisions, and what could be available with addiƟ onal focus on the development 

of SoSP?” In order to answer this quesƟ on, the ITG relied on the literature synthesis, the quesƟ onnaire, and 

its own experience. It fi rst idenƟfi ed the models, tools and metrics currently being used by Federal agencies, 

which included:

Models and Tools:

QuanƟ taƟ ve Analysis • DeterminisƟ c Models: Econometric; Risk Modeling; OpƟ ons Modeling; Cost 

Benefi t; Cost Eff ecƟ veness

• StochasƟ c Models: Agent Based; System Dynamics

QualitaƟ ve Analysis • Case Studies; Peer/Expert Review; Delphi; Strategic/Logic

VisualizaƟ on Tools • Network Analysis; Visual AnalyƟ cs; Science Mapping; Scientometrics

Data CollecƟ on Tools • Survey; Web Scraping; AdministraƟ ve Data; Data Mining

Metrics:

Outcome • ScienƟfi c/Micro Level: InnovaƟ on; CompeƟƟ veness; Knowledge Increase

• Program/Porƞ olio: Eff ecƟ veness; Value

• Systems Level: ProducƟ vity; Quality of Life; Workforce CharacterisƟ cs; GDP

Budget and Performance • Earned Value; Process Metrics; Effi ciency; Marginal Cost

Inputs • Bibliometrics: CitaƟ ons; Patents; ScienƟfi c Papers

• Community/Network: Network Value; Eff ecƟ veness; Structure; Workforce

The ITG then idenƟfi ed the dimensions along which the methods, tools and metrics had value for science 

policy as well as the dimensions for assessing the potenƟ al cost if any investment were to be required to 

bring them to full use. Five criteria were idenƟfi ed, three associated with the potenƟ al value of the element: 

Relevance for Science Policy, Breadth of Use, ScienƟfi c Rigor; and three associated with the potenƟ al cost: 

Maturity of the Method or Tool, Availability and Quality of Data Required.

Relevance to Vision: The degree to which the element provides a signifi cant contribuƟ on to 

resolving one or more of the 10 ScienƟfi c Challenges idenƟfi ed by the ITG.

Breadth of Use: The extent of the adopƟ on of the element in the Federal or academic 

science policy context.

ScienƟfi c Rigor: The quality of the scienƟfi c foundaƟ on of the element, in terms of 

publicaƟ ons, scienƟfi c openness, size of community and reproducibility.

Maturity of the Method 

or Tool:

The degree to which the element is used in the Federal or academic science 

policy context.

Availability and 

Qualityof Data Required:

The pracƟ cality of using the element to develop the empirically based 

plaƞ orm for decision making that is the goal of science policy.
26 T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ

T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ 26

In order to provide a visualiza

Ɵ on of state of play of each of these models, tools and metrics, the ITG then created two grids mapping each element. 

Green signi

fi es that the element is currently ful

fi lled with respect to the criterion; yellow signi

fi es that the element is on the way to ful

fi lling the 

criterion; red signi

fi es that there are substan

Ɵ al gaps. 

MODELS/TOOLS

Theme Poten

Ɵ al Value

Value of Investment

Poten

Ɵ al Cost Main Missing Element

Relevance to Vision

Breadth of Use

Scien

Ɵ

fi c 

Rigor

Maturity Access to Inputs

Quan

Ɵ ta

Ɵ ve 

Analysis

Determinis

Ɵ c Models

- Econometric 1,2,3 Data, Community

- Risk Modeling 2,3 Community- Op

Ɵ ons Modeling 2,3 Community

- Cost Bene

fi t2 Community

- Cost E

ff ec

Ɵ veness 2 Community

Stochas

Ɵ c Models

- Agent Based 2 Data, Community

- System Dynamics 2 Community

Qualita

Ɵ ve 

Analysis

- Case Studies 1, 2 Community- Peer/Expert Review 2 Community- Delphi 2 Community- Strategic/Logic 2 Community

Visualiza

Ɵ on 

Tools

- Network Analysis 2,3 Data, Community

- Visual Analy

Ɵ cs 2, 3 Community

- Science Mapping 2,3 Data, Community- Scientometrics 2, 3 Data, Community

Data Collec

Ɵ on 

ToolsTools

- Survey 1,2,3 Data, Community- Web Scraping 1, 2,3 Data, Community- Administra

Ɵ ve Data 1, 2,3 Community

- Data Mining 1,2,3 Data, Community
T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ 27

T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ 27 

METRICS Theme

Poten

Ɵ al Value

Value of Investment

Poten

Ɵ al Cost Main Missing 

Element(s)

Relevance to Vision

Breadth of Use Scien

Ɵ

fi c Rigor Maturity Availability of Inputs

Outcomes

Scien

Ɵ

fi c/Micro Level

Innova

Ɵ on 1,2,3 Data, Community

Compe

Ɵ

Ɵ veness 3 Community

Knowledge Increase 2,3 Data, CommunityProgram/Por

ƞ olio:

Eff ec

Ɵ veness 1,2,3 Community

Value 2,3 Data, CommunitySystems Level:Produc

Ɵ vity 1,2,3 Community

Quality of Life 3 CommunityWorkforce Characteris

Ɵ cs

2,3 Data, Community

GDP 1,2,3 Community

Budget and 

Performance

Earned Value 2 CommunityProcess Metrics 2 Data, CommunityEffi ciency 2 CommunityMarginal Cost 2 Community

Inputs

Bibliometrics:Cita

Ɵ ons 1,2,3 Data, Community

Patents 1,2,3 CommunityScien

Ɵ

fi c Papers 1,2,3 Data, Community

Community/Network:Network Value 1,2,3 CommunityEff ec

Ɵ veness 1,2,3 Community

Structure 1,2,3 Data, CommunityWorkforce 1,2,3
28 T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« ÝÙ« R ÊÃÖ
T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ 29

REFERENCES

1

 30th Annual AAAS Forum on Science and Technology Policy in Washington, D.C., April 21, 2005.

 Text of Dr. Marburger’s speech can be found at:

hƩ p://www.aaas.org/news/releases/2005/0421marburgerText.shtml.

2 AnalyƟ cal perspecƟ ves, Budget of the United States Government FY2009. See chapter on Crosscuƫ ng

 Programs at hƩ p://www.whitehouse.gov/omb/budget/fy2009/pdf/spec.pdf.

3 CommiƩ ee on Science, Engineering, and Public Policy, The NaƟ onal Academies. (2007). Rising Above the

 Gathering Storm: Energizing and Employing America for a Brighter Economic Future. Washington, D.C.:

The NaƟ onal Academies Press.

4

 Marburger, J.H. and McMillin, S.S. (August 14, 2007). FY 2009 AdministraƟ on Research and Development 

 Budget PrioriƟ es. Retrieved from hƩ p://www.ostp.gov/galleries/default-fi le/FY2009FINALOMB-

 OSTPRDPriorityMemo.pdf

5

Marburger, J.H. and Portman, R. (July 23, 2006). FY 2008 AdministraƟ on Research and Development 

 Budget PrioriƟ es. Retrieved from hƩ p://usgeo.gov/docs/OSTP-OMB%20Guidance%20Memo%20

 for%20FY08.pdf

6

 CommiƩ ee on Science, Engineering, and Public Policy, The NaƟ onal Academies. (1999). EvaluaƟ ng 

 Federal Research Programs: Research and the Government Performance and Results Act. Washington, 

 D.C.: The NaƟ onal Academies Press.

7

CommiƩ ee on EvaluaƟ ng the Effi ciency of Research and Development Programs at the U.S. Environmental 

 ProtecƟ on Agency, NaƟ onal Research Council. (2008). EvaluaƟ ng Research Effi ciency in the U.S. 

 Environmental ProtecƟ on Agency. Washington, D.C.: The NaƟ onal Academies Press.

8

OECD Global Science Forum (GSF) Workshop on Science of Science Policy: Developing our Understanding of 

 Public Investments in Science, 12 July 2006, Helsinki, Finland. Text of speech retrieved from hƩ p://

 www.oecd.org/dataoecd/42/63/37470200.pdf

9

CommiƩ ee on EvaluaƟ ng the Effi ciency of Research and Development Programs at the U.S. Environmental 

 ProtecƟ on Agency, NaƟ onal Research Council. (2008). EvaluaƟ ng Research Effi ciency in the U.S. 

 Environmental ProtecƟ on Agency. Washington, D.C.: The NaƟ onal Academies Press.

10Working Party on Technology and InnovaƟ on Policy, OrganisaƟ on for Economic Co-OperaƟ on and 

 Development. (June 11, 2008). Assessing the Socio-Economic Impacts of Public R&D Investment. 

 Retrieved from hƩ p://www.oecd.org/document/10/0,3343,en_2649_34273_40854218_1_1_1_1,00.

 html. 

11Griliches, Z. (1957). Hybrid Corn: An ExploraƟ on of the Economics of Technological Change. Econometrica. 25

 (4), pp. 501-522. 

12Dyer, J.H., Powell, B.C., Sakakibara, M., and Wang, A.J. NaƟ onal InsƟ tute of Standards and Technology. (2006). 

Case study on how R&D alliances between fi rms can accelerate innovaƟ on: DETERMINANTS OF 

 SUCCESS IN R&D ALLIANCES (NISTIR 7323). Washington, D.C.: NIST PrinƟ ng Offi ce.
30 T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ

Auerswald, P., Branscomb, L., Gorman, S., Kulkarni, R., and Schintler, L. NaƟ onal InsƟ tute of Standards and

Technology. (May 2007). Placing InnovaƟ on: IdenƟ fying Emergent Technological AcƟ vity (NIST GCR

06-902), Washington, D.C.: NIST PrinƟ ng Offi ce.

Nail, J. and Brown, H. NaƟ onal InsƟ tute of Standards and Technology. (April 2006). IdenƟ fy Technology Flows

and Spillovers Through NAICS Coding of ATP Project ParƟ cipants (NISTIR 7280). Washington, D.C.: NIST

PrinƟ ng Offi ce.

13Kay, J., “A true and fair view of producƟ vity”, Financial Times, March 2002

14Reed, J.H., Jordan, G. and Vine, E. Energy Effi ciency and Renewable Energy, U.S. Department of Energy. (July

2007). An Impact EvaluaƟ on Framework for Technology Deployment Programs: An Approach for

QuanƟ fying RetrospecƟ ve Energy Savings, Clean Energy Advances, and Market Eff ects. Retrieved

from hƩ ps://apps3.eere.energy.gov/ba/pba/analysis_database/docs/pdf/2007/impact_framework_

tech_deploy_2007_main.pdf

15Ruegg, R. and Feller, I. Advanced Technology Program, NaƟ onal InsƟ tute of Standards and Technology. (July

2003). A Toolkit for EvaluaƟ ng Public R&D Investment Models, Methods, and Findings from ATP’s First

Decade. Retrieved from hƩ p://www.atp.nist.gov/eao/gcr03-857/contents.htm.

16NIST’s labs use business fi nance measures to reveal economic impacts. Tassey, G. NaƟ onal InsƟ tute of

Standards and Technology. (September 2003). Methods for Assessing the Economic Impacts of

Government R&D. Retrieved from hƩ p://www.nist.gov/director/prog-ofc/report03-1.pdf.

17Jordan, G., Hage, J. and Mote, J. (November 1, 2006). A Theory-based Framework for EvaluaƟ ng Diverse

Porƞ olios of ScienƟfi c Work. Presented at the American EvaluaƟ on AssociaƟ on Annual Conference.

Retrieved from hƩ p://www.wren-network.net/resources/2006AEA.htm.

18Stockton, D.J. (March 2, 2002). What Makes a Good Model for the Central Bank to Use? WriƩ en comments

presented at the Conference of the Federal Reserve Bank of San Francisco and the Stanford InsƟ tute for

Economic Policy Research. Retrieved from hƩ p://www.frbsf.org/economics/conferences/0203/panel_

remarks.pdf.

19Solow, R.M. (1956). A ContribuƟ on to the Theory of Economic Growth. Quarterly Journal of Economics. 70 (1),

pp. 65–94.

20Romer, P.M. (1986). Increasing Returns and Long-Run Growth. Journal of PoliƟ cal Economy. 94 (5), pp.

1002-37.

21Jorgenson, D.W. and SƟ roh, K.J. (May 1999). InformaƟ on Technology and Growth. Papers and Proceedings of

the One Hundred Eleventh Annual MeeƟ ng of the American Economic AssociaƟ on. The American

Economic Review 89 (2), pp. 109-115.

22Brécarda, D., Fougeyrollasa, A., Le Mouëla, P., Lemialea, L. and Zagamé, P. (September 2006). Macroeconomic consequences of European research policy: Prospects of the Nemesis model in the year

2030. Research Policy. 35 (7), pp. 910-924.
T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ 31

23U.S. Department of Agriculture, CooperaƟ ve State Research, EducaƟ on, and Extension Service. Business 

 with CSREES-Intellectual Property. (March 10, 2008). Retrieved from www.csrees.usda.gov/business/

 awards/intellprop.htm.

24Centers for Disease Control and PrevenƟ on, NaƟ onal Center for Chronic Disease PrevenƟ on and Health 

 PromoƟ on, Case studies of successful PRC-tested intervenƟ ons. (October 10, 2007). Retrieved from 

hƩ p://www.cdc.gov/prc/tested-intervenƟ ons/index.htm. 

Centers for Disease Control and PrevenƟ on, PrevenƟ on Research Centers, Healthy Aging Research Network 

 Case Study Report. (2007). Retrieved from hƩ p://www.cdc.gov/aging/pdf/HAN_case_study_report.pdf.

25Council on CompeƟƟ veness. (2005). Innovate America. Washington, D.C.: Council on CompeƟƟ veness 

 PrinƟ ng Offi ce. 
32 T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« ÝÙ« R ÊÃÖ
T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ 33

ACRONYMS

ATP Advanced Technology Program (NIST)

CERN European OrganizaƟ on for Nuclear Research (Conseil Européen 

pour la Recherche Nucléaire)

CDC Centers for Disease Control and PrevenƟ on 

AMERICA COMPETES ACT America CreaƟ ng OpportuniƟ es to Meaningfully Promote 

Excellence in Technology, EducaƟ on, and Science Act, 2007

COSEPUP CommiƩ ee on Science, Engineering and Public Policy

CRIS Current Research InformaƟ on System (USDA)

CSREES CooperaƟ ve State Research, EducaƟ on, and Extension Research 

(USDA)

DHHS U.S. Department of Health and Human Services 

DHS U.S. Department of Homeland Security 

DOC U.S. Department of Commerce

DOD U.S. Department of Defense

DOE U.S. Department of Energy

DOEd U.S. Department of EducaƟ on

DOI U.S. Department of Interior

DOT U.S. Department of TransportaƟ on 

EPA Environmental ProtecƟ on Agency

ERS Economic Research Service

FS Forest Service (USDA)

GPRA Government Performance Results Act

GPT General Purpose Technology

IRS Internal Revenue Service
34 T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ

ITG Interagency Task Group

NAS NaƟ onal Academy of Sciences

NASA NaƟ onal AeronauƟ cs & Space AdministraƟ on

NIH NaƟ onal InsƟ tutes of Health

NIST NaƟ onal InsƟ tute of Standards and Technology 

NOAA NaƟ onal Oceanic and Atmospheric AdministraƟ on

NSF NaƟ onal Science FoundaƟ on

NSTC NaƟ onal Science and Technology CommiƩ ee

OMB Offi ce of Management and Budget

OSTP Offi ce of Science and Technology Policy

PART Program Assessment RaƟ ng Tool

R&D Research and Development

ROI Return on Investment

S&E Science and Engineering

S&T Science and Technology 

SciSIP Science of Science and InnovaƟ on Policy

SBE Social, Behavioral, and Economic Sciences

SoSP Science of Science Policy

STEM Science, Technology, Engineering, and MathemaƟ cs

USDA U.S. Department of Agriculture

VA U.S. Department of Veterans Aff airs 
T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ 35

APPENDIX A - CHARTER
36 T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ
T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ 37
38 T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« ÝÙ« R ÊÃÖ
T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« R ÊÃÖ 39

APPPENDIX B – SCIENCE MAGAZINE EDITORIAL
40 T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« ÝÙ« R ÊÃÖ
T«  S ®Ä Ê ¥ S ®Ä P Ê1⁄2®ù : A F Ù1⁄2 R ÝÙ« ÝÙ« R ÊÃÖ 55
Printed with soy ink on recycled paper