U.S. GOVERNMENT PRINTING OFFICE

WASHINGTON : 

For sale by the Superintendent of Documents, U.S. Government Printing Office

Internet: bookstore.gpo.gov Phone: toll free (866) 512–1800; DC area (202) 512–1800

Fax: (202) 512–2104 Mail: Stop IDCC, Washington, DC 20402–0001

67—522PDF 2011

THE MERIT REVIEW PROCESS: 

ENSURING LIMITED FEDERAL RESOURCES ARE 

INVESTED IN THE BEST SCIENCE 

HEARING 

BEFORE THE

SUBCOMMITTEE ON RESEARCH AND SCIENCE 

EDUCATION 

COMMITTEE ON SCIENCE, SPACE, AND 

TECHNOLOGY 

HOUSE OF REPRESENTATIVES 

ONE HUNDRED TWELFTH CONGRESS 

FIRST SESSION 

TUESDAY, JULY 26, 2011 

Serial No. 112–31 

Printed for the use of the Committee on Science, Space, and Technology 

( 

Available via the World Wide Web: http://science.house.gov 
(II) 

COMMITTEE ON SCIENCE, SPACE, AND TECHNOLOGY 

HON. RALPH M. HALL, Texas, Chair 

F. JAMES SENSENBRENNER, JR., 

Wisconsin 

LAMAR S. SMITH, Texas 

DANA ROHRABACHER, California 

ROSCOE G. BARTLETT, Maryland 

FRANK D. LUCAS, Oklahoma 

JUDY BIGGERT, Illinois 

W. TODD AKIN, Missouri 

RANDY NEUGEBAUER, Texas 

MICHAEL T. MCCAUL, Texas 

PAUL C. BROUN, Georgia 

SANDY ADAMS, Florida 

BENJAMIN QUAYLE, Arizona 

CHARLES J. ‘‘CHUCK’’ FLEISCHMANN, 

Tennessee 

E. SCOTT RIGELL, Virginia 

STEVEN M. PALAZZO, Mississippi 

MO BROOKS, Alabama 

ANDY HARRIS, Maryland 

RANDY HULTGREN, Illinois 

CHIP CRAVAACK, Minnesota 

LARRY BUCSHON, Indiana 

DAN BENISHEK, Michigan 

VACANCY 

EDDIE BERNICE JOHNSON, Texas 

JERRY F. COSTELLO, Illinois 

LYNN C. WOOLSEY, California 

ZOE LOFGREN, California 

DAVID WU, Oregon 

BRAD MILLER, North Carolina 

DANIEL LIPINSKI, Illinois 

GABRIELLE GIFFORDS, Arizona 

DONNA F. EDWARDS, Maryland 

MARCIA L. FUDGE, Ohio 

BEN R. LUJA ́ N, New Mexico 

PAUL D. TONKO, New York 

JERRY MCNERNEY, California 

JOHN P. SARBANES, Maryland 

TERRI A. SEWELL, Alabama 

FREDERICA S. WILSON, Florida 

HANSEN CLARKE, Michigan 

SUBCOMMITTEE ON RESEARCH AND SCIENCE EDUCATION

HON. MO BROOKS, Alabama, Chair 

ROSCOE G. BARTLETT, Maryland 

BENJAMIN QUAYLE, Arizona 

STEVEN M. PALAZZO, Mississippi 

ANDY HARRIS, Maryland 

RANDY HULTGREN, Illinois 

LARRY BUCSHON, Indiana 

DAN BENISHEK, Michigan 

RALPH M. HALL, Texas 

DANIEL LIPINSKI, Illinois 

HANSEN CLARKE, Michigan 

PAUL D. TONKO, New York 

JOHN P. SARBANES, Maryland 

TERRI A. SEWELL, Alabama 

EDDIE BERNICE JOHNSON, Texas 
(III) 

C O N T E N T S 

Tuesday, July 26, 2011 

Page 

Witness List ............................................................................................................. 2 

Hearing Charter ...................................................................................................... 3 

Opening Statements 

Statement by Representative Mo Brooks, Chairman, Subcommittee on Research and Science Education, Committee on Science, Space, and Technology, U.S. House of Representatives ............................................................... 10 

Written Statement ............................................................................................ 10 

Statement by Representative Daniel Lipinski, Ranking Minority Member, 

Subcommittee on Research and Science Education, Committee on Science, 

Space, and Technology, U.S. House of Representatives .................................... 11 

Written Statement ............................................................................................ 12 

Witnesses: 

Dr. Cora Marrett, Deputy Director, National Science Foundation 

Oral Statement ................................................................................................. 14 

Written Statement ............................................................................................ 15 

Dr. Keith R. Yamamoto, Vice Chancellor for Research, University of California San Francisco 

Oral Statement ................................................................................................. 21 

Written Statement ............................................................................................ 23 

Dr. Nancy B. Jackson, President, American Chemical Society 

Oral Statement ................................................................................................. 25 

Written Statement ............................................................................................ 27 

Dr. Jorge Jose ́, Vice President for Research, Indiana University 

Oral Statement ................................................................................................. 30 

Written Statement ............................................................................................ 32 

Discussion ................................................................................................................. 36 

Appendix: Answers to Post-Hearing Questions 

Dr. Cora Marrett, Deputy Director, National Science Foundation ...................... 58 

Dr. Keith R. Yamamoto, Vice Chancellor for Research, University of California San Francisco ............................................................................................ 64 

Dr. Nancy B. Jackson, President, American Chemical Society ........................... 65 

Dr. Jorge Jose ́, Vice President for Research, Indiana University ....................... 67 

(1) 

THE MERIT REVIEW PROCESS: 

ENSURING LIMITED FEDERAL RESOURCES 

ARE INVESTED IN THE BEST SCIENCE 

TUESDAY, JULY 26, 2011 

HOUSE OF REPRESENTATIVES, SUBCOMMITTEE ON RESEARCH AND SCIENCE EDUCATION, COMMITTEE ON SCIENCE, SPACE, AND TECHNOLOGY, Washington, DC. 

The Subcommittee met, pursuant to call, at 10:01 a.m., in Room 

2318 of the Rayburn House Office Building, Hon. Mo Brooks 

[Chairman of the Subcommittee] presiding. 
2 
3 

1 Report to the National Science Board on the National Science Foundation’s Merit Review 

Process Fiscal Year 2010, p. 7. 

HEARING CHARTER

COMMITTEE ON SCIENCE, SPACE, AND TECHNOLOGY 

SUBCOMMITTEE ON RESEARCH AND SCIENCE 

EDUCATION 

U.S. HOUSE OF REPRESENTATIVES 

The Merit Review Process: 

Ensuring Limited Federal Resources Are Invested 

in the Best Science 

TUESDAY, JULY 26, 2011

10:00 A.M.–12:00 P.M. 

2318 RAYBURN HOUSE OFFICE BUILDING

Purpose 

On Tuesday, July 26, 2011, the Subcommittee on Research and Science Education 

will hold a hearing to examine the merit review grant award process and its effect 

on federally funded scientific research, in an effort to understand the strengths and 

potential weaknesses of the process. 

Witnesses 

• Dr. Cora Marrett, Deputy Director, National Science Foundation 

• Dr. Keith Yamamoto, Vice Chancellor for Research, University of California San 

Francisco 

• Dr. Nancy Jackson, President, American Chemical Society 

• Dr. Jorge Jose ́, Vice President for Research, Indiana University 

Overview 

• A number of federal agencies, from the Department of Energy to the National 

Institutes of Health and the National Science Foundation, use various types of peer 

or merit review to evaluate proposals and make recommendations to award federal 

funds. 

• The National Science Foundation (NSF) has three funding mechanisms: grants, 

cooperative agreements, and contracts. NSF makes merit-based grant awards to researchers, educators, and students. In Fiscal Year 2010 (FY 10), NSF received 

55,542 proposals and awarded 12,996 grants, a 23 percent funding rate. Fifty percent of its budget was devoted to new and continuing grants. 1

• Approximately 96 percent of NSF grant proposals are evaluated through an external review process, commonly known as the NSF merit review process. The process utilizes subject matter experts to review proposals through the mail, in-person 

at a panel review, or through a combination of both (early-concept grants, rapid response grants, and small conferences and workshops are evaluated through an internal merit review process). 

• The NSF merit review process evaluates proposals based on two criteria, intellectual merit and broader impacts. A National Science Board Task Force is currently examining the two criteria and a report is forthcoming. Since 2007, NSF has 

also been promoting potentially transformative concepts through additional language added to the intellectual merit criteria. 

Background 

Federal funding is disbursed in a number of ways, including through contracts, 

cooperative agreements and grants. The process by which many federal agencies 

evaluate potential grant awards is often termed merit or peer review. This process 

can take several different forms or utilize different processes; however, in general, 
4 

2 Early-concept Grants for Exploratory Research (EAGER) are two-year awards for up to 

$300,000. These awards support quick-response research, or research having a severe urgency 

with regard to availability of data, facilities, or equipment. 3 Report to the National Science Board on the National Science Foundation’s Merit Review 

Process Fiscal Year 2010, p. 32. 

it requires that the grant proposals be reviewed and evaluated by subject matter 

experts not associated with the proposal. 

The National Science Foundation (NSF) uses grants for the majority of its funding 

disbursements. There are two basic grants. A standard award has a duration of 

typically one 09five years, but is fully funded in the first fiscal year. A continuing 

grant, also for a multi-year project, is provided in annual increments. The first year 

of funding for a continuing grant comes with a statement of intent to continue the 

funding with continuing grant increments (CGIs) through completion of the project, 

but the continuation is contingent on whether NSF deems satisfactory progress, 

availability of funds, and the receipt and approval of annual reports. 

Cooperative agreements are used when the project requires substantial agency involvement like research centers and multi-user facilities. 

NSF utilizes an internal merit review process for a fraction of its grant awards, 

including the Early-concept Grants for Exploratory Research (EAGER) and Grants 

for Rapid Response Research (RAPID). 2 However, the bulk of NSF funded grants 

are evaluated through an external grant review process, known as the NSF merit 

review process. 

NSF Merit Review Process 

Grant proposals are required to be submitted electronically. NSF program officers 

ensure each proposal has been assigned to the correct office for review and determine the appropriate level of review (internal or external). Proposals are returned 

without being reviewed if they do not comply with NSF regulations, including separately addressing the intellectual merit and broader impacts criteria. 

Program officers choose proposal reviewers and panel members from an NSF 

database of over 300,000 reviewers. They can also recruit reviewers based on literature searches, professional activities, and other reviewer recommendations. In 

addition, they also screen all reviewers for potential conflicts of interest and provide 

guidance and instructions. 

Reviewers provide comments by mail or through the meeting of a panel session 

(these are most often in person but panelists may also meet virtually). Once reviewers return comments, or a panel convenes, program officers are responsible for synthesizing comments and recommending the award or decline of each proposal. Reviewers provide narratives and categorical ratings which the program officer takes 

into account. An ‘‘Excellent’’ rating does not guarantee the award of funding. In FY 

2010, 3,743 proposals that received an average review of ‘‘Excellent’’ were funded 

and 1,312 were not, and 4,560 proposals that received an average review of ‘‘Very 

Good to Excellent’’ were funded while 6,318 were not. 3
5 

4 Report to the National Science Board on the Nationl Science Foundation’s Merit Review 

Process Fiscal Year 2010, page 29. 5 Ibid. 

The Division Director reviews the program officer’s recommendations and then 

passes them on to the Division of Grants and Awards, who vets the eligibility of 

the awardee, negotiates any necessary changes, and disburses the award. The Director’s Review Board reviews any award in excess of 2.5 percent of the awarding Division’s budget. The National Science Board (NSB) must approve any award in excess 

of $3 million dollars, or one percent or more of the awarding Directorate’s prior year 

current plan, whichever is greater. 

All those who submitted proposals receive notification as to whether or not an 

award will be made. Those to whom funding was declined receive copies of the reviews as well as information on the number of grants awarded and the number of 

proposals in each category. If a proposal is declined, the proposer may ask the program officer for further clarification. If he is still unsatisfied, he may make a reconsideration request to the relevant Assistant Director and a second request to the 

Deputy Director. (See Appendix A for the NSF Flow Chart.) 

Types of Review 

Proposals submitted to the merit review system are reviewed in three ways. 

Through ‘‘mail-only’’ reviews, proposals are sent to reviewers who are asked to submit written comments to NSF. Through ‘‘panel-only’’ reviews, reviewers serve on inperson (or virtual) panels to discuss reviews and provide advice to the program officer. Additionally, some proposals receive a combination of mail and panel review, 

which can take place in a number of ways. 

There is value in each type of review. Mail review allows for better matching between the expertise of reviewers and proposals. Panel review allows for interplay 

between reviewers in the evaluation of proposals and the integration of different 

perspectives in the review of proposals. According to NSF, ‘‘Using panels in the review process tends to reduce proposal processing time (time-to-decision), compared 

to mail-only reviews. For example, in FY 10, 78 percent of all proposals reviewed 

by panel-only were processed within six months, compared to 72 percent for mail 

plus panel and 55 percent for mail-only.’’ 4 While in-person panels are most common, ‘‘virtual panels’’ are being convened more often. Virtual panels allow reviewers 

to participate from remote locations using interactive technology. The combination 

of mail and panel reviews ‘‘is used frequently because it combines the in-depth expertise of mail review with the comparative analysis of panel review.’’ 5
6 

6 Report to the National Science Board on the Nationl Science Foundation’s Merit Review 

Process Fiscal Year 2010, page 34. 7 Ibid. 8 Report to the National Science Board on the Nationl Science Foundation’s Merit Review 

Process Fiscal Year 2010, p. 26. 9 Report to the National Science Board on the Nationl Science Foundation’s Merit Review 

Process Fiscal Year 2010, pp. 21 0922. 10 NSB Task Force on Merit Review, http://www.nsf.gov/nsb/committees/ 

tskforce¥mr¥charge.jsp. 

The Program Officer 

NSF program officers are made up of permanent (54 percent) and non-permanent 

(46 percent) employees; 6 all are subject matter experts in the areas they manage 

with advanced degrees or credentials. ‘‘Some non-permanent program officers are ‘on 

loan’ as ‘Visiting Scientists, Engineers, and Educators’ (VSEEs) for up to three years 

from their host institutions. Others are supported through grants to the home institutions under the terms of the Intergovernmental Personnel Act (IPA).’’ 7 These ‘‘rotators’’ ensure that new and fresh scientific ideas and specialties come through the 

Foundation and help to prevent institutional or innovative stagnation. One drawback, however, is the loss of institutional knowledge when a rotator leaves and the 

challenge of frequently bringing new rotators up to speed on NSF policies and processes. 

The Foundation expects program officers to administer balanced portfolios and requires them to utilize the advice and expertise of the proposal reviewers while assessing proposals in terms of each portfolio. In order to create a balanced portfolio, 

program officers are expected to additionally evaluate proposals for a number of criteria, including, but not limited to: geographic distribution; novel approaches to significant research questions; capacity building in a new and promising research area; 

potential impact on the development of human resources and infrastructure; and 

NSF core strategies, including integration of research and education, broadening 

participation, and promoting partnerships. 8

Merit Review Criteria 

Since initial approval in 1997, every NSF grant proposal has been reviewed based 

on two merit review criteria, intellectual merit and broader impacts. While additional consideration may be given for a number of reasons including special requirements of the program, intellectual merit and broader impacts remains the cornerstone of the NSF merit review process. 

• Intellectual Merit. What is the intellectual merit of the proposed activity? How 

important is the proposed activity to advancing knowledge and understanding within its own field or across different fields? How well qualified is the proposer (individual or team) to conduct the project? (If appropriate, the reviewer will comment 

on the quality of prior work.) To what extent does the proposed activity suggest and 

explore creative, original, or potentially transformative concepts? How well conceived and organized is the proposed activity? Is there sufficient access to resources? 

• Broader Impacts. What are the broader impacts of the proposed activity? How 

well does the activity advance discovery and understanding while promoting teaching, training, and learning? How well does the proposed activity broaden the participation of underrepresented groups (e.g., gender, ethnicity, disability, geographic, 

etc.)? To what extent will it enhance the infrastructure for research and education, 

such as facilities, instrumentation, networks and partnerships? Will the results be 

disseminated broadly to enhance scientific and technological understanding? What 

may be the benefits of the proposed activity to society? 9

The America COMPETES Reauthorization Act of 2010 expands the broader impacts criteria to include activities to achieve the following goals: (1) increase the economic competitiveness of the United States; (2) development a globally competitive 

STEM workforce; (3) increase participation of women and underrepresented minorities in STEM; (4) increase partnerships between academia and industry; (5) improve 

pre-K 0912 STEM education and teacher development; (6) improve undergraduate 

STEM education; (7) increase public scientific literacy; and (8) increase national security. 

In February 2010, the NSB reconstituted the Task Force on Merit Review. The 

Board charged the Task Force with ‘‘examining the two Merit Review Criteria and 

their effectiveness in achieving the goals for NSF support for science and engineering research and education.’’ 10
7 

11 Report to the National Science Board on the National Science Foundations’s Merit Review 

Process Fiscal Year 2010, p. 8. 12 Report to the National Science Board on the National Science Foundations’s Merit Review 

Process Fiscal Year 2010, pp. 10 0911. 

In June 2011, after a year of review, NSB and NSF put out a call for public comment (closed July 14, 2011) on proposed revisions to the merit review criteria. The 

proposed changes maintain the themes of intellectual merit and broader impacts 

while establishing key principles of the merit review criteria. The proposed changes 

include the identification of national goals which every NSF project should seek to 

advance, including but not limited to: the increased economic competitiveness of the 

United States; the increased participation of women, persons with disabilities, and 

underrepresented minorities in STEM; the increased public scientific literacy and 

public engagement with science and technology; and increased national security. 

(See Appendix B for the complete proposal.) 

Potential Challenges 

While the NSF merit review process is widely considered the most effective of its 

type for the awarding of federal funding, there are existing challenges to be considered in an effort to strengthen the process. Questions remain about the way in 

which scientific priorities are established and whether the process is truly supporting innovative research and researchers. Below are some additional challenges: 

• Transformative Research Research that questions existing science often faces 

additional hurdles when facing review by scientific researchers in that field, especially during lean economic times as experts favor more conservative funding approaches. Since 2007, NSF has been working to ensure transformative research is 

considered appropriately and such proposals are provided an opportunity to compete 

through the merit review process, including adding explicit language in the intellectual merit criteria for transformative concepts. EAGER grants are intended to be 

used, in part, to fund potentially transformative ideas for which there is little to no 

preliminary data and, as such, would fare poorly in the standard merit review process. NSF has also incorporated efforts to encourage transformative research in its 

training of program officers and reviewers. It is also experimenting with modifications in the review process to help identify transformative proposals. Are these efforts working? Is there more to be done, within the process, to encourage transformative science? 

• Ensuring a Pipeline for U.S. Students by Encouraging New Principal Investigators New Principal Investigators (PIs) often do not have the same level of experience 

or access to resources that established PIs have, both considerations included as 

part of the intellectual merit criteria. In FY 10, new PIs submitted 21,545 proposals 

and received 3,620 awards, a funding rate of 17 percent; prior PIs were funded at 

a rate of 28 percent. 11 A strong system properly encourages new investigators to 

participate in the scientific arena in order to ensure a pipeline for U.S. student participation in scientific endeavors. The Faculty Early Career Development (CAREER) 

Program offers specific funding opportunities for new PIs to help in this endeavor, 

but is this enough? Does the merit review process encourage the participation of 

new PIs? 

• Parity for Institutions Institutions that are not regular grant recipients do not 

always have the same resources or proficiencies as those institutions that consistently receive federal funding. ‘‘For FY 10, the average funding rate was 26 percent 

for the top 100 Ph.D.-granting institutions (classified according to the amount of FY 

10 funding received). In comparison, the rate was 17 percent for Ph.D.-granting institutions that are not in the top 100 NSF-funded category.’’ 12 Are those institutions, not regularly in receipt of federal funding, encouraged to submit grant proposals and participate in the merit review process? 

• Multidisciplinary Review As NSF seeks to grow its multidisciplinary projects, 

the merit review process must consider the management of reviews that incorporate 

a combination of scientific disciplines in order to fund the strongest multidisciplinary proposals. Is the current process able to effectively encourage and evaluate 

multidisciplinary projects? 

In all, the merit review process must continue to balance these challenges with 

the inherent need to fund the strongest science. 
8 
9 
10 

Chairman BROOKS. The Subcommittee on Research and Science 

Education will come to order. 

Good morning. Welcome to today’s hearing entitled ‘‘The Merit 

Review Process: Ensuring Limited Federal Resources Are Invested 

in the Best Science.’’ I am going to give an opening statement. 

Our hearing today presents us with an opportunity to examine 

the merit review process for awarding federal grant funds. It is our 

goal to highlight the benefits of the process, while acknowledging 

that no process involving human decision making is flawless. The 

focus of today’s hearing will primarily be on the merit review process at the National Science Foundation. 

The National Science Foundation Act of 1950 directs NSF to initiate and support basic scientific research and programs to 

strengthen scientific research potential and science education programs at all levels. NSF works to accomplish this Congressional directive through the issuance of merit-based awards to researchers, 

educators and students at approximately 1,900 U.S. colleges, universities and other institutions. In 1994, the National Academies 

touted it as among the best procedures known for insuring the 

technical excellence of research projects that receive public support, 

but the process has changed since then, and we need to make sure 

that is still the case. 

As we know, a large number of potentially fundable proposals 

are declined each year. The Foundation received over 55,000 proposals for funding in fiscal year 2010 and funded nearly 13,000, or 

23 percent, of them. Many of the proposals received were not worthy of federal funding, but it is also true that many were not funded because federal funds are limited. So given that those limited 

dollars should go to the very best scientific research, NSF must 

maintain a robust and transparent merit review process. 

Today, our witnesses will share their thoughts on how the process works and its strengths and weaknesses. We want to know if 

the current process spurs or stifles innovation, how award decisions 

are actually made after receiving peer review, and if there are 

flaws in the system that may be providing precious federal funds 

to lower rated proposals over more highly rated proposals. 

In exercising its oversight role, this Subcommittee must ensure 

that federal dollars are being spent on the best science. This examination of the merit review process will help us to understand how 

programmatic funding decisions are made and how, in turn, those 

decisions interact with Foundation-wide priorities. 

I look forward to hearing from each of our witnesses on this important topic. I thank you for joining us. 

[The prepared statement of Mr. Brooks follows:] 

PREPARED STATEMENT OF CHAIRMAN MO BROOKS

Good morning, and welcome to each of our witnesses. Our hearing today presents 

us with an opportunity to examine the merit review process for awarding federal 

grant funds. It is our goal to highlight the benefits of the process, while acknowledging that no process involving human decision making is flawless. The focus of 

today’s hearing will primarily be on the merit review process at the National 

Science Foundation (NSF). 

The National Science Foundation Act of 1950 directs NSF to ‘‘to initiate and support basic scientific research and programs to strengthen scientific research potential and science education programs at all levels.’’ NSF works to accomplish this 

Congressional directive through the issuance of merit-based awards to researchers, 
11 

educators and students at approximately 1,900 U.S. colleges, universities, and other 

institutions. In 1994, the National Academies touted it as being among ‘‘the best 

procedures known for insuring the technical excellence of research projects that receive public support,’’ but the process has changed since then, and we need to make 

sure that is still the case. 

As we know, a large number of potentially fundable proposals are declined each 

year. The Foundation received over 55,000 proposals for funding in Fiscal Year 2010 

and funded nearly 13,000, or 23 percent, of them. Many of the proposals received 

were not worthy of federal funding, but it is also true that many were not funded 

because federal funds are limited. So, given that those limited dollars should go to 

the very best scientific research, NSF must maintain a robust and transparent 

merit review process. 

Today, our witnesses will share their thoughts on how the process works and its 

strengths and weaknesses. We want to know if the current process spurs or stifles 

innovation, how award decisions are actually made after receiving peer review, and 

if there are flaws in the system that may be providing precious federal funds to 

lower rated proposals over more highly rated proposals. 

In exercising its oversight role, this Subcommittee must ensure that federal dollars are being spent on the best science. This examination of the merit review process will help us to understand how programmatic funding decisions are made and 

how, in turn, those decisions interact with Foundation-wide priorities. 

I look forward to hearing from each of our witnesses on this important topic; 

thank you for joining us. 

Chairman BROOKS. At this point, I recognize Mr. Lipinski, who 

is the Ranking Member of this Subcommittee. 

Mr. LIPINSKI. Thank you, Chairman Brooks, and thank you for 

holding the hearing this morning. 

It will not surprise you when I say that I have had a great interest in this topic since I first submitted a grant proposal to the National Science Foundation when I was a graduate student. That interest continued throughout my academic career, and when I was 

elected to Congress I joined what was then the Science and Technology Committee to a large extent—actually it was the Science 

Committee at the time when I joined—to a large extent because of 

my strong belief in the NSF and its mission. 

I agree with the statements of all of the witnesses here today 

that NSF’s merit review system remains the gold standard for the 

world. And I don’t say that just because that NSF grant application 

I submitted when I was in grad school was successful. 

At the same time, I recognize that there are challenges in any 

system for allocating limited research dollars. I agree with Chairman Brooks that it is our job on this Subcommittee to hold hearings such as this one to discuss these challenges and collectively 

imagine how we might continue to make NSF, and the merit-review system that it manages, even stronger. Particularly in this 

tight budget environment, it is incumbent upon us all to make sure 

that the system for funding excellent science is as efficient and effective as possible. 

In 2009, when I was Chair of this Subcommittee, we looked at 

a few slices of the broader topic being addressed today when we 

held a hearing on high-risk, high-reward, or what we also call 

transformative research. Dr. Yamamoto was not on the panel for 

that hearing, but a report that he helped author, the ARISE report, 

played a central role and remains relevant today. I look forward to 

learning from his deep expertise on this topic. 

But there are many issues that we have not examined in detail, 

including the extent to which faculty from lesser-resourced institutions face an uneven playing field. I am also interested in the ex-
12 

tent to which the institutional structures underlying NSF’s peer-review system influence decisions and the benefits and drawbacks of 

different approaches to peer review. 

For example, I am intrigued by proposals to conduct committee 

review in virtual environments such as Second Life. While I am 

perhaps a little skeptical, I recognize that virtual review has the 

potential to save taxpayers a lot of money in travel expenses, as 

well as broaden the pool of reviewers. So I am glad that the NSF 

is looking into innovative approaches. But we need to be confident 

that the group dynamics in a virtual environment, while certain to 

be different, do not in any way undermine the quality of NSF merit 

review. 

I am particularly interested in hearing the panel’s recommendations about some of the alternatives to standard merit review, not 

as a replacement of, but rather as a complementary approach. For 

example, in last year’s COMPETES Act, I authored a provision 

that authorizes prize programs at all of the science agencies. While 

NSF, as a basic research agency, would need to design and implement a prize program that looks very different from those run by 

DARPA or NASA, I continue to believe that the NSF should experiment with some pilot projects to award research prizes. I look forward to hearing an update from Dr. Marrett on the Foundation’s 

thoughts on this subject. 

So I see this hearing as an opportunity to examine the real challenges that do exist in a very strong review system and to discuss 

current and novel approaches to overcoming those challenges. 

Finally, Mr. Chairman, I think it is important to point out that 

the National Science Board is in the middle of a process to review 

and revise the existing merit review criteria. While I believe this 

hearing is important, it is critical that in the fall when the Board 

has finished its work and produced a new set of review criteria for 

the scientific community, this Committee should examine what the 

Board has come out with. Especially in light of the provision in last 

year’s COMPETES bill directing the agency to clarify the purpose 

and implementation of the broader impacts review criterion, it will 

be helpful at some point later this year or early next year to revisit 

this topic, and I certainly hope that we can do that. 

With that, Mr. Chairman, I thank all of the witnesses for being 

here today and I yield back. 

[The prepared statement of Mr. Lipinski follows:] 

PREPARED STATEMENT OF RANKING MEMBER DANIEL LIPINSKI

Thank you, Chairman Brooks, and thank you for holding this hearing this morning. It will not surprise anyone when I say that I have had a great interest in this 

topic since I first submitted a grant proposal to the National Science Foundation 

as a graduate student. That interest continued throughout my academic career and 

when I was elected to Congress I joined what was then the Science and Technology 

committee to a large extent because of my strong belief in the NSF and its mission. 

I agree with the statements of all of the witnesses here today that NSF’s merit-review system remains the gold standard for the world. And I don’t say that just because that NSF grant application I submitted when I was in grad school was successful. 

At the same time, I recognize that there are challenges in any system for allocating limited research dollars. I agree with Chairman Brooks that it is our job, on 

this Subcommittee, to hold hearings such as this one to discuss these challenges and 

collectively imagine how we might continue to make NSF, and the merit-review system that it manages, even stronger. Particularly in this tight budget environment 
13 

it is incumbent upon us all to make sure that the system for funding excellent 

science is as efficient and effective as possible. 

In 2009, when I was chair of this Subcommittee, we looked at a few slices ofthe 

broader topic being addressed today when we held a hearing on high-risk, high-reward, or what we also call transformative research. While Dr. Yamamoto was not 

on the panel for that hearing, a report that he helped author, the ARISE report, 

played a central role and remains relevant today. I look forward to learning from 

his deep expertise on this topic. 

But there are many issues that we have not examined in detail, including the extent to which faculty from lesser resourced institutions face an uneven playing field. 

I am also interested in the extent to which the institutional structures underlying 

NSF’s peer-review system influence decisions and the benefits and drawbacks of different approaches to peer review. 

For example, I am intrigued by proposals to conduct committee review in virtual 

environments such as Second Life. While I am perhaps a little skeptical, I recognize 

that virtual review has the potential to save taxpayers a lot of money in travel expenses, as well as to broaden the pool of reviewers. So I’m glad that the NSF is 

looking into innovative approaches. But we need to be confident that the group dynamics in a virtual enviromnent, while certain to be different, do not in any way 

undermine the quality of NSF merit review. 

I’m particularly interested in hearing the panel’s recommendations about some of 

the alternatives to standard merit review, not as a replacement of, but rather as 

a complementary approach. For example, in last year’s COMPETES Act, I authored 

a provision that authorizes prize programs at all ofthe science agencies. While NSF, 

as a basic research agency, would need to design and implement a prize program 

that looks very different from those run by DARPA or NASA, I continue to believe 

that the NSF should experiment with some pilot projects to award research prizes. 

I look forward to hearing an update from Dr. Marrett on the Foundation’s thoughts 

on this subject. 

So I see this hearing as an opportunity to examine the real challenges that do 

exist in a very strong review system and to discuss current and novel approaches 

to overcoming those challenges. 

Finally, Mr. Chairman (as you already mentioned), I think it’s important to point 

out that the National Science Board is in the middle of a process to review and revise the existing merit review criteria. While I believe this hearing is critically important, it is unfortunate that we couldn’t wait until this fall when the Board has 

finished its work and produced a new set of review criteria for the scientific community and this Committee to examine. Especially in light of the provision in last 

year’s COMPETES bill directing the agency to clarify the purpose and implementation of the Broader Impacts Review Criterion, it will he helpful at some point later 

this year or next year to revisit this topic. 

With that, Mr .Chairman, I thank all of the witnesses for being here today and 

I yield back. 

Chairman BROOKS. Thank you, Mr. Lipinski. 

At this time I would like to introduce our witnesses for today’s 

hearing. First, we have Dr. Cora B. Marrett, the Deputy Director 

of the National Science Foundation. Since January 2009, she has 

served as NSF’s Acting Deputy Director and Senior Advisor until 

her confirmation as Deputy Director in May 2011. Dr. Marrett 

holds a bachelor of arts degree from Virginia Union University, a 

master of arts and a doctorate from the University of WisconsinMadison, all in sociology. 

Dr. Keith Yamamoto is Vice Chancellor for Research at the University of California at San Francisco. Dr. Yamamoto served on 

grant review panels for the NSF Biology Directorate, and as an ad 

hoc member of the National Science Board Taskforce on Transformative Research. 

Dr. Nancy Jackson serves as President of the American Chemical 

Society. Dr. Jackson is employed at Sandia National Laboratories 

and earned her Ph.D. in chemical engineering from the University 

of Texas at Austin. 
14 

And Dr. Jorge Jose ́ is the Vice President for Research at Indiana 

University. A theoretical physicist and neurobiologist, Dr. Jose ́ received his undergraduate, master’s and Ph.D. in the Department of 

Physics at the National Autonomous University of Mexico. 

As our witnesses should know, spoken testimony is limited to 

five minutes each, after which the members of the committee will 

have five minutes each to ask questions. 

I now recognize our first witness, Dr. Cora Marrett, for five minutes. 

STATEMENT OF DR. CORA MARRETT, 

DEPUTY DIRECTOR, NATIONAL SCIENCE FOUNDATION 

Dr. MARRETT. Thank you, Chairman Brooks, Ranking Member, 

Lipinski, and the distinguished Members of the Subcommittee 

staff. I am pleased to have this opportunity to discuss the National 

Science Foundation’s merit review process. 

For over 60 years, NSF has been a steward of the Nation’s 

science and engineering enterprise with a proven track record for 

producing results. Despite its relatively small size, NSF has an important impact on scientific and engineering knowledge and academic capacity. NSF’s investments in discovery, learning and innovation have been important to increasing America’s competitiveness, our economic strength, national security and overall quality 

of life. 

NSF relies on a merit-based competitive process that is critical 

to fostering the highest standards of excellence and accountability. 

The process lies at the heart of the agency’s strategy for accomplishing its overall mission. 

Now, of the 256 American Nobel Prize recipients in science since 

NSF first began to award research grants, of those 256, approximately 75 percent have received NSF funding at some point in 

their careers. That includes 19 individuals in the last five years. 

The agency has a strong record of funding the most insightful 

ideas and visionary investigators. Our model of high-quality merit 

review has been emulated and replicated by other nations. Our 

merit review process helps assure that awards made by NSF are 

of the highest quality, are relevant to our goals and objectives and 

have an appropriate balance for the resulting portfolio. This assurance is critical as nearly 90 percent of NSF’s funding is allocated 

through merit review, allocated either as grants or cooperative 

agreements. 

As you have already noted, in the fiscal year 2010, NSF evaluated over 55,000 proposals through this process and made approximately 13,000 new awards. This entailed conducting about 287,000 

proposal reviews and engaging nearly 46,000 members of the 

science and engineering community as reviewers. Underlying these 

statistics is a strategy that helps ensure that each proposal submitted to NSF is reviewed in a fair, competitive, transparent, and 

in-depth manner. 

First, our program officers are experts in the scientific and engineering programs they manage. They hold strong credentials, usually a Ph.D. or equivalent in a STEM field, and their funding recommendations are subject to several additional layers of review. 
15 

Each proposal submitted to NSF is reviewed by these in-house experts, and they in turn rely extensively on external experts to keep 

NSF-funded research at the frontier. As we emphasize, this is a 

unique aspect of NSF, the knowledge of our program officers, who 

are then empowered to make recommendations on funding. 

Each proposal must meet the highest standards in terms of two 

merit review criteria: intellectual merit and broader impacts. Intellectual merit encompasses the potential of the research to advance 

knowledge, the originality and creativity of the proposed activity 

and the qualifications of the researchers. Broader impacts include 

technological innovation, societal benefits and opportunities to include a diversity of participants. 

Finally, the merit review process itself is constantly assessed 

through evaluations by committees of visitors, advisory committees 

and other stakeholders. NSF continuously strives to maintain and 

improve the quality and transparency of the process. 

So in summary, the National Science Foundation is dedicated to 

ensuring that the merit review process remains robust, rigorous, 

and beyond reproach. This process enables us to carry out our mission. I appreciate, then, this opportunity to appear before the Subcommittee on this important topic and would be pleased to answer 

any questions you might have. 

[The prepared statement of Dr. Marrett follows:] 

PREPARED STATEMENT OF DR. CORA MARRETT, 

DEPUTY DIRECTOR, NATIONAL SCIENCE FOUNDATION

Chairman Brooks, Ranking Member Lipinski, and distinguished Members of the 

Subcommittee, thank you for inviting me to participate in this hearing on ‘‘The 

Merit Review Process.’’ 

I am delighted to discuss the National Science Foundation’s (NSF) Merit Review 

Process with you. As you well know, NSF is the primary Federal agency supporting 

research at the frontiers of knowledge, across all fields of science and engineering 

(S&E) and all levels of S&E education. Its mission, vision and goals are designed 

to maintain and strengthen the vitality of the U.S. science and engineering enterprise. As part of the overall national R&D enterprise, the basic research and education activities supported by NSF are vital to the economic advancement of the 

U.S. and provide the know-how that allows the U.S. to respond rapidly and effectively to a range of unexpected challenges. The NSF merit review process lies at the 

heart of the agency’s strategy for accomplishing its overall mission. As such, NSF 

is continuously striving to maintain and improve the quality and transparency of 

the process. 

Before I begin my discussion of the unique elements of the NSF merit review system, let me first describe the essential features of merit review writ large. In general, merit review refers to an independent assessment of a plan’s worthiness. The 

Code of Federal Regulations (Section 600.13 of title 10) defines Merit Review as a 

‘‘thorough, consistent and objective examination of applications based on pre-established criteria by persons who are independent of those individuals submitting the 

applications and who are knowledgeable in the field of endeavor for which support 

is requested.’’ 

I would also like to note here that although the terms ‘‘merit review’’ and ‘‘peer 

review’’ are often used interchangeably, they are not equivalent terms. NSF made 

this distinction clear back in 1986, based on a report from an external Advisory 

Committee on Merit Review, established by then-director Erich Bloch at the request 

of the National Science Board. As is described by Marc Rothenberg, the NSF historian, in his 2010 article ‘‘Making Judgments about Grant Proposals: A Brief History 

of the Merit Review Criteria at the National Science Foundation:’’ 

• According to the committee, the term ‘‘peer review’’ was properly a restrictive 

term referring to the evaluation of the technical aspect of the proposal. However, 

for more and more federally funded research, ‘‘technical excellence’’ was, in the 

words of the committee, ‘‘a necessary but not fully sufficient criterion for research 
16 

funding.’’ Acknowledging that the NSF (as well as other federal agencies) was using 

a wide range of nontechnical criteria as part of the decision-making process, the 

committee suggested that the term ‘‘merit review’’ more accurately described the 

NSF selection process. 

• The committee’s recommendation was accepted by Director Bloch, and since 

then NSF has used the term ‘‘merit review’’ to describe our process. 

Since its founding, NSF has relied on the merit review process to allocate the vast 

majority of its funding. As in other agencies, this has involved the use of proposals 

from prospective researchers that are judged on their merits by knowledgeable persons. But there are several elements that give merit review at the NSF its distinct 

features. For one, right from the beginning, NSF utilized the project grant mechanism (as opposed to a contract mechanism) for providing funds. This was a rather 

radical concept back in 1951, when most government operations used contracts. 

Since that time, the use of the grant mechanism has been adopted by many federal 

extramural research funding organizations. 

NSF’s process for deciding which proposals to fund differs from the approach of 

a number of other funding agencies and organizations (such as philanthropic foundations) nationally and internationally. Perhaps the most distinctive differences are 

our reliance on expertise from both outside and within the Foundation, and the discretionary authority vested in the NSF program officer to make funding recommendations. Unlike many philanthropic foundations (and even some federal research funding programs), NSF policy requires that the program officers seek external expert advice before making most of their funding recommendations. However, 

in contrast to a number of other funding bodies, the external reviewers do not make 

binding recommendations that the program officer is obliged to follow, although program officers always pay close attention to all external reviews. Because of the responsibility we give our program officers, NSF sets a high standard for excellence 

in that position. Our program officers are subject matter experts in the scientific 

areas that they manage, and bring strong credentials with them, including advanced 

educational training (e.g., a Ph.D. or equivalent credentials) in science or engineering, and deep experience in research, education, and/or administration. 

NSF has chosen to give the program officer the responsibility for making funding 

recommendations to enable a more strategic and long-term approach for building 

the award portfolio. As important as the input of the external scientific experts is, 

they have only a snapshot view of the current set of proposals they are evaluating. 

The NSF program officer is responsible for putting that snapshot view into the larger context of the entire award portfolio they are managing, which can lead to a more 

diverse and robust portfolio overall. Together with the division directors, who have 

the authority to review and act on the program officers’ recommendations, program 

officer teams are poised to identify promising research that responds to national priorities identified by Congress and the Administration. In addition, program officers 

can incorporate agency or programmatic priorities, which are articulated in the annual agency budget, special solicitations, and standing program descriptions, all of 

which are available to the community via the NSF Web site. 

The NSF merit review process is described in full detail on the NSF Web site 

(http://www.nsf.gov/bfa/dias/policy/meritreview/). There is also a summary of the 

major steps in the merit review process in the annual Report to the National 

Science Board on the Merit Review Process (the most recent report covering activities in FY 2010 can be found at http://www.nsf.gov/nsb/publications/2011/ 

nsb1141.pdf). It is worth noting here that the key features of the NSF process have 

remained remarkably stable over time. Any changes that have been incorporated 

have sought primarily to clarify the process and make it more transparent. For example, initially only excerpts of the external reviews were shared with the proposal 

authors. Over time, NSF provided the verbatim reviews (but not the identities of 

the reviewers) to the applicant. Similarly, over time there have been modifications 

to the number and clarity of the review criteria. In the America COMPETES Reauthorization Act, the broader impacts criterion is specifically mentioned, and the National Science Board is in the process of analyzing the many comments received on 

this topic. 

A flowchart that graphically depicts the major steps in the merit review process 

and a timeline is attached to this testimony as Appendix I. These steps include: 

• Assignment to the appropriate program for review. Principal investigators initiate this process by selecting the program or programs to which they wish to submit their proposal. Once submitted, the cognizant program officers for those programs confirm that the assignment is appropriate. On occasion, a proposal may be 

reassigned to another program where there is a better fit. During this initial assignment process, it is not uncommon for proposals to be assigned to multiple programs 
17 

for review, if the subject is interdisciplinary in nature, or if the question is of interest and relevance to more than one program. 

• Administrative review of all proposals for compliance with NSF regulations. 

These regulations, which are intended to ensure fairness in the review process, are 

described in the Grant Proposal Guide, which is widely available to the NSF community on the NSF Web site (http://www.nsf.gov/pubs/policydocs/pappguide/ 

nsf11001/nsf11¥1.pdf). Proposals that do not comply with these regulations may be 

returned without review. 

• Merit review of all proposals that pass the administrative review. As noted 

above, a critical feature of NSF’s process is the use of both external review by experts in the field and internal review by NSF’s corps of program officers. The program officers are responsible for administering the merit review process from beginning to end, starting with identifying and recruiting appropriate peer reviewers 

from the external community to serve either as individual reviewers for a particular 

proposal (referred to as ‘‘ad hoc’’ reviewers) or as members of a panel of reviewers 

who evaluate a larger set of proposals. To ensure that they receive substantive reviews from a variety of perspectives, the program officers reach out to a broad range 

of experts for input—in fiscal year 2010, over 46,000 external peer reviewers from 

academia, government, and occasionally industry provided authoritative advice to 

the Foundation. Selection of expert peer reviewers may be based on the program 

officer’s knowledge, references listed in the proposal, individuals cited in recent publications or relevant journals, presentations at professional meetings, reviewer recommendations, bibliographic and citation databases, or suggestions from the proposal author (subject to the program officer’s discretion). In making these selections, 

program officers pay very careful attention to avoiding conflicts of interest, both real 

and perceived. 

NSF takes seriously its responsibility to ensure that the merit review process is 

fair and equitable. One of the ways in which we address this responsibility is 

through the briefings that are given to each review panel before it begins its work. 

In these briefings, panelists are instructed on NSF’s review criteria (Intellectual 

Merit and Broader Impacts), and on maintaining confidentiality and avoiding conflicts of interest. In addition, review panel briefings typically include alerting the 

reviewers to the phenomenon of implicit bias, which may adversely impact new investigators, smaller institutions, and underrepresented groups. By guarding against 

the effects of implicit bias in the review process, NSF is working to ensure that 

there are equitable opportunities for all investigators. 

I should note here that while the vast majority of the proposals received at NSF 

(96%) are subject to both external and internal merit review, for some proposals the 

external review requirement is waived. This waiver provides necessary flexibility for 

handling proposals for which most of the external community would be conflicted 

(such as proposals for small conferences, workshops, or symposia), those for which 

there is a severe urgency (submitted through the Grants for Rapid Response Research, or RAPID, mechanism used, for example, on rapid-response research to the 

Deepwater Horizon oil spill), and those that request support for high-risk, potentially transformative exploratory work (submitted through the Early Grants for Exploratory Research, or EAGER, mechanism). These proposals are usually only reviewed internally by program officers with appropriate expertise. 

• Development of funding recommendations. A central tenet of the NSF merit review process is that the reviewer input is advisory in nature. Funding recommendations are developed by the program officer, who is responsible for synthesizing the 

advice of the reviewers along with several other factors, with the goal of allocating 

funding to a diverse portfolio of projects that addresses a variety of considerations 

and objectives. In addition to their scientific expertise noted above, NSF program 

officers bring their own unique perspective born from their experience of working 

with hundreds, thousands, or—in some cases—tens of thousands of proposals. In developing recommendations within the larger context of their overall portfolio, program officers consider carefully the individual merits of each proposal with respect 

to both its intellectual merit and the potential broader impacts of the project, and 

how each proposal might help advance a variety of portfolio goals such as: 

• Achieving special program objectives and initiatives; 

• Fostering novel approaches to significant research and education questions; 

• Building capacity in a new and promising research area; 

• Supporting high-risk proposals with potential for transformative advances; 

• Supporting NSF’s core strategies of integration of research and education 

and integrating diversity into NSF’s programs; 
18 

• Potential impact on human resources and infrastructure; 

• Other available funding sources; and 

• Geographic distribution. 

NSF has set a goal for completing this process within six months, from the time 

the proposal is submitted to the point at which the proposal is either declined or 

recommended for funding and forwarded to the Division of Grants and Agreements 

for the final stages of review and processing. The proposal assignment and administrative review stage is typically complete within a few weeks. The bulk of the time 

is spent in the merit review stage, which can take three to four months to complete. 

Despite the volume of proposals that NSF receives annually (in FY 2010, over 

55,000 proposals were submitted, an increase of 23% over the previous year), NSF 

routinely processes the majority of these proposals (°75%) in fewer than six months. 

To ensure the integrity of the process, all program officer recommendations are 

reviewed by the division director (or other appropriate NSF official), who examines 

whether the process used to arrive at the decision has been executed in accordance 

with NSF’s policies and that the decision has been based on a thorough analysis 

of the merits of the proposal. Large awards may receive additional review, either 

by the Director’s Review Board (DRB) or additionally by the National Science Board 

(NSB). The DRB examines award recommendations with an average annual award 

amount of 2.5 percent or more of the awarding division’s prior year current plan. 

The NSB reviews recommended awards with an annual award amount of one percent or more of the awarding Directorate’s or Office’s prior year current plan, or less 

than one percent or more of the prior year total NSF budget at the enacted level. 

Once the funding recommendation is approved (at whatever level is appropriate), 

the Division of Grants and Agreements ensures that the award recommendation 

meets all of NSF’s requirements before officially issuing the award. 

In addition to having multiple layers of review of individual award recommendations, NSF requires that all programs undergo an external review by Committees 

of Visitors (COVs) every three years. COV reviews provide NSF with external expert 

assessments of the quality and integrity of program operations and program-level 

technical and managerial matters pertaining to the merit review and final proposal 

decisions. Finally, retrospective analysis of the process is periodically performed on 

a Foundation-wide basis, including the statistical reports submitted to the NSB 

every year and the Impact of Proposal and Award Management Mechanisms 

(IPAMM) report of 2007 (http://www.nsf.gov/pubs/2007/nsf0745/nsf0745.pdf). 

At the request of Congress, in 2005 the NSB undertook an examination of NSF’s 

Merit Review Process (http://www.nsf.gov/nsb/documents/2005/nsb05119.pdf). The report concludes that: 

‘‘The Board fully supports the current NSF system of merit review, which utilizes 

the peer review process as the principal driver in funding decisions. The Board also 

strongly endorses the role of NSF program officers’ discretionary authority, in concurrence with division directors, for ensuring the implementation and goals of both 

Merit Review Criteria, along with achieving a balanced portfolio of research and 

education awards, both within directorates and across the suite of NSF programs. 

Unlike a system based solely on peer reviews’ scores, NSF’s merit review process 

incorporates peer review in a system that also considers those attributes of a proposal (risk, multidisciplinary nature, novelty) that are not readily accommodated by 

a numerical score, but essential to identifying the most innovative proposals.’’ 

The National Academy of Sciences, in the 1994 report ‘‘Major Award Decisionmaking at the National Science Foundation,’’ stated that, ‘‘The United States has 

built the most successful research system in the world. The use of peer review to 

identify the best ideas for support has been a major ingredient in this success. Peer 

review-based procedures such as those in use at NSF, the National Institutes of 

Health, and other federal research agencies remain the best procedures known for 

ensuring the technical excellence of research projects that receive public support.’’ 

In November 2009, the Executive Director of the Transportation Research Board at 

the National Research Council, provided testimony before Congress on how to facilitate the implementation of research at the Department of Transportation. In that 

testimony, the Director endorsed strongly the fact that NSF’s merit review process 

is well suited to the mission of the agency. His observation: ‘‘The more applied mitigation and adaptation research topics should be steered by the concerns and needs 

of policy makers and practitioners, while the fundamental research topics should be 

organized along the NSF model in which scholars and experts are guiding the decisions about which projects are likely to be most promising.’’ 

NSF’s merit review process has served the agency, the scientific community, and 

indeed the country well for many years. Many Nobel Laureates, National Medal of 
19 

Science and Technology winners, and MacArthur Foundation Fellows (popularly 

known as recipients of Genius Grants) have been supported by NSF at various 

stages in their careers. Through separate programs and in the course of funding 

specific scientific progress, over the past 25 years NSF has also supported the training of hundreds of thousands of graduate and post-graduate scholars in STEM 

fields. Discoveries stemming from NSF-funded projects have led to advances across 

all areas of science, engineering and education, with far-reaching impacts in the 

fields of nanotechnology, information technology, environmental science, genomics, 

STEM education, and many others. 

The high quality of NSF’s merit review process is recognized globally, as evidenced by the fact that it has been used as a model by countries around the world 

that are newly establishing their own funding agencies. The merit review system 

for L’Agence Nationale de la Recherche (ANR), the French counterpart to NSF, is 

explicitly modeled after NSF, as is that of the Foundation for Polish Science. NSF 

helped the European Research Council establish its merit review system some five 

years ago, and was instrumental in helping Ireland establish Science Foundation 

Ireland. Back in 1986, a Chinese official came to NSF for six months to learn about 

our merit review and decision making processes, and subsequently incorporated 

what he had learned in establishing the National Natural Science Foundation of 

China (NSF 09C). These are just a few examples of international agencies where 

NSF has had an explicit role in helping develop their merit review systems, but 

there are literally dozens of others that have borrowed our approach over the years. 

As the nature of research and the scientific enterprise continues to change—becoming more interdisciplinary, technological, international and collaborative—NSF 

continues to explore ideas and strategies that could strengthen the merit review 

process by enlarging the range of tools that can be used in proposal evaluation. 

These ideas have come from a variety of sources—internally, from the research community, from the practices of other funding agencies, and from the scientific literature on merit review. One idea that we are actively exploring is a greater use 

of technology-mediated virtual panels when and where it makes sense, with the 

hope that decreasing the travel burden will expand the potential pool of reviewers. 

Among the benefits that NSF would derive from an expanded pool of reviewers are 

the inclusion of more and varied perspectives, increased opportunities for participation by underrepresented groups, decreased review burden per individual reviewer, 

and decreased travel costs for the agency. We have established an internal working 

group to identify other viable candidates for pilot activities, and to develop plans 

for running and evaluating those pilot activities. We will be discussing these with 

an advisory committee over the next few months to get their help in refining the 

processes. 

For over 60 years, NSF has been forward looking in terms of how the agency manages its research and education portfolio. Merit review fosters the ‘‘process of discovery,’’ the means by which researchers can identify emerging scientific challenges 

and innovative approaches for addressing them. NSF is dedicated to ensuring that 

the merit review process remains robust, rigorous, and beyond reproach, in support 

of our mission and enabling us to pursue our goal of funding the world’s best research in science, engineering and education. 

I appreciate the opportunity to appear before the Subcommittee to speak to you 

on this important topic. I would be pleased to answer any questions that you may 

have. 

Appendix I: NSF Proposal and Award Process and Timeline 
20 
21 

Chairman BROOKS. Thank you, Dr. Marrett. 

I now recognize our second witness, Dr. Keith Yamamoto, for five 

minutes. 

STATEMENT OF DR. KEITH R. YAMAMOTO, 

VICE CHANCELLOR FOR RESEARCH, 

UNIVERSITY OF CALIFORNIA SAN FRANCISCO 

Dr. YAMAMOTO. Good morning, Chairman Brooks, Ranking Member Lipinski, distinguished Members of the Subcommittee. Thank 

you for the invitation to present a statement before you today. I am 

Keith Yamamoto, a molecular biologist, professor, and administrator at the University of California San Francisco. My lab’s research has been funded continuously for 35 years by grants from 

the NIH and NSF. 

I have also been active in matters of science and public policy, 

leading or serving on dozens of committees focused on a broad 

range of these topics. One of these topics has been the federal merit 

review system for evaluation of biomedical and biological research 

grant applications, especially those overseen by NIH and to a lesser 

extent the NSF. 

Today, I shall describe the key conceptual and operational features of the merit review process, assess how well that process 

works from my point of view, consider the impact of a specific proposal to change the NSF process, and mention two modifications 

that might make merit review even stronger. 

Every application for NIH and NSF life sciences research support, and at NIH, that means 80,000 applications per year, we just 

heard the 55,000 at NSF, every application is rigorously reviewed 

and prioritized for scientific merit by special communities of experts. The details of the review process differ among the different 

federal agencies but there are two critical features that to my 

knowledge are common across agencies. 

The first is that the merit review is carried out by other scientists because only scientific peers have the knowledge and perspective required to assess the scientific significance, innovation, 

and impact of proposed research projects, and the second is that 

the review committees judge only scientific merit. It is left to others to assess relevance of the applications to the goals or portfolio 

of the funding agencies and to make the funding decisions themselves. Thus, peer review with a solitary focus on scientific merit. 

How does the process work and how well does it work? The review committees are federally chartered and populated by volunteer expert scientists who set aside time from their own research 

at institutions across the country to carefully evaluate written proposals for scientific investigation. The reviewers exercise their individual and collective scientific judgments, prioritizing the scientific 

merit of the applications. To help avoid nonscientific biases, both 

the applications and the reviews are standardized. For example, in 

the NIH process, every application is reviewed and scored according to the same set of criteria, significance, approach, investigator, 

environment and overall impact. By any measure, peer-driven 

merit review has been spectacularly successful at identifying and 

prioritizing the most interesting, innovative and significant scientific research, thus enhancing profoundly the strength of feder-
22 

ally funded research. In contrast, merit review is not intended to 

influence the breadth or type of research that is funded. Rather, 

breadth and type are affected by funding mechanisms and by the 

stated missions of the funding agencies. 

How might the merit review process be changed to further 

strengthen it? Let me mention first a change currently under consideration that would, in my view, not serve the process and with 

two examples of changes that might enhance it. 

The National Science Board Taskforce on Merit Review has proposed a revision of the broader impacts criteria for NSF merit review, and I am quoting, ‘‘to ensure consideration of how the proposed project advances national goals.’’ This criterion departs, in 

my view, from the singular focus on scientific merit that is essential to the merit review process and calls upon reviewers to judge 

grant applications by metrics outside of their expertise. 

Moreover, NSF is the Nation’s basic research agency as mandated by the NSF Act of 1950, which created the Foundation. The 

impact of individual basic research projects on broad national goals 

may be impossible to judge at the time of proposal but may emerge 

with time in unexpected and profound ways. 

Broad national goals of course are vital and need to be advanced 

collectively by federal science research agencies. They cannot, however, be mandated for individual NSF grant applications. Indeed, 

national goals can be addressed by development of funding mechanisms and by defining agency priorities, not by the merit review 

process. 

Now, while the current merit review process has great strengths, 

there are potential changes that might improve it further. I cite 

several in my written testimony. Let me mention just two here. 

Consider recognizing and validating that a grant application is not 

a contract for sure, in fact, not even a roadmap of experiments but 

rather an exercise in which the applicant demonstrates his or her 

capacity to pose a scientific problem and devise a research plan 

that would impact and advance a field. In reality, scientists pursue 

the implications of each day’s results rather than adhering to the 

course of experiments imagined in their grants. Thus, grant application formats and merit review criteria would closely assess the 

merits of a proposed idea and of the investigator while reducing the 

current focus on experimental detail and feasibility. 

Second, consider formally denoting two separate classes of research, one I will call innovative and one we have been referring 

to as transformative, both essential but each requiring distinct 

merit review mechanisms. Innovative research would advance and 

deepen our understanding of current paradigms. Transformative 

research would disrupt or destroy prevailing paradigms and force 

creation of new ones. 

In closing, the merit review process for federal support of research is indisputably the best system for identifying highest quality of science. Its primary features, peer-driven review and singular 

focus on merit, have been critical in identifying grant applications 

that describe the best science by the best scientists. A healthy and 

robust merit review process is critical to maintaining and extending U.S. leadership in scientific research and innovation, which in 

turn is essential to reaching broad national goals. 
23 

This concludes my testimony. I will be pleased to answer questions and address your comments. Thank you again for the opportunity to discuss this important matter with you. 

[The prepared statement of Dr. Yamamoto follows:] 

PREPARED STATEMENT OF DR. KEITH R. YAMAMOTO, 

VICE CHANCELLOR FOR RESEARCH, 

UNIVERSITY OF CALIFORNIA SAN FRANCISCO

Good morning, Chairman Brooks, Ranking Member Lipinski, and Members of the 

Subcommittee. Thank you for the invitation to present a statement before you 

today. I am Keith R. Yamamoto, Vice Chancellor for Research, Executive Vice Dean 

of the School of Medicine and Professor of Cellular and Molecular Pharmacology at 

the University of California, San Francisco. I received a Bachelor of Science from 

Iowa State University and a Ph.D. from Princeton University before migrating to 

San Francisco, where I have been on the faculty for 35 years. My molecular biology 

lab has been studying the detailed mechanisms by which small molecules made in 

our bodies, hormones, control important physiological processes such as metabolism, 

stress responses and immunity; in the course of that work, I have had primary responsibility for training approximately 100 graduate students and postdoctoral 

scholars. Our research has been funded throughout by grants from the NIH and 

NSF. 

For the past 30 years, I have also been active in matters of science and public 

policy, leading or serving on dozens of committees focused on a broad range of 

issues, challenges and opportunities. One of the major areas of emphasis for those 

activities has been federal merit review system for evaluation of biomedical and biological research grant applications, especially those overseen by the National Institutes of Health (NIH) and the National Science Foundation (NSF). For example, I 

co-chaired the NIH effort to assess and enhance its peer review process, and served 

on the National Science Board/NSF task force on transformative research. These extensive experiences have provided me with a rather deep perspective on the merit 

review process and its relationship to the U.S. life science research enterprise. 

In my testimony today, I shall: (1) describe the key operational and organizational 

features of the merit review process for biomedical and biological research; (2) assess how well that process works; (3) consider the impact of a particular change to 

the process that is currently being considered; and (4) enumerate some potential 

modifications that might further improve the process. 

Every application for NIH and NSF life sciences research support (NIH, for example, receives some 80,000 applications per year) undergoes rigorous review and 

prioritization of scientific merit by committees of peers, typically by scientists who 

themselves hold grants from the same agency, and whose research is in the same 

area of research as the proposed study. The details of the review process differ 

among the different federal agencies, but evaluation by peers is the crucial common 

feature. 

To populate the federally chartered merit review committees, agencies enlist volunteer service from expert scientists in each area of research, who agree to set aside 

time from their own scientific research at academic and research institutions around 

the country, typically several times a year, to carefully evaluate written proposals 

for scientific investigation. The reviewers exercise their individual and collective scientific judgments free of other biases, prioritizing the scientific merit of the applications. 

To help prevent nonscientific biases, the formats of both the applications and the 

reviews are tightly delineated, whereas the applicants have broad flexibility in 

choosing the scientific subject matter of the applications. In the NIH process, for example, a standardized set of five criteria (significance, approach, innovation, investigator, and environment) is mandated as the basis for rating every application. Assessment of the investigator is focused solely on past performance and qualification 

for carrying out the proposed study. Importantly, merit review committees are 

charged to judge only scientific merit. In particular, they do not assess relevance 

of the applications to the portfolio of the funding agency, nor do they not make funding decisions. This singular focus on merit (together with peer-driven review) is the 

second key feature of the merit review process. 

Thus, it is important that decisions of scientific merit are insulated and separated 

from decisions of funding. In addition, of course, the merit review process has no 

control over the level of funding allocated to support meritorious applications. When 

funding levels fall far below the capacity to support some of the very best applica-
24 

tions (as is presently the case), the merit review process appears to fail, i.e., outstanding science goes unfunded. This apparent failure instead reflects a misalignment between the number of highly meritorious applications and the number of dollars available to fund those applications. 

Any merit review process that depends upon peers to carry out evaluations must 

acknowledge and address at least two intrinsic and related conflicts of interest: reviewers might unfairly support applications from their friends to create an ‘‘old 

boys’’ network, or they might unfairly disadvantage applications from competitors 

or from those outside of some perceived ‘‘inner circle.’’ In general, these intrinsic 

conflicts have been addressed successfully by well-crafted regulations, and more importantly, by a universal ‘‘culture of respect’’ from the participating scientists who 

serve as reviewers. 

While peer-driven merit review plays a crucial role in identifying excellence 

among proposed ideas and research plans, it is also the case that an element of conservatism is intrinsic to peer review, which complicates recognition and 

prioritization of ‘‘transformative’’ ideas and approaches. This is because the majority 

of scientists, and therefore the majority of peer reviewers, embrace and extend prevailing scientific paradigms, whereas transformative research disrupts or destroys 

accepted paradigms and creates new ones. Because both types of research are essential, approaches to address intrinsic conservatism are important. One strategy is to 

adopt special funding mechanisms, and perhaps some special elements of the review 

process, designed to identify ideas. 

The current merit review processes, which have been in place in the U.S. for over 

65 years, recognize that only scientific peers have the knowledge and perspective 

required to assess the relevance, innovation and impact of proposed research 

projects. Indeed, the current system of peer-driven merit review is widely held to 

be not the ‘‘best good system’’ for evaluation and prioritization of merit, but ‘‘the 

only good system.’’ 

By any measure (e.g., quality of scientific publications resulting from support of 

meritorious applications, honors and prizes given in recognition of the highly meritorious research, products and services that are developed from the results of supported research, creation of an outstanding scientific workforce resulting from training of students and fellows supported), peer-driven merit review has been spectacularly successful at identifying and prioritizing the most interesting, innovative and 

significant scientific research projects. Hence, the merit review process enhances 

profoundly the strength of the research funded by the federal government. In contrast, the merit review process is not intended to influence the breadth or type of 

research that is funded. Rather, breadth and type are strongly influenced by funding mechanisms, and by the range and diversity of disciplinary foci that are chosen 

by the funding agencies. 

Are there specific changes that might further strengthen the merit review process 

to ensure support of the best science? Let me mention first a change currently under 

consideration that would in my view damage the process, and then end with a brief 

enumeration of some potential changes that might enhance it. 

The National Science Board (NSB) Task Force on Merit Review is currently reviewing the NSF’s merit review criteria, and has proposed a revision of both the ‘‘Intellectual Merit’’ and the ‘‘Broader Impacts’’ requirements, with the goal of clarifying 

their intent and the ways that they would be used in the merit review process. The 

purpose of the Broader Impacts criterion ‘‘is to ensure the consideration of how the 

proposed project advances a national goal(s).’’ This criterion, as stated, would in my 

view adversely affect the merit review process because it departs from the singular 

focus on scientific merit that is essential to the process, and because it obligates 

peer reviewers to judge grant applications by metrics outside of their expertise. 

This being said, I concur fully that broader national goals are essential, and as 

stated in the NSB Merit Review Principles, ‘‘collectively, NSF projects should help 

to advance a broad set of national goals.’’ It is important, however, to remain mindful of the language of the NSF Act of 1950, which directs the Foundation ‘‘to initiate 

and support basic scientific research and programs to strengthen scientific research 

potential and science education at all levels.’’ Indeed, NSF itself originated from 

‘‘Science the Endless Frontier,’’ the redoubtable 1945 policy initiative of Vannevar 

Bush, which called out untargeted basic research as ‘‘the pacemaker of technological 

progress’’ in which ‘‘new products and new processes are founded on new principles 

and new conceptions, which in turn are painstakingly developed by research in the 

purest realms of science.’’ 

This implies that broad national goals should be advanced by the composite federally funded scientific research endeavor, and in particular should not be mandated 

for individual NSF research grant applications. In general, such goals should be addressed by development of funding mechanisms and by defining agency priorities, 
25 

and not as a part of the merit review process. Moreover, it seems that broad national goals might specifically be mandates for mission-driven agencies that seek to 

support research relevant to health, environment, energy, food and agriculture, or 

national security, rather than for the National Science Foundation. 

While the current merit review process has great strength, there are conceptual 

and operational aspects that might improve it further. Some examples for consideration across a wide spectrum: 

• Reconfigure the grant application conceptually to be viewed less as a ‘‘contract,’’ 

and more as demonstrations of an investigator’s capacity to identify an important 

scientific problem and devise tests that could advance knowledge and understanding, and thereby impact the field. Thus, the grant application is not intended 

as a ‘‘roadmap of experiments’’ projected three to five years into the future. Grant 

application formats and merit review criteria should place greater focus on the merits of the proposed idea and of the investigator, whole reducing the current focus 

on proposed experimental details and feasibility. 

• Motivate top scientists to maintain active participation in the merit review 

process, in part by developing mechanisms that more effectively encourage applicants to submit bold scientific ideas. 

• Establish and formalize two separate investigator-initiated funding mechanisms, innovative and transformative, which invite, identify and support research 

that, respectively, advances and deepens our understanding of current paradigms, 

or disrupts and destroys prevailing paradigms, and forces creation of new ones. Consider unique aspects of merit review process for transformative applications. 

• The topical/disciplinary focus of review committees has been eroded by the demands of reviewing a rapidly increasing proportion of grant applications that include a remarkable diversity of experimental approaches. To recover the conceptual 

focus of merit review committees, institute a ‘‘focused external review’’ process in 

which ad hoc reviewers are requested to contribute electronically brief comments 

that address solely those technologies or approaches for which specific expertise is 

lacking on the chartered committee. Such external reviewers would not be asked to 

assess the overall scientific merit of the application; that responsibility would reside 

solely with the chartered committee. 

The merit review process used for review of federal grant applications for support 

of biological and biomedical research is indisputably the best system for managing 

this important responsibility. Its primary features of peer-driven review and singular focus on merit have been critical in identifying grant applications that describe the best science by the best scientists. Once unique in the world, the process 

is being widely emulated. 

This concludes my testimony. I would be pleased to answer your questions or address your comments. Thank you again for the opportunity to discuss this important 

matter with you. 

Chairman BROOKS. Thank you, Dr. Yamamoto. 

I recognize our next witness, Dr. Nancy Jackson, for her five 

minutes. 

STATEMENT OF DR. NANCY B. JACKSON, 

PRESIDENT, AMERICAN CHEMICAL SOCIETY 

Dr. JACKSON. Good morning, Chairman Brooks and distinguished 

Members of the Subcommittee. 

As President of the American Chemical Society (ACS) during 

2011, the International Year of Chemistry, it is my great pleasure 

to address the Subcommittee this morning. Founded in 1876, ACS 

is the world’s largest scientific society with more than 163,000 

members. A nonprofit organization, ACS was chartered by the Congress in 1937 to advance chemistry in all its branches, promote scientific research and inquiry, and foster public welfare and education. ACS members work in industry, academia and government. 

ACS is a long-time supporter and strong supporter of NSF and its 

merit review process, which is recognized globally as the gold 

standard for identifying the best research to fund. My testimony 
26 

today will concentrate on chemistry and NSF’s impact on our 

science. 

The Federal Government is an important source of support, particularly for basic research conducted within the chemical enterprise that also trains tomorrow’s researchers. By stimulating innovation, the Federal Government empowers a competitive U.S. 

chemical enterprise, which contributes to U.S. economic growth. 

Mr. Chairman, I would like to focus on three areas from my written testimony that I believe are of particular value in these times 

of constrained resources and economic challenges. Those three 

areas are: one, balancing the NSF research portfolio; two, streamlining the merit review process; and three, measuring return on 

NSF investment. 

My first point has to do with portfolio management. As anyone 

with a retirement fund has been told, managing a portfolio is critical to its long-term strength, finding the right balance between 

proven performing stocks and riskier, higher-return investments. 

Managing a research portfolio is similar, striving to find the right 

balance between more incremental research that builds on previous 

successes and those ideas that are riskier and could lead to gamechanging developments. In times of tight budgets and restricted 

funding, NSF mustn’t become too conservative. The agency must 

ensure adequate attention to providing opportunities for young researchers as well as for out-of-the-box research that may create 

economic renewal, produce jobs and train the U.S. scientific workforce of the future. 

Secondly, NSF should be able to triage the least competitive submission from the review process. At the present time, all proposals 

submitted to NSF must go through an extensive peer review, even 

if they are intellectually weak with no hope of ultimately getting 

NSF funding. For every 100 grant proposals a program officer reviews, a small number, perhaps around 10, will be of such high 

quality that it is obvious to everyone that it should be funded. Another 50 proposals easily will be recognized as not competitive. 

However, in the current system, they all take considerable time to 

review and process. The real agonizing choices must be made concerning the remaining 40 proposals, all very good proposals. Under 

current funding, only 13 of those 40 proposals would be funded. 

The merit review process should be focused on these choices. 

Over the last 10 years, the number of grant proposals submitted 

to the NSF chemistry division has doubled while the size of the 

NSF chemistry division staff has remained the same, thereby presenting a significant challenge to NSF to continue to perform its 

job of identifying the best science. 

ACS is responsible for a research fund, albeit small compared to 

NSF. It is the ACS Petroleum Research Fund (PRF). It has seen 

the number of grant proposals skyrocket in recent years as well. To 

help manage this increase, a new policy was recently implemented 

that empowers PRF research managers to reject proposals at the 

outset that they deem to be poor in quality. The practical result 

has been that more time is freed up for PRF staff and the volunteer peer reviewers to focus on selecting the right balance of research from the strong proposals. Empowering NSF research managers to likewise would be a practical step to focus a steadily in-
27 

creasing NSF workload and maintain excellence in its merit review 

process. 

And that brings me to my third point. We need new tools and 

methods to evaluate the success of NSF investment. While ACS 

strongly supports NSF’s broader impacts criteria, we need better 

tools to determine their overall effectiveness. Measuring how many 

fellowships or grants are funded is easy; measuring increased innovation, improved national security or broadening participation is 

complex and must take place over a longer time scale. 

It is difficult to single out how one individual effort has impacted 

a complex collection of national priorities. I understand that finding the right metrics for measuring these issues is very difficult but 

it is worthwhile. Empowering NSF to more easily gather data is 

necessary to ensure that research is successful in achieving the 

broader impacts and that the selection process is meeting its many 

goals. 

Chairman Brooks, I thank you for the opportunity to testify 

today and I would be happy to answer any questions you or your 

Subcommittee have. 

[The prepared statement of Dr. Jackson follows:] 

PREPARED STATEMENT OF DR. NANCY JACKSON, 

PRESIDENT, AMERICAN CHEMICAL SOCIETY

As president of the American Chemical Society, or ACS, it is my great pleasure 

to address the Subcommittee this morning on the topic of the merit review process 

of the National Science Foundation (NSF). 

Founded in 1876, ACS has grown to be the world’s largest scientific society with 

more than 163,000 members and one of the world’s leading sources of authoritative 

scientific information. A nonprofit organization, ACS was chartered by the U.S. Congress in 1937 to advance chemistry in all its branches, promote scientific research 

and inquiry, and foster public welfare and education. ACS members work in industry, universities and colleges, and at national laboratories. 

ACS is at the forefront of the evolving worldwide chemical enterprise. It is the 

premier professional home for chemists, chemical engineers and related professionals around the world as well as a global leader in chemical information. ACS 

publishes 41 world-class scientific journals and operates the Chemical Abstracts 

Service, which provides the most comprehensive databases of disclosed research in 

chemistry and related sciences. 

Every year, ACS gives more than $11 million in grants for basic research in petroleum and related fields through the Petroleum Research Fund (PRF). Twenty-five 

researchers, who were recipients of these grants, later went on to become Nobel 

Laureates. 

The Society also plays a leadership role in educating and communicating with 

public policy makers and the general public about the importance of chemistry in 

our lives. This includes identifying new solutions to global challenges, improving 

public health, protecting the environment, and contributing to the economy. 

ACS has been a strong, long-time supporter of the National Science Foundation, 

which is of particular importance at this critical time in our nation’s history. My 

testimony will concentrate on chemistry and NSF’s impact on our science. While I 

think my observations and recommendations are broadly applicable, chemistry is 

my area of expertise. 

Chemistry is the fundamental science that is at the heart of processes and products that meet our most fundamental needs for food, shelter, and health, as well 

as developments and materials that are vital to advances in biotechnology, computing, and telecommunications. It is a keystone of U.S. manufacturing and is essential to a range of industries. 

America’s $720 billion chemical industry is one of our nation’s top exporters, with 

$171 billion in annual exports, which accounts for more than 10 cents of every dollar in total U.S. merchandise exports. Within the United States, the chemical industry employs 784,000 people and is a driver of innovation. The industry invests $55 

billion in research and development annually, and one in five U.S. patents is chem-
28 

istry related. In addition, the industry contributes to human and environmental 

health. Drug innovations, made possible through chemistry, have helped increase 

life expectancy in the United States by 30 years over the past century. 

I mention this today because the success of the chemical enterprise is due largely 

to scientific and technological breakthroughs and advances made in industrial, academic, and government laboratories. Although much of the nation’s chemical research is carried out by scientists, engineers, and technicians employed in industry 

and academia, the Federal Government is an important source of support, particularly for basic research conducted by our nation’s universities and government laboratories. By stimulating the roots of innovation, the Federal Government plays a 

fundamental role in ensuring the ability of the U.S. chemical industry to stay competitive in the long term. And because so many other industries depend on chemicals, the federal investment enhances the ability of the United States to compete 

globally by enabling a high-tech, competitive chemical industry to supply new products at prices that give our nation’s producers an edge. 

The NSF plays a unique role in the U.S. scientific enterprise. While other federal 

agencies have missions directed at advancing specific science and technology in 

health or energy, for example, the core mission of NSF is to foster a healthy scientific enterprise here in America. Supporting the best ideas and exploring new 

frontiers across research disciplines have been the hallmark of NSF and the backbone of the American research system. 

NSF has played a pivotal role in paving the way for scientific discovery, in large 

part, by awarding grants to members of the scientific research community that have 

demonstrated outstanding merit. The Foundation accomplishes its mission by supporting fundamental research and education in science and engineering. From aircraft design, pioneering medical tools and robotics, to discovering how children can 

learn chemistry better, NSF has played a key role in funding discoveries that have 

driven the nation’s economy, improved our quality of life, and enhanced national security. It also supports high-risk research and novel collaborations that could deliver 

exceptionally high rewards. 

NSF is not just about research. It’s also about developing and training tomorrow’s 

scientific workforce. There is a symbiotic relationship between research and education. When a graduate student or a post-doctoral student works with a researcher 

funded by NSF, the student is honing skills and adding new scientific knowledge. 

In this way, the torch is passed from one generation of researchers to the next. To 

put it another way, this is how we keep pushing the edge of the envelope. If the 

United States is to continue to be a leader in science and technology, then we need 

to have the trained workforce working in that space. 

NSF provides more than 20 percent of the federal support for basic research at 

academic institutions and supports roughly 10,000 new awards per year through the 

merit reviews of over 40,000 proposals received. Every year, an estimated 200,000 

people, from undergraduates to senior faculty, participate directly in NSF research 

and education programs. 

The NSF merit review process is the gold standard worldwide, and is one of the 

reasons why U.S. science has been as successful as it is. When other countries seek 

to set up their own national research efforts, they often look to the U.S. NSF as 

the role model to emulate. 

At NSF, all proposals are evaluated for intellectual merit and broader impacts. 

NSF receives far more meritorious proposals than it could ever fund. While a proposal with weak intellectual merit has no hope of getting NSF funding, many proposals are rated ‘‘excellent’’ with strong intellectual merit and still do not get funded 

because of the stiff competition. The broader impacts criteria take into consideration 

which research is the most urgent or has the greatest relevance to improving the 

quality of life. This merit review process enables NSF to ensure that precious R&D 

money goes only to the most pressing R&D needs. 

As anyone with a retirement fund has been told, managing a portfolio is critical 

to its long-term strength. Financial advisors stress that it’s important to find the 

right balance between solid performing stocks and riskier investments that may provide higher returns. Managing a research portfolio is similar: the research manager, 

whether working in industry or at NSF, strives to find the right balance between 

science that will deliver steady advances and ideas that are out of the box, but could 

result in game-changing developments. This point is especially important in times 

of restricted funding. It’s human nature to make more conservative choices and be 

risk-aversive when times are tough. However, now more than ever, America needs 

pioneering research that will create economic renewal, produce jobs, and train the 

scientific workforce of the future. Extra efforts and attention must be paid to cultivating young researchers and game-changing ideas. 
29 

One of the reasons why the merit review process is so successful is because it 

draws from the collective wisdom of the scientific community. Many NSF personnel 

come directly from the scientific community and will return to their research institutions at the end of their two- or-three-year rotations. Relying on rotating directors 

means the managers are up to date on the most recent scientific developments. The 

panels that perform the peer review of proposals are fellow researchers in the field, 

and as such, are also up to speed on the latest developments. This scientific community service, whether performed by grant proposal reviewers or NSF program officers, is an integral part of scientific culture. Many scientists dedicate their time in 

this way because it provides an opportunity to remain in touch with and influence 

the cutting edge, as well as because they understand that the system only works 

if everyone volunteers to play their part. In a way, it is the science community’s 

way of 11paying it forward.’’ 

The merit review process requires significant efforts by both NSF employees and 

scientist volunteers. To better understand how the process plays out, consider this 

example from the NSF chemistry division. The division receives about 1,800 proposals annually. Each program officer in the division manages about 100 proposals 

a year. These managers are responsible for picking peer reviewers, and they must 

do so with an eye for diversity across a large number of factors such as ensuring 

that the reviewers reflect a balanced group based on type of institution (e.g., small 

undergraduate colleges vs. large research universities), geography, and racial and 

gender characteristics. 

Peer reviewers must also be experts within the proposal’s subfield of chemistry. 

Generally, a program officer approaches three reviewers to find one who will accept 

the call to serve. Since each proposal requires three to five reviewers, this means 

the officers approach six to 10 reviewers for each proposal. Therefore, on the average, the NSF chemistry division approaches between 10,000 to 18,000 researchers 

to serve as peer reviewers for the proposals submitted. 

For every 100 grant proposals a program officer reviews, a small number (perhaps 

10) will be of such high quality that it is obvious they should be funded. Another 

50 proposals will be recognized as clearly not competitive; however, they still must 

be considered through the process. An agonizing choice must then be made over the 

40 proposals in the middle. These include proposals that may be considered excellent or very good. In fiscal year 2010, the NSF award rate was 23 percent. In our 

analogy of 100 proposals, this would mean that 13 out of the remaining 40 would 

be funded. 

The broader impacts criteria include considerations about whether the research 

proposal would broaden underrepresented minorities’ participation in science, 

strengthen U.S. infrastructure, improve national security, or foster innovation. 

Some of these impacts are the result of language in the America COMPETES bill 

enacted last year. The broader impacts criteria take into consideration which research is the most urgent or has the greatest relevance to improving the quality of 

life. The broader impacts criteria enables NSF to choose between meritorious and 

even more meritorious proposals, and is a way to ensure that precious R&D money 

goes to the most pressing R&D needs. 

It should be added that the number of grant proposals submitted to the NSF 

chemistry division steadily increases each year, more than doubling from levels 10 

years ago. And while the number of proposals has doubled, the size of the NSF 

chemistry division staff has remained the same. NSF is challenged to continue to 

perform its job of supporting the best science, even as the sheer number of proposals 

competing for funding has ballooned. 

I mentioned in my introduction that ACS is responsible for the management and 

administration of the ACS Petroleum Research Fund (PRF), which was established 

in 1944 by seven oil companies as a perpetual trust to advance science education 

and fundamental research in the petroleum field. In 2010, the Fund provided $11.4 

million for research grants. 

Although PRF is a small research fund, like the NSF, it has seen the number of 

submitted grant proposals skyrocket in recent years. Perhaps some insights gleaned 

from PRF would be useful in considering how to strengthen the NSF merit review 

process. 

Several years ago, to relieve the growing peer review burden on the science community and to lighten the administrative load on the PRF staff, a policy was implemented to withdraw proposals from consideration that were deemed to be ‘‘poor’’ 

from the get-go: This includes those that are poorly written, use bad science, or do 

not address the specific scientific areas that were eligible for funding. In practice, 

this means that the managers now triage approximately 20 percent of the grants 

that come their way. These managers err on the side of caution: If there is any 

doubt that a proposal may have some merit, it is forwarded to the peer review pan-
30 

els for consideration and ranking. The result has been that, while some ‘‘poor’’ proposals are removed from the evaluation process, more time and energy is freed up 

for PRF staff and the volunteer peer reviewers to focus on selecting the right balance of research from the strongest proposals. 

Currently, NSF does not have the freedom to remove any proposals from the very 

bottom of the pile from consideration. Empowering NSF research managers to do 

so—provided that specific criteria are taken into consideration—would be a simple 

step to help NSF maintain excellence in its merit review process. If the average acceptance rate for an NSF proposal is about 23 percent, this means that 77 percent 

of funding proposals will be turned away. Enabling managers to remove the lowest 

20 percent of those that would normally be rejected from consideration is highly unlikely to result in a potentially great proposal not getting its due consideration. Instead, this approach may be a practical step to balancing a steadily increasing NSF 

workload. 

I have mentioned the important role the broader impacts criteria play in the NSR 

merit review process. NSF promotes broadening participation of underrepresented 

minorities and women, and persons with disabilities. This also includes increasing 

diversity in the NSF portfolio with respect to types of institutions supported and the 

geographic regions represented. Broadening participation is one way to address the 

broader impacts criteria; however, other activities are also appropriate. 

The importance of NSF efforts to broaden participation of underrepresented minorities in science and engineering is well understood and supported by the scientific community. We know that if the scientific work force doesn’t reflect the demographics of our country, we risk missing out on bringing the best minds and talents from every community to work on the scientific challenges that will impact all 

of our lives. 

In spite of NSF efforts in the broader impact criteria areas, we could use better 

tools to measure how effective these NSF efforts have been. One difficulty of measuring the long-term impact of the broader impact criteria is that it’s easier to measure the inputs than the outputs. Measuring how many fellowships or grants are 

funded is easy. Measuring increased national innovation, improved national security, or broadening participating is complex and must take place over a longer time 

scale. These differences make it difficult to single out how one individual effort has 

impacted a complex collection of national priorities. 

As a scientist, I want these efforts to be successful. As a senior administrator, I 

recognize that it’s nearly impossible to measure success if you can’t measure longterm outputs. In industry, we understand that finding the right metrics is very difficult, but it’s worthwhile to try. I do not know how to resolve this issue, but I do 

believe that empowering NSF to more easily gather the data needed to measure the 

success of the broader impacts criteria would be a necessary step to ensuring those 

efforts achieve their desired effect to the maximum extent possible. 

Chairman Brooks, I thank you for the opportunity to testify today and to share 

these thoughts with you. ACS believes the NSF is the cornerstone of the U.S. scientific enterprise, and we stand ready to assist you efforts to strengthen the agency 

for the benefit of the scientific community and the entire nation. 

Chairman BROOKS. Thank you, Dr. Jackson. 

Next, we recognize our final witness, Dr. Jose ́. 

STATEMENT OF DR. JORGE JOSE ́ , 

VICE PRESIDENT FOR RESEARCH, INDIANA UNIVERSITY 

Dr. JOSE ́. Good morning, Chairman Brooks, Ranking Member Lipinski and distinguished Members of the Subcommittee, and especially Representative Bucshon from Indiana. Thank you for allowing me the opportunity to speak with you today. 

My remarks arise from my experience as a primary investigator 

funded for many years by the National Science Foundation, as a 

member of NSF review panels, and an advisor for strategic planning at both the NSF and the National Institute of Health (NIH). 

I have been the Vice President for Research at two major research 

universities, for five years at the State University of New York in 

Buffalo, and since last August at Indiana University. Both institutions are members of the American Association for Universities and 

the Association for Public Land-Grant Universities. 
31 

There are three points I would like to emphasize this morning. 

First, merit review is the most effective process we have for ensuring that federal funds are used to support the most important and 

far-reaching scientific research. Second, merit review is the best 

way to ensure impartiality in awarding funding to research. And 

finally, federal funding for scientific research awarded through rigorous process of merit review is a necessary and important component of continued American preeminence in the world and is the 

foundation of our Nation’s long-term economic and national security. 

A very large number of scientific problems are interesting in 

principle but only a small number of those problems is important 

and deserves the investment necessary to search for and find solutions. I support what Dr. Jackson said about not all of the problems that people want to study are interesting or relevant for 

study. My broad experience in the process by which the Federal 

Government funds scientific research at universities leaves me to 

conclude while no review system is perfect, merit review is the best 

way to identify those problems that are important to support. 

One indication of the strength of the merit review system is the 

number of major scientific breakthroughs resulting from research 

that was awarded federal funding as a result of peer review by experts, people uniquely able to recognize the potential of proposed 

research. For instance, one can imagine how the average person in 

the street would respond if asked whether or not we should spend 

taxpayer dollars to study C. elegans, which is a tiny, transparent 

worm, a millimeter in size. Yet federally funding both at the NSF 

and the NIH on this worm has enabled scientists to identify key 

genes regulating organ development and programmed cell death 

and has shown that corresponding genes exist in high-end species 

including humans, of course. Nine recipients of Nobel Prizes since 

2002 focused their work on C. elegans. Research on C. elegans has 

led to clinical trials on the treatment of macular degeneration, 

asthma, diabetes and brain diseases. These advances occurred because experts in the field recognized that this tiny worm would 

make a good model system to study and could help us learn important things about humans. 

I would like to comment on the broader impact criteria that are 

part of the merit review. In my experience, the broader impacts criteria are used only to decide between proposals of equal scientific 

merit, and this is entirely appropriate. Having said that, I would 

like to stress the importance of expanding the participation of 

underrepresented groups in the sciences, a goal which over the past 

10 years has been the primary broader impact criteria for NSF. 

This is not a matter of establishing quotas or prioritizing this participation over the scientific importance of a proposal but the NSF 

emphasis on this is the demonstrated way of increasing the pool of 

talented and trained scientists which in turn enhances American 

economic competitiveness and leadership in scientific research, discovery and innovation. 

In these challenging times, it is more important than ever that 

federal funds are spent wisely. Federal research funding has for 70 

years been the cornerstone of American economic security, scientific and educational preeminence, maintaining our competitive-
32 

ness globally, continuing American leadership in scientific research 

and innovation and ensuring that our children and grandchildren 

enjoy a future in which American higher education and industry 

remain the envy of the world. All these rely on continued robust 

federal funding for scientific research and discovery, funding 

awarded through the merit review system as my colleagues have 

already stated. 

Thank you very much for inviting me here this morning and I 

look forward to answering any questions you may have. Thank you. 

[The prepared statement of Mr. Jose ́ follows:] 

PREPARED STATEMENT OF DR. JORGE JOSE ́, 

VICE PRESIDENT FOR RESEARCH, INDIANA UNIVERSITY

Chairman Brooks, Ranking Member Lipinski and distinguished Members of the 

Subcommittee, it is an honor to be here this morning to speak with you about the 

important topic of the merit review process in federal funding for scientific research, 

and in particular at the National Science Foundation. My remarks today arise from 

my experience as a primary investigator funded for many years by the NSF, as a 

member of NSF review panels, and as an advisor for strategic planning at both the 

National Science Foundation and the National Institutes of Health. I also speak 

from my experience as Vice President for Research at two major research universities, for five years at the State University of New York at Buffalo, and since last 

August, at Indiana University. Both institutions are members of the American Association of Universities and the Association of Public and Land-Grant Universities. 

This morning I would like to speak with you about the importance of the merit review process, specifically, its integral place in establishing American preeminence 

in hIgher education, scientific investigation, and economic innovation. 

My broad experience in different aspects of the process by which the federal government funds scientific research at universities leads me to conclude that while no 

system of review is perfect, nor guaranteed to fund only the best scientific research, 

the merit review system is the most effective process we have for ensuring that federal 

funds are used most effectively in support of scientific research, in particular at this 

time of limited resources when we need to prioritize how the taxpayer dollars are 

best invested. 

In some respects, the challenges facing federal funding agencies such as the NSF 

and the National Institutes of Health are very much like the challenges I face as 

Vice President for Research at Indiana University. As you may know, research universities often invest some of their limited resources to catalyze programs of research into issues that are of fundamental importance to our State, our Nation, and 

the world. My goal is to help our researchers identify and address the most important scientific, social and economic problems ofthe 21st century, such as energy security, health care, national security and our global competitiveness. Addressing 

these problems is not only valuable but is also a necessity because the problems will 

not solve themselves. Seemingly intractable problems can be solved only when the 

best minds with the appropriate expertise are brought together, and America’s current and future well-being depends in an essential way upon the results of research 

into these problems. 

As important as this work is, research is just one among many important areas 

of the University’s work and available resources—at IU as within the Federal Government—are limited. Given limited resources and given the importance of tbe problems, it is crucial that IU directs available resources to the projects with the strongest likelihood of being transformative and successful. I rely heavily on the ability 

of experts to assess each proposal, the work plan, and the potential of specific people 

to carry out a project successfully. This guidance helps to ensure that funding decisions are made on the basis of scientific merit rather than personal or political considerations. In short, merit review must be the foundation of funding decisions we 

make at IU because merit review enhances the likelihood that we will properly invest our limited available funds into research projects with the strongest potential 

for innovation, for transforming a field, or addressing successfully an important 

problem. 

We are, of course, following the path that was set up by Vannevar Bush in his 

‘‘Science—the endless frontier’’ developed right after World War II, that led to the 

formation of the National Science Foundation. NSF introduced the merit review 

process as an essential component to assess and determine how tax dollars should 
33 

be best allocated to scientific research. The merit review process has for the last 60 

years led to many NSF notable successes. It is very important to recognize that a 

very large number of scientific problems are interesting in principle but a much 

smaller subset of those problems is important and deserves the investment necessary to search and find solutions. 

Merit review is the best way to identify the important problems. It is the best 

way to ensure that federal funds are invested in a healthy array of important problems, covering a breadth of areas and approaches within a particular field. And 

merit review is the best way to ensure impartiality, so that the best science and 

the best scientists are funded. 

In his planning for the NSF, Vannevar Bush drew on the experience of wartime 

scientific research, organized through the National Defense Research Council 

(NDRC). Wartime federal investment in scientific research resulted in the development of penicillin, the radar, and most famously, the atomic bomb. NDRC research 

brought together the very best scientific minds from Europe and the U.S. to work 

on the most important problems of the time—and resulted in discoveries that helped 

America become the pre-eminent economic, military, and scientific power of the 20th 

century. Since the end of the Second World War, federal investment in research at 

American universities has been central to the development of universities that are 

the envy of the world, and merit review has been central to funding research that 

enables the United States to remain the leader in scientific inquiry, the development of new technologies, and the translation of fundamental research into applications that shape our lives every day. While there are many ways to assess and demonstrate the value of the merit review system, allow me to focus on one. The 

strength of relying upon merit review to determine what small percentage of proposals will be funded is demonstrated by its adoption by other countries across the 

world. Funding agencies in Europe, South America, and Asia, which are trying to 

emulate the research breakthroughs the U.S. has had in the last 60 years, all rely 

on a merit review system in which experts assess proposed research much as we 

do in the United States. 

I can give you a long list of items we use every day which were often developed 

as a result of the research that was funded by the federal investments in scientific 

research. One which the large majority of Americans use is the cell phone; another 

is the Global Positioning System, which was developed by the Department of Defense but which was based on a trail of research discoveries that started with the 

work by Einstein on his mathematical theory of gravity. GPS use is so common that 

it is hard to imagine how we found our way anywhere before it became commercially viable! We also hear the weather report, in particular during these hot days; 

the food we eat has been produced using scientific breeding techniques. We often 

don’t think about where all these things came from, but we do know that Americans 

as a whole are very proud of the long tradition of scientific research, inventiveness 

and innovation, a tradition which has made the U.S. the advanced technological society that it is today. 

It might be helpful to reflect for a moment on a couple of specific examples of the 

impact of federal funding for scientific research, because these examples point to the 

success of merit review. They also demonstrate that it is not always possible to anticipate what kind of impact research may have. For example, Caenorhabditis 

elegans (c. elegans), a transparent worm that most of us can go our whole lives without thinking too much about, but which has been the subject of significant research 

since the early 1960s. Only one millimeter in length, it was the C. elegans which 

first had its full genome sequenced, prior to the big achievement of decoding the 

human genome. For many reasons, C. elegans is a useful model organism, enabling 

researchers to learn about genetics, cell biology, and the pathogenesis that relate 

to many human diseases. Three times since 2002, Nobel Prizes have been awarded 

to researchers working with C. elegans. The 2002 Nobel Prize in Physiology or Medicine was shared by three researchers, Sidney Brenner (Berkeley), John Sulston 

(Cambridge), and Robert Horvitz (MIT), whose work was funded largely by the National Institutes of Health through its processes of merit review. The 2002 Laureates identified key genes regulating organ development and programmed cell death 

and showed that corresponding genes exist in higher species, including man. The 

2006 Nobel Prize in Physiology or Medicine was awarded to longtime NIH grantees 

Andrew Fire (Stanford) and Craig C. Mello (U. Massachusetts) for their discovery 

of RNA interference in C. elegans—work which has led to clinical trials in the treatment of macular degeneration, asthma, diabetes, and brain diseases. Further, the 

2008 Nobel Prize in Chemistry was awarded to Martin Chalfie (Columbia) for his 

work on green fluorescent protein in C. elegans (along with Roger Tsien and Osamu 

Shimomura, who studied these proteins in other contexts). I still read many C. 

elegans papers in the current scientific literature that keeps unraveling important 
34 

1 Distinguished Professor of Political Science, and Senior Research Director, Workshop in Political Theory and Policy Analysis, Indiana University; Research Professor and Founding Director, Center for the Study of Institutional Diversity, Arizona State University. 

new discoveries with very likely applications including useful drugs to treat diseases. You can imagine the response if you were to ask a regular person in the street 

if they felt that investing tax dollars would be justified to study a simple, almost 

insignificant worm. As the results show, this research has been a good investment— 

and it’s an investment that relied on the experts who recognized that this organism 

was indeed an animal model system which had many properties in common with 

higher organisms like humans and that it was worth studying. It is important to 

note that a significant percentage of the Nobel Prize winners for the last 50 years 

have been American or working in the U.S. This can be connected with the existence of our merit review system, which tries to fund and identify only the best and 

more promising ideas for funding. 

In the social and behavioral sciences, federal funding awarded through merit review has been at the foundation of substantial and important research. Elinor 

Ostrom (Indiana University) 1 and Oliver Williamson (Berkeley) won the Nobel 

Prize for their work on ‘‘Economic Governance.’’ Ostrom explored how communities 

often govern shared property and common resources more effectively than institutions do. Her research not only challenges the logical assumption about the inefficiency of informal groups, but also demonstrates that economic analysis can help 

understand myriad forms of social organization. Williamson provided a theory of 

why some economic transactions take place within firms and other similar transactions take place between firms in the marketplace. Their work informs us about 

how to handle one ofthe most basic choices in human organization: When should decision power be controlled inside an organization, and when should decisions be left 

to the market or governments themselves. Ostrom is the only woman who has won 

the Nohel Prize in Economics, and she benefited from long-term funding from the 

Social and Behavioral Sciences section ofthe National Science Foundation. Those 

funds were awarded through merit review. 

Why is the merit review process for awarding federal funds considered 

a strong or beneficial process? What is the impact of the merit review process on the breadth, type, and strength of research funded by the Federal 

Government? 

At the center of the merit review process is peer review, in which experts review 

each proposal for its importance, soundness, and possible transformational and 

broader impacts when considering if it should be funded or not. The experts who 

serve on review panels not only have expertise in specific research; they also have 

an understanding of the broader context of the discipline. They are therefore 

uniquely qualified to assess whether an idea is important as well as interesting; the 

extent to which the research methods proposed and the qualification of the researchers is appropriate for the problem under consideration; the degree to which a proposed research project has transformative potential. The strength of the merit review process rests largely on the service of subject experts, their willingness to read 

proposals carefully, often for free, so as to identify those most deserving of funding. 

It is important to recognize that the expected results from the proposed projects for 

funding are not known in advance; otherwise they would not be called research. It 

is for this reason that expert reviewers are able to best assess the probability of success of a project based on the previous track record of the investigators submitting 

the proposal and the track record of the reviewers on the subject matter under consideration. 

Of course, even the most well-intentioned group of subject experts is not immune 

from the biases, limitations, and agendas that are part and parcel of being human. 

However, in my experience as a reviewer for many years and after having submitted 

many proposals to the funding agencies, it is clear that the agencies try to reduce 

as much as they can all possible types of bias. By and large, most of the reviews 

by the experts are more than likely to arrive at the correct overall decision about 

whether or not it is appropriate to fund a proposal. There can be outliers to the general process, but reviewers for the most part are very strict about basing their decisions on scientific merit, which is why the percentage of proposals which are strongly recommended for funding is so low. 

At its best, the merit review process is a system of checks and balances not unlike 

our system of government, in that program officers can also offer a counterbalance 

to the limitations of the peer reviewers. Program officers bring to the decision-making process an awareness of agency priorities and funding trends that individual re-
35 

2 See attachment A. 

viewers or review panels will not have. In selecting reviewers, framing the review 

process, and interpreting the panel’s recommendations, program officers place funding decisions in a broader context than just subject expertise. Together, program officers and reviewers have been remarkably successful at identifying the best proposals across a wide variety of fields, basing their funding decisions first and foremost on the intellectual merit as assessed by the experts, the transformative potential, and intrinsic importance ofthe research proposed. 

With limited federal resources, what role does the merit review process 

play in ensuring that the best scientific and potentially transformative 

ideas receive funding? How do the broader impacts criteria requirements, 

in addition to intellectual merit, affect these funding decisions? 

The merit review process is the best way to ensure that limited resources are directed to the best ideas. As I stated above, the collaboration of subject experts and 

program officers enhances the likelihood that funded projects will be the ones addressing important questions, in ways that are methodologically appropriate and 

that can lead to transformative changes in the subject matter at hand. The merit 

review process is, in my view, the best way to minimize the potential for politicizing 

scientific research, and to ensure that limited funds are allocated as well as possible, in particular when the federal financial situation is as precarious as it is at 

the moment. 

The broader impacts considerations offer important additional criteria for funding. 

Allow me to stress that in my experience, the broader impacts criteria are used to 

decide between proposals of equal scientific merit—the contribution of a proposed 

project to achieving the extra-scientific goals included within these criteria does not 

outweigh scientific considerations. That is, assuming two proposals offer programs 

that are of equal importance and potential, then the broader impacts criteria should 

be used to distinguish them. In my view, funding decisions made in this way find 

the appropriate balance between intellectual merit and broader impacts of research. 

As members of the Subcommittee are likely aware, in response to the America 

COMPETES Act of 2010, the National Science Board recently proposed an expanded 

list of broader impact criteria for NSF proposals, a list that does give some cause 

for concern. I do not believe that this expanded list might alter the appropriate balance between intellectual merit and broader impacts in funding decisions. Rather, 

my concern is that the expanded list will diminish the National Science Foundation’s admirable and necessary leadership in promoting the participation of underrepresented groups (women, racial, and ethnic minorities) in the sciences. Expanding the participation of underrepresented groups is an economic and intellectual necessity. Science and scientific innovation are increasingly important to a strong 

economy, and so American economic security as well as American pre-eminence in 

scientific inquiry and higher education depend upon expanding scientific education, 

at both the K 0912 and higher education levels. To the extent that the expanded 

list of broader impacts criteria diverts attention from this priority, it risks damaging 

our economic and national security. 2

Expanding the participation of members of underrepresented groups in the 

sciences is not a matter of establishing quotas or prioritizing this participation over 

the scientific importance of a proposal. The NSF emphasis upon this is a demonstrated way of increasing the pool of talented and trained scientists. This in turn 

enhances overall American competitiveness, and over the long run, it will ensure 

that the United States continues to be the world leader in scientific research, discovery, and innovation. 

How does the merit review process work to ensure review impartiality 

for all applicants while maintaining high standards of excellence? 

Among the strengths of the merit review process is a degree of flexibility and 

breadth that other possible review systems would be unable to match. By this I 

refer to the ways in which our proposal review processes recognize that merit has 

many facets. For instance, an investigator at an early stage in her/his career cannot 

be expected to have the track record that a more senior investigator has—and yet, 

the early career proposal may be as meritorious and promising as the ones submitted by the senior investigator earlier in their career. The NSF and other agencies 

have worked to develop programs that attend to differences in seniority, to allow 

the junior investigator to develop and flourish since they represent our future. CAREER Awards, available through the NSF, are targeted specifically at researchers 

early in their academic careers. But outside of these programs, the merit review sys-
36 

3 James Turner, ‘‘Best Practices in Merit Review: A Report to the U.S. Department of Energy,’’ APLU, December 2010, p. 7. www.aplu.org/document.doc?id=2948. 4 Ibid., p. 10. 

tem contributes to impartiality of reviews because it brings a group of scientific experts together to review proposals. Barring personalities or idiosyncratic agendas 

that subvert the process, merit review increases the likelihood of fair and open review of all proposals over any alternative review process I am aware of. 

Please discuss any potentially novel ideas that should be considered in 

order to strengthen the process. 

I am, as you have likely gathered, a strong proponent of the merit review system. 

I believe it is the process more likely to ensure that limited federal funds for research are awarded to the most significant scientific research, with the highest likelihood of long-term impact. Nonetheless, the process is not perfect. I have alluded 

to the possibility that review panels are not free from bias that can influence their 

funding recommendations. One weakness of the peer review process is that scientists can be somewhat conservative. By this, I mean that it is easier to identify 

and fund proposals with the likelihood of incremental scientific advances than it is 

to identify and fund proposals with the potential for transformative breakthroughs. 

To the extent that bias toward the incremental means that we fail to fund innovative, paradigm-shifting research, we may be missing opportunities for precisely the 

kind of scientific advances that characterized federally funded research in World 

War II. 

This conservatism is recognized by funding agencies, which have occasionally responded by altering the charge given to review panels. ‘‘For example, when NIH 

concluded that it was not awarding enough high risk/high payoff grants, it changed 

its charge to panels accordingly.’’ 3 Adapting guidelines given to review panels is 

one way to ensure flexibility and strength within merit review processes. ARPA 

09E, the Department of Energy program that focuses on funding breakthrough research, offers another model that might strengthen the merit review process. By 

using a multi-part proposal and review process, ARPA 09E attempts to give reviewers a better understand of extraordinary proposals, and thus a better chance of recommending the best and most feasible proposals for funding. 4 Similar multipart 

proposal and review processes might effectively be adopted by other funding agencies, in specific programs if not throughout all funding programs. This might 

strengthen the merit review process by making funding of extraordinary projects 

with transformative potential increasingly likely. Of course, scientific research advances slowly and having transformative discoveries is not as common as one would 

wish. Thus I am not advocating that funding be directed solely to risky research. 

Rather, I am suggesting that the overall research funding apparatus could be improved by a measured emphasis on potentially disruptive discoveries and having 

mechanisms that allow them to be funded. 

In these challenging economic times, it is more important than ever that federal 

funds are spent wisely. Continued federal funding for scientific research remains an 

important priority. Federal research funding—via the National Defense Research 

Council during the Second World War, and via the NSF and other government agencies since then—has for 70 years been a cornerstone of American economic security, 

scientific and educational preeminence. Maintaining our competitiveness globally, 

continuing American leadership in scientific research and innovation, and ensuring 

that our children and grandchildren enjoy a future in which American higher education and industry remain the envy of the world—all of these rely on continued, 

robust federal funding for scientific research and discovery. Yet each dollar we 

spend must be spent wisely—and a strong, flexible and rigorous merit review process is the best possible guarantee that American scientific research will continue to 

lead the way forward and that every taxpayer dollar is spent with the highest possible return on investment. 

I thank the Subcommittee on Research and Science Education for allowing me to 

express my opinions about the merit review systems used by the federal agencies 

in general but in particular by the National Science Foundation, which funded a significant portion of my research for close to 25 years. I will gladly respond to any 

questions you might have. 

Chairman BROOKS. Thank you, Dr. Jose ́, and thank you, panel, 

for the information that you shared with us. I am reminding the 

Members that Committee rules limit questioning to five minutes. 

However, if time permits and there are Members who wish to en-
37 

gage in a second round of questions, we may do that. The Chair 

will at this point open the round of questions and as such I recognize myself. 

The first question is for Dr. Marrett. Based on fiscal year 2010 

data, please explain why over 2,700 proposals rated from poor to 

good to very good received federal funding over the 8,000-plus proposals that received very good to excellent or excellent ratings and 

were not funded. More particularly, in looking at the graph that we 

have been provided, there were some proposals, one that had a 

poor to fair rating but was funded, we had 98 proposals that received a fair to good rating that were funded, while on the other 

hand we had 1,312 proposals that received an excellent rating that 

were not funded, and similarly, 6,318 proposals that received a 

very good to excellent rating that were not funded. So generally 

speaking, why this kind of variation? 

Dr. MARRETT. Yes. Now, I have to explain, to go back to one of 

my key points, the role of program officers who have to bring their 

expertise to bear. What you are reporting on are the results of the 

reviews of external panels. The external panels look at, as already 

indicated, very much the technical merit of proposals. There are 

other things that have to be taken into account, and they have 

been the founding of the Foundation. Thus, a program officer has 

to ask about what else might be known about the proposal. The 

program officer also pays a lot of attention to what Dr. Jackson 

mentioned, a portfolio. Thus, if we are to be concerned about how 

well we are serving the entire United States, how well a portfolio 

represents within a discipline the varying ways in which work 

might be done in that discipline so that there isn’t complete overlap 

in what is being funded. So making sure that there this is this kind 

of balanced portfolio is a responsibility that leads then to the fact 

that our program officers who make their recommendations to the 

next levels of review in the process, program officers are not then 

bound by the kinds of assessments that might be given since those 

assessments are on technical merit. They are not bound to have to 

rely exclusively on those recommendations from the external reviewers. 

So let me close by saying, I will note again, those are recommendations that are made and they are not the final decisions 

or how the funding will be done. 

Chairman BROOKS. You mentioned other things taken into account and then you gave as an example overlap. Are there any 

other factors taken into account that you have not shared? 

Dr. MARRETT. Well, there can be a number of things. I think 

some of the questions from the Committee that we received asked 

about such questions as geographical balance. One of the questions 

commented on the fact that we know that talent is not limited to 

any specific part of the Nation and thus there is a question often 

of how are we looking across, but this is not to suggest that these 

are non-meritorious. You don’t start with the assumption that you 

will do away with the merit but the meritoriousness of proposals. 

So there are numbers of other things. The potential contributions 

to innovation can be another criterion that a program officer looks 

at. We then do have—and these don’t depart significantly from 

what will appear in all of the materials that we share with the 
38 

community, with what will appear in the broader impacts criteria, 

but all of these can be brought to bear as assessments are made 

about how to use the funding in the most effective ways for the Nation. 

Chairman BROOKS. These other criteria that you mentioned, are 

they required by statute, required by Code of Federal Regulations 

or internal to the way in which the NSF makes its decisions? 

Dr. MARRETT. They are internal to NSF. We do abide by the code 

that affects all the federal agencies when it comes to merit review 

so merit review in a broad sense. They are sets of principle shared 

by all federal agencies because of the code. 

But then there are the specific criteria approaches that the Foundation has outlined and those are the ones that we try to make 

sure—not try—we actually make sure they are communicated, understood by the community, thus our Web sites, our communications always are very explicit about what is going to be expected 

of those who submit proposals and how the process will work. 

Chairman BROOKS. And finally, the graph that I have talks 

about these proposals in terms of numbers but not in terms of dollars, so I know no score a number, poor to fair, a number of proposals that fit that criteria, fair to good, good to very good, very 

good to excellent, and excellent and so forth. Does the NSF have 

this information broken down by dollar figures or is it strictly on 

a number basis? 

Dr. MARRETT. We can provide you the information about dollars 

if you would like to know. 

Chairman BROOKS. Will you please provide our Subcommittee 

staff with that information? 

Dr. MARRETT. We will. 

Chairman BROOKS. Thank you. 

Next I will recognize Mr. Lipinski for his questions. 

Mr. LIPINSKI. Thank you, Mr. Chairman. 

Dr. Marrett, as I mentioned in my opening statement, I offered 

a provision in last year’s COMPETES Act that authorized prize 

programs at all science agencies. This language began with a bill 

I introduced with Congressman Frank Wolf that would have created an innovation inducement prize pilot program at NSF. We introduced this legislation based on a 2007 National Academies report which concluded that ‘‘an ambitious program of innovation inducement prize contests will be a sound investment in strengthening the infrastructure for U.S. innovation.’’ I have also worked 

with Congressman Wolf to include report language for the fiscal 

year 2012 Commerce, Justice, Science appropriations bill that asks 

NSF to make use of its new authority, especially the mechanism 

for funding high-risk, high-reward research projects. 

Dr. Marrett, can you tell me whether NSF has had any discussions on this topic and whether it has any plans to administer a 

prize program? 

Dr. MARRETT. Thank you, Mr. Lipinski. We have had extensive 

discussions on the topic and appreciate the intent, or we appreciate 

what has happened to give us the authority to try out other kinds 

of tools in the toolkit. I appreciate your earlier comment that we 

must make sure that what we do fits into the kind of support for 

fundamental work that lies at the heart of what the National 
39 

Science Foundation does and so we have been looking at what 

would be appropriate in that context, and we will be launching a 

series of experimental efforts to try to see how these fit within our 

context and so we are—we haven’t been ready to announce very 

broadly because these are highly experimental but we are giving a 

lot of attention to the use of prizes and awards as ways for advancing innovation and advancing knowledge in the Nation. So this 

would not be inconsistent with what our overall mission is but certainly we have paid close attention to the strong interest being expressed by this Subcommittee, members of the larger community in 

the fact that prizes and awards could be a mechanism for expanding our portfolio. 

Mr. LIPINSKI. Thank you. 

I want to turn now to Dr. Jose ́. One of the inherent challenges 

of NSF’s merit review process is overcoming potential conflict that 

exists when an established researcher reviews a transformative 

proposal that may question that reviewer’s work. Now, NSF proposes to address this challenge by awarding EAGER grants, retraining program officers and experimenting with alternative approaches to the merit review process. In your testimony, you mentioned a similar challenging experience by Indiana University and 

how crucial it is that IU directs available resources to projects with 

the strongest likelihood of being transformative and successful. 

Could you expand on how you ensure that funding decisions of 

transformative research are based on scientific merit rather than 

the personal considerations of reviewers, specifically, if such research conflicts with that of a reviewer? How does your process 

compare with NSF’s process, and if it differs, how do you recommend incorporating Indiana University’s process into NSF’s 

process? 

Dr. JOSE ́. Yes. Thank you very much for the question, Mr. Lipinski. One of the characteristics of a scientific endeavor is it moves 

slowly, usually moves slowly, and once in a while there is a big discovery and it has a big jump and then it continues. Scientists generally actually are conservative. When they review proposals, they 

are conservative and they are less willing often to risk the funds 

that they have to decide to award to a researcher because they are 

not sure that it is going to succeed. Now, doing research means 

that we don’t know what the answer is going to be. That is why 

we call it research because we pose a question that we would like 

to find an answer. Often it is not easy to see how to get the answer. 

We have to have a plan of how we are going to do it. 

Now, when—we had a similar problem, as you mentioned, at Indiana University. We also have limited resources that we want to 

invest in trying to catalyze the research that the faculty may want 

to do that is risky and is transformative. Now, the way of doing 

that is by requiring from the reviewers to consider what will happen if this project will succeed, how much is it going to change the 

nature of the field in which the proposal is being submitted, is it 

going to be transformative. What does it mean to be transformative? It is really not going to be just small change in the way 

things have been done but is really going to be significant in the 

way that it is being done. 
40 

The NSF and the NIH have recognized that that is a problem. 

In fact, I would say that the way they review proposals in the NSF 

is different than the way they review proposals in the NIH. NIH 

has review panels that are study sections that actually review panels for an extended period of time. In the National Science Foundation, the reviewers are usually confidential; their names are not 

known. And what happens is that the reviewers are often trying 

to make sure that the money is not going to be wasted so they are 

not as willing to risk funding a project that they are not so sure 

is going to work out. One needs to have some policies, and the NSF 

and NIH have done that. 

They have introduced projects, like the ones you have mentioned 

at the NSF, in which a certain amount of money is going to be 

given for projects that are risky but the payoff is going to be very 

high, if they succeed, it is going to be very high, money to be given 

for a project that actually going to be transformative. For example, 

the NIH has a program that is called Pioneer Awards. Pioneer 

Awards are given to projects that are very ambitious, very risky, 

that are really going to be transformative in the field in which they 

are given. So the agencies are trying to do that. It is not easy to 

do but it needs to be done. Having said that, funding proposals 

where the advance is slowly moving is also important because 

science doesn’t advance in jumps suddenly. Thank you. 

Mr. LIPINSKI. Thank you, Dr. Jose ́. Thank you, Mr. Chairman. 

Chairman BROOKS. The Chair recognizes next Mr. Bucshon from 

Indiana. 

Mr. BUCSHON. Thank you, and welcome, Dr. Jose ́. Dr. JOSE ́. Thank you. 

Mr. BUCSHON. A couple comments. First of all, I was a cardiovascular surgeon in my previous career, and so I know a little bit 

about research and I would just like to say that a lot of discoveries, 

as everyone would recognize, are made by students and people who 

are just getting their careers started in science and so that is kind 

of the area that I am going to focus on a little bit. 

Some areas of research may only have a few experts in the field, 

so to speak, and the review process, even though technically external may actually be consistent with an in-the-family-type review, 

and that happens a lot, as we know, in science. So I guess, Dr. 

Marrett, I would like to ask, how do we guard against the politics 

involved? Because everybody knows there is politics involved in 

this process. In a highly competitive academic environment, both at 

the institutional level and at the individual level, and is there a review process for the reviewers that shows their consistency and 

their willingness to assess projects that appear to be nonpartial, 

even in this setting? 

Dr. MARRETT. Let me note again first that what the reviewers, 

the panels that we call in or the mail reviewers, those are again 

advisory, and when you mention, we are very concerned to make 

sure that the process is free of lots of conflicts. There are a couple 

of ways in which those are handled. One is by having all reviewers 

do have to sign the conflict forms to indicate that they are not in 

conflict. Our program officers do training for ethics and conflict 

training. In fact, interestingly, we also have that now, ethics training, for students, undergraduates, graduates, post-doctoral stu-
41 

dents. There is also a requirement for institutions, that institutions 

must assure that there isn’t a conflict of interest that would be 

there. 

Even with all of that, we still have other sorts of processes to 

oversee how final awards are made and so the awards are not 

made exclusively on the basis of these recommendations. They go 

up through other levels for review. Then finally, if there still remain problems, those can be undertaken through our Office of the 

Inspector General. If there is evidence of waste, fraud or abuse, 

those we would report to the Inspector General and the Inspector 

General gets them through other processes. So we have had—we 

put in place checks and balances to reduce the likelihood that there 

will be the kind of personal considerations that I know would always be of concern, and that is why we have what I say again are 

the checks and balances to assist them that will not rely exclusively on any single individual. 

Mr. BUCSHON. Thank you, because looking over the process, as 

you know, there is a disparity in the percentage of grants being 

made to new investigators versus established as well as universities that traditionally get federal funding versus those that may 

be new to the process, and I understand all the reasons why those 

may occur. People honestly are more experienced at submitting 

their proposals and so they may be more likely to be successful. 

But the reality is, there probably is a little bit of this internal family-type thing that does come into play. Again, saying that, you 

know, students and new researchers are ones that frequently make 

the most discoveries. I am glad to hear your explanation that these 

people, you know, they appear to have the same opportunity. 

Dr. Jose ́, I will ask you a question. In your experience, many 

years of experience, has the review—I mean, is there a lot of political interplay in the review process, and you are free to—— 

Dr. JOSE ́. I wouldn’t call it political. I think that often people 

that have been working in the field for some time, they think that 

they have the right answer, and when someone else comes in and 

say well, maybe I have another way of looking at the problem with 

a different answer, they may say oh, no, no, no. There is some resistance at the beginning. But eventually one of the nice things 

about science and the review process is that other researchers will 

actually come up with results that will say yes, the new way of 

looking at the problem is the way to do it. But it is always, just 

as in human nature, I mean, all of a sudden you have a way of understanding a problem, you think you have the answer and someone else comes in with a more innovative way of doing it and says 

oh, maybe not, maybe it is not a good idea to support it but eventually I think the system works itself out to actually recognize new 

ways of doing things, and even when you have a group of people 

that are—what do you call them, in the family—that look at problems in one way, eventually they have to give credit to the new 

ways of looking at a problem and solving the problem. 

Now, let me say something about the young investigators. The 

NSF has a program for—and NIH too—for recognizing junior investigators that gives them some extra points, if you wish, when they 

start the running because they can’t compete with the ones that 

have been in the field for 40 years. So there are programs that are 
42 

working very well. They are not as good as they should be, but they 

have recognized those problems. 

Mr. BUCSHON. Great. Thank you. I yield back. 

Chairman BROOKS. Thank you, Mr. Bucshon. 

The Chair next recognizes Mr. Clarke. 

Mr. CLARKE. Thank you, Mr. Chair. We are a few days from possible government default, and to me, I am the new guy here, relatively, like a few months here in Congress. Many years ago, I was 

a staffer. So I am going to say that maybe I have a perspective that 

could give us a clearer picture on things because I haven’t been 

part of the system for a long time. But one underlying issue that 

I see that Members of Congress have regarding taxpayer-funded 

basic research as we are dealing with here is they want to make 

sure that we are not wasting money, that it also goes to make a 

constructive difference in our country, and the broader impacts criteria kind of lays out what our objectives would be. 

Dr. Jackson indicated, I think, a very important issue, though, 

is that how do we best measure whether that research is effectively 

achieving those broader impact goals, especially when it is research 

that is totally innovative and totally transformative. It is almost 

like, you know, asking the veteran Members of Congress to quit 

blaming each other and think of a different way of actually passing 

legislation by inspiring the public with an enlightened point of 

view. I am not sure if that would actually come from entrenched 

folks here in this system but it could. 

So in that sense, I am just posing this question to all of you. 

What type of measures do you think we should look at? Dr. Jackson indicated that maybe empowering the NSF to easily gather 

more data to measure the success of the research could be effective, 

but along those lines, we here in Congress want to make sure that 

when we are spending taxpayers’ dollars that the taxpayers are 

getting the best benefit for those dollars. How do we measure it? 

Dr. JOSE ́. If I may answer the question, I mean, there is a program now, a federal program that is called NSTAR where they are 

going to try to find out for every dollar that is spent by the Federal 

Government, how many jobs it will create, how many people are 

educated, what is the impact that those dollars have in the economy, for example, and the number of jobs and the competitiveness 

of the United States when comparing with other countries. It is a 

very difficult thing to do. It is a multi-variable type of problem. 

One answer is not that simple. We had this question asked of us 

when the stimulus money was given to universities, and we were 

asked, and I was interviewed a few times, and people would ask 

me, well, you are going to get $20 million so how many jobs will 

you create, how many jobs were created from that $20 million, and 

it was not an easy question to answer, particularly because the 

money had just arrived a few months before, and second, because 

in particular when it has to do with research, it takes some time 

actually to see the results. 

But I will say that in my view, the money that the United States 

has invested in funding research and particularly after the second 

World War has maintained the preeminence and the dominance of 

the United States economy. Furthermore, it has been the seed 

money that was used to create the best university system we have 
43 

in the world and is the envy of the rest of the world. I would call 

that evidence that everything has been working out. Now, having 

said that, we need to review constantly how we are doing things, 

how we are investing the money, because times change and we 

should look at things in a different way like today with the budget 

problem that we have, the Federal Government budget problem 

that we need to be very careful about how we invest money, federal 

research taxpayer money. 

Dr. MARRETT. I would just elaborate a bit on that. I think the 

points are very well taken. The National Science Foundation pays 

attention constantly to issues of metrics because we take seriously 

our responsibilities to be accountable to the larger public. Those 

metrics are reflected in many ways in the kinds of requirements for 

what must be done, broader impacts. There are the assessments 

that have to take place as people are reporting on the broader impacts. So we are constantly looking at, reviewing, examining what 

would tend to be approaches that are possible. 

I think the other issue, a key issue here is, it is much easier 

often to think about short-term returns for those things that might 

have—we can look at, then the longer-term returns, and it is the 

longer-term returns that are very important for an agency such as 

the National Science Foundation. We have done studies that have 

looked at—we take account of some of important developments and 

track them back. Sometimes those have been over an extended period of time. Thus, I think a part of Dr. Jackson’s comments had 

to do with ensuring that our attention on returns would not focus 

us so heavily on immediate returns that we are unable to think 

about and invest in what will have to be the longer terms and the 

metrics appropriate for the longer life of activities. 

Dr. JACKSON. And yes, I would say that is exactly true. When 

you analyze and measure the impact of fundamental and basic research, it takes a long time sometimes to see what those impacts 

are, so it is a long-term process, not just a short-term one. 

Dr. YAMAMOTO. I agree with that. I think you raise a really important question, and the way that I would look at it, as I said in 

my remarks, is that this must be viewed as a continuum that recognizes that application that really reaches a level of being evident 

in addressing national goals, requires a continuous input of fundamental information about complicated processes and that for any 

one grant application, it could be impossible actually to ascribe how 

they may be approaching national goals. 

Let me do one quick example. My colleague, Herb Boyer, in the 

early 1970s was working on a process that was sort of an esoteric 

process in which DNA that got into bacteria was broken down for 

some reason and he didn’t understand, and this was called DNA 

restriction. The kind of DNA that would get into bacteria would be 

restricted by breaking it down. And very few people, very few scientists were interested in this process. I would wager that if this 

were put up against the test of addressing national goals, it might 

not do very well. That discovery of what DNA restriction modification was turned into the capacity to make recombinant DNA, which 

then turned into the biotechnology industry worth billions and billions of dollars and leading the world. So it was impossible to as-
44 

cribe with that particular grant what its impact would be but there 

it was. 

So I think that the goal would be in fact not to look at these 

things individually, to ask collectively whether the federally funded 

scientific research endeavor in this country is actually moving 

things toward national goals, to recognize that we understand so 

little about these processes that a continued input of investigation 

at the fundamental level is essential and that that continuum really does work. 

The NSF is the basic research engine of the Federal Government 

and as such really needs to be celebrated and protected. There are 

mission-oriented agencies including the NIH in fact, the DOE, the 

DOD, Department of Agriculture, Commerce, elements of Commerce and so forth that have specific missions and I think that 

those should be tested in the short term against whether they are 

addressing national goals but always to recognize that input of fundamental science from the National Science Foundation and elsewhere is always going to be essential. 

Chairman BROOKS. The Chair next recognizes Mr. Hultgren. 

Mr. HULTGREN. Thank you, Mr. Chairman, and thank you all for 

being here. 

I have a question for Dr. Yamamoto and Dr. Jackson and Dr. 

Jose ́. Each of you has testified about broader impacts imposed on 

the Foundation by Congress in the 2010 American COMPETES Reauthorization Act and that it will adversely affect funding decisions 

by requiring reviewers to consider metrics far outside the scope of 

the intellectual merit. Dr. Jose ́ suggests that rather than enhancing competitiveness and national security, they risk damaging our 

economic and national security. I wondered, are reviewers even 

privy to the national priorities prior to the meeting of a panel, and 

just asking if you would comment, if each of you would comment 

on this. 

Dr. JOSE ́. I would like to say that all of us are aware of our environment. All of us are aware about what is going on with Congress 

today. For example, all of us read the news, all of us know what 

is important. All of us are very aware about what is happening in 

the growth of the economies in the rest of the world and how the 

United States is having to compete very strongly with China, 

which in particular is growing very fast and we know or we feel 

that one of the ways in which the United States is going to succeed 

is by innovation, by being innovative, by being creative and introducing new knowledge and new ideas, and I think that we should 

not limit the imagination of investigators and researchers by having a large number of restrictions about how the money is going 

to be awarded. I mean, we put too many restrictions that constrain 

the freedom that researchers have that often leads to the creation 

of an industry like biotechnology, as Dr. Yamamoto just said. It is 

the essence actually of the freedom of thinking freely that creates 

new innovations that were totally unexpected. 

Mr. HULTGREN. Dr. Yamamoto or Dr. Jackson? 

Mr. YAMAMOTO. As I said in my comments, I think that one of 

the critical features of merit review is a singular focus on scientific 

merit and that anything outside of that is valuable and important 

and in fact essential as it may be such as addressing broader na-
45 

tional goals should reside outside of that process and that asking 

merit reviewers to make an assessment of how well the fit is of a 

given project to national goals I think is inappropriate. I think the 

agency individually and collectively, as I said, the federal scientific 

research endeavor needs to be paying attention to whether national 

goals are being addressed but that neither reviewers nor applicants, in my opinion, should be mandated to make such a statement. If there is a clear application, of course, it would be to the 

advantage of the applicant to make that clear, but if not, and it is 

a fundamental question, then you are really asking reviewers to 

make guesses, and I think one of the great things about the merit 

review process is that there is not a lot of guesswork there and 

that we shouldn’t be asking reviewers to step outside of their expertise. 

Dr. JACKSON. As I mentioned in my testimony, there are a lot of 

proposals that are sent to NSF that are very, very good, excellent 

proposals that just cannot get funded because there is not enough 

money so that when the program officer at NSF looks at 40 proposals that are all great and could have a huge impact on this 

country and they have to decide which of those 40 to choose, then 

the broader impact does come as an advantage to choose between 

all these great proposals, and that is an important role, I think, for 

the broader impacts criteria and should be kept as such because 

there are clearly far more many proposals that are excellent that 

can be funded. 

Mr. JOSE ́. May I say one more thing? 

Mr. HULTGREN. Certainly. 

Mr. JOSE ́. Just going along with what you said, I just came to 

Indiana University last year, and we have limited resources but we 

introduced a seed funding program for people that would be willing 

in interdisciplinary research. We actually asked people first to have 

new projects, new collaborations and so on and so forth. We know 

of about 165 proposals and we could only fund 10 percent. There 

were many that were recommended for funding but we had to 

make a decision at the end. We decided never to fund something 

that was not recommended on merit, but then we have to use some 

criteria to decide which projects were more likely to be good and 

producing new results or new discoveries and so we had to reduce 

the total number of projects that we were going to fund using other 

criteria. So it is important to have a combination perhaps of the 

two. 

Mr. HULTGREN. My time is winding down. If I could just ask Dr. 

Marrett just to follow up quickly, Mr. Chairman? 

Chairman BROOKS. We are going to do a second round, given our 

time availability. Go ahead and—— 

Mr. HULTGREN. Let me just quickly, Dr. Marrett, I know this is 

something that Congress has directed you to do. It wasn’t necessarily something internal that you decided. I wonder if you would 

have any comment on what your colleagues at the table stated, 

what your feelings are about this. 

Dr. MARRETT. Well, it is not quite true that this is only, that we 

only are talking about broader impacts because of Congressional 

activity. No, from the outset, from the time that NSF was founded 

and started making awards, it has always considered both the tech-
46 

nical side and how we address the national goals. The idea, the terminology of broader impacts was really brought in in 1998 when 

the Foundation said we are now going to collapse several criteria 

down to the two criteria, intellectual merit and broader impacts. 

The broader impacts statement or the expectation is that the goals, 

these do have to reflect some of the kinds of goals that are expected 

by the Nation of the investments in science. So they would include 

economic competitiveness, development of a STEM workforce, increased participation by underrepresented groups, partnerships between academia and industry. In other words, we have seen these 

as very critical for the Nation and reasonable as an agency thinks 

about how it will allocate resources. I think the only difference between the NIH and NSF approach, NIH does much of the same 

kind of a thing, it is at a different level. So the advisory councils 

take up the questions of how should the portfolio and how should 

these other national considerations be taken into account. They 

happen to be done as a part of our regular merit review process 

in NSF. But this is not a new idea. 

Mr. HULTGREN. But wouldn’t you say, though, it is an—this expansion is mandated by COMPETES? Isn’t that true and that 

makes it unique? 

Dr. MARRETT. No, what COMPETES did was to say be much 

clearer and there are other things with reference to broader impacts, and so that is what the Science Board is looking at, that is 

what the expectation of the Foundation, and that is what we do believe we can be much clearer on what broader impacts should mean 

and how it would come up with indications appropriate metrics, 

but again, this was a part of an ongoing process and we appreciate 

then the interest of COMPETES, the interest of others in helping 

us continue to move forward in explaining and being accountable, 

again, to the Nation. 

Mr. HULTGREN. Mr. Chairman, you have been very gracious. I 

yield back. Thank you. 

Chairman BROOKS. Now for the second round of questions. The 

Chair will start first. 

Dr. Jose ́’s testimony stated that the merit review process is at 

best a system of checks and balances, and this question is for all 

of you. Being a member of a government body built to provide a 

system of checks and balances, I am interested in this theory. In 

the context of the financial circumstances that Congressman 

Clarke so notably described that we face here in Washington, DC, 

do you or any of your colleagues sitting with you today have any 

additional suggestions on how to strengthen the system based on 

our need for a balanced approach to federal funding, particularly 

to ensure the advancement of science as our sole or primary funding criteria? 

Dr. JOSE ́. Actually, let me actually say something. I mean, NSF 

has no specific mandate to do other than basic research and try to 

uncover new laws of biology or physics. The NIH has a very clear 

mandate to try to find the best way to improve American citizens’ 

health. The Department of Energy has very special goals, the Department of Defense, each one of them has a merit review system 

that works differently than the one at the NSF. The NSF is to 

some extent unique. 
47 

Having said that, I think that when I said checks and balances, 

what I mean is that the reviewing process of papers, not just proposals for funding—I mean, we are supposed to publish, many of 

us that work in science, our research, our results. It is very hard 

to get results published in the best journals if they are not of the 

highest quality. The same thing happens in the merit review process. You are not going to get funded unless you have the highest 

quality. So in a sense, we have like a voting system. Our colleagues 

actually vote about what we do, if they think it is good or not, as 

we get elected or reelected in Congress or not, I mean, the voters 

have to decide if we are doing a good job or not. As researchers, 

if we are doing a good job, our colleagues will vote that yes, get 

funded, yes, get published, yes, get evaluated for lectures and so 

on and so forth. So that is what I mean in the sense that we have 

checks and balances. 

When a recent result is wrong, the community will find out that 

it is wrong because when they try to reproduce that discovery and 

they cannot reproduce it, like cold fusion, for example, it just will 

die immediately. That is what I meant by checks and balances. 

That is how science advances, not in a straight line continuum. 

Chairman BROOKS. Thank you for the additional information 

with respect to checks and balances. Do you have any specific suggestions on how we can strengthen or improve the process or system in order to help ensure that these scarce federal dollars are 

being spent on basic science or research rather than being affected 

by the criteria that are extraneous to those primary functions? 

Dr. JOSE ́. Yes, of course. I think that, as I said, it is a human 

endeavor when people try to review proposals. As Representative 

Bucshon said, I mean, there is some element of human nature that 

interferes and how do we review proposals, who is reviewing the 

proposals, is the best investment that one can make or not. But we 

recognize that is a problem we have. We develop all kinds of checks 

and balances from the agencies, from you. You all the time are telling us you have to convince us that you are doing the right thing, 

otherwise we won’t give you money to do the research, and it is not 

perfect and we keep changing it and correcting it as we go. 

Chairman BROOKS. Well, my question is, do you have any specific 

suggestions on how to improve the process? 

Dr. JOSE ́. Yes. I think that—let me—yes. I think that—actually 

I have thought a lot about that. There are blind types of reviews 

because often the name of the scientist plays a role in if it is going 

to get funded or not going to get funded. Perhaps even the institution, perhaps the project and the area. I think that—let me see. 

How do improve the system? I think that we have to make sure 

that we choose the right reviewers. We have to make sure that 

there is no conflict of interest with the reviewers being part of the 

same group of people that are funding each other. Other than that, 

I think I have many things to say but it would take a while to describe. 

Chairman BROOKS. Well, thank you, Dr. Jose ́. You mentioned 

blind reviews. What are those? 

Dr. JOSE ́. Well, for example, if you submit a proposal, you don’t 

know the name of the author of the proposal, the institution of the 

proposal, as you don’t know the name of the referees, and then you 
48 

will just blindly decide if this is really good or not whereas the way 

we do it now, I mean, we know the name of the reviewer. We can 

know if they are very good or not very good and decide—that may 

influence it a little bit. 

Chairman BROOKS. Are you recommending that we have blind 

reviews? 

Dr. JOSE ́. No, no. We have tried to do that. In fact, there is a 

journal trying to do that and they have mixed reviews, mixed results. I mean, they tried to do that, not having the name of the author, not having the name of the reviewers, and they just review 

it blindly. I am not sure that the end result was satisfactory as expected. 

Chairman BROOKS. Dr. Jackson or Dr. Yamamoto or Dr. Marrett, 

do you all have any specific suggestions that you think we should 

look at in order to try to improve the process? 

Dr. JACKSON. Well, I do think—I will go back to what I mentioned in my testimony about giving NSF program officers the ability to triage proposals. At this point in time, all these proposals 

have to go through merit review. That taxes the scientific community, that taxes the program officers, and we need to make this— 

since we can’t fund all the proposals, and there are a lot of excellent proposals—there are more excellent proposals than we can 

fund—it makes sense for us to spend our time thinking about 

which of those excellent proposals we should fund rather than 

spreading our time across the whole gamut of proposals. 

Chairman BROOKS. I am sorry. I understand what you have already testified to, and my question should have been better. Do you 

have anything specific in addition to what you have already informed us of? 

Dr. JOSE ́. I am sorry. I must say something about that, about 

what Dr. Jackson has. I mean, at the NSF, there are increasing 

number of proposals that have been submitted to the NSF for review and they have decided to assign to the university to have a 

limited submission-type thing. You can only submit two or three 

grants when there is several million dollars rather than having 10 

or 15. That is a change that was made and it is a change also that 

has been made in the NIH as well. 

Chairman BROOKS. Thank you. 

Dr. Jackson, I don’t know if you have thought of anything in addition to what you have previously shared with us? 

Dr. JACKSON. It is a messy human process and it would be difficult to make it more efficient. It is always something you are 

struggling to improve. 

Chairman BROOKS. Well, I have to admit that when I heard it 

compared to the election process, that raised some concern. 

Dr. Yamamoto or Dr. Marrett, do you all have any other specific 

suggestions on what we can do to improve the process? 

Dr. YAMAMOTO. Let me make two comments that I want to frame 

as an expansion of what President Obama said when he addressed 

the National Academy of Sciences in April 2009. He said that a 

basic research project may not work for a year or a decade or ever, 

and it is for that reason that public funds, government funds, 

should support basic research because the private sector will always underinvest in that side because when a project works, it 
49 

works for everyone. When it doesn’t work, of course, the private 

sector loses. Maintaining that basic research and engine is really 

critical, and so let me make one suggestion, which is an expansion 

of what I said about transformative research, and I will try to 

frame briefly why I think it really needs to be a distinct sector of 

research that is supported in this country, and then a second comment that really approaches this question of broader impacts. 

So transformative research, I suggested in my comments, and I 

will expand very briefly, should be viewed as a different kind of research. It is destructive of current views rather than extending 

them or deepening our understanding of current paradigms, and to 

do that requires that we recognize that special characteristic and 

have a review process that is very distinctive. I proposed the transformative research track at the NIH in 2006. It has been installed 

in a limited way. I chaired the first two rounds of review of those 

applications when they came in, and an essential feature of I think 

making that program successful is actually carving out a different 

mode of merit review, one that is driven by reviewers that are not 

deep content experts in the areas that are being proposed but rather are generalists who think deeply, who are able to recognize really high-impact ideas when they see them and are able to celebrate 

the fact that those ideas may actually bring down their current 

ways of thinking about a process. 

Luckily, such scientists exist in this country. They can be found. 

They can be identified. They are willing to participate in this way. 

They are the kinds of people that are finally getting challenge 

grants from the Gates Foundation that are proposing—making proposals to the transformative track and are willing to review in that 

way. So it takes a different kind of review that recognizes that you 

don’t bring in content experts who made the paradigms because 

they are going to be immediately critical of someone who comes in 

and says I think the current paradigm is wrong. I could expand on 

that further. 

Let me move to the second point, something I didn’t mention, 

didn’t talk about at all, and in a sense, you could view this as being 

outside of the merit review process, but I think not, and that is the 

idea of recognizing, especially with this broader impacts mandate, 

that basic research has advanced to a point where it can increasingly be applied, and since I work in the biomedical sphere, I see 

this all of the time with the opportunities of our faculty in my own 

laboratory to be able to move fundamental research to application, 

but there is the so-called valley of death that I am sure you have 

heard about, that especially in the current economy has made it 

very difficult to be able to do the few experiments that are needed 

to make it evident to pharmaceutical companies and the biomedical 

sphere that there is a worthy investment there. I think that we 

have an opportunity to be able to extend the continuum to be able 

to build new interfaces between academic basic research funded by 

the NSF and other agencies and build public-private partnerships 

that can more effectively move ideas into application, and in that 

way be able to really directly approach this challenge of addressing 

broader impacts that the COMPETES Act really challenged all of 

us to be able to do. 

Chairman BROOKS. Thank you, Dr. Yamamoto. 
50 

And Dr. Marrett, I don’t know if you have any suggestions. It occurred to me that if there some, you probably would have already 

implemented them as Deputy Director and Acting Director but 

nonetheless, if you have any other suggestions on how you think 

we can improve the process, please share it. 

Dr. MARRETT. And I take your question to mean, what especially 

might the subcommittee do, and I have two recommendations 

there. One is to share broadly the understanding of the process of 

merit review. I am not sure how widely the process is understood 

by, I wouldn’t just say Congress, or general public, yet if we are 

drawing on public resources, we need to make sure that that process is understood. So this hearing represents a good approach for 

broadening knowledge about the process. 

The second recommendation I would make for the Subcommittee 

is, I hope you would be open to our sharing with you the experiments, the pilots that we are trying. We do say that there are 

areas where there can be improvement. Let me take the case of potentially transformative research. We are not all that satisfied but 

we are experimenting with, do we advance the potentially transformative research by identifying proposals once they have come in, 

by soliciting new ideas, by thinking of the sorts of experience from 

NIH. We are trying all of that. We would welcome then coming 

back as we have learned from those experiments to say more about 

what should be advanced. We are not quite ready yet to say these 

are the things that we think ought to be put in place. And finally 

there, I would say whatever, we think that at the core, that core 

has to be maintained. I mean by that core that commitment to 

merit and so whatever the sorts of changes, the changes are around 

the margins in many respects. They never should undermine the 

importance of relying on the meritoriousness of the idea and the 

approach that is being proposed. So thank you. 

Chairman BROOKS. Thank you, Dr. Marrett. 

Mr. Lipinski, thank you for being patient. 

Mr. LIPINSKI. Thank you, Mr. Chairman. I think that—I am glad 

we went to a second round of questions because I think we really 

got into some really good ideas here about how, you know, to make 

improvements to the system, and I think actually I would ask—the 

Chairman asked Dr. Marrett if we could possibly do, maybe a tutorial, and get into how exactly this process does work, because we 

are talking about it here but even someone who has presented a 

proposal and has gotten a grant, I would like to really know better 

how the whole system actually works and maybe the opportunity 

for members to sit down and go through a tutorial. I think that 

would be a great idea for all of us. 

Chairman BROOKS. That is a good suggestion, and I welcome it. 

Mr. LIPINSKI. Because I think there is a lot to the process, and 

we are talking about the process but I think there is a lot of details 

to the process and things that probably each of us doesn’t completely understand and might be able to get a better understanding 

of that. 

Two things I want to get into. The first one, Dr. Jackson had— 

one of the main proposals you had to sort of lessen the work is for 

the program officers to be able to essentially triage and put aside 

proposals that come in that the program officer doesn’t see there 
51 

being merit to. I wanted to get a better understanding of what you 

are saying how that would be done, and then I will get Dr. 

Marrett’s response on that. Are you saying then that immediately 

set aside these proposals without doing any sort of response to 

them? Because right now all the proposals get a response from the 

reviewers. It would seem the only way you could save time really 

is if the program officer just put these aside without any comments 

and just send it back and say rejected right off the top. In that 

way, you would not be getting—you would not be giving the researchers any feedback at all. I was just wondering how exactly 

you saw that working. 

Dr. JACKSON. Right, and you are right. That is the downside to 

triaging proposals that clearly do not—clearly aren’t up to snuff, so 

to speak, is that they miss the opportunity for a certain amount of 

feedback, and there are several ways you can handle that. Certainly for the petroleum research fund at ACS, there are certain 

ways we deal with that. One is that you can ask for a review from 

the program officer who in our case and in NSF’s case is an expert 

in the field so that program officer does have to write up something 

about what is a problem with the project and this sort of thing. Another thing that we do with the Petroleum Research Fund is to provide workshops particularly focused at younger researchers, but 

anybody can come, that talk about what makes for a good proposal 

and so we are training the new generation and new people to the 

field how to write proposals, what we are looking for and what 

needs to be done, so we are doing this to counteract maybe some 

of the individual feedback that could be—that would be given if it 

was sent out to review. 

But you have to understand that a proposal goes out not just to 

one reviewer but to a number of reviewers, so there are possibilities too you could have; if you realized that this is not going to be 

funded, you could ask one reviewer perhaps to provide a review, to 

provide advice to the proposal writer, or there are a number of 

other things you can do that would decrease the amount of work 

for each proposal. But in the PRF, we recognize that these are just 

not going to get funded and so we spend a limited amount of time 

with them and try to do our education separately. 

Mr. LIPINSKI. Yeah, I understand the need to save time and I 

think we all want to find ways, because there is—reviewers are 

overburdened and it is hard to get people to serve in that capacity. 

I just want to give—and I am not, you know, being critical of that 

proposal. I want to see what Dr. Marrett would have to say about 

that. 

Dr. JACKSON. Well, and we also—one more thing. We also encourage those proposal writers to call our program managers and 

talk to them about their proposal so there can be some kind of exchange and learning going on. 

Dr. MARRETT. We are absolutely intrigued by anything that could 

help reduce the burdens on program officers, on reviewers. The 

number of proposals we are getting, the number continues to climb 

and it is very difficult. On the other hand, there is the kind of view, 

a very pervasive view that a part of the whole process is to help 

give good feedback, especially help in the cultivation of new investigators and thus our program officers are hesitant simply to say 
52 

let us try to move this as quickly as possible. Because I think there 

is a responsibility to help people understand where the problems 

might have been, to help cultivate the ways in which they might 

develop stronger proposals. It is a dilemma then of how to have an 

effective, efficient strategy that is at the same time an educational 

strategy that we are all seeking to accomplish. 

So as I said, we welcome ideas but we are trying to examine 

them in the context of the larger kinds of things that we seek to 

do in the whole development of the science and engineering community of the United States. 

Mr. LIPINSKI. And I wanted to go on to another, if the Chairman 

will indulge me. 

Chairman BROOKS. Feel free. 

Mr. LIPINSKI. I wanted to ask about conducting the committee reviews in virtual environments. It is something I mentioned in my 

opening statement. I personally have had no experience at all. I 

have no avatar. I have not gotten into this so I can’t really speak 

from any expertise here, but I know this is something that has 

been looked at, and I was wondering if—I will start with Dr. 

Marrett. I want to ask all the other panelists about their opinions 

on using this as a way to make it easier in some ways in doing this 

in a virtual environment. You don’t have to fly people into Washington to do this, and other possible merits to using this system. 

I would also think that the blind review process—and also having 

people come together in a way that they don’t know each other that 

could impact, change the dynamics of the group. So, Dr. Marrett, 

we will start with you. 

Dr. MARRETT. Yes, we are looking very much at virtual environments. There are experiments that have been taking place, and 

these environments, thinking of virtual panels especially, we have 

been thinking of them for two reasons of reducing some of the cost, 

the cost for reviewers, and for potentially expanding the pool of reviewers. In other words, there are some people who would find it 

difficult to do the traveling since most of these are here in this 

area. We thought this would be a way then to bring more people 

into the process. 

What we are looking at, what are the sorts of things that have 

to be taken into account. There are matters of security, for example, that you have to make sure that you have been abiding by. 

You raised the questions about the kind of interpersonal dynamics. 

There, we are actually drawing some of the research being sponsored by our Directorate for Social Behavioral Economic Sciences 

where there are studies of human dynamics, enough for us to understand how those dynamics might come into play and actual 

studies specifically of virtual environments. All of this then we 

think we need to take into account and that is why we are experimenting but we are trying to bring the knowledge to bear as much 

as we can to ask what would be the consequences and where there 

might be some savings but where there might be some losses if we 

are not careful about how we try to institute the idea of virtual environments. 

Mr. LIPINSKI. Any other panelists have any opinions here? I will 

start with Dr. Jose ́. 
53 

Dr. JOSE ́. Yeah, let me say that since I am quite old and I was 

funded for many years, I can see the evolution of how proposals are 

reviewed by the NSF. For a long time, there was only by mail review, and they only waited until the reviews were done and they 

analyzed what the kind of grade each one of the reviewers gave. 

Then they went into having a combination of having mail and 

panel reviews. And I can tell you that by having done both, there 

is a significant change in the actual assessment of the proposals 

that arises from meeting with people live than just from reviewing 

it by yourself. 

Okay. Now, with the virtual analysis they have to do, it is very 

important to know what kind of human dynamics are going to influence to some extent the reviewing process, and the reason I am 

saying this is that one of the committees that I was a member of, 

the mail reviewers gave much better grades to the proposals than 

when we sat down and discussed the proposals and interacted and 

listened to everybody else live, okay, so what effect would it have 

to have it virtually and not face to face, one has to wonder. But 

it is course a very useful and important thing to explore, yes. 

Mr. LIPINSKI. Who else? Dr. Jackson? 

Dr. JACKSON. Living in Albuquerque, New Mexico, which seems 

to have no direct flights to anywhere, the idea of having virtual 

meetings is very exciting to me and to my colleagues because we 

are more likely to be able to be involved in these sort of—you 

know, become more—participate more in these sort of activities. 

But also because of that, I have done a lot more of this sort of virtual meetings and this sort of thing and I would say that yes, perhaps when you first start doing it, it is a little unusual but it is 

remarkable how with time you can become very accustomed to it, 

and I highly encourage NSF to consider these sorts of possibilities 

because I think the kind of expertise they could tap into will only 

become greater and richer. 

Mr. LIPINSKI. Yeah, obviously I brought up avatars. You don’t 

have to do it that way, but Dr. Yamamoto? 

Dr. YAMAMOTO. I agree. I think there are two drivers for this increased—rapid evolution toward electronic environment for review. 

The first is that those of us who didn’t grow up using those media 

are disappearing and being replaced fortunately by people who 

have screens in front of them their entire lives and are very comfortable working in that environment. And the second is the technology development is marching along very quickly and there are 

electronic video conferencing modes that you are looking across the 

room at a screen and it looks exactly like we are looking across the 

room at you. And so the opportunity for interactions, which I agree 

with Dr. Jose ́, is essential in being able to come to this sort of 

agreement about the scientific merit of an application that can 

really occur increasingly seamlessly in this sort of environment. 

Let me make one further comment, because you asked, Congressman Lipinski, about this idea of blind reviews that Dr. Jose ́ had 

mentioned, and just say there again here is the spectrum of experiments ongoing in different agencies. You may know that the Howard Hughes Medical Institute does actually the opposite of a blind 

review. In fact, rather than choosing scientific projects as the agencies that we all represent and are working and do, they simply 
54 

choose people, and so the identification of the person is the whole 

ballgame for the Howard Hughes Institute. The Gates Foundation 

in their initial round of reviews for their Grand Challenges grants, 

which I participated in several times, is blinded to the identity of 

the applicants, and after they make it through the first round, then 

that identity is revealed, and then NSF and NIH, the federal agencies that I am familiar with, the identity is always known. 

There is a risk of conflict of interest or other kinds of biases that 

can be introduced at that point, but let me say that having been 

experienced in review for at least NSF and NIH extensively that 

I don’t actually see that actually occurring, and instead knowing 

the identity of the investigator, in fact, building it in as an explicit 

criterion for the merit review process, turns out to be very important. There have been studies done and arguments made that in 

fact the strongest predictor of success of a given scientific proposal 

is in fact the track record of the investigator who is making the 

proposal. So in that sense, knowing that person, their past capacity 

for doing innovative work—and let me just insert here that this 

does not mean—because very often that statement that I just made 

raises concern that this would bias against young investigators. In 

fact, it does not, and I think the clearest view of that is that I have 

been involved, and I am sure my colleagues have as well, for years 

in making judgments about graduate fellowships, postdoctoral fellowships, young trainees who have very little in the way of conventional track records but it is very easy, I think leaving those meetings, those of us who are on those panels, feel very confident that 

we have been able to make the right choices for identifying the best 

candidates based on what they have done before, even though they 

have been in a training mode and very different from being an 

independent investigator, and data prove that out that those people 

that win such fellowships and awards go on to be able to do very 

good work. 

And so my view is that having that identify early on at least— 

I think the Gates Foundation strategy actually works quite well— 

but being able to unmask those individuals early on, to be able to 

see their track record and build it into the merit review process is 

important. 

Mr. LIPINSKI. Well, let me throw my own two cents in here. I am 

not always certain from—I think it depends on the field possibly. 

In my background as a political scientist, I was not always convinced that it was helpful to the advancement of knowledge to 

know who was proposing something because sometimes I think 

that had an influence that wasn’t good on review processes, and I 

am talking more about journal reviews. But that is my own two 

cents. It is very interesting to hear, though, Dr. Yamamoto’s and 

everyone else’s experiences with this, and I thank all of you very 

much and I especially thank the Chairman for his indulgence here 

and for this hearing. 

Chairman BROOKS. My pleasure, but we are not finished yet. I 

am going to follow up on something that Mr. Lipinski noted, and 

that is, we have talked about the virtual panels but what is the 

cost of in-person panels? 

Dr. Marrett, do we have any kind of transportation costs or hotel 

costs or is there a budget line item, or how much does it cost per 
55 

panel or how many panels are there per year? Can you share some 

information with us about that? 

Dr. MARRETT. We will get that information to you. We actually 

have done the calculations of what the costs are associated with 

panels, and those do include the travel costs, of course, and there 

are hotel issues there, and we had started as we were looking at 

what would be the savings from virtual panels, and I should mention that in the case of avatars, we have had reviews done in Second Life. But in terms of the sorts of costs associated with virtual 

panels, we are not quite sure because there are other kinds of 

things. We just haven’t explored enough now. We can’t do the comparative information but we can get to you how many panels, what 

the cost is of an average panel for the National Science Foundation. 

Chairman BROOKS. And if you can do a comparison format, too, 

I would anticipate that the virtual panels are significantly less expensive. But if we have some kind of comparison model, that would 

be beneficial. 

Dr. MARRETT. We will try that, but why I say it is not necessarily 

going to be much cheaper because there is the question of the technology that we might have it on our side, but it is also got to be 

at the other side, and one of the things that has been talked about 

will be using regional panels where there will be cost of travel to 

the regional panels as well, the enhanced security costs that come 

into play, so we are not quite sure at the outset that there would 

be lots of savings that could happen over time in a number of ways. 

That is why I said we won’t have for you—it will be very rough 

kinds of estimates when it comes to virtual panels but we can tell 

you about the investments currently made in the in-person panels. 

Chairman BROOKS. Thank you, Dr. Marrett. 

As a short concluding statement, I don’t know if you all have access to this, perhaps Dr. Marrett already has it, but some kind of 

compilation of success stories with NSF, the kind of information 

that this Subcommittee can share with the public or the Members 

can be familiar with so when asked, we can share it with the public. I am sure you are all familiar with the relatively famous 

shrimp on a treadmill commercial that has been running over the 

last month or two, and I am afraid that that may give the general 

public some dissatisfaction with the way their tax dollars are being 

spent, so I think it is important to also have information that we 

can share with the public that shows success stories and the kinds 

of returns on the investments that we are asking taxpayers to 

make. 

Having said that, I want to thank the witnesses for their valuable testimony and the Members for their questions. If the Members of the Subcommittee have additional questions for the witnesses, you are free to submit those, and we will ask you to respond to those in writing. The record will remain open for two 

weeks for additional comments from the Members. 

The witnesses are excused and this hearing is adjourned. 

[Whereupon, at 11:43 a.m., the Subcommittee was adjourned.] 

(57) 

ANSWERS TO POST-HEARING QUESTIONS
58 

ANSWERS TO POST-HEARING QUESTIONS

Responses by Dr. Cora Marrett, 

Deputy Director, National Science Foundation 

Questions submitted by Chairman Mo Brooks 

Q1. As requested during the hearing, please provide the total amount of federal 

funding awarded to those proposals rated from ‘‘Poor’’ to ‘‘Good or Very Good’’ for 

FY 10. While you touched on it briefly at the hearing, please also expand on why 

those proposals received funding over proposals rated ‘‘Very Good to Excellent’’ and 

‘‘Excellent.’’ 

A1. All funded proposals are determined to be highly meritorious based on a combination of reviews by individuals, panel deliberations and program officer evaluation. On average, NSF proposals are reviewed by four to six individuals, depending 

on the type of review. All reviewers are chosen for their specific expertise related 

to the subject, and the collection of persons brings different points of view to the 

decision-making process. When the average reviewer score is in the ‘‘good’’ range, 

it often represents a split of ‘‘excellent’’ or ‘‘very good’’ reviews with some ‘‘fair’’ or 

‘‘poor’’ review scores that lowered the average. It is important to note that the proposal rating data included in the annual NSB Merit Review Report reflects proposal 

ratings before panel deliberations and, therefore, not the final panel evaluation. The 

panel evaluation is based on a thorough discussion of the proposal’s strengths and 

weaknesses in the context of the full set of proposals being reviewed; this discussion 

forms the basis for placing a proposal in a particular category. These in-depth discussions can often clarify perceived weaknesses and result in a proposal being recommended for funding despite the initial average review score. Likewise, some proposals with high average review scores are not recommended by panels as a result 

of a detailed discussion that uncovers weaknesses that might not have been reflected in the initial reviews. 

The expertise of the NSF Program Officer making the final recommendation is 

also an important voice in the process. Program Officers take into consideration 

other factors that might not have been considered by expert reviewers. For example, 

proposals for innovative new ideas often use unproven methods or techniques that 

might be considered risky by reviewers and panelists. Risky proposals often result 

in transformative research that accelerates the pace of discovery. Although Program 

Officers consider concerns about risk expressed by panels, they also see the value 

of funding potentially transformative research. Proposals that do not review well at 

panel because the methods are unproven or risky can be given small awards to 

allow enough work for a ‘‘proof of concept.’’ Program Officers will also consider 

broader impacts that might not be obvious to reviewers, such as an infrastructure 

need that will serve a large number of people. There are also many dimensions of 

portfolio balance that influence the final recommendation. In addition to maintaining a diverse scientific portfolio, Program Officers strive to fund proposals from diverse institution types across the U.S., and from both young and experienced investigators. 

As explained above, the reviewer rating data reported in the Merit Review Report 

are only initial reviewer ratings, which is just the beginning of the merit review 

process leading to a final dertermination of whether any given proposal should be 

funded. Initial reviewer ratings do not reflect panel deliberations or Program Officer 

input. In FY 2010, NSF funded approximately $46K in proposals initially rated as 

poor, $21M in proposals initially rated as fair, $818M in proposals initially rated 

as good, and $1.6B in proposals initially rated as very good. Following panel discussion and analysis, all of these proposals were determined to be highly meritorious 

nothwithstanding their initial rankings. 

Q2. In your testimony, you described experimenting with innovative approaches to 

identify potentially transformative research. Please expand on the ‘‘ideas factory 

sandpit’’ approach and tell us what you are learning from it and other novel approaches. 

A2. NSF has experimented with an approach to identifying potentially transformative high-risk research that it is now calling ‘‘Ideas Lab.’’ The Ideas Lab is 

closely modeled on the ‘‘sandpit’’ process developed by the UK’s Engineering and 

Physical Sciences Research Council (EPSRC). The essential element of the Ideas 

Lab is an intensive interactive residential workshop involving 20–30 participants, 

with the aim of developing bold, often risky, new approaches to grand challenge 

questions in areas that could benefit from creative ‘‘out-of-the-box’’ thinking. A fun-
59 

damental aspect of the EPSRC sandpit that has been incorporated into the Ideas 

Lab is the use of a highly multidisciplinary mix of participants (including active researchers from diverse fields and potential users of research outcomes) to address 

specific research challenges. A description of the process used follows. Slight variations should be anticipated as NSF gains experience with the process and as it is 

adapted to different topics. 

To identify potential participants, a solicitation is issued that includes an open 

call for participants. Interested individuals submit short preliminary proposals that 

include concise descriptions of their pertinent experience and expertise as well as 

their communication skills, collaborative activities, and creative abilities. A panel of 

reviewers evaluates the applications and identifies a pool of potential participants 

from a range of disciplines and backgrounds who have high potential to work at the 

interface between disciplines and to develop new and highly original research ideas. 

NSF Program Officers make the final selection from the pool to ensure a diverse 

mix of participants. Industrial psychologists provide advice that guides but does not 

decide participant selection. 

During the multi-day Ideas Lab workshop, participants interact in unconventional 

new ways to develop innovative research project ideas on the selected topic area. 

Professional facilitators, experienced in sandpit-like activities, integrate creative 

problem-solving techniques, iterative project-development activities, and real-time 

peer review by both participants and a resident panel of experts (called the mentor 

group) to advance the most innovative ideas. Outcomes at the end of the workshop 

are research project concepts that vary in scale and scope in addressing the grand 

challenge topic of the Ideas Lab. At the end of the Ideas Lab, the panel of reviewers 

provides a consensus report summarizing its evaluation of each project concept. 

Based on this review, the Program Officers invite the submission of full proposals 

for some, none, or all of the project concepts. The invited groups have six to eight 

weeks to submit full proposals, which are then reviewed by the same panel of mentors using NSF’s two merit review criteria. Based on that review, NSF then makes 

a decision whether to fund some or all of the proposals. Taking part in the Ideas 

Lab does not mean that the participant is guaranteed to be funded under an award 

resulting from the Ideas Lab process. 

Experimentation with the Ideas Lab is still at an early stage. A total of four Ideas 

Labs have been conducted to date. The first three resulted in 12 awards and the 

fourth is currently awaiting full proposal submissions. Feedback from participants 

in the Ideas Labs has been positive. However, the resulting funded projects are still 

in their beginning phases. As they progress, NSF will look at the outcomes of these 

projects to evaluate whether they resulted in transformative research. 

Q3. Researchers will send in proposals whenever they have an idea that they would 

like to have funded. However, NSF also puts out solicitations for specific areas of 

research. Please explain how decisions are made on what types of research areas warrant a specific solicitation from the Foundation? What happens if the Foundation 

does not receive high quality proposals for a solicitation? Do you pick from what you 

have or do you rework the solicitation? 

A3. Solicitations are formal NSF publications that encourage the submission of 

proposals in specific program areas of interest to NSF. Solicitations are generally 

more focused than program announcements, and normally apply for a limited period 

of time. Ideas for new solicitations can be initiated within Divisions by Division Directors, at the Directorate level by Assistant Directors, or by groups of Assistant Directors who see the need for a new cross-cutting activity. Program Directors also 

commonly suggest ideas for new initiatives. The initial spark for a new initiative 

often comes from interaction with the scientific community through scientific meetings or other communications. When an idea for a new solicitation is suggested, a 

working group is formed that includes Program Officers who are expert in the research area. The working group collaborates on a detailed plan for the new solicitation, which is then discussed and reviewed by various levels of leadership before approval. 

Specific factors that are considered when deciding whether to develop a new solicitation include the following: 

• the intellectual reason for the Program, activity, or initiative; 

• whether the new activity(ies) will generate sufficient interest in the targeted 

community; 

• whether the Program, activity, or initiative is new, how it supports the longrange goals of the Directorate and/or NSF; 

• whether the size of the effort justifies a separate announcement and/or competition; 
60 

• the total funding available for the proposal competition, including estimated 

proposal receipts and anticipated number of awards and funding levels; 

• cross-Directorate participation (and implications) in the Program. 

Program Solicitations often specify submission limits, award conditions or reporting requirements, and provide supplemental proposal preparation guidance in addition to what is in the Grant Proposal Guide. Program solicitations also provide specific review criteria when reviewing proposals. In cases where the Foundation does 

not receive high quality proposals for a solicitation, we decline the proposals that 

are not of high quality. The solicitation could be revised and recompeted to attract 

high quality proposals. Because NSF funding opportunities generally attract more 

high quality proposals that we can fund, this would be a rare occurrence. 

Q4. After reviewing the flow chart for the proposal and award process and timeline, 

the Directorate Assistant Director seems conspicuously absent from the process. 

Please describe what the role and responsibilities of the Assistant Director are in the 

funding process, both from a programmatic and overall agency funding priorities 

perspective. 

A4. While Assistant Directors (ADs) are not involved in the day-to-day review and 

processing of proposals submitted to the Foundation, as described in the NSF Grant 

Proposal Guide Exhibit III 091 (http://www.nsf.gov/pubs/policydocs/pappguide/ 

nsf1100/gpg¥3ex1.pdf), they fulfill a vital role in the overall funding process. (See 

the attached referenced flowchart.) 

ADs are knowledgeable about the award portfolios in their directorates, but they 

are not involved in the decision-making process itself, because their role is to set 

the vision and strategic goals and objectives for the divisions/offices that report to 

them. 

Assistant Directors also play an important role in the formal reconsideration process. If a PI is dissatisfied with the explanation they receive for why a proposal has 

been declined, he/she may request a reconsideration of the decision. ADs/Office 

Heads are responsible for responding to these requests, and review the proposal 

record to determine whether NSF’s review of a declined proposal was fair and reasonable, substantively and procedurally. If they were involved in the decision-making process, the would have a conflict of interest in responding to any official reconsideration request. 

Q5. How does the Foundation leadership ensure that Program Officers ‘‘produce and 

manage a balanced portfolio of awards that address a variety of considerations and 

objectives’’ as the FY 10 NSB Report states? 

A5. Portfolio balance is reviewed at a variety of levels at different times during 

the decision-making process. Program Officers consider many dimensions of portfolio 

balance when they are making decisions about what proposals should be recommended for awards. Some of the factors that are considered include: balance 

across disciplines and subdisciplines, award size and duration, awards to new investigators, geographical distribution of awards, awards to different types of institutions, innovative/potentially transformative projects, projects with elements of risk, 

inter- and multidisciplinary projects, projects that integrate research and education, 

and projects that are relevant to agency mission or national priorities. Division Directors review the recommendations by Program Officers for portfolio balance before 

they concur with the award recommendations. Portfolio balance is also reviewed by 

our Committees of Visitors who review programs at three-year intervals. Some programs also contract for external evaluations of their portfolio periodically to inform 

how they might make changes to their programs. The results of both COV reports 

and external portfolio analyses are reveiwed by Directorate Advisory Comittees. 

Q6. According to the FY 10 Board Report, NSF awarded approximately five percent 

of its annual budget to federal agencies and laboratories. What kinds of awards were 

these and did they go through the formal merit review process? 

A6. The 2010 Merit Review Report to the National Science Board reported that 

NSF funded $351.2M in awards to Federally Funded Research and Development 

Centers (FFRDCs). The majority of this funding went to two organizations that 

build and manage large astronomy facilities for university consortia: Associated Universities Inc. (AUI) and the Association of Universities for Research in Astronomy 

(AURA). AUI received $111M in funding associated with the National Radio Astronomy Observatory, the Atacama Large Milimeter Array (ALMA), and other related 

projects. The Association for Universities for Research in Astronomy (AURA) received $234.3M for a number of projects including building the Advanced Technology Solar Telescope and the Advanced Technology Solar Telescope and operations 
61 

and management of the Gemini Observatory, the National Optical Astronomy Observatory, and the National Solar Observatory. In addition, $0.4M in funding went 

to fund several much smaller projects through another FFRDC, Aerospace Corporation. Proposals submitted to NSF by FFRDCs go through the same merit review 

process as other proposals. The large awards for building and operating large facilities go through a very lengthy and detailed review process that includes site visits, 

cost reviews, design reviews, and approval by the National Science Board. 

The $351.2M reported in the Merit Review Report also includes $5M in contracts 

to fund the Science and Technology Policy Institute (STP) operated by the Institute 

for Defense Analyses. STPI provides rigorous and objective analysis of science and 

technology (S&T) policy issues for the White House Office of Science and Technology 

Policy (OSTP) and other offices and councils within the Executive Branch of the 

U.S. Government and federal agencies. IDA was selected to operate STPI in 2003 

following a competition and undergoes reviews at five-year intervals. 

Note that the funding to FFRDCs described in the FY 2010 Merit Review Report 

did not include contract funds to or from other federal agencies through interagency 

agreements. 

Q7. What kind of peer reviewers are coming from industry, non-profits, and government? Do they all have Ph.D.s? What role does a panelist from the government play? 

What qualifications do they have? 

A7. Reviewers are chosen for their expertise in areas covered by the proposals that 

they are asked to review. For research proposals, reviewers are typically researchers 

in domains of science relevant to the topic of the proposal. In the review of proposals 

for facilities, in addition to the reviewers who can provide input on the research impacts, technical feasibility and soundness of the faciltity’s design, Program Officers 

may also include reviewers with expertise in other relevant fields such as project 

management, systems engineering, complex acquisition processes, architectural design, etc. 

In many scientific and engineering disciplines, some of the leading researchers 

work in industry, non-profits, government laboratories and FFRDCs. Examples include computer science and a number of others. Such researchers are very much the 

peers of their academic counterparts and are included in NSF’s pool of peer reviewers. Some of these researchers may have spent part of their research careers in academia and some in industry or an FFRDC, allowing them to bring important perspectives on the state of the art in the different environments and their potential 

broader impacts. These reviewers tend to have the qualifications that are typical for 

the research communities to which they belong. In many disciplines, this is often 

a Ph.D., although occasionally it is simply long experience doing cutting-edge research. Typically, what signals the expertise of a researcher is his or her record of 

research achievement, including significant publication in peer-reviewed journals 

and conference proceedings. 

Reviewers from government and industry are often more familiar with project 

management and complex acquisition processes than some of their academic counterparts, and so such individuals are sometimes asked to bring this expertise to review teams looking at proposals for research facilities. Such individuals may or may 

not have Ph.D.s. 

In general, what NSF looks for in its choices of reviewers is expertise in the topics 

under review. 

Q8. How does the Foundation train reviewers to prevent the phenomenon of implicit 

bias? 

A8. The frontline of the merit review process are the approximately 520 NSF Program Officers (POs) who select experts who can provide the information needed to 

make a recommendation in accordance with the National Science Board (NSB) approved criteria for selection of projects. Program Officers are trained on conflicts of 

interest, the importance of getting a diversity of perspectives, and guarding against 

the influence of subjective or biased input. 

Proposals submitted to NSF receive rigorous and objective treatment and POs ensure that this takes place. Proposals are evaluated by independent reviewers consisting of scientists, engineers, and educators who do not work at NSF or for the 

institution that employs the proposing researchers. NSF selects the reviewers from 

among a pool of experts in each field, and their evaluations are anonymous. On average, about 50,000 experts give their time to serve on review panels each year. POs 

ensure that there is diverse representation within the review group. The goal is to 

achieve a balance among various characteristics, including type of organization represented, reviewer diversity, age distribution and geographic balance. 
62 

The reviewer’s job is to provide advice to NSF on which projects are the highest 

priorities. This competitive process ensures that many voices are heard and that 

only the best projects make it to the funding stage. When someone is asked to review a proposal (either as an ad hoc or panel reviewer), they are provided with information on the confidentiality of the process and the potential for conflicts of interest. Panelists sign a ‘‘Conflict-of-interests and Confidentiality Statement’’ whenever they participate in a panel. For ad hoc reviewers, by submitting their review, 

they are acknowledging that they’ve been informed of such policies. Again, NSF POs 

are responsible for assuring that appropriate, qualified merit reviewers are selected 

and the entire process is overseen by Section Heads and/or Division Directors who 

supervise the Program Officers. 

Q9. The 2010 reauthorization of the America COMPETES Act required the Foundation to ‘‘Apply a Broader Impacts Review Criterion to achieve’’ various goals. Witnesses at the hearing raised some concerns with the draft criteria that is currently 

being weighed by the Board. Have the goals, now specified in statute, been considered 

in the past when making funding decisions? Are the peer reviewers taking these goals 

into consideration during their review or are the Program Officers simply tasked 

with this responsibility? Based on the work being conducted by NSB and NSF and 

your experience with the merit review process, is the legislative requirement 

achiveable and is it necessary? 

A9. NSF strives to invest in a robust and diverse portfolio of projects that creates 

new knowledge and enables braekthroughs in understanding across all areas of 

science and engineering research and education. To identify which projects to support, NSF relies on a a merit review process that incorporates consideration of both 

the technical merits of a proposed project and its potential to contribute more broadly to advancing NSF’s mission ‘‘to promote the progress of science; to advance the 

national health, prosperity, and welfare; to secure the national defense; and for 

other purposes.’’ In 1997, these considerations were put into action through the two 

primary merit review criteria of Intellectual Merit and Broader Impacts. Each reviewer must consider, and address, both merit review criteria for each proposal. 

As noted in your question, the importance of incorporating consideration of potential broader impacts in deciding which projects to fund was re-emphasized in the 

America COMPETES Reauthorization Act of 2010. Having the reinforcement of 

Congressional support on the fundamental nature of the Foundation’s Organic Act 

is always an important, and appreciated, development. However, there is a danger 

of viewing the Broader Impacts criterion as a ‘‘one size fits all’’ checklist, which 

would be a mistake. 

This COMPETES Reauthorization identified a number of societally relevant outcomes that may result as a consequence of NSF-funded research. Stated more 

broadly, these outcomes include (but are not limited to) increased participation of 

women, persons with disabilities, and underrepresented minorities in STEM; improved STEM education at all levels; increased public scientific literacy and public 

engagement with science and technology; improved well being of individuals in society; development of a globally competitive STEM workforce; increased partnerships 

between academia, industry, and others; increased national security; increased economic competitiveness of the United States; and enhanced infrastructure for research and education. These represent examples of societally relevant outcomes. The 

NSF will strive to clarify that these examples should not be considered either comprehensive or prescriptive, and that investigators may include appropriate outcomes 

not covered by these examples. 
63 
64 

Responses by Dr. Keith Yamamoto, 

Vice Chancellor for Research, University of California San Francisco 

Questions submitted by Chairman Mo Brooks 

Thank you for these insightful questions, Chairman Brooks. 

Q1. In basic science research, when one cannot possibly know what the outcome may 

be, how can one establish what the broader impacts will be? 

A1. Basic science research, by definition, is untargeted, driven only by the curiosity, intuition, background and expertise of the investigator(s) who define a question and develop a plan to answer it. For such investigations, it is impossible to ascribe broader impacts at the time of merit review. Only after the work has been 

completed, often long after its completion, can the wisdom of retrospection be applied to define broader impacts. Thus, inclusion of the broader impacts criterion in 

the merit review of grant applications is misguided and damaging to the integrity 

of the merit review process. 

Q2. Having sat on review panels, please tell me how heavily the broader impacts of 

a proposal are weighed when rating the proposals you were considering? Did the 

evaluation of broader impacts have any effect on the evaluation of the intellectual 

merit of the proposals? 

A2. My experience on NSF review panels predated the 1998 development of the 

broader impacts criterion, so I cannot provide experience-based perspective here. 

However, as indicated in my testimony, any departure from singular focus on scientific merit, as mandated by this assessment of broader impacts, erodes the evaluation of the intellectual merit of the proposals. 

In this regard, I wish to comment in particular on the point raised by Ms. Jackson 

that the broader impact criterion provides a metric to distinguish proposals that are 

equally meritorious on scientific grounds. She viewed this metric as ‘‘an important 

role for the broader impacts criteria and should be kept as such because there are 

clearly far more proposals that are excellent than can be funded.’’ With due respect, 

my opinion is sharply different. Application of this approach would discriminate 

against true basic science research as defined in my response to your first question 

above, and favor funding of work with a clear application. To the extent that this 

evaluation metric would become known in the scientific community, this approach 

would lead to a decline in basic science proposals. Fundamental discovery remains 

absolutely essential for scientific progress, and would suffer significantly from negative discrimination applied at the merit review stage. 

In NIH peer review, as noted by Dr. Marrett, this conflict is avoided by placing 

broader impact considerations in a separate review process carried out by separate 

review committees, thus maintaining the singular focus of merit review. 
65 

Responses by Dr. Nancy B. Jackson, 

President, American Chemical Society 

Questions submitted by Chairman Mo Brooks 

Q1. In basic science research, when one cannot possibly know what the outcome may 

be, how can one establish what the broader impacts will be? 

A1. At the National Science Foundation (NSF), all proposals are evaluated for intellectual merit and broader impacts. Two major determinate factors in estimating 

the long-term broad impacts of scientific proposals are the amount of resources 

available to fund grants and the standards of the Request for Proposal (RFP). 

NSF receives many more meritorious proposals than it could ever fund. While a 

proposal with weak intellectual merit has no hope of getting NSF funding, many 

proposals are rated ‘‘excellent’’ with strong intellectual merit and still do not get 

funded because of the stiff competition. The broader impacts criteria allow decision 

makers to gauge which research is the most urgent or has the greatest relevance 

to improving the quality of life. This merit review process enables NSF to ensure 

that precious R&D money goes only to the most relevant R&D needs. 

As anyone with a retirement fund has been told, managing a portfolio is critical 

to its long-term strength. Financial advisors stress that it’s important to find the 

right balance between solid performing stocks and riskier investments that may provide higher returns. Managing a research portfolio is similar. 

The research manager, whether working in industry or at a government agency, 

strives to find the right balance between science that will deliver steady advances 

and ideas that are out of the box, but could result in game-changing developments. 

This point is especially important in times of restricted funding. It’s human nature 

to make more conservative choices and be risk-aversive when times are tough. Expanding resources allows grant reviewers to ‘‘take a chance’’ on promising but less 

established scientific ideas that may result in a major scientific leap when compared 

to a safer but more conservative proposal. 

For every 100 grant proposals a program officer reviews, a small number (perhaps 

10) will be of such high quality that it is obvious they should be funded. Another 

50 proposals will be recognized as clearly not competitive; however, they still must 

be addressed through the process. An agonizing choice must then be made over the 

40 proposals in the middle. These include proposals that may be considered excellent or very good. In fiscal year 2010, the NSF award rate was 23 percent. In our 

analogy of 100 proposals, this would mean that 13 out of the remaining 40 would 

be funded. 

Unfortunately, it has proven difficult to pinpoint from where the next gamechanging scientific moment will occur. In 1994, when NSF funded the Stanford Integrated Digital Library Project, the Web was a tiny portion of the Internet, which 

was dominated by file sharing and gopher sites. This innovative grant to fund research into developing methods for searching Internet databases eventually lead to 

the creation of a $200 billion dollar company (Google, Inc.), which revolutionized almost every aspect of the way humans interact with each other and information. 

However, it would have been difficult to pinpoint that particular project as the most 

promising grant NSF would fund in 1994. 

Similarly, Dr. Robert Grubbs won the 2005 Nobel Prize in chemistry for his work 

in olefin metathesis. This complex organic reaction essentially allows molecules to 

swap components and is broadly used today in the pharmaceutical industry. The 

basic understanding of olefin metathesis dates back to the 1950s; however, at that 

time it was deemed expensive and dangerous for industrial use. For four decades 

corporate and academic researchers labored to refine and improve the reaction, 

which did not become industrially productive until the late 1990s. NSF was a major 

contributor to Dr. Grubbs’ work, which is revolutionizing the way drug companies 

improve the environmental footprint of chemical reactions. Without NSF’s long-term 

dedication to the rigor and quality of Dr. Grubbs’ (along with many others) work, 

safe and economically viable olefin metathesis may not have occurred. 

These two examples (one based on a single grant that revolutionized the information age, the other, the result of four decades of research by multiple teams of researchers) show the need for sustained, predictable research funding. Ensuring a 

steady stream of research funds would allow NSF to fund not only the most rigorous 

and reliable research, but also take risks in cutting-edge ideas based on promising 

data. By doing so, the government would be encouraging the broadest possible impacts of taxpayers’ precious dollars. 

One of the reasons why the merit review process is so successful is because it 

draws from the collective wisdom of the scientific community. Many NSF personnel 
66 

come directly from the scientific community and return to their research institutions 

at the end of their two- or three-year rotations. Relying on rotating directors means 

the managers are up to date on the most recent scientific developments. The panels 

that perform the peer review of proposals are fellow researchers in the field, and 

as such, are also up to speed on the latest developments. This scientific community 

service, whether performed by grant proposal reviewers or NSF program officers, is 

an integral part of scientific culture. Many scientists dedicate their time in this way 

because it provides an opportunity to remain in touch with and influence the cutting 

edge, as well as because they understand that the system only works if everyone 

volunteers to play their part. In a way, it is the science community’s way of ‘‘paying 

it forward.’’ 

The broader impacts criteria include considerations about whether the research 

proposal would broaden underrepresented minorities’ participation in science, 

strengthen U.S. infrastructure, improve national security, or foster innovation. Including some of these impacts is the result of language in the America COMPETES 

bill enacted last year. The broader impacts criteria take into consideration which 

research is the most urgent or has the greatest relevance to improving the quality 

of life. The broader impacts criteria enables NSF to choose between meritorious and 

even more meritorious proposals and is a way to ensure that precious R&D money 

goes to the most pressing R&D needs. 
67 

Responses by Dr. Jorge Jose ́, 

Vice President for Research, Indiana University 

Questions submitted by Chairman Mo Brooks 

Q1. In basic science research, when one cannot possibly know what the outcome may 

be, how can one establish what the broader impacts will be? 

A1. Thank you, Chairman Brooks, for the question. As you correctly note, in basic 

science research, it is not possible to know in advance what the outcome of research 

will be. We pose a question for which we would like to find an answer, and we lay 

out a research plan that in principle should lead us to an answer. The research plan 

is based on prior knowledge and an educated hypothesis of the expected results. A 

researcher may not know for certain that a particular plan will yield the results 

they hope for, but they do know aspects of a plan that is likely to do so. Reviewers, 

in fact, will base their review on whether the hypothesis and the research plan seem 

to be an appropriate way to ascertain the ansers to the question being posed. In 

the course of an investigation, other questions or an unexpected discovery may arise 

that may lead the researcher to significantly change the direction of the research. 

My sense is that a similar awareness of cause and effect can shape our expectations for what the broader impacts of any given research project might be. Take, 

for example, the broader impact goal of expanding the participation of underrepresented minorities in science, technology, engineering, and mathematics (STEM). 

I cannot know for certain that a research project will have that result, but I can 

know some aspects of a research plan that are likely to do so. If my research plan, 

for instance, includes some form of collaboration with colleagues from an HBCU, 

then it is more likely to expand the participation of underrepresented minorities 

than if it does not include that collaboration. Of course the collaboration is not a 

failsafe guarantee what the broader impact will be, but it increases the likelihood— 

just as certain elements of the scientific research plan will increase the likelihood 

of certain experimental outcomes. 

Q2. Having sat on review panels, please tell me how heavily the broader impacts of 

a proposal are weighed when rating the proposals you were considering. Did the evaluation of broader impacts have any effect on the evaluation of the intellectual merit 

of the proposals? 

A2. Thank you, Chairman Brooks, for that question. As Dr. Marrett noted, the 

NSF has many processes in place to guard against bias within review committees— 

to make sure, in other words, that as far as is possible, the process identifies the 

projects with the most significant intellectual merit. Ideally, questions of broader 

impact are used only to decide afterwards between proposals of equal intellectual 

merit. Certainly at Indiana University, when we are allocating resources, we make 

every effort to do the same thing—to remove conflicts of interest, to focus on the 

intellectual merit of competing proposals, and to leave ‘‘broader impact’’ and strategic considerations for later. 

Realistically, of course, it is impossible to guard against all variables that might 

prioritize some other aspect of a proposal over its intellectual merit. Indeed, one 

might worry that the expansion of broader impact goals as laid out in the America 

Competes Act increases that difficulty. But my experience is that more often than 

not, the intellectual merit of a proposal is the issue that is most determinative in 

funding recommendations, since consideration of broader impacts is generally a second-level concern. 

Æ