**A REVIEW OF THE INFORMAL SCIENCE EDUCATION (ISE) PROGRAM**

**by**

**Committee of Visitors (COV)**

**April 2-3, 2001**

Date of COV: April 2-3, 2001

Program: Informal Science Education (ISE)

Cluster, Division: Elementary, Secondary, and Informal Education (ESIE)

Directorate: Education and Human Resources (EHR)

Number of jackets reviewed: 30 for declined proposals, 42 awarded
proposals[^1]

**Table of Contents**

 

[Introduction](http://www.ehr.nsf.gov/gpra/cov/2001/ESIEResponse2001.htm#__RefHeading___Toc517852875)
2

[A. Integrity And Efficiency of the Program's Processes &
Management](http://www.ehr.nsf.gov/gpra/cov/2001/ESIEResponse2001.htm#__RefHeading___Toc517852876)
5

[1. Effectiveness of the program's use of merit review
procedures](http://www.ehr.nsf.gov/gpra/cov/2001/ESIEResponse2001.htm#__RefHeading___Toc517852877)
5

[2. The program's use of the NSF Merit Review
Criteria](http://www.ehr.nsf.gov/gpra/cov/2001/ESIEResponse2001.htm#__RefHeading___Toc517852878)
7

[3. Reviewer
selection](http://www.ehr.nsf.gov/gpra/cov/2001/ESIEResponse2001.htm#__RefHeading___Toc517852879)
9

[4. Resulting portfolio of
awards](http://www.ehr.nsf.gov/gpra/cov/2001/ESIEResponse2001.htm#__RefHeading___Toc517852880)
10

[B. Results: Outputs and Outcomes of NSF
Investments](http://www.ehr.nsf.gov/gpra/cov/2001/ESIEResponse2001.htm#__RefHeading___Toc517852881)
13

[5. PEOPLE Strategic Outcome
Goal](http://www.ehr.nsf.gov/gpra/cov/2001/ESIEResponse2001.htm#__RefHeading___Toc517852882)
13

[6. IDEAS Strategic Outcome
Goal](http://www.ehr.nsf.gov/gpra/cov/2001/ESIEResponse2001.htm#__RefHeading___Toc517852883)
15

[7. TOOLS Strategic Outcome
Goal](http://www.ehr.nsf.gov/gpra/cov/2001/ESIEResponse2001.htm#__RefHeading___Toc517852884)
16

[8. Areas of
Emphasis](http://www.ehr.nsf.gov/gpra/cov/2001/ESIEResponse2001.htm#__RefHeading___Toc517852885)
17

[9. Comments on program areas that the COV believes needs
improvement](http://www.ehr.nsf.gov/gpra/cov/2001/ESIEResponse2001.htm#__RefHeading___Toc517852886)
18

[10. Comments on the program's performance in meeting program-specific
goals and objectives (non-GPRA
outcomes)](http://www.ehr.nsf.gov/gpra/cov/2001/ESIEResponse2001.htm#__RefHeading___Toc517852887)
19

[11. Feedback on the COV review process, format and core
question](http://www.ehr.nsf.gov/gpra/cov/2001/ESIEResponse2001.htm#__RefHeading___Toc517852888)
19

[C. Supplementary Questions for Informal Science Education
COV](http://www.ehr.nsf.gov/gpra/cov/2001/ESIEResponse2001.htm#__RefHeading___Toc517852889)
20

# Introduction

#####  

##### Composition of the Committee of Visitors

Susan B. Millar, a member of the EHR Advisory Council, chaired the COV.
Other COV members were as follows:

[Subcommittee on Museums]{.underline}

> William Booth, COSI, (Subcommittee chair)
>
> James Peterson, Science Museum of Minnesota
>
> Gwendolyn K. Crider, Austin Children's Museum

[Subcommittee on Media]{.underline}

> Gerry Wheeler, National Science Teacher's Association, (Subcommittee
> Chair)
>
> Greg Andorfer, Maryland Science Center
>
> Bunny Lester, World Media Foundation

[Subcommittee on Youth & Community-based Groups]{.underline}

> Eric Jolly, Education Development Center (Subcommittee Chair)
>
> Ellen Wahl, American Museum of Natural History
>
> Alberto Ramirez, Frederick Community College

##### Materials Reviewed

The Informal Science Education (ISE) program staff groups its projects
into three categories by the focus of awards: museum awards, media
awards, and youth & community awards. They further group their program
activity by type of grant, including standard, continuing, planning, and
conference grants, and small grants for exploratory research (SGER).

In conducting its work, the COV considered a comprehensive set of
materials provided by the ISE program officers, including:

-   a detailed set of program management data compiled expressly for the
    COV presenting the award statistics by year, broken out for
    preliminary proposals, awards (showing investment by program area,
    PI demographics, and commitment by state), declinations and
    withdrawals, supplements, and increments, and statistics on proposal
    processing ("dwell") time

-   1998 ISE COV Report

-   Guidance for NSF Staff and COV Members, FY 2001

-   ESIE Strategic Plan, FY 1995 -- FY 2000

-   FY 1998, FY 1999, and FY 2000 Elementary, Secondary, and Informal
    Science (ESIE) Annual Reports

-   Recent ESIE Program Solicitation and Guidelines Booklets

-   The COSMOS Corporation 1998 "Report on the Evaluation of the NSF's
    Informal Science Education Program"

-   Examples of Program Contributions toward Relevant Government
    Performance Results Act (GPRA) Outcomes from the NSF Feb. 11-12,
    1999 Program Effectiveness Review of the ISE Program

-   Instructions for Panel Reviews on the use of Fastlane

-   Sample of 90 proposal jackets. The ISE staff selected the jackets at
    random from the full set of 1998, 1999 and 2000 jackets so as to
    result in 45 (15 for each subgroup, with 5 for each year) and 45
    declinations/withdrawals (15 for each subgroup, with 5 for each
    year), and so as to provide examples in each category of the five
    types of grants (standard, continuing, planning, and conference
    grants, and Small Grants for Exploratory Research).

-   A set of program summaries submitted to Fastlane, organized by
    subarea

-   A short report of the ISE Outreach meetings held during 2000-01.

**COV Meeting Process**

Members of the EHR staff, including Dr. Jane Stutsman from the Assistant
Director's Office, and Dr. Jane Kahle and Ms. Jean Vanski from the
Elementary, Secondary, and Informal Science (ESIE) Division Director's
Office, welcomed the COV and emphasized the importance of the
contributions that COVs make to the Directorate. Dr. Elmima Johnson,
from the Division of Research, Evaluation and Communication, reviewed
with the committee relevant features of the Government Performance and
Results Act (GPRA). Dr. Kahle emphasized how important it is to the
Division and the Directorate that COVs include numerous "nuggets," that
is, project activities and outcomes that illustrate the points that the
committee wishes to make. Dr. Johnson noted that, as have previous COVs,
ours is asked to review of the effectiveness and efficiency of the ISE
program's processes and management by considering the following four
factors:

> •        use of merit review procedures
>
> •        use of the NSF Merit Review Criteria (intellectual merit and
> broader impacts) by both proposal reviewers and program officers
>
> •        reviewer selection
>
> •        resulting portfolio of awards.

Dr. Johnson then gave special attention to recent changes to the GPRA
requirements and to the subset of performance goals articulated in the
NSF 2001 Performance Plan that relate to the ISE program. In particular,
she noted that ESIE and the Informal Science Education program officers
are asking the COV to determine the success of the ISE program by
assessing whether, in the aggregate, results reported in the 1998 --
2000 period demonstrate significant achievement in one or more of the
following indicators:

With respect to the "People" outcome:

> •        Improved mathematics, science, and technology skills for U.S.
> students at the K-12 level and for citizens of all ages, so that they
> can be competitive in a technological society
>
> •        A public that is provided access to the benefits of science
> and engineering research and education

With respect to the "Ideas" outcome:

> •     Partnerships connecting discovery to innovation, learning, and
> societal advancement
>
> •           Research and education processes that are synergistic

With respect to the "Tools" outcome:

> •           Networking and connectivity that take full advantage of
> the Internet and make science, mathematics, engineering and technology
> information available to all citizens.

Mr. Barry Van Deman, the ESIE Section Head in charge of the ISE program,
reviewed with the committee the primary goals of their program, which
are to:

> •        increase public understanding of science, mathematics, and
> technology;
>
> •        increase the number of youth, particularly underrepresented
> (e.g., minorities, girls, the physically disabled) and underserved
> (e.g., rural communities), who are excited about science, mathematics
> and technology (SMT) and who pursue such activities both in and out of
> school;
>
> •        bring informal science education programs and activities to
> relatively large areas currently without, or minimally reached by,
> such opportunities, e.g., rural areas and inner-city environments; and
>
> •        enrich the quality of life by improving the scientific and
> technological literacy of children and adults so they are informed
> about the implications of SMT in their everyday lives; are motivated
> to pursue further experiences in these areas; and are aided in making
> informed, responsible decisions about related policy issues having
> societal implications.

He also noted that, characteristically, ISE projects are designed to:

> •        provide rich and stimulating opportunities outside formal
> classroom settings whereby individuals of all ages, interests, and
> backgrounds can increase their appreciation and understanding of
> science, mathematics, and their applications;
>
> •        take place in diverse environments, such as, museums, zoos,
> botanic gardens, community organizations, and involve the use of
> various media, such as broadcast, film, exhibits, interactive
> technology, and print;
>
> •        promote linkages between informal and formal education
> resulting in improved and creative SMT education in all learning
> environments;
>
> •        stimulate parents and other adults to become effective
> proponents for better quality and more universally available SMT
> education in formal and informal settings;
>
> •        either serve large audiences or to have significant impact at
> national or regional levels; and
>
> •        promote collaborations among organizations that have similar
> goals, especially when projects bridge informal and formal education
> communities.

Barry Van Deman also noted that the program developed five Supplementary
Questions for Informal Science Education for consideration by the COV.
Ms. Sylvia James, an ESIE program officer, provided additional
clarification about the types of information provided to the COV.

In preparing to break into subgroup meetings, the COV decided that each
subgroup would attempt to review all 30 of the jackets selected for
their review. The COV reasoned that by taking this approach, rather than
reviewing a smaller number of jackets more intensively, we would be
better able to address the supplementary questions pertaining to optimal
ways to design the ISE portfolio in the future.

During its time together, the ISE program staff responded rapidly and
effectively to all COV requests for additional material, and listened
responsively to the informal feedback session that we presented to both
ISE and other ESIE program officers on April 3, 2001.

**Note to readers**: Text below in italics is from the Report Template.

# A. Integrity And Efficiency of the Program's Processes & Management

>  

This section of the report is devoted to the committee's assessment of,
and comments about, factors that the NSF deems important for maintaining
the integrity and efficiency of the ISE program's processes and
management.

 

 

## 1. Effectiveness of the program's use of merit review procedures

 

***a. Overall design***

Overall, the ISE program merit review design is very strong. From the
jackets reviewed, it appears that documentation is complete, and that
reviewers' comments are thorough, thoughtful, and intended to help the
PI successfully implement the project or, in the case of
declinations/withdrawals, develop a stronger resubmission. Comments by
program officers were equally constructive and thoughtful. The
effectiveness of the ISE program's review process should serve as a
model for other merit review processes.

 

The program officer role is clearly central to the design process. The
evidence we gleaned from the jackets showed an excellent performance by
the program officers.

 

***b. Effectiveness of the program review process***

Overall, the ISE program's review process is exceptional, an outcome we
attribute to the program officers' ability to transmit review panel
concerns into negotiation dialogues in ways that align the project more
closely with NSF's ISE goals and, thus, maximize NSF's investment. It
appears that the funded projects are consistently of high quality,
respond to the program's requests for proposals (RFPs) and thus are
consistent with its goals, and share a set of defining characteristics
that were identified during the review process. In cases where panel
reviews were mixed, program officers actively gathered clarifying
information and supplemental reviews. In the matter of declinations, the
outcomes of the review process also seemed very consistent, generally
well-documented, and appropriate.

 

We recommend that the important role of program officer in effectively
transmitting review panel concerns to proposers during negotiation
dialogues be given special attention during new program officer
orientation.

 

Considered by type of proposal, the following differences in the review
process were noted by the youth & community subcommittee.

> [Full panels]{.underline}. When full panel reviews are organized for a
> solicitation, the process appeared to be thorough and well-documented.
> Review panels included a range of perspectives and expertise. Panels
> seem to be appropriately guided by input from the RFP, the program
> officer, and NSF's review criteria. The panel summaries written by
> program officers were uniformly excellent distillations of the writing
> of the panels. Program officer correspondence and negotiations with
> PIs were similarly well-documented and an asset in supporting project
> development. However, with notable exceptions, there was little
> evidence of correspondence and oversight (for example, through program
> officer site visits) during the award follow-up period. We found
> exceptions, for example, where the program officer added a stipulation
> to the funding or made site visits. In these cases, the follow-up was
> excellent and well-documented.
>
>  
>
> [Preliminary proposals]{.underline}. Preliminary proposals were
> reviewed internally by two or more program officers, and were managed
> efficiently and in a way that provided clear communication to the PIs.
>
>  
>
> [SGER and conference proposals]{.underline}*.* The review process for
> SGER and Conference grants is not as strong as for the preliminary
> proposal reviews. Consistent with NSF rules (cf. Proposal and Award
> Manual, April 2001, Chapter VIII, Sec. E, Part. 1), these proposals
> were reviewed by only one program officer. However, the jackets we
> reviewed were so concise as to suggest that not all the communication
> between the proposer and program officer was documented. While
> recognizing the already high demands on their time that program
> officers face, we nonetheless urge the program to ask a second program
> officer to review SGER proposals. This second review would strengthen
> the credibility of recommendations for what appear to be worthwhile
> submissions.

 

***c. Efficiency, time to decision***

The COV examined dates of submission, dates of review, dates of
communication and negotiation, and date of program officer analysis. In
general, there appeared to be a trend over the three-year period toward
greater efficiency and a shorter duration in the time between panel
review and making an award. There were multiple examples of rapid,
efficient, and careful responses in which notification of declination
was made within two months of review, and this efficiency was the norm
rather than the exception.

 

While we commend the program on its generally successful effort to
shorten the time to decision, this process can be improved. The evidence
we reviewed indicates that some of the problem lies at the point after
ISE program officers submit their recommendations. The resulting lag
time to receiving formal notification is serious for many grantees,
creating burdens that, in fact, raise the costs of initiatives.

 

We also noted that the inadequate performance of a program officer no
longer with the division contributed to this problem. For example, three
of the 15 declinations that the youth and community subcommittee
reviewed had unacceptably long delays in receiving notification of
declination. There was also a long delay in the process for one proposal
that resulted in an award. A different program officer completed the
process for a \$50,000 planning grant in short order. To prevent
recurrence of this type of situation, the ISE program should present
clear efficiency standards in its new program officer orientation, and
establish a process of on-going oversight.

 

***d. Completeness of documentation making recommendations***

The ISE staff does a consistently excellent job of documentation. Most
jackets had exemplary documentation, and a few, such as ESI 98-15021
(University of Pittsburgh, *Responding to the Gender Gap in Informal
Science Education*, youth & community project), and ESI 98-04318 (Miami
University, *Dragonfly Quest*, youth & community project), were
particularly outstanding. Program officers provided informative notes
when there was an anomaly in process, such as a case in which the
incoming program officer had served on the review panel and another
program officer documented how the process would be handled. Some
program officers documented telephone contact, a process that we
recommend all follow.

>  

***e. Consistency with priorities and criteria stated in the program's
solicitations, announcements and guidelines***

We found the ISE program's solicitations, announcements, and guidelines
to be remarkably consistent, clearly stating the priorities and criteria
of the program. The merit review procedures were uniformly consistent
with program priorities and criteria. As important, all evidence of
verbal communication by the program officers appeared consistently
aligned with the ISE program criteria and goals.

 

Because of the critical role that program officers play in the merit
review process, the COV is concerned that an overloaded staff could
result in less time for the early negotiations with grantees and for
ongoing monitoring of active projects.

##  

 

## 2. The program's use of the NSF Merit Review Criteria 

 

***a. Performance goal: implementation of merit review criteria by
reviewers***

*1) Did reviewers adequately address the [intellectual
merit]{.underline} criterion in their reviews?*

The COV found that all reviewers gave careful attention to the
intellectual merit ("first") criterion.

> *2) Did reviewers adequately address the [broader impacts]{.underline}
> criterion in their reviews?*
>
>  
>
> Overall, the COV found little consistency across panels in how fully
> and effectively the broader impacts ("second") criterion was
> addressed. In some panels every panelist addressed the issue, while in
> others only one or none addressed it. This problem may be aggravated
> by two factors:
>
> 1)      Some reviewers' comments incorporated discussion of both
> review criteria without explicitly distinguishing the effectiveness
> with which each was addressed. For example, in one proposal, while the
> reviewers gave no explicit attention to the second criterion, they
> made reference to issues that we can infer to be related to broader
> impacts by inference.
>
> 2)      Some reviewers did not appear to consider the impact on
> underserved groups to be an important element of the way in which the
> NSF defines the second criterion.[^2]
>
>  
>
> The COV found that in about three-quarters of the jackets reviewed, at
> least one member of every panel addressed the criterion. We estimate
> that fewer than half of the panelists directly addressed this
> criterion in their reviews. However, there was some evidence that
> increased attention was given to the second criterion over the course
> of the reporting period. We believe that the second criterion is
> essential to the goals of the ISE, and are concerned about the
> inconsistency with which this criterion is addressed. The subcommittee
> on youth and community projects gave special attention to the way
> reviewers address the second criterion in the jackets they reviewed.
>
>  

***b. Performance goal: implementation of merit review criteria by
program officers***

*1) Did program officers adequately address the [intellectual
merit]{.underline} criterion in their reviews?*

The COV found that all program officers gave careful attention to the
intellectual merit ("first") criterion.

> *2) Did program officers adequately address the [broader
> impacts]{.underline} criterion in their reviews?*
>
>  

Program officers addressed both criteria in their reviews, often taking
pains to make explicit indirect or incomplete attention by panelists to
the second criterion.

###  

***c. Discuss any concerns the COV has with respect to NSF's merit
review system***

The information available for the COV review process did not make clear
how program officers introduced the merit review criteria to the panels.
This is of note because when reviewers addressed "broader impacts" it
was not clear whether they had in mind merely the size of the audience
served, or also the attention given to underrepresented groups or
underserved geographic areas.

 

In light of our findings, the COV recommends that instruction to
panelists more carefully explain the meaning of the second
criterion,[^3] more emphatically require that explicit attention be
given to each criterion, and clearly stipulate that reviewers should
assign high ranks only to proposals that they rank high on both
criteria. In particular, we recommend that the entire text of the second
criterion, as specified in Dr. Colwell's "Dear Colleague Letter" (NSF
99-172), be presented to review panelists at the outset of their
deliberations.

## 3. Reviewer selection

***a. Use of adequate number for balanced review***

We found that the numbers of review panelists were adequate to obtain
balanced reviews.

***b. Use of reviewers having appropriate expertise/qualifications***

We found a good balance among reviewers with respect to diversity of
content specialties and expertise in pedagogy and community outreach.
The COV felt that in some cases panel composition could have been
improved by more scientists, but caution that scientists should not be
added unless they also have expertise in the ISE field. Overall, the
quality and appropriateness of reviewer expertise and qualifications was
evident in the above-noted excellence of reviewer comments. For
instance, reviewers noted an aspect of a proposal that members of the
project's intended audience might have found culturally
offensive/inappropriate.

 

***c. Use of reviewers reflecting balance among characteristics such as
geography, type of institution, and underrepresented groups***

First, we want to commend the ISE staff for an excellent job of creating
balance on our COV panel. The balanced composition of the COV is
consistent with our previous experiences as reviewers for the ISE
program.

 

Our review of the jackets shows a good balance among reviewers with
respect to gender, type and size of institution (museum, university,
media outlet, community or youth organization), and geographic
distribution. Although we know from our own experiences as review
panelists that ISE panels include ethnically diverse members (but not
persons with disabilities), it was not possible to determine
ethic/racial, or disability \[or language background from information
contained in the jackets. The COV recognizes that there may be legal
considerations in how demographic data are collected and presented. We
therefore recommend that the staff request the panelists themselves to
submit a paragraph describing relevant points about their backgrounds
and expertise. This information would greatly support the panel process
by ensuring and documenting diversity and balance, and by further
facilitating the exchange of views that makes the panel process so
valuable and contributes to better projects.

>  

***d. As appropriate, recognition and resolution of conflicts of
interest by NSF staff and adequacy of documentation justifying actions
taken***

Based on the evidence provided in the jackets, it is difficult to assess
attention to conflict of interest monitoring. However, we know from
personal experiences on review panels that the staff is very attentive
to these concerns.

 

## 4. Resulting portfolio of awards

***a. Overall quality of science/engineering***

The quality of the science in the awarded projects was very high,
showing strict attention to the adequacy of science content. Those
proposals that did not meet these standards were either withdrawn or
declined.

***b. Appropriateness of award scope, size, and duration***

We judge the scope, size, and duration of the awards to be appropriate.
However, if summative data is to be more effective in informing the ISE
field, the duration of many ISE grants will need to be expanded. This
said, we note that there is a trend in the Foundation's RFPs toward
placing ever greater importance on summative data demonstrating the
outcomes achieved by each project, and that the duration of most
projects reviewed was too short to enable PIs to produce convincing
summative data now called for in the RFPs. Thus, if this trend in the
expectations stated in RFPs continues, we believe the duration of
individual grants should be increased.

***c. Effective identification of and support for emerging
opportunities***

If considered in terms of the recent efforts to support the development
of proposals from rural and small communities, support for emerging
opportunities is to be commended. And if considered in terms of a larger
timeline, certain emerging opportunities, such as innovative use of
electronic networks, are being supported by the ISE program. The COV
found the matter more complex, however, when considering support for
emerging opportunities in terms of a shorter timeline. We consider this
topic below in our discussion of achieving a portfolio balanced with
respect to high-risk and innovative projects.

***d. Appropriate attention to maintaining openness in the system, for
example, through the support of new investigators***

The COV's assessment of the program's success in maintaining openness in
its grant system varied by subcommittee. The museum subgroup found the
program generally successful in this regard, noting examples of openness
to emerging opportunities in grants proposing research on how learning
occurs in informal settings (e.g., ESI 99-06982, Association of Science
and Technology Centers, *Mathematics Exhibits in Science Museums,*
museum project, ESI 00-00527*,* Institute for Learning Innovation*, A
Multi-Factor Investigation of Variables Affecting Informal Science
Learning*, museum project, and ESI 93-55504*,* Philadelphia Informal
Science Education Collaborative, *Family Learning in Museums -- The
PISEC Perspective*, museum project), and in the newly-funded program for
smaller communities. The media subgroup found that the staff has
maintained an openness to new investigators through their use of
Planning Grants. They noted that, within the media area, the larger
national producers are more likely to have the facilities and
infrastructure to propose/mount large-scale media productions and for
this reason are more frequent media award recipients. The youth and
community subgroup, however, found too little evidence of new PIs in the
program portfolio, noting that the current distribution of grants is
heavily weighted toward organizations with a long history of success in
seeking NSF funding.

***e. Evidence that proposers have addressed the integration of research
and education in proposals***

In order to conduct our review in a manner that is appropriate for the
informal education field, the COV chose to define "research" in terms of
three elements: 1) the degree to which the proposals rest on a solid
research and evaluation base; 2) the degree to which new scientific
research is integrated into the content of programs and activities; and
3) the degree to which new knowledge about how people learn is used in
the methods and strategies for informal science education. We reviewed
the jackets in terms of each of these elements.

> •        Many, but not all, ISE projects successfully integrate
> research and education. The research and evidence base cited in some
> proposals was weak. While this may be due in part to the page limit
> requirement, the COV is concerned that these ISE projects are weak in
> their efforts to learn from the existing knowledge base and contribute
> to the emerging knowledge base. We address this point at more length
> below, as the problem may be related to the scarcity of outcome
> evaluation and a need to dedicate deliberate attention and more
> resources to the challenge of building a reliable research and
> evaluation foundation for the field as a whole.
>
> •        We noted limited evidence that new scientific research is
> being incorporated into the content of programs and activities.
>
> •        Although projects generally demonstrated a solid base of
> understanding of human development and developmental appropriateness,
> we noted limited attention to the recent research on how people learn.
> In stating this, we acknowledge that national interest in linking
> cognition and learning to program and educational strategies only
> emerged midway through the three-year period under review.

 

***f. Evidence of increased numbers of applications from
underrepresented groups***

The jackets provided did not allow the subcommittee to adequately assess
a change over time in the proportion of applicants from underrepresented
groups. We did note, however, that application rates from PIs are
overwhelmingly white, and primarily male. We found only one proposal
from a minority serving organization---a planning grant---that was
funded. The museum group noted, however, that virtually every project
included discussion of how underrepresented groups would be reached and
served.

#######  

***g. Balance of projects***

We address various features of a balanced portfolio in the bullets
below.

 

•        ***High-risk***. The COV notes that there are multiple
definitions of a "high-risk" grant. In traditional research, proposals
that effectively offer high-risk/high-gain ideas are funded because
positive results would add so significantly to our body of knowledge. In
the ISE field, however, a truly innovative project from a highly
experienced institution or PI might not be considered "high risk" (due
to the "track record"), whereas the same project proposed by a less
experienced entity might be. Finally, we note that the meaning of
high-risk changes with time: not so long ago, the whole ISE program was
considered a high-risk initiative.

>  
>
> In light of these definitions, the media subgroup found very little
> evidence that that the ISE program is funding high-risk work. This is,
> in part, understandable because, in contrast to traditional research
> fields, it is more difficult to identify "high-gain" in the informal
> science education field. The museum and the youth and community
> subgroup also found few examples of funded high-risk or innovative
> projects, and noted that the problem lies not in the inappropriate
> rejection of quality high-risk proposals, but in the fact that the
> applicant pool includes very few proposals from either high-risk
> applicants or known performers with high-risk ideas.

 

> The committee believes that, due to the nature of informal science
> education, proposals pursuing ideas that are outside of the bounds of
> traditional ISE activities should be considered cautiously, but that
> proposals from new PIs and institutions should be aggressively sought,
> as these represent an important means for extending the field and
> growing a community of experts capable of continuing this important
> work. In this regard, we wonder if the program is sending a mixed
> message to the field, thereby discouraging new applicants. To wit, the
> current process requires applicants to demonstrate that they already
> have the capacity (including implementation experience) to launch a
> project. Given this, prospective PIs who are new and inexperienced,
> (or for that matter, who are considering totally new projects) are
> likely to assess their likelihood of success too low to warrant the
> effort.
>
>  
>
> We recommend that the program continue to pursue strategies to
> encourage and cultivate greater participation from broader and more
> diverse potential applicants. The current efforts by the ISE program
> to encourage participation from smaller communities (commended under
> c, above) could increase the number of "high-risk" projects. To
> further this effort, we recommend that the program seek input from the
> field to better define innovation and high-risk in the context of
> informal education.

 

•        ***Multidisciplinary***. The subgroups differed in their
assessment of the degree to which the portfolio has a good balance with
respect to multidisciplinary activities. The media and the museum
subgroups were satisfied that this criterion is being met. However,
while the youth and community group found a number of projects that
combined science and literacy, they found few that investigated multiple
scientific disciplines, or focused on how the traditional boundaries
between disciplines are changing.

 

•        ***Innovative***. We interpreted "innovative" to mean a
tried-and-tested idea that is being attempted with a new or critical
population that has not been served before, or a new idea, strategy, or
model, and judged the portfolio to be balanced in this regard. For
example, ESI 98-15021 (The University of Pittsburgh, *Responding to the
Gender Gap in Informal Science Education,* youth & community program),
which examines the gender gap in informal science, takes an innovative
approach in its investigation of a persistent equity and participation
problem through the lens of parent-child interaction. It is also an
example of cross-project fertilization, as this problem was identified
in a project conducted by the San Jose Children's Museum and the Family
Science Learning Project (described in the ISE Performance Effectiveness
Review). (See also our comments on this matter in the bullet on
high-risk, above.)

 

 

# B. Results: Outputs and Outcomes of NSF Investments

## 5. PEOPLE Strategic Outcome Goal

*Development of a diverse, internationally-competitive and
globally-engaged workforce of scientists, engineers, and well-prepared
citizens.*

Because the purpose of informal science projects is to create or figure
out how to create a scientifically literate populace, this program is
explicitly designed to support this strategic goal.

*[FY 2001 Performance Goal:]{.underline} The program is
[successful]{.underline} when, in the aggregate, results reported in the
period demonstrate significant achievement in one or more of the
following indicators:*

***a. Improved mathematics, science, and technology skills for U.S.
students at the K-12 level***

The COV agreed that it is not appropriate to use this indicator of
success to assess the success of the ISE program. However, the majority
of the funded ISE projects entail strong ties to formal education in the
form of, for example, curriculum guides, teacher training activities and
materials, and ancillary materials that are aligned with the national
science standards.

***b. Improved mathematics, science, and technology skills for citizens
of all ages, so that they can be competitive in a technological
society***

The COV considers the ISE program **successful** in demonstrating
significant achievement of this indicator. *For example, ISE programs
(e.g., ESI 00-00589,* Montshire Museum of Science, *TEAMS Traveling
Exhibit Collaborative and ESI 97-30087,* Rutgers University, *Science
Education During Recreation in Wilderness Parks*) are improving SMT
skills for citizens of all ages by addressing how families learn in
informal settings and by involving parents as participants in children's
science and math education. Projects targeting early childhood learners
also demonstrate achievement of this indicator. *ZOOM* (ESI 98-14956,
WGBH Educational Foundation, media project) and *BUSYTOWN (*ESI
94-53790, Oregon Museum of Science and Industry, museum project) are
outstanding examples of early childhood projects funded by the program
that support achievement of this outcome goal. Other examples of ISE
projects that demonstrate achievement in improving math, science and
technology skills for citizens of all ages are:

> •        ESI 99-01923 (Soundprint Media Center, *Exploring Space
> Science,* media project) reports evaluation findings based on a focus
> group study that showed that:
>
> •        listeners related information from the program to their own
> life experiences;
>
> •        teachers found the series to be useful for both classroom and
> independent student use; and
>
> •        listeners report that an essential role of public
> broadcasting is to provide in-depth information on a variety of topics
> that is not available through other media.
>
> •        ESI 99-01975 (Lichtenstein Creative Media, *The Infinite Mind
> Radio Series,* media project), is currently broadcast in 150 public
> radio markets across the country.

***c. A science and technology and instructional workforce that reflects
America\'s diversity***

The COV agreed that it is not appropriate to use this indicator of
success to assess the success of the ISE program. However, elements of
the program relate to this indicator. For example, ESI 95-52551
(Astronomical Society of the Pacific, *Project Astro*, youth & community
project), cited below as an exemplar in promoting the Ideas Strategic
Outcome Goal, also exemplifies a commitment to the principles of NSF's
second criterion by working in multilingual communities, developing
bilingual materials and targeting high minority school districts.
Another example of an ISE project that serves diverse students and
engages diverse communities directly in the scientific enterprise is ESI
97-05519 (New England Aquarium, *Sounds of the Sea,* museum project).
This project empowers a generation of students and teachers to explore
scientific concepts and engage in understanding an array of scientific
professions that are not a traditional part of the K-12 experience. It
created a national model for engaging blind and visually impaired
students and adults in experiencing hands-on science. The program
provided curricula to urban school districts that will reach 15,000
teachers annually through the Teacher Resource Center, and remain in use
beyond the grant period. A legacy of the project is a four-page
"advanced organizer," as well as in-home, in-school, and after-school
activity guides distributed to over 10,000 educators to serve students'
exploration in marine science.

**d. *Globally engaged science and engineering professionals who are
among the best in the world***

The COV agreed that it is not appropriate to use this indicator of
success to assess the success of the ISE program. However, elements of
the program relate to this indicator. For example, ESI 98-15232 (WGBH
Educational Foundation, *World Congress of Science Producers*, media
project), brought together exemplary leaders in their fields to share
professional knowledge and experiences.

**e. *A public that is provided access to the processes and benefits of
science and engineering research and education***

The COV considers the ISE program **successful** in demonstrating
significant achievement of this indicator. We cite various projects that
illustrate how ISE projects provide the public with access to the
processes and benefits of science and engineering research and
education.

-   Evaluation findings for ESI 96-14742 (KCTS Television, *Bill Nye,
    the Science Guy,* media project), include the following as results
    of viewing the Bill Nye TV program:

> •        ninety-four percent of children interviewed conducted at
> least one experiment from the science kits, and
>
> •        children more frequently suggested science activities after
> receiving the kits and parents felt more confident about where to find
> science information and resource.
>
> •      ESI 96-14737 (American Association for Microbiology, *Microbial
> Literacy Collaborative,* media project) produced a four-part TV series
> that was viewed by 1.6 million households each week. As of June 2000,
> 20,000 copies of this project's publication of hands-on activities,
> "Meet the Microbe," have been distributed nationwide (over half to
> individuals and organization upon specific request), and an additional
> 17,000 activities were downloaded from the www.microbeworld.org web
> site.
>
> •        ESI 96-14743 (WGBH Educational Foundation, *ZOOM*, media
> project) reported the following outcome based on evaluation research:
> "Kids are learning science content and process from ZOOM -- there were
> dramatic changes in kids' understanding of several specific ZOOM
> science concepts..."

######### •        ESI 97-05298 (Smithsonian Institution, *Galapagos III,* media project) Imax exemplified public access to scientific discovery in the making: during filming 15 new species of vertebrates and invertebrates were identified, resulting in an NSF-funded media project that directly advanced science. 

## 6. IDEAS Strategic Outcome Goal

*Enabling discovery across the frontier of science and engineering,
connected to learning, innovation and service to society. [FY 2001
Performance Goal]{.underline}: The program is [successful]{.underline}
when, **in the aggregate**, results reported in the period demonstrate
significant achievement in one or more of the indicators below.*

The indicators specified for this goal have little in common with the
ISE program goals stated in the ESIE Annual Reports (e.g., pp. 9-11,
1999 Annual Report) and the Program Solicitation Guidelines for the
years covered by the COV. The evidence in the jackets we reviewed
strongly supports the judgment that the ISE program successfully
accomplished its own stated goals. We strongly recommendation that the
ISE goals in the Annual Report be used as the criteria for future COV
review of the Ideas strategic outcome goal.

 

**a. *A robust and growing fundamental knowledge base that enhances
progress in all science and engineering areas including the science of
learning***

The COV agreed that it is not appropriate to use this indicator of
success to assess the success of the ISE program. However, our comments
below about building the knowledge base for the ISE field are relevant
to this indicator.

**b. *Discoveries that advance the frontiers of science, engineering,
and technology***

The COV agreed that it is not appropriate to use this indicator of
success to assess the success of the ISE program.

***c. Partnerships connecting discovery to innovation, learning, and
societal advancement***

The COV considers the ISE program **successful** in demonstrating
significant achievement of this indicator. While many projects exemplify
how ISE projects focus on scientific discoveries and establish
synergistic partnerships between education, research, and practicing
scientists, we choose to cite the "Project Astro Coalitions" (ESI
95-52551, Astronomical Society of the Pacific, youth & community
project) as a particularly stunning example of this kind of work. This
project links amateur and professional astronomers, school children,
their teachers, and families around a variety of activities, such as
creation of scientific records, production of student research journals,
and collaborative investigation of astrophysical phenomena. In addition
to having formed partnerships that connect groups who have never worked
together before, this project has created a system that has engendered
new coalitions beyond those targeted in the funding cycle. This ISE
project has launched a new idea for public collaboration in science that
is regenerating itself in a variety of forms across the nation.

***d. Research and education processes that are synergistic***

The COV considers the ISE program **successful** in demonstrating
significant achievement of this indicator. We found that some SGER
grants and that the Research Supplement projects were particularly
notable in achieving this type of outcome, as these enable researchers
to communicate their work to the public. The "South Dakota School of
Mines Antarctic Fossils" project (ESI 98-15231, museum project) provides
an excellent example of how the program is achieving this Ideas goal.
Other examples of achievement of synergistic research and education
processes include "Math Activities in Science Museums," awarded to the
Association of Science and Technology Centers, (ESI 99-06982) which
supported a study of how science centers involved audiences in learning
mathematics, and the Exploratorium's "Memory" exhibit (ESI 99-80498) and
the Museum of Science and Industry's "Genetics Decoding" exhibit (ESI
96-14298), both of which effectively incorporate research components in
the exhibit development process.

 

 

## 7. TOOLS Strategic Outcome Goal

*Providing broadly accessible, state-of-the-art information-bases and
shared research and education tools. [FY 2001 Performance
Goal:]{.underline} The program is [successful]{.underline} when, **in
the aggregate**, as a result of its investments, results reported in the
period demonstrate significant progress in achieving one or more of the
indicators below.*

***a. Shared use platforms, facilities, instruments, and databases that
enable discovery***

The COV agreed that it is not appropriate to use this indicator of
success to assess the success of the ISE program.

***b. Shared use platforms, facilities, instruments, and databases that
enhance the productivity and effectiveness of the science and
engineering workforce***

The COV agreed that it is not appropriate to use this indicator of
success to assess the success of the ISE program.

***c. Networking and connectivity that takes full advantage of the
Internet and makes SMET information available to all citizens***

The COV considers the ISE program **successful** in demonstrating
significant achievement of this indicator. Evidence in the jackets
reviewed showed that ISE projects have successfully created broadly
accessible web sites, and made other uses of the Internet to reach broad
audiences. For example, ESI 99-01985 (Exploratorium, *The Electronic
Guidebook*, museum project), is exploring new ways to use the Internet
to make information available to larger audiences. ESI 96-14737
(Microbial Literacy Collaborative, media project), provides a
particularly excellent example of an ISE program that is using Internet
connectivity to make SMET information available to all citizens: the
[www.microbeworld.org](http://www.microbeworld.org/) website created by
this project received 689,620 hits and recorded visits by 29,779
individuals during the first month. The site received over 3 million
hits in the first year.

***d. Information and policy analyses that contribute to the effective
use of science and engineering resources***

The COV agreed that it is not appropriate to use this indicator of
success to assess the success of the ISE program. However, we note that
certain ISE projects produced outcomes and impacts related to this
indicator. For example, ESI 96-27091 (Education Development Center,
*Access by Design*, youth & community project), created tools that help
networks of scientists, community-based organizations, technology
providers, end users, and public policy leaders understand and take
effective action pertaining to equitable access to technology. This
project produced nine booklets for community groups to use in planning
for technology use in service of community, one research report
summarizing the obstacles to access and recommendations for action, four
papers, and a comprehensive guide to resources and information. These
materials and the groups that have been convened to develop and reflect
on them have resulted in continuing collaborations and modifications of
practice among several foundations and community-serving organizations
including, for example, the Rhode Island Indian Council, National Urban
League, and the Benton Foundation. They have influenced the public
policy dialogue of CEOS, Radcliffe School of Public Policy, and the UCLA
School of Public Policy.

## 8. Areas of Emphasis

*For each area specified below, determine whether the investments and
available results demonstrate the likelihood of strong performance in
the future?*

***a. Strategic Outcome: People***

The ISE program is not designed to address the K-12 systemic activities,
instructional workforce/professional development, near-term workforce
needs (Advanced Technology Education program), or broadening
participation through the Tribal Colleges or Partnerships for Innovation
(PFI) programs.

***b. Strategic Outcome: Ideas***

We consider the balance of the ISE portfolio (high risk,
multidisciplinary, or innovative research) above. The ISE program is not
designed to address specific NSF initiatives in information technology
research, nanoscale science and engineering, biocomplexity in the
environment, or fundamental research in the mathematical sciences,
functional genomics, or cognitive neuroscience.

***c. Strategic Outcome: Tools***

The ISE program is not designed to address the NSF's investments
equipment for Terascale Computing Systems, Major Research
Instrumentation Program, Science and Engineering
Information/reports/databases, or new types of scientific databases and
the tools for using them.

##  

## 9. Comments on program areas that the COV believes needs improvement

##  

We commend the ISE staff for the high quality of their work assessing
and managing proposals. We were particularly impressed by their
documentation of success within the ESIE annual reports and suggest that
the ISE program consider publishing a version of this report to help
inform the public of the enormous contribution that ISE makes to our
nation's scientific and technological literacy.

This said, we highlight a number of problems, and in some cases also
provide recommendations for addressing those problems.

> •        We believe that the staff is too small for the tasks required
> of them. One result of this is that program officers are unable to
> devote sufficient time to the early negotiations with grantees and to
> the ongoing monitoring of active projects.

-   We believe that the success of the program review process is largely
    due to the excellent performance of the program officers. However,
    excellent performance of this type cannot be taken for granted, and
    should be developed through new program officer orientation, and
    maintained and further developed through a process of on-going
    oversight.

-   Although the program's practice is consistent with NSF rules, which
    direct all SGER proposals to be reviewed by only one program
    officer, we nonetheless urge the program to ask a second program
    officer to review SGER proposals, in order to strengthen the
    credibility of recommendations on these submissions.

-   We believe that the program needs to develop procedures that ensure
    more careful reviewer attention to the "broader impact" merit
    criterion. These changes might entail more clear and emphatic
    instruction to panelists, including the standard that only proposals
    that they rank high on both criteria should be ranked high.

-   Because it is difficult to ascertain the ethnic backgrounds of
    reviewers, we recommend that the staff request the panelists to
    submit a paragraph describing relevant points about their
    backgrounds and expertise.

-   Attention needs to be paid to reducing the lag time in notification
    once the program officer recommends award of a grant.

-   We believe that the ISE field lacks a "progress model," and
    recommend that the program create an RFP that is designed to foster
    the sharing of knowledge gained through the NSF investment in the
    ISE field. This sharing is necessary in order to build the capacity
    of this field. We suggest that SGER grants be used to further
    encourage capacity building within underrepresented communities. For
    clarification, see C.1, C.2.b. and C.2.c, below.

## 10. Comments on the program's performance in meeting program-specific goals and objectives (non-GPRA outcomes)

######### As stated elsewhere in this report, the ISE program should allocate more attention and resources to: 

######### •        outreach to, and support of groups and communities underrepresented and underserved in science, math, and technology; 

######### •        the development of evaluation plans, and use of formative and summative evaluation outcomes for individuals and for capacity building of the infrastructure; and 

######### •        assistance for first time PIs, especially those from underrepresented groups. 

## 11. Feedback on the COV review process, format and core question

1\. The following factors limited our ability to provide in-depth
commentary with regard to information demonstrating achievement of the
strategic outcome goals:

> •        It is critical---for internal knowledge, external
> credibility, and to educate the public and policymakers about the
> measurable contribution of the informal educational enterprise to the
> health of science---that the ISE program address the matter of
> outcomes in the language of GPRA. However, many elements of the COV
> template are inappropriate to the ISE initiatives, and we are
> concerned that not answering these questions will threaten continued
> support of this important element of the NSF portfolio. We recommend
> that the NSF restate, or provide interpretation of, the template
> questions and indicators so that they are aligned with the educational
> enterprise under review.
>
> •        The time allotted at the COV meeting did not allow us to give
> the outcomes and impact questions the systematic review and analysis
> they deserve. Such a review requires a careful and fairly extensive
> examination of final reports, summative evaluations, and products.
> Those of us who participated in the COV process in prior years found
> that the addition of the GPRA template and the need to evaluate
> outcomes as well as process more than doubled the workload. We had
> sufficient time to examine the jackets in order to answer only the
> questions in Part A. We needed additional time to examine final
> reports and products in order to systematically address Part B. We
> recommend expanding the time for the COV from two to three days.
> Because the field recognizes the seriousness with which the NSF
> regards the peer review process, we believe the extended time will not
> deter potential COV participants.

######### •        In light of the fact that sustainability and longer-term impact are important aspects of the outcomes question, we question whether the typical three-year project time limit, even including project closures, is adequate for assessing outcomes. 

 

2\. We found the "efficiency, time to notification" criterion (A. 1.c.)
difficult to assess because the documentation within the jackets
indicating contact dates for a variety of activities was inconsistent or
confusing. For example, in the box on Form 1, the "return by" was listed
as "ASAP" in one case, and in another, the declination date was listed
as earlier than the submission date. While occasional typographical
errors are to be expected, we suggest that, by including a log sheet on
the left side of the jacket, errors of this type may be more easily
avoided.

 

3\. We found the idea of dividing the COV into three subcommittees
corresponding to the main funding areas of ISE useful and recommend this
approach for future meetings. Based on our experience, we also recommend
that special effort be made to ensure that the award jackets provided
each subcommittee are sorted correctly by funding area.

# C. Supplementary Questions for Informal Science Education COV

***1. Are we gathering appropriate and sufficient data from projects
that would allow us to assess the extent to which the programs are
successfully achieving their intended objectives and outcomes? What
outcomes might we require be documented in final reports and summative
evaluation reports?***

Currently, most projects document outputs, and many projects provide
anecdotal data that indicate success. But only a few of the projects
reviewed provided high quality evaluation data on outcomes, whether in
the form of intermediate or direct indicators of success. We strongly
recommend that the program require each proposal to include clear
statements of intended outcomes and a set of credible indicators by
which they will measure successful attainment of these outcomes. These
indicators should go beyond the gathering of basic data (such as numbers
served). In line with this change, plans for assessment of the proposed
outcomes should be a key factor in panel and program officer reviews of
proposals.

We understand that it will be challenging for both the program and the
field to act on these recommendations. To address this challenge, it is
essential to begin by *acknowledging the difficulty* of measuring
outcomes, especially for media and museum projects where there is no
direct or sustained interaction with the audience. A second challenge is
to *admit* that we have too few appropriate methods and that we need to
find new and creative approaches to capturing the critical data, among
which "intermediate indicators" of change in infrastructure and capacity
should be included. (For example, a national children's science
television program would not be able to show a direct connection between
their product and an increase in student achievement. They could,
however, identify intermediate indicators such as audience ratings,
cognitive gains with small control groups, and data on capacity
building, such as programs that "train the trainers"). A third challenge
is obtaining longitudinal data: rarely can the true impact of a project
be assessed at the end of the funding period because the real impacts
are yet to be realized.

In light of these challenges, we also recommend that the ISE program
develop tools for the field by developing and making easily available a
tested matrix of evaluation variables and protocols that could guide PIs
in the development of their evaluation plans. (One model for a matrix
might be in the six drivers developed for the Systemic Initiatives.)
Systematic use of these variables would support future ISE research that
seeks to aggregate the knowledge gained from individual projects. While
the program could not expect every project to use each element within
this matrix of evaluation variables, the matrix would nonetheless
provide an organizing framework that would support the development of
partially comparable datasets.

We also recommend that the program widely disseminate their projects'
interim reports on progress towards intended outcomes and on new
knowledge developed while implementing a project (e.g., what worked and
what did not), as this information will enhance the body of knowledge
within the field.

***2. How should the ISE program take a more active role in shaping the
direction of the field of informal science education? For example,
should the submission of more applied research projects which assess the
effectiveness and impact of informal science education be encouraged?***

The ISE program already has significantly impacted the field of informal
science education by encouraging and funding quality exhibits, films and
programs, by connecting informal education with schools and the
curricula, and by requiring evaluation. The ISE program plays a major
role in shaping the direction of the field and we are confident it will
continue to support the development of research into the impacts of
informal learning as well as learning in education. This said, we have
several recommendations, which we present by theme.

***a. More clearly articulate your vision***

Following on a recommendation in the 1998 COV Report, we believe the ISE
program is still in need of a more well-articulated vision.

***b. Build capacity for research and evaluation***

The ISE field, as a discipline, lacks a "progress model" that allows the
knowledge base gained from its projects to inform future efforts. We
suggest that you capitalize on your capacity to fund, convene, and
publish in order to develop the ISE research and evidence base. We also
suggest viewing this issue in terms of two action strategies: (1)
develop new ways to measure outcomes, and (2) foster transmission and
use of new knowledge that shapes the field.

> •        Develop a mechanism that "mines" the formative and summative
> data in NSF-sponsored projects, and compiles the literature,
> knowledge, research and evidence about informal science education.[^4]
> We suggest, in particular, development of an RFP for producing a
> repository of gained knowledge, and encourage proposals from
> partnerships comprising ISE "producers" and evaluation experts.
>
> •        Support conferences, commission papers, and create a working
> group to tackle the challenges of informal science evaluation, and
> particularly outcome and impact evaluation. One model might be the
> Evaluation Roundtable on Comprehensive Community Initiatives that was
> supported and administered by the Aspen Institute.
>
> •        Require projects that seek to improve individuals' learning
> or acquisition of skills to cast at least some of their goals in terms
> of participant outcomes, as the ASCEND guidelines demanded. Require
> projects that seek to build infrastructure to make their logic model
> and model of change explicit, and identify a set of intermediate
> outcomes or benchmarks that will provide credible evidence.
>
> •        Provide support to projects as they are developing their
> evaluations. In the ideal, attach to ISE a team of evaluation
> consultants who can be available by email and telephone to work with
> projects. Recognizing that resources will be limited, provision of
> this resource might be feasible if made available only to with
> projects that have received favorable reviews and are at the stage of
> questions and negotiation.
>
> •        Build bridges between the most effective research and
> evaluation practitioners in other similarly complex and confounding
> arenas, such as community development and youth development. Identify
> and work with evaluators who have focused on examining impact, such as
> Patricia Campbell of Campbell-Kibler Associates or Beatriz Chu Clewell
> of the Urban Institute; practitioners who have successfully collected
> impact data, such as Morton Slater of Mount Sinai Medical School and
> the founder of the Gateway to Higher Education Program; and experts in
> methods and approaches for assessing the effects of social action
> programs and reform efforts, such as Carol Weiss of Harvard or
> Elizabeth Stage.

***c. Build the field***

We believe the ISE program should create expectations for, and provide
infrastructure to enable, the sharing of new knowledge developed since
its establishment in 1984. To build the field, we recommend that ISE:

-   Hold meetings for project directors

-   Set up a web page on the NSF site that includes everything from
    research and evaluation designs and findings, to descriptions of
    exemplary programs, to the theoretical and pedagogical strategies
    underpinning informal science education. The Howard Hughes Medical
    Institute (HHMI) fall 2000 pre-college program produced such a
    website following its project directors meeting. (Joseph Perpich,
    formerly of HHMI and now at AAAS, may be a good source of
    information and ideas on this.)

-   Fund, commission, publish and disseminate papers based on a list of
    topics related to informal science education.

-   Organize or co-sponsor major presentations at conferences, both
    inside and outside science education, including for example, annual
    meetings of youth organizations, minority-serving institutions, and
    fields such as literacy education.

***d.* *Capitalize on the burgeoning after-school movement***

After-school programs are proliferating at a rapid rate, with funding
widely available from private foundations at the local and national
levels, compared to miniscule amounts five years ago. But staff in these
programs are not tapping the store of excellent science, math, and
technology resources that have already been developed, rarely understand
the pedagogy and potential of informal science education (and thus
resort to test preparation and drill-and-kill), and generally are not
building on the resources of the community, school, and scientific
institutions in their neighborhood. They offer the ISE a tremendous
opportunity for piloting, demonstrating, implementing, and evaluating
informal educational approaches, for using standards-aligned content and
skills, and for helping academic performance and narrow the gap in
achievement. To capitalize on this opportunity, we recommend that the
ISE program build partnerships with the major players and experts in
this arena, develop a strategic plan for working together over the next
five years, and support collaborative projects between science-rich,
content-rich institutions and after-school programs through the use of
high quality content materials, staff development strategies, and
evaluation methods.

***e. Promote public understanding of and appreciation for informal
science education***

To further this ISE's quest to promote appreciation for the learning
that happens out of school, and make more permeable the boundaries
between school, community, and scientific and cultural institutions:

-   engage evaluators to assess whether, and if so, how and to what
    extent synergistic relationships across these boundaries improve
    science literacy, and

-   collaborate with communications experts on how to promote not just
    the process of informal education, but the kind of scientific
    knowledge we will all need to survive in this technologically and
    genetically complicated century.

***f. Keep a focus on content***

Programs in all three ISE areas, but especially in the youth and
community group, must focus as much on content as on process and skills.
This is not just to help children learn and reduce the achievement gap,
but to meet the general public's need to be informed about topics
ranging from HIV research and prevention to cloning to pharmaceutical
testing to environmental policy.

***g. Expand Youth & Community Initiatives***

We recommend that ISE expand its support for major initiatives by
community and youth coalitions that view math and science
literacy/education as essential to their constituencies' health and
futures. This element of the ISE portfolio is the main locus for
redressing inequities, and the projects in this portfolio should be used
to integrate the equity agenda into other initiatives, both within ISE
and throughout NSF. For details, see the footnote, below.[^5]

***3. How should ISE shape its portfolio to take advantage of emerging
opportunities, such as the World Wide Web? Do the ISE guidelines address
the full range of activities seen in the field?***

Before presenting suggestions with regard to emerging electronic
opportunities, we want to emphasize that the Web should be viewed as a
means rather than an end, and should not be relied on for communication
to all groups, as the communities most underrepresented in science are
also those least likely to have access. It also is important that NSF
use its position to insist that all web pages and technology efforts
they support are fully accessible to persons with disabilities and poor
communities.

This said, the Web is an important emerging informal science "conduit"
that should be explicitly identified in solicitations as a potential
stand-alone vehicle for reaching the informal education populations. The
Web has reached 25% of the American public in less than a decade, faster
than all previous technological innovations. While this technology has
challenges, such as the digital divide, it remains a major untapped
opportunity. Because the potential of the new web technology for
communication, dissemination, and co-creation is so tremendous, ISE is
facing and needs to decide whether to seize the potentially fruitful
opportunity to examine how people learn science and math in and through
the online environment. The COV believes that the ISE program could
greatly assist the field by supporting research and evaluation on how
the Internet can best support learning.

We also note the following other needs/opportunities:

-   a need for ways to "re-purpose" project-specific Web sites for more
    permanent access, perhaps through an NSF web portal;

-   a need to teach children and the general public to better use the
    Web; and

-   opportunities to develop web-casting, distance learning,
    person-to-person electronic interactions, and virtual field trips to
    research sites.

***4. In 2001, ISE was appropriated an additional \$10 million dollars
to provide resources to "expand the pool of ISE grantees to providers in
smaller communities." What strategies are suggested for ensuring that
our programs impact communities with metropolitan area populations under
3 million?***

We note first that the charge to increase the pool of grantees is more
likely to be successful in the museums or youth & community portions of
the ISE portfolio because these venues are strategically placed to
enhance the science literacy of local communities. That is, the real
opportunity here is to support local projects that could have a larger
impact in a region, rather than national efforts "broadcasting" into a
region.

We commend the type of outreach activities recently undertaken by ISE
program officers and encourage their continuation. We recommend other
ideas---all of which depend heavily on partnerships---for encouraging
smaller communities to participate in the ISE program:

-   Partner potential applicants from small communities with more
    experienced project leaders;

> •        Encourage proposals that increase the capacity of smaller
> institutions to develop and manage projects; including collaboratives
> of small organizations;
>
> •        Call upon national organizations and leaders to help identify
> strategies for expanding the pool of grantees, and
>
> •        Develop a "capacity-building" grant program that would
> provide a maximum of \$5,000 for experienced PIs and others to serve
> as consultants to help smaller communities that have never received
> funding understand and tap into the existing field of knowledge about
> what makes a viable project worthy of NSF support. We do not believe
> that this initiative can occur with the current staffing levels. New
> models of dissemination of the opportunities to new potential grantees
> must be established. For example, an RFP might be created to encourage
> existing groups (e./g., ASTC, et al) to use their national reach to
> conduct local awareness and grantsmanship workshops.

-   Develop a program designed to increase the capacity of
    community-building organizations (CBOs) in smaller communities to
    meaningfully engage in informal math, science, and technology
    programming. This program should support CBOs' capacity to vie for
    funding from both federal and non-federal sources. Similarly, we
    would suggest ISE consider developing an initiative that helps
    increase the capacity of organizations with regional and national
    scopes of work, as well as those based in large metropolitan
    communities, to design program mechanisms that can successfully
    reach small metropolitan areas.

-   Target communities by, for example:

> •         Extending programs through already existing partners that
> serve these areas, such as the rural systemic initiatives;
>
> •         Creating inter-governmental programs with the USDA and its
> national 4-H division;
>
> •         requiring grants from a consortium of three to five CBOs
> from across a geographic region (perhaps a model similar to LS-AMP);
>
> •         Supporting initiatives for organizations with a national
> scope to provide programming for smaller and underserved communities.

***5. Do you have any recommendations for ways in which ISE staff can
improve program processes and management based on the results drawn from
the GPRA report template for COVs?***

The ISE proposal management process is very positive and the attention
to documentation and conflict of interest procedures are models for the
field. ISE staff are doing an exceptionally good job maximizing the
outputs and outcomes of grantee initiatives. This excellence, however,
may not be evident in Part B of COV reports, which is shaped by the
NSF's current GPRA template which, in turn, is aligned with the NSF's
strong culture of discipline-based research. The GPRA measure focuses on
ultimate outcomes -- people, ideas, and tools, whereas the ISE program
generally focuses on building infrastructure. This mismatch leaves ISE
COV reviewers with the dilemma of having to use program review criteria
designed to measure ultimate outcomes---people, tools, ideas, whereas
the ISE program should be measured by criteria designed to demonstrate
improvements in cultural change, organizational structures and
processes, and so forth.

To obtain useful insights from the Committee of Visitors, NSF would need
to translate major portions of Part B of the COV template, aligning the
criteria and evaluation guidelines with the ISE outputs and outcomes
goals appearing in its program solicitation manual. It would be wise, in
addition, to be sure that these "translated" indicators are reflected in
the a) proposal guidelines, b) charge panel review, c) charge to
external evaluators, d) and guidelines for producing final reports and
products.

The following additional recommendations for ways to improve program
processes and management are offered:

-   Additional staff and travel funds would facilitate the ability of
    program officers to provide more support to smaller communities and
    first-time grant seekers. It is clear from reviewing the jackets
    that the role of the program officer is extremely beneficial to the
    project results.

-   Stronger outcome reporting, as described earlier, would help to
    build the field.

-   More direction/guidance on how to sort through and prioritize the
    wealth of resources and information provided to ISE panelists (both
    for grant reviews and COV reviews) would be helpful to panelists.

-   It might be helpful to survey PIs (both funded and declined) to
    obtain feedback on the process.

**Staff Response to the Informal Science Education Program**

##### Committee of Visitors Report

**July 2001**

 

The Informal Science Education (ISE) program staff wishes to thank the
Committee of Visitors (COV) for the thorough report and its many
positive findings. We appreciate, too, the careful attention to
recommendations and responses to ISE staff questions.

This report is divided into three major categories: Program Operations
and Management, Supplementary Questions asked by ISE Staff, and
Recommendations on the Review Process for Future COV Meetings. In
reviewing the report, several sections appeared to address related
comments. In these instances, staff took the initiative to combine
responses.

### Program Operations and Management

> •         **The COV believes that staff is too small for tasks
> required of them. One result of this is that Program Officers (POs)
> are unable to devote sufficient time to early negotiations with
> grantees and to ongoing monitoring of active projects. The COV further
> believes that the success of the program review process is largely due
> to the excellent performance of POs. However, excellent performance of
> this type cannot be taken for granted, and should be developed through
> new PO orientations, and maintained and further developed through a
> process of on-going oversight.**

**Response**. The Division regularly monitors workload issues for
reasons cited above. At the time of this COV, the ISE program was
understaffed: one PO had left NSF; another was on medical leave. Also,
additional responsibility for the ASCEND program contributed to the
workload of the Science Literacy Section. A number of recent staffing
changes will alleviate some of the workload issues. The Section has
brought on a full-time and an intermittent PO for museum projects, as
well as a full-time PO for media projects. In addition, an Einstein
Fellow with experience in K-12 and informal education has been assigned
to the Section.

This year, both the EHR Directorate and ESIE are providing training to
new POs; new ISE staff will attend these sessions and focus on topics
highlighted by the COV. In addition, each new PO within ESIE is given an
experienced PO as a mentor as soon as they arrive. ISE, like all
programs within ESIE, has a process of on-going oversight of funding
decisions and program operations by the Division Director and Deputy
Division Director that helps ensure quality across the Program.

•         **The COV recommends that the program develop procedures to
ensure more careful reviewer attention to the "broader impact" merit
criterion. It recommends that instruction to panelists more carefully
explain the meaning of the second criterion, more emphatically require
that explicit attention be given to each criterion, and clearly
stipulate that reviewers should assign high ranks only to proposals
scoring high on both criteria. In particular, the COV recommends that
the entire text of the second criterion, as specified in Dr. Colwell's
"Dear Colleague Letter" (NSF 99-172), be presented to review panelists
at the outset of their deliberations.**

**Response.** The ISE staff is very sensitive to the issue raised by the
COV recommendation. Staff has already provided more explicit attention
to both NSB review criteria (intellectual merit and broader impacts) in
briefings to review panels. Special attention has also been given to the
contents of Dr. Colwell's letter (NSF 99-172). The ISE staff will
continue to review procedures for ensuring that clear guidance is given
to panelists at the time of the review panels and in written materials
sent to panelists in advance of panel meetings.

•         **The COV's review of jackets shows good balance among
reviewers with respect to gender, type and size of institution (museum,
university, media outlet, community, or youth organization), and
geographic distribution. Although members of the COV know from their own
experiences as review panelists that ISE panels include ethnically
diverse members, it was not possible to determine ethnic/racial, or
disability from information contained in jackets. The COV recognizes
that there may be legal considerations in how demographic data are
collected and presented. It therefore recommends that staff request
panelists themselves to submit a paragraph describing relevant points
about their backgrounds and expertise. This information would greatly
support the panel process by ensuring and documenting diversity and
balance, and by further facilitating the exchange of views that makes
the panel process so valuable and contributes to better projects.**

**Response.** The ISE staff always seeks balance and broad
representation among reviewers with respect to gender, type and size of
institution, geographic location, ethnic and racial diversity, and
persons with disabilities. Reviewers are required to provide a recent
curriculum vita and requested to provide demographic information on a
voluntary basis. The selection of panel reviewers is subject to approval
by the ESIE Division Director, who also looks for the diversity noted
above. While some information (discipline, geographic representation,
institutional affiliation, gender) is provided on the Form 1 in each
jacket, other information (race, ethnicity) is not. ESIE is putting in
place formal procedures to systematically collect information on the
characteristics of reviewers after each panel that will be made
available to subsequent COVs. Also, it should be noted that disciplinary
data used by NSF does not adequately capture the expertise (exhibit
design, large-format film production) requisite for ISE. We will try
again to explore opportunities for getting NSF to expand its coding for
expertise for ISE.

•         **The COV recommends that attention be paid to reducing the
lag time in notification once the PO recommends an award. The COV
examined dates of submission, dates of review, dates of communication
and negotiation, and date of PO review analysis. In general, there
appeared to be a trend over the three-year period toward greater
efficiency and a shorter duration in the time between panel review and
making an award. While the program is commended on its generally
successful effort to shorten the time to decision, this process can be
improved. Evidence indicates that some of the problem lies after ISE POs
submit their recommendations.**

**Response.** The ISE staff places high priority on moving funding
decisions forward in a timely manner. Because of the number of proposals
handled by ESIE and the many other tasks that must be completed,
reducing lag time is sometimes difficult. ESIE is working on strategies
for streamlining proposal processing, while maintaining the integrity of
funding decisions. The Division has no control over timeliness of awards
once jackets are forwarded beyond the Division.

•         **The COV found the "efficiency, time to notification"
criterion difficult to assess because the documentation within the
jackets indicating contact dates for a variety of activities was
inconsistent or confusing. It suggests that, by including a log sheet on
the left side of the jacket, errors of this type may be more easily
avoided.**

**Response.** Efficiency and time to notification are always a
challenge, but nonetheless of great importance to ISE staff. The ESIE
Division currently uses a routing slip that tracks the progress of
jackets through the Division. This routing slip is removed after the
award is processed. Including a log sheet within the jacket might help
demonstrate the history of the proposal within ISE and ESIE, and we will
consider its use. However, because NSF is moving towards electronic
jackets, the log sheet might prove unnecessary.

•         **The COV noted that ISE staff does a consistently excellent
job of documentation. Most jackets had exemplary documentation. Program
Officers provided informative notes when there was an anomaly in
process. Some POs documented telephone contact, a process that we
recommend all follow.**

**Response.** The ISE staff concurs that good documentation is essential
to effective project monitoring. All POs have been informed (1) of the
importance of preparing diary notes as records of telephone
conversations or other communications with Principal Investigators (PIs)
and (2) of placing such documentation in award jackets. As the use of
email grows, documentation will be easier to record; copies will be
placed in jackets as documentation of decisions and project milestones.
FastLane is including capabilities for adding documentation of
communications with PIs to electronic jackets. The ISE staff will
familiarize themselves with that capability once available and use it on
a systematic and regular basis.

•         **The COV believes that the informal science education field
lacks a "progress model" and recommends that the program create an RFP
that is designed to foster the sharing of knowledge gained through NSF's
investments in this arena. This sharing is necessary in order to build
the capacity of this field. The COV suggests that ISE capitalize on its
capacity to fund, convene, and publish in order to develop the ISE
research and evidence base. It also suggests viewing this issue in terms
of two action strategies: (1) to develop new ways to measure outcomes,
and (2) to foster transmission and use of new knowledge that shapes the
field.**

**Response.** The ISE staff agrees with the COV that the field lacks a
progress model. As stated previously, ISE is exploring new ways of
measuring outcomes and is encouraging submission of proposals that build
capacity in the field. Some of these proposals were funded as conference
grants and resulted, or will result, in publications demonstrating best
practices and reflecting the latest research. In FY 2001, ESIE's funded
a new Centers for Learning and Teaching (CLT), *Center for Informal
Learning and Schools*, that will conduct research on the intersection of
science learning in formal and informal settings.

In addition, ISE has already put several efforts in place to increase
capacity in the field, including workshops and other efforts that will
help institutions develop better projects through shared knowledge of
best practices. As part of its review of program guidelines in summer
2001, ISE began to think about effective ways for encouraging projects
whose purpose is to disseminate knowledge among individuals and
organizations that effective informal science education providers.

•         **The COV recommends that ISE allocate more attention and
resources for (1) outreach to, and support of, groups and communities
underrepresented and underserved in science, mathematics, and
technology; (2) assistance for first time PIs, especially those from
underrepresented groups; and (3) development of evaluation plans, and
use of formative and summative evaluation outcomes for individuals and
for capacity building of the infrastructure. Moreover, it suggests that
SGER grants be used to further encourage capacity building within
underrepresented communities.**

**Response.** The ISE staff routinely provides assistance to prospective
PIs as they develop ideas and proposals, primarily through the
preliminary proposal process. Preliminary proposals provide Program
Officers the opportunity to mentor prospective PIs on essential elements
of good proposal and project design. Recently, additional measures have
been implemented to reach out to underrepresented and underserved groups
and communities. For example, a series of outreach workshops was held
across the nation last year. Staff is currently discussing other ways to
serve underrepresented audiences. With respect to SGER grants, NSF
policy provides strict guidance on their use. This funding mechanism is
unlikely to fill the need suggested by the COV. The ISE staff will
continue to discuss strategies for increasing capacity for promoting
science literacy in underrepresented communities.

ISE proposals must incorporate plans for both formative and summative
evaluation. While these have helped inform projects and document program
impact, it is clear that evaluation of informal science education
activities is difficult and requires development of new indicators and
methodologies. This is especially true when trying to assess
effectiveness in developing and/or strengthening an education
infrastructure. This past year, ISE has enlisted support of an
evaluation contractor to review project evaluations and to begin the
process of developing indicators to track success of the program. Over
the next several years, ISE plans to move forward to strengthen
evaluation capabilities at both program and project levels.

•         **The COV believes that, due to the nature of informal science
education, proposals pursuing ideas outside the bounds of traditional
ISE activities should be considered cautiously, but that proposals from
new PIs and institutions should be aggressively sought, as these
represent an important means for extending the field and growing a
community of experts. The COV recommends that the program continue to
pursue strategies to encourage and cultivate greater participation from
broader and more diverse potential applicants. The current efforts by
ISE to encourage participation from smaller communities could increase
the number of "high-risk" projects. To further this effort, it
recommends that ISE seek input from the field to better define
innovation and high-risk in the context of informal education.**

**Response.** The ISE staff concurs with the COV that new performers
should be aggressively sought. The Program will pursue aggressive and
creative opportunities for outreach to professional organizations and
new constituencies that constitute the broader ISE community, as well as
provide technical assistance to the field. ISE will be reviewing and
revising the ISE program guidelines with an effort to encourage new
performers and cultivate broader participation. Also, it will solicit
new performers through outreach efforts and dissemination to
professional organizations.

•         **The COV commends ISE staff for the high quality of their
work assessing and managing proposals. It is particularly impressed by
documentation of success within the ESIE annual reports and suggests
that ISE consider publishing a version of this report to help inform the
public of the enormous contribution that ISE makes to our nation's
scientific and technological literacy.**

**Response.** The ISE staff will consider how it might document program
success for public distribution and access. It feels that informing the
public of NSF's and ISE's contribution to scientific literacy is
important. It should be noted that ISE requires awarded projects to
recognize NSF's support in all publications, in media, on Websites, and
in exhibitions.

######### •         **In light of the fact that sustainability and longer-term impact are important aspects of the outcomes question, the COV questions whether the typical three-year project time limit, even including project closures, is adequate for assessing outcomes.** 

**Response.** ISE does not impose three-year project duration. It is
generally requested by most projects that find the time period adequate
for completing a project. ISE staff has encouraged PI's to extend their
duration where it is appropriate and prudent to include a longer time
period for summative evaluation. Also, the program is actively exploring
various evaluation strategies that would document project impact in ways
that facilitate broader program evaluation.

•         **The COV recommends the ISE program seek a second Program
Officer's review of all SGER proposals. While acknowledging that ISE's
practice is consistent with NSF rules that SGER proposals can be
reviewed by only one PO, the COV nonetheless urged ISE to ask a second
PO to review SGER proposals, in order to strengthen the credibility of
recommendations on these submissions.**

**Response.** NSF policy exempts Small Grants for Exploratory Research
(SGER) from external merit review and does not specify the number of
required reviews. The staff understands and fully supports the reasoning
behind the COV's suggestion for securing an additional review. In order
to strengthen documentation of funding decisions on SGER proposals, ISE
will ensure review by a second PO.

# [Supplementary Questions asked by ISE Staff]{.underline}

•         **The COV strongly recommends that ISE require each proposal
to include clear statements of intended outcomes and a set of credible
indicators by which they will measure successful attainment of these
outcomes. These indicators should go beyond the gathering of basic data
(such as numbers served). In line with this change, plans for assessment
of the proposed outcomes should be a key factor in panel and Program
Officer reviews of proposals.**

Response. The ISE staff will review program guidelines with the idea of
including specific language that addresses this recommendation. Through
the Division of Research, Evaluation, and Communication (REC), ISE is
currently working with two contractors in an effort to identify
outcomes, outputs, and indicators of ISE projects. The ISE staff has
also discussed options for collecting data that should contribute to a
larger data pool that can be used to assess overall program impact. A
recent panel of experts called into discuss future program directions
has recommended that guidelines suggest indicators for assessing impact
that challenge the field as a means of promoting development of
evaluation capabilities. The Program will pursue this strategy over the
next several years.

•         **The COV recommends that ISE provide tools for the field by
developing and making easily available a tested matrix of evaluation
variables and protocols that could guide PIs in the development of their
evaluation plans. While the program could not expect every project to
use each element within this matrix of evaluation variables, the matrix
would nonetheless provide an organizing framework that would support the
development of partially comparable datasets.**

**Response.** The ISE staff is in discussion with the REC staff and an
outside contractor regarding evaluation variables and protocols for ISE
projects (see above). At this point, ISE staff is not certain that this
effort will result in the suggested matrix, but it will consider such a
tool for the field.

•         **The COV recommends that ISE widely disseminate projects'
interim reports on progress towards intended outcomes and on new
knowledge developed while implementing a project (e.g., what worked and
what did not), as this information will enhance the body of knowledge
within the field.**

**Response.** The ISE staff does not feel it appropriate to distribute
interim reports or annual reports submitted from PI's. The purpose of
these reports is to inform POs of progress on specific projects, which
may be the basis for decisions regarding their continued funding.
Formative evaluation is, by its very nature, designed to identify
problems in the design or implementation of a project. Consequently, it
might present a rather negative impression of a project at a particular
point in time. It may also contain proprietary information.

The ISE staff, however, fully concurs with the need for building and
disseminating a body of knowledge that will move the field forward.
While ISE encourages PIs to share findings with the field through
presentations at professional meetings, journal articles, posting of
summative evaluations on Websites, etc., it also realizes that these
efforts are too *ad hoc* and sporadic as to be greatly effective. ESIE,
as well as other EHR divisions, is exploring strategies for achieving
the objectives embodied in the COV's recommendation. Such strategies
might include Web-based dissemination of project outcomes, preparation
of monographs for practitioners, publication of results of summative
evaluations and case studies, etc.

•         **Following on a recommendation in the 1998 COV Report, the
COV believes ISE is still in need of a better-articulated vision.**

**Response.** The ISE staff agrees that new opportunities and changes in
the field challenge the program in updating its vision for informal
science education. On September 6-7, 2001, ISE staff met with experts in
museum, media, and community-based programs to discuss directions in the
field. It will use results of this dialogue, as well as communications
with merit reviewers and others in the field, to articulate program
directions that are aligned with the goals and objectives of NSF, EHR,
and ESIE.

•         **The COV recommends that ISE create expectations for, and
provide infrastructure to enable, the sharing of new knowledge developed
since its establishment in 1984. To build the field, they recommend that
ISE: (1) hold meetings for project directors; (2) set up an NSF Website
that includes everything from research and evaluation designs and
findings, to descriptions of exemplary programs, to the theoretical and
pedagogical strategies underpinning informal science education; (3)
fund, commission, publish, and disseminate papers based on a list of
topics related to informal science education; and (4) organize or
co-sponsor major presentations at conferences, both inside and outside
science education, including for example, annual meetings of youth
organizations, minority-serving institutions, and fields such as
literacy education.**

**Response.** ISE does organize and present sessions at conferences,
though the number of presentations has been limited by availability of
travel funds. The ISE staff is working on strategies to encourage and
enable the sharing of new knowledge about informal science education,
exemplified in the many successful ISE projects. As indicated
previously, the Web is intended to play an important role in the
program's dissemination strategy. Because ISE has a small staff whose
major work is reviewing and recommending proposals, there is a limit to
how much can be done by ISE staff. The Program will explore use of
contract support, special grants, and mechanisms for enlisting efforts
of PIs to dissemination new knowledge.

•         **The COV recommends that ISE build partnerships with the
major players and experts in the after-school arena, develop a strategic
plan for working together over the next five years, and support
collaborative projects between science-rich, content-rich institutions
and after-school programs through the use of high quality content
materials, staff development strategies, and evaluation methods.**

**Response.** The NSF After-School Centers for Exploration and New
Discovery, which operates under the general umbrella of the Science
Literacy Section, calls for projects that provide after-school, weekend,
and summer programs. This November, the second ASCEND competition will
be held. ASCEND is supported through funds provided to NSF under the
H1-B Visa legislation. If funds are available to support
student-centered programs under the more recent H1-B legislation, ISE
and other ESIE staff will work to reshape the concept and strengthen
program guidelines based on experience with proposals submitted over the
last year. ISE staff will certainly consider the advice of the COV in
establishing partnerships with major players and experts in this arena.

•         **To further ISE's quest to promote appreciation for the
learning that happens out of school and to make more permeable the
boundaries between school, community, and scientific and cultural
institutions, the COV recommends that ISE engage evaluators to assess
whether, and if so, how and to what extent synergistic relationships
across these boundaries improve science literacy, and collaborate with
communications experts on how to promote not just the process of
informal education, but the kind of scientific knowledge needed to
survive in this technologically and genetically complicated century.**

**Response.** The ISE staff agrees that there is much to learn about
learning in both formal and informal settings. Staff is exploring ways
of assessing, or encouraging proposals that assess and build upon the
research base, learning across boundaries of informal and formal
learning; and that study synergistic relationships between formal and
informal institutions.

•         **Programs in all three ISE areas (i.e., museums, media,
community groups), but especially in the youth and community groups,
must focus as much on content as on process and skills. This is not just
to help children learn and reduce the achievement gap, but to meet the
general public'sneed to be informed about topics ranging from HIV
research and prevention to cloning to pharmaceutical testing to
environmental policy.**

**Response.** The ISE staff agrees that content is very important in ISE
projects, and has been diligent in ensuring that proposed content is
adequately reviewed before making award recommendations. Often, youth
and community projects have a stronger emphasis on science process and
skill development because participants are involved for longer periods
of time. This emphasis, however, does not mitigate the need for
appropriate address of content. The ISE staff will therefore review ISE
guidelines and revise them, as necessary, to present a clearer focus on
SMET content. It will also investigate strengthening requirements for
developing partnerships and meaningful collaboration between informal
science providers and higher education or other content-rich
organizations.

•         **The COV believes that ISE could greatly assist the field by
supporting research and evaluation on how the Internet can best support
learning. It also noted the following other needs/opportunities: (1) a
need for ways to "re-purpose" project-specific Websites for more
permanent access, perhaps through an NSF web portal; (2) a need to teach
children and the general public to better use the Web; and (3)
opportunities to develop Web-casting, distance learning,
person-to-person electronic interactions, and virtual field trips to
research sites.**

**Response.** As one might expect, ISE has seen an increasing number of
proposals that either include a Web-based component or have a Web-based
activity at their core. The ISE staff has been concerned about many of
these proposed projects because they do not define their target
audiences well and do not provide adequate evaluation plans for
assessing impact on target audiences. Among successfully awarded ISE
projects are ones that include much opportunity for interactions, such
as, citizen-scientist programs and Web-casting that include virtual
field trips to research sites. The Program will continue to encourage
innovative projects on how the Internet can best support learning, while
looking for opportunities to build capacity in the field to develop such
proposals and projects and to develop evaluation strategies for
documenting their effectiveness.

•         **The COV commends the type of outreach activities recently
undertaken by ISE POs and encourages their continuation. It recommends
other ideas---all of which depend heavily on partnerships---for
encouraging smaller communities to participate in the ISE program: (1)
partnering potential applicants from small communities with more
experienced project leaders; (2) encouraging proposals that increase the
capacity of smaller institutions to develop and manage projects,
including collaboration of small organizations; (3) calling upon
national organizations and leaders to help identify strategies for
expanding the pool of grantees, and (4) developing a "capacity-building"
grant program.**

**Response.** The ISE staff agrees with the recommendation. Indeed, it
has already put some of these efforts into action. The upcoming revision
of ISE program guidelines will encourage capacity-building activities,
including the partnership of more experienced institutions with less
experienced institutions.

#### [Recommendations on the Review Process for Future COV Meetings]{.underline}

•         **It is critical---for internal knowledge, external
credibility, and to educate the public and policymakers about the
measurable contribution of the informal education enterprise to the
health of science---that ISE address the matter of outcomes in the
language of GPRA. Many elements of the COV template are inappropriate to
ISE initiatives, and the COV is concerned that not answering these
questions will threaten continued support of this important element of
the NSF portfolio. The COV recommends that NSF restate, or provide
interpretation of, template questions and indicators so that they are
aligned with the educational enterprise under review.**

**Response.** Because the GPRA template is used to document the impact
of program efforts across NSF, it cannot be tailored to one program
alone. Under GPRA, NSF is now encouraging programs to modify COV report
templates to include program-specific indicators of success. The ISE
staff, however, is in full concurrence with COV concerns about GPRA
performance reports. The division has already expressed its concerns to
the EHR Directorate, which has conveyed them to those responsible for
implementing GPRA Foundation-wide.

•         **The time allotted to the COV did not allow for the
systematic review and analysis merited by outcomes and impact questions.
Such a review requires a careful and fairly extensive examination of
final reports, summative evaluations, and products. Those who
participated in the COV process in prior years found that the addition
of the GPRA template and the need to evaluate outcomes as well as
process more than doubled the workload. The COV members recommend
expanding the time for the meeting from two to three days. Because the
field recognizes the seriousness with which the NSF regards this
oversight process, we believe the extended time will not deter potential
COV participants.**

**Response.** The ISE staff commends the COV on its outstanding work
within the two-day period allotted. We agree with the COV that three
days might prove more beneficial. In addition, ISE and the Division will
explore ways to synthesize, and distribute early, information that will
facilitate COV deliberations.

•         **The COV found the idea of dividing the COV into three
subcommittees corresponding to the main funding areas of ISE useful and
recommends this approach for future meetings.**

**Response.** The ISE staff agrees that the subcommittee structure
worked well and will recommend its use in the future. Proposals
submitted to ISE generally fall into three major categories (i.e.,
media, museum projects, youth/community-based projects); the
subcommittees were organized around these three categories. In the
future, ISE might consider use of other subcommittees to address special
categories of proposals (e.g., Web-based projects).

National Science Foundation

Education and Human Resources

Elementary, Secondary, and Informal Education

**Staff Response to the Informal Science Education Program**

##### Committee of Visitors Report

**July 2001**

 

The Informal Science Education (ISE) program staff wishes to thank the
Committee of Visitors (COV) for the thorough report and its many
positive findings. We appreciate, too, the careful attention to
recommendations and responses to ISE staff questions.

This report is divided into three major categories: Program Operations
and Management, Supplementary Questions asked by ISE Staff, and
Recommendations on the Review Process for Future COV Meetings. In
reviewing the report, several sections appeared to address related
comments. In these instances, staff took the initiative to combine
responses.

### Program Operations and Management

> •         **The COV believes that staff is too small for tasks
> required of them. One result of this is that Program Officers (POs)
> are unable to devote sufficient time to early negotiations with
> grantees and to ongoing monitoring of active projects. The COV further
> believes that the success of the program review process is largely due
> to the excellent performance of POs. However, excellent performance of
> this type cannot be taken for granted, and should be developed through
> new PO orientations, and maintained and further developed through a
> process of on-going oversight.**

**Response**. The Division regularly monitors workload issues for
reasons cited above. At the time of this COV, the ISE program was
understaffed: one PO had left NSF; another was on medical leave. Also,
additional responsibility for the ASCEND program contributed to the
workload of the Science Literacy Section. A number of recent staffing
changes will alleviate some of the workload issues. The Section has
brought on a full-time and an intermittent PO for museum projects, as
well as a full-time PO for media projects. In addition, an Einstein
Fellow with experience in K-12 and informal education has been assigned
to the Section.

This year, both the EHR Directorate and ESIE are providing training to
new POs; new ISE staff will attend these sessions and focus on topics
highlighted by the COV. In addition, each new PO within ESIE is given an
experienced PO as a mentor as soon as they arrive. ISE, like all
programs within ESIE, has a process of on-going oversight of funding
decisions and program operations by the Division Director and Deputy
Division Director that helps ensure quality across the Program.

•         **The COV recommends that the program develop procedures to
ensure more careful reviewer attention to the "broader impact" merit
criterion. It recommends that instruction to panelists more carefully
explain the meaning of the second criterion, more emphatically require
that explicit attention be given to each criterion, and clearly
stipulate that reviewers should assign high ranks only to proposals
scoring high on both criteria. In particular, the COV recommends that
the entire text of the second criterion, as specified in Dr. Colwell's
"Dear Colleague Letter" (NSF 99-172), be presented to review panelists
at the outset of their deliberations.**

**Response.** The ISE staff is very sensitive to the issue raised by the
COV recommendation. Staff has already provided more explicit attention
to both NSB review criteria (intellectual merit and broader impacts) in
briefings to review panels. Special attention has also been given to the
contents of Dr. Colwell's letter (NSF 99-172). The ISE staff will
continue to review procedures for ensuring that clear guidance is given
to panelists at the time of the review panels and in written materials
sent to panelists in advance of panel meetings.

•         **The COV's review of jackets shows good balance among
reviewers with respect to gender, type and size of institution (museum,
university, media outlet, community, or youth organization), and
geographic distribution. Although members of the COV know from their own
experiences as review panelists that ISE panels include ethnically
diverse members, it was not possible to determine ethnic/racial, or
disability from information contained in jackets. The COV recognizes
that there may be legal considerations in how demographic data are
collected and presented. It therefore recommends that staff request
panelists themselves to submit a paragraph describing relevant points
about their backgrounds and expertise. This information would greatly
support the panel process by ensuring and documenting diversity and
balance, and by further facilitating the exchange of views that makes
the panel process so valuable and contributes to better projects.**

**Response.** The ISE staff always seeks balance and broad
representation among reviewers with respect to gender, type and size of
institution, geographic location, ethnic and racial diversity, and
persons with disabilities. Reviewers are required to provide a recent
curriculum vita and requested to provide demographic information on a
voluntary basis. The selection of panel reviewers is subject to approval
by the ESIE Division Director, who also looks for the diversity noted
above. While some information (discipline, geographic representation,
institutional affiliation, gender) is provided on the Form 1 in each
jacket, other information (race, ethnicity) is not. ESIE is putting in
place formal procedures to systematically collect information on the
characteristics of reviewers after each panel that will be made
available to subsequent COVs. Also, it should be noted that disciplinary
data used by NSF does not adequately capture the expertise (exhibit
design, large-format film production) requisite for ISE. We will try
again to explore opportunities for getting NSF to expand its coding for
expertise for ISE.

•         **The COV recommends that attention be paid to reducing the
lag time in notification once the PO recommends an award. The COV
examined dates of submission, dates of review, dates of communication
and negotiation, and date of PO review analysis. In general, there
appeared to be a trend over the three-year period toward greater
efficiency and a shorter duration in the time between panel review and
making an award. While the program is commended on its generally
successful effort to shorten the time to decision, this process can be
improved. Evidence indicates that some of the problem lies after ISE POs
submit their recommendations.**

**Response.** The ISE staff places high priority on moving funding
decisions forward in a timely manner. Because of the number of proposals
handled by ESIE and the many other tasks that must be completed,
reducing lag time is sometimes difficult. ESIE is working on strategies
for streamlining proposal processing, while maintaining the integrity of
funding decisions. The Division has no control over timeliness of awards
once jackets are forwarded beyond the Division.

•         **The COV found the "efficiency, time to notification"
criterion difficult to assess because the documentation within the
jackets indicating contact dates for a variety of activities was
inconsistent or confusing. It suggests that, by including a log sheet on
the left side of the jacket, errors of this type may be more easily
avoided.**

**Response.** Efficiency and time to notification are always a
challenge, but nonetheless of great importance to ISE staff. The ESIE
Division currently uses a routing slip that tracks the progress of
jackets through the Division. This routing slip is removed after the
award is processed. Including a log sheet within the jacket might help
demonstrate the history of the proposal within ISE and ESIE, and we will
consider its use. However, because NSF is moving towards electronic
jackets, the log sheet might prove unnecessary.

•         **The COV noted that ISE staff does a consistently excellent
job of documentation. Most jackets had exemplary documentation. Program
Officers provided informative notes when there was an anomaly in
process. Some POs documented telephone contact, a process that we
recommend all follow.**

**Response.** The ISE staff concurs that good documentation is essential
to effective project monitoring. All POs have been informed (1) of the
importance of preparing diary notes as records of telephone
conversations or other communications with Principal Investigators (PIs)
and (2) of placing such documentation in award jackets. As the use of
email grows, documentation will be easier to record; copies will be
placed in jackets as documentation of decisions and project milestones.
FastLane is including capabilities for adding documentation of
communications with PIs to electronic jackets. The ISE staff will
familiarize themselves with that capability once available and use it on
a systematic and regular basis.

•         **The COV believes that the informal science education field
lacks a "progress model" and recommends that the program create an RFP
that is designed to foster the sharing of knowledge gained through NSF's
investments in this arena. This sharing is necessary in order to build
the capacity of this field. The COV suggests that ISE capitalize on its
capacity to fund, convene, and publish in order to develop the ISE
research and evidence base. It also suggests viewing this issue in terms
of two action strategies: (1) to develop new ways to measure outcomes,
and (2) to foster transmission and use of new knowledge that shapes the
field.**

**Response.** The ISE staff agrees with the COV that the field lacks a
progress model. As stated previously, ISE is exploring new ways of
measuring outcomes and is encouraging submission of proposals that build
capacity in the field. Some of these proposals were funded as conference
grants and resulted, or will result, in publications demonstrating best
practices and reflecting the latest research. In FY 2001, ESIE's funded
a new Centers for Learning and Teaching (CLT), *Center for Informal
Learning and Schools*, that will conduct research on the intersection of
science learning in formal and informal settings.

In addition, ISE has already put several efforts in place to increase
capacity in the field, including workshops and other efforts that will
help institutions develop better projects through shared knowledge of
best practices. As part of its review of program guidelines in summer
2001, ISE began to think about effective ways for encouraging projects
whose purpose is to disseminate knowledge among individuals and
organizations that effective informal science education providers.

•         **The COV recommends that ISE allocate more attention and
resources for (1) outreach to, and support of, groups and communities
underrepresented and underserved in science, mathematics, and
technology; (2) assistance for first time PIs, especially those from
underrepresented groups; and (3) development of evaluation plans, and
use of formative and summative evaluation outcomes for individuals and
for capacity building of the infrastructure. Moreover, it suggests that
SGER grants be used to further encourage capacity building within
underrepresented communities.**

**Response.** The ISE staff routinely provides assistance to prospective
PIs as they develop ideas and proposals, primarily through the
preliminary proposal process. Preliminary proposals provide Program
Officers the opportunity to mentor prospective PIs on essential elements
of good proposal and project design. Recently, additional measures have
been implemented to reach out to underrepresented and underserved groups
and communities. For example, a series of outreach workshops was held
across the nation last year. Staff is currently discussing other ways to
serve underrepresented audiences. With respect to SGER grants, NSF
policy provides strict guidance on their use. This funding mechanism is
unlikely to fill the need suggested by the COV. The ISE staff will
continue to discuss strategies for increasing capacity for promoting
science literacy in underrepresented communities.

ISE proposals must incorporate plans for both formative and summative
evaluation. While these have helped inform projects and document program
impact, it is clear that evaluation of informal science education
activities is difficult and requires development of new indicators and
methodologies. This is especially true when trying to assess
effectiveness in developing and/or strengthening an education
infrastructure. This past year, ISE has enlisted support of an
evaluation contractor to review project evaluations and to begin the
process of developing indicators to track success of the program. Over
the next several years, ISE plans to move forward to strengthen
evaluation capabilities at both program and project levels.

•         **The COV believes that, due to the nature of informal science
education, proposals pursuing ideas outside the bounds of traditional
ISE activities should be considered cautiously, but that proposals from
new PIs and institutions should be aggressively sought, as these
represent an important means for extending the field and growing a
community of experts. The COV recommends that the program continue to
pursue strategies to encourage and cultivate greater participation from
broader and more diverse potential applicants. The current efforts by
ISE to encourage participation from smaller communities could increase
the number of "high-risk" projects. To further this effort, it
recommends that ISE seek input from the field to better define
innovation and high-risk in the context of informal education.**

**Response.** The ISE staff concurs with the COV that new performers
should be aggressively sought. The Program will pursue aggressive and
creative opportunities for outreach to professional organizations and
new constituencies that constitute the broader ISE community, as well as
provide technical assistance to the field. ISE will be reviewing and
revising the ISE program guidelines with an effort to encourage new
performers and cultivate broader participation. Also, it will solicit
new performers through outreach efforts and dissemination to
professional organizations.

•         **The COV commends ISE staff for the high quality of their
work assessing and managing proposals. It is particularly impressed by
documentation of success within the ESIE annual reports and suggests
that ISE consider publishing a version of this report to help inform the
public of the enormous contribution that ISE makes to our nation's
scientific and technological literacy.**

**Response.** The ISE staff will consider how it might document program
success for public distribution and access. It feels that informing the
public of NSF's and ISE's contribution to scientific literacy is
important. It should be noted that ISE requires awarded projects to
recognize NSF's support in all publications, in media, on Websites, and
in exhibitions.

######### •         **In light of the fact that sustainability and longer-term impact are important aspects of the outcomes question, the COV questions whether the typical three-year project time limit, even including project closures, is adequate for assessing outcomes.** 

**Response.** ISE does not impose three-year project duration. It is
generally requested by most projects that find the time period adequate
for completing a project. ISE staff has encouraged PI's to extend their
duration where it is appropriate and prudent to include a longer time
period for summative evaluation. Also, the program is actively exploring
various evaluation strategies that would document project impact in ways
that facilitate broader program evaluation.

•         **The COV recommends the ISE program seek a second Program
Officer's review of all SGER proposals. While acknowledging that ISE's
practice is consistent with NSF rules that SGER proposals can be
reviewed by only one PO, the COV nonetheless urged ISE to ask a second
PO to review SGER proposals, in order to strengthen the credibility of
recommendations on these submissions.**

**Response.** NSF policy exempts Small Grants for Exploratory Research
(SGER) from external merit review and does not specify the number of
required reviews. The staff understands and fully supports the reasoning
behind the COV's suggestion for securing an additional review. In order
to strengthen documentation of funding decisions on SGER proposals, ISE
will ensure review by a second PO.

# [Supplementary Questions asked by ISE Staff]{.underline}

•         **The COV strongly recommends that ISE require each proposal
to include clear statements of intended outcomes and a set of credible
indicators by which they will measure successful attainment of these
outcomes. These indicators should go beyond the gathering of basic data
(such as numbers served). In line with this change, plans for assessment
of the proposed outcomes should be a key factor in panel and Program
Officer reviews of proposals.**

Response. The ISE staff will review program guidelines with the idea of
including specific language that addresses this recommendation. Through
the Division of Research, Evaluation, and Communication (REC), ISE is
currently working with two contractors in an effort to identify
outcomes, outputs, and indicators of ISE projects. The ISE staff has
also discussed options for collecting data that should contribute to a
larger data pool that can be used to assess overall program impact. A
recent panel of experts called into discuss future program directions
has recommended that guidelines suggest indicators for assessing impact
that challenge the field as a means of promoting development of
evaluation capabilities. The Program will pursue this strategy over the
next several years.

•         **The COV recommends that ISE provide tools for the field by
developing and making easily available a tested matrix of evaluation
variables and protocols that could guide PIs in the development of their
evaluation plans. While the program could not expect every project to
use each element within this matrix of evaluation variables, the matrix
would nonetheless provide an organizing framework that would support the
development of partially comparable datasets.**

**Response.** The ISE staff is in discussion with the REC staff and an
outside contractor regarding evaluation variables and protocols for ISE
projects (see above). At this point, ISE staff is not certain that this
effort will result in the suggested matrix, but it will consider such a
tool for the field.

•         **The COV recommends that ISE widely disseminate projects'
interim reports on progress towards intended outcomes and on new
knowledge developed while implementing a project (e.g., what worked and
what did not), as this information will enhance the body of knowledge
within the field.**

**Response.** The ISE staff does not feel it appropriate to distribute
interim reports or annual reports submitted from PI's. The purpose of
these reports is to inform POs of progress on specific projects, which
may be the basis for decisions regarding their continued funding.
Formative evaluation is, by its very nature, designed to identify
problems in the design or implementation of a project. Consequently, it
might present a rather negative impression of a project at a particular
point in time. It may also contain proprietary information.

The ISE staff, however, fully concurs with the need for building and
disseminating a body of knowledge that will move the field forward.
While ISE encourages PIs to share findings with the field through
presentations at professional meetings, journal articles, posting of
summative evaluations on Websites, etc., it also realizes that these
efforts are too *ad hoc* and sporadic as to be greatly effective. ESIE,
as well as other EHR divisions, is exploring strategies for achieving
the objectives embodied in the COV's recommendation. Such strategies
might include Web-based dissemination of project outcomes, preparation
of monographs for practitioners, publication of results of summative
evaluations and case studies, etc.

•         **Following on a recommendation in the 1998 COV Report, the
COV believes ISE is still in need of a better-articulated vision.**

**Response.** The ISE staff agrees that new opportunities and changes in
the field challenge the program in updating its vision for informal
science education. On September 6-7, 2001, ISE staff met with experts in
museum, media, and community-based programs to discuss directions in the
field. It will use results of this dialogue, as well as communications
with merit reviewers and others in the field, to articulate program
directions that are aligned with the goals and objectives of NSF, EHR,
and ESIE.

•         **The COV recommends that ISE create expectations for, and
provide infrastructure to enable, the sharing of new knowledge developed
since its establishment in 1984. To build the field, they recommend that
ISE: (1) hold meetings for project directors; (2) set up an NSF Website
that includes everything from research and evaluation designs and
findings, to descriptions of exemplary programs, to the theoretical and
pedagogical strategies underpinning informal science education; (3)
fund, commission, publish, and disseminate papers based on a list of
topics related to informal science education; and (4) organize or
co-sponsor major presentations at conferences, both inside and outside
science education, including for example, annual meetings of youth
organizations, minority-serving institutions, and fields such as
literacy education.**

**Response.** ISE does organize and present sessions at conferences,
though the number of presentations has been limited by availability of
travel funds. The ISE staff is working on strategies to encourage and
enable the sharing of new knowledge about informal science education,
exemplified in the many successful ISE projects. As indicated
previously, the Web is intended to play an important role in the
program's dissemination strategy. Because ISE has a small staff whose
major work is reviewing and recommending proposals, there is a limit to
how much can be done by ISE staff. The Program will explore use of
contract support, special grants, and mechanisms for enlisting efforts
of PIs to dissemination new knowledge.

•         **The COV recommends that ISE build partnerships with the
major players and experts in the after-school arena, develop a strategic
plan for working together over the next five years, and support
collaborative projects between science-rich, content-rich institutions
and after-school programs through the use of high quality content
materials, staff development strategies, and evaluation methods.**

**Response.** The NSF After-School Centers for Exploration and New
Discovery, which operates under the general umbrella of the Science
Literacy Section, calls for projects that provide after-school, weekend,
and summer programs. This November, the second ASCEND competition will
be held. ASCEND is supported through funds provided to NSF under the
H1-B Visa legislation. If funds are available to support
student-centered programs under the more recent H1-B legislation, ISE
and other ESIE staff will work to reshape the concept and strengthen
program guidelines based on experience with proposals submitted over the
last year. ISE staff will certainly consider the advice of the COV in
establishing partnerships with major players and experts in this arena.

•         **To further ISE's quest to promote appreciation for the
learning that happens out of school and to make more permeable the
boundaries between school, community, and scientific and cultural
institutions, the COV recommends that ISE engage evaluators to assess
whether, and if so, how and to what extent synergistic relationships
across these boundaries improve science literacy, and collaborate with
communications experts on how to promote not just the process of
informal education, but the kind of scientific knowledge needed to
survive in this technologically and genetically complicated century.**

**Response.** The ISE staff agrees that there is much to learn about
learning in both formal and informal settings. Staff is exploring ways
of assessing, or encouraging proposals that assess and build upon the
research base, learning across boundaries of informal and formal
learning; and that study synergistic relationships between formal and
informal institutions.

•         **Programs in all three ISE areas (i.e., museums, media,
community groups), but especially in the youth and community groups,
must focus as much on content as on process and skills. This is not just
to help children learn and reduce the achievement gap, but to meet the
general public'sneed to be informed about topics ranging from HIV
research and prevention to cloning to pharmaceutical testing to
environmental policy.**

**Response.** The ISE staff agrees that content is very important in ISE
projects, and has been diligent in ensuring that proposed content is
adequately reviewed before making award recommendations. Often, youth
and community projects have a stronger emphasis on science process and
skill development because participants are involved for longer periods
of time. This emphasis, however, does not mitigate the need for
appropriate address of content. The ISE staff will therefore review ISE
guidelines and revise them, as necessary, to present a clearer focus on
SMET content. It will also investigate strengthening requirements for
developing partnerships and meaningful collaboration between informal
science providers and higher education or other content-rich
organizations.

•         **The COV believes that ISE could greatly assist the field by
supporting research and evaluation on how the Internet can best support
learning. It also noted the following other needs/opportunities: (1) a
need for ways to "re-purpose" project-specific Websites for more
permanent access, perhaps through an NSF web portal; (2) a need to teach
children and the general public to better use the Web; and (3)
opportunities to develop Web-casting, distance learning,
person-to-person electronic interactions, and virtual field trips to
research sites.**

**Response.** As one might expect, ISE has seen an increasing number of
proposals that either include a Web-based component or have a Web-based
activity at their core. The ISE staff has been concerned about many of
these proposed projects because they do not define their target
audiences well and do not provide adequate evaluation plans for
assessing impact on target audiences. Among successfully awarded ISE
projects are ones that include much opportunity for interactions, such
as, citizen-scientist programs and Web-casting that include virtual
field trips to research sites. The Program will continue to encourage
innovative projects on how the Internet can best support learning, while
looking for opportunities to build capacity in the field to develop such
proposals and projects and to develop evaluation strategies for
documenting their effectiveness.

•         **The COV commends the type of outreach activities recently
undertaken by ISE POs and encourages their continuation. It recommends
other ideas---all of which depend heavily on partnerships---for
encouraging smaller communities to participate in the ISE program: (1)
partnering potential applicants from small communities with more
experienced project leaders; (2) encouraging proposals that increase the
capacity of smaller institutions to develop and manage projects,
including collaboration of small organizations; (3) calling upon
national organizations and leaders to help identify strategies for
expanding the pool of grantees, and (4) developing a "capacity-building"
grant program.**

**Response.** The ISE staff agrees with the recommendation. Indeed, it
has already put some of these efforts into action. The upcoming revision
of ISE program guidelines will encourage capacity-building activities,
including the partnership of more experienced institutions with less
experienced institutions.

#### [Recommendations on the Review Process for Future COV Meetings]{.underline}

•         **It is critical---for internal knowledge, external
credibility, and to educate the public and policymakers about the
measurable contribution of the informal education enterprise to the
health of science---that ISE address the matter of outcomes in the
language of GPRA. Many elements of the COV template are inappropriate to
ISE initiatives, and the COV is concerned that not answering these
questions will threaten continued support of this important element of
the NSF portfolio. The COV recommends that NSF restate, or provide
interpretation of, template questions and indicators so that they are
aligned with the educational enterprise under review.**

**Response.** Because the GPRA template is used to document the impact
of program efforts across NSF, it cannot be tailored to one program
alone. Under GPRA, NSF is now encouraging programs to modify COV report
templates to include program-specific indicators of success. The ISE
staff, however, is in full concurrence with COV concerns about GPRA
performance reports. The division has already expressed its concerns to
the EHR Directorate, which has conveyed them to those responsible for
implementing GPRA Foundation-wide.

•         **The time allotted to the COV did not allow for the
systematic review and analysis merited by outcomes and impact questions.
Such a review requires a careful and fairly extensive examination of
final reports, summative evaluations, and products. Those who
participated in the COV process in prior years found that the addition
of the GPRA template and the need to evaluate outcomes as well as
process more than doubled the workload. The COV members recommend
expanding the time for the meeting from two to three days. Because the
field recognizes the seriousness with which the NSF regards this
oversight process, we believe the extended time will not deter potential
COV participants.**

**Response.** The ISE staff commends the COV on its outstanding work
within the two-day period allotted. We agree with the COV that three
days might prove more beneficial. In addition, ISE and the Division will
explore ways to synthesize, and distribute early, information that will
facilitate COV deliberations.

•         **The COV found the idea of dividing the COV into three
subcommittees corresponding to the main funding areas of ISE useful and
recommends this approach for future meetings.**

**Response.** The ISE staff agrees that the subcommittee structure
worked well and will recommend its use in the future. Proposals
submitted to ISE generally fall into three major categories (i.e.,
media, museum projects, youth/community-based projects); the
subcommittees were organized around these three categories. In the
future, ISE might consider use of other subcommittees to address special
categories of proposals (e.g., Web-based projects).

 

 

[^1]: The program officers provided a sample of 45 declined and 45
    awarded jackets. Time prevented review of all of the declination
    jackets. Three of the award jackets s were not reviewed due to
    committee COIs.

[^2]: The NSF Proposal and Award Manual (Ch. 5, Sec. 2) defines broader
    impacts as follows:

    *"What are the broader impacts of the proposed activity? How well
    does the activity advance discovery and understanding while
    promoting teaching, training, and learning? How well does the*

    *proposed activity broaden the participation of underrepresented
    groups (e.g.,gender, ethnicity, disability, geographic, etc.)? To
    what extent will it enhance the infrastructure for research and
    education, such as facilities, instrumentation, networks, and
    partnerships? Will the results be disseminated broadly to enhance
    scientific and technological understanding? What may be the benefits
    of the proposed activity to society?*

    *"Reviewers are asked to provide an overall rating, as shown on the
    review form, as well*

    *as separate statements addressing each of the two criteria. The
    statements also should include comments about the relative
    importance of the two criteria in assigning the rating. Note that
    the criteria need not be weighed equally. PIs also are reminded to
    address the following in their proposal to provide reviewers with
    the information necessary to respond fully to the above described
    merit review criteria. NSF Program staff must give these elements
    careful consideration in making funding decisions."*

     

[^3]: Relevant to this point is the March 6, 1997 memo by Susan Cozzens:
    "Analysis of Responses to the NSB/NSF Report on Merit Review
    Criteria."

[^4]: We need a resource that goes beyond Valerie Crane's edited book
    and provide a comprehensive base that informs the field about where
    we are, what we know and don't know, what difference it makes, the
    research and evaluation challenges, and possible approaches and
    solutions.

     

[^5]: If one examines the range of offerings in the youth & community
    portfolio, it appears to be underdefined and quite limited in scope.
    The investment shows a sharp decline in community awards between FY
    1999 and FY 2000. The number of programs being funded is quite
    small, and the range of organizations involved does not begin to
    reflect the diversity of groups that could and should be active in
    promoting participation in informal science education. Program
    closures in these years reflect a similar dearth of involvement by
    community-based or youth organizations, and also indicate that the
    bulk of the community projects that had been funded were part of the
    parent involvement initiative.

    We suggest that ISE examine how it can increase the diversity of
    proposers, audience, and focus areas. We recommend that ISE:

    •          Clarify for community serving organizations and the
    informal science education field at large the range and type of
    program that it is seeking to support. Make clear the ISE/NSF
    mission and how community and youth groups can forward that effort.
    Mailings, presentations, telephone and on-line communication could
    reach groups that serve the target audiences but have not been NSF
    insiders, as well as current PIs, to raise awareness about how they
    can increase their attention to community, youth,
    underrepresentation, underservice, and public understanding in its
    broadest form.

    > Early in the life of ISE, former ISE head George Tressel was
    > proactive in contacting these groups and the "other" portfolio
    > grew from one to multiple projects during his tenure. In those and
    > in the ensuing years, ISE supported organizations such as the
    > National Urban League, La Raza, ASPIRA, Girls Incorporated, Girl
    > Scouts, and NACME.
    >
    >  

    These efforts were then further leveraged by the role that AAAS has
    played in community based science, and AAAS's leadership role in
    creating linkages between the scientific community and
    community-based organizations to get science on the agendas of youth
    and community groups. AAAS in turn was able to support community
    groups in the proposal preparation and project development process,
    and many of the above grants were the result of AAAS's capacity
    building efforts. Their role as an intermediary might be one
    strategy for ISE to consider as it strengthens its community/youth
    portfolio.