[MEMORANDUM]{.underline}

February 25, 2004

TO: Judith A. Ramaley

Assistant Director, EHR

VIA: James H.Lightbourne

> Senior Advisor for Planning, Analysis and Policy, EHR

VIA: Rosemary Haggett

> Division Director, DUE

FROM: Joan Prival

> Program Director, DUE

SUBJECT: Response to Committee of Visitors report on CETP and STEMTP
programs

This memorandum responds to the COV report submitted to EHR by Committee
Chairman, Robert Devaney.

The COV convened at NSF on October 9-10, 2003 to review the NSF
Collaboratives for Excellence in Teacher Preparation (CETP) and Science,
Technology, Engineering, and Mathematics Teacher Preparation (STEMTP)
Programs. The COV\'s review process followed the requirements of the
Government Performance and Results Act (GPRA). In addition, the COV
responded to a set of questions developed by the Division of
Undergraduate Education. This response addresses comments and issues
that were included in the COV report. Responses are organized in
accordance with the order provided by GPRA template and page references
refer to the page numbers of the COV report.

**A.1. QUALITY AND EFFECTIVENESS OF MERIT REVIEW PROCEDURES**

Core Question: (Page 4) \"Do panel summaries provide sufficient
information for the principal investigator(s) to understand the basis
for the panel recommendation?\"

> COV Comment: *\"* *In general the panel summaries did provide
> sufficient information for the PIs. The panel chair's comments usually
> reflected the reviewers' concerns and supportive comments. One
> exception was a special project, which was not as strong as some of
> the others the COV looked at. In the review, there was a failure to
> mention that one of the strong concerns was the apparent lack of
> awareness of the national standards.\"*

**Program Response:**

DUE appreciates the COV\'s comments regarding the sufficiency of the
panel summaries in general. The exception is noted and serves as a
reminder of the need to emphasize to panelists the importance of
incorporating panelists\' observations into the panel summary.

Core question (Page 4): \"Is the documentation for recommendations
complete, and does the program officer provide sufficient information
and justification for her/his recommendation?\"

> COV Comments:
>
> *\"The COV looked in detail at several proposals that were awarded
> even though the panels rated them significantly lower than other
> proposals that were declined. For example, in the STEMTP 03
> competition, the top 7 proposals were rated between 4.6 and 5.0 and
> were all funded. One other project was rated 3.83 and ranked number 24
> overall but was also funded. There was no rationale for this action
> given in the program director's narrative. Nonetheless, comparisons
> with several of the funded projects indicate that this award seems to
> be justified. NSF staff later provided verbal information indicating
> why this project was moved ahead of the other projects.\"*

And Core Question (page 5): Issues identified by the COV concerning the
quality and effectiveness of the program\'s use of merit review
procedures:

> *\"The one flaw in the system as noted above involves full
> documentation of the rationale for funding proposals whose ratings are
> lower than other projects that are not funded. In the STEMTP Proposal
> Recommendation guidelines, the FY2003 recommendations include a
> reference to an explanation of funding decisions when they are counter
> to panel review analysis. The COV noted that one project was funded
> with a 3.83 overall rating while 12 proposals with higher reviews were
> not funded. However, the jacket for the project in question did not
> include a "fully documented" rationale. There are no doubt appropriate
> rationales for this decision; e.g., that this was the only physics
> proposal. But the integrity of process would dictate that the
> rationale for these decisions be clearly documented. The Review
> Analysis Form includes a section for "Reviews in Conflict with
> Recommendations". Either this section should be used to include
> statements from program officers that make the rationale for support
> when looking at "the big picture" or a new section should be created
> for this purpose.*

*In addition, while there is extensive external review prior to initial
funding decisions, the degree to which the external review process
impacts continued funding is less well defined. For example, review
scores to extend follow-up awards in the CETP were significantly lower
than initial funding awards (For example, CETP 2001 awards ranked from
3.5-2.75 while their initial funding awards ranked from 4.63-3.0).\"*

**Program response:**

As the COV noted, in general, ranking of proposals by reviewers\' scores
may correspond to funding decisions; however, funding decisions are not
based purely on a numerical ranking of proposals. Competitive proposals
worthy of funding may fall below the highly competitive range and may be
funded if funds are available. Such decisions are typically made in the
interests of balancing or expanding the portfolio. DUE acknowledges the
COV\'s comments regarding the need to provide a clearer rationale for
funding decisions, particularly when decisions may appear to be
inconsistent with the rank order associated with reviewers\' scores. A
more complete explanation of funding decisions can be obtained through
examination of the collection of review analyses for the set of
proposals considered highly competitive, yet not funded. The review
analyses, in these cases, provide the rationale for declining the
proposal, and when considered in conjunction with the rationale for
funding other proposals, a more comprehensive basis for the award
decision becomes evident. Nevertheless, we acknowledge the need for
providing clarification about such funding decisions. To this end, we
are exploring options for providing further documentation, beyond what
is currently provided through the Proposal Recommendation Process
document referenced in the review analysis. Specifically, we will ensure
each review analysis provides a clear justification of the decision to
fund a proposal that falls below the group of proposals considered
highly competitive. In addition, recognizing that each review analysis
addresses a particular proposal, rather than a group of proposals, we
will pursue the development of an addendum to the Proposal
Recommendation Process document for inclusion in the Program Book that
provides an overview of the final portfolio of projects for a given
competition, fully explaining specific funding decisions that may appear
to deviate from simple rank order funding.

Regarding the distinction between external review for initial funding
decisions and \"follow-on\" funding under the CETP program, these are
separate competitions, with the former consisting of panel review
followed by reverse site review, and the latter consisting of mail
review, with varying numbers of proposals and different review criteria
in each case. Reviewers\' scores may not be comparable from one
competition to another.

**Section A.2 IMPLEMENTATION OF NSF MERIT REVIEW CRITERIA**

Core Question (page 6): Have the individual reviews (either mail or
panel) addressed whether the proposal contributes to both merit review
criteria?

COV Comments:

> *\"The CETP preproposal review forms clearly address the intellectual
> merit and broader impacts issue. However, while the proposal reviews
> for CETP clearly cited strengths and weaknesses of the proposal, they
> often did not specifically address the merit review criteria. On the
> other hand, all of the STEMTP proposals that members of the COV
> examined did address both criteria. The COV suspects that this is the
> case with most of the early proposals versus later proposals. The
> increased emphasis in later years on the merit criteria is evident.\"*

**Program response:** Proposals submitted prior to the strengthening of
NSF requirements regarding Intellectual Merit and Broader Impacts (CETP
proposals), may not have explicitly addressed these criteria. With the
introduction of Fastlane reviews with specific text boxes for
Intellectual Merit and Broader Impacts, the reviewer form prompts the
reviewer to address these criteria separately. FastLane reviews were in
place for STEMTP proposals and this increased attention to the review
criteria was noted by the COV.

Core question (page 6): Have the panel summary reviews addressed whether
the proposal contributes to both merit review criteria?

> COV Comment: *\"As above, the answer is no for many of the CETP
> proposals but the answer is yes for the STEMTP proposals examined by
> the COV.\"*

**Program response:**

As noted above, the difference between CETP proposals and STEMTP
proposals is explained by the advent of the NSF requirement that panel
summaries address both review criteria, which was implemented in October
1, 2002, whereas the last proposal deadline for CETP proposals was
September, 1999.

Core question (Page 7): Discuss any issues or concerns the COV has
identified with respect to NSF's merit review system.

> COV Comments: *\"The COV found that both consistency among reviewers
> as well as the "baseline" knowledge of what the merit review criteria
> mean continue to be a challenge.* *The latter is especially confusing
> because the NSF documentation is clear on this point. The review
> process relies on a high level of inter-reviewer reliability. Further,
> the review process provides a means for developing and supporting new
> investigators. It is easy to recognize this need as a concern.*
>
> *Beyond raising this issue and concern, the COV suggests the following
> approaches to improve these issues.*

1.  *Create a Fast Lane based system of certification for individuals to
    serve on panel review teams. The certification process could include
    panel member demographics and provide an IRB style tutorial and
    assessment to prepare panel reviewers to respond to the reviews with
    sufficient details to be beneficial to the PIs and the Program
    Officers.*

2.  *Present Panel Review Workshops at national meetings with examples
    of proposals and the difference between excellent, very good, etc.
    reviews (both in terms of the proposal and the degree to which the
    review itself informs PI's and Program Officers). The participants
    in the workshop could turn in a form that Program Officers could use
    to assist them in adding to panels. The COV understands that this is
    already being done to some extent.\"*

> Also, page 9: \"*Clearly, program officers place a high premium on
> selection of reviewers. The overall quality of the reviews is
> apparent. One can notice improvements in the reviews over time (from
> early CETP days to recent STEMTP days). The individual reports in the
> STEMTP and more recent CETP jackets sincerely attempt to address the
> criteria, and the summaries appear to represent consensus. NSF is to
> be commended for these obvious improvements in the selection of
> reviewers and the improvements in the overall quality of the reviews
> themselves.*
>
> *However, the COV did still see some problems with certain reviewers
> and suggests that NSF look into ways of better preparing new reviewers
> (as suggested above).\"*

**Program Response:**

DUE appreciates the COV\'s observations regarding the improved quality
of reviews over time and shares the concern about inter-reviewer
reliability. We strive to include approximately one third new reviewers
in each competition and, understandably, new reviewers may not always
provide high quality reviews. To address this issue, consistent with the
suggestion of the COV, we periodically conduct proposal writing
workshops, which often include writing reviews of sample proposals. This
year, the ATE program provided sample reviews to reviewers as models of
quality reviews and we are exploring the possibility of extending this
approach in other programs, based on feedback from the ATE program.

The COV offers an interesting suggestion regarding the creation of a
FastLane based system for training and certification of reviewers that
merits further discussion. The program will explore the feasibility of
developing such a system.

**A.3 SELECTION OF REVIEWERS**

Core Question (page 8): Did the program make appropriate use of
reviewers to reflect balance among characteristics such as geography,
type of institution, and underrepresented groups?

> COV Comment: *\"The COV sampled numerous CETPs (and follow-ons) and
> STEMTPs. The review teams seem to be balanced by gender, ethnicity,
> and discipline as shown in Table 12. It did appear, however, that an
> overwhelming number of panelists were drawn from baccalaureate
> institutions or universities, not school districts or two-year
> colleges.\"*

**Program response:** An effort is made to include community college
faculty and K-12 teachers and administrators on each panel, while the
reviewer pool also reflects the types of institutions that have
submitted proposals to these two programs. DUE will continue to add
reviewers from these populations.

Core Question (page 9): Did the program recognize and resolve conflicts
of interest when appropriate?

COV Comments:

> *\"NSF might be well served to explore some opportunity to provide
> evidence that the foundation indeed does document how conflict of
> interest is handled. At present no effort is made to indicate when a
> panel member was removed from discussions because of a conflict. At
> best, the Program Officers might indicate the names of individuals
> contacted to serve on panels who were not included because* *of
> potential conflicts. However, it is not clear that anyone would
> benefit from this.\"*

**Program response:**

Every effort is made to exclude potential conflicts of interest at the
time the reviewers are invited and placed on panels. That the COV did
not find evidence of conflicts in the sample of proposals examined is an
indication that such conflicts were avoided prior to the panel meeting.
In those rare cases when reviewers identified conflicts at the panel
meeting and were removed from the discussion, the designation of a
conflict is indicated on the Form 7 and the particular code that appears
next to the reviewer\'s name indicating a conflict was explained to the
COV. The reviewer in conflict explains the nature of the conflict in
FastLane and this is placed in the proposal jacket as the documentation
of a conflict. Furthermore, if a reviewer has a conflict of interest, it
is noted in the review analysis. Reviewers who indicate a conflict at
the time they are invited are not placed on a panel and while this
information is retained during the panel invitation period, program
management agrees with the COV that there is no clear benefit in
retaining this information once the panels have been established or
including information about reviewers who did not serve in the proposal
jackets.

**A.4 RESULTING PORTFOLIO OF AWARDS**

Overall quality of the research and/or education projects supported by
the program.

COV Comment (page 10) :

> *\"The overall quality of the projects seemed appropriate especially
> in how they addressed national and local needs, but research aspects
> were nonexistent in most of the CETP and STEMTP proposals. It appears
> that this is being addressed in the new TPCs.\"*

**Program response:**

CETP and STEMTP were not designed as research programs. The COV is
correct in noting that the TPC program supports research projects.

Core Question (page 12): Does the program portfolio have an appropriate
balance of Institutional Types?

> COV Comment: \"*There was a noticeable lack of HBCUs and HSIs in the
> pool of STEMTP awardees. In addition, 9 out of 37 awards in the CETP
> and STEMTP programs were made to master's degree institutions, and
> only 1 out of 37 went to a bachelor's degree only institution.\"*
>
> Also, COV Comment (page 13): *\"The only major concerns were the
> absence of funding for HBCUs and HSIs, as well as very limited
> participation by two-year colleges in the STEMTP program.\"*

**Program response:** It should be noted that there were only two STEMTP
competitions and the awardees reflected the composition of the proposal
pool. DUE continues to engage in outreach efforts that typically include
HBCUs and HSIs in order to increase awareness of NSF funding
opportunities among minority serving institutions and increase the
number of successful proposals from these institutions. As noted by the
COV in section B1 (NSF OUTCOME GOAL for PEOPLE), a STEMTP award
supporting the MET Summit II conference has an emphasis on participation
by HBCUs and other institutions that produce minority teachers. It is
not surprising that Masters degree institutions, as traditional
producers of teachers, would comprise a larger portion of submitting
institutions and awardees than baccalaureate institutions. With a
success rate of less than 10% for STEMTP program, only the most
competitive proposals can be funded. While no awards were made to
community colleges under the STEMTP program, several awards do include
community college partners. There continues to be an effort to support
teacher preparation at two-year colleges and such projects have been
funded by DUE through the ATE program, which includes a track for
two-year college teacher preparation proposals.

**C1. Program areas in need of improvement or gaps (if any) within
program areas.**

COV Comments (Page 18): *\"To summarize some of the specific findings
above, the COV noted that the following areas could use some
improvement:*

*Better training of reviewers.*

*More two-year college presence in STEMTP*

*More of a HBCU and HSI presence in STEMTP*

*Better documentation of projects that are funded even though their
panel rankings are lower than those of other, non-funded proposals\"*

Program Response: These issues have been addressed in responses to
previous sections.

COV Comments (Page 18): *\"More generally, the education enterprise
needs a progress mechanism. Unlike what happens in traditional research,
each new gain in knowledge in education is not necessarily transmitted
to the common shared vision. All the projects within these two programs
were to develop models. Those models are the outcomes (or products) of
these investments. Where are the models? Can the NSF list them? To say
that CETP and STEMTP were successful implies that there are models.
Where is the plan to bring those models into the common shared vision?
The addition of the CETP follow-on was timely and innovative but will be
useless if there is not a vehicle to add the insights into the field.\"*

Program Response: The models produced by the CETP and STEMTP programs
are disseminated through publications and conferences with the
expectation that the successful models will provide convincing evidence
of their effectiveness and encourage others to adopt or adapt the
models. DUE acknowledges the continuing need for articulating the models
and results that are produced by our programs. Although the shared
vision for these models is reflected in the program solicitations that
guided the proposal development and award decisions and promoted through
annual PI meetings, DUE recognizes the need to facilitate the
dissemination of promising practices developed under these two programs
to the greater STEM community. This need is not unique to CETP and
STEMTP. The issue of building on prior investments is a continuing
concern in DUE and is reflected in the Adaptation and Implementation
track of the Course, Curriculum, and Laboratory Improvement program,
which supports efforts to adapt and implement undergraduate STEM
educational strategies developed elsewhere. This approach that
encourages institutions to adapt or build on work developed by others
provides a mechanism for informing the community of the results of prior
work, which can then lead to new areas of implementation, thereby
advancing the field. By extending the knowledge base on effective
practices, the TPC program is designed to provide a mechanism for
informing the community of strategies worthy of replication or
adaptation through the results of research. It is anticipated that TPC
will provide opportunities for further study of the models and
strategies developed under the CETP and STEMTP programs. With the
increasing emphasis on research-based approaches in DUE and other EHR
solicitations, it is expected that the education enterprise will more
closely mirror the traditional scientific research community with
knowledge \"transmitted to the common shared vision\", as suggested by
the COV.

> COV Comments (page 18): *\"The CETP PI meetings seem to have been
> quite successful in bringing together individuals involved in teacher
> prep. Is there now any occasion where all PIs from teacher education
> programs funded by NSF are brought together?\"*

**Program Response**: DUE recognizes the value of PI conferences that
provide opportunities for networking, professional development, and
knowledge exchange. A PI conference is planned for Spring, 2004 bringing
together CETP, STEMTP and ATE teacher preparation awardees.

C.2 Please provide comments as appropriate on the program's performance
in meeting program-specific goals and objectives that are not covered by
the above questions.

COV Comments (page 8): *\"Both the CETPs and the STEMTPs accomplished
most of their stated goals. However, like nearly all other funded
projects, it is difficult to see evidence of the sustainability or
institutionalization of these efforts. A random search of the websites
of the early CETPs showed that most are out of date.\"*

Program Response: DUE agrees that the issues of sustainability and
institutionalization are extremely important and difficult to document.
For this reason, the issue of sustainability is one component of the
ongoing CETP Core evaluation. Looking beyond websites, to the
institutionalization of courses reformed under CETP and faculty
practices, the 2003 evaluation report cites the establishment and
institutionalization of reformed courses as well as improved
interactions within and among STEM and education faculty and K-12
schools as indicators of the sustained effects of CETP.

Since the two programs under review have been discontinued, the COV was
asked to consider recommendations for the future of STEM teacher
preparation programs and to comment on DUE\'s role in STEM teacher
preparation. These issues were addressed in their comments in the
following section:

**C.3 Please identify agency-wide issues that should be addressed by NSF
to help improve the program\'s performance.**

Regarding the replacement of STEMTP and Teacher Enhancement with the new
Teacher Professional Continuum program, the COV stated (pp 18-19):

*\"The COV has two concerns about this:*

1.  *The teacher enhancement portion of these programs may now overwhelm
    the teacher preparation portion.*

2.  *The important role of DUE in teacher preparation may now be
    marginalized.*

    *More specifically, there is a concern that salient components of
    specific programs currently funded and overseen by DUE be continued
    in the future. Since CETP and STEMTP are being discontinued, a
    potential void has been created. For example, explicit in the CETP
    solicitation is the expectation that teacher preparation programs
    have full institutional support and the concerted effort of many
    stakeholders, including faculty and administration from two-year,
    four-year and research institutions; school districts; the business
    community; and state departments of education. There is good
    evidence that teacher preparation programs in CETP worked to ensure
    these components. Will future TP programs work to ensure these
    collaborations? The CETP solicitation is specific in numerous other
    ways about what the continuum of teacher preparation means, and
    about how projects should be designed to include described features
    and components. It is important that TP program solicitations be
    explicit about the inclusion of (indeed, a focus on) these features,
    that future TP programs be configured to include these components in
    one funded program, and that funding of future TP programs be robust
    enough to support these activities.*

    *DUE program officers are well-qualified and experienced in areas
    related to undergraduate teacher preparation. It is important that
    future TP programs enjoy significant involvement of DUE and other
    well-qualified program officers, who are well-positioned in the
    review process to ensure appropriate attention to undergraduate
    education.*

    *Attention needs to be given to the proper balance for the total EHR
    programs as these DUE programs fade away and new ones arise in other
    divisions. For example, the CETPs and STEMTPs had the "call" to
    engage the faculty from both the arts and sciences and the education
    schools. The MSPs presumably cover this (the TPCs don't seem to) but
    those are much larger grants. What has shifted over the past 10
    years is a drift away from smaller projects and a (seemingly)
    intentional abandonment of preparing the elementary teacher (with
    the exception of the master teacher).\"*

**Program response:** The program appreciates the support the COV has
expressed for continuing involvement of DUE in teacher preparation and
attention to teacher preparation at the undergraduate level, and their
concern as to whether this effort will be sustained as STEMTP and CETP
are discontinued. DUE continues to participate in teacher preparation
efforts as co-manager of the new TPC program with ESIE. It is expected
that the lessons learned through CETP and STEMTP will influence the
development of new programs. The TPC program provides an opportunity for
researchers to study the various models supported under CETP and STEMTP
as well as others and to investigate key issues relevant to the teacher
preparation and teacher enhancement. As such it should further the
knowledge base upon which future efforts are built.

**C.4 Comments on other issues the COV feels are relevant:**

COV Comments (Page 19): *\"Accountability for the success of a program
should be linked to the degree to which the goals of the projects it
supports articulate the goals of program. The accountability for each
individual project's success should then be easily linked to these
initial goals. The annual report and final report systems should require
PI's to address the formative, summative and confirming evidence used to
show the extent to which a goal is reached. One possible way to expedite
this would be to have the initial proposals submitted in data base
bites. Goals statements, for example, could be placed in individual
cells at the time a proposal is submitted. Once a project was funded,
these cells could be coded by a Program Officer to link them to the
specific goals of a program. At the time of an annual report, Fast Lane
would provide the PI's with their stated goals (one-at-a-time). PI's
would be required to address the formative or summative evidence of the
degree to which the project has reached its stated goals. Any
modifications made in the ways in which the project has attempted to
reach the goals should also be detailed.\"*

Program response: PIs are expected to provide formative and summative
evaluation data as part of the annual and final reporting process with
outcomes related to proposed goals and objectives. FastLane reporting is
designed for use throughout NSF and currently provides only limited
program-level modifications. Nevertheless, DUE agrees there is a need
for the type of accountability system described by the COV. A pilot
project for providing end-to-end portfolio management, currently under
development for use in the Informal Science Education program in ESIE,
would provide clear linkages between proposals and outcomes through
modifications in FastLane reporting.