##### CORE QUESTIONS and REPORT TEMPLATE

#####  for 

##### FY 2003 NSF COMMITTEE OF VISITOR (COV) REVIEWS

**Guidance to NSF Staff:** This document includes the FY 2003 set of
Core Questions and the COV Report Template for use by NSF staff when
preparing and conducting COVs during FY 2003. Specific guidance for NSF
staff describing the COV review process is described in Subchapter
300-Committee of Visitors Reviews (NSF Manual 1, Section VIII) that can
be obtained at
[http://www.inside.nsf.gov/od/gpra/.](http://www.inside.nsf.gov/od/gpra/. )

NSF relies on the judgment of external experts to maintain high
standards of program management, to provide advice for continuous
improvement of NSF performance, and to ensure openness to the research
and education community served by the Foundation. Committee of Visitor
(COV) reviews provide NSF with external expert judgments in two areas:
(1) assessments of the quality and integrity of program operations and
program-level technical and managerial matters pertaining to proposal
decisions; and (2) comments on how the outputs and outcomes generated by
awardees have contributed to the attainment of NSF's mission and
strategic outcome goals.

Many of the Core Questions developed for FY 2003 are derived, in part,
from the OMB-approved FY 2003 performance goals and apply to the
portfolio of activities represented in the program(s) under review. The
program(s) under review may include several subactivities as well as
NSF-wide activities. The directorate or division may instruct the COV to
provide answers addressing a cluster or group of programs -- a portfolio
of activities integrated as a whole -- or to provide answers specific to
the subactivities of the program, with the latter requiring more time
but providing more detailed information.

The Division or Directorate may choose to add questions relevant to the
activities under review. NSF staff should work with the COV members in
advance of the meeting to provide them with the report template,
organized background materials, and to identify questions/goals that
apply to the program(s) under review.

**Guidance to the COV:** The COV report should provide a balanced
assessment of NSF's performance in two primary areas: (A) the integrity
and efficiency of the ***processes*** related to proposal review; and
(B) the quality of the ***results*** of NSF's investments in the form of
outputs and outcomes that appear over time. The COV also explores the
relationships between award decisions and program/NSF-wide goals in
order to determine the likelihood that the portfolio will lead to the
desired results in the future. Discussions leading to answers for Part A
of the Core Questions will require study of confidential material such
as declined proposals and reviewer comments. *COV reports should not
contain confidential material or specific information about declined
proposals**.*** Discussions leading to answers for Part B of the Core
Questions will involve study of non-confidential material such as
results of NSF-funded projects. It is important to recognize that the
reports generated by COVs are used in assessing agency progress in order
to meet government-wide performance reporting requirements, and are made
available to the public. Since material from COV reports is used in NSF
performance reports, the COV report may be subject to an audit.

*We encourage COV members to provide comments to NSF on how to improve
in all areas, as well as suggestions for the COV process, format, and
questions.*

##### FY 2003 REPORT TEMPLATE FOR

#####  NSF COMMITTEES OF VISITORS (COVs)

+-----------------------------------------------------------------------+
| ##### Date of COV: May 28-29, 2003                                    |
+-----------------------------------------------------------------------+
| **Program/Cluster: Course Curriculum and Laboratory Improvement**     |
+-----------------------------------------------------------------------+
| **Division: Division of Undergraduate Education**                     |
+-----------------------------------------------------------------------+
| **Directorate: Education and Human Resources**                        |
+-----------------------------------------------------------------------+
| **Number of actions reviewed by COV**[^1]**: Awards:14 (random); 19   |
| (exemplary projects) Declinations:17 (random) Other:**                |
+-----------------------------------------------------------------------+
| **Total number of actions within Program/Cluster/Division during      |
| period being reviewed by COV**[^2]**: 3820 Awards: 1074 Declinations: |
| 2746 Other:**                                                         |
+-----------------------------------------------------------------------+
| **Manner in which reviewed actions were selected:**                   |
|                                                                       |
| **The Chair of the COV selected a random number and jackets were      |
| selected using that formula. All members of the COV reviewed and      |
| discussed the jackets. In addition, 19 exemplary projects were        |
| selected by the CCLI staff for COV review.**                          |
+-----------------------------------------------------------------------+

**PART A. INTEGRITY AND EFFICIENCY OF THE PROGRAM'S PROCESSES AND
MANAGEMENT**

Briefly discuss and provide comments for *each* relevant aspect of the
program\'s review process and management. Comments should be based on a
review of proposal actions (awards, declinations, and withdrawals) that
were *completed within the past three fiscal years*. Provide comments
for *each* program being reviewed and for those questions that are
relevant to the program under review. Quantitative information may be
required for some questions. Constructive comments noting areas in need
of improvement are encouraged. Please do not take time to answer
questions if they do not apply to the program.

A.1 Questions about the quality and effectiveness of the program's use
of merit review procedures. Provide comments in the space below the
question. Discuss areas of concern in the space provided.

+-----------------------------------------------------------+----------+
| **QUALITY AND EFFECTIVENESS OF MERIT REVIEW PROCEDURES**  | **YES,   |
|                                                           | NO,**    |
|                                                           |          |
|                                                           | **DATA   |
|                                                           | NOT      |
|                                                           | AV       |
|                                                           | AILABLE, |
|                                                           | or NOT   |
|                                                           | APPL     |
|                                                           | ICABLE** |
+-----------------------------------------------------------+----------+
| Is the review mechanism appropriate? (panels, ad hoc      | Yes      |
| reviews, site visits)                                     |          |
|                                                           |          |
| Comments: Review panels at NSF are used for the review of |          |
| proposals. In the opinion of the COV this is very         |          |
| appropriate and has produced thorough, balanced and fair  |          |
| reviews.                                                  |          |
|                                                           |          |
| Please refer to proposal "jackets" available during the   |          |
| COV meeting.                                              |          |
+-----------------------------------------------------------+----------+
| Is the review process efficient and effective?            | Yes      |
|                                                           |          |
| Comments: For approximately 70% of the proposals, the     |          |
| review process is completed and decisions communicated to |          |
| applicants within six months. This is the current target  |          |
| for the staff and it is difficult to see how this         |          |
| timetable would be likely to improve without additional   |          |
| staff.                                                    |          |
|                                                           |          |
| Please refer to proposal "jackets" available during the   |          |
| COV meeting.                                              |          |
+-----------------------------------------------------------+----------+
| Are reviews consistent with priorities and criteria       | Yes      |
| stated in the program's solicitations, announcements, and |          |
| guidelines?                                               |          |
|                                                           |          |
| Comments:                                                 |          |
|                                                           |          |
| While there is good evidence that the intellectual        |          |
| quality of the project is always addressed, the broader   |          |
| impact criteria was not consistently addressed by         |          |
| individual reviewers; however, there is evidence in the   |          |
| more recent reviews that review panels and staff have     |          |
| given equal weight to criterion 2-broader impact. We      |          |
| recommend continued efforts to communicate the importance |          |
| of broader impact, and we found ample evidence of the     |          |
| weight that the staff has attached to guiding PIs to the  |          |
| research on student learning.                             |          |
|                                                           |          |
| Please refer to proposal "jackets" available during the   |          |
| COV meeting.                                              |          |
+-----------------------------------------------------------+----------+
| Do the individual reviews (either mail or panel) provide  | Yes      |
| sufficient information for the principal investigator(s)  |          |
| to understand the basis for the reviewer's                |          |
| recommendation?                                           |          |
|                                                           |          |
| Comments:                                                 |          |
|                                                           |          |
| Please refer to proposal "jackets" available during the   |          |
| COV meeting.                                              |          |
+-----------------------------------------------------------+----------+
| Do the panel summaries provide sufficient information for | Yes      |
| the principal investigator(s) to understand the basis for |          |
| the panel recommendation?                                 |          |
|                                                           |          |
| Comments: The COV was impressed with the high quality of  |          |
| the panel summaries. In every instance of the jackets     |          |
| reviewed, members of the COV felt it understood why a     |          |
| particular decision (either negative or positive) was     |          |
| made. The summaries often accomplished a synthetic        |          |
| reading of the proposal and were clearly intended to be   |          |
| useful to the PI, guiding the PI to a resubmission or a   |          |
| rethinking and restructuring of the project. The COV was  |          |
| impressed with the efforts of both reviewers and staff to |          |
| provide constructive criticism as well as guidance.       |          |
|                                                           |          |
| Please refer to proposal "jackets" available during the   |          |
| COV meeting.                                              |          |
+-----------------------------------------------------------+----------+
| Is the documentation for recommendations complete, and    | Yes      |
| does the program officer provide sufficient information   |          |
| and justification for her/his recommendation?             |          |
|                                                           |          |
| Comments: See previous comment.                           |          |
|                                                           |          |
| Please refer to proposal "jackets" available during the   |          |
| COV meeting.                                              |          |
+-----------------------------------------------------------+----------+
| Is the time to decision appropriate?                      | Yes, see |
|                                                           | comment. |
| Comments: An initial target of 90% in six months proved   |          |
| to be overly optimistic. This is clearly a reflection of  |          |
| the limited staff and increased volume. The staff has     |          |
| attempted to address this problem by staggering the due   |          |
| dates for proposals. Recent data indicates that they are  |          |
| processing 70% of the proposals in 6 months and that has  |          |
| become the target.                                        |          |
|                                                           |          |
| Please refer to [Section Two of the CCLI COV              |          |
| binder](../../Section%20Two/Proposal%20Flow%20.xls) for   |          |
| additional information.                                   |          |
+-----------------------------------------------------------+----------+
| Discuss issues identified by the COV concerning the       |          |
| quality and effectiveness of the program's use of merit   |          |
| review procedures:                                        |          |
|                                                           |          |
| We found the current review process to be very strong     |          |
| indeed. The practice of reading all proposals at a        |          |
| meeting in Washington should be continued; however, we    |          |
| suggest that the option of reviewers pre-viewing          |          |
| proposals via the web should be added to the current      |          |
| system.                                                   |          |
|                                                           |          |
| The COV observed that the quality conversation and review |          |
| that resulted from the face-to-face meeting and           |          |
| interaction was superior to that elicited from individual |          |
| reviews. The COV found that panel summaries produced      |          |
| excellent feedback for PIs.                               |          |
+-----------------------------------------------------------+----------+

A.2 Questions concerning the implementation of the NSF Merit Review
Criteria (intellectual merit and broader impacts) by reviewers and
program officers. Provide comments in the space below the question.
Discuss issues or concerns in the space provided.

+----------------------------------------------------------+-----------+
| **IMPLEMENTATION OF NSF MERIT REVIEW CRITERIA**          | **YES,    |
|                                                          | NO,**     |
|                                                          |           |
|                                                          | **DATA    |
|                                                          | NOT       |
|                                                          | A         |
|                                                          | VAILABLE, |
|                                                          | or NOT    |
|                                                          | APP       |
|                                                          | LICABLE** |
+----------------------------------------------------------+-----------+
| Have the individual reviews (either mail or panel)       | Mixed     |
| addressed whether the proposal contributes to both merit |           |
| review criteria?                                         |           |
|                                                          |           |
| Comments:                                                |           |
|                                                          |           |
| The intellectual significance of the project was always  |           |
| addressed, and addressed well by individual reviewers.   |           |
| Individual reviews were inconsistent in their attention  |           |
| to criterion 2. The COV noted improved attention to      |           |
| criterion 2 in the most recent reviews. At least in the  |           |
| CCLI program, the emphasis on broader impact (criterion  |           |
| 2) is finding its way into the review process.           |           |
|                                                          |           |
| Please refer to proposal "jackets" available during the  |           |
| COV meeting.                                             |           |
+----------------------------------------------------------+-----------+
| Have the panel summary reviews addressed whether the     | Yes       |
| proposal contributes to both merit review criteria?      |           |
|                                                          |           |
| Comments:                                                |           |
|                                                          |           |
| In contrast to the individual reviewers we found that    |           |
| the summary reviews paid close attention to both review  |           |
| criteria.                                                |           |
|                                                          |           |
| Please refer to proposal "jackets" available during the  |           |
| COV meeting.                                             |           |
+----------------------------------------------------------+-----------+
| Have the *review analyses* (Form 7s) addressed whether   | Yes       |
| the proposal contributes to both merit review criteria?  |           |
|                                                          |           |
| Comments: Yes. In general the COV was very impressed     |           |
| with the quality of the review analyses. The staff       |           |
| comments provided clear reasoning for the decisions that |           |
| were made and offered positive constructive comments for |           |
| the improvement of proposals.                            |           |
|                                                          |           |
| Please refer to proposal "jackets" available *during the |           |
| COV meeting.*                                            |           |
+----------------------------------------------------------+-----------+
| Discuss any issues or concerns the COV has identified    |           |
| with respect to NSF's merit review system.               |           |
|                                                          |           |
| The COV was impressed with the high quality of the peer  |           |
| review system operating in the CCLI decisions. We find   |           |
| its current structure and organization produces          |           |
| excellent results as measured by the educational         |           |
| materials and products available for our review during   |           |
| the COV meeting.                                         |           |
+----------------------------------------------------------+-----------+

A.3 Questions concerning the selection of reviewers. Provide comments in
the space below the question. Discuss areas of concern in the space
provided.

+---------------------------------------------------------+------------+
| Selection of Reviewers                                  | YES , NO,  |
|                                                         |            |
|                                                         | DATA NOT   |
|                                                         | AVAILABLE, |
|                                                         |            |
|                                                         | or NOT     |
|                                                         | APPLICABLE |
+---------------------------------------------------------+------------+
| Did the program make use of an adequate number of       | Yes        |
| reviewers for a balanced review?                        |            |
|                                                         |            |
| Comments:                                               |            |
|                                                         |            |
| We reviewed the size of the review panel and the        |            |
| composition of the review teams in the summary          |            |
| materials provided for us and in the materials in the   |            |
| binders in the workroom. We found the teams well        |            |
| balanced and able to produce fair and equitable         |            |
| reviews.                                                |            |
|                                                         |            |
| *Please refer to* [*Section Two of the CCLI COV         |            |
| bind                                                    |            |
| er*](../../Section%20Two/CCLI%20Reviewer%20Profile.xls) |            |
| *for additional information*                            |            |
+---------------------------------------------------------+------------+
| Did the program make use of reviewers having            | Yes        |
| appropriate expertise and/or qualifications?            |            |
|                                                         |            |
| Comments:                                               |            |
|                                                         |            |
| Same as above.                                          |            |
|                                                         |            |
| *Please see [Section Two of the CCLI COV                |            |
| bin                                                     |            |
| der](../../Section%20Two/CCLI%20Reviewer%20Profile.xls) |            |
| for additional information*                             |            |
+---------------------------------------------------------+------------+
| Did the program make appropriate use of reviewers to    | Yes        |
| reflect balance among characteristics such as           |            |
| geography, type of institution, and underrepresented    |            |
| groups?                                                 |            |
|                                                         |            |
| Comments:                                               |            |
|                                                         |            |
| While the staff has made excellent efforts to increase  |            |
| the number of women on their review panels, we hope     |            |
| they can do more to increase the numbers of             |            |
| underrepresented minorities. (See our comments at C. 2) |            |
| The COV feels that representatives of underrepresented  |            |
| minority groups can be especially helpful on the review |            |
| of proposals that are targeted at increasing the        |            |
| numbers of underrepresented groups. The experience of   |            |
| minority reviewers might provide important advice and   |            |
| feedback to PIs on these projects. Thus special effort  |            |
| should be made to include members of those              |            |
| underrepresented groups in the review of such           |            |
| proposals.                                              |            |
|                                                         |            |
| *Please see [Section Two of the CCLI COV                |            |
| bin                                                     |            |
| der](../../Section%20Two/CCLI%20Reviewer%20Profile.xls) |            |
| for additional information*                             |            |
+---------------------------------------------------------+------------+
| Did the program recognize and resolve conflicts of      | Yes        |
| interest when appropriate?                              |            |
|                                                         |            |
| Comments:                                               |            |
|                                                         |            |
| It was clear that careful attention was paid to         |            |
| identify conflicts of interest and when they occurred   |            |
| they were addressed.                                    |            |
|                                                         |            |
| Please refer to proposal "jackets" available *during    |            |
| the COV meeting.*                                       |            |
+---------------------------------------------------------+------------+
| Discuss any concerns identified that are relevant to    |            |
| selection of reviewers.                                 |            |
|                                                         |            |
| We think that the CCLI is doing an excellent job of     |            |
| securing high quality reviews and balanced review       |            |
| panels. We encourage the staff to continue to expand    |            |
| the number of underrepresented minorities and women on  |            |
| review panels.                                          |            |
+---------------------------------------------------------+------------+

A.4 Questions concerning the resulting portfolio of awards under review.
Provide comments in the space below the question. Discuss areas of
concern in the space provided.

+------------------------------------------------------+---------------+
| **RESULTING PORTFOLIO OF AWARDS**                    | **A           |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **NOT         |
|                                                      | A             |
|                                                      | PPROPRIATE,** |
|                                                      |               |
|                                                      | **OR DATA NOT |
|                                                      | AVAILABLE**   |
+------------------------------------------------------+---------------+
| Overall quality of the research and/or education     | Appropriate   |
| projects supported by the program.                   |               |
|                                                      |               |
| Comments: It was clear from our review that CCLI     |               |
| receives more high quality proposals than money      |               |
| available to support such projects. It is also clear |               |
| to us that the best proposals are being funded.      |               |
|                                                      |               |
| *Please see [Section Three of the CCLI COV           |               |
| bind                                                 |               |
| er](../../Section%20Three/Awards%20Distribution.xls) |               |
| for additional information*                          |               |
+------------------------------------------------------+---------------+
| Are awards appropriate in size and duration for the  | Appropriate   |
| scope of the projects?                               | with respect  |
|                                                      | to size. The  |
| Comments:                                            | duration of   |
|                                                      | awards is     |
| NSF is certainly getting its money's worth. CCLI is  | often too     |
| funding a large number of creative ideas that often  | short.        |
| serve as catalysts for further work. We found a      |               |
| pattern of flexibility with respect to timing of the |               |
| projects. The COV was concerned about the ability of |               |
| CCLI to demonstrate the strength of the portfolio,   |               |
| given the sporadic submission of annual project      |               |
| reports. Final reports, however, are required for    |               |
| further NSF funding and that seems to the COV an     |               |
| appropriate policy to insure appropriate tracking of |               |
| projects.                                            |               |
|                                                      |               |
| The duration of the awards is often too short for    |               |
| completion of the project. Extensions are obtained   |               |
| with relative ease when no additional costs are      |               |
| incurred and this seems to the COV the correct       |               |
| policy.                                              |               |
|                                                      |               |
| *Please see [Section Three of the CCLI COV           |               |
| bind                                                 |               |
| er](../../Section%20Three/Awards%20Distribution.xls) |               |
| for additional information*                          |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   High Risk Proposals?                             |               |
|                                                      |               |
| Comments: We found the definition of high risk       |               |
| appropriate and we found an acceptable number of     |               |
| awards in this category. We were particularly        |               |
| impressed by the high percentage of first-time grant |               |
| recipients funded by CCLI.                           |               |
|                                                      |               |
| Please refer to proposal "jackets" available during  |               |
| the COV meeting.                                     |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   Multidisciplinary Proposals?                     |               |
|                                                      |               |
| Comments:                                            |               |
|                                                      |               |
| The staff is aware that these are difficult          |               |
| proposals to write and difficult projects to run. We |               |
| think they have appropriately funded these projects  |               |
| so as to encourage PIs to write and submit such      |               |
| proposals.                                           |               |
|                                                      |               |
| *Please see [Section Three of the CCLI COV           |               |
| bind                                                 |               |
| er](../../Section%20Three/Awards%20Distribution.xls) |               |
| for additional information*                          |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   Innovative Proposals?                            |               |
|                                                      |               |
| Comments:                                            |               |
|                                                      |               |
| CCLI has innovation as a key component in each of    |               |
| its three tracks. The program is ideal for           |               |
| individuals with good testable ideas to receive      |               |
| small initial funding to test the idea with a "proof |               |
| of concept" proposal. Moreover, in each of the other |               |
| areas the COV found evidence that innovative ideas   |               |
| and projects were encouraged.                        |               |
|                                                      |               |
| Please refer to proposal "jackets" available during  |               |
| the COV meeting.                                     |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   Funding for centers, groups and awards to        |               |
|     individuals?                                     |               |
|                                                      |               |
| Comments: Most CCLI proposals are from individuals   |               |
| or small groups of PIs. However, there is also       |               |
| evidence of group proposals, and proposals and       |               |
| projects from larger collective entities.            |               |
|                                                      |               |
| *Please see [Section Three of the CCLI COV           |               |
| bind                                                 |               |
| er](../../Section%20Three/Awards%20Distribution.xls) |               |
| for additional information*                          |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   Awards to new investigators?                     |               |
|                                                      |               |
| Comments:                                            |               |
|                                                      |               |
| CCLI awards a very high percent of its grants to new |               |
| investigators. The percent was as high as 45% in     |               |
| 2001.                                                |               |
|                                                      |               |
| *Please see [Section Four of the CCLI COV            |               |
| binder](../.                                         |               |
| ./Section%20Four/PI%20and%20Co-PI%20Information.xls) |               |
| for additional information*                          |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   Geographical distribution of Principal           |               |
|     Investigators?                                   |               |
|                                                      |               |
| Comments:                                            |               |
|                                                      |               |
| *Please see [Section Three of the CCLI COV           |               |
| binder](../../Section%20Three/Awards-Geographic.ppt) |               |
| for additional information*                          |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   Institutional types?                             |               |
|                                                      |               |
| Comments: The COV is especially concerned that there |               |
| be representation of community colleges, HBCUs,      |               |
| MSIs, and liberal arts colleges. CCLI programs are   |               |
| one of the main sources of funding for science       |               |
| education in these institutions and, if anything,    |               |
| they should be over represented among those funded.  |               |
|                                                      |               |
| *Please see [Section Three of the CCLI COV           |               |
| bind                                                 |               |
| er](../../Section%20Three/Awards%20Distribution.xls) |               |
| for additional information*                          |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance of:                                          |               |
|                                                      |               |
| -   Projects that integrate research and education?  |               |
|                                                      |               |
| Comments:                                            |               |
|                                                      |               |
| This is a special focus of the new track             |               |
| organization in CCLI. We are particularly impressed  |               |
| that CCLI has made this an important priority.       |               |
|                                                      |               |
| Please refer to proposal "jackets" available during  |               |
| the COV meeting.                                     |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have an appropriate       | Appropriate   |
| balance:                                             |               |
|                                                      |               |
| -   Across disciplines and subdisciplines of the     |               |
|     activity and of emerging opportunities?          |               |
|                                                      |               |
| Comments: The funded projects are representative of  |               |
| the targeted disciplines and subdisciplines. By its  |               |
| very design, the program is responsive to innovation |               |
| and emerging opportunities.                          |               |
|                                                      |               |
| *Please see [Section Three of the CCLI COV           |               |
| bind                                                 |               |
| er](../../Section%20Three/Awards%20Distribution.xls) |               |
| for additional information*                          |               |
+------------------------------------------------------+---------------+
| Does the program portfolio have appropriate          | Appropriate   |
| participation of underrepresented groups?            |               |
|                                                      |               |
| Comments:                                            |               |
|                                                      |               |
| The COV looked closely at this data and discussed it |               |
| extensively with the staff. Women's representation   |               |
| appears to be solid and growing, but the numbers of  |               |
| African-American and Hispanic applicants is smaller  |               |
| than we or the staff would like to see. Although the |               |
| COV recognizes the challenge of a relatively small   |               |
| number of minority faculty members available in      |               |
| national pool, the number of funded minorities must  |               |
| grow if we are to provide role models to attract     |               |
| minority students to the sciences.                   |               |
|                                                      |               |
| *Please see [Section Four of the CCLI COV            |               |
| binder](../.                                         |               |
| ./Section%20Four/PI%20and%20Co-PI%20Information.xls) |               |
| for additional information*                          |               |
+------------------------------------------------------+---------------+
| Is the program relevant to national priorities,      | Appropriate   |
| agency mission, relevant fields and other customer   |               |
| needs? Include citations of relevant external        |               |
| reports.                                             |               |
|                                                      |               |
| Comments:                                            |               |
|                                                      |               |
| The COV is confident that CCLI's goals and programs  |               |
| address directly national priorities, in particular, |               |
| national concern over the high drop out rate in the  |               |
| STEM fields and declining student interest in        |               |
| pursuing these fields. Please see the report of the  |               |
| National Research Council, Science and Engineering   |               |
| Indicators 2002, especially pp.5-19 and the          |               |
| discussion of funding for undergraduate science      |               |
| education. See as well the recent ACT Report on the  |               |
| lack of interest in STEM courses discussed in        |               |
| *[Maintaining a Strong Engineering                   |               |
| Workforce]{.underline}* and the in-depth study by    |               |
| Elaine Seymour and Nancy M. Hewitt [*Talking About   |               |
| Leaving: Why Undergraduates Leave the                |               |
| Sciences*]{.underline} . The recent report (2002) of |               |
| the AACU *[Greater Expectations: A New Vision of     |               |
| Learning as a Nation Goes to College]{.underline}*,  |               |
| identifies similar problems and issues.              |               |
+------------------------------------------------------+---------------+
| Discuss any concerns identified that are relevant to |               |
| the quality of the projects or the balance of the    |               |
| portfolio.                                           |               |
+------------------------------------------------------+---------------+

A.5 Management of the program under review. Please comment on:

+-----------------------------------------------------------------------+
| Management of the program.                                            |
|                                                                       |
| Comments:                                                             |
|                                                                       |
| The November 2, 2001 Management Plan reflects careful planning for    |
| each of the tracks. We found ample evidence of thoughtful planning.   |
|                                                                       |
| *Please see Section Two of the CCLI COV binder for additional         |
| information*                                                          |
+-----------------------------------------------------------------------+
| Responsiveness of the program to emerging research and education      |
| trends.                                                               |
|                                                                       |
| Comments:                                                             |
|                                                                       |
| > There are several features of the CCLI program that promote NSF\'s  |
| > ability to respond to needs in emerging areas. A base of permanent  |
| > staff members helps institutional memory and assists in recognition |
| > of new ideas. This coupled with an influx of rotating specialists   |
| > provides the correct mix of creative disruption to the process. The |
| > flexibility of the staff in promoting combinations of funding       |
| > possibilities is important, as is their ability to communicate that |
| > flexibility to reviewers and to challenge reviewers to find new     |
| > concepts and approaches to education. The entire peer review        |
| > process seems especially effective at finding both \"gems\" and     |
| > \"flaws\" in the proposals.                                         |
| >                                                                     |
| > NSF\'s newly directed focus in the program announcement encouraging |
| > projects that contribute to the research base on science education  |
| > is especially significant if undergraduate education is to          |
| > progress. If a faculty member is to move from entrenched teaching   |
| > methods to embrace new ideas, he or she must be convinced of the    |
| > validity of those methods. The COV was most impressed with the      |
| > variety and caliber of new ideas and products developed with        |
| > program funds, but more is needed.                                  |
| >                                                                     |
| > The issue is not so much that change is needed in the policies and  |
| > procedures of DUE as it is the need for increased funding of this   |
| > important program to better meet the goals of NSF. The COV notes    |
| > that CCLI is the main avenue of NSF funding for colleges and        |
| > universities that wish to encourage creativity and innovation in    |
| > undergraduate STEM education. When one considers that there are     |
| > 4,720 such institutions, a FY03 budget of \$43.63 million, or under |
| > \$10,000 per institution, appears woefully inadequate. The          |
| > continued erosion of funds from this program is an unacceptable     |
| > trend that risks our nation\'s future.                              |
+-----------------------------------------------------------------------+
| Program planning and prioritization process (internal and external)   |
| that guided the development of the portfolio under review.            |
|                                                                       |
| Comments:                                                             |
|                                                                       |
| The NSF staff has obviously spent considerable time planning the      |
| distribution of its funds for maximal impact. More time consuming     |
| projects, such as National Dissemination proposals, justifiably       |
| receive more funds per project while Adaptation and Implementation    |
| proposals receive less per project with more projects being           |
| supported. The management team considers factors such as relative     |
| percentages of undergraduate student enrollments by discipline and    |
| numbers of proposals per discipline to further break down             |
| allocations. This is done after funds are set aside for special       |
| multidisciplinary projects that the directorate wishes to encourage.  |
| The COV wishes to note, however, that analysis of funds directed      |
| toward faculty development is difficult to judge since all categories |
| of spending have elements of faculty development included and since   |
| the data are not broken down in that manner.                          |
|                                                                       |
| > A continuing issue for the COV is that the number of CCLI proposals |
| > from community colleges continues to be far less than the needs of  |
| > these colleges. Fewer than 10% of all community colleges receive    |
| > federal science and engineering dollars from any source. Since 52%  |
| > of all undergraduates begin their undergraduate education at a      |
| > community college, this issue speaks directly to the NSF strategic  |
| > plan to develop \"a diverse, internationally competitive, and       |
| > globally engaged workforce of scientists, engineers, and            |
| > well-prepared citizens.\" Beginning with the FY03 competition, the  |
| > CCLI program announcement for the EMD/ND track encourages a         |
| > collaboration between community college faculty and others in the   |
| > development of educational materials by permitting a \$25,000       |
| > increase in funding requested for a proof of concept grant to a     |
| > maximum of \$100,000 if community college faculty are included in   |
| > the proposal. This is an excellent incentive, but more should be    |
| > done to build awareness of the CCLI program in community colleges.  |
| > Since many community colleges do not have grants officers, there is |
| > often not a logical contact person in the institution, making this  |
| > a challenging problem indeed.                                       |
+-----------------------------------------------------------------------+
| Discuss any concerns identified that are relevant to the management   |
| of the program.                                                       |
|                                                                       |
| Low staffing levels and the overall inadequacy of the budget are of   |
| concern---at the same time we would not want to see fewer projects    |
| funded.                                                               |
+-----------------------------------------------------------------------+

##### PART B. RESULTS : OUTPUTS AND OUTCOMES OF NSF INVESTMENTS

NSF investments produce results that appear over time. The answers to
questions for this section are to be based on the COV's study of award
results, which are direct and indirect accomplishments of projects
supported by the program. These projects may be currently active or
closed out during the previous three fiscal years. The COV review may
also include consideration of significant impacts and advances that have
developed since the previous COV review and are demonstrably linked to
NSF investments, regardless of when the investments were made.
Incremental progress made on results reported in prior fiscal years may
also be considered.

The following questions are developed using the NSF outcome goals in the
FY 2003 Performance Plan. The COV should look carefully at and comment
on (1) noteworthy achievements of the year based on NSF awards; (2) the
ways in which funded projects have collectively affected progress toward
NSF's mission and strategic outcomes; and (3) expectations for future
performance based on the current set of awards. NSF asks the COV to
provide comments on the degree to which past investments in research and
education have contributed to NSF's progress towards its annual
strategic outcome goals and to its mission:

-   To promote the progress of science.

-   To advance national health, prosperity, and welfare.

-   To secure the national defense.

-   And for other purposes.

B. Please provide comments on the activity as it relates to NSF's
Strategic Outcome Goals. Provide examples of outcomes (nuggets) as
appropriate. Examples should reference the NSF award number, the
Principal Investigator(s) names, and their institution

+-----------------------------------------------------------------------+
| **B.1 [NSF OUTCOME GOAL for PEOPLE]{.underline}: Developing "a        |
| diverse, internationally competitive and globally engaged workforce   |
| of scientists, engineers, and well-prepared citizens."**              |
|                                                                       |
| Comments:                                                             |
|                                                                       |
| The CCLI program funds projects in the areas of science, technology,  |
| engineering, and mathematics (STEM) that lead to the development,     |
| adoption, adaptation, implementation, assessment, and dissemination   |
| of effective educational models, products, and practices. The focus   |
| is on learning by students with diverse backgrounds and career goals. |
| In fiscal year 2002 (Table A14, Annual Report 2002), 56% of student   |
| participants were from underrepresented groups (women, minorities,    |
| persons with disabilities). The following projects further elucidate  |
| CCLI\'s efforts in funding projects that meet this outcome goal:      |
|                                                                       |
| -   Award Number 9952369, Alexander Grushow, Rider University, NMR    |
|     Collaborative Training Partnership: This project targets the      |
|     training of students in the acquisition of raw spectral data and  |
|     analysis of it on a network-capable Fourier transform nuclear     |
|     magnetic resonance (FT-NMR) spectrometer. The project developed   |
|     materials to train students, and it involved a number of two- and |
|     four-year institutions. Because of this project, these            |
|     institutions were able to include in their curriculum             |
|     state-of-the-art experimental techniques and analysis using       |
|     FT-NMR.                                                           |
|                                                                       |
| -   Award Number 9972457, David Gosser, CUNY City College, Peer-Led   |
|     Team-Learning (PLTL): This project developed a model for peer-led |
|     team learning. Student leaders guide activities of small groups   |
|     of students in weekly workshop meetings. In 2001-2002, 14,330     |
|     students used PLTL with 1,292 peer leaders in courses taught by   |
|     137 faculty in 51 different colleges and universities. PLTL is    |
|     applicable across disciplines; currently, it has been developed   |
|     for Biology, Physics, Mathematics, and Chemistry. Through faculty |
|     development and involvement of teaching assistant personnel and   |
|     students, the project has contributed significantly to human      |
|     resource and leadership skills development. Assessment of the     |
|     project has shown positive gains in performance for students in   |
|     PLTL courses as compared with those in non-PLTL classes.          |
+-----------------------------------------------------------------------+
| **B.2 [NSF OUTCOME GOAL for IDEAS]{.underline}: Enabling "discovery   |
| across the frontier of science and engineering, connected to          |
| learning, innovation, and service to society."**                      |
|                                                                       |
| Comments:                                                             |
|                                                                       |
| The CCLI program supports the advancement of knowledge in student     |
| learning and the integration of research advances into courses and    |
| curricula. The program develops and disseminates innovative and       |
| effective instructional methods as well as the application of         |
| technology to support instruction and student learning and            |
| achievement. The Educational Materials Development (EMD) track of     |
| CCLI supports the development and evaluation of innovative ideas to   |
| determine the potential for impact early. The Adaptation and          |
| Implementation (A & I) and National Dissemination (ND) tracks support |
| further research on development and adaptation of innovative          |
| materials and practices and dissemination of these innovations to a   |
| broader audience. The following projects demonstrate CCLI\'s efforts  |
| in funding projects that meet this outcome goal:                      |
|                                                                       |
| -   Award Number 9972437, Brian MacWhinney, Carnegie Mellon           |
|     University, STEP\--A System for Teaching Experimental Psychology: |
|     the focus of this project is in teaching students how to design   |
|     and analyze experiments, preparing them for empirical research in |
|     psychology. The project collected a critical mass number of       |
|     web-based experiments for student research in psychology and      |
|     created tutorials on generating experiments.                      |
|                                                                       |
| -   Award Number 0088986, James Hutchinson, University of Oregon,     |
|     Eugene, An Environmentally-benign (\'Green\') Organic Chemistry   |
|     Curriculum: this project exemplifies the integration of research  |
|     and education. It is developing new organic chemistry laboratory  |
|     experiments that teach the fundamental concepts and skills of     |
|     organic chemistry in a safer manner, while teaching the tools and |
|     strategies of \"green\" chemistry. The materials are being        |
|     disseminated through workshops, a laboratory textbook, a          |
|     searchable web-based database, and other publications. 25 new     |
|     laboratory experiments have been developed and classroom tested.  |
|     These new experiments are safer, reduce waste, and inspire        |
|     students to use chemistry to solve environmental problems.        |
|                                                                       |
| -   Award Number 0040142s, Kim Vandiver and Felice Frankel, MIT,      |
|     Scientific, Imaging and Visual Expression: A New and Powerful     |
|     Approach to Learning Teaching, and Communicating Science: This is |
|     an example of an innovative and high-risk project that has been   |
|     highly successful. The work promotes visual representations of    |
|     scientific data and ideas. The project developed a curriculum and |
|     methodology to make use of the power of imaging to communicate    |
|     science and provide a new channel for scientific creativity. In   |
|     addition, the PI published a quality handbook entitled            |
|     *Envisioning Science.*                                            |
+-----------------------------------------------------------------------+
| **B.3 [OUTCOME GOAL for TOOLS:]{.underline} Providing "broadly        |
| accessible, state-of-the-art and shared research and education        |
| tools."**                                                             |
|                                                                       |
| Comments:                                                             |
|                                                                       |
| The A & I and ND tracks of the CCLI program support the adaptation    |
| and dissemination of effective educational models, products, and      |
| practices. Dissemination is typically accomplished through            |
| presentations, mentoring, workshops and collaborations. There are     |
| other effective avenues for providing broad access including the      |
| digital library and the Project Information Resource System (PIRS).   |
| The Johns Hopkins University\'s project, (Award Number 9972273, under |
| the direction of Wilson Rugh) Interactive Visualization via Java for  |
| Signals and Systems, is an excellent example of a state-of-the-art    |
| education tool that received national recognition and dissemination   |
| to engineering programs.                                              |
|                                                                       |
| This project developed interactive educational materials using        |
| web-based technologies for the typical sophomore/junior course in     |
| Signals and Systems offered in engineering departments. This project  |
| received the 2001 Premier Award for Excellence in Engineering         |
| Education Courseware that is awarded by the National Engineering      |
| /Education Delivery Systems (NEEDS). The award is a national          |
| competition that recognizes high quality, non-commercial courseware   |
| designed to enhance engineering education.                            |
+-----------------------------------------------------------------------+

##### PART C. OTHER TOPICS

C.1 Please comment on any program areas in need of improvement or gaps
(if any) within program areas.

The COV joins the CCLI staff in its concerns about three areas, two in
management (the timeliness of the review process and the limited ability
to monitor projects) and one in award distribution.

The stated goal of responding to 90% of proposals within 6 months (FY 02
Management Plan for CCLI, Nov. 2, 2001 from Susan Hixson, Myles Boylan
and Herb Levitan) is now identified as unrealistic. This is related to a
number of factors---including budgetary constraints and uncertainty, an
increase in programs outside of CCLI for which staff members are
responsible and increased numbers of applications in the three and soon
to be four programs areas of CCLI. The staff has attempted to address
this problem by staggering the due dates for proposals. Recent data
indicates that they are processing 70% of the proposals in 6 months and
that has become the target.

With regard to the monitoring of projects, the current travel budgets of
program officers are very limited. Given the budgetary constraints at
NSF, this will continue to be a challenge. We support the staff's
efforts to use professional meetings for conversations and exchange with
PIs, to continue to think about PI meetings at NSF and to explore
additional mechanisms for electronic communication with PIs. Given the
relatively small size of most grants in this division, monitoring
expenses should be kept to a minimum. Rather we would like to see
ongoing assessment of the adequacy of the questionnaire currently in
use, continued efforts to elicit annual reports from investigators and
the use of final reports from PIs to monitor project outcomes.

The COV is particularly struck by the relatively few proposals from
minority investigators. It notes, however, that the percentage of
proposals nearly matches the percentage of minorities in the faculty
pool. However understandable, the COV feels CCLI must set higher goals
to involve minority role models who can encourage minority students to
enter STEM subjects. Extra efforts should be made to attract minority
reviewers, perhaps from adjunct pools, in order to train new
investigators and encourage new proposals.

C.2 Please provide comments as appropriate on the program's performance
in meeting program-specific goals and objectives that are not covered by
the above questions.

**C.3 Please identify agency-wide issues that should be addressed by NSF
to help improve the program\'s performance.**

We are concerned about the decline in the CCLI budget. Given the health
of the four tracks within CCLI and their responsiveness to perceived and
changing needs within undergraduate education, we are concerned to see
the relatively small budget of CCLI further decreased. We are impressed
with the types of efforts supported by the four strands of CCLI and see
the program's contributions as among those most closely linked to the
over all objectives of the National Science Foundation --the preparation
of scientists and engineers\--and to the objectives of the Division of
Undergraduate Education. Moreover, the materials generated through the
strands of the program have important lessons for all divisions of NSF.
In the COV's estimation, the CCLI program is one of the primary sites
for the integration of research and education at NSF and consequently
one of the most important arenas for achieving its goals. It touches the
lives of many scientists who are strategically placed in the nation's
undergraduate institutions and therefore closely in touch with the
population we must attract and retain in the sciences. It needs to be
adequately funded.

C.4 Please provide comments on any other issues the COV feels are
relevant.

**C.5 NSF would appreciate your comments on how to improve the COV
review process, format and report template.**

> The notebook that was given to the COV was extremely helpful during
> the review. The committee did have trouble navigating through the
> notebook. It would have helped if the Table of Contents had been
> placed at the beginning of the notebook and if dividers or different
> colored paper was used to separate the materials in each of the
> sections.

**SIGNATURE BLOCK:**

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

For the CCLI COV

Susan C. Bourque

Provost and Dean of the Faculty

Smith College

Chair

[^1]: To be provided by NSF staff.

[^2]: To be provided by NSF staff.