##### COMMITTEE OF VISITORS REPORT

**May 9-11, 2001**

**Plant Genome Research Program**

**Division of Biological Infrastructure**

**Directorate for Biological Sciences**

**Number of actions reviewed: 277**

**Signatures:**

**Dr. Andrew Binns, Chair
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_**

**Dr. Burt Ensley, BIO/AC liaison
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_**

**Dr. Winslow Briggs
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_**

**Dr. Phil Bourne
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_**

**Dr. Roger Hangarter
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_**

**Dr. Tim Helentjaris
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_**

**Dr. Diana Horvath
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_**

**Dr. Robert J. Jones
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_**

**Dr. Thomas McCoy
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_**

**Dr. Cindy Putnam-Evans
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_**

**Dr. Ray Rodriguez
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_**

**Dr. Anne Simon
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_**

**Dr. Thomas Slezak
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_**

**Dr. Ann E. Stapleton
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_**

**Dr. Anne Vidaver
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_**

# Committee of Visitors (COV) Executive Summary 

The life sciences are undergoing a revolution in the examination of
fundamental biological questions. Large-scale identification of genes
and gene products involved in important biological processes is leading
to a better understanding of such processes and how they can be utilized
in ways that will help humankind. The National Plant Genome Initiative
recognized that the application of these "genomics" approaches to the
study of plant biology is crucial to our agricultural systems and to the
developing biotechnology industry. In this context the initiative
outlined a series of goals to facilitate the discovery of new knowledge
and application thereof. The Plant Genome Research Program (PGR) in the
Biological Sciences Directorate of the National Science Foundation was
established as part of this initiative and has been in operation since
1998. This Committee of Visitors has been charged with evaluating the
success of the program.

[COV Conclusion:]{.underline} **The committee finds that the PGR has
succeeded in bringing plant and agricultural biology into the genomic
era.** **We find that the processes of reviewing, identifying and
awarding the best proposals are fair and transparently executed. The
program has worked diligently to ensure that awarded projects are
maximally productive and, in the limited period of time in which the
program has existed, project progress has been very satisfactory.** A
wide variety of large-scale projects, most using a series of
interdisciplinary approaches, have been initiated and, assuming success,
will result in the development of ideas, tools and human resources that
could not have otherwise occurred. For the plant and agricultural
community this must be looked upon as nothing short of revolutionary.
Heretofore, the vast majority of projects were on the scale of
individual investigators or, at best, two or three collaborating
laboratories. The capacity to support large-scale projects has revealed
that numerous groups of investigators are eager and able to work
together on problems that had not been usefully explored via other
funding mechanisms. At the same time, we believe the infrastructure
being developed in these larger, collaborative projects will
significantly enhance the capabilities of individual principal
investigator projects, which would otherwise not have been able to
develop and utilize these newer technologies. Beyond the impacts upon
basic research in public institutions, the business of actually applying
the results of this research to improving the human condition is even
more fundamentally affected. Applied researchers will now be able to
address a wider array of complex problems. We expect a dramatic
improvement in the nature of benefits akin to that being seen in human
pharmaceuticals as a result of the Human Genome Project.

[COV Recommendation:]{.underline} **Efforts by the Program Directors to
enhance elements of program management in virtual centers and other
larger collaborations should continue in order to maximize project
success. Specifically, Project Managers may be required for many of the
larger projects.** As might be expected, the transition to large-scale
projects has not been without problems. The committee notes that project
management has been an issue. This problem attracted the immediate
attention of Program Directors and has been dealt with by utilizing
"cooperative research agreements" containing milestones for deliverables
in those cases where appropriate and by instituting External Advisory
Boards that have served to streamline projects and make them more useful
to the broader community. These efforts are laudable but must be
supplemented by implementation of additional appropriate management
strategies.

[COV Recommendation:]{.underline} **The PGR Program Directors need to
move aggressively to institute systematic procedures for all aspects of
data acquisition, archiving, and distribution by appointing a
program-wide bioinformatics coordinator, adopting community-wide
standards, and funding specific proposals or contracts to accomplish
this goal before the next COV review.** A second major issue is in the
general area of bioinformatics. The tremendous amounts of data generated
by these projects must be managed internally, archived centrally,
integrated with the results across projects, accessed by researchers
from around the world and finally converted into useful knowledge at
thousands of distant sites. The COV acknowledges that the Program
Directors recognize and are working diligently on these problems. We
feel this issue represents the most important of our concerns and
propose that significantly more resources and effort be devoted to
obtaining workable solutions.

The Committee also notes other broad areas of concern that need to be
addressed, some at the highest levels of NSF.

[COV Recommendation:]{.underline} **Program Directors should be
commended on previous efforts to increase the diversity of participants
at all levels but are encouraged to be even more aggressive in reminding
participants about the importance and necessity of improving
diversity.**

[COV Recommendation:]{.underline} **NSF should take a leadership
position in providing guidelines for industry/public relationships in
the context of this program. Where possible NSF should encourage the
participation of companies in these programs while maintaining the
standards for open and rapid release of data.**

[COV Recommendation:]{.underline} **While we recognize that shared
funding across international boundaries for this program is problematic,
NSF could take a leadership role in moving the most important of these
efforts to the level of an internationally collaborative effort akin to
the original Arabidopsis Genome Sequencing Project.**

[COV Recommendation:]{.underline} **To facilitate program oversight**,
**NSF should make one of the two Program Directors a permanent position.
This will ensure program continuity while maintaining the advantages of
new insights brought in by rotating external Directors.**

A.  **INTEGRITY AND EFFICIENCY OF THE PROGRAM'S PROCESSES & MANAGEMENT**

```{=html}
<!-- -->
```
1.  **Effectiveness of the program's use of merit review procedures:**

```{=html}
<!-- -->
```
a.  **Overall design, including appropriateness of review mechanism
    (panels, ad hoc reviews, site visits):**

> The overall design of the program's merit review procedures follows
> the standard-peer review process used by NSF. This is a well-designed
> and highly effective procedure. An appropriate number of reviewers
> were used both for the panels and the ad hoc reviews. In addition, on
> the larger grants, site visits are utilized to provide an additional
> level of review and address any of the reviewer's comments. An example
> of the effective use of a reverse site visit is the Osborn proposal
> (DBI-0077774), whereby "the reservations of the reviewers about the
> weaknesses of the project have been clearly addressed by the site
> visit review process and the choice of funding instrument."

b.  **Effectiveness of program's review process:**

> The COV believes the program review process was effective. All
> proposals, whether selected for an award or declined, received an
> adequate level of review. In FY98 there were 4.7 written reviews per
> proposal, with an average of 4.7 written reviews per award and 4.8 per
> decline. In FY99 and FY00 there were 5.7 and 4.6 written reviews per
> proposal respectfully, and 4.5 and 4.1 reviews per award and 4.7 and
> 4.3 reviews per decline. The average return rate of 53% in FY98, 61%
> in FY99 and 49% in FY00 is adequate; however, mechanisms to attempt to
> improve this return rate should be pursued, especially given the
> difficulty of identifying reviewers without conflicts.

c.  **Efficiency; time to decision (please provide numerical data):**

-   Average dwell time (defined as the period from proposal deadline to
    funding decision) in FY00 was 6.1 months with 21% of the proposals
    having a dwell time of less than six months and 79% with a dwell
    time of 6 to 9 months.

-   Average dwell time in FY99 was 6.7 months with 10% of the proposals
    having a dwell time of less than six months and 89% with a dwell
    time of 6 to 9 months.

```{=html}
<!-- -->
```
-   Average dwell time in FY98 was 5.1 months with 96% having a dwell
    time of less than six months and 4% with a dwell time of 6 to 9
    months.

> Keeping the dwell time to six months or less would be desirable;
> however, many of these large proposals require extensive negotiations,
> including site visits and the development of cooperative agreements.
> Considering the complexity of the proposals, the short dwell time is
> to be highly commended.

d.  **Completeness of documentation making recommendations (please
    provide numerical data):**

> Nearly 100% of the proposals had complete documentation regarding the
> recommendation for award or decline. In fact, only one proposal jacket
> was identified as incomplete (Gill, DBI 9872681): it was missing a
> panel review summary. The COV does suggest, however, the inclusion in
> the jacket of a summary from the Program Director justifying the
> funding recommendations made to DD/DBI and AD/BIO. This summary would
> be an internal document that could be shared with a future COV. The
> COV clearly understands and appreciates the necessity of discretion at
> this level of review; however, such a summary would assist future COV
> and Program Officer's understanding of the final funding decisions.

e.  **Consistency with priorities and criteria stated in the program's
    solicitations, announcements, and guidelines:**

> The proposal review criteria are clearly articulated and then followed
> in the review process. The COV recognizes that the first year of the
> program was challenging given the novelty of the program and the
> unprecedented award size. In addition, the FY98 RFP was vague causing
> problems for not just PI's, but for the reviewers and panel members.
> As a result of this lack of clarity the FY98 program received many
> inappropriate proposals, including ones that were nothing more than
> extensions of regular research programs, and proposals using genomic
> approaches but were not actual genomics proposals. The RFP's improved
> immensely in FY99 and FY00. Overall the COV found the panel reviews
> and awards made were consistent with the priorities stated in the
> program's solicitations, announcements and guidelines.

2.  **The program's use of the NSF Merit Review Criteria (intellectual
    merit and broader impacts)**:

```{=html}
<!-- -->
```
a.  **Performance Goal:** **Implementation of Merit Review Criteria by
    Reviewers:**

```{=html}
<!-- -->
```
1)  Did reviewers adequately address the [intellectual
    merit]{.underline} criterion in their reviews?

> The reviewers took this part of the merit review very seriously. The
> reviews were very thorough and paid attention to the quality of the
> science, project management, whether or not the proposed work was
> innovative, potential problems and how the proposer suggested solving
> potential problems.

2)  Did reviewers adequately address the [broader impacts]{.underline}
    criterion in their reviews?

> The reviewers [did not]{.underline} adequately address the broader
> impacts criterion (criterion 2). We found that only between 25% and
> 35% of reviewers addressed criterion 2 in their reviews. Our
> observations mirror a recent Science article citing a report on the
> NSF review process where "Some 73% (of reviewers) disregard criterion
> 2 \[social relevance\] altogether or simply merge it into scientific
> merit...". One example of this is found in the proposal by Ventner
> (DBI-9813392). The intellectual merit of this proposal was thoroughly
> reviewed but the reviewers failed to address criterion 2 beyond
> training graduate students. A second example is found in the proposal
> by Preuss (DBI-9872641). This is an example of a very successful
> proposal. Intellectual merit was thoroughly evaluated by the
> reviewers; however, criterion 2 was not addressed adequately by any of
> the reviewers. A typical example is found in the proposal by Sederoff
> (DBI-9975806). Of 5 reviewer's comments, 1 reviewer made an attempt to
> address criterion 2, 1 reviewer cited the commercial impact of the
> proposal, and the other 3 reviewers failed to address criterion 2 at
> all. Another typical example is found in Wessler's proposal
> (DBI-0077709). Of 4 ad hoc reviewers, 1 reviewer cited teaching and
> training as components of a broader impact, the other three reviewers
> failed to mention criterion 2. Teaching and training per se do not
> constitute "broader impacts."

b.  **Performance Goal: Implementation of Merit Review Criteria by
    Program Officers:**

```{=html}
<!-- -->
```
1)  Did program officers adequately address the [intellectual
    merit]{.underline} criterion in their decisions?

> The program directors were very diligent in addressing the
> intellectual merit criterion in all proposals. For example, in the
> proposal by Collmer (DBI-0077622), the program director addressed the
> following intellectual merit points: the quality of the research
> group, the diversity of the group in terms of expertise and stage of
> their careers, and the importance of sequencing this species of plant
> pathogen both due to its economic impact and as a model for other
> studies. The program director also noted the hypothesis testing in the
> proposal and the value of the potential outcomes of the research.
> Another excellent example is found in the proposal by Estelle
> (DBI-0077769) where the program director discussed the novelty and
> scope of the proposal and noted that it was highly ambitious,
> discussed the qualifications of the PIs and the likelihood of their
> success. The program director also noted the geographical diversity
> and complementary skills of the research team.
>
> The program director should be commended in actions on the proposals
> by Baker and Ronning which were merged into one proposal with Baker as
> PI (DBI-9975866). The program director recognized that combining them
> into a single entity would strengthen the proposals and that research
> objectives of both proposals were met in manner that reduced the
> overall cost.

2)  Did program officers adequately address the [broader
    impacts]{.underline} criterion in their decisions?

> For the most part all of the program directors made an attempt to
> address the broader impacts of the proposed work, but some were more
> detailed in their assessment than others. An excellent example was
> found in the proposal by Sederoff (DBI-9976836). The program director
> carefully addressed criterion 2 and noted that the PI and Co-PI
> participated in the BRITE program and that minority students were
> hired into this lab every year. An example of an adequate review by a
> program director is found in the proposal by Wessler (DBI-0077709).
> The program director cited the training of 4 undergraduate students as
> part of the broader impact, and the value of the genome data that
> would be generated as a broader impact of the proposed work.
>
> A rare example of the program director inadequately addressing the
> broader impacts issue was found in the proposal by Doebly
> (DBI-9872631). The program director's treatment of criterion 2 was
> limited to citing the training aspects addressed in the site-visit
> report.

c.  **Discuss any concerns the COV has with respect to NSF's merit
    review system.**

> The committee noted that most of the PIs made an effort to address
> criterion 2 in their proposals. For example, Wessler (DBI-0077709) did
> a good job of addressing criterion 2, having an informatics component,
> dissemination of the data to be generated by the project, training of
> 4 undergraduate students through fellowships and post-doctoral and
> graduate student training. At the same time, the panel summary
> concerning criterion 2 for this same proposal was weak. The panel only
> cited dissemination of information about rice.
>
> Reviewers should emphasize criterion 2, including diversity, impacts
> to society as a whole, quality of life, improving scientific skills of
> the workforce. The committee felt that one possible approach to the
> problem would be modification of the Fast Lane template for reviewers
> such that a greater emphasis is placed on criterion 2 and it becomes
> very clear that this item must be included in any review that is
> submitted. It might be possible to construct the template such that
> the review cannot be sent until this criterion is addressed.
>
> Although not specifically requested, we would like to comment on the
> panel summary treatment of criterion 2. The panel summaries in general
> were not as thorough in addressing the broader impacts criterion as
> either the PI's or program director's reviews. Most panel summaries
> were short and tended to emphasize the value and dissemination of the
> data, and the importance of understanding mechanisms as a broader
> impact of the proposed research.

3.  Reviewer selection (please provide numerical data):

```{=html}
<!-- -->
```
a.  **Use of adequate number for balanced review:**

> Because many of the proposals have numerous PIs and CoPIs, one of the
> most difficult jobs for the Program Directors has been in finding
> reviewers that do not have conflicts of interest. Nevertheless, The
> program has done a good job of finding adequate numbers of qualified
> reviewers. On average for the three years of the program, there have
> been about 5 reviews for each proposal. The COV feels that a larger
> number of ad hoc reviews would be beneficial but given the conflict of
> interest issue, the COV appreciates the difficulty of obtaining more.
> We do not know how these numbers compare with other NSF programs.

b.  **Use of reviewers having appropriate expertise/qualifications:**

> The reviewers represent a wide range of expertise and qualifications
> that is appropriate for the extreme breadth of topics covered by the
> program.

c.  **Use of reviewers reflecting balance among characteristics such as
    geography, type of institution, and underrepresented groups:**

> Specific data were not available for the diversity of reviewers with
> regard to underrepresented groups. However, a survey of proposal
> jackets showed that the reviewer pool is well balanced, especially
> with respect to geographical distribution, which covers the entire USA
> and included many international experts. In addition, the reviewer
> pool included institutions of nearly all types.

d.  **As appropriate, recognition and resolution of conflicts of
    interest by NSF staff and adequacy of documentation justifying
    actions taken:**

> The COV felt that the NSF staff has done an excellent job of
> identifying conflicts of interest. Overall, the conflicts have been
> well explained and appropriate actions taken. For example, in the
> Raikhel proposal (DBI 9975815), after reviews were received, the
> program officer recognized that one reviewer had a conflict and took
> appropriate actions to disallow that review.

4.  **Resulting portfolio of awards (please provide numerical data):**

```{=html}
<!-- -->
```
a.  **Overall quality of science/engineering**:

The COV was very impressed with the overall quality of the award
portfolio, especially in the second and third years of the program. In
functional genomics, proposals from Preuss (DBI-9872641), Estelle
(DBI-0077769), Doebley (DBI-9872631), Lam (DBI-9872636), and Raikhel
(DBI-9975815) were among a number of outstanding grants receiving
funding from the program. Many of the funded proposals were to develop
services for use by the plant biology community. These proposals
involved extensive organization, and include collaborations across
disciplines and institutions that previously could not have existed
without the Plant Genome Program. Green (DBI-9872638), Wing
(DBI-9872676), Bohnert (DBI-9813360), Ecker (DBI-9975718), Coe
(DBI-9872655), Walbot (DBI-9872657), and Tanksley (DBI-9872617) head
some centers that have outstanding potential. While we recognize that
some centers got off to a slow start because of the managerial problems,
the oversight by the program directors and external advisory boards is
commendable and has clearly improved the centers' subsequent progress in
achieving defined benchmarks. Having these big center grants as
cooperative research agreements is viewed positively.

> One area for which the COV voiced significant concern is in the area
> of bioinformatics. The COV felt that easy to use, community-accessible
> databases and data management tools were not always available. The
> current program directors clearly recognize this as a problem and have
> been working with awardees in the following ways: funding supplements
> to improve the bioinformatics component of the project, e.g., Guerinot
> (DBI-0077378); discussion of the problems at awardees meetings;
> running joint workshops for the bioinformatic specialists from the
> various centers to define solutions to issues such as linkages,
> accessibility, and standardization of databases. Nevertheless, further
> steps must be taken to resolve these issues, and the COV outlines
> further thoughts about this in sections 7 and 9.

b.  **Appropriateness of award scope, size, and duration:**

The COV felt that the breadth of research represented by the awards was
appropriate to fulfill the goals of the Plant Genome Initiative. The
awards represented a wide variety of both important crops and model
plant systems. The approaches ranged from basic genome mapping to novel
technologies for gene identification and functional analysis. Given the
complexities of the projects, the size and duration of the awards (2 to
5 years) was viewed as appropriate. It was noted that the start up time
and management issues of the large centers means that initial grants
need at least 4 years of effort to fully judge the progress for future
renewals. The COV was impressed that the panels and program directors
were able to identify specific components of some proposals for funding.
This allowed the support of some high-risk areas through the use of
proof of concept awards. Examples of this include Lam (DBI-9872636),
Estelle (DBI-0077769), Carpita (DBI-0077719), Dinesh-Kumar
(DBI-0077510), Pruess (DBI-9872641) and Schnable (DBI-9975868). Funding
of the large centers may need to be enhanced to fully achieve the
benefits to the community of the proposed research, particularly in
terms of database and management issues.

c.  **Effective identification of and support for emerging
    opportunities:**

In a global context, the entire program was viewed as an exceptional
emerging opportunity to develop tools and biological and human resources
for use by a large and growing plant community in the USA and around the
world. These resources should make possible rapid advances in areas
ranging from the characterization of basic plant processes to the
improvement of crop productivity. In order to exploit these resources
fully, the issues of data base management and utilization discussed
above must be more fully addressed. In addition, the COV feels that the
program directors should continue their successful practice of
identifying components of proposals that warrant special consideration.

> Dr. Jane Silverthorne, Program Director, provided the committee with
> an analysis of the 3-year investment in the Plant Genome Research
> Program. The committee noted and agrees with her assessment of program
> highlights, which include:

-   EST and other mapping technologies for maize, wheat, soybean,
    cotton, and pine

-   Comparative mapping for cereal genomes, and between Arabidopsis and
    tomato

-   Functional genomics expression studies

-   Technology advancement projects in a variety of areas:

> Gene Mapping
>
> Tracking chromosome position using "beacons"
>
> Gene regulation
>
> Centromere analysis and artificial chromosomes
>
> Improved surface technologies for microarrays
>
> Rapid selection of gene point mutants
>
> Gene silencing
>
> Distribution of genome data and resources
>
> Outreach and Training
>
> The COV appreciated the thorough self evaluation performed by the
> Program Director. We recommend that a formal 'self-evaluation' such as
> this be a standard part of the COV process.

d.  **Appropriate attention to maintaining openness in the system, for
    example, through the support of new investigators:**

> There is significant evidence of new investigator inclusion and
> support during the first years of the program (15 of 76 awards). This
> generally comes in the form of sub-awards from large center grants.
> While the COV does not encourage new or inexperienced PIs to lead
> large center proposals, the COV would like to encourage the
> appropriate inclusion of these investigators and that they are given
> recognition commensurate with their contributions. This is
> particularly important when these investigators are evaluated for
> promotion at academic institutions. The COV commends the program
> directors for trying to broaden the applicant pool for the program by
> making informal presentations at meetings, site visits and other
> venues.

e.  **Evidence that proposers have addressed the integration of research
    and education in proposals:**

The participation of graduate students and postdocs is well documented
in proposals. In addition to the outreach activities described below
(5A), several proposals have included workshops on tool utilization
\[e.g., Green (DBI-9872638), Coe (DBI-9872655), Walbot (DBI-9872657) and
Paterson (DBI-9872649)\]. The COV encourages all large center projects
to hold public workshops in tool use and information to assist the
community in understanding and making use of the resources generated.

f.  **Evidence of increased numbers of applications from
    underrepresented groups:**

There is no evidence of an increase in numbers of applications from
underrepresented groups. The COV feels that this reflects more on the
limited number of scientists in the field and not on the review process.
However, the committee urges that projects, particularly the large
center projects, include investigators from underrepresented
institutions. Such interactions could include summer paid sabbaticals at
the center for the investigator prior to formal involvement as a co-PI.
NSF should ensure that there are sufficient incentives to make such
contacts.

g.  **Balance of projects characterized as**

-   High-risk

-   Multidisciplinary

-   Innovative

The COV felt that there was a good mix of proposals funded to date. As
stated above, the COV encourages the program directors to continue to
identify and fund high-risk proposals.

##### RESULTS: OUTPUTS AND OUTCOMES OF NSF INVESTMENTS

# Strategic Outcome Goals: For each of the strategic outcome goals listed below, has the program demonstrated success in achieving the outcome goal? The program is [successful]{.underline} when, *in the aggregate*, results reported in the period demonstrate significant achievement in one or more of the listed indicators. 

# 

# Provide specific NSF-supported examples which demonstrate your judgement, and explain why they are relevant or important to the outcome goal. If performance is not successful, comment on the steps that the program should take to improve performance. It is important to note if a specific outcome goal indicator is not relevant to the program and provide a brief explanation.

5.  **PEOPLE Strategic Outcome Goal: Development of a diverse,
    internationally-competitive and globally-engaged workforce of
    scientists, engineers, and well-prepared citizens.**

**Please place an X in the box next to one of the following
statements:**

**X** The program met the strategic outcome goal.

[or]{.smallcaps}

> The program did not meet the strategic outcome goal.

a.  **Improved mathematics, science, and technology skills for U.S.
    students at the K-12 level:**

At least eight programs have made modest contributions involving high
school students and/or teachers. For example, Sederoff (DBI 9975806) has
developed a summer course for high school teachers. Ronning (DBI
9813392) has summer internships both for teachers and students. Both Coe
(DBI 9872655) and Cook (DBI 9872664) regularly recruit high school
students to work in their laboratories as does Schwartz (DBI 9975606)
who actively recruits minority students into his laboratory. While we
realize that this is not a major thrust of the program, we find it
commendable that a number of PI's do make an effort to connect with high
school level and contribute at that educational level.

b.  **Improved mathematics, science, and technology skills for citizens
    of all ages, so that they can be competitive in a technological
    society:**

> Clearly the program is contributing to a major information base for
> all citizens. We interpret this question to address education at the
> undergraduate, graduate, and postdoctoral levels. The program is
> definitely having a major positive educational impact especially at
> the graduate and postdoctoral levels. For example, the Bohnert program
> (DBI 9813360) reports participation by 26 postdoctoral fellows, 22
> graduate students, and 61 undergraduates over the duration of the
> grant. However, the COV noted that some of the large service-oriented
> programs are not always suitable to produce plant biology thesis
> projects or postdoctoral research projects, though they could do so in
> the area of informatics.

c.  **A science and technology and instructional workforce that reflects
    America\'s diversity:**

> Several projects have developed outreach programs with historically
> black colleges. For example, Delmer (DBI 9872649) obtained a
> supplement to support one graduate student and two undergraduates from
> Alabama A&M in her program. Both Gelvin (DBI 9975715) AND Vodkin (DBI
> 9872565) have established relationships with Tuskegee Institute, the
> former having positions for two undergraduates in his laboratory, and
> the latter having reserved slots in a workshop for Tuskegee students.
> The students with Gelvin are encouraged to take selected mutants back
> to Tuskegee for further study during the academic year. As mentioned
> above, Schwartz (DBI 9975606) actively recruits students from
> underrepresented groups into his laboratory. Finally, Paterson has a
> total of nine African American scientists working on his project. Four
> of the students were identified through a special recruitment and
> retention program at the University of Georgia. The COV commends these
> activities but feels that NSF should encourage PI's to establish such
> relationships with minority students in their own institutions. Many
> of these institutions have developed their own programs to identify
> interested students from underrepresented groups. It is also important
> the PI's to establish individual relationships by going to
> institutions that serve underrepresented groups as a way of bringing
> students and faculty into their laboratories and initiate a two-way
> exchange.

d.  **Globally engaged science and engineering professionals who are
    among the best in the world**:

> The PI's are among the top plant scientists in the world and attract
> an international cadre of graduate students, postdoctoral fellows and
> visiting faculty members to their projects. The Plant Genome
> Initiative also supports and encourages participation in international
> meetings by providing travel support for younger investigators. For
> example, Qualset (DBI 9975989) has been involved in establishing
> workshops at the International Triticeae Mapping Initiative, involving
> a variety of genome and EST projects related to the cereal crops
> especially projects in Europe and Australia. Both Michelmore (DBI
> 9975971) and Paterson (DBI 9872649) have collaborations with the
> International Rice Research Institute in the Philippines. Paterson
> also has formal collaborations with groups in Japan and Brazil. Cook
> (DBI 9872664) heads up a project that has become the hub for research
> on the model legume system *Medicago truncatula*. This activity
> includes collaborations at CNRS/INRA in France, the Agricultural
> University of Wageningen, the Netherlands. Gelvin's project (DBI
> 9975715) involves active collaboration with scientists at the
> Friedrich Miescher Institute in Basel, Switzerland.

e.  **A public that is provided access to the processes and benefits of
    science and engineering research and education**:

Major data bases are being made available to the public and the private
sector on the internet, one way to make the processes and benefits of
the science available to the public. A number of PI's in addition are
making efforts to speak in public on issues such as GMO's, informing
legislators and journalists to their own activities and activities of
others in the field. Pruess (DBI 9872641) worked with the American
Society of Cell Biologists to develop a policy statement on GMO's. She
also has been active in speaking to members of congress on GMO issues.
Lightfoot (DBI 9872635) has been extremely active in interviews with
journalists, has written popular articles about the GMO controversy from
a European viewpoint, and has hosted laboratory tours from a number of
visiting politicians and administrators. The COV feels that it is
critical for PI's to continue and expand this kind of public outreach.
We realize that it is difficult for many scientists, but it is essential
if we expect to keep the public informed about the course of the science
and the risks and benefits to be derived from it. Finally, society will
clearly benefit as the databases grow to the point where companies can
effectively use this information to produce new products and processes.

6.  **IDEAS Strategic Outcome Goal: Enabling discovery across the
    frontier of science and engineering, connected to learning,
    innovation and service to society.**

**Please place an X in the box next to one of the following
statements:**

**X** The program met the strategic outcome goal.

[or]{.smallcaps}

> The program did not meet the strategic outcome goal.

a.  **A robust and growing fundamental knowledge base that enhances
    progress in all science and engineering areas including the science
    of learning**:

> The Committee found the contributions to the knowledge base to be
> perhaps the area of greatest success in the Plant Genome Project. Many
> of the awards made contributions in the form of direct data in
> databases or tools for the development of databases. Examples are:
> Bohnert (DBI-9813360), Coe (DBI-9872655), Preuss (DBI-9872641),
> Tanksley (DBI-9872617), Vodkin (DBI-9872565), Walbot (DBI-9872657),
> Wilkins (DBI-9872630), Baker (DBI-9975866), Ecker (DBI-9975718),
> Qualset (DBI-9975989), Sederoff (DBI-9975806), and Collmer
> (DBI-0077622).
>
> One example of particular significance is the Bohnert project
> (DBI-9813360), which has been identifying genes involved in plant
> stress using several approaches. So far, they have identified greater
> than 12,000 ESTs for 4 species, generated \> 130,000 tagged
> Arabidopsis lines, sent some 15,000 seed pools to the Ohio stock
> center, developed over 40 knockout strains of Saccharomyces, and
> developed microarrays of up to 2500 elements for 5 species. A second
> strong example is the Ronning (DBI-9813392) and Tanksley (DBI-9872617)
> projects. These have addressed questions in tomato functional genomics
> and produced over 60,000 tomato ESTs (available on the web at TIGR and
> Genbank), \>15 cDNA libraries (widely distributed), produced some 6
> publications and book sections, and developed 2 web sites focusing on
> solanaceous genomes and microarrays.

b.  **Discoveries that advance the frontiers of science, engineering,
    and technology:**

> Several awards used particularly innovative approaches to answer
> fundamental biological questions. Specific examples include the Lam
> project (DBI-9872636, 0077617), which has demonstrated important
> information about movements of chromosomes by labeling each with a
> different colored molecular beacon. In the Guerinot project
> (DBI-0077378) rapid spectroscopic methods are used to analyze large
> numbers of essential and toxic metal ions. These data will yield
> information on both plant and human nutrition. Henikoff (DBI-0077737)
> developed the "TILLING" method to introduce point mutations in any
> gene in any plant, and ultimately in other organisms.

c.  **Partnerships connecting discovery to innovation, learning, and
    societal advancement:**

> The Committee noted several awardees made use of partnerships with
> research institutes, companies, and commodity groups. Examples:
> Tanksley (DBI-9872617) and Baker (DBI-9975866) partnered with The
> Institute for Genome Research (TIGR) for sequencing tomato and potato
> ESTs; Vodkin (DBI-9872565) partnered with a soybean commodity group in
> association with a project on functional genomics in soybean; Wing
> (DBI-9872676) has received funding for contract research from Novartis
> to produce and distribute BAC libraries for rice; and Ecker
> (DBI-9975718) is using Affymetrix chips for Arabidopsis gene
> expression analysis.
>
> One issue that concerned the COV was the logistics of interactions of
> awardees with commercial partners. In some jackets, we observed that,
> in the absence of a clearly stated policy by the NSF, problems could
> arise when PIs interacted with companies. For example, Coe
> (DBI-9872655) lost valuable time by initially working with a BAC
> library developed by one of the PIs in a project with Novartis, only
> to find that they were unable to negotiate an agreement to use it in
> the scope of the PGR grant. In another example (Lemieux (DBI-9872677),
> the PI described in two progress reports the intention to contract
> with DNA Landmarks, resulting in a call from the Program Director to
> request a copy of the agreement between the two parties. Although the
> deal had fallen though, this example demonstrates that there was not
> an understanding by the PI to stay in contact with the NSF in regard
> to commercial partners.
>
> There should also be a clear statement about expected time of release
> for database information. Moreover a mechanism for monitoring these
> releases should be agreed upon. Timely release of data should be a
> condition of further funding.
>
> Finally, the COV recommends that the NSF should be active in
> approaching companies to release database information to the public.
> This is a particularly tough issue, but one that requires attention,
> discussion and resolution. Members of the highest levels of the NSF
> may have success in working out such agreements with the industry.

d.  **Research and education processes that are synergistic**:

This criterion is quite vague. The COV noted the examples that pertained
quite specifically to special efforts combining research and education
such as Bohnert (DBI-9813360), who has trained some 25 postdocs, 22 grad
students, and 61 undergraduates in genomics. Another is Wing
(DBI-9872676) who has instructed a range of high school students and
visiting scientists in their BAC library production process. See also
section 5e of this COV report.

7.  **TOOLS Strategic Outcome Goal: Providing broadly accessible,
    state-of-the-art information-bases and shared research and education
    tools.**

**Please place an X in the box next to one of the following
statements:**

**X** The program met the strategic outcome goal.

[or]{.smallcaps}

> The program did not meet the strategic outcome goal.

a.  **Shared use platforms, facilities, instruments, and databases that
    enable discovery:**

> The COV had multiple interpretations of the meanings of "tools" and
> "shared use platforms" in the context of this question. Under a broad
> definition of both terms, which would include things like BAC
> libraries, microarrays, ESTs, etc. in addition to computational tools,
> we found that the program definitely meets the criteria. The COV also
> notes that there has been relatively little time for many of the
> projects to reach milestones because funding started in the very
> recent past.
>
> A few major examples of funded projects that have outputs enabling
> discovery include:

-   Tanksley (9872617) Comparative mapping database.

-   Walbot (DBI-9872657) Novel algorithm for spliced DNA alignment, made
    available on public Web site

-   Wing (9872676) BAC libraries used by multiple researchers world-wide

> We anticipate many more such examples as projects initiating in
> FY99/00 begin to approach the point where data becomes available and
> made public.
>
> Under a tighter definition of information-bases and computational
> tools the outlook is not as optimistic. We note that nearly all the
> funded projects have sub-critical-mass funding for necessary
> informatics, and as a direct result, there is inadequate planning and
> engineering to produce readily usable databases and tools. Moreover,
> the calls for proposals appear to ignore the need for standards across
> the portfolio. (Examples: the Gene Ontology program has spent years
> defining standards to be used for annotating gene expression data. The
> Plant Genome Program should be joining this effort to extend it to
> plants, and requiring all projects to use this ontology as appropriate
> within their local systems.) Another recent example is the Distributed
> Annotation System (DAS, URL: <http://stein.cshl.org/DAS>) that is an
> emerging standard for community annotation, which the Plant Genome
> Program should consider encouraging. Finally, it is insufficient to
> simply make data available with an arbitrary ad hoc Web interface. A
> set of well-defined standard URLs for querying Web-enabled databases
> needs to be developed, returning information using appropriate XML
> definitions. These URLs should be supported by every Plant Genome
> Project with data on a Web site, allowing both human and
> program-driven data extraction.
>
> The COV also has concerns about data and tools developed under PGR
> funding having continuing availability after project funding ends. A
> central data warehouse and tool repository would accomplish this, and
> also allow for a high degree of information integration across
> species. This would be a major, multi-year project focused on
> providing these services, and not on novel bioinformatics research.
>
> **b. Shared use platforms, facilities, instruments, and databases that
> enhance the productivity and effectiveness of the science and
> engineering workforce**:
>
> The COV had trouble distinguishing between "enabling discovery" and
> "enhancing the productivity and effectiveness of the science and
> engineering workforce." One interpretation of the distinction led to
> the view that getting raw data out on the Web, even with differing
> syntax and semantics, would enable discovery but did not lead to
> effectiveness (i.e., expert users could sort out the problems and make
> discoveries by mining the raw data.) Others argued that it would have
> to be good tools that made sense out of the raw data that would lead
> to discovery.
>
> Whatever the case, we feel that the answer to 7(a) above applies
> equally here. Improving the bioinformatics components of individual
> projects, and providing better oversight of the entire bioinformatics
> portfolio will certainly improve both the discovery and effectiveness
> components of these projects.
>
> **c. Networking and connectivity that takes full advantage of the
> Internet and makes SMET information available to all citizens:**
>
> The web sites developed with Plant Genome Program funding are open and
> available to the world's citizens. The committee believes that the
> Program should continue to lead the way in educating citizens about
> the value of the research it funds. We note that a small
> proof-of-concept grant to Gribskov (0086252) is pointing in the right
> direction of providing a more centralized Web-based "portal" to data
> and tools for Program participants and the public. This effort, if it
> shows success, should be competed openly and fully reviewed before
> being scaled up.

d.  **Information and policy analyses that contribute to the effective
    use of science and engineering resources:**

> The COV was confused as to the meaning and scope of this question. We
> will take advantage of the inscrutability of this question to note the
> following:

-   The Interagency Working Group (IWG) that led to the formation of the
    Plant Genome Project is itself the best example of how information
    and policy led to a much more effective use of science and
    engineering resources.

-   The COV was pleased to note that the use of advisory committees for
    the larger projects appeared to be extremely successful. Examples of
    this include:

-   Jorgensen (DBI-9975930)

-   Coe (DBI-9872655)

-   Baker (DBI-9975866)

-   Qualset (DBI-9975989)

8.  **Areas of Emphasis:** For [each]{.underline} relevant area of
    emphasis shown, do the investments and available results demonstrate
    the likelihood of strong performance in the future? **Explain and
    provide NSF-supported examples that relate to or demonstrate the
    relevant strategic outcomes.**

```{=html}
<!-- -->
```
a.  **Strategic Outcome: People**

**Please place an X in the box next to one of the following
statements:**

**X** The program achieved the strategic outcome.

[or]{.smallcaps}

> The program did not meet the strategic outcome.

-   K-12 systemic activities (n/a -- Although education is integral to
    the majority of awards in this program, there was no call for
    systemic K-12 activities.)

-   Enhancing instructional workforce/professional development:

```{=html}
<!-- -->
```
-   Centers for Learning and Teaching (CLT) (n/a -- Although education
    is integral to a majority of awards in this program, there was no
    call for forming CLTs)

-   Graduate Teaching Fellows in K-12 Education (n/a -- Although
    graduate education is important in this program and many awards
    support it, the Graduate Teaching Fellows in K-12 Education are not
    part of this program.)

```{=html}
<!-- -->
```
-   Broadening participation:

```{=html}
<!-- -->
```
-   Tribal Colleges:

> The Guerinot (DBI-0077378) proposal has an ongoing recruitment effort
> for Native Americans. In particular, one of the PI's on that project,
> Dr. Salt at Northern Arizona University, coordinates with regional
> Native American recruitment groups to identify appropriate summer
> students. These students will go to one of the other campuses for a
> defined period in the summers, starting with Summer 2001. These
> students are to be sent in groups of at least two per campus and
> training will be coordinated with other undergraduate summer programs
> and support programs to make the experience away from home less
> intimidating and more positive. The host PIs plan to visit NAU to give
> a seminar and meet with potential students ahead of time so that the
> transition will be easier. The possibility of having the workshops at
> NAU to make it more inclusive for these undergraduate students was
> discussed at the site visit and will be implemented. Dartmouth has the
> largest Native American student body of the Ivy League schools and a
> Native American house and advisor on campus. The participation of
> Northern Arizona University brings a unique opportunity to broaden
> participation of Native American students in a plant genome project.

-   Partnerships for Innovation (PFI) (n/a -- There are many
    interactions in these awards between industry and academic
    institutions, however, no awards were made under the PFI
    announcement.)

```{=html}
<!-- -->
```
-   Addressing near-term workforce needs:

```{=html}
<!-- -->
```
-   Advanced Technological Workforce program (ATE) (n/a -- Training in
    new technologies is a very important part of the Plant Genome
    program, but no awards were made under the ATE announcement.)

b.  **Strategic Outcome: Ideas**

**Please place an X in the box next to one of the following
statements:**

**X** The program achieved the strategic outcome.

[or]{.smallcaps}

> The program did not meet the strategic outcome.

-   Appropriate Balance of Portfolio (high-risk, multidisciplinary, or
    innovative research) for the program:

> The COV generally felt there was a good balance to the portfolio but
> more high-risk projects should be considered and funding.

-   Investment in three initiatives:

```{=html}
<!-- -->
```
-   Information Technology Research (ITR)

> Many of the proposals called for addressing needs in bioinformatics.
> Presently we observed no particular interactions with ITR initiatives.
> We encourage PGR to seek suitable opportunities to co-fund projects
> with ITR initiatives.

-   Nanoscale Science and Engineering (n/a -- The PGR program focuses on
    plant genomics research.)

-   Biocomplexity in the Environment

> As noted in Dr. Jane Silverthorne's review, PGR projects contribute
> indirectly to the goals of the Biocomplexity Initiative by generating
> tools to understand the ecological roles of plants and also by
> developing data on a substantial range of plant species.

-   Investments in non-initiative fundamental research:

```{=html}
<!-- -->
```
-   Mathematical Sciences Research

> Generally this area was not a main component of PGR awards, with the
> single exception of Osborne (DBI-0077774), who uses statistical
> approaches to address functional genomics in polyploids. The COV did
> feel that this is an area that should be looked at, possibly in the
> conjunction with other funding programs.

-   Functional Genomics

> Many awards address the topic of functional genomics through the use
> of microarrays and mutations (a table of these projects can be found
> in Dr. Silverthorne's Selected Program Highlights). The COV suggests
> that additional award areas should be considered, such as funding
> projects addressing proteomics and metabolomics. We learned that, to
> date, the proposals received in these areas were not suitable for
> funding. However a greater number of incoming grant proposals focus on
> these techniques and high quality projects in these areas should be
> forthcoming. This is a goal that the next COV should monitor.

-   Cognitive neuroscience (n/a -- The PGR program focuses on plant
    research.)

c.  **Strategic Outcome: Tools**

**Please place an X in the box next to one of the following
statements:**

**X** The program achieved the strategic outcome.

[or]{.smallcaps}

> The program did not meet the strategic outcome.

-   Investments in Major Research Equipment:

```{=html}
<!-- -->
```
-   Terascale Computing System

> The COV would like to explicitly point out that although the PGR
> program focuses on plant research, it is only a short time before the
> PGR program will need to use terascale computing resources to solve
> its problems. This will certainly be the case for various types of
> comparative genomics analyses (i.e., data mining across gene
> expression data from multiple plant species) that will exceed the
> available computing capacity. It is also the case that many of the
> smaller awardees will not have access to this scale of computing
> power. The PGR program needs to anticipate providing access to
> required computing and storage resources in a cost-effective way
> within 2 years, either as a user-facility or through some form of an
> outsourcing contract.
>
> Making this scale of computing power available to applicants will
> allow them to think outside of the constraints that their current
> computing environments place upon them.

-   Continuing investments:

```{=html}
<!-- -->
```
-   Major Research Instrumentation Program (MRI):

> The Plant Genome Research program has an appropriate level of
> participation in the MRI; one project (DBI-9977155) was co-funded by
> the PGR Venture fund. The availability of funds for future co-funding
> will ensure the future potential for strong performance.

-   Science and Engineering Information/reports/databases:

> Investments in databases have made substantial progress toward meeting
> the strategic goals between 1998 and 2000. Examples of NSF-funded
> projects with strong potential for future growth are those projects
> which use TIGR resources, especially the tomato and potato projects
> (Tanksley 9872617 and Baker 9975866). The second set of projects which
> leverage existing databases are those collaborating with USDA-ARS
> (Qualset 9975989 and Sederoff 9975806). Future potential is best for
> those projects that build on current strengths and those which have
> acknowledged the complexity and expense of database management.
>
> \- New types of scientific databases and tools for using them:
>
> As discussed above, the PGR Program will need to find ways of
> attracting innovative bioinformatics proposals. It will then face the
> problem of finding reviewers who are competent enough to judge them on
> their computational merits in addition to their biological relevance.
> Finally, the PGR leadership must reach the point where the need to
> elevate bioinformatics from a minor support effort to major projects
> becomes obvious.

**\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\--**

9.  **Please comment on program areas that the COV believes need
    improvement.**

As noted in the Executive Summary, The COV commends the program for the
high quality of the science and the profound impact that this program is
currently having and will have on the plant and agricultural sciences.
The COV did identify some program areas that could be improved. A brief
description of these recommendations follows, several of which have been
discussed above.

-   The COV felt that one of the primary areas for improvement in the
    Plant Genome Research Program is in the general area of
    bioinformatics. Paramount amongst the concerns is the recognition
    that genomic projects in general generate extremely large amounts of
    data that must be archived, accessed, and interpreted by the
    original researchers. Subsequently, this information must be
    accessible by users beyond the original community who need this
    information in the context of their own research questions. There
    was a realization that even large virtual centers are unable to
    solve the issues involved with general interoperability of genomic
    databases. NSF needs to take a lead role in conjunction with
    awardees to develop the requirements and eventual structure such
    that the entire plant research community can take advantage of the
    fruits of this work. The development of integrated databases may
    need to be addressed in terms of specific grants and/or contracts
    specified by the NSF PGP in consultation with the research
    community. Given that very few major bioinformatics proposals have
    yet been received, and none funded, little progress has been made in
    this area. Proposals in this area are confounded by the breadth and
    novelty of the field resulting in difficulties for the program to
    receive informed reviews. Given there has not been any specific
    funding in this area, it is also not surprising that training of
    students and post-doctorals has lagged and that the infusion of true
    computational expertise into these programs has been slight.
    Co-review and co-funding with other DBI programs may be the best
    solution.

```{=html}
<!-- -->
```
-   The COV believes that project management is a critical element to
    the success of the large distributed projects. We recommend that
    supplemental funds be provided to existing grants to hire a high
    level program manager responsible for hiring, coordination within
    the project and coordination between projects. Future proposals
    should include detailed management plans, including a project
    manager for the large center grants, and details of how multiple
    sites will be coordinated.

```{=html}
<!-- -->
```
-   The Program must work diligently to increase the use of tools, data,
    and results across the plant research community beyond the original
    awardees. The power and impact of the supported projects will be
    profound transformation only if the entire research community is
    able to build, collectively, upon the results developed by these
    approaches. There have been some initial stumbling blocks, for
    example in the need (in some instances) to charge for the
    distribution of biologicals, clones, microarrays, transgenics, etc.
    While this may seem appropriate, it is clear that this can be a
    significant impediment to users beyond the direct NSF awardees.
    Policies in this area should be continually re-examined for novel
    solutions, such as out-sourcing to companies when practical.
    Examples would be the use of Research Genetics to distribute clones
    or Rosetta/Agilent to distribute microarrays. A related issue is
    that the majority of the awardees were funded primarily due to their
    research capability and few have experience in providing long-term
    service to their communities via distribution of biologicals and
    documentation on the scale generated in these projects. NSF will
    have to monitor the proficiency of virtual center efforts to
    distribute the results of their efforts and offer recommendations
    where appropriate. The use of advisory groups for major projects
    consisting in many cases of eventual users in this role is a logical
    approach.

```{=html}
<!-- -->
```
-   The NSF could provide leadership with the project involvement with
    companies, which has been a thorny issue in the past and is likely
    to continue so for some time. PI's and their home institutions are
    often inexperienced with setting up interactions with companies. NSF
    policies with regard to the PGP are vague as to whether these are
    encouraged, discouraged, or indifferent. At the same time, companies
    contain an immense set of data and talent in this regard, that if
    martialed to support public awardees, the impact could be magnified
    several fold. General policies need to be hammered out with regard
    to ownership of intellectual property and the timely release of data
    and results. Perhaps approaching the corporate community as a group
    to resolve this issue is one area where NSF could provide leadership
    for public institutions and individual PI's. The COV also noted that
    while qualified reviewers were at a premium, due both to expertise
    and lack of conflicts, corporate scientists offer a ready pool that
    might alleviate some aspects of this problem.

```{=html}
<!-- -->
```
-   More emphasis on increasing the involvement of underrepresented
    groups is necessary, especially with the large genomics proposals.
    The COV recommends that PI's on large proposals work with their
    respective institution's minority recruitment offices to identify a
    plan and a timeline for increasing the involvement of
    underrepresented groups. The Program should require a specific
    timeline as part of the RFP. If an award is made then the Program
    Officer must insist on a detailed assessment of the accomplishments
    in increasing involvement of underrepresented groups as part of the
    annual report. If the assessment is inadequate then the Program
    Officer should not continue the award.

```{=html}
<!-- -->
```
-   An increased investment in high-risk projects in areas like
    proteomics and metabolomics is desirable, especially since no
    proposals have yet been funded in this area. The program was
    successful with the investment in the Lam proposal in FY98
    (DBI 9872636) and several other high-risk investments look
    promising. The COV feels that since the Program's initial focus was
    on large genomics proposals it is now appropriate for increased
    investments in smaller high-risk projects of shorter duration (less
    than \$1 million for no more than two years). Therefore, the COV
    recommends that the Program emphasize its intent to fund more
    high-risk proposals (if they are meritorious) in the next RFP.

-   The COV recommends a strong statement regarding data access be
    included in the RFP. The RFP should prominently state that data
    generated must be made available to the public as soon as possible.
    Non-compliance with this requirement will result in termination of
    award.

```{=html}
<!-- -->
```
-   Review teams should be augmented with people chosen to focus on the
    "broader impact" aspects; too many reviewers appear to be solely
    focused on the science and uninterested in Criterion 2. The need for
    PI's, reviewers, and panel members to address Criterion 2 needs to
    be strongly emphasized and enforced. Also annual reports should not
    be approved unless Criterion 2 is adequately addressed.

```{=html}
<!-- -->
```
-   As part of the documentation process, the COV recommends the Program
    Director prepare a brief written summary justifying her/his funding
    recommendations to the DD/DBI and the AD/BIO. This summary would be
    an internal document that would be available to future COV. The COV
    clearly recognizes and understands the need for discretion and the
    Program Director must be able to fine-tune final funding decisions;
    however, this summary would help clarify the final funding
    decisions.

```{=html}
<!-- -->
```
-   An additional concern relates to the fact that international
    collaborations in many of the supported areas would be an obvious
    benefit, but the mechanisms of achieving such collaborations are not
    clear. Extensive discussions with international science agencies on
    developing such mechanisms are likely to be difficult but viewed as
    necessary to maximize the success of many projects. While we
    recognize that shared funding across international boundaries for
    this program is problematic, NSF could take a leadership role in
    moving these efforts to the level of an internationally
    collaborative effort akin to the original Arabidopsis Genome
    Sequencing Project.

```{=html}
<!-- -->
```
-   Finally, given the complexity of this program the COV recommends
    that NSF pursue hiring a permanent Program Director for the Plant
    Genome Research Program. An additional Program Director serving in a
    rotating mode must also be utilized.

10. Comment as appropriate on the program's performance in meeting
    program-specific goals and objectives (non-GPRA outcomes).

The program goals and objectives are stated as supporting "projects that
make significant contributions to our understanding of plant genome
structure and function. Emphasis is placed on plants of economic
importance and appropriate model systems that link genome information to
function at the cellular, organismal, and evolutionary levels". These
goals are most certainly being met. Numerous examples are cited
throughout this COV report and the PGR annual reports.

11. **NSF would appreciate your feedback on the COV review process,
    format and core questions.**

```{=html}
<!-- -->
```
1.  The COV commends the NSF staff for the high quality of the materials
    prepared for us, and for the outstanding assistance staff members
    provided during the meeting. The letter of invitation to COV members
    included passwords allowing access to annual reports, the list of
    questions to be answered by the COV, and to general NSF information
    such as annual reports, lists of grantees and charge to the COV. Of
    particular value to the COV in their review process was the material
    submitted by the Program Officers citing direct examples relating to
    questions in the report template. This saved an invaluable effort by
    the COV in allowing them to focus their time on more substantive
    discussions rather than trying to run down examples.

2.  The COV notes that the questions in the review template were often
    redundant and/or ambiguous. This had the effect of hindering
    assessment of the broader issues.

3.  This is the first COV for the Plant Genome Program. To assist future
    COVs, we describe our process:

The COV, consisting of 15 members, met at the NSF from May 9, 2001 to
May 11, 2001 to conduct the first on-site, external review of the Plant
Genome Research (PGR) Program. The review began with an overview of the
program by the DBI Director, Dr. Machi Dilworth. This overview included
a brief history of the program, organizational structure, program goals
and highlights, evaluation mechanism and charge to the COV.

The COV was then provided with extensive program documentation which
included compilations of grant awards and declinations, statistics on
reviewers, dwell times, geographical distributions of awards and award
rates over the past three years. Files, organized with helpful tabs
(jackets) for all 1998-2000 awarded proposals (58) and declinations (52)
were provided to the COV for review. The COV was divided into three
subgroups of five members and carefully reviewed the jackets. The group
then considered the various questions posed in the review assignment and
split into smaller groups, each addressing one or more questions.
Periodically, the group reconvened as a whole for further discussion.
Findings were summarized using the "FY2001 Report Outline for the NSF
COV". In the course of developing the review summary, the COV had the
opportunity to meet with the Program Directors, Program staff and Dr.
Mary Clutter, Assistant Director of the Directorate of Biological
Science. These meetings provided the COV with clarification on NSF
policies and valuable background information the the Program and awarded
projects.

The COV FY2001 Report was finalized and signed on May 11, 2001.

## Plant Genome Research Program -- Specific Questions

1.  **How can the program encourage participation of a broader community
    of scientists in the program?**

PI's should continue and expand current outreach efforts. NSF should
require and not just strongly encourage a letter of intent to encourage
appropriate outreach to be considered by PI's sufficiently early in the
process. The NSF could then supply case studies where different kinds of
outreach programs have been successful. For example, see sections 5c and
8a for examples of involvement of underrepresented groups; other
examples should include awards in which large and small institutions
enter into partnerships and examples in which younger investigators who
probably shouldn't try to manage a full program are included as co-PI's
on larger proposals.

2.  **How can we encourage and facilitate training of informatics
    personnel?**

It is necessary to recognize that there are different types of
informatics personnel needed by these projects and thus would be trained
in different areas. The scope of training needs to range from coding
scientific application programs, to software engineering, to algorithm
development and implementation, to all areas of bioinformatics and
computational biology. Given the overall lack of qualified teachers in
these areas in general, and as faculty associated with the funded
projects in particular, this represents a significant challenge.
Moreover, the training of application programmers and software engineers
may not be the primary mandate of the organization in which the award
resides.

Where qualified faculty exist on these projects they should be
encouraged to develop curricula that revolve around informatics aspects
of the plant genome project. This would both train and draw new students
to the projects. The NSF could assist with course development
supplements. Bohnert (DBI-9813360) has developed a bioinformatics course
that is an example of this approach.

Further supplements are desirable to encourage the hiring of student
interns to work with existing informatics personnel in funded
laboratories.

Another option is funding focused hands-on workshops that span centers
and projects and which provide specific training in aspects of plant
genome informatics. Analogies to the Cold Spring Harbor (CSH) workshops
come to mind.

The traditional approach is to accelerate conventional undergraduate,
graduate, and post-doctoral training. Certainly, there is an increasing
pool of students interested in this area and supplements should be
available to encourage training on these projects. At the post-doctoral
level a different model could apply. A hybrid model might be one in
which the individual is paid more like a software engineer and performs
that task for the project but at the same time is engaged in
postgraduate work on a part time basis. At the end of this training the
individual could pursue either a research or computer-oriented career.
Consider the analogy to a MD/PhD program, albeit at a postdoctoral
level.

3.  **How can we encourage and facilitate better coordination between
    projects focused on related problems or systems? The PIs interact at
    the annual meeting but appear to work in a vacuum for the rest of
    the year. We have established a web site to coordinate activities
    but it is just getting under way.**

The COV believes that the key to better coordination between centers is
better communication between centers. Communication channels need to be
established between comparable personnel at each site. The suggested
assignment of project managers to each large project should have the
coordination of this communication effort as part of their job
description. Without the availability of a project manager the PIs
should assume the responsibility for this task. They would then foster
communication between PIs as well as recognize developments relevant to
more than their own project and communicate such information. They would
also be responsible for coordinating workshops on topics relevant to
large areas of the PGR.

The Web site is to be commended, but at the same time relevant
information needs to be actively distributed (pushed) to PIs, co-PIs and
their staff. This could include information on papers published through
the program, upcoming workshops, availability of reagents and so forth,
in other words, a point of central communication. It is desirable that
this central site be competed, rather than awarded as a supplement to a
single center. Such proposals should include support for at least one
FTE and show innovation in methods of information dissemination. It is
recommended that NSF staff review the progress of NIH glue projects that
are large-scale efforts that include fostering communication between a
group of researchers working on a common project and note their
successes and failures.

Finally projects should be encouraged to use the latest in communication
technology under advice of their respective informatics staff to foster
communications. This should include videoconferencing, Web broadcasts,
and emerging collaborative technologies for holding virtual meetings.