**Executive Summary: August 2004 EAR/IF Committee of Visitors Report**

The Committee of Visitors (COV) for the Earth Sciences Instrumentation
and Facilities (EAR/IF) program reviewed proposal jackets, analyzed IF
data, spoke with Program Officers (POs) and administrators, and
considered several national publications concerning research and
infrastructure. We were very impressed that the POs represent a
hard-working, hands-on, fair, and efficient team dedicated to funding
and managing a large and diverse set of instrumentation and facilities
projects for quality research and education in the Earth Sciences. The
COV was pleased that the POs consider both Intellectual Merit and
Broader Impacts of their awards, and that they work to increase the
diversity of the population they serve. IF is performing a critical and
important function congruent with national priorities and agency
mission.

Of particular note is the complexity of the IF portfolio, with grants
ranging from \$13 million to \<\$20,000 per annum. The COV inferred that
facility support (large awards) are generally serving the geophysical
community well, based upon the observation that fewer geophysical
requests are made for small equipment grants; similarly, we inferred
that equipment acquisitions/instrument development/technician support
(smaller awards) are generally serving the geochemical community well,
based upon the large number of such geochemical requests, their
diversity, and the relatively high success rates for these proposals
(\~50% or higher). Our assessment of geophysical and geochemical awards
lumped chemical disciplines (e.g. geomicrobiology, petrology,
mineralogy, environmental chemistry) into \"geochemistry\" and physical
disciplines (e.g. quantitative structure, geomorphology, geophysics)
into \"geophysics\". Success rates have been maintained through
successful efforts by the POs to leverage money from outside of EAR/IF
for IF awards.

The COV makes suggestions within the report body about these
programmatic aspects:

• Disciplinary representation on panels may be improved so as to be
congruent with the

distribution of proposals across disciplines.

• Reviewers, panels, and occasionally the POs can continue to improve
the treatment of

the two review criteria (Intellectual Merit and Broader Impacts).

• Site visits, often used for large facilities, are deemed particularly
important for

evaluation of large expenditures of facility funding and should be
documented in site

visit reports.

• Because it is difficult for a COV to assess the appropriateness of
award balance among

facilities and smaller awards given the short assessment period, other
mechanisms to

allow longer term analysis of this balance should be considered.

• Especially where disciplinary communities are not receiving award
support for large

facilities, POs should continue to promote mechanisms to educate these
communities

about such opportunities, and should consider funding support for
technical personnel

to help multi-user facilities grow.

• Current and future budget pressures should be exerted across the
board, so that

budgetary changes do not disproportionately affect one type of awardee
more than

another: this is particularly important inasmuch as different
communities are largely

served by the large and small awards.

• Money leveraged from outside of IF (e.g. from the Major Research
Instrumentation

program) has allowed maintenance of high success rates, but has left
EAR/IF vulnerable

should outside funding decrease in the future.

> • The COV process forces a small committee up a steep learning curve
> in a short time
>
> and mechanisms to achieve more effective reviews might be considered.

##### FY 2004 REPORT TEMPLATE FOR

#####  NSF COMMITTEES OF VISITORS (COVs)

+-----------------------------------------------------------------------+
| ##### Date of COV: 18-20 August 2004                                  |
+-----------------------------------------------------------------------+
| **Program/Cluster:** Instrumentation & Facilities Program (IF)        |
+-----------------------------------------------------------------------+
| **Division:** Earth Sciences (EAR)                                    |
+-----------------------------------------------------------------------+
| **Directorate:** Geosciences (GEO)                                    |
+-----------------------------------------------------------------------+
| **Number of actions reviewed by COV**[^1]**: Awards: Declinations:    |
| Other:**                                                              |
+-----------------------------------------------------------------------+
| **Total number of actions within Program/Cluster/Division during      |
| period being reviewed by COV**[^2]**: Awards: Declinations: Other:**  |
+-----------------------------------------------------------------------+
| **Manner in which reviewed actions were selected: Randomly or with    |
| criteria as explained below**                                         |
+-----------------------------------------------------------------------+

**PART A - INTEGRITY AND EFFICIENCY OF THE PROGRAM'S PROCESSES AND
MANAGEMENT**

Briefly discuss and provide comments for *each* relevant aspect of the
program\'s review process and management. Comments should be based on a
review of proposal actions (awards, declinations, and withdrawals) that
were *completed within the past three fiscal years*. Provide comments
for *each* program being reviewed and for those questions that are
relevant to the program under review. Quantitative information may be
required for some questions. Constructive comments noting areas in need
of improvement are encouraged. Please do not take time to answer
questions if they do not apply to the program.

**[QUALITY AND EFFECTIVENESS OF MERIT REVIEW PROCEDURES]{.underline}**

**A1a** - **Is the review mechanism appropriate? (panels, ad hoc
reviews, site visits)**

The review process for IF proposals includes some combination of one or
more of the following processes: program officer (PO) review, *ad hoc*
review, panel review, and site visit review. Given the diversity of
proposals funded by the IF program, this variety of review mechanisms is
appropriate.

In all cases, PO review of proposals occurred and was deemed to be
outstanding by the COV. In most cases PO review was also well
documented. We discuss *ad hoc*, panel, and site review in the following
sections.

*Ad hoc Reviews*: The COV recognizes that obtaining a sufficient number
of *ad hoc* reviews for every proposal is difficult and we applaud the
EAR/IF program officers' efforts to obtain the greatest number of *ad
hoc* reviews possible for every proposal. Over the three-year review
period for this COV (2001-2003), 396 proposals were handled by EAR/IF.
The number of *ad hoc* reviews for these proposals is summarized in
Table 1.

**Table 1. Number of *ad hoc* reviews per proposal, 2001-2003**

> **No. of *Ad Hoc* Number of No. of *Ad Hoc* Number of**
>
> **Reviews Proposals Reviews Proposals**
>
> 2 5 9 17
>
> 3 47 10 6
>
> 4 53 11 2
>
> 5 85 12 3
>
> 6 65 13 2
>
> 7 60 14 1
>
> 8 48 21 1
>
> 31 1

The COV is impressed that 87% of proposals received 4 or more external
reviews: given the difficulties of garnering reviews from an
increasingly busy scientific professoriate, this is an impressive return
rate, and is undoubtedly indicative of good management on the part of
the POs.

The COV was concerned, however, with the fact that the *ad hoc* reviews
often did not address the Broader Impacts category adequately, despite
specific requests for discussion of this proposal criterion in the
EAR/IF solicitation. As discussed more fully in A2, we recommend that
the Program Officers consider whether they could include specific
examples of how Broader Impacts should be mentioned in individual
reviews within the formal request for *ad hoc* reviews, so as to
stimulate better response on this criterion within *ad hoc* reviews.

*Panels*: After three days of discussions and examination of proposal
jackets, the COV developed the strong impression that the IF program is
a diverse portfolio of projects that serves the EAR community well and
is managed by a professional, efficient, and hard-working pair of
program officers (POs).

It is very apparent that the clientele of EAR/IF represents a wide range
of scientific disciplines, with each proposal cycle representing the
potential to yield applications more weighted toward one area of
specialization than another. This presents a challenge for POs to
appoint a balanced and representative panel to meet the demands of any
one round. Specifically, it is a challenge for the program directors to
maintain the right balance of reviewers and panelists, as it is not
possible to anticipate in advance all the fields that will be
represented in the next submission round. It appears to this COV that
flexibility in the composition of the panel is required in order to
properly and fairly rate the proposals for a given round. Furthermore,
the mechanisms used to replace panel members as they rotate off may be
perceived as prone to introducing a slight bias generating an "old
boys/girls" network (professor nominates his/her student, colleagues,
etc.). In other words, the model of how the EAR/IF panel is constructed
needs to be re-thought. For example, special emphasis panels could be
encouraged for *each* multi-user facility proposal renewal; for other
proposals, the panel make-up should be flexible to reflect the types of
proposals received for a given round. This and other ideas will be
further explored in A5b, Part B, and elsewhere.

*Site Visits*: The types of activities covered during a typical site
visit were not readily apparent in the jackets for the two National
Multi-user Facilities we examined. We therefore invited the program
officers to our session and discussed with them the details of what
constitutes a site visit and how often they are conducted. We learned
that these occur during panel meetings, which are located at the site of
a specific multi-user facility. For example, sites visits conducted
during 2001-2003 are summarized in Table 2.

**Table 2. Site visits 2001-2003**

# **Year Site or Facility**

> 2001 Tucson, AZ: AMS Facility
>
> 2001 Stony Brook, NY: COMPRES Facility
>
> 2002 Minneapolis, MN: IRM Facility
>
> 2002 W. Lafayette, IN: PRIME lab
>
> 2003 Austin, TX: X-ray Computed Tomography
>
> 2003 Los Angeles, CA: National Ion Microprobe

Unfortunately, there are no formal reports of these site visits,
although data may be gleaned from panel summaries. The COV unanimously
agreed that a formal report for each site visit should be drafted and
this report should specifically note what happened during the site visit
(with respect to the facility), and what conclusions were reached
concerning successes and deficiencies for each particular multi-user
facility. In this way, a detailed and specific record is maintained.

Site visits are critical to understand how resources are being invested,
to recognize strategic needs, and to streamline budgets in a rational,
well thought-out manner. As funded multi-user facilities form a large
proportion of the EAR/IF budget, this COV feels that it is essential
that more time be allowed the POs to handle thorough site visits or
reverse site visits, often with independent panels of experts for the
larger facilities. These site visits might or might not include the POs.

*PO decisions*. In the set of proposals this COV examined, it was most
common that the panel, *ad hoc* reviewers, and the Program Officers all
agreed on the proposal ratings. The COV was thus satisfied that
decisions made by PO officers was largely driven by the panel and *ad
hoc* reviewers, as would be hoped.

However, the COV also observed cases when PO decisions were incongruent
with panel and/or *ad hoc* reviews. For example, in a couple of
examples, the Panel and *ad hoc* reviews both suggested the proposal
should not be funded, but the Program Officers funded the proposal,
albeit at a reduced level from the original request. The COV observed
that in such cases, the reasons for funding (e.g. availability of money
ear-marked for undergraduate institutions, PO desire to broaden
diversity of the population of institutions or PIs funded, importance
and ephemeral nature of a given data-set, etc.) were deemed appropriate
for funding action. It was gratifying to see that documentation was
available to support these decisions, that such discrepancies were
anomalies rather than high-frequency events, and that such decisions
generally resulted in small awards comprising only an extremely small
fraction of the overall IF budget.

**A1b - Is the review process efficient and effective?**

It is evident that the EAR/IF program is well managed and the Program
Officers are continuously trying to distribute resources within the EAR
community fairly. A few suggestions on how the panel and *ad hoc* review
process can be made more effective have been made above and in Part B.

The efficiency and effectiveness of the review process should not be
allowed to decrease as the proposal load or number of site visits
handled by the Program Officers increases: EAR must be vigilant in
allowing the POs enough time and personnel to handle their proposal
load. Over 300 proposals per year (comprising EAR/IF, MRI, and splits
with other NSF programs) were processed during 2001-2003, representing a
significant increase from the \~200 proposals per year over the last
review period. The consensus of this COV is that additional
administrative support for the EAR/IF program be considered in order for
the program to continue to function in a fair and efficient manner.

**A1c - Are reviews consistent with priorities and criteria stated in
the program's**

**solicitations, announcements, and guidelines?**

We found from the proposals selected for review by this COV that the
reviews were consistent with the IF priorities and criteria as stated in
the program's solicitations, with a few exceptions \-- and one important
deficiency related to Broader Impacts \-- as noted below.

The use of Broader Impacts as a new criterion for evaluation of
proposals for funding was approved by the National Science Board for
implementation Oct 1997, and the full text of the two merit criteria
appeared in NSF Grant Proposal Guide (GPG) 00-2. A subsequent GPG, NSF
02-2 (effective January 1 2002), stated that PIs must clearly address
Broader Impacts of a project both within the Project Summary and the
Project Description. It was stated in that guide (effective Oct 1 2002)
that proposals that did not address the two merit review criteria in
separate statements in the Project Summary would be returned without
review. This statement has been reiterated in subsequent GPGs.

Through both proposal evaluation and discussions with Program Officers,
we discovered that the wider EAR community requires more education as to
what constitutes a "Broader Impact". We note that over the 2001-2003
timeframe, both principal investigators and reviewers have gradually
paid more attention to this review criterion, but that this is by no
means universal. For example, we were pleased to note that this
criterion was adequately addressed by both PIs and reviewers in all the
proposals rated \"excellent\" that the COV has examined (n = 8).
However, many reviewers only commented upon the broader impacts of the
science. We suggest that the Program Officers specifically include, in
their communications with potential reviewers, a request for assessment
of the Broader Impacts of a proposal. It would be instructive to the
reviewer if this communication included examples of what constitutes a
broader impact. This will help educate the EAR community and will lead
to more comprehensive reviews of both review criteria for the EAR/IF
proposals.

The 2001-2003 review period also coincided with the introduction of new
opportunities in the MRI program to provide funding for equipment at
undergraduate institutions. While Intellectual Merit generally prevailed
as the principal criterion in reaching decisions, we note that this has
not always been the case for the MRI proposals from undergraduate
institutions. In the cases reviewed by this COV, it appears that success
in some cases was largely based on the significance of the Broader
Impacts. We urge vigilance in the evaluation of proposals from
undergraduate institutions to ensure that both "Intellectual Merit" and
"Broader Impacts" are weighted appropriately within programmatic
constraints. The COV recommends proactive solicitation of proposals to
expand the pool of applicants, thereby increasing competition and the
quality of the proposals that receive funding. We heard from the POs
that such activity is in progress.

A1d - Do the individual reviews (either mail or panel) provide
sufficient information for the principal investigator(s) to understand
the basis for the reviewer's recommendation?

The COV found abundant instances where *ad hoc* reviews provided
adequate information to assess the basis for the reviewer\'s
recommendation. In general, the review community takes the review
responsibility seriously. Several examples were found where reviewers
provided helpful comments to young PIs to help improve future proposal
writing. However, the COV also noted that occasionally there is a
discrepancy between the overall rating of a proposal by an *ad hoc*
reviewer and the tone of the written comments. This is a continual
frustration for the Program Officers and can be awkward in
communications with the unfunded PI. We are pleased to see that in the
examples evaluated by the COV, the "Review Analysis" section of the
proposal jackets contained sufficient documentation to highlight this
discrepancy, and was usually very useful in understanding the decisions
made on the proposal.

**A1e - Do the panel summaries provide sufficient information for
investigator(s) to**

**understand the basis for the panel recommendation?**

The usefulness of the panel summary depends upon the panel composition
and the load incurred by the particular panel member responsible for a
particular proposal type. We feel that the effectiveness of the panel
summaries could be improved if the process by which a panel is created
is tailored to the needs of a particular round of proposals and if
panels are encouraged to comment upon Broader Impacts (see above). In
addition, the COV found several examples where no panel summary was
contained in the panel jacket. Panel summaries are useful in reviewing
the peer review process, and their inclusion in jackets could be
increased.

**A1f - Is the documentation for recommendations complete, and does the
program officer**

**provide sufficient information and justification for her/his
recommendation?**

The documentation for recommendations on almost all proposals was
exemplary, except for the lack of panel summaries as noted above in some
cases. The decisions made by the Program Officers, whether consistent
with or contrary to the panel and *ad hoc* reviews, were extremely well
justified. We applaud the POs for their vigilant work to document the
peer review and decision-making process.

**A1g - Is the time to decision appropriate?**

Most proposals are reviewed and decisions are reached and communicated
to the PI within 6 months, although the full processing may take longer.
Where the processing time has gone beyond 6 months, we found
documentation that justified the delay. It was not clear, however,
whether the PI was always kept informed of the process. We encourage
rapid notification of PIs as well as the inclusion of documentation
within jackets to document why the decision time exceeds the normal
6--month period if notification is slow.

A1h - Discuss issues identified by the COV concerning the quality and
effectiveness of the program's use of merit review procedures:

The areas which the Committee identified for improvement in the review
process are summarized here. The COV suggests that the POs should
consider

> 1\) Encouraging better discussion of broader impacts within *ad hoc*
> reviews, perhaps by
>
> tailoring the review request letter to include specific examples of
> Broader Impacts;
>
> 2\) Insisting that panel reviews include discussion of Broader
> Impacts;
>
> 3\) Increasing the frequency of site visits or reverse site visits,
> utilizing committees of experts
>
> for such visits, and instituting a requirement for reporting of such
> visits;
>
> 4\) Seeking a more flexible and fluid mechanism for appointing the
> review panel, such as
>
> having six standing members on three-year rotations and two short-term
> members
>
> appointed on the basis of proposals received for each round of
> solicitations;
>
> 5\) Seeking more administrative staff support especially in order to
> increase the number of
>
> facilities that receive site visits;
>
> 6\) Taking a proactive stance on soliciting proposals from non-Ph.D.
> granting institutions (or
>
> other populations that can be funded from special reservoirs of money)
> in order to foster
>
> competition that will lead to higher caliber proposals.

A2 -Questions concerning the implementation of the NSF Merit Review
Criteria (intellectual merit and broader impacts) by reviewers and
program officers (PO's).

To answer the Broader Impact (BI) questions in this section, this COV
randomly chose and read 5 of the top-rated proposals (rating = 5.0) and
6 other proposals ranked between 2.0 and 4.25, encompassing a total of
48 ad hoc reviews. We particularly focused on the top-rated proposals to
ensure that, in those proposals, both criteria were considered in a well
balanced fashion. These proposals were randomly distributed with respect
to year (2001-2003). All of the proposals reviewed were accepted and
funded at some level. We read all the *ad hoc* reviews, the PO review
assessments (Form 7) and the panel summaries when they existed.

In addition, the COV also spoke with the IF POs to see if the perception
gained by looking at the reviews of this small subset of proposals was
representative of 2001-2003 proposals overall: we were assured that our
observations were congruent with that of the POs. We therefore feel that
our assessment of the questions posed in this section is reasonably
valid for fiscal years 01, 02, and 03.

One important evolution has occurred with respect to reviews for IF
during the time period under investigation. The BI merit criterion has
been in effect for the whole period under consideration. However, it was
only in 2003 that the IF solicitation specifically requested that BI be
explicitly stated in a proposal abstract and proposal body. Thus, while
proposers and reviewers should have been discussing BI before the 2003
proposal round, they were not explicitly required to do so until that
last round. GPRA statistics for IF proposals indicate that the number of
proposals to IF that included BI justifications increased from 79% to
88% from 2002 to 2003. The POs of IF told the COV that from this point
forward, proposals that do not explicitly outline the BI criteria will
be returned.

**[IMPLEMENTATION OF NSF MERIT REVIEW CRITERIA]{.underline}**

**A2a - Have the individual reviews (either mail or panel) addressed
whether the proposal**

**contributed to both merit review criteria?**

*Intellectual merit criterion*: All of the reviewers do a reasonable to
excellent job of evaluating the proposals based on the IM of the
proposals. They defend their assessment by addressing most or all
aspects of intellectual merit (importance of work, PI qualifications,
quality of writing, access to resources, etc.).

*Broader impacts criterion*: For the *ad hoc* reviews that were analyzed
from 2003, the reviewers almost always addressed the issue of BI. This
was not the case for 2001 and 2002, indicating the change in IF policy
in this regard in 2003 (see above). This is most likely a result of the
Format of the Fast Lane review which explicitly requests comments on the
BI of the work. We believe this will prompt the *ad hoc* reviewers to
address this issue specifically in the future.

Unfortunately, a problem remains with respect to the reviewers\'
interpretation of the BI criteria. Of the 48 reviews analyzed, close to
30% of the reviewers responded to the BI question in terms of extending
the science, as opposed to the intended interpretation of the criterion
regarding societal impacts of the proposal (i.e. dissemination,
teaching, training, benefits to society, underrepresented groups).
Assessing BI of IF proposals is particularly difficult in that
\"...consolidating and expanding infrastructure...\" is in fact a BI
criterion. The COV agreed that PIs will be unsure of how to treat BI for
IF proposals, and that further guidance on this point is needed within
the program solicitation. One possible approach might be to include
examples of how to treat BI within the text of the request letter sent
out by POs requesting reviews. One reassuring finding of our analysis is
that the proposals receiving the highest ranking (all Excellent) were
also found to be exceptional in the area of BI and that all the *ad hoc*
reviewers of these proposals addressed the BI criterion.

**A2b - Have the panel summary reviews addressed whether the proposal
contributes to**

**both merit review criteria?**

*Intellectual merit criterion*: The panel reviews always address the IM
of every proposal reviewed.

*Broader impacts criterion*: Because we focused on the highly rated
proposals, the decision to fund these proposals is made by the POs
without requiring the input of the panel. Thus, not many panel summary
reviews were analyzed. On the other hand, of the panel assessments that
were analyzed, none addressed the BI issue. This is an area of the
review process that could be improved.

**A2c -** **Did the review analyses (Form 7s) address whether the
proposal contributes to both merit review criteria?**

*Intellectual merit criterion*: In every case that we reviewed, the POs
very carefully documented most or all aspects of intellectual merit
based on all available mail and panel reviews. The COV was especially
impressed with the care used by IF officers in preparing these review
assessments. The assessments generally were well organized, thorough,
and thoughtful.

*Broader impacts criterion*: Except for the Review Analysis section of
Form 7s, there is no specific request for BI input. However, in almost
every case that we have reviewed, the POs very carefully document most
or all aspects of broader impact merit based on all available mail and
panel reviews.

**A2d** - **Discuss any issues or concerns the COV has identified with
respect to NSF's merit**

**review system.**

> 1\) The BI criterion is not uniformly applied and is often
> misunderstood by the *ad hoc*
>
> reviewers: perhaps the POs could tailor the review request letter to
> spell out how a
>
> reviewer might evaluate BI, especially with respect to infrastructure
> proposals.

2\) Based on the COV's investigation of the PO Review Analyses, we found
it noteworthy

that the POs are very aware and supportive of the BI criterion. The COV
agrees with

the POs that consistency in treatment of BI is desirable and it urges
the POs to remain

thoughtful and balanced in applying both criteria in judgments.

> 3\) The Panel Summary Reports we analyzed did not contain information
> on the BI of
>
> proposal merit. Especially for the border-line proposals, we would
> suggest that the
>
> results of the BI merit of the proposal be included in the summary.

4\) To ensure that high intellectual merit standards remain the primary
requirement for

funding, the POs are encouraged to be proactive in soliciting proposals
from diverse

PIs and institutions, to be proactive in \"match-making\" among PIs and
institutions

where appropriate to increase diversity, and to be proactive in
educating the EAR

community about BI.

**[SELECTION OF REVIEWERS (MAIL AND PANEL]{.underline}**)

**A3a -- Did the program make use of an adequate number of reviewers for
a balanced**

**review**?

Over the 3-year review period a total of 396 proposals received a total
of 2345 *ad hoc* reviews or an average of 5.9 reviews per proposal (see
also Table 1, section A1a). Of the total, only 5 proposals (1.3%)
received lower that the mandatory minimum of 3 reviews, and 47 reviews
(12%) received the minimum. Therefore, 87% of proposals received more
than the mandatory minimum number of reviews. At the upper end, 16
proposals (4%) received 10 or more reviews. Two national multi-user
facilities proposals -- IRIS (31 reviews) and Arizona AMS Facility (21
reviews)\-- received the highest number of reviews. The median number of
reviews per proposal is 6. It should be noted that many more
solicitations were made in order to produce the median result of 6
reviews due to un-responsiveness of some reviewers. Overall, the COV was
satisfied that the POs made a determined and largely successful attempt
to gather a substantial number of *ad hoc* reviews for each proposal
under consideration.

In the case of panel reviewers, the expertise of panel members during
2001-2003 was spread over 8 sub-disciplines in the geosciences. It can
be assumed that, in the vast majority of cases, the 'lead' panel member
assigned to a particular proposal had sufficient expertise to present a
meaningful interpretation of that proposal.

Assuming that the panel members read all proposals presented at their
meetings (albeit at different levels of detail), it can be assumed that
a majority of proposals are seen by 14 reviewers. This is an impressive
number: the COV therefore concluded that this process has generally
produced balanced reviews and that the POs are doing an excellent job
facilitating this process.

**A3b -- Did the program make use of appropriate expertise and/or
qualifications of reviewers?**

Identifying appropriate *ad hoc* reviewers is largely a matter of
experience of the program managers or their accumulated familiarity with
the research community. Given the large number of requests made (and
reviews garnered), it is the opinion of the COV that the program
managers have made full use of their talent and training in making
telling selections of appropriate *ad hoc* reviewers. In addition, given
the large average number of *ad hoc* reviews received for each proposal,
the expectation of the COV is that there is sufficient expertise among
the reviewer pool that each proposal receives expert review. This is
borne out by considering a random selection of proposals: within this
proposal set, the COV identified many examples where the background of
specific reviewers qualified them as experts whose opinions mattered in
the decision-making process. This is a crucial aspect of the peer-review
system whereby appropriate weight can be given to the opinion of certain
individuals in the reviewer pool.

The expertise of the panel reviewers is spread out over different
sub-disciplines. It is expected that the level of expertise among the
panel -- to review any particular proposal -- is therefore less than
that of the *ad hoc* reviewers. On the one hand, the COV recognizes that
the role of panel members is to weigh the relative merits of different
proposals as opposed to focusing on the quality of one particular
proposal. On the other hand, the majority of proposals under panel
review had a strong geochemistry emphasis and it was felt that greater
panel expertise in the area of geochemistry would better serve the
community in making informed decisions. Therefore, selection of
panelists is an issue that deserves more consideration and, as discussed
previously in this report (section A1a), a more flexible approach with
some *ad hoc* and some standing panelists might be desirable.

**A3c -- Did the program make appropriate use of reviewers to reflect
balance amongst reviewers (geography, institution type, underrepresented
groups)**

There are two aspects to this issue: *ad hoc* and panel reviewers. In
order to answer this question with respect to the case of *ad hoc*
reviewers, the COV had access to GPRA performance indicators and formed
a few interpretations based upon a random read of a selection of
proposals by COV members.

According to GPRA statistics, 5 and 13 *ad hoc* reviewers of IF
proposals were reported to be derived from underrepresented groups
during 2003 and 2002, respectively. These very low numbers in comparison
to the several hundreds of reviews received probably is due to the low
numbers of EAR professionals deriving from underrepresented groups, as
well as the low number of reviewers that reported demographic
information each year (38 and 53, respectively). The lack of statistics
on most of the reviewers makes it difficult to impossible to assess
whether the diversity of the pool of *ad hoc* reviewers is appropriate.

However, the prime selection criterion for reviewers by the POs appears
to be the ability of reviewers to evaluate the scientific (intellectual)
merits and broader impacts of the proposal. Geographical, institutional,
and diversity balance appears to be a secondary consideration in many
cases, although some geographic balance is achieved through the use of
international reviewers. Of course, some proposals specifically review
programs affecting Ph.D.-granting (research-oriented) institutions on
the one hand, or non-PhD granting (teaching-oriented) institutions on
the other hand, and reviewers for those programs are appropriately
composed of individuals from similar institutions. The COV encourages
the POs to select reviewers from underrepresented groups or from non-PhD
granting institutions whenever populations with appropriate expertise
are available: the review process not only provides peer review but also
serves to educate the reviewer population about science and programmatic
opportunities. The COV found no evidence that the POs were not already
cognizant of this important task although its implementation is
recognized to be difficult.

For the case of panel reviewers, the COV is satisfied that the panel is
selected to reflect diversity in the research/teaching community in
terms of geography, institutional type and gender. Rotations on/off the
panel appear to be made to retain this diversity.

**A3d -- Did the program recognize and resolve conflicts of interest
when appropriate?**

It is the opinion of the COV that this is an issue well-managed by the
program. In the majority of cases, the reviewers themselves have been
educated to raise this issue at the first possible hint of conflict. The
extensive experience of the program managers catches any potential
conflicts which are not obvious to the reviewers (panel and *ad hoc*
reviewers). Resolution is also well managed in that reviews/reviewers
are removed from the decision-making process where appropriate. This
activity was well-documented by the program managers.

**A3e -- Discuss any concerns identified that are relevant to selection
of reviewers.**

The COV has no serious concerns regarding the selection of *ad hoc*
reviewers. The program manages to receive a substantive number of *ad
hoc* reviews to help guide the decision process. It is, however, the
opinion of the committee that improvements may be possible in the
selection of panel reviewers as discussed previously in A1a and A1h. To
reiterate, we suggest that the program should respond to the types of
proposals submitted in each round by identifying expertise that could be
drafted onto a panel on an 'as needed' basis. For example, if the number
of 'standing' panelists were reduced to 6 (instead of 8) then 2 experts
would be able to join the panel to provide additional expertise in a
particular area. A pool of *ad hoc* panelists would provide the program
with increased flexibility to respond to proposal pressure.

**[RESULTING PORTFOLIO OF AWARDS]{.underline}**

Questions concerning the resulting portfolio of awards under review.
Provide comments in the space below the question. Discuss areas of
concern in the space provided.

A4a - Overall quality of the research and/or education projects
supported by the program.

The conclusion of this COV, based upon our analysis of the *ad hoc*
reviews, the panel comments, and the experience and knowledge of the
POs, is that the IF Program generally selects, encourages, and funds the
proposals that are highest ranked in terms of the project's Intellectual
Merit, while taking into consideration the Broader Impacts
appropriately.

We reviewed a number of proposals from the period 2001-2003. A prime
example of one of the more exceptional scientific opportunities that
have come about as a result of IF decisions during this or previous
funding periods includes the UNAVCO Inc. response to the 2002 Magnitude
7.9 Denali Earthquake. It was the largest strike-slip earthquake in
North America in almost 150 years. Within days, technicians from the
UNAVCO Inc. Boulder facility (Prescott, EAR-0321760, Freymueller,
EAR-0323156, Freymueller, EAR-0075680, Davis, EAR-0229402) were
mobilized. This group quickly installed a large number of permanent
receivers in and around the 340 km of surface deformation. Because the
facility existed, the community was able to collect post-seismic surface
deformation data that allowed for a greater understanding of the
dynamics of strike-slip earthquakes in general and Denali Fault hazards
and mechanics in particular. The UNAVCO Inc. response to this earthquake
has set the standard for immediate deployment of GPS receivers after a
large event. Such data collection will allow understanding of the
post-seismic deformation in much finer temporal detail. The importance
of this work was recognized in a publication in *Science* (The 2002
Denali Fault Earthquake, Alaska: A Large Magnitude, Slip-partitioned
event, Eberhart-Phillips, D. and 28 other authors, *Science, 300*,
2003).

While outstanding examples such as the Denali fault investigation
document the direct contribution of IF decisions to the advancement of
science, numerous indirect contributions from IF funding are harder to
describe or quantify explicitly. The instrumentation, field-work,
personnel, facilities, and centers funded by IF contribute in a broad
disseminated fashion toward the advancement of geophysics, geology, and
geochemistry in ways that are hard to pinpoint. These disseminated
impacts also allow enhanced educational experiences and opportunities
including hands-on experience for graduate and undergraduate students.

One outstanding example of such impact is the work of D. Whitney-King of
Colby College, who was funded by IF (EAR-0115900 \$99k) for a unique
field laboratory fully mounted on a boat for use in field measurements
by undergraduates in Maine. With 7 water column sensors, an on-board
chemistry laboratory, and a vibracorer for lake studies, this
pontoon-mounted system has provided extraordinarily fun and exciting
geochemical opportunities for undergraduate students at a non-PhD
granting institution.

The "phase in" of the Broader Impacts (BI) criterion for proposal
reviews in 1999 to the strict requirement that BI be addressed in the
proposal abstract in 2003 will no doubt increase the participation of
graduate and undergraduate students, women and underrepresented ethnic
groups in NSF-sponsored projects. In this respect, the COV would like to
highlight the exemplary project by A. Schwalm of Oglala Lakota Tribal
College. The project has received funds to support the acquisition of a
mobile laboratory of analytical instruments to be employed in
environmental geosciences research. Oglala Lakota Tribal College is a
non-PhD granting institute on the Pine Ridge American Indian Reservation
in South Dakota. The equipment is used by undergraduates also within a
K-12 outreach program on the Reservation. The COV recognizes the
importance of these types of projects and encourages the IF to continue
supporting projects like this into the future.

The COV also discovered some excellent examples of IF funding for the
development of new instrumentation. The IF program provided \$99K of
funding to establish a multi-user Computed Tomography (CT) laboratory at
the University of Texas under the direction of William Carlson. X-ray CT
allows non-destructive imaging of geological samples in three dimensions
with micron-scale resolution. CT imaging is revolutionizing our ability
to image and quantify porosity and permeability structure in rocks, our
capability to identify structures and functions of fossilized remains,
and our ability to quantify size distribution of crystals and other
textures.

Important instrumentation developments often developed as a result of
IF- supported collaborations between multiple research groups. We would
like to point to the development of a high-accuracy GPS calibration
system. No one institute could tackle the problem of development of this
system alone. J. Davis of Harvard-Smithsonian Astrophysical Observatory
was funded to work with scientists and engineers from MIT, UNAVCO Inc.
Facility, NASA, and Trimble to develop the Antenna and Multipath
Calibration System, a movable parabolic antenna capable of tracking an
individual satellite used to characterize the multipath characteristics
of a particular site. Such characteristics remain a significant source
of noise in GPS observations. Characterizing these effects will allow
significantly improved estimates of crustal deformation.

In conclusion, it is the consensus of this committee that the IF program
funds world-class science and educational programs. The IF portfolio
manifests awards for individual investigator instruments, technicians,
educational opportunities, several investigator facilities, and
multi-user, multiple institution facilities.

**A4b - Are awards appropriate in size and duration for the scope of the
projects?**

The majority of awards for instrumentation are for a two-year period,
which seems appropriate. On the other hand, awards for Technical
Support, Phase I (3 years) and Phase II (2 years NSF, 2 years
institutional), are longer in duration than those for Equipment
acquisition, as expected for support for personnel. Facilities
necessarily have a 3-5 yr duration as continuing grants or cooperative
agreements, but these are negotiated case by case. While GPRA targets a
three-year award duration, on the whole this is not appropriate for IF
awards since most IF grants are Equipment acquisition grants and two
years is generally ample time to research and purchase instrumentation.
The COV concluded that award durations were appropriate within the IF
program.

The average award sizes vary from \$74K (2001) to \$191K (2003),
bracketing the GPRA target of \$128K but with a very large standard
deviation. During 2001- 2003, the IF program funded a large number of
smaller awards and 13 facilities: award sizes of facility support
comprised one each at about \$13, 3, 3, and 1.5 million, and eight at
about \$0.25 million, for an average of \$1.9 million for facilities
including increments. Recognizing the large number of small awards and
small number of large awards, the median (\$50-70K) is a better
indicator of award size. Technical support awards average \$75K per
year: this is the highest award allowed in this category, and appears to
be an appropriate amount to cover both salary and overhead at many
institutions. For institutions where cost of living is higher, the
institution must provide additional funds to bring salary up to a
competitive level. The COV agreed that the IF program was awarding fund
levels appropriately given GPRA targets and given the fact that
instrumentation needs vary in size and cost over a very large range.

The COV generated a lot of discussion about the balance between awards
to facilities and individuals, simply because of the large size of
awards to facilities. Facilities garner the largest share of funds, but
reading of the jackets show that their budgets are scrutinized closely
and appropriately. This issue is a difficult one for the COV to assess
given the length of time we had to look at the data. This topic is
further discussed in A4f below.

**A4c -- Does the program portfolio have an appropriate balance of high
risk proposals?**

If we define a high-risk proposal as a proposal concerned with
instrument or technique development, then more than 30 proposals out of
a total of 396 over the 2001-2003 review period fit this category. Of
these proposals, 14 were funded with an average *ad hoc* review score of
4.2. Appropriately, the declined proposal population had a lower average
score of 3.7. In addition, some of the more than 155 acquisition
proposals reviewed may also be classified as high-risk. For example, the
EAR/IF program implemented a new funding category in 2003 to provide
funding to early-career PIs. The new category bundles instrument
acquisition with technical support and addresses the need to fund
innovative research proposed by unproven PIs. It is the opinion of the
COV that these proposals may also be deemed High Risk in many cases.
Furthermore, approximately 25% of funded proposals are submitted by new
investigators to EAR/IF and may also comprise high-risk projects. Given
these imperfect measurements of high risk in projects, we concluded that
the program funds an appropriate number of high-risk proposals and has
implemented changes to ensure support to innovative proposals by new
researchers. However, clearly a better metric is needed to answer this
question if it is to be answered precisely.

One example of High Risk is proposal EAR 0236799 (J.N. Aurnou). This
project was funded to promote development of a cylindrical geophysical
dynamics lab to simulate processes occurring in the Earth's core. This
involves a scientist early in his career, high-risk innovative research,
and significant broader impacts. Another example is 0352119 and 0352134
(P. Silver, F. Niu) designed to develop a methodology to directly
measure temporal variations in subsurface tectonic stress based on
changes in seismic velocities. They propose to use unique and new data
from the Earthscope SAFOD project. This project will develop a working
methodology, which, although risky, could have a high community impact.
A third example is 0318473 (K. Farley), a collaborative proposal to
improve the capabilities of noble gas sector mass spectrometry.
Reviewers identified the proposal as high risk, yet potentially
rewarding.

**A4d -- Does the program portfolio have an appropriate balance of
multidisciplinary proposals?**

The facilities and instruments funded by EAR/IF support PIs from many
different disciplines. UNAVCO supports work on glaciers, volcanoes, and
tectonically active areas, and, likewise, IRIS supports a broad
community of scientists to investigate data collected from around the
world. The IRIS and UNAVCO communities hosted several multi-disciplinary
meetings during the development of the Earthscope proposal to allow
input from a broad spectrum of scientists.

An example of the large number of disciplines that use an EAR/IF funded
geochemical instrument can be found in Table 1 of the EAR-0114641
(Schiffman) proposal. Although about half of the use of Schiffman\'s
electron microprobe is by the Geology Department, the remainder of the
use is spread between Chemistry, Veterinary Medicine, Chemical
Engineering, Civil Engineering, Physics, University Extensions, and
outside non-profit institutions. Instruments and facilities funded by
the EAR/IF program are commonly used by a broad range of researchers.

**A4e -- Does the program portfolio have an appropriate balance of
innovative proposals?**

Innovation is required to develop world-class facilities, instruments,
and methods. The intellectual merit criteria ensure a high degree of
innovation because science advances most rapidly by allowing the best
researchers to make cutting edge measurements with advanced instruments.
EAR/IF has consistently pushed the envelope and as a result top
scientists from all over the world now work in the USA. Large facilities
like IRIS set global standards with projects like the Global Seismic
Network and the Data Management Center. The community-driven and newly
funded Earthscope initiative, initiated with help from personnel within
IRIS and UNAVCO as well as other institutions, effectively blends
seismology and geodesy to examine ongoing tectonic processes. Some
innovations result from the flexibility to respond to events. Rapid
responses to recent earthquakes are capturing possible strain transients
to better characterize the earthquake cycle. Others improve measurement
precision, such as the development of an instrument to make in-situ GPS
antenna phase center calibrations. Finally, innovation is fostered by
the feedback mechanism between P.I.s and the POs that results in
improved proposals.

**A4f - Does the program portfolio have an appropriate balance of
funding for centers,**

**groups and awards to individuals?**

The COV spent considerable time discussing this question. The IF program
supports a portfolio of programs that spans many small projects funded
at levels as low as several tens of thousands of dollars per year to one
large project funded at over \$13 million/year (data from 2003). The
distribution of awards collated by funding amount therefore comprises a
skewed distribution with the majority of awards less than \$100,000 per
year. The tail of the distribution represents a few very large Facility
Support awards of which the three largest are IRIS, UNAVCO, and COMPRES.
To investigate this distribution thoroughly in order to assess whether
the distribution is appropriate would take more time than allotted for
the COV process. The COV feels that considerable discussion of this
question occurs at NSF and is fostered by the POs; however, the POs
might consider promoting a broader community discussion of the topic
through workshops, published articles, or town meetings at national
conferences.

Facility-support (FS) dollars comprise approximately 72% of the total IF
budget and generally fund large multi-user centers and instrumentation.
This fraction has stayed roughly constant from 2001 to 2003. Other
smaller funded projects (nonFS projects) comprise 16% for Equipment
Acquisition (EA), 6% for Instrument and Technique Development (ITD), and
3% for Technician Support (TS). These percent values represent fractions
of the total budget and have each remained roughly constant from 2001 to
2003. The few projects (3%) funded by IF that do not fall in these
categories are special projects or workshops convened with respect to
infrastructure questions or goals.

All of the FS awards benefit relatively large groups of people
(centers), and many of the awards to single PIs benefit multiple
constituencies at one or more universities. The IF program thus
demonstrates great agility in attracting, awarding, and managing a wide
distribution of single- and multi-PI (group or center) types. While many
of the smaller EA, ITD, and TS grants are awarded to individuals, these
grants to single PIs generally benefit large groups of users on a single
or on a regional set of campuses. Therefore, the COV points out that few
if any of these IF awards benefit single individuals, and the COV argues
that this is appropriate for infrastructure grants. We feel that the
more people that have access to infrastructure, the larger the benefit
of the funding.

A harder question is whether the IF funding benefits an appropriate
balance of small versus large groups. In addressing this question, the
COV looked at the balance among FS, EA, ITD, and TS proposals and
awards. To assess whether the portfolio has an appropriate balance of
these categories, the COV discussed the balance with POs, looked at
proposal pressure and success rate, analyzed the percent of funding per
award, and looked at where POs drew the line between funded and
non-funded proposals.

One way to assess whether an appropriate balance is maintained is to ask
the question, Is the success rate for each proposal type kept at an
appropriate level? Success rates for EA, ITD, and TS proposals over the
timeframe 2001 to 2003 average 61, 54, and 53% respectively. The COV
felt that the success rates for EA, ITD and TS proposals should remain
above 50% given the large population served by this projects, and we
applaud the diligence of the POs in maintaining these rates.

In contrast, the success rate for FS proposals is hard to document given
the iterative nature of the proposal development process for FS.
However, these proposals generally have a higher success rate because
they are thoroughly vetted by the community through workshops, *ad hoc*
reviews, and panel reviews. While the COV found ample evidence that the
POs interact extensively with some PIs for smaller grants, the
significant interaction among FS personnel and IF Program Officers
probably results in higher success rates for FS proposals. The COV felt
this higher success rate for FS proposals to be appropriate.

The success rates of the different proposal types was observed to be
generally stable with time with the exception that over the period 01 to
03, success rates of TS proposals decreased from 71 to 22%. The COV was
assured that this low rate of funding of TS proposals in the last year
was an anomaly and does not signify an important trend. To assess
whether these success rates were appropriate, the COV carefully looked
at the projects that were just above or below the award/declination line
for each year, and agreed with POs in the positioning of the declination
line. A large fraction of the nonFS awards are geochemical awards (see
section A4k) \-- for example, 2/3 of nonFS proposals awarded in 2003
have a strong geochemistry component \-- and the COV felt this
observation is explained by the fact that most of the needs of the other
communities are met through FS awards such as IRIS. The steady success
rate of these largely geochemical EA/TS/ITD proposals, along with our
observation that the declination line is drawn in roughly the correct
position, led the COV to the conclusion that the geochemical community
is largely well-served by the IF program.

However, the COV was very aware that budgetary considerations may
produce pressure on success rates. We observe that, if this pressure is
largely exerted on EA/TS/ITD proposals, then the pressure may
disproportionately and inappropriately hurt the geochemical community.
The COV pointed out that high success rates within the IF program are
probably well-warranted given the fact that these proposals put
infrastructure into the hands of a diverse and talented pool of
scientists, and without such infrastructure, science projects and future
proposals are impossible to complete.

Budgetary pressures may be felt not only in success rates but also in
the percent of budgets awarded (fraction of the \"ask\" price that is
awarded). The COV therefore looked in FY 03 to determine what percent of
the request for funds was funded for successful awards. For FS
proposals, we calculated that awards ranged between 80 and 100% of the
request, with an average of 90% (in other words, on a year by year
basis, on average, each successful FS PI receives 90% of what was
requested per year). In contrast, EA and TS were funded on average at 83
and 100% respectively (this represents the fraction of the total request
that was given as the total award). The similarity of these numbers may
suggest that the IF is doing a good job of treating FS, EA, and TS
proposals equivalently; however, more thorough analysis of this metic is
warranted. The COV applauds the POs with respect to these numbers, and
suggests that these values be considered in the future as budgetary
decisions are made so that large and small awards are treated
appropriately.

On balance, given the amount of time we had to look at the data, the COV
felt that the portfolio is well-balanced between large and small awards.
We recognize that some communities have organized and nucleated groups
that are able to garner large awards: these awards allow efficient use
of resources, and allow the community itself to self-police funding.
Where other communities (largely geochemical communities) have not yet
self-organized, the COV feels that EAR has been trying hard to help this
process along. The COV encourages this process, and encourages the IF
program officers to fund workshops and meetings that might enable
instrumentation centers or instrumentation developments to foster
organization of communities. The POs must also continue to try to make
sure that balance is maintained between large and small awards by
endeavoring to maintain high success rates among EA/ITD/TS proposals and
by endeavoring to treat small and large awards congruently with respect
to the fraction of the \"ask\" price that is awarded.

Finally, the COV notes that the high success rates have been maintained
largely on the basis of funding managed by IF from outside of IF (MRI,
ITR, undergraduate institution funding). The COV is concerned that if
this outside funding were to be removed, then success rates would drop
to inappropriately low levels.

**A4g - Does the program portfolio have an appropriate balance of Awards
to New**

**Investigators?**

The COV was very impressed with the fact that for each year of this
review, \~25% of awards were given to new investigators. We feel that
this was a very appropriate number, allowing new ideas to be funded and
assuring the healthy development of EAR infrastructure.

**A4h -- Does the program portfolio have an appropriate balance of
Geographical distribution of P.I.'s?**

Funding to the three accounts managed by EAR/IF (MRI, FS, IF) has been
awarded to 41 different states during this review period. This
distribution is remarkable and reflects careful management by the
program.

**A4i -- Does the program portfolio have an appropriate balance of
Institutional types?**

The large research universities continue to receive the largest share of
IF funds during the review period. MRI funding is more evenly split. It
is not clear how support for smaller institutions can be improved within
the constraints imposed by the Intellectual Merit criterion. A stronger
emphasis on Broader Impacts could raise the scores of some proposals,
but this may not increase funding for nonPh.D.-granting institutions.
Recently, some funds have been allocated by Congress for institutions
whose primary mission is undergraduate education, and EAR/IF has removed
the requirement for 30 per cent matching funds for the first \$50K
awarded. We expect that a better balance in institutional types will
result. Wherever and whenever possible, the IF POs should try to
increase the number of proposals from smaller institutions so that
competition will promote funding of the best ideas.

**A4j - Does the program portfolio have an appropriate balance of
projects that integrate**

**research and education?**

Large facilities funded by EAR/IF all have an education and outreach
component, as specified by the program announcement. In addition, more
and more of the EA/ITD/TS proposals include research and education
components as part of the Broader Impacts criterion. Furthermore, most
of the funded infrastructure from the IF program is accessed by students
at a variety of institutions: thus the research component is often
integrated dynamically with the education component. Improved
participation from non-PhD granting institutions, likely to result from
recent policy change, should improve undergraduate use of EAR/IF funded
instruments.

There are many examples spanning across a variety of types of projects
that clearly demonstrate integration of research and education. Flagship
among the large facilities is IRIS, which has put significant effort
into the development of a vibrant E/O program, including in particular a
summer research internship program for undergraduates at member
institutions. Another example is UNAVCO: a facility that assists PIs
with almost every aspect of GPS data collection (recommending
appropriate instrumentation, establishing permanent installations,
organizing and conducting field experiments, maintaining an equipment
pool, monitoring the health and status of the various permanent
installations, archiving data into an on line database, etc.) and runs a
successful education and outreach program. UNAVCO Inc. encourages
discussion amongst its members by holding an annual workshop where GPS
scientists (PI's and students) are encouraged to share and discuss their
latest results. The outreach program places particular emphasis on K-12
students and teachers. UNAVCO Inc. also coordinates the PBO component of
EarthScope. The UNAVCO Inc. research has been outstandingly successful
in enabling US scientists to participate in forefront research in the
geosciences using state-of-the-art GPS instrumentation while educating a
broader community about GPS. This is a unique resource, unanimously
supported by the community, which is continuing to advance our
understanding of Earth's internal structure and function.

At the other end of the spectrum, an example of successful integration
fostered by the IF is the grant to Albrecht (0321299) at Oglala Lakota
Tribal College. This is a Native American college on Pine Ridge
Reservation, South Dakota. It integrates teaching and research not only
in undergraduate education, but also in high schools where it is
critically needed. Another example of undergraduate participation in
research at a small college can be found in the final report from D.
Whitney-King (Colby College, 0115900) where 16 students are listed as
using an integrated field laboratory.

With the continued development of the Broader Impacts criterion in
proposals, especially after 2003 when it was mandated that this subject
had to be broken out into a specific section, we expect education to be
better integrated into every EAR/IF proposal.

# 

# **A4k -- Does the program portfolio have an appropriate balance across disciplines/sub-disciplines of the activity and of emerging opportunities ?**

The IF program supports infrastructure in a dazzling array of
disciplines in the geosciences. The national, multi-user facilities
section of the program portfolio (FS) alone supports research in 12
themes under the broad umbrellas of geochemistry, geophysics, geodesy.
The other 4 areas of support (EA, ITD, TS, EC) cover other disciplinary
areas of research, more at the individual PI level. The committee
applauds the fact that the program is sufficiently flexible to offer
support across such a broad swathe of the geosciences.

For the 2003 IF awards, a total of 75 awards could be divided as
follows: 47 geochemistry; 14 geophysics; 9 cyberinformatics; 1
geobiology and 4 geology. Therefore, two-thirds of proposals can be
classified as having a dominant geochemistry component. To the COV, this
reflects the fact that the geophysics community is well served by the FS
portfolio (particularly IRIS) whereas much of the community
geochemistry/geology infra-structure lies in the hands of individual
PIs. Given the nature of the science pursued by the community and their
different equipment needs, the COV debated the apparent disparity in (a)
the large number of geochemistry proposals, and (b) the budgetary skew
towards the FS section of the program. However, the geosciences
community as a whole appears to be well served by the IF program, if the
relatively constant success rate of EA proposals can be used as an
indicator (see question A4f). The question of balancing community needs
and prioritizing goals is an issue that needs to be addressed at the
community level: the COV agreed that time constraints for the COV
process do not allow adequate investigation of this question.

**A4l - Does the program portfolio have appropriate participation of
underrepresented**

**groups?**

The COV looked at the percentage of women and minorities that received
IF support in the period 2001-2002. We found that women and minorities
represented approximately 10% and 6% of the funded awards. If we compare
these percentages to the percentage of women and minorities receiving
their PhD's in 2000, 30% and 21% respectively (NSF Division of Science
Resources Statistics, http://www.nsf.gov/sbe/srs/), we find that the
percentages are below the percentages of women and minorities among new
PhD's. This comparison is no doubt somewhat unfair as the percentage of
PhD's in each of these underrepresented groups is probably somewhat
larger than the percentage represented in new faculty. However, this
latter figure is not easily found in most databases. The COV discussed
this with the POs and was pleased to find that the IF program is putting
and will continue to put a large effort into increasing the percentage
of women and minorities garnering awards, in order to match the makeup
of university faculties across the United States.

**A4m - Is the program relevant to national priorities, agency mission,
relevant fields and**

**other customer needs? Include citations of relevant external
reports.**

The EAR division allocates more than 28% of its budget to investment in
infrastructure, when contributions from the Major Research
Instrumentation (MRI) and Geoinformatics funds are included. This
relatively high investment in infrastructure is slightly above the value
that the National Science Board Committee on Programs and Plans Task
Force on Science and Engineering Infrastructure recommended as the
fraction of the NSF budget that should be spent on small- and
medium-scale infrastructure and cyberinfrastructure: 22-27% (Setting
Priorities for Larger Research Facility Projects supported by that
National Science Foundation). We feel that this allocation is
appropriate, given the needs for instrumentation and infrastructure in
geological and environmental sciences. In this respect, IF is performing
a critical and important function congruent with national priorities and
agency mission.

The infrastructure supported by the EAR/IF program is an essential and
basic component for advancing research in all of the 6 opportunity areas
identified in the 2001 report of the National Research Council, \"Basic
Research Opportunities in Earth Science\". For example, the IRIS
consortium comprises over 100 member institutions and serves the
infrastructure needs of a broad community of research seismologists
engaged in studies of the earth\'s deep interior in an efficient,
integrated, and standardized manner and with implications for areas of
national need. Specifically, the IRIS global seismographic network (GSN)
contributes significantly to the global nuclear test ban treaty
monitoring system. IRIS brings to treaty monitoring an established
structure for the collection and distribution of data from a global
network of seismic stations.

**A4n - Discuss any concerns identified that are relevant to the quality
of the projects or the balance of the portfolio.**

1\) Because it is difficult for a COV to assess the appropriateness of
award balance

among facilities and smaller awards given the short COV assessment
period, other

mechanisms to allow longer-term analysis of this balance should be
considered. For

example, workshops or town hall meetings could be facilitated to look at
award

balance among large and small awards.

2\) Where disciplinary communities or populations of institution types
are not receiving

award support for large facilities, POs should consider mechanisms to
educate these

> communities about such opportunities where appropriate. One way to
> help nucleation
>
> and growth of larger facilities is to consider more ample funding for
> technicians for
>
> successful multi-user facilities as opposed to new support for new
> instrumentation in
>
> single PI laboratories. Education of the community of earth scientists
> to increase the
>
> pool of small colleges applying for infrastructure would also benefit
> the IF program.

3\) Budget pressures should be exerted on both small and big awardees
appropriately, so

that budgetary cutbacks or increases do not disproportionately affect
one type of

awardee more than another: this is particularly important given that
large facilities may

be able to muster more political muscle in times of fiscal cutbacks as
compared to

small awardees and given that two different communities are largely
served by the

large and small awards.

4\) Money leveraged from outside of IF (e.g. from the Major Research
Instrumentation

program) has allowed maintenance of high success rates, possibly making
EAR/IF

more vulnerable to future problems if such outside funding decreases.

**[MANAGEMENT OF THE PROGRAM UNDER REVIEW]{.underline}**

**A5a - Management of the program.**

In reviewing a selection of awards over the years 2001, 2002, and 2003,
the documentation supporting each decision was exemplary. This COV noted
a few borderline cases and also those with a low number of *ad hoc*
reviews to investigate further. In each of these case, between 8 and 11
*ad hoc* reviews had been requested, even though fewer reviews had been
received. In most cases, proposals with a low number of *ad hoc* reviews
were also reviewed by several panel members. Thus the POs were making
reasonable efforts to garner an adequate number of reviews per proposal.
In the Review Analyses presented by the Program Officers, the reasons
for the funding decisions were detailed and well presented, such that
the PI could easily see the rationale for the outcome. This COV commends
the Program Officers for their diligence in this process.

**A5b -** **Responsiveness of the Program to emerging research and
education trends.**

The COV addressed this question by looking at two categories of
proposals. Those that represented a "New Investigator" (i.e., new to the
program), and those that represented development of new instruments and
techniques (ITD). Of the awards made during the period covered by this
COV, \~25% were to new investigators. Between 2001 and 2003, 30 ITD
proposals were received and 14 were funded, including the following
examples: Thonnard (Ear-0119110) "RIMS for Kr isotope analysis"; Aurnou
(EAR-0236799) "Development of an experimental geophysics fluid dynamics
lab"; Farley (EAR-0318473) "Development of a Pauli ion trap for noble
gas mass spectrometry". For some of the IF awards, the EAR/IF program
makes good use of leveraging funds from other NSF programs and other
agencies for developing such emerging technologies and techniques. The
COV applauds this effort and strongly encourages it to continue.

In one high-growth and high-impact emerging area, low-temperature
geochemistry and geobiology, the COV noted that this expertise was not
well represented among the panel reviewers. The POs pointed out that new
additions to panels for 2004 had been made to encompass this area, and
the COV encourages careful attention to such emerging disciplines. As
discussed previously, if the panel members rotate off and are allowed to
suggest new panel members, this might focus panels in mature areas of
science rather than emerging areas of science. Careful attention to
panel composition is clearly warranted.

The COV also noted that some awards were made because of a strong
educational component (e.g., Halfman EAR-0116078) and that the supported
national, multi-user facilities (e.g., IRIS, COMPRES, NCALM, etc.)
contain a large education and public outreach component. We believe that
these examples demonstrate that the EAR/IF program is being responsive
to emerging education trends and urge that it continue to do so.

Finally, the interaction of IF support with other programs and agencies
is important for obtaining support for education through research
assistantships and fellowships, whereas student use of IF-funded
equipment/facilities is critical for developing the next generation of
scientific talent. As with the previous COV, we view it as essential
that support of graduate student training be accompanied by access to
state-of-the-art instrumentation and that such support be increased
wherever possible.

**A5c -** **Program planning and prioritization process (internal and
external) that guided the**

**development of the portfolio under review.**

The COV is aware of strategic planning occurring within EAR, including
the IF program, and feels that this exercise has been important in
helping the POs to facilitate excellent proposals within IF. Such
planning should allow Program Officers to maintain awareness of emerging
technologies (e.g., NanoSIMS) and opportunities (e.g.,
multi-disciplinary research) that could dramatically impact EAR
infrastructure.

In assessing planning and prioritization, we also refer to the
recommendations made by the last COV report in 2001. The recommendations
made in that report and the responses by the POs included:

> 1\) The EAR/IF program might give consideration to a process where the
> Early Career

Researchers (ECR) status of all PIs are specifically identified. *This
has been*

*implemented*.

> 2\) It is important that the program managers protect the smaller,
> non-facilities portion of
>
> the budget and allow it to grow in proportion to the increased needs
> that are developed
>
> by large comprehensive programs. *This COV notes that the bulk of the
> increase in the*
>
> *EAR/IF budget over the last 3 years has gone to existing or new
> multi-user facilities,*
>
> *while funding for other areas of the EAR/IF program has remained
> relatively constant*.
>
> *The former is not surprising, given that, to a large extent, the
> emergence of new multi-*
>
> *user facilities or the expansion of existing ones was anticipated,
> and additional funds*
>
> *in their support were actively sought and obtained for them in
> advance. Maintenance*
>
> *of level support for the smaller awards appears to have been attained
> through*
>
> *leveraging of IF money with outside money: while the COV commends
> this successful*
>
> *leveraging, it foresees that changes in outside funding might leave
> IF vulnerable to*
>
> *significant decreases in success rates.*
>
> 3\) The program should encourage bold evolution of the scope and
> capabilities of those
>
> multi-user facilities that request continued funding. *In this review
> period, the largest*
>
> *pre- existing facilities, IRIS, COMPRES, and UNAVCO, have engaged in
> a substantia*
>
> *evolution process involving partial reorganization in anticipation of
> their role in the*
>
> *implementation of Earthscope.* *However, they maintain a* *strong
> level of activity in*
>
> *their \"traditional\" areas, the need for which should continue to be
> thoroughly*
>
> *reviewed. In particular, in the current overall restricted budget
> climate, the continued*
>
> *program budgets should be carefully scrutinized.*
>
> 4\) Advent of Earthscope must be accompanied by a substantial influx
> of new funds to the
>
> IF program. *This issue is important - especially in the current
> economic climate -- for*
>
> *EAR, but it is now irrelevant to IF, since a special program and
> panel have been*
>
> *created within EAR to handle Earthscope related projects. On the
> other hand, as far*
>
> *as projects supported by IF are concerned, we urge that the Program
> Officers be*
>
> *extra vigilant in their efforts to leverage funding from other
> agencies and NSF*
>
> *programs, as well as in maintaining a better balance between the
> different EAR/IF*
>
> *emphasis areas.*
>
> 5\) Dwell time: decisions made by the program should continue to be
> driven primarily by
>
> the integrity and thoroughness of the review process, rather than
> efficiency targets.
>
> *This recommendation remains valid. In particular, the Program
> Directors should*
>
> *continue to strive to receive 4 or more expert reviews for every
> proposal considered.*

A5d - Discuss any concerns identified that are relevant to the
management of the program.

The funding balance between different areas of the EAR/IF program needs
to be carefully assessed, especially with respect to the amount going to
national multi-user facilities versus other areas of the program (see
A4f). One focus of this concern is the observed lack of growth in the
amounts spent for Equipment Acquisition and Technician Support over the
last three years. This level funding did not result in a decrease of
proposal success rate (although Technician Support funding showed an
anomalously low funding rate in the last year considered); however, it
appears that outside money brought into IF from MRI and other programs
may be part of the reason success rate has remained high. Given the
expense of equipment and support, it will become increasingly useful to
encourage communities to nucleate centers of excellence and to seek
Facility Support for communal equipment. The COV suggests that IF POs
continue to help the geochemical community nucleate centers and
organize. In this regard, it may be useful to consider allocating more
money to Technical Support for existing facilities in the near future.

Furthermore, to keep facilities vibrant and growing, we reiterate the
recommendation of the previous COV that IF should encourage bold
evolution of the scope and capabilities of those multi-user facilities
that request continued funding.

Part B - RESULTS: OUTPUTS AND OUTCOMES OF NSF INVESTMENTS

B1 - [OUTCOME GOAL FOR PEOPLE]{.underline}: Developing "a diverse
competitive and globally engaged workforce of scientists, engineers,
technologists and well-prepared citizens."

The IF program officers have a unique opportunity to use IF funding to
attain the goal of a diverse, competitive and globally engaged
workforce. The COV observed that the POs were actively engaged in this
opportunity and were amply successful in providing infrastructure
funding to a diverse group of people. We provide a few examples
pertinent to diversity.

D. Whitney-King of Colby College was funded by IF (0115900) for a unique
field laboratory fully mounted on a boat. With 7 water column sensors,
an on-board chemistry laboratory, and a vibracorer for lake studies by
undergraduates in Maine, this pontoon-mounted system has provided
extraordinarily fun and exciting geochemical opportunities for young
scientists at an undergraduate-only institution.

Dr A. Schwalm of Oglala Lakota Tribal College was funded to support the
acquisition of a mobile laboratory of analytical instruments for
environmental geosciences research. Oglala Lakota Tribal College is a
non-PhD granting institute on the Pine Ridge American Indian Reservation
in South Dakota. The equipment is used by undergraduates as well as
within a K-12 outreach program on the Reservation.

Finally, development of people is complex: the IF program can respond to
this complexity by funding multiple institutions within a given project.
One of the best examples of this is the work by S. Shirey (DTM,
Washington, DC) and co-workers who published a study in Science relating
the formation of diamonds and evolution of the Kaapvaal-Zimbabwe Craton
to the seismic structure of the lithospheric mantle. This work utilized
multiple awards related to the IF program including the following
components: FS (to IRIS), EA to PI Carlson and Deines, and TS (to
Garnero). This study and publication was made possible by taking
advantage of multi-disciplinary infrastructure set up by the IF program.

**B2 -- [OUTCOME GOAL FOR IDEAS:]{.underline} Enabling \"discovery
across the frontier of science and engineering, connected to learning,
innovation, and service to society\".**

To develop new ideas about earth materials and processes, new
measurements are often necessary. Toward this end, most of the IF
funding, while generally expended for tools or tool development, creates
the environment necessary to enable discovery across the frontier of
science and engineering. We provide a few examples.

The IF program has provided funding to establish a multi-user Computed
Tomography (CT) laboratory at the University of Texas under the
direction of William Carlson. X-ray CT allows non-destructive imaging of
geological samples in three dimensions with micron-scale resolution. CT
imaging is revolutionizing our ability to image and quantify porosity
structure in rocks, our capability to identify structures and functions
of fossilized remains, and our ability to quantify size distribution of
crystals and other textures. Distributions of porosity in rocks are now
enabling theoreticians to develop new models to predict permeability in
rocks, which will enable the development of better models of groundwater
and hydrocarbon flow. This CT laboratory was the location of
measurements that led to a recent Nature article documenting the
characteristics of the oldest fossil bird\'s brain. Thus, an
infrastructure investment made by IF has implications for both the
applied sciences of water and oil resources and for the curiosity-driven
science of paleobiology.

The IF program also provided funding for the development of an
integrated multi-parameter regional observatory for the study of plate
boundary deformation (B. Romanowicz, UC Berkeley), to serve as prototype
for the Plate Boundary Observatory of Earthscope. This distributed
observatory across central California combines bore-hole strainmeters,
seismometers, and pressure gauges, as well as GPS receivers, from which
data are continuously telemetered, archived and freely distributed to
the community, providing a uniquely wide bandwidth of data to improve
our understanding of the physics of spatial and temporal distribution of
tectonic related strain, and ultimately to contribute to earthquake
hazards reduction.

**B3 - [Outcome Goal for TOOLS]{.underline}: Providing "broadly
accessible, state-of-the-art S&E facilities, tools and other
infrastructure that enable discovery, learning and innovation."**

The Program Officers are applauded for maintaining the high rate of
funding success of proposals (\~ 50%), which allows a diverse suite of
research and education tools to be made available to the EAR community
(e.g., EA, ITD, and FS proposals) as well as promoting the continued
building of a solid infrastructure (e.g., ECR and TS proposals). This
COV believes that the careful management of the EAR/IF program has
played a pivotal role in placing Earth Sciences in the United States at
the forefront of EAR research and education in the world. Funding of new
equipment and development of new techniques and instruments are both
essential. Strategic development of multi-user facilities throughout the
country has allowed a larger portion of the community access to
cutting-edge research and education opportunities. The innovative use of
funds leveraging (e.g., through the MRI program, cost-sharing with other
NSF programs and with other agencies) has been exemplary and has
increased the effectiveness of the EAR/IF. We are confident this will
continue. We do urge the Program Officers to keep a careful eye on
program balance because we note a continued increase in EA proposals
while the absolute dollar amount allocated to such proposals has
remained relatively constant.

Some examples of prominent tools developed during this funding period
are notable. For example, Dr Russ Hemley of Carnegie\'s Geophysical Lab
was funded to develop a successful method to grow very hard and tough
diamond single crystals rapidly by microwave plasma chemical vapor
deposition. This type of development will benefit the entire
high-pressure geoscience research community, which relies heavily on the
availability of high quality diamonds. Under the IRIS umbrella (D.
Simpson, PI), and as part of the Global Seismic Network, a unique
seismological observatory was deployed in boreholes located 8 km from
the South Pole (SPRESSO= South Pole Remote Earth Science and
Seismological Observatory). In addition to providing crucial coverage in
the southern hemisphere for studies of deep earth structure, this
observatory serves as an outpost for seismological and other geophysical
experiments requiring quiet conditions.

**B4 - [OUTCOME GOAL for ORGANIZATIONAL EXCELLENCE]{.underline}:
Providing "an agile, innovative organization that fulfills its mission
through leadership in state-of-the-art business practices."**

It has been educational for this COV to see how the EAR/IF program is
managed. We extend our thanks to the Program Officers for their
ceaseless efforts in ensuring that the EAR/IF dollars have the greatest
impact on the EAR community. We are also greatly impressed with the
leveraging of funds that occurs between different EAR programs within
NSF and with other agencies (e.g., NASA). We urge continued expansion of
this practice, especially in light of the current economic climate where
budget cuts and/or level funding are likely for the next couple of
years. For example, we urge the Program Officers to continue to develop
relationships with other GEO programs and also with other NSF
directorates such that EAR/IF dollars can be leveraged effectively. We
also urge that relationships with more agencies be formulated and we are
pleased to hear that the Program Officers are trying to re-establish
such a relationship with the Keck Foundation.

**PART C - OTHER TOPICS**

C1 - Please comment on any program areas in need of improvement or gaps
(if any)

within program areas.

1\) Disciplinary representation on panels does not match the
disciplinary balance of

awards, possibly because the current procedure for choosing panelists
does not take

into account the distribution of proposals across emphasis areas for a
given proposal

cycle. Suggestions from the COV include having a reduced number of
standing panel

review members that could be supplemented from a pool of alternates,
depending upon

the types of proposals received.

2\) Reviewers, panels, and occasionally the POs act inconsistently with
respect to

balancing Intellectual Merit and Broader Impacts. Mechanisms for
increasing the

awareness of reviewers and panel members in terms of Broader Impacts are
outlined in

this report.

3\) Site visits, often used for large facilities, are deemed
particularly important for

evaluation of large expenditures of facility support money, but are not
officially

documented through site visit reports. We strongly recommend official
documentation

of each site visit by the POs.

4\) Because it is difficult for a COV to assess the appropriateness of
award balance among

facilities and smaller awards given the short assessment period, other
mechanisms to

allow longer-term analysis of this balance should be considered.

5\) Where disciplinary communities are not receiving award support for
large facilities,

POs should consider mechanisms to educate these communities about such

opportunities. One way to help nucleation and growth of larger
facilities is to consider

more ample funding for technicians for successful multi-user facilities
as opposed to

new support for new instrumentation in single PI laboratories.

6\) Current and future budget pressures should be exerted on both small
and big awardees

appropriately, so that budgetary cutbacks or increases do not
disproportionately affect

one type of awardee more than another: this is particularly important
inasmuch as

awardees from large facilities may be able to muster more political
muscle in times of

fiscal cutbacks compared to small awardees. This latter consideration is
particularly

important in that different communities are largely served by the large
and small

awards.

7\) Money leveraged from outside of IF (e.g. from the Major Research
Instrumentation

program) has allowed maintenance of high success rates but has left
EAR/IF

vulnerable should outside funding decrease in the future. We urge
continued vigilance

with respect to this matter.

8\) The COV process forces a small committee up a steep learning curve
in a short time:

targeted collation of data before the meeting to address the specific
COV questions as

well as some overlap in committee membership from review to review could
improve

efficiency and effectiveness.

C2 - Please provide comments as appropriate on the program's performance
in meeting

program-specific goals and objectives that are not covered by the above
questions.

One statistic that is especially relevant to assessing the health of the
program would be the amount of investment in obtaining/building
facilities (EA, FS, ITD proposals) compared to that invested in
maintaining such facilities (TS proposals). At this time, the relative
dollar amounts invested in TS is trivial compared to the amount going to
EA, FS and ITD. The reason for this may be the lack of competitive
proposals in the TS area. We encourage the Program Officers to advertise
this aspect of the program to the greater EAR community, as it is a
superb way to protect and extend the infrastructure investment already
made by NSF EAR/IF.

C3 - Please identify agency-wide issues that should be addressed by NSF
to help

improve the program's performance.

It is evident that the Program Officers have done a fantastic job at
creating cooperative relationships with and leveraging funds from some
other NSF GEO programs. We urge that the POs continue to collaborate and
leverage funds within the GEO directorate where appropriate.
Furthermore, we think it is essential that this collaboration also occur
between the *different* directorates within NSF, as research projects
and facilities are becoming increasingly more inter-disciplinary. This
will likely be the future of research and education infrastructure, as
can be seen by the creation of NSF/DOE Environmental Molecular Science
Institutes. The EAR/IF program needs to be prepared for this.

C4 - Please provide comments on any other issues the COV feels are
relevant.

This COV feels that all pertinent issues have been addressed by our
comments to other questions.

C5 - NSF would appreciate your comments on how to improve the COV review
process,

format, and report template.

The COV appreciates the tremendous amount of work put in by the Program
Officers in preparing for this review. As a suggestion to help future
COVs, however, we feel that the program statistics could be collated and
printed up in a way so as to tailor them to the specific questions on
the COV template (e.g., Section A. 4). Toward this end, it would also be
helpful if the template questions were numbered/lettered throughout,
since any COV will be assigning these questions for writing among
committee members. (This COV added its own lettering system as a guide).
It would help both COVs and POs if an electronic database were developed
to keep track of these statistics.

The COV was also provided with some project reports, but we saw no
collated statistics from such reports. Are these data collated and used
in performance evaluations? Although such reports may not be
representative of the award impact in many cases (for example, the
impact of equipment acquisition is felt well beyond the lifetime of the
EAR/IF award period), collated statistics from final reports would be an
indicator of success or failure of the IF program. Some mechanism for
judging the overall long-term impact based on information such as number
of users of a facility (including minority status); publications; the
number of graduate and undergraduate students using the facility
(including minority status) would help a future COV to assess
infrastructure funded by IF. Using such indicators could help to assess
the IF program.

In addition, we also note the steep learning curve for all COV members
since none had experience with the process. It might be appropriate that
one or two members from the previous COV be included in the next one for
\"corporate\" memory (for example, the chair of the next COV panel could
be selected from the membership of this one). Obviously, this is
difficult in that a member of the COV must also be on the GEO Advisory
Committee. Staggering membership in some way so that members remained on
the committee for more than one rotation would greatly help in the COV
review process and in the acquisition of the most relevant statistics
and specific proposals needed for the review.

In conclusion, every member of the COV learned about NSF and about IF,
and we wish to convey our thanks for the stimulating three days spent at
NSF during the evaluation.

**SIGNATURE BLOCK:**

**\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\
**

For the *EAR/Instrumentation & Facilities COV*

Professor Susan Brantley, Chair

**EAR/IF COV MEMBERS (August 18-20, 2004)**

### NAME Expertise

Dr. Sue Brantley (**[Chair]{.underline}**) Environmental biogeochemistry

Department of Geosciences \[AC/GEO member\]

Pennsylvania State University

University Park, PA 16802

# Email: <brantley@essc.psu.edu>

Dr. David Hilton Stable/noble isotope geochemistry

Geosciences Research Division

Scripps Institution of Oceanography, UCSD

La Jolla, CA 92093-0244

# Email: <drhilton@ucsd.edu>

# 

# 

# Dr. Michael Lisowski Volcanology/geophysics

U.S. Geological Survey

1300 SE Cardinal Court, Bldg. 10, Suite 100

Vancouver, WA 98683

Email: <mlisowski@usgs.gov>

# 

# Dr. Sam Mukasa Radiogenic isotope geochemistry

Department of Geological Sciences

# University of Michigan 

Ann Arbor, MI 48109-1063

Email: <mukasa@umich.edu>

# 

# Dr. Clive Neal Igneous petrology/geochemistry

Department of Civil Engineering & Geological Sciences

# University of Notre Dame 

Notre Dame, IN 46556

Email: <neal.1@nd.edu>

# 

# Dr. Barbara Romanowicz Seismology

Department of Earth & Planetary Science

# University of California - Berkeley 

Berkeley, CA 94720-4767

Email: <barbara@seismo.berkeley.edu>

Dr. Tonie M. van Dam Geodesy

European Center for Geodynamics & Seismology

19 Rue Josy Welter

L-7256 Walferdange

Luxembourg

Email: <tvd@ecgs.lu>

The COV also was to include one further member, listed below, who could
not attend due to personal reasons. Dr Burnley did not participate in
the formatin of the report.

Dr. Pam Burnley Mineral physics/education

Department of Geology

Georgia State University

Atlanta, Georgia 30303

# Email: <burnley@gsu.edu>

[^1]: To be provided by NSF staff.

[^2]: To be provided by NSF staff.