**National Oceanic and**

**Atmospheric Administration**

**Office of**

**Oceanic and Atmospheric Research**

**Guidelines for Laboratory**

**Review Team Members**

**January 2009**

**1. Introduction**

Laboratory scientific reviews are conducted every four years to evaluate
the **quality**, **relevance**, and **performance** of research
conducted in Oceanic and Atmospheric Research (OAR) laboratories to both
internal and external interests, and to help strategically position the
laboratory in its planning of its future science. These reviews are
intended to ensure that OAR laboratory research is linked to the
National Oceanic and Atmospheric Administration (NOAA) Strategic Plan,
relevant to NOAA Research mission and priorities, and consistent with
NOAA planning, programming, and budgeting.

These guidelines have been prepared using experience gained from
previous laboratory reviews. The goal of the guidelines is to clarify
your role and assist in the organization of the work of the review team.
The guidelines cover the process from when you receive the invitation
letter to participate on the review team to submission of the summary
report of the review team.

**2**. **Science Areas in Review and Charge to the Review Team**

Each member of the review team should have received the "charge to the
reviewers" document. The charge covers the following topics: purpose of
the review, scope of the review, focus areas for the review including
questions to be addressed by the review team, proposed schedule
including the dates of the review and time frame for delivery of the
review report as well as the time commitment for reviewers and review
team resources. Each member is asked to complete an individual review
report on two or more review areas and provide the reviews to the Chair.
The Chair, as a federal review team member, will summarize the
individual reports of the review team, but will not attempt to seek a
consensus of the review team on any findings or recommendations. We are
asking for a summary, not consensus, report to conform to the Federal
Advisory Committee Act (FACA) regulations. FACA regulations allow
consensus reports only from external groups established under FACA, and
our short-term review teams do not meet these criteria.

Each member of the review team should also have received a conflict of
interest disclosure form which should be returned to OAR headquarters.

**3. Resources for the Review Team**

A Deputy Assistant Administrator of OAR will provide the resources
necessary for you and the review team to complete its work. All
laboratory review materials and presentations for the review will be
posted to a website in advance of the review. The web site will contain
background documents from NOAA (e.g., NOAA Research 5 Year Plan) and
background data on the lab, including several "indicators of
preeminence" (e.g., publications, awards, scientific leadership,
patents). The presentation files will be provided to review team members
on the first day of the review. In addition, reviewers will be asked if
they would like to receive a binder with printed copies of presentations
in advance of the review. You will also be provided a template (form) on
which to complete your review observations, findings, and
recommendations.

**4. Logistics and Agenda for the Review**

Laboratory staff will contact you to arrange travel to the review and
all travel arrangements will be paid for by OAR. Please provide the
laboratory with your intended dates of travel and other particulars by
the requested due dates to ensure all arrangements can be made
satisfactorily. The laboratory will reserve a block of hotel rooms for
the reviewers, but you will be asked to cover all your travel expenses
(except air fare) upfront and will be reimbursed, usually through direct
deposit to your bank, after laboratory staff complete the travel
reimbursement forms with your help. Some receipts may be needed for
reimbursement. If you have not been the recipient of federal travel
reimbursement before, you will need to register as a U.S. government
vendor to receive your travel reimbursement. The laboratory travel staff
will do that for you, but you will have to provide them with some
personal identifying information, including the routing number for your
bank account for direct deposit of the reimbursement check. For non-U.S.
reviewers, you will be sent a check for travel cost reimbursement.
Travel schedules should be chosen to allow you to attend all scheduled
review sessions. The review agenda will include presentations and
discussions that will provide information on the science areas to be
reviewed and the questions to be addressed by the review team.

Laboratory staff may also ask for information for building security in
advance of the review, particularly for reviewers who are not U.S.
citizens.

**5. Teleconferences Prior to the Review**

Two teleconferences will be scheduled to discuss the review process and
answer any questions you may have approximately two months and one month
prior to the review. In addition to the review team members, attendees
will include the OAR Deputy Assistant Administrator (DAA), the OAR
Headquarters coordinator and management from the laboratory. On the
first call, the charge to the review team and the draft agenda for the
review will be discussed as well as any other questions reviewers may
have on the process. The focus for the second call will include
information provided on the website, presentation materials, the final
review agenda, and the review reports.

**6. During the Review**

Reviews are held over a three to four day period. On the morning of the
first day, you will meet at breakfast with the OAR Assistant
Administrator (AA) and DAA to discuss any final issues before the
review. Generally the first morning will include an overview presented
by the laboratory director. Material is then presented for each of the
primary science areas of the laboratory. These presentations may include
PowerPoint presentations, poster sessions and/or facility tours. Time
will be built into the review schedules for questions and discussion
following presentations. Interactive dialogue and discussion during all
of the sessions is strongly encouraged.

Time will also be set aside for reviewer closed sessions to meet with
laboratory management, as well as with bench scientists without
management present. The laboratory will facilitate separate sessions
with bench scientists and the laboratory management team that are
designed for you to address the research **quality**, **relevance,** and
**performance**. Please use these closed sessions to probe more deeply
into the operations of the laboratory.

Time will also be set aside for reviewers-only closed session. The goal
of the reviewers-only sessions is to provide time for the review team to
discuss any presentations or information provided and to identify
additional information needed or issues that need to be clarified. The
closed sessions also provide an opportunity to discuss the
process/timeline for preparing reports and feedback for the preliminary
report to laboratory management at the end of the third day. At any time
during the review, you should feel free to request additional
information or clarifications from laboratory staff.

**7. Preparation and Submission of the Review Report**

We ask that you complete your individual report on two or more science
areas covered in the review. A reporting form is attached that provides
the questions to be assessed for each science area in the review and
expandable text boxes for you to enter your observations/findings as
well as specific recommendations for the laboratory to review and
consider incorporating in its research and operations. The chair, a
federal reviewer, will compile a summary report from the individual
reports. The summary report is requested within 45 days of the review
and should be submitted by the Chair to the Deputy Assistant
Administrator and the Laboratories and Cooperative Institutes (LCI)
coordinator. OAR will have 30 days from the submission of the draft
report to review it for technical/factual corrections. Any
technical/factual corrections will be sent back by OAR to the review
team members to make adjustments, as appropriate, to the final
individual and summary reports. An example of a recent review report
will be provided to the review team for their information.

**8. Uses for and Distribution of the Review Report**

As outlined in the "purpose of the review" section of the "charge to
reviewers", laboratory scientific reviews are conducted to help
strategically position the laboratory in planning its future science and
to ensure that laboratory research is linked to the National Oceanic and
Atmospheric Administration (NOAA) Strategic Plan, relevant to NOAA
Research mission and priorities, and consistent with NOAA planning,
programming, and budgeting. After submission of the final report by the
review team, the laboratory will be asked to review the report and
prepare a plan, to be discussed with OAR management, to incorporate
recommendations into laboratory research and operations.

The summary report will be a public document and may be distributed to
internal NOAA and external audiences. Your individual reports will not
be made public, and will only be used by OAR as background for the
summary report. Internal distribution of the individual reports will be
limited.

Attachments:

Reporting Template

Contact Information

National Severe Storms Laboratory

**Science Research Review**

February 17-19, 2009

Norman, Oklahoma

**Evaluation Forms**

**Evaluation Focus**

**(**from "Charge to Reviewers" document)

> 1\. ***Quality:*** Assess the quality of the laboratory's research and
> development. Assess whether appropriate approaches are in place to
> ensure that high quality work will be performed in the future. Assess
> progress toward meeting OAR's goal to conduct preeminent research as
> listed in the "Indicators of Preeminence."

-   How does the quality of the laboratory's research and development
    rank among Research and Development (R&D) programs in other U.S.
    federal agencies? Other science agencies/institutions?

-   Are appropriate approaches in place to ensure that high quality work
    will be done in the future?

> Indicators of Preeminence: Types of Indicators can include the
> following; not all may be relevant to each laboratory.
>
> a\. A lab's total number of refereed publications per unit time and/or
> per scientific Full Time Equivalent staff (FTE).
>
> b\. A list of technologies (e.g. observing systems, information
> technology, numerical modeling algorithms) transferred to
> operations/application and an assessment of their significance/impact
> on operations.
>
> c\. The number of citations for a lab's scientific staff by individual
> or some aggregate.
>
> d\. A list of awards won by groups and individuals for research,
> development, and/or application.
>
> e\. Memberships and involvement in prestigious organizations (e.g.,
> the National Academy of Sciences, National Academy of Engineering, or
> fellowship in the American Meteorological Society, American
> Geophysical Union or the American Association for the Advancement of
> Science etc.).
>
> f\. Service of individuals in technical and scientific societies such
> as journal editorships, election to boards or executive level offices,
> service on U.S. interagency groups, service of individuals on boards
> and committees of international research-coordination organizations.
>
> g\. A list of research products, information and services and an
> assessment of their impact by end users, including participation or
> leadership in national and international state-of-science assessments.
>
> h\. Evidence of collaboration with other national and international
> research groups, both inside and outside of NOAA as well as
> reimbursable support from non-NOAA sponsors.
>
> i\. Significance and impact of involvement with patents, Cooperative
> Research and Development Agreements (CRADAs) and other activities with
> industry.
>
> j\. Other forms of recognition from NOAA information customers such as
> decision makers in government, private industry, the media, education
> communities, and the public.
>
> k\. Contributions of data to national and Global Earth Observing
> System of Systems (GEOSS)-related data bases and programs, and
> involvement in international quality-control activities to ensure
> accuracy, precision, inter-comparability, and accessibility of global
> data sets.
>
> 2\. ***Relevance:*** Assess the degree to which the research and
> development is relevant to NOAA's mission and of value to the Nation.

-   Does the research address existing (or future) societally-relevant
    needs (national and international)?

-   How well does it address issues identified in the NOAA research
    plans or other policy or guiding documents?

-   Are customers engaged to ensure relevance of the research?

-   Are there R&D topics relevant to national needs that the laboratory
    should be pursuing but is not? Are there R&D topics in NOAA and OAR
    plans that the laboratory should be pursuing but is not?

> 3\. ***Performance:*** Assess the overall effectiveness with which the
> laboratory plans and conducts its research and development, given the
> resources provided, to meet NOAA Strategic Plan objectives and the
> needs of the nation. The evaluation will be conducted within the
> context of three sub-categories: research leadership and planning,
> effectiveness, and transition of research to applications.
>
> 3a. *[Research Leadership and Planning]{.underline}.* Assess whether
> the laboratory has clearly defined objectives, scope, and
> methodologies for its key projects.

-   Does the laboratory have clearly defined and documented scientific
    objectives, rationale and methodologies for key projects?

-   Has the scope of key projects been identified including methods for
    determining when areas of investigation should end or be
    transitioned to operations or information services?

> *3b. [Efficiency and Effectiveness]{.underline}*[.]{.underline} Assess
> the efficiency and effectiveness of the laboratory's research and
> development, given the laboratory's goals, resources, and constraints
> and how effective the laboratory is in obtaining needed resources
> through NOAA and other sources.

-   Does the laboratory execute its research in an efficient and
    effective manner?

-   Is the laboratory organized and managed to optimize the conduct and
    planning of research, including the support of creativity?

```{=html}
<!-- -->
```
-   How well integrated is the work with NOAA's planning and execution
    activities? Are there adequate inputs to the planning process of
    NOAA's Programming, Planning and Budgeting and Execution System
    (PPBES)?

    -   Is the proportion of the external funding appropriate relative
        to its NOAA funding?

    -   Are human resources adequate to meet current and future needs?
        Is the laboratory organized and managed to ensure diversity in
        its workforce?

    -   Are appropriate resources and support services available?

> 3c. *[Transition: How well has the laboratory delivered
> products?]{.underline}* Assess laboratory's effectiveness in
> transitioning and/or disseminating its research into applications
> (operations and /or information services).

-   How well is the transition of research to applications and/or
    dissemination of knowledge planned and executed?

-   Are there appropriate interactions with stakeholders and customers?
    Are end users of the research and development involved in the
    planning and delivery of applications and/or information services?

-   Are the research results communicated to stakeholders and the
    public?

**Evaluation Worksheet**

(Note in WORD the boxes below will expand to fit the text)

+---------------+------------------------------+-----------------+---+
| **Research    | Weather Radar Research       |                 |   |
| Area:**       |                              |                 |   |
+---------------+------------------------------+-----------------+---+
| **Reviewer:** |                              |                 |   |
+---------------+------------------------------+-----------------+---+
| **QUALITY**   |                              |                 |   |
| (Reference    |                              |                 |   |
| material      |                              |                 |   |
| provided at   |                              |                 |   |
| web site and  |                              |                 |   |
| in the        |                              |                 |   |
| binders       |                              |                 |   |
| provided.)    |                              |                 |   |
|               |                              |                 |   |
| **Comments    |                              |                 |   |
| and           |                              |                 |   |
| observation   |                              |                 |   |
| s/findings:** |                              |                 |   |
+---------------+------------------------------+-----------------+---+
|               |                              |                 |   |
+---------------+------------------------------+-----------------+---+
| **RELEVANCE** |                              |                 |   |
| (Reference    |                              |                 |   |
| material      |                              |                 |   |
| provided      |                              |                 |   |
| during the    |                              |                 |   |
| briefings.)   |                              |                 |   |
|               |                              |                 |   |
| **Comments    |                              |                 |   |
| and           |                              |                 |   |
| observation   |                              |                 |   |
| s/findings:** |                              |                 |   |
+---------------+------------------------------+-----------------+---+
|               |                              |                 |   |
+---------------+------------------------------+-----------------+---+
| **            |                              |                 |   |
| PERFORMANCE** |                              |                 |   |
| (Reference    |                              |                 |   |
| material      |                              |                 |   |
| provided      |                              |                 |   |
| during the    |                              |                 |   |
| briefings.)   |                              |                 |   |
|               |                              |                 |   |
| **Comments    |                              |                 |   |
| and           |                              |                 |   |
| ob            |                              |                 |   |
| servations:** |                              |                 |   |
+---------------+------------------------------+-----------------+---+
|               |                              |                 |   |
+---------------+------------------------------+-----------------+---+
| **Re          |                              |                 |   |
| commendations |                              |                 |   |
| for Weather   |                              |                 |   |
| Radar         |                              |                 |   |
| Research**    |                              |                 |   |
|               |                              |                 |   |
| **Please      |                              |                 |   |
| provide       |                              |                 |   |
| specific      |                              |                 |   |
| re            |                              |                 |   |
| commendations |                              |                 |   |
| for your      |                              |                 |   |
| observatio    |                              |                 |   |
| ns/findings** |                              |                 |   |
+---------------+------------------------------+-----------------+---+
|               |                              |                 |   |
+---------------+------------------------------+-----------------+---+

Evaluation Worksheet

+---------------+------------------------------+-----------------+---+
| **Research    | Hazardous Weather Forecasts  |                 |   |
| Area:**       | and Warnings Research        |                 |   |
+---------------+------------------------------+-----------------+---+
| **Reviewer:** |                              |                 |   |
+---------------+------------------------------+-----------------+---+
| **QUALITY**   |                              |                 |   |
| (Reference    |                              |                 |   |
| material      |                              |                 |   |
| provided at   |                              |                 |   |
| web site.)    |                              |                 |   |
|               |                              |                 |   |
| **Comments    |                              |                 |   |
| and           |                              |                 |   |
| ob            |                              |                 |   |
| servations:** |                              |                 |   |
+---------------+------------------------------+-----------------+---+
|               |                              |                 |   |
+---------------+------------------------------+-----------------+---+
| **RELEVANCE** |                              |                 |   |
| (Reference    |                              |                 |   |
| material      |                              |                 |   |
| provided      |                              |                 |   |
| during the    |                              |                 |   |
| briefings.)   |                              |                 |   |
|               |                              |                 |   |
| **Comments    |                              |                 |   |
| and           |                              |                 |   |
| ob            |                              |                 |   |
| servations:** |                              |                 |   |
+---------------+------------------------------+-----------------+---+
|               |                              |                 |   |
+---------------+------------------------------+-----------------+---+
| **            |                              |                 |   |
| PERFORMANCE** |                              |                 |   |
| (Reference    |                              |                 |   |
| material      |                              |                 |   |
| provided      |                              |                 |   |
| during the    |                              |                 |   |
| briefings.)   |                              |                 |   |
|               |                              |                 |   |
| **Comments    |                              |                 |   |
| and           |                              |                 |   |
| ob            |                              |                 |   |
| servations:** |                              |                 |   |
+---------------+------------------------------+-----------------+---+
|               |                              |                 |   |
+---------------+------------------------------+-----------------+---+
| **Re          |                              |                 |   |
| commendations |                              |                 |   |
| for Hazardous |                              |                 |   |
| Weather       |                              |                 |   |
| Forecasts and |                              |                 |   |
| Warnings**    |                              |                 |   |
|               |                              |                 |   |
| **Please      |                              |                 |   |
| provide       |                              |                 |   |
| specific      |                              |                 |   |
| re            |                              |                 |   |
| commendations |                              |                 |   |
| for your      |                              |                 |   |
| observatio    |                              |                 |   |
| ns/findings** |                              |                 |   |
+---------------+------------------------------+-----------------+---+
|               |                              |                 |   |
+---------------+------------------------------+-----------------+---+

Evaluation Worksheet

+---------------+------------------------------+-----------------+---+
| **Research    | Hydrometeorology Research    |                 |   |
| Area:**       |                              |                 |   |
+---------------+------------------------------+-----------------+---+
| **Reviewer:** |                              |                 |   |
+---------------+------------------------------+-----------------+---+
| **QUALITY**   |                              |                 |   |
| (Reference    |                              |                 |   |
| material      |                              |                 |   |
| provided at   |                              |                 |   |
| web site)     |                              |                 |   |
|               |                              |                 |   |
| **Comments    |                              |                 |   |
| and           |                              |                 |   |
| ob            |                              |                 |   |
| servations:** |                              |                 |   |
+---------------+------------------------------+-----------------+---+
|               |                              |                 |   |
+---------------+------------------------------+-----------------+---+
| **RELEVANCE** |                              |                 |   |
| (Reference    |                              |                 |   |
| material      |                              |                 |   |
| provided      |                              |                 |   |
| during the    |                              |                 |   |
| briefings.)   |                              |                 |   |
|               |                              |                 |   |
| **Comments    |                              |                 |   |
| and           |                              |                 |   |
| ob            |                              |                 |   |
| servations:** |                              |                 |   |
+---------------+------------------------------+-----------------+---+
|               |                              |                 |   |
+---------------+------------------------------+-----------------+---+
| **            |                              |                 |   |
| PERFORMANCE** |                              |                 |   |
| (Reference    |                              |                 |   |
| material      |                              |                 |   |
| provided      |                              |                 |   |
| during the    |                              |                 |   |
| briefings.)   |                              |                 |   |
|               |                              |                 |   |
| **Comments    |                              |                 |   |
| and           |                              |                 |   |
| ob            |                              |                 |   |
| servations:** |                              |                 |   |
+---------------+------------------------------+-----------------+---+
|               |                              |                 |   |
+---------------+------------------------------+-----------------+---+
| **Re          |                              |                 |   |
| commendations |                              |                 |   |
| for           |                              |                 |   |
| Hyd           |                              |                 |   |
| rometeorology |                              |                 |   |
| Research**    |                              |                 |   |
|               |                              |                 |   |
| **Please      |                              |                 |   |
| provide       |                              |                 |   |
| specific      |                              |                 |   |
| re            |                              |                 |   |
| commendations |                              |                 |   |
| for your      |                              |                 |   |
| observatio    |                              |                 |   |
| ns/findings** |                              |                 |   |
+---------------+------------------------------+-----------------+---+
|               |                              |                 |   |
+---------------+------------------------------+-----------------+---+

Evaluation Worksheet -- Additional Comments and Recommendations

+---------------+------------------------------+-----------------+---+
| **Reviewer:** |                              |                 |   |
+---------------+------------------------------+-----------------+---+
| **Additional  |                              |                 |   |
| comments for  |                              |                 |   |
| OAR and       |                              |                 |   |
| laboratory    |                              |                 |   |
| management**  |                              |                 |   |
+---------------+------------------------------+-----------------+---+
|               |                              |                 |   |
+---------------+------------------------------+-----------------+---+
| **Additional  |                              |                 |   |
| comments and  |                              |                 |   |
| suggestions   |                              |                 |   |
| on conduct of |                              |                 |   |
| the review    |                              |                 |   |
| for use in    |                              |                 |   |
| future        |                              |                 |   |
| laboratory    |                              |                 |   |
| reviews**     |                              |                 |   |
+---------------+------------------------------+-----------------+---+
|               |                              |                 |   |
+---------------+------------------------------+-----------------+---+
| **Reco        |                              |                 |   |
| mmendations** |                              |                 |   |
|               |                              |                 |   |
| **Please      |                              |                 |   |
| provide       |                              |                 |   |
| specific      |                              |                 |   |
| re            |                              |                 |   |
| commendations |                              |                 |   |
| for your      |                              |                 |   |
| observatio    |                              |                 |   |
| ns/findings** |                              |                 |   |
+---------------+------------------------------+-----------------+---+
|               |                              |                 |   |
+---------------+------------------------------+-----------------+---+

Contact Information for the NSSL Science Review

OAR HQ LCI Coordinator: Michael Uhart (Acting)

<michael.uhart@noaa.gov>

301-734-1177

NSSL Review Coordinator: Kevin Kelleher, Deputy Director, NSSL

<kevin.kelleher@noaa.gov>

405-325-6900

NSSL Travel Coordinator for the Review Team- Kelly Lynn, NSSL

<Kelly.Lynn@noaa.gov>

405-325-6907