# Presentation: 425278

## The Research Scientist and ARS Project Peer Review

- Office of Scientific Quality Review
- (OSQR)
- Scientific Officer
- Frank C. Greene 2003 -
- Program Coordinator
- Marcia Moore

**OSQR**

## Goal of Project Peer Review

- Independent and expert evaluation of the scientific and technical merit of research project plans, and advice to enable their improvement
- Not evaluation of the Mission or National Programs of ARS, or of project budgets or personnel management

**OSQR**

## Foundations of ARS Peer Review

-  Scientific:  Scrutiny, advice and endorsement of experts in the field.
-  Business:  Enhanced knowledge and recognition of the excellence of ARS as an institution.
-  Law:  The Secretary shall establish procedures that ensure scientific peer review of all activities conducted by the Department.  (Public Law 105-185).

**OSQR**

## Office of Scientific Quality Review

- Mission:
- OSQR is responsible
- for planning and facilitating high quality scientific and technical peer review of all ARS prospective research project plans.

**OSQR**

## National Program Cycle

## ARS Research Project Approval 

- National Program Workshop develops Program Action Plan

- Research Scientists develop Project Plan

- OSQR Panel reviews Project Plan

- SQR Officer certifies Project Plan

- Research Scientists implement Project Research

**OSQR**

## Context of ARS Peer Review

**OSQR**

- Research must be relevant to an ARS National Program Action Plan
- Primary driver is the need to solve a problem, not investigator curiosity or idea novelty
- Projects are not in competition for funding
- Evaluation generates an “Action Class” recommendation, not a funding priority determination

## Review Criteria

- The Project Plan, as a stand-alone document, must convince a panel of experts that it describes research which has:
-  Suitable Experimental Approaches and Experimental Procedures
-  High Probability of Success
-  Merit and Significance

**OSQR**

## Things You Should Do

- Know the capabilities of MU/project scientists.  Challenge them to stretch.
- Know your NPL and National Program Action Plan.  Consult both thoroughly to confirm areas of great potential impact.
- Know the OSQR system.  Consult the web site references and staff.
- Organize and participate in MU-wide discussions to critique project plan drafts.

**OSQR**

## The Prospectus

-  It is a skeletal project plan for solution of problems in support of National Program objectives.
-  It contains all the justification, conceptual foundation, and experimental design of the full plan.
-  It influences the selection of panel Chairs, panel members, and primary/secondary reviewers.

**OSQR**

## The Project Plan

- Frames objectives to generate significant and definitive results
- Develops sound and modern experimental plans, testing hypotheses as appropriate
- Plans for and details contingencies
- Details and documents collaborations
- Should be subjected to informal peer review before submitting to OSQR

**OSQR**

## Project Plan Template

- Title and Investigator(s).......i
- Signature Page..................ii
- Table of Contents..............iii
- Project summary (250 words)....iv
- Objectives (< 1 p)...............1
- Need for research  (1-2 p)
- Scientific Background  (5-7 p)
- Approach & Procedures (6-15 p)
- Milestones & Outcomes ( 1 p)
- Literature Cited
- Past Accomplishments
- Issues of Concern statements
- Appendices (letters plus other material)

***15 - 30 pages***

**OSQR**

## Chief Criticisms to Date

- Apparent redundancy in research
- Perfunctory Contingency Plans
- Poor description of connection between disparate project objectives
- Incomplete documentation of collaborations

**OSQR**

##    Project Plan Components

- Express need both scientifically and in context of NP Action Plan
- Be concise in statement of research purpose
- Discuss potential benefits and anticipated products
- Identify relevant customers and stakeholders

- NEED FOR  RESEARCH: 1-2 pages

**OSQR**

##    Project Plan Components

- A* critical* review of *relevant* literature
- An overview of what is known and what is not known in the general project area
- Information showing how project “fills a gap”
- A CRIS search – with citations of major relevant projects (approx. 5)
- Preliminary data, if available

- SCIENTIFIC BACKGROUND: 5 to 7 pages

**OSQR**

## Project Plan Components

-   APPROACHES & PROCEDURES: 6 to12 pages

- Describe general experimental strategy.
- Illustrate clearly how stated objectives can be achieved (Who will do what, how, and when.).
- Utilize approaches and methods that are appropriate, modern, and technologically sound.
- Discuss advantages and limitations as appropriate
- Include contingencies and collaborations
*The single most critical section of the project plan.*

**OSQR**

##    Project Plan Components

- Identify overall results and impact of the completed project
- Predict significant intermediate research accomplishments expected
- Reinforce overall logic and organization of Experimental Plan
- State in discrete modules rather than as a continuum

-    Project Plan Components

**OSQR**

## Preparation for Panel

- Panelists selected by Chair
- Primary/Secondary Reviewers selected by Panel Chair
- Possible additional *ad hoc * reviewers selected
- Project Plans sent by OSQR to all Panelists six weeks before panel
- OSQR sends orientation letter to each panelist
- OSQR holds interactive panel orientation

- Preparation for Panel

**OSQR**

## The Panel Peer Review Process

- Primary and Secondary reviewers analyze plan in-depth, followed by all-panel discussion

- Panelists and Panel Chair assign  Action Classes to the project.

- Turn in
- to OSQR

- Primary reviewer prepares panel recommendations on disk

-  1 hour

- On breaks, evenings

- Panel Chair Approval

- Turn in
- to OSQR

**OSQR**

***Debriefing***

## Action Classes

**No revision required****.**  Already at highest quality level.  Project plan is feasible as written.  *[excellent].*
**Minor revision required** to increase quality to highest level. Project Plan is feasible as written*.  [very good].*
**Moderate revision required** to increase quality to highest level.  Plan is feasible as written.  *[good]*
**Major revision required** to achieve highest quality level. Plan is not feasible as written*.  [fair]*
**Not feasible**.  Project plan has major flaws or deficiencies*.*  Cannot be simply revised** **to produce a sound product.  *[poor]*

**OSQR**

## After Panel - ARS Response

- If the composite Action Class is:** **
- No revision required
- Minor revision required
- Moderate revision required
**Major**** ****revision**** **required
**Not feasible**

- Revise for immediate re-review; **delay revision** until major resource changes (*e.g*., personnel) are made; or terminate

- Revised Plans are re-reviewed

- Scientists solicit comments and revise the Plan within 6 weeks

## Proposal Score Distribution     n=636

**11/03**

**OSQR**

## Peer Review Reference Sources

- OSQR Web Site
-    (http://www.ars.usda.gov/osqr)
-    Program Overview
-    OSQR Project Plan Writing Video
-    Project Plan Information, Format
-    Peer Review Manual
- OSQR Staff
- National Program Staff

**OSQR**

## Pass Go, Collect $300/SY!

**OSQR**