# Presentation: 404117

## Peer Reviewof ARS Research Projects

- Office of Scientific Quality Review
- Steven Huber, Scientific Quality Review Officer**	**

- A Brief Overview of the Process

## OSQR  Web Sitehttp://www.osqr.ars.usda.gov

- National Programs
- Schedules of Reviews
- Forms and Resources *(TIPS!)*
- Reviewer Information
- Peer Review status
- Manual (online comments soon)
- Staff

## Topics to Cover

- Background
- Role of the Research Leader
- The Prospectus & Project Plan
- General writing tips
- The panel process
- Outcomes

## New Mandate for this process

- The Agricultural Research, Extension and Education Reform Act of 1998 (public law 105-185):
- ARS projects be evaluated by panels composed predominantly of non-ARS scientists
- With oversight from the REE Advisory Committee

## Intent of Peer Review

- To obtain constructive, independent, expert feedback and advice on ways to improve the scientific and technical merit of each project plan
- NOT funding issues

## Fundamental Questions

- Panelists are asked to evaluate:
**MERIT** **AND** **SIGNIFICANCE **
-  	(*Relevance to NP)*
**APPROACHES** **AND** **PROCEDURES**
**LIKELIHOOD OF SUCCESS**
*	**(feasibility)*
- Suggestions for improvements or solutions to problems are solicited

## Steps in the Process

**RL and Scientists develop Prospectus**

**RLs Prepare Project Plan with LSs**
**Submit Plan for approval**

**Peer Review**

**Response and Revision**
**Submit revised project plan**

**Re-review?**

- NPS

- AD

- OSQR

- AD/NPS

- OSQR

- 16 wk
- (24 wk)

- 18 wk
- (14 wk)

- 6 wk

- 6 wk

-  6 wk

## Scientist’s Responsibilities

- Communicate with the NP Team.
- Scope of the project.
- Coordination with other projects.
- QUALITY of the Project Plan.
- Many are ‘poorly’ written.

## Project Plan Template

- Title and Investigator(s)
- TOC (coming soon!)
- Project summary
- Objectives (< 1⁄2 p)
- Need for research  (1p)
- Scientific Background  (5-6 p)
- Approach & Procedures (6-12 p)
- Milestones & Outcomes ( 1 p)
- Literature Cited
- Past Accomplishments
- Issues of Concern statements
- Appendices

***15 - 25 pages***

## Scientific Background 

- Focus on the *pertinent* literature
- Show linkage/coordination with other similar projects in the NP (or other NPs) and with minority-coded projects in the MU.

## GENERAL TIP

- Use illustrations (figures, schemes, etc) to enhance the Plan. (Up to 2 pages does not count against page limit).
- Preliminary data
- How your project fits with others
- Working models
- Experimental design/treatments

## Approach and Procedures—Tips

- Lack of *necessary* detail is the most common criticism.
- Tell what will be done, by whom, and what will result. Identify staff involved with key portions of the project.
- If an objective is supported by a grant, mention that.

## The Panel

**Panelists**
- 	**Primary reviewers**
**	****Secondary reviewers**

- Panel Chair

- Ad-hoc reviewers may be used

- OSQR Officer
- Provides orientation
- Observes
- Debriefs
- Receives results

## Panel Chair Selection

- SUGGESTIONS
- for Non-ARS Panel Chair
- from NPS, ADs, etc.

- OSQR Officer
- Selects Chair

- OSQR Officer
- Consults inside
- and outside
- ARS

## Panelist Selection

- SUGGESTIONS for Panelists
- from NPS, ADs, RLs, Lead Scientists,
- University Scientists, Customers, *etc*.

- OSQR Officer
- Approves Panel*

- Panel Chair (PC)
- Makes
- recommendations to
- OSQR Officer

- *Based on PC recommendations and diversity requirements

## Suggested Panelists

- Use this form!

## The Panel Meeting Process

_**Panel Discussion**_
**Primary reviewer presents the plan input from secondary reviewer, panelists, and ad hoc reviewer(s).**

**Panelists and Panel Chair assign an ****action class**** to the project**.

**Turn in**
**to SQR **
**Officer**

**Primary reviewer modifies the ****review**** ****recommendation**** on disk**

****** ****1 hour**

**On breaks, evenings**

**Panel Chair**
** ****Approval**

**Turn in**
**to SQR **
**Officer**

## Action Classes*

- Each panelist individually provides overall ‘Action Class’ assignment:
**No revision required**. Minor revision might be made.
**Minor revision required**. Objectives fit the NP action plan; approaches to all objectives are sound. Project is feasible.
**Moderate revision required**.  Moderate revision of an objective and/or one approach needed.  Project is feasible.
**Major revision required**. Should be sound and feasible after major revision. (May be re-reviewed).
**Not feasible**. Deficiencies in expertise or facilities or major flaw.

- *Action Class Matrix will be provided

**Notes:**
Pass out the Action Class matrix.  Also see page 53 of the OSQR Manual.

## After the Panel Meeting

- If the composite Action Class is:
- No revision required
- Minor revision required
- Moderate revision required
- Major revision required
- Not feasible

**Scientists solicit comments and revise the Plan within 6 weeks**

**Revised Plans will be revised AND re-reviewed**

**Revise for immediate re-review; delay revision until major resource changes (*****e.g*****., personnel) are made; or terminate**

## Responding to the  Review Recommendations 

- Project Title:_____________________________________________________________
- CRIS #_______________________    National Program__________________________
- Lead Scientist:_________________________________ Reviewer #_________________
- PEER REVIEW OF ARS RESEARCH PROJECT
- 2. Adequacy of Approach and Procedures:
- Please comment on the following:  Are the hypotheses and/or plan of work well conceived?  Are the experiments, analytical methods, and approaches and procedures appropriate and sufficient to accomplish the objectives?  How could the approach or research procedures be improved?
**Please list and number each significant recommendation being made.  Be sure to briefly state the rational or basis for suggestions made or questions raised.  Each recommendation can  include several specific questions you believe should be addressed by the lead scientist.**
**1. Objective 1--The hypothesis being tested is only one of several that should be considered.  For example......**
- ARS Response:

## ARS Responses to Review Recommendations

- ARS scientists must respond to each major recommendation.
-  TONE of the response:  should be receptive, 	not defensive or condescending.
- CONTENT: make all reasonable efforts to 		accommodate suggestions made.  Lack of 	adoption must be justified.  Should ‘stand 	alone.’

- The completed forms go back to the panelists!

## ‘Re-Review’ of Project Plans

- Purpose:
- 1) independent evaluation of response, and
- 2) maintain credibility with
-   community.

## NP108, NP302, NP202  Panel Statistics

- 7 Panel Chairs and 44 Panelists (51 total)
- 1 ARS employee
- 33 employed by universities
- 7 employed by private industry, trades groups, or self-employed
- 10 Other federal agencies (U.S. and Canada)

## NP108, NP302, NP202 Panel results

-                       Action Class

- Number of Projects

## Comments from Panelists

- Think the peer review process is important for the agency
- Appreciated the positive approach of the ARS peer review process.
- Enjoyed the experience (opportunity for them to discuss science with peers).
- Hope the review recommendations will be useful and incorporated.

## Benefits of Peer Review

- Better short- and long-term thinking and planning of research (if done properly!).
- Increased communication among scientists, ADs and NPLs.
- Increased collaboration (especially among ARS labs).
- Increased quality—perspectives from peer reviewers.
- Greater ability to compete for competitive funds.

## OSQR Web Sitehttp://osqr.ars.usda.gov/

-  National Programs
-  Schedules of Reviews
-  Forms and Resources
-  Reviewer Information

-  Peer Review Status
-  Manual
-  Staff