# Presentation: 009859

## Peer Review Process

- Changes in the Process

## Changes in OSQR

- Manual
- Project Plan Elements
- Project Management and Evaluation
- Prior Accomplishments
- Milestone Table
- Panel Briefings
- Re-Reviews
- Ad Hoc Reviews

## Components

- Process
- What are we learning from reviews about the process and ARS Science?
- Procedures
- What have we changed to improve the process?

## What are we learning?

- Scores in the second round haven’t significantly changed from the first round.
- Need to place an emphasis on quality of the project plan
- Quality begins with and rests with the project team
- Signature page should carry a reflection of review

## Where are the problems?

**Where are the problems?**

## Comment Sampling “No Revision”

**Comment Sampling ****“No Revision”**

*...**studies will fill rather substantial knowledge gaps.*
*...**well written and comprehensive*
*...**well written project plan*
*...**well organized, cogently prioritized and comprehensive research plan*...

## Comment Sampling “Minor Revision”

**Comment Sampling ****“Minor Revision”**

*...**studies will fill rather substantial knowledge  **	**gaps.*
*...**well written and comprehensive*
*...**well written project plan*
*...**well organized, cogently prioritized and **	**comprehensive research plan*...
*...**well written, well organized, and easy to **	**follow.*

## Comment Sampling “Moderate Revision”

**Comment Sampling ****“Moderate Revision”**

*one of the better written*
*excellent discussion of technology *
*large research project plan but poorly organized.*
*concerns that some of the objectives can be completed by this team.*
*work and ability described as "adequate"*
*missing current information*
*hypotheses and plan well conceived, approaches appropriate*

## Comment Sampling “Major Revision”

**Comment Sampling ****“Major Revision”**

*...**a heroic course that...will not achieve the stated objectives*
*serious flaws in experimental designs...*
*General Lack of focus. Plan is so broad that it leaves out important details...not clear data will be relevant or interpretable.*
*lacks cohesion and clear direction*
*writing style, quality, and organization significantly detract...*
*Short on details. Not well planned. Need to take process seriously.*

## Comment Sampling “Not Feasible”

**Comment Sampling ****“Not Feasible”**

*approaches highly flawed, lack detail, many experiments duplicative*
*poorly written...difficult to discern what the authors want to do. *
*proposed research contradicts intention of the proposal.*
*exaggerations, incomplete reviews..., and misleading comments, as well as poor grammar and organization.*
*lack of knowledge...incomplete understanding ...fundamental misunderstanding...*
*Work does not address stated objectives.*

## Where are the problems?

- (Content)

## Where are the problems?

- (Content)

## Processes

- What are we learning that can help us do a better job in research planning?
- What can we do in the process to develop better plans?

## Panel Training

- Web-based panel training to introduce chairs and panelists to OSQR process
- Review of project objective development
- Updated National Program Action Plan
- Introduction to the National Program components

## 2. National Program developed.

***5. ******OSQR Review and Certification.***

***8. ******Retrospective evaluation.***

***1.****** Stakeholder Workshop (customers, scientists, National Program Staff, Center Directors, Area Directors).***

- Input

- Implementation

- Planning

- Assessment

- Input

***3. ******Program Priorities set (PDRAM).***

***4. ******Research  Project Plan describing research prepared.***

***6. ******Research initiated***

***7. ******Annual progress reviews***

***Congressional Mandate***

## Procedures

- Manual
- Project Plan
- 		*Milestone Table*
*	**	**Project Evaluation and Management*
*	**	**Prior Accomplishments*
- Web-based briefings and introduction to OSQR and National Programs
- Re-Reviews
- Ad Hoc Reviews

## The Goal of Peer Review

**    ****Enhance research through independent, expert examination of PROSPECTIVE plans for *****scientific and technical merit.***

## Importance to ARS

**Congressional intent was to enhance research.**
**OMB Analyses. President’s initiative places high emphasis on peer review.**
**Provides for Agency Accountability**
**Enhances ARS Image**
***A strong image for ARS can translate***
***into enhanced funding for research***

## What is a Research Plan?

- A

- The “story” of your research
- An integrated and logical presentation
- A clearly presented argument
- A “Marketing” brochure (and more!)

*Your plan should present your work in a clear, logical fashion that will transmit enthusiasm for the work and convince the reader that your team is best suited to perform it.*

## Project Review Criteria

- Adequacy of Approach and Procedures
- Probability of Successfully Accomplishing the Project’s Objectives
- Merit and Significance

## Reviewers want to know...

- What is the problem?
- Why is it important?
- Where are you going with it?
- How are you going to get there?
- And how will you know you have arrived?

## Document Outline

- Signature Page..........................page 2
- Table of Contents.......................page 3
- Project summary (250 words)...page 4
- Objectives.................................page 5
- Need for research  (1-2 p)
- Scientific Background  (5-7 p)
- Prior Accomplishments (2 p)
- Approach & Procedures (6-15 p)
**Milestone Table** (1-3 p)
- Literature Cited
- Past Accomplishments of Project Team Members
- Issues of Concern statements
- Appendices (letters plus other material)

***15 - 30 pages***

**Document Outline**

##    Project Plan Components

- Express need scientifically AND in the context of NP Action Plan
- Be concise in statement of research purpose
- Discuss potential benefits and anticipated products
- Identify relevant customers and stakeholders
- Briefly note the principle methods you will utilize (e.g., ...using microarray technologies we will elucidate...”

- NEED FOR  RESEARCH: 1-2 pages
*Where are you going?*

##  Project Plan Components

- Highlight what is known and **not** known
- Show how this creates a Rationale for the proposed objectives
- Show how project  will fill identified knowledge gaps
- Not more than 1/3 of project plan length
- Include mention of other, similar projects within and outside CRIS
- Cite preliminary data, if available

- Scientific Background: 5-7 pages
*Why are you going there?*

## Project Plan Components

- Prior Accomplishments: 1-2 pages
*What have you done before?*

- Name prior project terminated within two years
- Major objectives and accomplishments
- Prior project investigators
- Impact of prior work (science, technology, users)
- Pertinent publications
-     Mark all publications from this research on the investigator
-      publication list

## Project Plan Components

-   Approaches & Procedures: 6-12 pages
***How are you going to get there?***

**Set out your Experimental design**
**Describe approaches and methods that are **
**     ****appropriate, modern, and sound **
**Discuss advantages and limitations (especially**
**    ****important if methods are “risky”)**
**Illustrate how objectives can be  achieved ****	**
**	**Who will do what, how, and when
**Include management, evaluation and contingencies **
**Describe nature and extent of collaborations **
**         **Letters in Appendix need to confirm what you say!

## Milestone Table

- Link to the NP Action Plan
- Significant Components
- SY’s involved in each subobjective/hypothesis
- Critical milestones
- Progress/changes (link to 421)
- Products

##    Project Plan Components

-    Project Plan Components

- MILESTONES TABLE

**Summarizes the project**
**Dynamic over project lifecycle**

## Good vs Excellent Project Plans

- 	 _**Good Plans**_
- Description of the research problem
- Defined objectives
- Concise hypotheses
- Adequate research procedures
- Solvable problems with potential impact
- Clear milestones

- 	 _**Excellent Plans**_
**Show the gap in the current knowledge and the project role in filling the gap**
**Clear, linked objectives**
**Well defined research procedures**
**Clear, compelling, description of project plan and path to success**

## Action Class Ratings

**No Revision**
- Excellent project. No changes or additions are required.
**Minor Revision**
- Approach sound. Some minor changes required
**Moderate Revision**
- Some change to an approach required but project is generally feasible.
**Major revision**
- Requires significant revision.  Major gaps in plan or information.** **
**Not feasible**
- Major flaws, omissions.  Unfeasible or not possible to assess.

## Project Plans in ARS

## Project Plan

- Performance

- Planning

- Progress

- Accountability
- Stewardship
- Leadership
- Oversight

- Annual
- Reviews,
- RPES

- ARMPS
- SCA’s
- Grants

- AD421’s, Publications, Tech Transfer

## Summary

- Improved training process for panelists and the products they produce
- Improved manual to help ARS project teams understand the scope of the project plan