# Presentation: 053790

## The OSQR Process

- Meant as a guide to help you through the peer review challenges

## Problem:  Scores have been lower in the second five year cycle

## Slide 3

## Slide 4

## Panel members feel they spend more time reviewing the project plan than the scientists spent developing it.
Why can’t ARS scientists write a testable hypothesis?
Vacancies–scientists need to tell the panel how the vacant position will contribute to the project’s objectives.
What impact have OSQR reviews had on the quality of our science?
Why have scores stayed the same?
Culture of ARS has not made the change
Lack of ownership by our scientists in the process
Project plans do not represent the scientists’ best effort
Lack of science clarity, i.e. “stove piped”

- Why can’t ARS scientists write a testable hypothesis?
- Vacancies–scientists need to tell the panel how the vacant position will contribute to the project’s objectives.
- What impact have OSQR reviews had on the quality of our science?
- Why have scores stayed the same?
- Culture of ARS has not made the change
- Lack of ownership by our scientists in the process
- Project plans do not represent the scientists’ best effort
- Lack of science clarity, i.e. “stove piped”

- Recent Observations from the OSQR Staff

## Project plans begin and end with the project team (lack of ownership).
The team needs to write the plan as a narrative for their peers not something to try and impress their bosses (NPS & AD).
Literature review should be written as a gap analysis; the team is the best one to fill in the gaps.

- The team needs to write the plan as a narrative for their peers not something to try and impress their bosses (NPS & AD).
- Literature review should be written as a gap analysis; the team is the best one to fill in the gaps.

## Getting There

_**OSQR Goal**_
- Initial review at Moderate or above for 85% by 2011
_**How to achieve it**_
- Problems that can be caught with internal review
- Flawed science
- Weak write-up
- Poorly proofread
- Lack of clarity
_**This is everyone’s issue**_
**Ownership at all levels **(real internal review)
**Cooperation among all levels **(all working to best plan)
**Poor quality plans impact all!**

## INTERNAL REVIEW 

*(Getting it right the first time)*

**  ****AREA OFFICE****	****	****   NATIONAL PROGRAM**
**	****	****	****	****STAFF**

**RESEARCH TEAM**

**COOPERATION BETWEEN ALL LEVELS**
**DURING PROJECT PLAN PREPARATION PROCESS**

**EXTERNAL REVIEW**

**OSQR OUTSIDE REVIEW PANELS**

***Internal dialogue ***
***and cooperation ***
***is essential***

*Review is a dialogue:*
*Panel Recommendations *
*and ARS Responses*

##     1. Program Direction and Resource 

**    ****1. Program Direction and Resource **
**        ****Allocation Memo (PDRAM)**
**    ****2. Project Plan Outline (PPO)**
**        ***Objectives, hypotheses, general approach, outcomes agreed *
*         **upon by all parties (Research Team, NPS, Area Office). *
**     ****3. Project Plan**
*        **Internal review is essential. Be sure this is well-written, *
*        **logical, and clear.*
**    ****4. Response to review ***(especially if objectives change)*

- Where does Area Office review?

## Cohesive objectives

**Cohesive objectives**
- Plans should explain how objectives fit together. **Schematics or diagrams are helpful.**
**Proofreading**
- Errors leave panel wondering if anyone cares!
**Poor hypotheses**
- A frequent complaint. We involve Mark West. Don’t just restate objectives.
**Unnecessary hypothesis**
- If it’s not hypothesis driven...don’t force it
**Weak introductory pages **(marketing the research)
**       **Panel reviewers should see what, why, how, and expected results in brief *in the first few *
*      **pages *(before the Background section).
**Difficult to follow narrative**
- The text should expand increasingly in detail through each section in document using common themes from the beginning.
**Milestones**
**Incomplete responses to recommendations  **
**       **Serious consideration of recommendations may prevent failure during re-review.
*(continued)*

- What to look for

## THE HUMAN ELEMENT

**Predicting failure**
**	**Language Project Plan or Responses implying one cannot fulfill objectives (lack of funding, personnel, etc.)—the panel may agree!
**“****Dirty Laundry” or irritation**
**	**Signs of disgruntlement picked up on by Panel either in the Project
- 	Plan or Responses.  Even if the review is discouraging, work to support the process. Don’t air grievances in your project plan! Treat like responding to manuscript reviewers.
**Format over Content**
**	***Readability trumps format!*
- 	Dogged adherence to format rules which may confound an easy
- 	reading plan narrative.  Many are sticklers for the “rules” but may be misinterpreting some items meant to be a guide not a requirement.  For some plans altering the format can improve the flow narrative. This a case of one size not fitting all.  Obtain examples of Project Plans from your Area.

- What to look for (continued)

## Hypotheses should be well constructed and appropriate

*Don’t force a hypothesis when it’s not justified*.
**Objectives should be clear, focused, and complementary**
- Provide more explanation of relevance when disparate objectives
- appear in the same project plan. *(Work out issues with NPS in PPO)*
**3. Over-long literature reviews**
- 	Keep literature pertinent to the objectives and problem addressed.
- 	*Focus on the information gaps* research will address.
**Better documentation of methods/procedure**
- 	Explain why certain methods have been chosen, provide more details on what will be done, clarify statistical tests to be used.
**5.**  **More details on who does what**
- Show percent effort of both SY’s and external collaborators, as well as
- number and commitment of support staff that will be involved with a
- project. A diagram can help here.
*(Continued)*

- What Panels tell us

## Provide better documentation of timelines for objectives

**Provide better documentation of timelines for objectives**
- 	Specify work already in progress.
**7.****	****Collaborations are important**
- 	Some projects would benefit from additional collaborations (within and outside of ARS), especially when important expertise is missing.
**Better Documenting of Collaborations**
- 	Letters; should include details on who the collaborator is, what they will do, resources, timeline, etc. Explain in text, confirm in letter.
**Contingency Statements**
**	**These show you have considered potential new directions.
**10. *****Plans should be readable!***
*	**A*ssure the plan presents a clear, logical narrative  based upon the theme(s) set at the beginning of the document and then expanded throughout.

- What Panels tell us (Continued)

## Document Length

**   **Limited in the original plan submission
-    NO page limits for revisions...but clarity is most important, not bulk!
**Preparation of Panelists**
- Panelists receive extensive preparation: 60-90 minute orientation online, printed reviewer guidelines, 30-45 minute briefing at the beginning of their meeting. They tell us they are better prepared than for any other review.
**Objectives**
- Changes to objectives in revision, or review comments on them, must be discussed with NPS. Typically, panels are concerned when objectives and proposed execution don’t match.
**Schedule overruns**** (e.g., when PDRAM  is late)**
- Time is taken from NPS review, not plan development.
**Are panels more rigorous?**
- Short answer: yes. Because in the first cycle they appeared to have been generally more lenient, knowing it was a first time for many.

- Recent Questions