# Presentation: 673519

## How to Prepare an Excellent Project PlanNP 104: Veterinary, Medical, and Urban Entomology

**Ed Cleveland and Mike Strauss**
**Office of Scientific Quality Review**

**Surviving Peer Review**

***Black Hole***

**Notes:**
Our purpose is to give you an understanding of the purpose of peer review and to show you how you, personally, and the Agency as a whole benefit from it...and to provide some advice on preparing plans that are successful in review. We also hope to show you that peer review is NOT one more government black hole!

##    1998 Farm Bill

ARS research peer-reviewed every 5 years

Most review panelists external to ARS

Satisfactory review before beginning research

**ARS research peer-reviewed every 5 years**
**Most review panelists external to ARS**
**Satisfactory review before beginning research**

**Why OSQR Review?**

**It’s not our fault!**

**Notes:**
The National Agricultural Research, Extension, Education, and Economics Advisory Board (NAREEEAB) provides advice to the Secretary of Agriculture and land-grant colleges and universities on top priorities and policies for food and agricultural research, education, extension and economics. The Board is made up of 31 members, each of which represents a specific category of U.S. agricultural stakeholders, as mandated by Congress. 

## Importance to ARS

**Congressional intent was to enhance research.**
**OMB Analyses. ARS must report statistics for review quarterly. *****OMB is looking for average score improvement...***
**Provides for Agency Accountability ...especially to Congress**
**Enhances ARS Image**
***Improving peer review success***
***could mean more support for ARS research***

## 2. National Program developed.

***5. ******OSQR Review and Certification.***

***8. ******Retrospective evaluation.***

***1.****** ******Stakeholder Workshop***

- Input

**Implement**

**Plan**

**Assess**

**Input**

***3. ******Program Priorities set (PDRAM).***

***4. ******Project Plan prepared.***

***6. ******Research initiated***

***7. ******Annual progress reviews***

***Congressional Mandate***

**You are   here**

*[PPO to NPS]*

## What is OSQR Review?

***A Dialogue****** ***
***And an...***
***	******External Review***
***	******Scientific Review***
***	******Prospective Review***
***	******Peer Review***
***	******Competitive Review***

**Notes:**
It is External because those involved are from outside ARS
It is Scientific because the reviewers are active or recently retired scientists
It is prospective because it examines work you intend to perform
It is peer based because the reviewers are YOUR scientific peers...the same ones who may review your papers for publication.
Is it perfect? No, but it is a dialogue...between you and the reviewers...so there is opportunity for respectful and informed disagreement. 
It is an opportunity to benefit from outside views and advice to improve your research plan.

## Review Panels

- Panelists are your colleagues.
- 	They read your peer-reviewed papers.
- Panelists are active scientists.
- 	Most are academics (per the Farm Bill).
- Panelist often know your work.
- 	And are often familiar with your excellence.
- Panelists take their task very seriously.
- 	The devote many hours to each review.
*They don’t want to give low scores!*

## How a Panel is Selected

- OSQR Receives suggestions/nominations from NPS, Areas, others.
- All potential chairs are screened for COIs.
- SQRO Interviews potential candidates.
- SQRO selects and invites Chairs.
- Chairs work with OSQR to develop a balanced, proposed list of panel members.
- SQRO reviews and approves final list of panelists.

## What is an Excellent Plan?

**Doesn’t just extend past work **
**Is important and significant**
**Breaks new ground**
**Pushes the frontiers of knowledge**
**Builds on new knowledge**
**Displays original ideas**
**Risks breakthroughs**

- Presented in a logical, clear, easy-to-follow, form.
*Don’t “tack” varying bits from different authors together...write a consistent, smooth, narrative from those pieces.*

**Notes:**
Good research is well conceived and well planned...exceptional research is creative and can take significant risk...but promises tremendous payoffs. Don’t be afraid to propose taking risks...that is the advantage of base funded research...just let your reveiwers know that you understand the risks and benefits...and even if you score low, don’t be afraid to justify your choice of approaches to the panel.

## What do Reviewers Want?

- Reviewers need to see a coherent story...
- 	What are the problems you are researching?
- 	Why are they important?
- 	What do you hypothesize?
- 	   (or why a hypothesis is not appropriate)
- 	How will you test your hypothesis?
- 	Why are you the best group to do this?
- Don’t let reviewers get another perspective...
- 	“My reputation should be enough”
- 	“This process isn’t important”
- 	“Who are these reviewers anyway!”
- 	“What’s the minimum I can do to pass?”
- 	“I’m not really taking this process seriously”

## The foundation of research within ARS
Treat this like a competitive grant request...reviewers do.

Link to performance and impact of individual and team
Congress and OMB use review scores to track overall success of the Agency...so your score makes a difference.

Reflect project team’s scientific expertise
	Reviewers often know you...so they expect to see quality that matches your reputation.

	They can’t score you on reputation.

- Treat this like a competitive grant request...reviewers do.
**Link to performance and impact of individual and team**
- Congress and OMB use review scores to track overall success of the Agency...so your score makes a difference.
**Reflect project team’s scientific** **expertise**
- 	Reviewers often know you...so they expect to see quality that matches your reputation.
- 	*They can’t score you on reputation.*

**ARS Project Plans**

## The Goal of Peer Review

**    ****Enhance research through Independent, expert examination of PROSPECTIVE plans for *****scientific and technical merit and relevance.***

## What is a Research Plan?

- A

- Tells the “story” of your research
- An integrated and logical presentation
- A clearly presented argument
- A “Marketing” brochure (and more!)

*Your plan should present your work in a clear, logical fashion that will transmit enthusiasm for the work and convince the reader that your team is best suited to perform it.*

## Project Review Criteria

- Adequacy of Approach and Procedures
- Probability of Successfully Accomplishing the Project’s Objectives
- Merit and Significance

## Reviewers want to know...

- What is the problem?
- Why is it important?
- Where are you going with it?
- How are you going to get there?
- And how will you know you have arrived?

*Don’t make them hunt for this!*
*Poor organization can wear their patience...and reduce your score.*

## Review Products

- Action Class Score
- Consensus review comments

## Action Class Ratings

**No Revision**
- Excellent project. No changes or additions are required.
**Minor Revision**
- Approach sound. Some minor changes required
**Moderate Revision**
- Some change to an approach required but project is generally feasible.
**Major revision**
- Requires significant revision.  Major gaps in plan or information.** **
**Not feasible**
- Major flaws, omissions.  Unfeasible or not possible to assess.

## Where are the problems?

**Where are the problems?**
**(Science, Data, Writing)**

**Little difference**

**The problem is writing and **
**gaps in the narrative.**

## OSQR SURVEY:Who Reads Plans Before Review?

**Other than those with filing or sign-off authority, how many others reviewed your project plan prior to the formal OSQR panel?**

| No. of Reviewers
0
1
2
3
4
5
>5 | Percentage
24.6
10.6
18.2
16.9
11.9
5.9
11.8
 |
| --- | --- |

**Notes:**
Mike, Percentages are those passing 1st round?  What are those in red?  Otherwise there appears to be a pattern here
NO. THIS IS RESPONSE TO THE SURVEY QUESTION ON THE LEFT. THE PERCENTAGE REPRESENTS THE PROPORTION OF WHICH ANSWER WAS MADE. SO HERE 24.6 PERCENT OF THOSE RESPONDING INDICATED THAT 0 IN RESPONSE THE QUESTION ON THE LEFT. I MARKED THE FIRST TWO RED TO UNDERSCORE THAT MORE THAN 1/3 OF RESEARCHERS HAD ONE OR NO OTHERS LOOK OVER THEIR PLAN BEFORE IT WAS SENT TO OSQR...AND THEN FOLLOWED THAT BY SAYING THAT PANELS CONFIRMED THIS IN THEIR COMPLAINTS IN ALMOST EVERY MEETING BEFORE SEPTEMBER 2007 THAT WE NEEDED A “TRIAGE” SYSTEM TO PREVENT CLEARLY BADLY WRITTEN PLANS TO MAKE IT TO REVIEW (PUT ANOTHER WAY THE QUESTION FROM THEM WAS OFTEN “DOES ANYONE READ THESE BEFORE THEY COME TO US???”).

## Like Review of a Manuscript

**“****Editor” = SQRO**
**	****Three outcomes**
**	****	****Publish as presented ****(no revision)**
**	****	****Publish after revision as monitored by the “editor” ****	****(SQRO). Reviewers clear on what researchers are planning ****	****(minor gaps in info).**
**	****	****(minor, moderate revision)**
**	****	**
**	****	****Publish after revision and reexamination by both reviewers ****	****and SQRO.  Reviewers not at all clear about what ****	****researchers are planning (major gaps in info). **
**	****	**** ****(major revision, not feasible)**
***Initial***** *****and final *****review are tracked and **
**included in your annual performance evaluation.**

##   Now let’s look at the “nuts and bolts” of a plan...

Remember
You are composing a narrative...
...not  completing a form!

Your plan is not reviewed by bureaucrats.

Reviewers are active scientists in your field.

They often know you and your work.

Reviewers devote, on average, 6 to 8 hours reviewing each plan BEFORE the panel meets.

- Remember
- You are composing a *narrative...*
- ...not  completing a form!
- Your plan is not reviewed by bureaucrats.
- Reviewers are active scientists in your field.
- They often know you and your work.
*Reviewers devote, on average, 6 to 8 hours reviewing each plan BEFORE the panel meets.*

## Document Outline

- Signature Page..........................page 2
- Table of Contents.......................page 3
- Project summary (250 words)...page 4
- Objectives.................................page 5
- Need for research  (1-2 p)
- Scientific Background  (5-7 p)
- Prior Accomplishments (2 p)
- Approach & Procedures (6-15 p)
- Literature Cited
- Milestone Table (1-3 p)
- Past Accomplishments of Project Team Members
- Issues of Concern statements
- Appendices (letters plus other material)

***15 - 30 pages***

**Document Outline**

*These are not boxes...they are guides to your narrative flow.*

## Your Plan is a Marketing Tool...and the reviewers are your customers

- By page 7 reviewers should know:
- 		The **subject** of your research
- 		**Why** it is important
- 		**What** it will produce
- 		**How** you are going to get there
- The rest of the document will “flesh out” this but if the reviewers don’t know it by page 7, they probably won’t get it easily from the rest of the document.

## Title and Investigators...............page 1

- Signature Page..........................page 2
- Table of Contents.......................page 3
- Project summary (250 words)...page 4
- Objectives.................................page 5
- Need for research  (1-2 p)
- Scientific Background  (5-7 p)
- Prior Accomplishments (2 p)
- Approach & Procedures (6-15 p)
- Literature Cited
- Milestone Table (1-3 p)
- Past Accomplishments of Project Team Members
- Issues of Concern statements
- Appendices (letters plus other material)

## The project summary should present the overall thread of your work, the major gaps being addressed, how you will address them, and what you will produce...in no more than a dozen sentences.

The Objectives should be followed by one to two paragraphs that explain how they are integrated and how they address the overall goals of the work. A chart illustrating this and showing where individuals on the project team fit is most helpful.

- The Objectives should be followed by one to two paragraphs that explain how they are integrated and how they address the overall goals of the work. A chart illustrating this and showing where individuals on the project team fit is most helpful.

##    Project Plan Components

- Write this in active voice.
- State the essential problem and why it is important.
- What have you done to date (1-2 sentences)?
- How will you address the issue?
- Why is this important?
*This is where you capture the interest of the reviewer. Make it compelling!*

- Project Summary – 250 words
*The “Abstract”*

##    Project Plan Components

- Express need scientifically AND in the context of NP Action Plan.
- Be concise in statement of research purpose.
- Discuss potential benefits and anticipated products.
- Identify relevant customers and stakeholders.
- Briefly note the principle methods you will utilize (e.g., ...using microarray technologies we will elucidate...”
- Include comment on outcomes, impacts, and products.
*Build upon, don’t repeat, the overview!*

- NEED FOR  RESEARCH: 1-2 pages
*Where are you going?*

##  Project Plan Components

- Highlight knowledge gaps.
- Literature demonstrates understanding and gaps. Not an exhaustive.
- Show a rationale for the objectives.
- How will this fill knowledge gaps?
- Limit to 1/3 of project plan length
- Note similar projects within and outside ARS and how your past work prepares for or leads to this *(provide details in the Prior Accomplishments section...but say enough to convince reviewers you know the area).*
- _Cite preliminary data from your projects_, if available

- Scientific Background: 5-7 pages
*Why are you going there?*

## Project Plan Components

- Prior Accomplishments: 1-2 pages
*What have you done before?*

*Highlighted briefly in the Background.*
- Name prior project terminated within two years
- Major objectives and accomplishments
- Prior project investigators
- Impact of prior work (science, technology, users)
- Pertinent publications
- A table or chart of past data can be very helpful.

## Project Plan Components

-   Approaches & Procedures: 6-12 pages
***How are you going to get there?***

- Set out your Experimental design.
-    *Formulate REAL, testable, hypotheses!*
- Describe approaches and methods any why they are appropriate.
- Discuss advantages and limitations (important if methods are “risky”).
- Illustrate how objectives can be  achieved.
-    Who will do what, how, and when (including collaborators and SCAs!)
- Describe nature and extent of collaborations, including SCAs
**   **Letters in Appendix need to confirm what you say!
-    For SCAs, a copy of the agreement is sufficient.
- Include management, evaluation and contingencies.
-    What is your path to success? How will you monitor it?

- Expanded from PPO!

## Hypotheses

- Most plans should have hypotheses.
- Hypotheses must be *falsifiable and testable.*
- Hypotheses are not restatements of objectives.
- A hypothesis may not be appropriate for work like breeding or germplasm characterization...but explain that!
- Ask your Area statistician to review your hypotheses

##    Project Plan Components

-    Project Plan Components

*Summarizes the project*

*Dynamic over the project lifecycle*

## Common Criticisms

- Unclear hypotheses
- 	*(...or they aren’t hypotheses)*
- Project objectives not connected
- 	*(Why are these put together...don’t know? Ask your NPL!)*
- Approach and Procedures lack appropriate detail
- 	*(More than in PPO...the longest portion of your plan)*
- Project team member roles poorly defined
- 	*(Who is doing what? A figure can help here)*
- Weak writing/poorly proofread
*	**(What do you think when you get a poorly written manuscript!)*
- Lack of statistically sound experimental design
- 	*(n=1?!)*
- Data accumulation with no analysis
- 	*(Gathering data is only the first step!)*

## Slide 33

## Comment Sampling “No Revision”

**Comment Sampling**** ****“No Revision”**

*...**studies will fill rather substantial knowledge gaps.*
*...**well written and comprehensive*
*...**well written project plan*
*...**well organized, cogently prioritized and comprehensive research plan*...

## Comment Sampling “Minor Revision”

**Comment Sampling**** ****“Minor Revision”**

*...**studies will fill rather substantial **	**knowledge  **	**gaps.*
*...**well written and comprehensive*
*...**well written project plan*
*...**well organized, cogently prioritized and **	**comprehensive research plan*...
*...**well written, well organized, and easy to **	**follow.*

## Comment Sampling “Moderate Revision”

**Comment Sampling**** ****“Moderate Revision”**

*...**one of the better written*
*...**excellent discussion of technology *
*...**large research project plan but poorly organized.*
*...**concerns that some of the objectives can be completed by this team.*
*...**work and ability described as "adequate"*
*...**missing current information*
*...**hypotheses and plan well conceived, approaches appropriate*

## Comment Sampling 

**Comment Sampling**** **
**“****Major Revision”**

*...**a heroic course that...will not achieve the stated objectives*
*...**serious flaws in experimental designs...*
*General Lack of focus. Plan is so broad that it leaves out important details...not clear data will be relevant or interpretable.*
*...**lacks cohesion and clear direction*
*...**writing style, quality, and organization significantly detract...*
*Short on details. Not well planned. Need to take process seriously.*

## Comment Sampling 

**Comment Sampling**** **
**“****Not Feasible”**

*...**approaches highly flawed, lack detail, many experiments duplicative*
*...**poorly written...difficult to discern what the authors want to do. *
*...**proposed research contradicts intention of the proposal.*
*...**exaggerations, incomplete reviews..., and misleading comments, as well as poor grammar and organization.*
*...**lack of knowledge...incomplete understanding*
*...**fundamental misunderstanding...*
*Work does not address stated objectives.*

## Recent Changes

- Single Re-Review
- Stringent postponement guidelines
- Re-review by full panel (online)
- Smaller panels but more time/plan
- Discharge of panel after re-review meeting
- Project Plan Outline
**Internal review by Areas before submission**
- New OSQR Hanbook (replaces Manual)

## Slide 40

## FY 2008 Reviews

| NP
Rangeland, Pasture & Forages

Plant & Microbe  Genetics & Breeding

Integrated Agriculture

Crop Production

FY08 to date | No. Plans
33

124

17

22

196 | % “Pass”
93.9%

88.7%

70.6%

86.4%

87.8% | 1st Cycle
57.7%

82.1%

75%

84.2%

78.1% |
| --- | --- | --- | --- |

**Notes:**
THIS TABLE HAS BEEN UPDATED BASED ON COMPLETION OF NP305 REVIEW. NOTE ON INTEGRATED AGRICULTURE. THIS IS THE RESULT OF ONE PANEL THAT HAD A FEW PLANS “ON THE EDGE” AND CHOSE TO GO DOWN INSTEAD OF UP IN SCORE. IF THEY HAD PASSED JUST ONE OF THESE THE AVERAGE WOULD BE MORE THAN 76%...SO THE % “PASS” NUMBER DOES NOT REPRESENT A MAJOR DECLINE FROM THE FIRST CYCLE.

## Conclusions

- The reviewers need to see the logical “thread” through your work.
- Don’t make readers “search” for what you are doing!
- Be clear, accurate, and correct.
- Don’t assume reviewers know you and your work...(a poor plan may not be saved even if they do!)
*    **A NONTECHNICAL reader should be able to understand the general goals of your research.*
*	**A TECHNICAL reader should be confident you understand the problem and can accomplish the work.*

## Some hints to success...

***Proofread Your Plan***
- Ask a nontechnical person to read your plan
- Ask someone who hasn’t seen it to read and proofread your plan
- Ask a highly critical colleague to read it thoroughly.
- (Hint: Ask them the same questions reviewers are asked)
- Are collaborations documented appropriately?
- Check hypotheses...
*Treat this the same care you would *
*a competitive proposal. The reviewers will!*

## What if I have several SYs?

- Everyone has a contribution to make
- but don’t follow this recipe for failure...
- Ask for a portion from each
- Assemble those in order
- Consider the plan done...(it’s just the start!)

- Use the information to write a flowing, logical, and consistent narrative.
- 	Settle scientific differences...then write!
- 	You may not like a colleague...don’t let the panel know that!
- 	You may not like some of the work...don’t let the panel see that!
- Alternatively, you may wish to write an initial draft and edit in others’ additions...keep the writing consistent throughout.

## What Happens After Review?

**  ****No, Minor or Moderate Revision**
- Lead Scientist responds to comments. Scientific Quality Review Officer certifies compliance with recommendations.
**  ****Major Revision or Not Feasible**
- Lead Scientist revises and responds to comments.
- Panel performs a second review assessing response to their comments and assigns a new Action Class Score. If still Major or Not Feasible, project is returned for administrative action. No further review.
**Projects are reviewed no more than ****two times**
- (There are *no* page limits for revised plans)

## Can I disagree with the panel?

**This is a dialogue**
- If you really disagree...put it away for a few days!
- Then...
**Honestly consider panel opinions.**
**Be polite but if you disagree say why**
**DON’T skip changes to plan**
**DON’T insult or impugn panelists**
**DO provide justification for your alternative view**
*Panels are NOT perfect...they are fellow scientists*

## The Impact of Peer Review

**Notes:**
NOTE: JUMP ABOUT 1996 PROBABLY DUE TO CHANGES IN SCOPUS ADDING MANY NEW JOURNALS. WHAT IS SIGNIFICANT ABOUT THE LINE AFTER 2001 IS ITS STEADY SLOPE UPWARDS AND NOT A ONE-TIME JUMP.

## Last Words

- Proofread
- Seek Review
- then
- proofread and seek more review
- And lastly
**Proofread and Seek Review**
*However...*

## ...Correct Grammar and Spelling are Important -- but not enough

*`Twas brillig, and the slithy toves**  Did gyre and gimble in the wabe;**All mimsy were the borogoves,**  And the mome raths outgrabe.*
- 					**Lewis Carroll**

- Yes, proofread but also...
- Be sure your plan presents a clear, logical, path to success...at the outset and through the document.
**Good grammar doesn’t **
**guarantee clear writing.**

## Peer Reviewer

## Input

- Implementation

- Planning

- Assessment

- Review