# Presentation: 700180

## ARS Peer Review Process Making scientific quality review work for you

- Ed Cleveland
- Scientific Quality Review Officer (SQRO)
- Office of Scientific Quality Review (OSQR)

## 1998 Farm Bill

ARS research peer-
reviewed every 5 years

Most review panelists external to ARS

Satisfactory review
before beginning
research

**ARS research peer-**
**reviewed every 5 years**
**Most review panelists external to ARS**
**Satisfactory review**
**before beginning**
**research**

- Creation of OSQR

**Notes:**
The National Agricultural Research, Extension, Education, and Economics Advisory Board (NAREEEAB) provides advice to the Secretary of Agriculture and land-grant colleges and universities on top priorities and policies for food and agricultural research, education, extension and economics. The Board is made up of 31 members, each of which represents a specific category of U.S. agricultural stakeholders, as mandated by Congress. 

## National Program Action Plan

***Peer Review***

***       ******Program Assessment***

***Stakeholder Workshop***

- Input

**Implement**

**Plan**

**Assess**

**Input**

***PDRAM***

***Research  Project Plan***

***Research initiated***

***Annual ***
***review***

***Congressional Mandate***

***Project Plan Outline***

***Certification***

## COOPERATION BETWEEN ALL LEVELS

*  **AREA OFFICE**	**	**   NATIONAL PROGRAM*
*	**	**	**	**STAFF*

*RESEARCH TEAM*
*(individual scientists)*

*COOPERATION BETWEEN ALL LEVELS*
*DURING PROJECT PLAN PREPARATION PROCESS*

*EXTERNAL REVIEW*

*OSQR OUTSIDE REVIEW PANELS*

*Internal dialogue *
*and cooperation *
*is essential*

*Review is a dialogue:*
*Panel Recommendations *
*and ARS Responses*

## National Program Action Plan

***Peer Review***

***       ******Program Assessment***

***Stakeholder Workshop***

- Input

**Implement**

**Plan**

**Assess**

**Input**

***PDRAM***

***Research  Project Plan***

***Research initiated***

***Annual ***
***review***

***Congressional Mandate***

***Project Plan Outline***

***Certification***

##     Goal of Peer Review

**	**Enhance research through independent, expert examination of PROSPECTIVE plans for *scientific and technical merit.*
*	**Not an evaluation of the ARS, its mission, National Programs, project budgets, or personnel management*

**OSQR**

## Context of Peer Review

**OSQR**

- Research must be relevant to an ARS National Program Action Plan
- Primary driver is the need to solve a problem, not investigator curiosity or idea novelty
- Projects are not in competition for funding
- Evaluation generates an “Action Class” and recommendations for improving.
- Research plan must receive a passing Action Class in order to proceed.

## Peer Reviews

- Provide external review by peers of the quality of a _prospective_ project plan
- Identify potential areas where impact of scientific effort can be increased
- Increase the awareness of the quality and extent of ARS research programs

## What is Reviewed?

- Adequacy of Approach and Procedures
- Probability of Successfully Accomplishing the Project’s Objectives
- Merit and Significance

**OSQR**

## Action Classes

**No revision**
*Excellent, no change needed*
**Minor revision**
*very good, a few modifications required*
**Moderate revision**
*Good, but has some important areas to address*
**Major revision required**
*Requires significant changes or additions*
**Not feasible**
*Major flaws or not possible to assess*

**OSQR**

## Project Plan Review vs. Peer Review of an Original Research Article

**	****	****(strong advisory component)**
**	****- “Editor” = SQRO**
**	****- Three outcomes**
**	****	**1. Publish as presented (no revision)
**	****	**2. Publish after revision as monitored
- 		by the “editor” (SQRO). Reviewers clear on what
- 		researchers are planning (minor gaps in info).
- 		(minor, moderate revision)
- 		3. Publish after revision and reexamination by 			both reviewers and SQRO.  Reviewers not at all
- 		clear about what researchers are planning (major
- 		gaps in info).  (major revision, not feasible)

## Lead Scientist and Scientist

- Responsible for plan development and implementation (Lead scientist)
- Evaluate and document progress through the five-year cycle
- Interface with stakeholders providing information on impacts
- Prepare research papers and summaries of findings

**Roles and Responsibilities**

## Roles and Responsibilities

**Research Team**

**Lead Scientist**

**Research Leader**

**OSQR Panel**

**Area Leadership**

**Laboratory Director**

**NPS--Program Direction **
**(Dialogue, Coordination, Synthesize Objectives)**

**Roles and Responsibilities**

**Ensure Quality, Science Input**

**Ensure Quality, Management Viewpoint**

## Project Plans are Linked to Team and Individual Performance

- Impact of Science,
- RPES

**Project Plan development forces thoughtful attention to project planning (hypotheses, experimental design, statistics, milestones and  contingencies) which can enhance research and career success.**

**Notes:**
The better the plan agreed upon (feasibility), the better performance you will have (meeting agency goals (ann. Reports), individual performance, ultimately the most important thing—promotion (RPES)

## Project Plans

- The foundation of  ARS research
- Link to performance and impact of individual and team
- Reflect project team’s scientific expertise

**OSQR**

## Foundation of Project Plan Development

## Roles and Responsibilities

- _Program Direction_
- National Program Leaders
*Set the objectives*
*(in dialogue with research team)*
- _Project Team_
- Scientists
-  Lead Scientists
-  Research Leaders
*Prepare Project Plan*
- _Management_
- Research Leaders
- Center/Laboratory Directors
- Area Directors
*Ensure quality*

**OSQR**

## The Impact of Internal Review

## The Impact of Internal Review

- Moderate Revision

- Minor Revision

## Another way to look at it...

- You may enjoy a 30-year career with ARS.
- Over that time the government may support project research with $20-30 million.
- Six times in that career you will be asked what you are doing with the government’s money.

## Slide 21

## How to succeed in Peer Review

Michael Strauss
Office of Scientific Quality Review

- Michael Strauss
- Office of Scientific Quality Review

## Little difference

**Little difference**

**The problem is writing and **
**gaps in the narrative!**

## Your Plan is a Marketing Tool...and the reviewers are your customers(this is true for all proposals)

- In the first few pages reviewers should know:
- 		The **subject** of your research
- 		**Why** it is important
- 		**What** it will produce
- 		**How** you are going to get there
- The rest of the document will “flesh out” this but if the reviewers don’t know it early, they probably won’t get it easily from the rest of the document.

## Common Panel Concerns

- Lack of appropriate detail in Approach, Procedure and Experimental Design
- Weak or no connection between project objectives
- Readability, narrative flow, rationale for research not developed
- Role of project team members, including collaborators, not well-defined
- Milestones and timelines vague
- Perfunctory Contingency Plans
- Project management and progress evaluation not documented

**Common Panel Concerns**

**OSQR**

**See Statistician**

**Provide Diagram**

**Tie Contingencies**
**to Milestones**

## Most Frequent Issues

- Lack of appropriate detail in Approach, Procedure and Experimental Design
- Weak or no **connection between project objectives**
- Readability, narrative flow, rationale for research not developed
- Role of project team members, including collaborators, not well-defined
**Milestones** and timelines vague
- Perfunctory **Contingency Plans**
- Project management and progress evaluation not documented

**Most Frequent Issues**

**OSQR**

**See Statistician**

**Provide Diagram**

**Tie Contingencies**
**to Milestones**

***Hypotheses***
***Milestones***
***Contingencies***
***Project Integration – using a diagram***

## Hypotheses

- Most research in ARS is hypothesis-driven. Make sure these are credible, scientifically testable (i.e., falsifiable) hypotheses related to the objectives.
- One of the most frequent comments OSQR receives from reviewers is that the plans do not contain real, testable, hypotheses.

**Hypotheses**

## What Is a Real Hypothesis?

- Definitions:
-  “A hypothesis is a tentative statement that proposes a possible explanation to some phenomenon or event.”
- “It is an assumption written in a clear, concise manner about what you think will happen in your project”
- “A hypothesis is a logical supposition, a reasonable guess, an educated conjecture”

## The value of a well constructed hypothesis is to provide direction for your project, keep your investigation focused,  and forces one to think about what results to look for in an experiment. 

- The development of a good hypothesis is not always an easy task, but without it, you may collect aimless data. Take the time to refine your hypothesis so you collect pertinent data. Remember the hypothesis keeps you a seeker of pertinent knowledge.
- Debbie Boykin, Statistician MSA

## Hypothesis Problems

- Wiggle words. A hypothesis with “*may*” or “*might*” or “*could*”    cannot be rejected; it’s true no matter what result you get.
- Misdirected hypotheses about the researchers themselves. These say things like “*Discovering the mechanism behind X will enable us to.......”* This tests the abilities of the *researchers *to take information and do something with it. Instead, the hypothesis should focus on the experimental system itself.

**Hypothesis Problems**

## Hypotheses that are statements of the obvious, or are    scientifically trivial.  “Disease results from expression of    genes for virulence in the pathogen and genes for    susceptibility in the host.”

- Too global. *“Quantifying X will provide significant    increases in income for the industry.”* Can any 5-year project plan in ARS really test this?

## Some research is not hypothesis-driven.

- This is acceptable.
- (But most ARS research should be hypothesis-driven)
- Examples are some types of engineering work, model development, or service work (e.g. germplasm grow-out)
- 	Even in these, however, there may be a basis for 	hypothesis testing.
- If stating a hypothesis is not appropriate, be sure the goal or target of the work is clear.

## Milestones

- Complete a database on ...
- Determine the accuracy and bounds of uncertainty of a model...
- Complete all work for a paper on...
- Complete the second year of a two-year experiment on...
- Complete the laboratory analyses for field samples collected last summer...
- Deliver data from resistance trials to a breeder who will...

***A Milestone is a MARKER that allows you ***
***to measure or assess your progress.***
***Used in the ancient world to gauge distance from Rome***

**Milestones**

## Weak Milestones

- 	Cannot tell what threshold would determine success on this
*Develop understanding of... *
- 	Understanding is a fleeting goal easily overturned by new 	information
*Plan a study that... *
- 	Planning is an ongoing activity for all scientists.
*Initiate experiment on... *
- 	Could be as simple as a dated entry in a notebook.

- Milestones should be *measures *of progress

**Weak Milestones**

## Contingencies

**Contingencies**

## What is a picture worth?

## ......does the plan credibly describe a cohesive, integrated project, or does it look “stove piped” with respect to how the objectives and personnel interact? It is important to describe a multi-personnel project in which the work hangs together into an integrated whole. Your plan should reflect how the work all comes together to accomplish the overall goals and objectives of the project.

## Writing a Clear PlanA well-done plan presents the “take home” message from its opening pages.

**What is the problem?**
**Why is it important?**
**Where are you going with it?**
**How are you going to get there?**
**And how will you know you have arrived?**

## Correct Grammar and Spelling are Important...but not enough

- Be sure your plan presents a clear, logical, path to success...at the outset and through the document.
- Have scientific peers outside your project and Unit read the plan for understandability BEFORE you send it to review.

## OSQR (and other) Resources

**Training after PDRAMs Issued**
**Training for SYs, RLs**
**Web-site:    **_[www.ars.usda/osqr](http://www.ars.usda/osqr)_
**Area colleagues and line management**
**NPLs**
**OSQR Manual**
**On web site...Area PAs have draft copies of the latest version.**

**OSQR (and other) Resources**

## Peer Reviewer