# Presentation: 169091

## Peer Reviews

- Tony Farnham
- SBN

- PDS MC Presentation

- 3 Jun 2004

## Goals

- Key goals of peer review
- Ensure compatibility with PDS system
- 	 Formats, standards, etc.
- 2) Ensure usability by community
- 	 Suitability for archiving
- 	 Evaluate quality and completeness of the data and
- 		   supporting documentation, etc.
- Participants
- 1) Discipline Node and CN representatives
- 2) Discipline Node scientists and external reviewers

- PDS MC Presentation

- 3 Jun 2004

## Face-to-Face Reviews

- Face-to-face reviews work best
- Reviewers focus their attention in the same way as on a proposal review
- Interaction between reviewers generates ideas and reveals problems that wouldn’t surface in email or telecon reviews
- Variety of datasets in a single review helps everyone understand broader issues in archiving
- Occasional call-in reviewer may be allowed to get expertise for a single dataset

- PDS MC Presentation

- 3 Jun 2004

## Review Process

- Encourage reviewers to be thorough
- Treat the data review as they would a review of a  paper for a journal or grant panel
- Citations of the data should reflect high-quality results
- Give reviewers ample time to look at the data
- At least three weeks at SBN
- Freeze dataset at that point
- Include list of known deficiencies if they are discovered before the review

- PDS MC Presentation

- 3 Jun 2004

## Review Process

- Assign a lead reviewer for every dataset
- Expect reviewers to look at datasets beyond the ones for which they are lead
- Expect node scientists to look in a similar way
- Ask leading questions
- Pose specific scientific questions for reviewers to consider in looking at the data
- Ask reviewers to come up with additional questions
- Strongly encourage reviewers to work with the data
- To answer the questions posed
- To explore how they might use the data
- To determine if there is anything missing/broken

- PDS MC Presentation

- 3 Jun 2004

## Review Process

- Carefully record liens
- Discuss broad issues in detail to resolve what needs addressed
- Detailed liens may be sent by email
- Compile a clear, concise list of liens
- Read back list to ensure that everything is recorded accurately and sufficiently well to explain the problem
- Circulate complete list of liens to everyone involved in the review
- After liens have been addressed
- Confirm with reviewers to verify satisfactory resolution

- PDS MC Presentation

- 3 Jun 2004

## Post-Review Issues

- Track the quality of reviewers
- Ask the good ones again in the future, but not the poor ones
- Keep a large pool of reviewers
- No single person is requested too frequently
- Large pool of expertise to draw from for different types of dataset
- Track liens from one review to the next
- Avoids repeating the same problems
- Look for patterns that might be worth incorporating into archiving guidelines

- PDS MC Presentation

- 3 Jun 2004

## Cost Issues

- External/Face-to-face reviews cost more
- Pay outside reviewers an honorarium
- Travel costs (minimize whenever possible)
- Hold reviews at sites where reviewers are plentiful
- Try to match reviews with other travel schedules (combine trips)
- Costs are incremental
- Small compared to the overall costs of preparing the data for archiving
- Benefits are well worth the extra costs

- PDS MC Presentation

- 3 Jun 2004

## Additional Benefits

- External reviewers
- Bring different areas of expertise to reviews
- Provide feedback from the general science community
- Address usability and value of PDS products
- Raise general issues that PDS can change or improve
- Advertise PDS to the community
- Suggest other/new data sets that should be included in archive

- PDS MC Presentation

- 3 Jun 2004