# Presentation: 749983

## Report of COV for BES Scientific User Facilities Division

- Presented to BESAC Meeting
- August 5, 2004
- J. Michael Rowe

## Committee Members

- Ian Anderson, ORNL
- Nora Berrah, W. Michigan University
- Martin Blume, APS
- Miles Klein, UIUC
- Richard Osgood, Columbia University
- Thomas Russell, University of Massachusetts
- Sunil Sinha, University of California, San Diego
- Michael Rowe, NIST (Chair)
- Julia Phillips, Sandia National Laboratories

## Charge to Committee

- For the scientific user facilities, assess the efficacy and quality of the processes used to:
- Solicit, review, recommend, and document actions leading to upgrade or construction of facilities or to special research activities related to facilities such as detector development or accelerator physics, and
- Monitor operating facilities
- Comment on how this review process has affected the national and international standing of the individual facilities and the collection of facilities operated by BES.

## The Review Process 1

- This is a new division, still developing policies and procedures, and the scale of projects is quite different from the other two divisions.
- The peer review process is, of necessity, different from PI grant processes
- Either full BESAC sub-panel review (Birgeneau/Shen)
- Or independent reviewers submitting individual reports (e.g. APS, HFIR)

## Review Process 2

- The charge is quite broad, and with limited time we chose to interpret it broadly and focus on
- the establishment of the new Division and issues relating to this
- the review process itself, as applied to synchrotron sources, neutron sources and nano centers

## Review Process 3

- Two day meeting in Germantown, 3/9-10
- Presentations by Pat Dehmer and Pedro Montano
- Three groups looked at review history for the three major types of facilities, with individual cross-checking of other groups
- Extensive executive sessions, including discussions with Pat

## Review Process 4

- Letters to each BES facility asking for comments (anonymous)
- All agreed it was fair; all said it helped but...
- Timing?
- Differing levels of enthusiasm
- Definition of metrics
- Metrics are good, but need careful thought
- Facilities should be partners in developing metrics
- Independent reviewers
- Concern about anecdotes becoming data
- Appearance of potential for bias
- Lack of consensus *could* lead to ill-considered statements and recommendations (JMR)

## COV Issues

- Complete timelines for each review should be maintained and available for future COVs
- There were difficulties tracing the process from review to recommendations to results (including facility responses) to re-reviews when necessary
- Cross-references to earlier reviews would help the COV better grasp the full history

## Documentation of Facility Reviews

- Necessary elements (inclusion was not uniform)
- Executive summary that accurately and succinctly reflects tone and substance
- Letter responses to reviews to facility and laboratory management
- Reporting of review outcomes to reviewers
- Care in informing reviewers for non BESAC reviews of procedures, and that COV will have access in future
- Re-reviews should be scheduled in cases where the normal schedule is too “leisurely”

## Users of Facilities

- Success = happy users and good science
- User needs are changing, and definitions must follow e.g. mail in samples, remote operation, nano centers
- Must come to acceptable definition of users, publications, and acknowledgements,  when only (small) part of research depends on facility
- Nano centers will accelerate these trends

## Metrics

- The COV approves of the use of well defined metrics in evaluating facilities, with the provisos
- A countable item may not be a good metric
- Terms must be clearly defined
- Metrics for neutron and photon facilities may be different from those for nano centers
- All metrics should be reviewed regularly, in direct consultation with facility managers

## Facility Review Process 1

- We conclude that reviews are fair and seen to be so, but there is unease about individual reports
- May give inordinate weight to minor points
- Perception by outsiders and facilities
- COV review and allowance for adequate facility responses help alleviate these concerns

## Facility Review Process 2

- Suggestions for improvement
- Better definition and choice of metrics
- User surveys should be required (by user groups)
- The agenda should be controlled to allow more executive discussion
- Time should always be provided for direct reviewer contact with users and staff
- When serious deficiencies are identified, we recommend re-review in a short time in addition to written responses
- Time for discussion of laboratory-wide issues should be included
- Ensure research program representation at reviews

## Nanoscale Science Research Centers

- Recommendations
- Require inter-center collaboration
- Closely involve users at the beginning
- Establish appropriate agreements with other laboratory activities and facilities
- Coordinate between laboratories to ensure a national, not regional resource
- Carefully integrate with science programs

## General Comments on Division

- New structure is good for all involved
- Relieves science program managers from details of facility operation
- Allows research program managers to focus on science
- Allows facilities to receive proper management attention, commensurate with their budget and impact
- Reduces scope for budgetary arguments

## Then Why Only Now?

- There are actual and potential negatives in separating facilities from science
- Loss of sense that facilities serve science
- Decoupling of facility management from research programs
- Enhanced visibility for good or ill
- We recommend that both research divisions have at least one program manager at every facility review

## Conclusions

- The new Scientific User Facilities Division is well launched, building on past facility management
- COV strongly supports the new structure
- Facility reviews are working, and we have recommended some changes & improvements
- We strongly support NSRCs, and recommend national planning and cooperation