# Presentation: 337634

## COV Report to BESAC

- W. Carl Lineberger
- February 25, 2002

## COV for DOE Chemistry Programs

- The history
- The charge
- The committee
- The process
- The recommendations
- General
- Program specific

## History of  COV at DOE 

- This is the first external evaluation

## The COV Charge (I) 

*From Director of the Office of Science to Chair, BESAC:*
- Establish a “committees of visitors (COV) through which BESAC can provide an assessment on a regular basis of matters pertaining to program decisions. ... The COV should provide an assessment of the processes used to solicit, review, recommend, and document proposal actions and monitor active projects and programs.”

## The COV Charge (II) 

*From Chair, BESAC to Chair, COV:*
- For both the DOE laboratory projects and the university projects, assess the efficacy and quality of the processes used to:
- Solicit, review, recommend and document proposal actions and
- Monitor active projects and programs
- Within the boundaries defined by DOE missions and available funding, comment on how the award process has affected:
- The breadth and depth of portfolio elements, and
- The national and international standing of the portfolio elements.

## The COV Charge (II) 

- Comment on future directions proposed by the division and BES management and on opportunities that might not have been presented.
- Comment on how the process for these reviews might be improved

## Programs to Review

** ****Fundamental Interactions **
- 	**Atomic, Molecular and Optical Physics**
**	****Chemical Physics**
**	****Photochemical and Radiation Sciences**
** ****Molecular Processes and Geosciences **
- 	**Catalysis and Chemical Transformations**
**	****Separations and Analysis**
**	****Heavy Element Chemistry**
- 	**Chemical Energy**** and ****Chemical Engineering**

- Group 1
- Group 2
- .
- Group 3
- Group 4

## The COV Itself

- Twenty experienced, well respected individuals
- About 25 % receive no DOE support
- Attempt balance among many factors
- Research area
- University investigators
- National Laboratory investigators
- Prior DOE/NSF program management experience
- Divided into four disciplinary groups
- Scheduled September 18th meeting cancelled, with resulting replacement of a few members

## Group 1: Atomic, Molecular and Optical & Chemical Physics

- Lew Cocke
- Steve Leone
- Marsha Lester
- Ann Orel

*Group 1: Atomic, Molecular and Optical **& Chemical Physics*

## Group 2: Separations and Analysis & Heavy Element Chemistry

- Carol Burns
- Charles Byers
- Graham Cooks
- Robert Paine

*Group 2: Separations and Analysis **& Heavy Element Chemistry*

## Group 3: Photochemical and Radiation Sciences & Electrochemistry

- Mostafa El-Sayed
- Rick Osgood
- Nick Serpone
- Henry White

*Group 3: Photochemical and Radiation Sciences & Electrochemistry*

## Group 4: Catalysis and Chem Transformations & Chemical Engineering

- Dennis Bennett
- Malcolm Chisolm
- Gary Haller
- Larry Scott

*Group 4: Catalysis and Chem Transformations **& Chemical Engineering*

## The 2 1⁄2 Day COV Process

- General BES presentation to full COV
- Separate into “expert” breakout groups, e.g., 1
- Meet with  appropriate program managers
- Evaluate jackets by requested criteria
- Draft report – chair briefs second read group
- Second read – members from other groups (e.g., 2,3,4)
- Review same materials to provide crosscut “reality check”
- Research quality standards
- Procedural standards
- Merge to produce a final group report
- Meet as a whole to identify common issues

## Materials Provided  for Each Group

- BES mission relevance for all projects predetermined
- FY 1998, 1999, 2000
- Approximately 12 University  program jackets
- Listing of **all** active projects
- Several easy fund and easy “decline” cases
- Most jackets were “at the decision margin”
- All National Laboratory jackets
- New Initiatives
- Proposals, site visits, recommendations
- Continuing projects
- Site visit reports, reviews, staff recommendations,lab responses

## The COV Recommendations

- General observations
- General recommendations
- Program specific recommendations
- This is a preliminary report, to obtain feedback from BESAC on areas to emphasize, clarify . . .

## General Observations

- COV strongly commends Pat Dehmer for taking the lead in establishing this oversight process to improve and open programs.
- Staff support for COV was excellent.
- Research portfolios are of very high quality.
- Research programs are in general well managed and carrying out important, relevant science for DOE.

## Classes of General Recommendations

- Standardized program documentation
- Program evaluation mechanisms
- Program quality conclusions
- BES staffing and program support issues
- The COV process itself

## Standardized Documentation (I)

- Develop common forms for referee reports, solicitations, and all other repetitive activities. Make the report form such as to force a recommendation.
- Develop a time/document line page for the front of every project jacket.  This enables the ready determination of what actions, responses and materials are present.
- Include all important events in project.
- Make apparent when something did or did not take place.

## Standardized Documentation (II)

- University program documentation
- Following long-established procedures
- Project documentation generally appropriate and complete
- National Laboratory program documentation
- New initiatives generally well-documented
- Continuing program documentation less complete
- Laboratory responses to site visits /DOE Recommendations are often verbal, rather than written
-  Accomplishment-based emphasis for laboratory programs
- More recent continuation decisions are better documented
- Both university and laboratory activities will benefit from standardization

## Program evaluation mechanisms

- Contractor meetings and workshops
- Extremely valuable for evaluation and mission issues
- Consider expanding to more programs
- Invite some unfunded and/or young scientists to help bring new blood into programs
- Mail reviews to supplement laboratory site visits will be useful in some cases
- An effective information management system could be invaluable to staff in assisting the selection of the best program reviewers

## Program quality conclusions

- Top DOE research programs are world-class, competitive and outstanding in any environment
- For some areas, increasing grant size at the expense of programs at the decision margin might well produce better science for DOE
- This is a complex area, as programmatic relevance is not a simple binary choice, as COV has assumed.
- New initiatives can invigorate programs, especially when they can assist the core programs
- AMO, nanoscience . . .

## Staffing and Program Support Issues

- Current Office of Science information management system
- Fails to provide needed reviewer data
- Selection, frequency of usage, responsiveness, quality of judgments
- Fails to provide current or historical program support data in searchable, analyzable forms
- Forces program officers to develop parallel systems
- Makes it very difficult for program rotators to take advantage of prior experience, contributing to referee selection issues
- Staff is already very overworked, and adding new reporting responsibilities is not obviously helpful
- Urgent need to fill BES staff vacancies
- Urgent need to find ways to recruit new rotators, and to find mechanisms to allow them to function effectively and rapidly in their  new environment

## The COV Process Itself

- Viewed as informative and useful
- Should be continued, if DOE can utilize the COV views
- Provides a GPRA-like overview of programs
- Next COV  will benefit from this first round
- Receive materials better focussed on task
- Receive prior COV report and BES responses
- COV would likely benefit most other areas in DOE

## Program-specific Recommendations

- AMO Physics and Chemical Physics
- Very strong, active, well managed programs
- Science viewed as first rate
- Significant numbers of new investigators and programs
- DOE-sponsored AMO Workshop in 1997, together with new initiative, has produced dramatic improvement
- Chemical Engineering
- Small program, with some strong projects
- Activities are narrow compared with program name
- Both new starts and turnover are extremely low
- Perhaps  better to incorporate into larger BES programs

## Program-specific Recommendations

- Photochemistry and Radiation Science
- Very strong, well-managed program with first-rate science
- Very high quality, thoughtful proposal reviews
- Very successful annual contractor meetings
- Separations and Analysis
- Top programs are outstanding
- Well managed
- Possibly better science would result from increasing some of the best programs using funds from programs at the margin

## Program-specific Recommendations

- Electrochemistry
- Fairly small program
- Very important basic/applied program area for BES
- Needs strengthening on fundamental aspects of field
- Broadly based regular workshops with outside participation could expand program vision, as occurred with AMO
- Funding decision documentation needs to more clearly address basis for actions taken

## Program-specific Recommendations

- Heavy Element Chemistry
- Relatively small program
- Very important for DOE programs
- Small numbers of investigators in field
- Would benefit from regular workshops
- Catalysis and Chemical Transformations
- Very good balance of programs
- Strong and healthy
- Well-managed

## In Conclusion

- The Committee of Visitors for Chemistry programs has appreciated this opportunity to work with BES leadership in this first external review of their procedures.  We have learned an enormous amount, and believe that all mission-driven science programs would benefit tremendously from such open evaluation.

## Office of Basic Energy Sciences

**Chemical Sciences, Geosciences, and **
**Biosciences Division**

**Vacant, William Kirchhoff, Acting Director**
- Karen Talamini, Program Analyst
- Carolyn Dorsey, Secretary

**William Millman**
- Diane Matthews, Proc. Tech.

**Molecular Processes **
**and Geosciences**

**Fundamental **
**Interactions**

**Allan Laufer**
- Sharon Bowser, Proc. Tech.

**Energy Biosciences ****Research**

**Gregory Dilworth**
- Patricia Snyder, Proc. Tech.

**Condensed Matter Physics**
**and Materials Chemistry**
**X-Ray and Neutron Scattering**

**William Oosterhuis**
- Melanie Becker, Proc. Tech.

**Metal, Ceramic, and**
**Engineering Sciences**

**Robert Gottschall**
- Vacant, Proc. Tech.

**Structure and Composition**
**of Materials**

**Engineering Research**

**Mechanical Behavior of**
**Materials & Radiation Effects**

**Physical Behavior **
**of Materials **

**Synthesis and Processing Science**

- Altaf (Tof) Carim
- Robert Hwang, SNL
-  John Vetrano, PNNL

- Yok Chen
- Harriet Kung, LANL
- Michael Kassner, OR State
- Robert Hwang, SNL
- John Vetrano, PNNL

- Vacant FTE
- Robert Hwang, SNL

- Vacant FTE
- Timothy Fitzsimmons
- Bassem Armaly, U. of MO

- Timothy Fitzsimmons
- Bassem Armaly, U. of MO

**X-ray and Neutron Scattering**

- Helen Kerch

**Spallation Neutron Source****(Construction)**

- Jeffrey Hoy

**X-ray and Neutron Scattering**
**Facilities**

- Pedro Montano

**Experimental Condensed **
**Matter Physics**

**Theoretical Condensed **
**Matter Physics**

**Materials Chemistry &**
**Biomolecular Materials**

- Jerry Smith
- David Zehner, ORNL

- Manfred Leiser
- Dale Koelling

- Dick Kelley
- Aravinda Kini
- Matesh Varma

**Catalysis and Chemical **
**Transformation**

**Separations and Analysis**

**Chemical Energy and**
**Chemical Engineering**

**Heavy Element Chemistry**

- Raul Miranda
- Julie d'Itri, U. Pittsburgh

- Paul Maupin

- Paul Maupin
- Vacant FTE
****Henry Shaw, LLNL

- Norman Edelstein, LBNL
- Henry Shaw, LLNL
- Vacant FTE

- Nicholas Woodward
- Roger Turpening,  MTU

**Geosciences Research**

**Photochemistry &**
**Radiation Research**

**Chemical Physics**

**Atomic, Molecular, and**
** ****Optical Science**

**Facility Operations**

- Eric Rohlfing

- William Kirchhoff
****Frank Tully, SNL

- Mary Gress
- Walter Stevens

- William Millman
- William Kirchhoff

**Plant Sciences**

**Biochemistry and Biophysics**

- Gregory Dilworth
- James Tavares

- Walter Stevens
- Sharlene Weatherwax

-  **Iran Thomas, Director**
- Christie Ashton, Program Analyst
- Tarra Hardeman, Secretary

**Materials Sciences and **
**Engineering Division**

- Robert Astheimer
- F. Don Freeburn
- Stanley Staten
- Sharon Long

**Associate Director's Office**
**Staff Contacts**

-   Dual Capacity
-   IPA
-   Detailee
-   Detailee, 1/4 time, not at HQ

**Patricia Dehmer, Director**
-  **Iran Thomas, Deputy Director**
**Mary Jo Martin, Secretary**

**Office of Basic Energy Sciences**

**January 2002**

****Vacant Detailee

**Nanoscale Science Research Centers (Construction)**

- Vacant FTE

**Experimental Program to Stimulate Competitive Research (EPSCoR)**

- Matesh Varma