# Presentation: 464555

## Strength Through Science

**Program Assessment Rating Tool**
**Long Term Performance Measures**
**And BERAC**
- Christine Chalk
- Office of Budget and Planning
- Office of Science
- Department of Energy
*July 10, 2006*

## “Federal programs should receive taxpayer dollars only when they prove they achieve results.”

- OMB created the Program Assessment Rating Tool  for FY04 Budget “to assess and improve program performance” and “to hold agencies accountable for accomplishing results”.
- Seven separate PART tools for each of the seven types of government Programs – SC PART is for R&D programs with select questions added from Capital Assets & Grants.
- Four PART sections for all programs – Program Structure and Organization, Program Planning, Management, Results (half of the points are in Results).
- Without “meaningful” annual and long-term measures in place programs are automatically rated “Results not Demonstrated”. Impossible to score above 50% without measures in place.
- For R&D programs PART assessments include the key elements of the R&D Investment Criteria.

## Critical Elements of PART

- Independently Validated Management Processes:
- Excellence - Peer Review (project, portfolio & processes (COVs))
- Relevance – Planning – Community and Program – that is Transparent; Realistic; and Tied to Budgets
- Performance
- Long-term and Annual Performance Measures that are: meaningful (to Programs, Community, OMB, Congress); quantitative and trendable (when possible); tied to budgets and plans; included in performance ratings of senior managers.
- Project and Facility Management

## For FY03, programs given red, yellow, or green light score for program performance based on OMB analyst assessment.  
For FY04, PART was developed to improve consistency in ratings and to help agencies improve.
For FY04, all Office of Science Programs were rated “Results not Demonstrated” due to lack of “meaningful” performance measures.
For FY05, three Office of Science Programs were rated “Effective” (including BER) and three were rated  “Moderately Effective”.

- For FY04, PART was developed to improve consistency in ratings and to help agencies improve.
- For FY04, all Office of Science Programs were rated “Results not Demonstrated” due to lack of “meaningful” performance measures.
- For FY05, three Office of Science Programs were rated “Effective” (including BER) and three were rated  “Moderately Effective”.

**Office of Science and BER ****Experience with PART**

## Significant PART Improvements

- For the FY05 PART, Science developed new long-term and annual performance measures.
- These measures are the _only measures_ in the Office of Science.
- Developed a website to help explain the measures and report progress. (www.sc.doe.gov/measures)
- Most annual measures are quantitative and trendable. They are included in budget, tracked Quarterly by the department and audited annually by independent accounting firm.
- Progress toward the qualitative long-term measures is evaluated, by independent scientific advisory committees, every 3 years.
- Developed roadmaps for the measure.
- Developed definitions for excellent, good, fair & poor progress.
- Committees of Visitors were established to provide independent validation of peer review and other management processes.

## Discussion

- DRAFT

- =Key Intermediate Objective from DOE Strategic Plan
- =Long Term Success Measure from PART

**2006**

**2008**

**2010**

**2012**

**2014**

**2016**

***Life Sciences***

***Climate Change Research***

***Environmental Remediation***

***Medical Sciences***

**A Roadmap for Biological and Environmental Research: Objectives and Performance Targets**

- Department-sponsored research into viewing the makeup of genes in living cells, tissues, and organisms is used by clinicians as a new, sensitive tool for diagnosing disease and for monitoring the efficacy of disease therapies that target the  products of specific genes. **(2013)**

- Characterize the multi protein complexes (or the lack thereof) involving a scientifically significant fraction of a microbe’s proteins. Develop computational models to direct the use and design of microbial communities to clean up waste, sequester carbon, or produce hydrogen. **(2015)**

- Mathematical model for microbial community that detoxifies uranium **(2007)**

- New strategies for CO2 capture **(2012)**

- Enhanced biobased sources of fuel and electricity **(2012)**

- Photosynthetic microbe for continuous hydrogen production **(2008)**

- Artificial chromosome **(2006)**

- New knowledge base for cost-effective cleanup **(2012)**

**Protein Production and Tags Facility:** construction begins **(2006)**

**Characterization and Imaging of Molecular Machines Facility:** construction begins **(2006)**

- Deliver new measurements of clouds where observations are missing **(2006)**

- Provide a climate model that links the Earth climate system with Earth’s biological systems **(2010)**

- Include realistic cloud simulations in a climate model **(2007)**

- Measure ecosystem responses to climate change **(2008)**

- New technologies for in situ characterization **(2008)**

- Bioremediation
- of metals and rads validated in the field **(2008)**

- Suite of field characterization techniques for long-term monitoring of closed sites **(2012)**

- Alternative cesium separations process for High Level Waste **(2006)**

- Technical basis for leaving in place radionuclides beneath tank farm at the Hanford Site **(2007)**

- Microfabrication of 1000-electrode prototype artificial retina **(2006)**

- 60-electrode retinal devices implanted in humans **(2005)**

- In vitro testing of 1000-electrode artificial retina devices in dogs **(2007)**

- Preclinical tests of radiolabeled probes for imaging defective genes **(2006)**

*=with Commerce and other contributing  agencies*

*=with EM and between two cleanup timeline elements, as noted.*

**EM**

- New capabilities for using microbial communities to solve complex energy challenges **(2010)**

- Develop/validate new models predicting effect of aerosols on climate forcing **(2010)**

- Validate new long-term monitoring tools at multiple field sites that are inexpensive and fast **(2010)**

- Nuclear medicine centers “graduate” first class of new radio tracer chemists to support U.S. nuclear medicine **(2010)**

***Future Facilities***
***(Cross cut and support multiple objectives and targets):***

***Interdependencies:***
***(Descriptions)***

*Broadly with**** ASCR ****on computational developments, both hardware and software, affecting all facets of basic research and  advanced instrumentation.*

*=with Agriculture and EPA.*

**NOAA**

**EM**

**AG**

**NIH**

*=with National Institutes of Health*

**EM**

**EM**

**EM**

**EM**

**EM**

**NIH**

**NIH**

**NIH**

**NOAA**

**NOAA**

**AG**

**AG**

**Protein Facility:** operation begins **(2011) **

**Characterization and Imaging Facility:** operation begins **(2010)**

**Analysis and Modeling of Cellular Systems Facility:** construction begins **(2006) **

**Whole Proteome Analysis Facility:** construction begins **(2006)**

**Cellular Systems Facility:** operation begins **(2010)**

**Proteome Facility:** operation begins **(2010) **

*Future facilities directly support performance targets in the life sciences, and through this, subsequent targets in climate change and environmental remediation.*

- This timeline is for planning purposes only and does not constitute financial or contractual commitments by the Federal Government.

- Deliver improved climate data & models for policy makers to determine safe levels of greenhouse gases **(2015).**  Reduce differences between observed temperature & model simulations at subcontinental scales using several decades of recent data **(2013).**

**NOAA**

- Develop science-based solutions for cleanup and long-term monitoring of DOE contaminated sites **(2015)**. A significant fraction of DOE’s long-term stewardship sites will employ advanced biology-based clean up solutions and science-based monitors **(2013).**

**EM**

## Improvements Were Noted

*“ **Performance Measures.*--The Committee commends the Office of Science for its efforts to develop quantifiable performance measures for its research activities. Some of the measures (e.g., inverse picobarns) are less comprehensible to Congress than others, but the overall approach to quantitative performance measurement is worthwhile.”
- House Report 108-554
- Energy and Water Development Appropriations Bill 2005

## BERAC and PART

- BERAC reviewed the long-term and annual PART measures developed for FY05 PART.
- Found goals generally meaningful, ambitious but achievable,  and measurable with a few suggested changes that have been incorporated.
- First review of progress toward the long-term measures is current charge.
- Assess progress - four point scale and roadmap milestones
- Comment on whether goals are still ambitious but achievable, meaningful, and measurable.
- Suggest improvements
- BER will need a complete report before next PART         re-assessment (scheduled for Spring 2007)