# Presentation: 292822

## Independent Oversight Office Evaluation ofBNL Self-Assessment (SA) Programs

- Presentation to Extended Operations Council
- December 5, 2000
- John Usher, Self-Assessment Oversight

## IO Evaluation of BNL SA ProgramsOverview

- Background
- What is IO’s Role?
- FY99 and FY00 Evaluation Results
- Strengths
- Areas for Improvement
- FY01 Evaluations

**Notes:**
The slide provides a table of contents for this presentation.

Background provides a very brief narrative of the drivers of the self-assessment process, how requirements are met using the Integrated Assessment Program, how organizations implement the self-assessment process, and how organizations improve their performance.

IO’s Role describes how the Independent Oversight Office fits into IAP by evaluating organizational self-assessment programs, and describes the basis for these evalauations.

Evaluation Results depicts SA Program ratings determined over the past two fiscal years, and briefly discusses performance trends observed and reasons for these trends.

Strengths and Areas for Improvements highlight findings from the IO Evalauations.  These findings resulted from most, if not all, of the evaluations and are meant to represent global strengths and areas for improvement.

The final slide presents the schedule for FY01 Evaluations.

## IO Evaluation of BNL SA ProgramsBackground

*What drives SA Programs?*
- ISM: Feedback and Improvement
- QA Program
- BNL Critical Outcomes
- Regulatory Requirements

**Notes:**
What are the requirements bases for self-assessment at BNL?

ISM Core Function 5, Provide Feedback and Continuous Improvement. feedback information on the adequacy of controls is gathered, opportunities for improving the definition and planning of work are identified and implemented, line and independent oversight is conducted, and, if necessary, regulatory enforcement actions occur. 

In the BNL QA Program, Criterion 9, Management Assessment, and Criterion 10, Independent Assessment, are met primarily through the implementation of the appropriate elements of the IAP.

“ Provisions within the Integrated Assessment Program (IAP) provide BNL with the necessary feedback processes to support performance-based management. Self-Assessment, an element of IAP, responds directly to the management assessment
criterion.”

“Provisions within the Integrated Assessment Program (IAP) ensure that BNL has the necessary feedback processes to support performance-based management. Internal Audit, Independent Oversight, Peer Review, as well as Corporate Oversight, respond directly to this criterion.”

The achievement of contractual critical outcomes, objectives, and performance measures is determined largely by BNL’s self-assessment process.

Some BNL organizations (e.g., Safeguards and Security, Procurement and Property Management, Internal Audit) are required by government regulations and/or DOE Orders to conduct specific assessment activities, analyze and report these results, and use these results to improve performance.

## IO Evaluation of BNL SA ProgramsBackground

*How are requirements met?*
- BNL’s Integrated Assessment Program (IAP) provides the means to measure performance against established objectives, to analyze performance, and to identify opportunities for improvement.

**Notes:**
IAP describes a process for determining performance objectives, planning and conducting assessment activities, analyzing results from assessments, measuring performance, and identifying opportunities for improvement.

## IO Evaluation of BNL SA ProgramsBackground

*How do BNL organizations implement IAP?*
- BNL organizations develop Self-Assessment Plans, conduct assessment activities, and analyze assessment results in conformance with the SBMS Subject Area: **Integrated Assessment**.

**Notes:**
The subject area for Integrated Assessment requires the determination of performance objectives, planning of assessment activities to measure performance against these objectives, and the analysis of results from assessments.

BNL organizations build the “cycle” of self-assessment:
Planning based on Critical Outcomes, Objectives and Performance Measures 
Monitoring performance against these and other objectives and performance measures 
Evaluating overall performance and identifying areas for improvement (Self-Evaluation Report)
Implementing actions for improvements 

The Annual Self-Evaluation Report represents the fruition of the organization’s Self-Assessment Program.  The report contains a critical evaluation of performance versus stated objectives, lists identified opportunities for improvement, and lays the foundation for establishing next year’s performance objectives.

## IO Evaluation of BNL SA ProgramsBackground

*How do BNL organizations improve performance?*
- BNL organizations develop (corrective) action plans based on identified improvement opportunities and then carry out these action plans to effect improvements in performance.

**Notes:**
The IAP and related subject areas require the design of improvement actions, the implementation of these actions, and verification that improvements have been effected.

## IO Evaluation of BNL SA ProgramsWhat is IO’s Role?

- The IO Office evaluates the effectiveness, efficiency and adequacy of organizational self-assessment programs:
- Verifies that the organization’s SA Plan conforms to IAP expectations (approach)
- Verifies that the organization’s SA process is implemented as planned (deployment)
- Verifies that the organization makes effective use of the results of self-assessment

**Notes:**
“The purpose of Independent Oversight is to independently verify the effectiveness, efficiency, and adequacy of the Laboratory's Self-Assessment Program at the Department, Division, Directorate, and Laboratory levels. The Independent Oversight Office evaluates each of the organizations indicated above annually in accordance with requirements and guidelines provided in Subject Area Descriptions for Integrated Assessment and Independent Oversight. Assessments are identified and scheduled in the Fiscal Year Independent Oversight Assessment Plan.”

## IO Evaluation of BNL SA ProgramsWhat is IO’s Role?

*How does IO evaluate SA programs?*
- Evaluators measure the organizational SA process against established criteria based on expectations derived from IAP, etc.
- Criteria are listed at _http://www.io.bnl.gov/safy01.htm_

**Notes:**
“The Independent Oversight Office evaluates the performance of the assessed organization using criteria identified in Independent Oversight Evaluation Criteria. The criteria used can be grouped into three categories: Approach, Development, and Results. Approach refers to how effectively the assessed organization addresses the planning     of assessments. Development refers to the extent to which Integrated Assessment is used by all parts of the assessed organization. Results refer to the outcomes of the organization's assessment activities and how they are used to sustain or improve performance.”

## IO Evaluation of BNL SA ProgramsFY99 and FY00 Evaluation Results

**FY99        FY00**

**Outstanding**

**Excellent**

**Good**

**Program Ratings**

**Notes:**
Program evaluations began in March 1999, and the 14th concluded in August 2000.  All Directorates were reviewed in FY99 or FY00 along with three other organizations reporting to the BNL Director.

The first six organizations were evaluated primarily against expectations for their approach to self-assessment in FY99.  These are indicated using a hashed pattern on the chart to represent the fact that their ratings are somewhat “fuzzy” when compared to the ratings for the eight organizations evaluated in FY00.  These eight organizations were evaluated against expectations for approach to, and deployment of, the self-assessment process. A detailed point-rating system was used for the FY00 evaluations but not for the FY99 evaluations.

No organizational programs received ratings of marginal or unsatisfactory.

## IO Evaluation of BNL SA ProgramsFY99 and FY00 Evaluation Results

- Organizational programs improved dramatically from FY99 to FY00.
- One Directorate (EM) improved from **good** to **outstanding** by turning “opportunities for improvement” into “strengths”.

**Notes:**
There appears to be a trend present showing improving performance.  It would likely be a statistically significant improvement if it weren’t for the IO Evaluator’s personal prejudice regarding the quality of the data.

“The [Environmental Management] Directorate, as a whole, demonstrated effective use of the results of assessment activities, in particular through their responsiveness to the FY99 IO Evaluation of EMD’s Self-Assessment Programs.  It is remarkable that many of the areas for improvement identified in the FY99 Evaluation Report are now considered organizational strengths.”

## IO Evaluation of BNL SA ProgramsFY99 and FY00 Evaluation Results

*What enabled these improvements?*
- Improved direction and guidance
- Technical assistance
- Organizational experience (learning)

**Notes:**
Even though the SBMS subject area for Integrated Assessment was issued (Sept. 2000) after the conclusion of the FY00 evaluations, it is evident that organizations have benefited from involvement of managers and staff on this subject area working group.

The IAP Administrator has issued periodic guidance to assist organizations in carrying out key IA elements such as developing Self-Evaluation Reports.  The IAP Administrator has also met with many of the managers of BNL organizations to personally provide guidance and answer questions relating to the self-assessment process.

The IO Evaluator also likes to think that he has been able to provide guidance to the managers and staff that he has interviewed in the course of the FY99 and FY00 evaluations. 

Most importantly, BNL organizations have obviously shared information among them to enable learning and improvement based upon the experience of organizations that have been evaluated.

## IO Evaluation of BNL SA ProgramsStrengths

- Management involvement in, and commitment to, self-assessment and continuous improvement was uniformly observed.
- Managers expressed “pride of ownership”, stressed the importance of serving as positive examples and even as “salespersons”.

**Notes:**
These strengths are noted site-wide though not necessarily in 100% of the organizations evaluated.

Many managers were deeply involved in the SA process primarily through assessment activities, though, notably, also in the analysis of assessment results and in the establishment of performance objectives.  Managers routinely participate in walkthroughs and in meetings with direct reports and staff.

Managers were eager to discuss their improvement and proud to describe the improvements implemented in their organizations.

## IO Evaluation of BNL SA ProgramsStrengths

- Almost all BNL organizations have developed comprehensive self-assessment programs covering research/operations, business functions, ES&H, and human resources.
- Organizations largely built their SA programs on a broad spectrum of existing assessment activities.

**Notes:**
By using the IAP (Baldrige) or other (Balanced Score Card, Environmental Management System, Work Breakdown Structure) framework many organizations have established comprehensive SA programs covering the gamut of the organization’s operations and business functions.

Organizations planned appropriately to take advantage of what already existed in the form of assessment activities within their organizations.  Organizations have not had to develop a set of new assessments to measure their performance.  

Organizations also realize that ongoing activities such as project and/or staff meetings provide valuable measures of performance.

## IO Evaluation of BNL SA ProgramsStrengths

- Organizational performance objectives are clearly linked to Critical Outcomes, IAP framework, other higher level goals/objectives, and regulations.
- Assessment activities are linked to organizational objectives and include assessments required by BNL.

**Notes:**
Organizations have also identified other objectives not directly related to Critical Outcomes but that are important to their operations/business functions.  

Organizations have developed strategies and means for measuring performance against documented objectives.

Most organizations identify assessments required by BNL and integrate these assessments into their SA programs.  Required assessments are formally listed in the SBMS subject area for Integrated Assessment.

## IO Evaluation of BNL SA ProgramsStrengths

- Organizations exhibit a strong customer focus in their SA programs and objectives by appropriately considering customer expectations.
- Organizations actively solicit customer feedback regarding the organization’s performance, services, or products.

**Notes:**
Some organizations involve DOE in the review of performance objectives.  DOE is formally involved in review and approval of the Balanced Score Card plans.

Organizations appropriately consider customer feedback from the previous year when developing objectives.

Many organizations solicit customer (user) feedback as an integral part of their SA process.  Organizations use surveys, focus groups, user meetings, and one-on-one meetings to solicit customer feedback.

Some organizations survey the organization's employees as part of their SA process.

## IO Evaluation of BNL SA ProgramsAreas for Improvement

- Some SA Plans do not contain a description of the entire SA process (cycle) or each process element.
- Responsibilities for SA elements (except assessments) are not clearly described including those for stewardship of the SA process.

**Notes:**
These areas for improvement were observed site-wide, though not necessarily in all organizations.  In fact, in some organizations, these findings may have been considered strengths.  Specific inquiries regarding assistance with improvements are welcomed by the IO evaluator.

Some SA Plans (or referenced documents) do not describe the overall SA cycle: identifying performance objectives, conducting assessment activities, analyzing/evaluating results, planning improvements, and using results to plan next year’s SA.

Some Plans do not include R2A2s for supporting and/or managing/coordinating the SA process.

## IO Evaluation of BNL SA ProgramsAreas for Improvement

- Processes for analyzing and evaluating the results of assessment activities are not clearly documented.
- Except for quantitative measures, bases for “rating” of performance are not documented.

**Notes:**
SA Plans do not contain a description of how results are analyzed/evaluated.  Many organizations use a management team approach for this process but often do not describe the process used.

Some organizations do not list ranges for rating performance. For “qualitative” evaluations, differences between “excellent” and “outstanding”, for example, are not explained.

## IO Evaluation of BNL SA ProgramsAreas for Improvement

- In some organizations, there is a need to improve:
- Tracking of assessment activities
- Tracking of corrective actions
- Records management

**Notes:**
Some organizations confused the tracking of corrective actions with tracking of assessment activities.

Prior to the issuance of “family-version” ATS, organizations used a variety of tracking “systems” some computerized but many not.  It was not always clear that personnel responsible for actions were always aware of their responsibility or scheduled due date.

Records resulting from assessment activities were not always managed through a defined process.

## IO Evaluation of BNL SA ProgramsAreas for Improvement

*What about improvements in IAP?*
- Recommendations for both specific and programmatic improvements have been communicated and included in:
- Integrated Assessment subject area
- IO Evaluation Criteria

**Notes:**
“Findings” regarding IAP programmatic issues were documented in each evaluation report and communicated to the IAP Administrator.

The SBMS subject area was issued in Sept. 2000.

FY01 criteria were developed and reviewed incorporating the two years of BNL experience.

The FY01 Evaluation Plan was sent to all Level 0, Level 1 and Level 2 Managers on November 20, 2000.

## IO Evaluation of BNL SA ProgramsFY01 Evaluations