![](media/image1.png){width="6.439583333333333in"
height="7.991666666666666in"}

## Scope

The Independent Oversight (IO) Office performs an annual review of
self-assessment programs of selected BNL organizations (Offices,
Departments, Divisions, and Directorates). IO conducts these reviews in
accordance with requirements and guidelines provided in the Management
System Description for BNL's Integrated Assessment Program (IAP), the
Integrated Assessment Subject Area, and the IO Office Procedure for
Conducting Independent Oversight Assessments.

## 

## Purpose

The purpose of the annual review is to independently verify the
effectiveness, efficiency, and adequacy of self-assessment programs for
organizations and, thereby, to support evaluation of the effectiveness,
efficiency, and adequacy of the Laboratory's Self-Assessment Program as
a whole. The reviews also support the continuous improvement of
self-assessment programs throughout the laboratory. Results of
individual reviews will be documented and communicated to responsible
organizational management, the IAP Management System Steward and
Point-of-Contact, the Lab Director, the Deputy Lab Directors, the
Manager of the Internal Audit Office, the Manager of the Quality
Programs and Services Office, and the Price-Anderson Amendments Act
(PAAA) Coordinator.

## 

## Methodology

The IO Office reviews the performance of the assessed organization using
identified criteria grouped into three categories (dimensions):

-   Approach: Refers to how effectively the assessed organization
    addresses the planning of assessments.

-   Deployment: Refers to the extent to which self-assessment plans are
    executed and used by all parts of the assessed organization.

-   Results: Refers to the outcomes of the organization's assessment
    activities and how they are fed back into operations and planning
    processes to sustain or improve performance.

The IO Office performs document reviews and conducts interviews of
selected organizational personnel in order to review performance using
specified criteria. Specifically, the IO Office will review
implementation of the self-assessment process in relation to the
assessed organization's Self-Assessment (SA) Program Plan \[Plan\].

The IO Office will then identify and document strengths and areas for
improvement in the assessed organization's Self-Assessment Program and
recommend improvements to responsible management.

## 

## Criteria

The following criteria constitute the basis on which IO will identify
strengths and areas for improvement. The criteria are derived primarily
from the requirements in the IAP Management System Description and the
Integrated Assessment Subject Area. Criteria have also been modified in
consideration of experience gained during previous reviews.

**Criterion 1.0 -- Approach**

1.  The scopes of planned assessment activities are comprehensive, have
    a strong technical basis, and are balanced with work activities.

-   Assessment activities are prioritized based on alignment with BNL
    and organizational strategic plans, results of previous assessments,
    business and operational risk, and available resources.

-   Plan design is based on the IAP (Baldrige) framework or alternative
    framework that ensures a comprehensive perspective of organizational
    performance such as the Balanced Score Card or the Laboratory
    Critical Outcome tree.

-   The Plan is comprehensive in that it covers the full scope of the
    organization's research, operational, business, and financial
    activities.

-   The Plan evidences appropriate attention to customer
    requirements/expectations.

-   The effectiveness of previously completed corrective actions is
    appropriately considered during planning.

-   Results from external assessments are incorporated into the
    organization's self-assessment and annual self-evaluation process.

-   External lessons learned are incorporated into the organization's
    self-assessment process.

    2.  The methods for conducting key scheduled assessments are defined
        and are commensurate with types of assessments planned and
        performance information desired.

```{=html}
<!-- -->
```
-   Assessment activities are assigned to qualified personnel including
    appropriate use of subject matter experts.

-   Managers effectively communicate their expectations such that all
    employees understand their roles and responsibilities for
    self-assessment.

-   Managers ensure that independent assessments and peer reviews are
    considered and incorporated into the self-assessment process, as
    appropriate.

-   The assessment approach uses an appropriate mix of information
    sources including performance observation.

-   The assessment approach minimizes the impact to work activities.

3.  Management and staff involvement commensurate with their
    responsibilities is evident.

-   Planning involves managers and staff members as appropriate to
    ensure that all aspects of the organization's operations are
    evaluated.

-   Organizational managers review and approve Self-Assessment Plans
    according to BNL's required schedule.

    4.  Key supporting organizational processes (tracking/trending
        systems, causal analysis, critiques, etc.) and tools are
        developed.

-   The organization has an effective process for analyzing and trending
    the results of assessment activities.

-   The organization has an effective process for tracking corrective,
    preventive, and improvement actions to completion.

-   Records are maintained that demonstrate effective planning, timely
    completion of assessments, and analysis of results.

-   Available information systems are used to effectively support
    assessment activities.

5.  Organizational plans/procedures address regulatory and/or SBMS
    drivers for assessment activities.

    -   Assessment activities are based on and linked to the
        organization's performance objectives and measures as well as
        the BNL Critical Outcomes and associated Performance
        Objectives/Measures.

    -   All applicable "Required Assessments" are identified in the Plan
        demonstrating appropriate consideration of integration of SBMS
        Management System requirements.

**Criterion 2.0 -- Deployment**

1.  Assessments are completed as scheduled. Planned assessment
    activities have been revised as appropriate based on new or changing
    information.

-   Assessment activities are completed as scheduled.

-   The SA Plan is updated to reflect changes in scope.

-   Scope and schedule of planned assessments are adjusted when
    appropriate based on significant performance information.

-   Customer feedback is solicited as part of the organization's
    assessment activities.

    2.  Assessments are documented and communicated as planned.

```{=html}
<!-- -->
```
-   Assessment activities are documented and results are provided to
    responsible managers.

-   The assessment schedule is managed to ensure information is
    available to support timely management decisions.

    3.  Assessment results (internal and external) are
        evaluated/analyzed to a degree commensurate with the type of
        assessment. Strengths are identified. Corrective/improvement
        actions are identified, prioritized, assigned to specific
        owners, and assigned due dates. All conditions are tracked to
        closure. Actions are validated as improving performance.

```{=html}
<!-- -->
```
-   The organization has an effective process for identifying
    opportunities for improvement.

-   Improvements are prioritized in accordance with business and
    operational risks and cost benefit.

-   Improvement actions are developed with consideration of causal
    factors as commensurate with the level of hazard/risk.

-   Change control for action due dates is timely and clearly reflects
    consideration to balance priorities.

-   Improvements are validated, as commensurate with the level of
    hazard/risk, to ensure desired outcomes are achieved including
    prevention of recurrence of adverse events.

    4.  Evidence of timely self-identification of issues exists.
        Significant issues are brought to the attention of management
        and disclosed to regulatory/oversight agencies in accordance
        with contractual obligations.

```{=html}
<!-- -->
```
-   Results are analyzed to identify potential noncompliances with
    nuclear safety rules (Price-Anderson Amendments Act).

-   Results are analyzed to determine potential impacts on SBMS
    documents.

```{=html}
<!-- -->
```
-   Results are appropriately considered for publication in the BNL
    Lessons Learned Program.

    5.  Management involvement is evident.

```{=html}
<!-- -->
```
-   Managers participate in planning and conducting assessment
    activities.

-   Managers participate in observing work practices.

```{=html}
<!-- -->
```
-   Managers participate in the analysis and evaluation of assessment
    results.

-   Managers verify that needed improvements are implemented.

-   Managers validate the effectiveness of improvements.

**\
**

**Criterion 3.0 -- Results**

1.  Sustained excellence and/or improved operational performance are
    evident for key areas of Laboratory operations, such as mission
    achievement and retention/expansion of core competencies.

-   Progress towards achievement of near-term and long-term goals and
    objectives is evident.

```{=html}
<!-- -->
```
-   Conduct of operations/research has been enhanced.

-   Evidence exists that customers and other stakeholders value the
    products and services of the organization.

```{=html}
<!-- -->
```
-   Organizations can provide specific examples of improvements
    resulting from the self-assessment process including improvements to
    the self-assessment process itself.

2.  Evidence exists that there is an appropriate connection between
    results of organizational self-evaluation and development of
    strategic/institutional plans.

-   Organizational strategic plans are modified as necessary based on
    evaluation of assessment results and evidence consideration of DOE
    Strategic Plans and the BNL Institutional Plan.

-   Improvement actions are either acted upon at the
    organization/Directorate level or communicated to BNL senior
    management to be included in the Laboratory's improvement agenda, as
    appropriate.

## Review Process and Schedule

IO reviews will be formally scheduled after consultation with management
of affected BNL organizations. The process begins with a review of
relevant documentation (e.g., SA Plans, self-evaluation reports,
assessment procedures, organizational performance measures)
approximately one week prior to conducting an in-briefing with the
management of the reviewed organization. The detailed schedule and scope
for the IO review will be discussed at this briefing. Interviews of
management and staff from the organization will be concluded during the
two weeks following the briefing. The IO Office will then analyze the
data gathered during the review and prepare a draft report of results.
An out-briefing will be held with the organization's management to
summarize review results and to submit the draft report to management
for factual accuracy review and comment. After addressing feedback from
management, the report will be finalized and distributed as discussed
previously.

The following schedule lists the organizations and a proposed
in-briefing date for FY 2004. An updated schedule including the status
of each review will be available at <http://www.io.bnl.gov/sched04.htm>

Self-Assessment Program Review Schedule -- FY 2004

+---------------------------------------+------------------------------+
| # **Organization**                    | # **In-Briefing Date**       |
+---------------------------------------+------------------------------+
|                                       |                              |
+---------------------------------------+------------------------------+
| Life Sciences Directorate             | February 9, 2004             |
+---------------------------------------+------------------------------+
| Energy, Environment, and National     | April 5, 2004                |
| Security Directorate                  |                              |
+---------------------------------------+------------------------------+
| Environment, Safety, Health, and      | July 12, 2004                |
| Quality Directorate                   |                              |
+---------------------------------------+------------------------------+